Group 6 - Augmented Reality

Augmented reality (AR) merges data from the real environment with virtual information. In the past, surgeons primarily used preoperative imaging, which could be classified as virtual reality technology. In these, the surgical navigation could be planned, leading to a virtual navigation system (cp. section Image Guided Surgery in article Imaging). But it is not ergonomic to switch between virtual navigation and real patient.

As minimally invasive procedures are increasingly demanded, surgeons are required to work within the smallest possible access to the brain. This leads to a small workspace, in which the surgeon has to concentrate not to hurt blood vessels or neural structures. Augmented reality addresses this situation and allows and ergonomic, clearly arranged workspace and should be an enabling technology regarding more minimally invasive procedures.

Currently, no randomized studies on the usage of devices equipped with augmented reality have been published yet, though. [1]

Surgical devices already equipped with augmented reality

Below three surgical devices that are already equipped with augmented reality are presented.

Microscope-assisted guided interventions

Microscope-assisted guided interventions (MAGI) require skull-fixed fiducial landmarks and/or dental stents. Non-rigid registration methods may change this, though (cp. section Image Registration/Image Fusion in article Imaging). 3D projections from preoperative surgical images are overlayed with the optics of the operating microscope (cp. fig. 1 for the principle of usage of an operating microscope with augmented reality). MAGI represents a coherent fusion between real images and virtual images by monitoring microscope optics pose, focus, zoom and internal camera parameters. Current microscopes equipped with augmented reality feature automatic alignment with the surgical field.

MAGI has two disadvantages, though: during the macroscopic part of a brain surgery (i.e. craniotomy) it is not useful and depth perception is deranged because of the monoscopic microscope perception and the projected virtual image. [1]

Figure 1: usage of an operating microscope during Image Guided Surgery [2]

Cameras

Cameras can be used to record the surgical workspace and display the real patient with additional information on a separate screen. One device that is currently used is the Dex-Ray: a lipstick-shaped video camera, mounted on a hand-held-pointer. The projection is in 3D on the monoscopic perception through the operating microscope. Compare figure 3: The big screen displays the combination of the video stream and 3D graphics, whereas the upper windows show the axial, coronal and sagittal planes intersecting at the tip of the camera pointer. [5]

The are two main advantages of this system compared to MAGI. The surgeon is aware of the spatical distance between the camera and the surgical workspace and it is easy to change usage modes between the macroscopic and the microscopic part of the operation. But there are disadvantages, too: in narrow corridors, the camera has limited sight and does not receive enough light, the surgeon’s point of view is different than that form the camera and the surgeon has to look away from the patient.

There exist other set-ups that partially address the disadvantages of the two different points of view: sometimes the camera is positioned either on the surgeon's hand or on the surgeon's head. [1]

Figure 2: Intraoperative evaluation of DEX-Ray [5]

Direct patient view with or without the interposition of a semitransparent mirror

Semitransparent mirrors are currently only used in oral surgery. The mirror is positioned at 45° in front of a light field display, the technology enables a full-parallax view of the virtual image. The advantage is that no additional display is needed, but there are nor implementations for operating microscopes of neural surgery yet. [1]

Figure 3: direct patient view with or without the interposition of a semitransparent mirror [3]

Augmented reality based on iMRI

Intraoperational MRI is a good method to measure and manage brain shift during surgery, enabling better tumor removal (cp. section Research on intraoperative MRI, article Imaging). One important task to make iMRI work is the application of real-time capable non-rigid registration algorithms. It was shown that this is feasible with today’s technology and that there is a significant improvement in alignment accuracy with these new algorithms. [6] Therefore it is likely that non-rigid registration augmented reality applications are going to be developed in the future.

Virtual reality and augmented reality for surgical training

Whereas augmented reality improves the surgical procedures, virtual reality (VR) enables medical students to assume the role of a surgeon and take over their perspective in an operation. With virtual reality, medical students can memorize procedural techniques, train operation room communication and become familiar with the entire operating room environment. Main advantages are the best point of view for learning, rather than a backseat in the operating room, and a big amount of people that can benefit from the same learning material. 360° videos that can be experienced with devices like the Oculus Rift are one example for VR learning material. [4]