Augmented reality aims to improve our perception of reality with useful data which is not visible to the naked eye.

The most prominent use-case of augmented reality in relation to brain tumors is in-operation visualization of brain areas in 3D from previously acquired patient-specific scans. This allows for minimally invasive operations to take place by providing the surgeon with exact position data about the tumor in 3D.

This can be achieved in several ways. We discuss projector-based AR, monitor-based AR and also head-mounted displays.

All augmented reality techniques have four steps, each technique combines different methods for each step:

1. virtual image creation from pre-operative data - MRI, CT, PET data converted to 3D models
2. real environment acquisition - hand-held cameras, head-mounted cameras, depth-cameras
3. registration - placed markers, feature-based registration
4. projection - projecting onto the patient, display on a monitor, display on a video see-through or optical see-through head mounted display

Projector-Based AR

Projector-based artificial reality works by taking preoperative data, registering the position of the patient (by placed fiducial markers) and then projecting the registered data into the real-world. In brain tumor surgery, this can be used to plan the skin incision and the craniotomy and to visualize the tumor borders on the brain surface after breaching the dura. The method also helps by visualizing the position of inner brain structures (e.g. veins) on the surface of the brain. [1]

The advantages of this technique are the low price of commercially available projectors, quick registration of pre-op data to intraoperative markers and no hindrance of the surgeon by HMDs or by having to look away at a monitor - all data is directly on the patient.

Of course, the disadvantage is that data can only be presented on the surface, so depth information may be lost. Monitor-based AR and video see-through HDMs can negate this by providing virtual transparency. 

Optical and Video See-through Head-Mounted displays

The two different approaches to overlaying information over the real world with head-mounted displays are Optical See-Through and Video See-Through. Both of these have their advantages and disadvantages.

Optical see-through works by letting the user see the real world through semi-transparent mirrors and the visualized data is overlayed over the real world.

Video see-through, on the other hand, completely isolates the user from the real world and a collection of cameras captures the real world from the user's perspective. This camera data is then fused with additional data to be visualized on LCDs located directly in front of the user's eyes.

Video see-through head-mounted displays applications

For neurosurgical applications, the video see-through approach appears to be  the currently more wide-spread, as it allows for more extensive image manipulation and the downsides of video see-through are not so limiting for neurosurgical applications.[[3]]

There are several ways to improve the quality of fusing real and virtual objects by different shading methods and stereoscopic and kinetic cues, resulting in very good perception of overlayed data by the surgeon.[5]

Trajectory planning and AR surgery simulation

As mentioned in their article, Navab et. al. [6] use advanced methods to propose several different trajectories for minimally invasive "keyhole" neurosurgery. This takes into account 

 A surgeon is later equipped with a video see-through HMD which shows possible trajectories and entry points on a real-world head phantom. The surgeon can then assess the viability of each possible trajectory, taking into account the ease-of-access for given patient position, instrument placement uncertainty and possible damage to specified risk areas such as blood vessels and ventricles.

Training for planning a tumor resection

In their article, Abhari et al. [7] show, that optical see-through HMDs are a viable training method for training novice neurosurgeons the correct placements of incision points and angles for tumor resection. They conclude that skills learned in augmented reality environment can be transferred into real-world scenarios and that AR-driven approach to neuronavigation is at least as good as any other tested modality.

Bibliography

1) Tabrizi L.B. et al. (2015) Augmented reality–guided neurosurgery: accuracy and intraoperative application of an image projection technique, J Neurosurg.123(1):206-11. DOI: 10.3171/2014.9.JNS141001, http://thejns.org/doi/pdf/10.3171/2014.9.JNS141001 (access 06/06/17)