The typical way of learning neurosurgery is through observation of an experienced surgeon in an OR over the course of several hundred operations performed. This approach is increasingly being hindered by legal and ethical concerns for patient safety, time requirements on experienced surgeons and the cost of OR time. [1]
Luckily, in recent years the field of VR and AR has been advancing quickly and we will discuss several promising neurosurgical simulators capable of imparting surgical and procedural skills on future surgeons. These methods allow for surgical skill acquisition without the enormous logistical, ethical, legal and financial requirements needed for hands-on training of neurosurgery.
Another use for surgical simulators is the training and rehearsal of a concrete operation for a concrete patient, where an already experienced surgeon will rehearse the approach for an operation beforehand to minimize risks for the patient in the actual surgery.
Types of Simulators
There are 3 main types of surgical simulators, these are physical models, virtual reality and mixed-reality simulators. [2]
Physical models have been used in surgery training for centuries and consist of either human or animal cadavers, living animals, or artificial models made from wood, metal, plastic or other materials.
They come with severe applicability limitations, as cadavers and artificial models do not represent the physiology, such as bleeding, of a working organism well. Animals have different anatomy from humans, so the skills acquired on animal surgery are not transferable to human operations. The costs of human and animal cadavers and moral concerns also complicate practice on these models.
Virtual reality simulators usually combine a visual aspect provided by a pair of stereoscopic goggles and haptic feedback to immerse the user into a fully virtual world. The main difficulty of these systems is the creation of a realistic environment that "looks and feels" right and accurately represents the underlying structures. Virtual simulators may or may not use physical models to assist in generating haptic feedback.
Mixed-reality simulator combine a virtual and real aspects into a whole that shows the real world with additional data overlaid to help understand the composition of the anatomy better or highlight the important structures. The main difficulty with these systems is real-to-virtual image registration, lag and contrast issues. You may visit our page on AR/VR to see the comparison.
Simulator Design
Simulators must fulfill certain criteria to be useful as a tool for surgery training or planning. These criteria are: [2]
- Validity - being a truthful representation of reality
- Face validity - if the simulator "looks and feels" like the real thing to an experienced surgeon - no obvious errors at first glance
- Content validity - if the simulator actually represents the real world in the physical sense - for example the hardness of bones, the stretch mechanics of soft tissue etc.
- Construct validity - "the degree to which a test measures what it claims, or purports, to be measuring" - for example, the assessment is able to discern experienced and beginner surgeons
- Criterion validity - how precisely different criteria are weighted and if their weights represent their real-world importance
- Reliability - whether same results are observed under consistent conditions (the simulator not having random factors)
Virtual Simulator Components
A VR simulator is composed of 3 main parts, all of these parts are dependent both on software implementation and capabilities provided and limitations imposed by the hardware. [4]
Today's simulators are far from perfect, as both correct and precise software implementation is practically impossible given our limited understanding of brain as a whole and hardware limitations not allowing us to simulate a neural network with ~100 billion neurons to see the outcomes of virtual operations at a precise level.
Graphics / Volume creation
Arguably the most developed aspect of surgical simulators is the graphical rendering part. This can be explained by the fact that computer-generated graphics and 3D computer games are a huge industry that is moving forward very quickly and a huge number of experienced graphics programmers is available on the labor market.
Using game engines as VR graphics/physics model
There has been some research into using game engines as physics/graphics basis for clinical simulators in the past [12]. This could have the effect that using advanced and very realistic rendering engines would be very easy (compared to other aspects of simulators) today and their realism and optimization is incomparable to graphics used barely a decade ago. It allows for realistic graphics to run even in VR stereoscopic, real-time mode on consumer-grade hardware. A great example of an engine that could be used in the most modern simulators and will probably see use in the future of VR surgery simulation is the Unreal Engine 4.[5]
UE is originally a game engine, but as VR trainers in all practical aspects are computer games (albeit a bit more serious ones) it is perfect for quick development of very realistic-looking simulators - it already has VR support for HMDs from the get-go, has great performance optimization, realistic graphics and a very active community of developers. It also has a built-in physics system, meaning that instead of a team having to spent years implementing graphics and physics simulation, they can learn UE4 in mere months and be able to develop on a completely different level. [6]
The downside of using an external solution for graphics/physics might be that if the team is not intimately familiar with the implementation of the engine, they might encounter unexpected and complicatedly debuggable issues.
See examples of laparoscopic simulator created by a very small team in UE4. [7]
Model volume creation
One approach to creating the volume of the virtual model is using actual patient data, in which case the simulator might be more suitable for surgery preparation and rehearsal, if the data is created from an atlas or a generalized model of the brain, then the simulation is suitable for training of young residents.
In the article [4], the two opposing approaches to creating the volumes rendered is described.
- Direct creation - takes MRI or other data and creates a true 3D representation based on intensity differences including all the inner parts, creating a true representation of the data
- Indirect (Surface) creation - only creates a model of the surface of objects - the various structures of the brain are segmented apart and simulated as filled surfaces with homogeneous stiffness and flex
The difference in performance and veracity of each approach can be easily deduced - direct creation is much more representative of the actual organ at the cost of performance requirements that are much greater. Indirect creation is more easily computed, but the accuracy of simulation may be compromised.
Side note - in the same article, they use the term rendering to refer to the creation of underlying model for simulation. This may lead to confusion, as in computer graphics this term is very well defined and means transforming a 3D scene to 2D pixel field displayed on a monitor, not having anything to do with the underlying model, we use the word creation instead, as rendering is the part that follows after the model is calculated and is to be displayed, not before or throughout the creation. [8]
Model response/tissue deformation
After the model is created, the rules of physics governing the behavior must be implemented. Volumetric models must respond to user manipulation with virtual tissue deformation. Again in [4], two main approaches are described
- Mass-spring method - describes each voxel as a mass linked to adjacent points by a spring - the characteristic of the model is the mass of points and the springiness of attached spring - the downside is its inability to accurately depict a surgical cut
- Finite-element method - simulates the structure as a grid of connected elements, with each element having simulated real-world characteristics such as hardness, stiffness, weight etc. and approximating natural laws such as effects of gravity for each element in every step of the simulation - better represents tissue bio-mechanics at the cost of increased performance
As mentioned in the previous section, graphical engines often incorporate different types of physically simulated objects, which can be readily used if their properties are found to be acceptably close to reality. The advantage is again - good optimization, usage of graphical cards for physics calculation (which increases computational speed by an order of magnitude compared to CPU physics calculation[9]) and a large community which means more support available than with in-house solutions. [10]
An advantage for modeling tissue deformation of the brain is that brain matter is fairly homogeneous and thus computation can be done fairly efficiently.
Haptics
Haptic feedback is the collective name for non-visual and non-auditory feedback given back to the trainee. The most important part of this feedback is the tool resistance to movement, which is crucial for an immersive surgical simulator, as the forces needed to be applied on the tissue are a critical part of an operation. The haptic feedback is arguably the least developed area of VR simulators, which could again be explained by a fairly narrow field of use in the past, but it can be expected to improve drastically as VR systems become more wide-spread and their audiovisual aspect will be perfected. The haptic feedback is generated by calculating collisions between the virtual representation of real tracked tools and virtual structures in the model.[1]
The hardware for haptic feedback nowadays is the most limiting factor for VR simulator development, as the hardware is very specialized and thus expensive[4]. Also, precise haptic feedback imposes severe computational burdens on the simulator, as the interaction between virtual tissue and virtual tool must be calculated at a very high frequency to provide adequate feedback quality.[4]
Advantage of neurosurgery in context of haptic feedback generation is that minimally invasive operations are fairly prevalent and these laparoscopic procedures can be simulated with an entry-port-mounted tool, which has less degrees of freedom than a free-hand tool. This allows for decreased computational intensity[4].
Recent Devices
In this section, we introduce and describe some modern devices used for either pre-op rehearsal or training in neurosurgery. These devices are usually composed of a stereoscopic display system, a haptic interface hardware and a high-performance computer. The software used can vary on each device as the devices may be used to train/rehearse different surgical actions.
NeuroTouch Endo/Cranio
Released in 2009 in Canada, the NeuroTouch is arguably the most advanced surgical VR simulator of today. It is composed of a 2 monitors which are viewed through a stereoscopic angled mirror set and a 6DOF haptic-feedback bimanual tool handles.
The NeuroTouch simulation software can emulate the tissue mechanics process. Tissue deforms to account for blood accumulation and local effects from surgical tools. Dissection of vascularized tissue triggers bleeding. The amount of bleeding depends on vicinity of large blood vessels. Tissue pulsates as if under effect of 60 bpm heart-beat, blood exits the tissue and then acts according to fluid dynamics - flowing down with gravity and accumulating in pools. [12]
Graphics runs at smooth 60 FPS, haptics at 1000 Hz and model simulation at 100 Hz.
The software is able to simulate multiple tool-sets and scenarios, depending on the selected suite. Of interest to our applications are the Endo and Cranio suites.
In the Cranio version, intended for simulation of brain tumor resection, the handles may simulate tools including the aspirator, cavitron ultrasonic surgical aspirator, bipolar, and micro scissors[[2]]. Training modules for the cranial portion are based on brain tumor patients and measure performance metrics providing an objective assessment of technical skills. [11] The Cranio version includes tumor debulking, tumor cauterization and craniotomy training. [12]
Tumor debulking is a task was built upon images of a patient with a left frontal meningioma. The goal of the simulation is to remove the tumor completely, while removing as little healthy tissue as possible. The simulator evaluates metrics of the trainee performance - percentage of tumor volume removed, total healthy tissue volume removed and time to complete. [12]
Tumor cauterization is a task built from images of a patient with a left frontal oligoastrocytoma that was soft and vascular. The goal of the simulation is to remove as much tumor tissue as possible while minimizing blood loss. The simulator evaluates only 2 metrics - the volume of blood and the time to complete. [12]
The Endo version adds a virtual 4mm drill tool and a software suite for transsphenoidal neurosurgery. This simulator has structured levels of difficulty by varying the size of the nasal passages. It simulates endoscopic view including lens distortion effects, blurring, and tissue deformation. The endoscopic portion includes modules for endoscopic third ventriculostomy and navigation in the nasal cavity with performance metrics. The ventriculostomy is performed when a tumor increases the CSF pressure in the brain to dangerous levels, to drain the CSF. [[11]] The Endo version may be simplified not to contain 2 monitors and a mirror set, as the simulated output from an endoscope may be viewed on a single screen without loss of information. [13]
Scoring for ventriculostomy performance using these VR–simulators is typically measured by entry point for the burr hole, catheter trajectory, length of catheter inserted, and time to complete procedure. [11]
In their article [14], Thawani et al. show (albeit with a small sample size), that using NeuroTouch Endo suite greatly increased performance in subsequent simulated surgeries, but more importantly, considerably increased the quality of real-world operations performed by the trained group against an untrained group, proving that Endo suite skills are transferable to real-world scenarios.
ImmersiveTouch
Released in 2009, the ImmersiveTouch VR simulator consists of a semitransparent mirror, a monitor, active stereoscopic goggles and dual 6DOF haptic-feedback bimanual tool handles.
The usage of a mirror and goggles allows for head movements not possible in the NeuroTouch device. Again, the software is able to simulate multiple scenarios relevant to neurosurgery, such as ventriculostomy.
Banerjee et al. demonstrate in their article [14], that trainees improve their skills in placing a catheter closer to the target location after repeated simulation attempts in patients with pathological ventricular cavities. This suggests that use of the ImmersiveTouch system leads to an increase in understanding of the abnormal ventricular anatomy. Real-world performance was not measured, so they conclude this method to be an effective first step before attempting real surgery.
Future Development
With VR headsets such as Occulus Rift and HTC Vive becoming more and more widespread, the field of medical simulators will probably move to these devices instead of stationary displays with mirrors, as seen in NeuroTouch and ImmersiveTouch, as commercial HMDs will provide better immersion at a lower cost.
As graphics in games and simulations keep improving, the visual aspect of simulators will improve too, with very realistic graphics being already possible today. Increased computing power of future hardware will also allow for more complex model properties to be implemented while retaining simulation fluidity.
Better understanding of the brain and the capability to simulate the effects of brain segments will allow for more precise performance metrics. These could include the assessment of damage to patient's vessels and functional areas, with different areas being given different weights.
Finally, haptic feedback systems will improve and presumably decrease in price, further limiting the hardware limitations imposed on developing neurosurgery simulators.
Example
CAE Healthcare NeuroVR
Bibliography
1) Alaraj A et al. (2011) Virtual reality training in neurosurgery: Review of current status and future applications, Surgical Neurology International. 2:52. doi:10.4103/2152-7806.80117
2) Rehder R. et al. (2015) The role of simulation in neurosurgery,
3) Dr. Wong, University of Hongkong, using the NeuroTouch - credit http://www.neurosurgical.tv/neurosurgical-simulation-training-center-using-virtual-realitly/
4) Malone H.R. et al. (2011) Simulation in neurosurgery: A review of computer-based simulation environments and their surgical applications,
5) Documentation of Unreal Engine 4 features, (C) Epic Games, https://docs.unrealengine.com/latest/INT/Engine/index.html (access 20/6/17)
6) Pros and cons of UE4 compared to other engines, http://staraban.com/en/undeal-engine-4-overview-features-cons-and-pros/ (access 20/6/17)
7) Youtube channel of Edward Quian, all videos from https://www.youtube.com/channel/UCVq5O7gpNSCOSmKXYH3fhiA - homepage at https://www.kunqiancg.com/ (access 20/6/17)
8) Rendering (computer graphics), https://en.wikipedia.org/wiki/Rendering_(computer_graphics) (access 20/6/17)
9) PhysX FAQ - Why physics is a good problem to solve on GPUs - (C) NVidia, http://www.nvidia.com/object/physx_faq.html (access 20/6/17)
10) Documentation of Unreal Engine 4 basic Physics object, (C) Epic Games, https://docs.unrealengine.com/latest/INT/Engine/Physics/PhysicsBodies/Reference/index.html (accessed 20/6/17)
11) Paramita Das, et al. (2014) Simulation Training in Neurological Surgery, Austin Neurosurg Open Access. 1(1): 1004.
12) Marks S. et al., Evaluation of Game Engines for Simulated Clinical Training
13) Delorme S. et al. (2012) NeuroTouch: A Physics-Based Virtual Simulator for Cranial Microneurosurgery Training, Congress of Neurological Surgeons
14) Resident simulation training in endoscopic endonasal surgery utilizing haptic feedback technology, Journal of Clinical Neuroscience, Volume 34, Pages 112-116
15) Simulation in Neurosurgery: A Review of Computer-Based Simulation Environments and Their Surgical Applications,
CAE NeuroVR™ Neurosurgical Training Simulator, https://caehealthcare.com/surgical-simulation/neurovr (access 14/06/17)