Mixed Reality And Neurosurgery: Removing Brain Tumors With 3D Precision

The human brain is staggeringly complex and fragile. So, removing a tumor from one is among modern medicine’s trickiest procedures. Some tumors grow deep inside gray matter and can be difficult to see and get at. Others can spread out awkwardly in tentacle-like projections, making the need for precision surgery paramount.

Now, an academic research team from Singapore’s National University Health System (NUHS) is exploring how digitally assisted procedures could one day give neurosurgeons an enhanced and augmented 3D view of a patient’s brain by using the mixed-reality capabilities of Microsoft HoloLens 2.

The project is part of NUHS’s wider Holomedicine Program, which is looking at how HoloLens 2 might be applied in a range of future clinical situations. As such, it is adding to the work of physicians in a number of other countries who are already using mixed reality technology in a growing list of ways.

The Singapore Government’s Health Sciences Authority (HSA) has granted a Class A License and Singapore Standard SS620 certification for the solution’s software – developed by apoQlar, a German company specializing in holomedicine technologies.

The NUHS team is now conducting studies to compare the potential of HoloLens 2-assisted neurosurgery to current clinical practice in several NUHS operating theaters in Singapore.

Work on the project is still at the research and development stage and more regulatory approvals will be needed before the new technology can be used as a primary tool in operations with real patients.

HoloLens 2 works with hand gestures and voice commands, enabling a surgeon to view 3D holographic images of a patient’s anatomy created from scans taken earlier. These images are displayed on the visor of the headset and give the effect of being superimposed on the patient during an operation. Surgeons can move those virtual images around to see them from different angles. They can also use the HoloLens 2 to access patient data during surgery, call up videos or documents to help solve problems and contact other specialists for advice.

Right now, the NUHS research team wants neurosurgeons to experience and assess HoloLens 2 as they operate but not rely on it as a primary tool. Instead, the doctors must adhere to standard clinical practices and procedures. Currently, surgeons constantly refer to CT or MRI scans displayed on screens as they operate.

As they look up at the screens and then down at the patient, they are in effect mentally reconstructing a 2D image (a scan) onto a 3D object (the patient’s head). Even with modern intra-operative navigation devices, this is not an easy task as the navigation screen is always some distance away from the actual operating field.

READ: HoloLens project enables collaboration among surgeons worldwide

“You have to look at the screen. Then you look at the object and you go: ‘I think it’s here,’” explains Associate Professor Ngiam Kee Yuan, who oversees NUHS’s development and implementation of the Holomedicine Program and is also the Group Chief Technology Officer of NUHS.

The aim is for HoloLens 2 to replace this constant mental back-and-forth, he says. Instead, a 3D map of a brain – created from scans – is displayed onto a headset visor and appears superimposed over the surgeon’s real-life view of the patient in the form of an interactive hologram.

This precise augmented view effectively will enable the surgeon to “look” inside the patient’s head and through skull and brain tissue. The surgeon can “see” the tumor in situ and in detail as the operation proceeds.

It has the potential to be a highly precise guide that, until now, would have been the stuff of science fiction or a superhero movie.

“If a doctor had Iron Man abilities, that’s what you would expect them to see,” says Professor Ngiam.

In a surgical setting using HoloLens 2, the holographic image of the patient’s brain is digitally locked into position over the patient’s head. It appears to stay in place from any angle, even if the wearer walks around the operating theater. In fact, multiple surgeons wearing headsets can look at the same image at the same time, and the image should be accurate for all, providing the potential to enhance teamwork and coordination.

Dr. Gao Yujia, NUHS program lead for the Holomedicine Program, recalls an immediate “wow factor” when he first put on the HoloLens 2 headset.

“What I felt was that this is a device that may enable us to do a lot of things that we have wanted to do, but have been unable to, because we just don’t have the capability. It potentially provides us a platform to do research and development on capabilities that we’ve always wanted,” he says.

Dr. Gao and Professor Ngiam expect HoloLens 2 may lead to enhanced clinical outcomes and better patient safety and recovery. Apart from neurosurgery, this technology can also be applied to other types of surgery, such as complex spinal operations, liver surgery, facial reconstruction surgeries and many others. However, due to the complexity of some of these procedures, it may take a few years or additional research and development before we see a more widespread use of this technology in the operating theater.

READ the NUHS media release 

There have been significant advances in the last decade to help doctors navigate complex organs. But brain surgery remains extremely tricky, and outcomes depend on the skill of the surgeon. Using HoloLens 2, surgeons can also use a ‘slicer box’ that provides 2D views of relevant areas. It’s a little bit like looking out from inside a square fish tank at the relevant slice of the brain on the other side of the glass.

“It gives you a slice, so you can have a very detailed look during the operation of how deep the tumor itself is (and) of its 3D location. I believe that none of the existing guidance systems can match it,” says Professor Ngiam.

Successful brain surgery requires more than just removal of the tumor, he says. “The brain is a very sensitive organ and minimizing the damage to the surrounding tissue is equally important. And the HoloLens 2 may help to do that too.”

Professor Ngiam believes that once the HoloLens 2 is in widespread use, specialists are very likely to find new uses for it. Already, this has happened. When neurosurgeons at NUHS started testing the technology, one neurosurgeon found a way to integrate data from an international brain connection project. “This would not have happened if this technology was not available. Making this available has opened up this possibility,” he said.

The project identifies connections between specific locations in the brain and the body part it controls. Using this information, a neurosurgeon might be aware that a procedure may, for example, involve a higher risk of paralyzing someone’s fingers or affecting their speech.

Surgeons won’t be the only ones who get to try out HoloLens 2. It’s potentially useful for patients too. During a pre-surgery consultation, a patient could wear a headset and see exactly where the tumor is, and what risks surgery might pose.

“With more information and more understanding, it is easier for them to process the whole situation and come to an informed decision,” said Professor Ngiam.

It also helps families. The HoloLens 2 may offer a much more detailed understanding of the challenges the patient faces, and they may be far better equipped to make necessary decisions.

ALSO READ: HoloLens and house calls: Telehealth technology delivers virtual consultations

“This is particularly important for ensuring optimal care in a culture like Singapore’s, where relatives play a large part in the decision-making process,” says Professor Ngiam.

NUHS hopes to become an active developer to help find new uses for the technology, which can be shared internationally.

“We would work with our partners to make holomedicine customizable, more user-friendly and clinically applicable,” said Professor Ngiam.