Apple Vision Pro Finds New Purpose in Operating Rooms

Apple's augmented reality (AR) headset, the Vision Pro, initially met with market failure but is now being explored for a surprising new application: operating rooms. Researchers at Sheba Medical Center are investigating how the device can enable surgeons to visualize patient anatomy in 3D, including tumors and internal structures, to enhance surgical planning and precision. Concurrently, another development based on the headset has received unprecedented approval from the U.S. Food and Drug Administration (FDA).

Dr. Eyal Mor Hader, an oncological surgeon specializing in sarcoma and melanoma at Sheba, is leading a study where he uses the Apple Vision Pro during complex surgeries to remove large tumors. The headset allows him to view the tumor and surrounding anatomical structures from various angles, overcoming the challenge of distorted anatomy caused by the tumor. "One of the things we see in these complex surgeries is that the tumor actually distorts the normal anatomy," Dr. Hader explained. "It presents a great challenge for the surgeon to identify which organs we want to preserve and which areas we want to remove."

Traditionally, surgeons rely on 2D CT scans, requiring their brains to translate the images into a 3D understanding. The AR headset offers a direct 3D visualization that mimics reality. Dr. Hader noted that ongoing research aims to project models during surgery, providing surgeons with maximum information and accuracy regarding critical structures. Currently, these applications are still in the research phase, with surgeries not yet performed while the surgeon wears the headset.

In another department at Sheba, focusing on head, neck, and jaw surgeries, a separate development is underway. This project aims to address the limitation of surgeons needing to divert their gaze from the patient to consult medical imaging and plans. The goal is to create an alert system, akin to navigation apps warning of traffic, to assist surgeons. "We have reached a situation where we know how to take imaging, CT, MRI, ultrasound, and create good, beautiful 3D renderings, but their clinical use is limited," a doctor involved stated. "In practice, we enter the operating room, and our computerized 3D plans are left on the wall on A4 pages, usually printed in black and white. We constantly have to disconnect from the surgical field to look at them."

The new system being developed will allow surgeons to view patient imaging data in real-time without looking away. Through the headset, they can see both the patient and projected imaging layers, such as orbital cavities, nasal passages, the skull base, and planned jawbone sections. "I have the ability to understand and orient myself optimally in space, which did not exist until today," a doctor explained. This development is part of research initiatives, including collaboration with Technion researchers, with hopes of transforming the surgical world by providing real-time feedback and precision.

In parallel to the Sheba research, Stryker Corporation's development using Apple's headset has received FDA clearance for arthroscopic hip surgery. This system is the first to be officially approved for use with the Vision Pro. It addresses the need for surgeons to look away from the patient to view camera feeds on monitors during the procedure. Dr. Hader likened these advancements to the evolution of navigation systems, moving from paper maps to apps like Waze and Google Maps, suggesting surgeons now need similar tools. Future iterations may include systems that alert surgeons to sensitive areas during operations, similar to how navigation apps warn drivers of hazards.