Collaboration Leads to High-Tech Surgical Visualization Device at OU Health

A conversation that began at the launch party for the Oklahoma City Innovation District led to a novel collaboration between the OU Health Sciences Center and energy technology company Baker Hughes. The result, an advanced surgical visualization device now being used in craniofacial reconstruction surgeries, embodies the purpose of the district -- to convene people from Oklahoma’s diverse sectors to share their expertise and, together, create something new.

Representatives from Baker Hughes and the OU Health Sciences Center started talking that day in 2018 and soon began collaborating on a prototype device. The work leveraged augmented reality/mixed reality technology that Baker Hughes developed to create 3D reconstructions of rock specimens from computed tomography (CT) scans. Such visualization is highly valuable to geologists and oil and gas companies as they determine where to drill oil wells. That capability caught the attention of OU Health plastic and reconstructive surgeon Christian El Amm, M.D., who envisioned a headset/visor that he could wear during surgery that would allow him to both see his patient as well as 3D data that was superimposed on the patient, such as CT scans and reconstruction steps he prepared for the procedure.

“Our device represents the value of collaborative spaces – these types of innovations can only happen when you have an open forum where you can discuss and collaborate,” El Amm said. “Developing a device like this requires engineering skills, computer technology skills, and a profound knowledge of what you’re doing surgically.”

The Innovation District encompasses about 1.3 square miles just east of downtown Oklahoma City and includes the OU Health Sciences Center campus, the city’s biosciences sector and many other diverse companies. Katy Boren, CEO and President of the Innovation District, said the collaboration between the OU Health Sciences Center and Baker Hughes exemplifies the mission of the district.

“We convene and introduce people across industries who normally would not know each other, and we provide them an opportunity to advance their discussions,” she said. “From those conversations, new inventions are created, new companies are started, new patents are earned, and it grows jobs and our economy. That’s what we do every day – create interactions that allow innovations to spark and grow.”

The Baker Hughes/OU Health Sciences Center collaboration has indeed flourished. Baker Hughes had advanced the technology to achieve mixed reality, which allows the person wearing the headset to see what is around him, as well as 3D images that appear as if they were part of real life. The technology became a virtual field visit for the oil and gas industry, providing extensive information about rock, like how porous it was.

“There were obvious analogies to the healthcare sector,” said Jeff Potts, Advanced Analytics Leader for Baker Hughes. “We talked at length with Dr. El Amm and others from the OU Health Sciences Center and began a collaboration agreement. There were several key challenges that we needed to solve to move forward with a prototype that could be used in a surgical setting.”

During the year-long collaboration, the team created solutions for those challenges which, until that time, were unsolved. Using artificial intelligence techniques, they “taught” the device how to recognize features of a human face, which was necessary to be able to superimpose 3D models on the patient in real time. The team also enhanced the accuracy of that process in order to meet the precision needed for surgery, and they enabled the device to track the surgeon’s instruments in real time.

Since the official collaboration ended, El Amm has continued to develop the device, with significant input from Mohammad Abdul Mukit, Ph.D., a graduate fellow in electrical and computer engineering on the OU-Tulsa campus. Notably, they have programmed the device to respond to voice commands. That allows the surgeon to keep his focus on the patient rather than turning away to look at a computer screen or clicking a mouse. They also developed “markerless tracking,” which allows the CT scan or other images to be superimposed using artificial intelligence instead of cumbersome markers to guide the way. El Amm compares the headset to what a fighter pilot wears – both need ready access to a lot of complex information, especially if they must make a split-second decision.

El Amm has begun using the device during surgical cases to enhance the safety and efficiency of complex reconstructions. Many of his patients come to him for craniofacial reconstruction after a traumatic injury; others have congenital deformities. Thus far, he has used the device for several cases, including surgery on a patient who was born without a right ear. The system took a mirror image of the patient’s left ear, then the device overlaid it on the right side, allowing El Amm to precisely attach a reconstructed ear. In the past, he would cut a template of the ear and aim for precision using the naked eye.

In another surgical case, which required an 18-step reconstruction of the face, the device overlaid the patient’s CT scan on top of his real bones. “Each one of those bones needed to be cut and moved in a precise direction,” he said. “The device allowed us to see the bones individually, then it displayed each of the cuts and each of the movements, which allowed the surgeon to verify that he had gone through all those steps. It’s basically walking through the steps of surgery in virtual reality.”

The OU Office of Technology Commercialization is pursuing intellectual property protection for the invention. In addition, El Amm has begun partnering with his colleagues in neurosurgery, orthopedic surgery and OB-GYN to develop further applications. The future is promising for a groundbreaking device that was launched from a conversation and a desire to collaborate.

“This was a highly complex and challenging project, which made it exciting for everyone involved,” Potts said. “We solved technical problems that no one else has, and we’ve done it right here in Oklahoma City. Our partnership was mutually beneficial, and it was only possible because the Innovation District provided the forum where we could discuss and collaborate.”