Metaverse for Healthcare

Continuing the Metaverse use cases in various industries it is time to take a look at the Healthcare. Industrial Metaverse is more matured than other areas of the Metaverse: enterprise and consumer. This is because industry has had a need and a clear use cases for the Metaverse for a really long time. This journey started a long before the term Metaverse was even invented. The process industry used and uses lots of sensors and computer/logic controlled equipment to make the facility productive and as automated as possible. These components are called today with terms of IoT, Digital Twins and AI. On top of that there is a big number of telemetric data transfers and collaboration in the real time, as well as visualizing the process in either 2D, 3D or Mixed Reality. Thus – we have the Industrial Metaverse.

As all industrial solutions, these components must be robust, reliable, easy to maintain and secure. These are also typically crossing multiple technology platforms since very rarely one can supply for all needs. That is why integrations play a pivotal role in industry solutions. All this applies to the Healthcare industry as well – when dealing with people’s health and lives the reliability and security standards are even higher than on factories. With all this background, there are already great opportunities to be executed at the Healthcare Metaverse today but a lot more will be possible in days to come.

Recently I had the honor to work with various healthcare organizations to plan how they will be utilizing the Metaverse. I can not to to customer details or solutions, instead I will share generic views and areas how Metaverse and the Metaverse Technology can and could be utilized in the Healthcare. This post is divided to 5 areas.

  1. Training, learning and guiding
  2. Artificial Intelligence
  3. Examination and Operations
  4. Patient meetings ja rehabilitation
  5. Smart Hospitals

Training, learning and guiding

This is one of those basic examples where a lot of people can see the value. Learning in the virtual or mixed reality can boost up productivity and cut down learning time an every industry. “During a pilot project at their San Antonio plant, Toyota used HoloLens 2 and Guides to train employees on how to assemble a new version of the Toyota Tundra. The data showed it was a success, Kleiner said. Defects were cut in half. Depending on the individual, training time fell between 20% and 50% Quote source article

With the latest update to Microsoft Dynamics 365 Guides, HoloLens 2 users can summon an array of holographic windows with a Teams call or chat, a Power BI dashboard, a Word document, a PDF or video, their OneDrive folder, or their calendar and operate in an immersive, 3D experience. (Image and image label from Microsoft)

Read about HoloLens 2 brings new immersive collaboration tools to industrial metaverse customers and Microsoft Customer Story: Toyota simplifies complex repairs and accelerates training with mixed reality on HoloLens 2.

When you are using holograms and mixing information to the real environment (mixed reality) it can concretize the process and operation. Using digital twins of real environments and equipment in the virtual training room lets you have a very nearly real experience. And the benefits of virtual training is that you don’t need to be afraid to make mistakes and you can rehearse the process multiple times. In the mixed reality you can have AI or scripted process guiding you through the process. For guiding process, please take a look at my Dynamics 365 Guides blog post.

Video by Microsoft Dynamics 365 Guides

The training and learning in the healthcare can include various situations, such as procedures and processes that needs to be followed on emergency rooms, in the reception, practicing for exceptional situations or even fire drills. It could be part of the on boarding process, where new nurses and staff can learn about the facility and locations for emergency medical equipment and how to use them, for example. When you rehearse something, in the real life or virtual environment with realistic digital twins, you will learn it better than just by reading a manual or watching a video passively. 

Visualizing the information can be used to explain how specific treatment, medicine, drugs or virus is affecting the body. How this new medicine is changing the experience from the previous version? You could read the paper, or learn about it in the visual way. How do you practice difficult social encounters ahead? How do you respond to patients or to angry people? I have heard various examples how Metaverse has helped to practice those situations for better results, when compared to video education or meetings.

I see that training, onboarding, learning and practicing process is a good way to get started with the Metaverse for various healthcare operators. It helps to train and maintain people’s skills, as well as expand understanding.

Artificial Intelligence

Use of AI is of course a huge opportunity. Hospitals and examinations produce a huge number of data, that could be also used to teach AI. Machine Learning can be used to assist doctors and nurses when making observations of the patient and data, once the AI has been educated to understand those use cases and situations. I have read about the use of AI used to discover potential issues from radiological or MRI scans. Not all AI findings are true, but the value is that they can spot issues that human doesn’t always notice. Then the doctor can verify those findings and make decision if that is indeed something that needs to be examined further or it can be marked as false positive.

Picture from video.

When the AI has enough data it can also assist in the decision making – suggesting next steps for doctors and nurses. Again, human needs to verify is the suggestion realistic. Combining all the information together from examinations, patient history and similar cases the AI can look the big picture with the data.

AI will be used on various other ways as well. Speech to text services can enable healthcare personnel to work hands-free. Speech to text and text to speech can be also combined with real-time translation services to reduce language barriers between patients and healthcare staff. This is important for understanding to both parties and for accessibility and inclusivity. Microsoft Teams already provides real time live caption translations during meetings, for example.

AI is already used widely when analyzing viruses or genes and when new medicines are being created. I recommend watching AI for Health / AI for Good video below.

Artificial Intelligence use in the healthcare is a huge opportunity and a major Metaverse Technology use example. This also requires a lot of architecting regarding the data collection, storage, analyzing, compliance, governance and security. Just think GDPR and other policies. For the future, I see those issues solved and the use of AI in healthcare will skyrocket because it increases productivity and efficiency while it reduces number of errors.

Examination and Operations

Using the Metaverse is opening new opportunities for examination and operations in the healthcare. Doctors can examine results in a visual way, in a way of “diving into the data”. Visualizing the data and results is giving new perspectives when compared to looking at the data in 2D.  Mixed Reality holograms can bring results in 3D for viewing and doctors can manipulate it with their hands, and add notes as needed. Nurses could use AI and Mixed Reality when examining the patient, for example AI analyzing the wound and sharing relevant information to the nurse real time. Or perhaps assisting finding a vein or how to correctly place sensors to the patient to reduce errors. Perhaps using intelligent edge cameras that can analyze procedure in the operating room and help automate checklists – for example by checking how procedure is progressing and keeping count of used items. AI would be in the core of that application as well.

Image Source Microsoft HoloLens.

AI can assist doctor in the process, or there could be multiple doctors and specialists in a metaverse meeting looking at the same hologram together. Doctors and experts could be from all around the world and could still work together when viewing the model and hearing & seeing what others hear and see. They could plan a difficult operation using the metaverse. This is something that is already used today. How about a surgical robots that could be operated by a doctor using a mixed reality device? That doctor may be physically in an another country. Or the robot may be performing the surgery on its own. Robotic Surgery is already reality, and for example HUS (biggest healthcare organization in Finland) has Da Vinci Xi robot in use. This is tech combining Robots, AI, sensors, videos and 3D together.

Image Source Microsoft HoloLens.

With the Metaverse and Metaverse Technology we can expect solutions such as this taking steps forward. Think about a specialist surgeon assisting operating surgeon from the another country. Being able to share the view and data, doctors could collaborate together during the operation. 

Image Source Microsoft HoloLens.

Patient meetings ja rehabilitation

This area has a lot of use cases for the Metaverse technology. For example using Mixed or Virtual Reality to show patients, in a visual way, how the operation is going to proceed, or what are rehabilitation or treatment next steps. Giving patients a better understanding what is going to happen is helping them to adopt to the situation and reduce uncertainty and anxiety of the situation. They can see how the operation is going to happen and what is going to be done. And virtual reality can be used during the operation to keep patient’s mind focused in something else but on the operation. This can reduce anxiety and stress in those operations where you are awake during the procedure. Perhaps they could be watching a tv series or just enjoying the nature to take their mind off from the actual procedure.

Image Source Microsoft HoloLens.

For the rehabilitation virtual or mixed reality can be used to help patient to recover from the accident or from the loss of limb. Learning how to walk again could be more motivational when you could do that in the forest or other location you can’t visit in the moment. This also opens more self-help options, when the AI nurse can help patient to make right movements and trainings – and also keep the records up to date about the progress. Using Virtual or Mixed Reality could be used when doing exercises outdoors.

On the mental health virtual and mixed reality could help to cope with the trauma, pain and enhance recovery. Perhaps reducing stress by using meditation apps (like TRIPP) or anxiety. If the patient has fear of going to crowded places, then they could be practicing that in the safe virtual environment where the experience can adopt to their progress.

When you add more sensors to the patient, then the digital twin could get real time information about the progress. This way the AI could start helping the nurse or patient on correct physical movements – or a remote expert doctor could be in the same session to monitor and analyze the patient condition. Either real time or asynchronously looking those results and AI assisted observations. 

Smart Hospitals

The final piece is of course Smart Hospitals and buildings. In the easiest way you could have the hospital building scanned to create a visual digital twin that can be used during onboarding, train for emergency situations or helping people and patients to navigate inside the facility using Mixed Reality. On the smart buildings part you combine and integrate that digital twin with synchronized building automation systems to include heat, water, power, air conditioning, access, security, video cameras, hospital equipment and other devices and sensors to the model. This way  you can start to visualize and optimize the whole hospital, making possible to reduce costs and make it more productive and sustainable.

And what would be a Metaverse in healthcare article, if I didn’t mention that you should also expand the Hospital to the Metaverse? Creating the hospital experience to the Metaverse makes it possible to people to learn about the building, operations. You can think here adding to customer / patient understanding of procedures, the results and other services and of course virtual appointments. In many ways this will start as 3D web page version, but eventually the Metaverse will help us to use more technology to make it possible to have various visits (like therapy session or remote rehab) in the Metaverse.

Read about Medcare Hospitals & Medical Centres Metaverse Hospital in UAE that was launched during October 2022.

The Metaverse is already reality in the Healthcare and Industrial use and it will only grow bigger

One thought on “Metaverse for Healthcare

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.