Hybrid meetings in the Metaverse connect real rooms with virtual reality spaces

It is easy to envision how meetings in the immersive Metaverse (virtual reality) are managed and organized. Once you start thinking about Hybrid Meetings more, you can soon realize that hybrid isn’t just about physical location of people attending using personal or room devices: it is about having a meeting where people attend from meeting rooms, homes and also from 3D meetings – aka Immersive Mesh Metaverse. Or perhaps you have multiple physical meeting rooms present in the meeting – along with that virtual room. How all these could even be working together – and on top of that: how this could be effortless and easy for attendees so they work, innovate and create together? Of course this doesn’t happen every time but in the Hybrid World we will be seeing all this more and more often. For example a workshop or ideation event would benefit from the hybrid setup that includes also virtual meeting spaces.

And if you think this is going to take quite a big investment from Microsoft to accomplish, it does look that Microsoft is investing in the Metaverse. Microsoft acquired 18th of Jan 2022 Activision with $69 billion. Microsoft is definitely investing in the Metaverse and with that acquisition they will have experience and expertise to take the Metaverse and 3D worlds & spaces further.

Gaming is the most dynamic and exciting category in entertainment across all platforms today and will play a key role in the development of metaverse platforms,

Satya Nadella, chairman and CEO, Microsoft

What tech solutions would be needed?

Just a disclaimer: this is my speculation and insight I could see this progressing.

Thinking this challenge I see there are a few things that doesn’t have tech available yet – at least not cost-effectively.
Virtualizing the physical meeting room is the key challenge.

How people at virtual space can interact and see the physical room attendees and content?
This will progress in steps over time. Step zero / day one solution would be that attendees in physical room join the immersive Mesh Metaverse meeting using their devices – just like any other virtual attendee. This will give them presence and ability to see what’s happening in the virtual world. But I don’t think this provides much value – in fact, the idea is to make working together east and seamless: truly hybrid. In this thinking we would want to bring the physical room and virtual attendees together in the Metaverse.
First, let’s think how virtual attendees could see and interact with physical room attendees.

1. I vision that the first real step is going to be a Virtual Teams Room view in the Mesh Metaverse. It would be the most easy way to do: just create a display & audio setup in virtual space that shows meeting room camera output, audio and shared content from the meeting room into virtual space. 2D view “a window” to physical world – just like you would be in a meeting in another Teams Room. This same tech could be used with people joining on their personal devices but who don’t want to use avatars (or don’t have one) and don’t participate in immersive side. Audio would come out from a single source (virtual side of Microsoft Teams Room). You could this this is enough, but let’s not stop in our visioning here.
2. Next, I would like to see a way to virtualize people in the physical meeting room. A setup of, perhaps, a 2-4 cameras along with microphones that are seeing the meeting room from different angles. Their feed could be analyzed by Azure AI and services to create a virtualized holo-view or avatars (if people in room are recognized and they have avatars room could use) that could be planted into Mesh Metaverse. This could be even something similar to pictures we see about Microsoft Mesh related to holopresence – but without mixed reality glasses involved. There should be some level of spatial audio available so walking to people in the virtualized physical meeting room would amplify these people audio over others. People joining in their own devices could see virtual attendees who are close to them (in the meeting room) and interact.

3. Taking the step 2 to next level: identifying people and using their Mesh avatars and virtual meeting room in the Mesh space so virtual attendees could walk near to some persons and talk to them. This would also include spatial audio and also gives virtual attendees ability to speak back to physical room people – with 3D spatial audio. Just like they would be in the real room. The way I vision this it would require 3D audio and enable people to talk with attendees two-way between mixed and virtual worlds. Adding with content camera and other features those interactive parts of the physical meeting room could be virtualized for other attendees. Something like this would have been way-over future scifi a few years back but now it is not unreasonable to think something like this may be just couple (or few) of years away.
4. Perhaps a sidestep for steps 2 and 3: using an app in a mobile phone to track person’s location in a room to make them appear in the immersive metaverse (virtual) room. This would require location service by WiFi or other way to accurately point person’s place in the metaverse. Perhaps then mobile app and audio & cam would be used to convey audio and expressions to the digital counterparts – and to “see” who else is in the room.

There is also the other way around. How physical meeting room people can interact with virtual attendees.

Step zero is that people use their own devices to look into the virtual meeting space. Attendees using HoloLens / Mixed Reality set can see people (avatars) in the virtual space and also communicate with them, just like in AltSpace VR now. This doesn’t really connect to the physical meeting room but optionally room occupants can join the virtual side. Again, I don’t think this progresses the idea of Hybrid work the way it is intended, so a few extra tech steps will help to take this forward.

1. Microsoft Teams Rooms will enable a view in one of it’s screens to virtual space – so all physical room attendees can see what’s happening in the virtual side. This would be just like Teams Immersive Meeting option in Teams Rooms as it will be on Teams clients.
2. Attendees using HoloLens / Mixed Reality device can see people (avatars) in the virtual space and also communicate with them. Or perhaps just do this with a mobile client.
   If you are in the meeting room you can see avatars along with other attendees. This is the very holopresence situation – but requires a device for attendees “to see to the virtual world”. The difference here to step zero is that the meeting room has also a virtualized version of itself so Metaverse virtual attendees won’t be walking through the table – at least not all the time.
   This is kind of classical holomeeting we have seen in scifi and also in Mesh pictures and videos. This step would also allow people using Hololenses to work together with virtual attendees, especially with 3D models displaying data on virtual space.
   Of course: what would be the point of people wearing headsets when some of them sit in the physical meeting room together? Unless those are only used for specific needs – like ideation in smaller groups, working on a 3D design, co-creating or learning together.
3. What is the next step from that? Holo-projection to meeting room, so you wouldn’t need a headset? That seems very future-scifi even for me to vision that happening any time soon. Perhaps it would be more about smarter use of their own device (mobile, laptop) to connect with the virtual world attendees while being able to speak to them in a small group if needed.

I am quite confident we will see some of step 1 during next year at latest. Bringing in Teams Rooms to work with immersive Mesh Metaverse would make lots of sense and also take steps in the Hybrid World path.

So that was visioning, what about now? How Metaverse are our Hybrid meetings anyway already?

This is the fun part, what we can do already? When we sit in a meeting at the meeting room we may not realize that we are already in Microsoft Metaverse. No – we don’t see virtual worlds or enter 3D meetings there, but the essential core is there: co-create and author content.

When people participate in the workshop we tend to ideate, plan, write and update documents, speak and use different tools together. You can add a variety of applications to Teams meetings to interact with people as well. Will these applications be present in virtual 3D spaces as well? I don’t know when, but I am very very disappointment if you could not add and interact these applications in Mesh Metaverse. Perhaps they drop to step 1 and not be there when immersive meetings launch – and it would be acceptable of course.

What’s already there?

• Microsoft Whiteboard can be edited by people attending from the room and online – just like with documents. If you have Surface Hub or similar device in the room showing the whiteboard this immersion steps one step forward and is truly a hybrid / metaverse experience: both physical and virtual people can edit the whiteboard and see results live. (people attending has currently to be from the same organization so they can view and edit the Whiteboard). This means Whiteboard is already a Metaverse tool.
• Documents like Word, PowerPoint, Excel, OneNote and so on. People online and in the room can edit them: everyone can see changes happening if the document is being shared.
• Collaboration applications: everyone can work with apps – they just have to use their own devices.
• .. and every other application that can be worked together.

Microsoft Loop is going to expand the list even further – enabling everyone to edit and ideate together even easier than before. It is not “out there” yet in regards of Whiteboard, Context IQ and so on – but it is clearly a step forward in Hybrid work and world journey: enabling working together in smaller pieces of content – making it very versatile to co-author and ideate together across physical, virtual or organization bounds.