2023 could well be the year of mixed reality

At the start of a new year, the public is still largely confused by one of last year’s biggest buzzwords: The metaverse. After being promised a society-changing technology by breathless media influencers, many people actually ended up with either (a) cartoonish virtual worlds full of creepy avatars or (b) opportunistic platforms selling “virtual real estate” through questionable NFT schemes.

It would be an understatement to say that the industry has over-promised and under-promised in 2022.

Fortunately, the metaverse really does have the potential to be a society-changing technology. But to get there, we need to move beyond the cartoonish worlds of today and deploy immersive experiences that are deeply realistic, wildly artistic, and much more focused on unleash creativity and productivity than hitting NFT landlords. In addition, the industry must overcome one of the biggest misconceptions about the metaverse: the flawed idea that we will live our daily lives in virtual worlds that will replace our physical environment. This is not how the metaverse will unfold.

Don’t get me wrong, there will be popular metaverse platforms that are fully simulated worlds, but these will be temporary “escapes” that users sink into for a few hours at a time, similar to how we watch movies or play video games today. On the other hand, the real one other way around, the one that will affect our days from the moment we wake up to the moment we go to sleep will not remove us from our physical environment.

Instead, the true metaverse will usually be one mixed reality (MR) in which immersive virtual content seamlessly blends with the physical world, expanding and beautifying our everyday lives with the power and flexibility of digital content.

A mixed reality arms race

I know some will push back on this prediction, but 2023 will prove them wrong. That’s because a new wave of products is coming our way that will bring the magic of MR to mainstream markets.

The first step in this direction was the recent release of the Meta Quest Pro, which is hardware-ready for high-end mixed reality color pass-through cameras that capture the real world and can combine it with spatially registered virtual content. It’s an impressive device, but so far there’s little software available that showcases its mixed reality capabilities in useful and engaging ways. That said, we can expect the real potential to be unleashed in 2023 as the software rolls out.

Also, HTC plans to release a headset in 2023 that seems even more powerful than the Meta Quest Pro for mixed reality experiences. It will reportedly be unveiled at CES in January and reportedly has pass-through color cameras with such high fidelity that you can look at a real phone in your hand and read text messages in mixed reality. Whether consumers prefer HTC’s new hardware or Meta’s, one thing is clear: an MR arms race is underway and it’s getting busier.

That’s because Apple is expected to launch its own MR headset in 2023. It’s rumored to be a premium device shipping mid-year, but it will probably be the most powerful mixed reality product the world has ever seen. It is claimed to feature high-performance pass-through cameras, along with LiDAR sensors for profiling real-world distances.

If the LiDAR rumor comes true, it could mean that the Apple device is the first MR/augmented reality (AR) glasses product to enable highly accurate real-world 3D registration of virtual content. Accurate registration is critical to suspending disbelief, especially when users can manually interact with real and virtual objects.

Why so much momentum toward mixed reality?

Simple. We humans do not like to be cut off from our physical environment. Sure, you can give someone a short demo in virtual reality (VR), and they’ll love it. But letting that same person spend an hour in fully immersive VR can make them feel uncomfortable.

Approach two hours, and for many people (myself included) it’s too much. I first noticed this phenomenon in 1991, when I was working as a VR researcher at Stanford and NASA studying how to improve depth perception in early vision systems. Back then, the technology was crude and uncomfortable, with low-fidelity graphics and lag so bad it could make you feel sick. Therefore, many researchers believed that the barrier to long-term use was its clunky design and poor reliability. We just needed better hardware and people wouldn’t feel uncomfortable.

I didn’t quite agree. Sure, better hardware would help, but I was pretty sure something else was going on, at least for me personally – a tension in my brain between the virtual world I could see and the real world I could see around me. could feel (and feel). . It was this conflict between two opposing mental models that made me uncomfortable and made the virtual world seem less real than it should.

To address this, I really wanted to harness the power of VR and combine it with my physical environment to create one immersive experience where my visual, spatial and physical senses were all perfectly aligned. My hunch was that the mental tension would disappear if we could let users interact with the real and the virtual as if they lived in the same perceptual reality.

A stroke of luck gave me the opportunity to pitch to the US Air Force and was funded to build a prototype mixed reality system at Wright Patterson Air Force Base. Called the Virtual Fixtures platform, it supported not only sight and sound, but also touch and feel (3D haptics), adding virtual objects to the physical world that felt so authentic that they could help users perform manual tasks with more speed and agility. The hope was that this new technology could one day support a wide variety of useful activities, from assisting surgeons during delicate procedures to helping technicians repair orbiting satellites. telerobotic control.

Two worlds that come together

So early, of course Air Force system did not support surgery or satellite repair. It was developed to test whether virtual objects can be added to real-world tasks and improve human performance. To measure this, I used a simple task that involved moving metal pegs between metal holes on a large wooden pegboard. Then I wrote software to create several virtual machines that can help you get the job done.

The fixtures ranged from virtual surfaces to virtual cones to simulated tracks you could slide the pin along, as early passthrough cameras aligned activity. And it worked, allowing users to complete manual tasks much faster speed and precision.

I give this background because of the impact it had on me. I can still remember the first time I moved a real stake into a real hole and automatically turned on a virtual surface. Although simulated, it felt real, allowing me to glide along the contours. At that point, the real world and the virtual world became one reality, a unified mixed reality where the physical and digital were combined into a single perceptual experience that satisfied all your spatial senses: visual, audio, proprioception, kinesthesia, and haptics.

Of course, both worlds had to be accurately aligned in 3D, but once you succeeded, you immediately stopped thinking about which part was physical and which was simulated.

That was the first time I experienced a real mixed reality. It may have been the first time anyone had it. I say that because once you experience the real and virtual combined into a single unified experience, all your senses aligned, the two worlds actually converge in your mind. It’s almost like one of those visual illusions where there’s a hidden face you can’t see, and then something clicks and it appears. This is what a true mixed reality experience should be like: a seamless fusion of the real and the virtual that is so natural and authentic that you immediately realize that our technological future will not be real or virtual, but both. A world; one reality.

Looking ahead, I’m impressed with how far the industry has come, especially in recent years. The image above (on the left) shows me in 1992 in an Air Force lab working on AR/MR technology. The image on the right shows me wearing a Meta Quest Pro headset today.

What the picture doesn’t show are the many large computers that ran thirty years ago to conduct my experiments, or the cameras mounted on the ceiling, or the huge wire harness draped behind me with cables leading to various machines were running. That’s what makes this new wave of modern headsets so impressive. Everything stands on its own: the computer, the cameras, the display, the trackers. And it’s all comfortable, lightweight and battery-powered. It is remarkable.

And it’s only just begun. Mixed reality technology is about to take off, and it’s not just the impressive new headsets from Meta, HTC and (possibly) Apple that will push this vision forward, but glasses and software from Magic Leap, Snap , MicrosoftGoogle, LenovoUnreal, Unity and many other major players.

At the same time, more and more developers will push the boundaries of creativity and artistry, unlocking what’s possible when you combine the real and the virtual – from new types of board games (Tilt Five) and powerful medical applications (Mediview XR), to remarkable outdoor experiences from Niantic Labs.

That’s why I’m convinced that the metaverse, the true metaverse, will be a fusion of the real and the virtual, combined so seamlessly that users will stop thinking about which elements are physical and which are digital. We will just go about our daily lives and deal with a single reality. It has been a long time in the making, but 2023 will be the year that this future really starts to take shape.

Louis Rosenberg is founder and CEO of swarm intelligence company Unanimous AI.