Apple is about to launch its most important new product in a decade. Reality Pro, as they say it’s called, will be a VR headset like no other. No, I mean an AR headset. Wait, make it an MR helmet with a dial switch between AR and VR at will…
Despite years of headsets from major companies like Google, Microsoft, Sony, Facebook/Meta, etc., most people still don’t understand AR, VR, or MR. Before Apple unleashes Reality Pro in our faces, let’s clarify what these things mean and discuss what we can expect from Apple’s new platform.
Virtual reality
Let’s start with the easiest. Virtual reality (VR) products place screens in front of your eyes with lenses to expand their field of vision. There are usually two screens (or one split in two), each displaying the perspective of each of your eyes to give a true 3D effect.
The movement and position of the headset should be tracked with gyroscopes and sensors to adjust your view to every little movement of what you see on screen. In order to feel completely immersed and to avoid motion sickness, this must be done extremely quickly, with very low “photon motion latency” (the time between the movement of your head and this change emitted by the screens) .
The end effect is that it looks like you’re in a completely computer-generated place. It can be cartoonish or realistic, but everything – the environment and everything in it – is made of computer graphics. That’s VR: no more real world, a whole virtual world.
Examples of products: MetaQuest, PlayStation VR, HTC Vive, Valve Index
Meta
Augmented reality
Augmented Reality (AR) does not replace the real world with a virtual world, but rather integrates digital objects into the real world. This can be done on a screen, like in an AR app on your iPhone, in a headset, or using a pair of glasses.
The basic idea is that you see the real world, but computer generated things added to it: icons, arrows, text, floating video screens, people, whatever.
An important note is that graphics should appear present in the real world. An AR object can stay in the same place in the real world as you move around it, and can even interact with real-world geometry like the floor, tables, walls, or even people. Simply showing floating graphics on a transparent screen in front of your eyes is not augmented reality.
AR requires at least rudimentary 3D mapping of your immediate surroundings, which is usually done using multiple cameras and sensors like LIDAR. Many products market themselves as AR but are more accurately called “heads-up displays” because they can’t actually integrate computer graphics into the real world. These products don’t make a 3D map of your environment to embed graphics, they just float things in front of you. Examples are Google Glass, Nreal Air and Vuzix Blade 2.
Pokemon Go is often referred to as an AR app, and that’s often confused with the fact that you’re playing it in the real world. It’s just location based game, not AR. There is a small AR part of the game: when you throw your Poké Ball to catch a Pokémon, you can activate an “AR mode” which shows your target Pokémon in the real world on our phone screen. This part, and this part only, is called AR. (And most players turn it off.)
An AR headset could look like a VR headset, completely blocking out your view of the outside world, if it uses external cameras to transmit real-time video of your surroundings to displays inside. That’s how Apple’s Reality Pro should work, and it’s in contrast to something like Microsoft HoloLens which has a transparent screen with an integrated display, so you’re viewing the real world itself directly.
Yet as long as you see the real world around you in real time, with computer generated graphics integrated in it and not just floating in front of you, it’s considered augmented reality.
Examples of products: Microsoft HoloLens, Magic Leap, and upcoming next-gen Snap Spectacles are considered true AR glasses or headsets.
Microsoft
mixed reality
This is where it gets confusing. Apparently augmented reality is not augmented enough so the industry has come up with a term that they can’t even clearly define to separate real AR of all those virtual displays and heads up displays as well as AR phone app stuff.
Look up mixed reality (MR) and you’ll find a dozen different definitions full of vague phrases like “brings the real and virtual worlds together in a more immersive way” or “allows multiple users to interact.” together in a virtual space. Some say it requires you to manipulate virtual objects with your own hands, while others say mixed reality can use controllers.
Intel says MR “brings together real-world and digital elements” (which is what AR does). “It offers the possibility of having one foot (or one hand) in the real world and the other in an imaginary place, breaking down the basic concepts between real and imaginary, providing an experience that can change the way you play and work today.” Poetic, but not particularly useful.
Meanwhile, Microsoft says, “It frees us from screen-bound experiences by providing instinctive interactions with data in our living spaces and with our friends.” I don’t know what “instinctive interaction” with data is, but it sounds incredibly boring. “People may not even realize that the AR filters they use on Instagram are mixed reality experiences.”
So even among the biggest tech companies, there doesn’t seem to be agreement on how to define MR, or what minimum standards allow a product or experience to be labeled mixed reality.
I think a good way to think about it is this: mixed reality is a superset of augmented reality. It’s AR with a minimum level of interaction and higher immersion. It’s viewed with your own eyes (through clear goggles or a headset with video passthrough from external cameras) rather than displayed on a phone or tablet screen. And it not only displays embedded information in the real world, it allows you to interact with these virtual objects and for objects to interact with the environment.
What to expect from Apple Reality Pro
From what we know of Apple’s headset, it will deliver an unparalleled VR, AR, and MR experience. It is said to be a helmet that looks like a pair of ski goggles, which will obscure your view of the outside world.
Multiple cameras will feed a real-time view of the world around you, and a fleet of sensors will create a 3D map of your surroundings so that virtual objects can fit in and interact with it. You’ll control apps and manipulate objects with your hands, and internal sensors will even detect your eye movements.
A dial on the outside, similar to the digital crown on an Apple Watch, will let you control how much of the outside world you want to see. Turn it one way and the world is blocked, leaving only virtual environments: VR. Turn it the other way and you see the full real world (powered by the headset displays from multiple external cameras), with integrated computer generated graphics (AR/MR).
We’ll find out more about Reality Pro’s capabilities when it’s officially announced, likely on June 5 during the WWDC keynote. If the current rumors are true, it will set a new bar for quality, detail, immersion, interaction… and price, potentially costing up to $3,000.
Table of Contents
