What It's Like Using Apple Vision Pro

The eye-tracking as a navigation mechanism is unreal.

I wrote earlier this week about the Vision Pro announcement from WWDC. Since then, I’ve had a chance to actually use it during a half-hour demo. The experience was unlike anything I’ve been a part of. I wasn’t able to take photos or record video of the experience, but I’m free to share anything else, and this is my attempt to do just that. The photos in this post are from Apple, but are exactly what I experienced.

Note: This is long, and there are lots of photos. You may want to read it on the web instead.

Also, to be completely candid, I wanted to write this down because it’s an experience I expect to look back on in the future as a moment when a lot of my expectations of technology were challenged and expanded. More importantly, I’m going to try to explain why I think Apple got it exactly right with Vision Pro.

First, an acknowledgment: I’ve said several times that I have no idea if wearing a computer on your face is ever going to be cool. I still have no idea. Apple, however, does have an idea—at least, it has an idea about what that computer should be like. The result is a thing I would absolutely use. Would I spend $3,499 to use one every day? That’s a very difficult question.

Pre-Demo

On Monday, Apple announced the Vision Pro during its WWDC keynote. Then, over the next few days, the company did hands-on demos with some number of journalists and, I presume, other guests Apple wanted to experience the new device.

I tried out the headset Tuesday morning in a Field House Apple constructed specifically for the Vision Pro demos. When I arrived, the first step was to have two scans taken, similar to setting up FaceID on an iPhone. In fact, the scan was done on an iPhone. The first was of my face, which give Apple measurements for the light shield. The second involved moving my head from side to side to get a measurement of my ears for the Spatial Audio feature of the Vision Pro.

Once that was done, I gave my corrective eyeglass prescription to a second person who put them in the computer so that the Vision Pro could be fitted with a set of Zeiss lenses. For those who didn’t have their prescription handy, Apple had a device that would analyze their glasses and provide the data.

Once finished with those two steps, which took no more than five minutes, I was led into a room, maybe 200 to 300 square feet in total, or, about the size of a modest living room. I sat on a couch and two Apple representatives sat in armchairs on either side of the couch.

When you first put the headset on, it’s surreal. You’re definitely looking at two screens, but what you see is exactly what you saw before you put it on. It’s as though you’re looking through a pair of untinted ski goggles, not a computer on your face. I cannot say enough that it was the most incredible computing experience of my life.

I distinctly remember the first time I unboxed and set up an original iPhone in November of 2007. It was delightful in the way it just worked. As you held it in your hand and thought about what you might do with it—or how you might interact—it did exactly what you expected. If you wanted to scroll, you just dragged your finger up or down.

Nothing else had ever been quite like the iPhone, and it was transformative. It wasn’t a phone as much as it was a completely different type of device, something that had never existed before, but that unleashed entirely new ways of interacting with a computer.

The Vision Pro—even as a pre-production piece of hardware—is that, only more so.

The forward-facing cameras and displays do such a good job of providing pass-through video that it’s like you’re just looking at the room as you were before you put it on. There is no perceptible delay or lag. As the product manager from Apple gave me instructions, I turned to look at him, and there was no sense that anything was off.

The Experience

When you put it on, the device takes you through a short setup process. The first is to align the displays with your eyes, which the Vision Pro does on its own. Then, it calibrates the eye tracking by showing and asking you to follow a series of dots.

We have to talk for a minute about the Vision Pro’s eye tracking.

The entire interface of VisionOS, the software powering Vision Pro, is navigated using your eyes and a few simple hand gestures. You simply look at elements and they “pop,” to tell you they are active. Then, tap your forefinger to your thumb to select something. You can also use a pinch and toss motion to scroll. Finally, you can use voice to enter text or ask Siri to open apps.

The eye-tracking is so good it worked flawlessly every time. All I had to do to select something was to look at it. Even small elements, like a circle with an “x” that would close an app window, were easy to navigate simply by looking at it and tapping my fingers together.

I have tried to think of another way to describe the experience, but I keep coming back to one simple thought: it’s basically magic. My brain understands that it’s not magic, but it definitely feels like it. It is as intuitive and effective as a way of navigating a software interface as multi-touch on the iPhone, or the mouse on a Mac.

To accomplish all of this, Vision Pro has infrared LED illuminators, and two IR cameras pointed at each eye. There are also cameras on the front and underside of the Vision Pro that feed the pass-through video, but more importantly, they track the movements of your hand. I wrote before that the technology behind it all is impressive, but not nearly as much so as the experience.

Experiences

Admittedly, the demo I experienced was exactly what Apple wanted me to experience. There was no free time with the Vision Pro to poke around at things and see what it could do. Instead, I was guided through various apps and experiences. There are still things about Vision Pro that are clearly not done, but what is done is breathtaking.

Looking back, the half-hour clearly started with what is the least impressive part of the demo, which makes sense—if you want to show off your revolutionary platform, you build to the best part. I opened the photos app and swiped through a series of photos, ending with a panorama of Iceland that wrapped all the way around my field of view - as if I were standing there.

The thing I can’t quite shake is that even that, the least impressive part, was still more impressive than any other technology I can think of.

After the photos, I opened a Safari window and scrolled through a website. The text was clear and sharp despite the fact that the entire window was floating a half dozen feet in front of my face. I spend most of my working day with about a dozen or so tabs open at a given time reading, researching, and writing, and I’m very proficient at all of those things on a Mac, or even on an iPad.

Could I see myself doing that with a Vision Pro? I’m not sure. Certainly, a laptop with a keyboard seems like a better experience for those things. Of course, during the keynote, Apple showed that if you have a Mac in your workspace, you can simply look at it and the display will be transferred into your virtual view, and you can interact with a Bluetooth keyboard and mouse.

Then again, if your primary way of working is using your laptop in a virtual space that looks exactly like your physical space, it might just make more sense to use it in the physical space. The main advantage seems to be that your virtual display can be a lot larger than the 14-inch version on your MacBook Pro.

Next, I watched part of a 3D movie (Avatar: The Way of Water). I was able to view it as if it were a giant TV screen in the living room, or in a completely dark, theater-like setting. I also saw a demo of the 3D videos that the Vision Pro is capable of recording. It’s impossible to relate to this without experiencing it, but the sense of presence was palpable.

In one of the scenes, a group of children gather around a cake and blow out candles. As they did, I found myself moving my head to the side as the smoke came directly at me. The image above really doesn’t convey that feeling in a 2D still, but it was wild. That said, I’m not sure how I feel about the idea of wearing something over my face during a seven-year-old’s birthday party.

There were other parts of the demo that involved Apple’s new video format that places you in the center of a virtual scene, as well as what it calls Environments. If, say, you’d rather not be in your office, you can change your environment. For example, the one I experienced was of Mt. Hood.

Then came a sports demo, where it appeared that I was directly behind and slightly above the backboard at an NBA game between the Denver Nuggets and Phoenix Suns. Another scene was what appeared as though I was standing in a dugout during an MLB game. Sports is an incredibly compelling demo because there are a lot of people who would pay to feel as though they were sitting courtside at a basketball game.

At one point, I had a FaceTime call with another person wearing a Vision Pro. They appeared to me as what Apple calls a persona, which is its term for a digital avatar. I think Apple intentionally chose another word because most people think of an avatar as an animated cartoonish representation of a person, whereas Apple’s personas are an attempt to make your digital representation as lifelike as possible.

You set up your persona by scanning your face with the front cameras and Lidar scanner on the Vision Pro, which maps your face. Then, when you’re wearing the headset, the cameras facing your eyes, as well as the ones facing down in front of you, watch the movements of your face to animate your persona. The effect is a bit eerie, but it does mostly work.

As for other people, Vision Pro is definitely designed not to isolate you. If someone else walks into the room where you’re wearing the headset, Vision Pro recognizes that a person is nearby and they break through the user interface and appear slightly transparent. At the same time, your eyes appear on the front of the device to let them know you can see them.

During the keynote, when Tim Cook announced Vision Pro and an image appeared with the headset revealing the eyes of the woman wearing it, the audience gasped. I don’t know how I feel about the idea that there is an OLED screen on the front of the display to show other people your eyes, but I do get why Apple did it.

Finally, the pièce de résistance came at the end when Apple shared what it called Encounter Dinosaurs, which is exactly what you expect. It started with one of the two people from Apple encouraging me to stick out my finger, and a butterfly flew out and landed on it. Even if I moved my hand, it stayed on my finger.

Then, I stood up and approached the wall on the other side of the room as a dinosaur appeared. It was visceral. As I moved side to side, the dinosaur reacted and followed me. The closer I got, the more I could see the scales on its body, and the reflection of light in its eyes. If someone had blown a puff of air at me as it “breathed,” I might have fallen over.

After each part of the demo, a simple press of the digital crown took me back to the home screen of apps. A long press on the digital crown would re-center all of the open windows in my field of view. Aside from that, those apps remain exactly where you leave them. They don’t wobble or move around. They remain as fixed as the physical elements in the real room I was sitting in.

That alone is incredibly noteworthy because it’s what gives Vision Pro a sense of augmenting reality. The stability of the virtual interface overlayed on the physical world that you see is what makes spatial computing viable as a platform.

Impressions

If you’ve read this far, you probably sense that I was overwhelmed and impressed. That’s absolutely the case. This isn’t like any piece of technology I’ve experienced, and I’ve used most of the current VR headsets available to buy. We have a Quest 2 sitting—mostly unused—in our basement. It’s fine for the occasional game or experience, but it’s not compelling in a way that would make me get up and wear it on any kind of regular basis.

The Vision Pro is not that. The Vision Pro is compelling precisely because it does not feel like the experience of using a VR headset. It’s useful that Apple talks about spatial computing because it helps to create the context for what this thing really is. It’s a device that transforms the space around you into a canvas where you can do any number of different things.

I only used it for a half hour, but it didn’t feel like using a computer. It felt like stepping into a futuristic world where virtual and physical elements overlap and “computing” isn’t defined by a device, but by ambient interactions that you can place anywhere in your environment.

I’m not ready to say that Apple has a hit—the Vision Pro is too expensive to make that a sure thing. That said, it did not feel overpriced at $3,499. Several people I spoke with who also tried Vision Pro said they would have paid good money just for another 30 minutes with the device.

Also, the primary use case of almost every VR headset on the market is gaming, a topic Apple did not touch on at all beyond a quick throwaway line during the keynote. There was no gaming part of the demo I experienced.

There are other barriers to success for a computer you wear on your face—the most obvious being the fact that humans get very self-conscious about wearing things that attract attention. The bar is high if the success of your product requires people to get over any self-consciousness they might feel using it on a regular basis.

The original AirPods had the same problem—you looked kind of silly wearing little white stems hanging out of your ears. People quickly got over that because the experience of wearing AirPods is definitely a net benefit They continue to be the best Bluetooth earbuds not because they are the best sounding speakers but because of features enabled by their overall integration with Apple’s other products, as well as their ease of use.

Vision Pro has an even higher bar to clear because it does not hang out of your ears, it hangs on the front of your face. I think Apple has put a lot of effort into solving the problem of someone wearing this device around other people, and I give them credit for the attempt. I’m just not convinced that wearing a headset computer will ever become quite as normal as wearing little white earbuds.

That said, Apple has definitely done all of the things you would need to do to make that happen. The technology is compelling enough to enable an entirely new way of interacting with computers. Its success will largely depend on the software created to let people do that—which is why Apple rolled it out at a developer conference. I’m still not sure it’s a thing anyone needs in their life, but it was incredibly fun giving it a try