Apple Vision Pro A New Era in Virtual Reality

Apple Vision Pro is an upcoming virtual reality headset developed by Apple Inc., announced on June 5, 2023, at the Apple Worldwide Developers Conference. It is scheduled to be available in early 2024 in the United States and later that year internationally. This marks Apple's first major new product category since the Apple Watch in 2015.

Apple markets Vision Pro as a "spatial computer" that integrates digital media with the real world. Users can interact with the system through physical inputs like gestures, eye tracking, and voice commands.

Primarily designed as a standalone device, Vision Pro runs its own operating system, VisionOS, derived from iOS, tailored for extended reality applications. It can also connect wirelessly to a Mac.

Development of Apple Vision Pro

In May 2015, Apple acquired the German augmented reality company Metaio, originally a Volkswagen spin-off. It’s rumored that this acquisition was initially linked to Project Titan. In the same year, Apple hired Mike Rockwell from Dolby Labs. Rockwell formed a team including Metaio co-founder Peter Meier and Apple Watch manager Fletcher Rothkopf. They developed an AR demo in 2016 but faced opposition from then-Chief Design Officer Jony Ive and his team.

Augmented reality and VR expert, former NASA specialist Jeff Norris, joined Apple in April 2017. Rockwell’s team helped launch ARKit with iOS 11 in 2017. They aimed to create a headset, collaborating with Ive's team; the decision to reveal the wearer’s eyes through an external display was well-received by the industrial design team.

The headset’s development saw a period of uncertainty with Ive’s departure in 2019, followed by his successor Evans Hankey leaving in 2023. Senior Engineering Manager Jeff Stahl, reporting to Rockwell, led the development of Vision Pro’s operating system, having previously worked on gaming and graphics technologies at Apple.

Apple's extended reality headset is seen as a precursor to lightweight AR glasses, which are not yet technically feasible. In November 2017, Apple acquired Canadian XR company Vrvana for $30 million. Vrvana’s Totem could overlay fully opaque, true-color animations onto the real world, unlike other AR headsets which couldn’t display black. It used infrared lights and cameras for spatial and hand tracking while maintaining a 120-degree field of view at 90 Hz.

According to leaker Wayne Ma, Apple initially planned to pull macOS programs into the user environment, but this was canceled early due to iOS dependency limitations and imprecise hand tracking for gaming. Collaborations with brands like Nike for workout use were also explored, but canceled due to the battery pack and fragile display. A feature called "co-presence" showing the full body of a FaceTime user was also canceled for unknown reasons.

How Apple Vision Pro Works

Apple Vision Pro is a "mixed reality" headset, capable of displaying both virtual and augmented reality content. Unlike AR glasses, it is designed to fully cover the eyes and block out surroundings.

The device uses external and internal cameras for its AR component. These cameras map the surrounding space and display the real world, allowing users to see an accurate image of their actual environment through the headset.

This capability enables Apple to overlay windows, objects, and 3D content in the user’s physical space or shut off the cameras for a fully virtual setup. Apple focuses on the headset's potential for work, content consumption, gaming, communication, and other activities "anywhere" using Vision Pro alone.

Design of Apple Vision Pro

Apple Vision Pro is a wearable "spatial computing" device. Though Apple avoids calling it a headset, that’s essentially what it is. It is worn over the head, resembling the PlayStation VR or Meta Quest headsets.

Design-wise, Vision Pro looks similar to a pair of advanced ski goggles, featuring a laminated glass front, "three-dimensionally formed," connected to a curved aluminum alloy frame that wraps around the face. Currently, the headset is available only in silver with matching gray seals and straps.

The frame magnetically attaches to the face’s Light Seal, which Apple offers in various sizes and shapes to fit different faces. A tight fit of the Light Seal is crucial for light blocking and proper headset operation. Apple designed the Light Seal to be flexible, conforming to individual facial shapes.

Audio straps with spatial audio speakers are attached to the sides of the headset. These straps connect to a braided textile headband securing the headset at the back of the head, with an adjustable dial for a custom fit. The main braided headband is detachable, allowing users to switch sizes and possibly designs in the future.

Apple designed the ribbed structure of the headband to provide cushioning, ventilation, and stretch for comfortable long-term wear.

An optional secondary strap can be placed over the head to better distribute weight. Apple has provided limited details about this secondary strap, noting the design was not finalized. The secondary strap was added after discovering that people with smaller body and head sizes had difficulty wearing the headset for more than half an hour.

Physical controls include a digital crown on top that changes your "immersion" level, showing more or less of your real-world surroundings through "environments," and a top button acting as a camera button to capture "spatial" 3D videos and photos.

On the left side of the headset, there is a space to connect a braided cable providing power to the device. The circular charging disk looks like an Apple Watch charging disk but clicks securely into the headset with a twist gesture.

The charging cable connects to either an external battery pack worn at the waist or a wall-connected power adapter.

Display

Apple Vision Pro features custom micro-OLED displays with a total of 23 million pixels, which Apple says is "more pixels than a 4K TV." The headset displays have a 90 Hz refresh rate, with a 96 Hz refresh rate available when watching 24 fps movies.

The displays are described as postage-stamp-sized, but specific hardware details are not yet known.

There is also an external "EyeSight" display that shows the wearer’s eyes to people around them. This feature allows others to know if the wearer is using the less immersive AR mode or the fully immersive VR mode.

The eye display comes from an internal camera monitoring the wearer’s eyes. It can also alert people when video recording is happening with the external camera.

Lens Inserts, Cameras, and Sensors

Apple Vision Pro customers who wear glasses can order Zeiss Optical inserts with their prescription. These will be an additional cost, but pricing has not been announced. The lenses will magnetically attach to the displays inside the headset.

Vision Pro includes 12 cameras and five sensors for hand gesture tracking and external environment mapping.

Two of the cameras transmit over a billion pixels per second to the display to depict the real world when using AR mode, while other cameras are used for head tracking, hand tracking, and real-time 3D mapping.

Infrared flood illuminators improve hand tracking in low-light conditions, and LiDAR depth sensors determine the size and location of objects in the room around you.

Four infrared cameras and LED lights inside the headset are for eye tracking. Using this technology, Vision Pro can precisely determine where the wearer is looking for navigation purposes.

There are no physical controls for Vision Pro. Navigation is performed through eye movements, hand gestures, and voice commands. For example, in VisionOS, you can highlight an item like an app icon by looking at it and using a quick hand gesture to launch the app.

Bluetooth keyboards, mice, trackpads, and game controllers can be connected to Vision Pro as alternative navigation methods, though it is primarily designed for use with Mac connectivity and Apple Arcade games.

Apple Vision Pro and Optic ID

The infrared cameras and LED lights inside Vision Pro are used for a security feature called Optic ID. Optic ID is similar to Touch ID or Face ID, but it uses iris scanning instead of facial recognition or fingerprint scanning.

Each person has a unique iris pattern, which the headset can detect to keep sensitive data on the headset secure. Optic ID can be used like Face ID to unlock the device, authenticate purchases, and substitute for passwords.

3D Camera

An external camera can capture 3D videos and photos that can be viewed in 3D on Vision Pro. Apple claims these are like reliving a memory in person due to the "incredible depth" available.

The camera provides a clear indicator when recording is active so the headset cannot be used for secret video recording.

Several media members have tested the spatial video feature on Vision Pro, describing it as so vivid that it’s almost eerie. Joanna Stern from The Wall Street Journal said it creates an intimacy that 2D pictures do not, while CNET's Scott Stein said the videos are convincingly realistic.

Audio, Processors, and RAM

Speakers are on both sides of the headset, with built-in speakers in the straps connected to the frame. The speakers feature dual-driver audio pods placed next to each ear, capable of analyzing room acoustics to adapt the sound to the space.

The speakers support spatial audio for an immersive surround sound experience, and there are six microphones for phone calls, video calls, and voice commands.

Vision Pro will pair with AirPods for a superior audio experience, with USB-C AirPods Pro released in September 2023 supporting 20-bit 48 kHz lossless audio with ultra-low latency when connected to Vision Pro.

Vision Pro is powered by two chips. The primary processor is the M2, the same chip first introduced in the 2022 MacBook Air. It handles content processing, running VisionOS, executing computer vision algorithms, and providing graphical content.

A secondary R1 chip is responsible for all incoming information from cameras, sensors, and microphones. Apple claims it can stream images to the displays within 12 milliseconds, providing an "almost lag-free" view of the world.

An internal thermal system is designed to quietly move air through the headset to manage heat and

keep the components inside cool.

According to a leaker, Vision Pro has 16GB of RAM. The RAM is paired with the M2 chip.

Battery Life

Vision Pro relies on an external battery that is meant to be kept in a pocket or at the waist, connected via a cable. The battery pack is the size of an iPhone, providing two hours of usage per charge.

The battery can be swapped out when it runs out of juice, or the headset can be plugged into the wall for non-portable use.

Vision Pro is designed for at-home use, not portable use, due to the limited battery life. The battery is for convenience so that users can switch locations without plugging in but isn't meant for long periods of use outside the home.

The battery pack is 69 watt-hours, comparable to the energy usage of a high-performance laptop.

VisionOS

Apple designed Vision Pro to run VisionOS, a new operating system derived from iOS and optimized for extended reality. It will run a dedicated App Store for VisionOS apps and programs, supporting more than 100 Apple Arcade titles.

VisionOS allows multiple apps to be open simultaneously, which can be arranged around the user’s environment. The home screen with apps is similar to the iPad's.

Apple designed VisionOS for head and hand tracking as well as external hardware input like keyboards and controllers. It integrates with other Apple devices, allowing iPhone users to display their screens on the headset and Mac users to use Vision Pro as a high-resolution display.

VisionOS is primarily for entertainment, communication, and productivity, with apps including Apple TV+, Safari, Photos, Apple Music, Freeform, Apple Arcade, Fitness+, and FaceTime.

Apple intends Vision Pro to be used for everyday activities like browsing the web, listening to music, watching movies, playing games, working, and communicating.

Apple Vision Pro Pricing

Vision Pro is priced at $3,499 in the United States, significantly more than most available AR and VR headsets.

According to The Information, the headset's components cost $1,500, including the external display, internal displays, and chipsets. The rest of the price comes from other components, development, and manufacturing costs. Apple also includes a fee to ensure its traditional profit margin.

Apple's entry price suggests it aims to keep Vision Pro as an expensive product for early adopters and professionals. Apple is developing lower-priced headsets, but these will not be available for several years.

Vision Pro International Availability

Apple will release Vision Pro in early 2024, initially only in the United States. Global sales will begin later in 2024, with specific countries and regions to be announced. Reports indicate Apple has a more targeted launch approach compared to previous products due to the headset's complexity and high price.

Apple Vision Pro Successors

Apple is working on two Vision Pro successors: a second-generation headset with a lighter and smaller design, and a lower-cost model.

According to Bloomberg, the second-generation model will launch in 2025, addressing issues like the weight and size of the first-generation headset. The lower-cost version will target a wider audience with fewer features and cheaper components.

Apple plans to improve the lens system, make it more comfortable, and improve battery life with the second-gen model. The affordable version will aim to maintain high resolution and performance at a reduced price, which could involve compromises in materials, sensors, and hardware.

Apple Vision Pro Criticism

Early media reviews of Vision Pro have been positive, with reviewers praising the impressive display, comfortable fit, and smooth operation. Vision Pro's first hands-on impressions were positive, with high praise for the stunning visuals.

However, concerns have been raised regarding the price and complexity. The $3,500 price tag is considered expensive, limiting the market to early adopters and professionals.

Battery life and comfort are also concerns. A battery life of just two hours and the need to carry an external battery pack is seen as inconvenient, and the device's bulk and weight might deter some users.

The Vision Pro's design may not appeal to everyone due to its appearance resembling a pair of ski goggles. However, Apple is working on future iterations to address these concerns.