Unit
- Definition of XR: Extended Reality (XR) is an umbrella term for immersive technologies including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). VR completely replaces the user's physical environment with a digital one, AR overlays digital information onto the real world, and MR allows digital and physical objects to coexist and interact in real time.
- Core components: An XR system comprises a display device (head-mounted display or glasses), sensors for tracking head and body movement, input devices like controllers or hand-tracking cameras, and a computing unit that renders responsive content based on user actions. These components work together to create a sense of presence.
- Key metrics: Important performance indicators include field of view, resolution, refresh rate, latency, and tracking accuracy. For a convincing experience, latency must be under 20 milliseconds, refresh rate at least 90 Hz, and positional tracking within millimeter accuracy.
- Applications across industries: XR is used in healthcare for surgical simulation, in manufacturing for assembly guidance, in education for immersive learning, in retail for virtual try-ons, and in entertainment for interactive experiences. Each application imposes specific requirements on the XR system.
- User experience goals: The primary goals are presence (feeling of being in the virtual environment), immersion (engagement of multiple senses), and usability (intuitive interaction). Designers balance visual fidelity with performance to maintain comfort.
- Challenges: Current XR faces challenges such as motion sickness due to latency, high cost of quality hardware, limited battery life in standalone devices, and the need for compelling content. Ongoing research addresses these issues through better optics, eye tracking, and foveated rendering.
- Standards and frameworks: Industry standards include OpenXR for cross-platform compatibility, WebXR for browser-based XR experiences, and Khronos standards for graphics. These enable developers to target multiple devices with a single codebase.
- Historical evolution: XR evolved from early head-mounted displays in the 1960s through the VR boom of the 1990s to modern consumer devices like the Oculus Quest and Microsoft HoloLens. Advances in mobile processing and display technology have made XR accessible to a broad audience.
- Spatial computing concepts: XR relies on understanding the physical space through spatial mapping and anchor systems. Devices create 3D meshes of the environment, allowing virtual objects to be placed on real surfaces. This enables occlusion, physics, and realistic interactions between digital and physical elements, reinforcing the illusion of co-existence.
- Interaction paradigms: Beyond controllers, XR supports gaze based selection, hand tracking, voice commands, and haptic feedback. Each paradigm suits different tasks. For example, hand tracking is intuitive for manipulation while gaze is efficient for pointing. Designers choose interaction methods based on context and user comfort to reduce fatigue.
- Development platforms: XR applications are built using engines like Unity and Unreal Engine with XR plugins. These platforms provide prebuilt components for locomotion, input, and rendering. Developers also use XR Interaction Toolkit for common interactions and platform specific SDKs for features like passthrough AR and hand tracking.
System Architecture
Pattern: XR Application Stack
- Input Layer: Sensors and controllers capture user actions and environmental data. This includes 6-DOF tracking, hand gestures, and voice commands.
- Processing Layer: The computing unit runs the XR engine, which interprets inputs, updates the virtual world state, and renders frames. It manages physics, logic, and networking.
- Rendering Layer: The graphics pipeline generates stereoscopic images with proper perspective and lighting. It applies optimizations like foveated rendering to reduce GPU load.
- Display Layer: The output devices present the rendered frames to the user through lenses and screens. Optical design corrects distortion and provides a wide field of view.
Quick Revision
- XR encompasses VR, AR, and MR, each blending digital and physical reality to different degrees.
- An XR system includes displays, sensors, input devices, and computing units working together.
- Key performance metrics like latency and field of view directly impact user comfort and presence.
- Standards such as OpenXR and WebXR facilitate cross-platform XR development.