Unit
- Mixed reality (MR): MR refers to the blending of physical and digital environments where real and virtual objects coexist and interact in real time. This concept extends beyond augmented reality by allowing virtual objects to respond to physical surroundings with spatial awareness, creating a seamless interplay between the two realms.
- The reality-virtuality continuum: Paul Milgram coined this spectrum with purely physical environments on one end and fully virtual environments on the opposite end. Augmented reality sits closer to the physical side by overlaying digital content, while augmented virtuality places real objects into virtual worlds. MR occupies the middle ground where both realms are equally interactive.
- Key attributes of MR systems: Three essential characteristics define MR: (1) spatial mapping that understands room geometry, (2) occlusion between virtual and real objects, and (3) persistent anchoring that holds digital content in place relative to physical surfaces. These attributes differentiate MR from simpler overlay approaches.
- Display technologies: Head-mounted displays (HMDs) like the Microsoft HoloLens and Magic Leap use waveguide optics to project holograms directly into the user's field of view. Passthrough VR headsets, such as Apple Vision Pro, use outward-facing cameras to show the real world on internal screens while adding digital elements with high fidelity.
- Sensor arrays: MR devices rely on depth cameras, inertial measurement units (IMUs), and eye-tracking sensors. Depth cameras capture environment geometry in real time, IMUs track head rotation and acceleration, and eye trackers enable foveated rendering and intent detection.
- Tracking and registration: Accurate tracking requires solving six degrees of freedom (6DoF) for the headset and controllers. Inside-out tracking uses on-board cameras to detect features in the environment without external markers, while outside-in tracking uses fixed cameras or lighthouses for sub-millimeter precision.
- Spatial awareness: MR systems build a real-time 3D mesh of the environment, enabling virtual objects to sit on tables, stick to walls, or be occluded by real-world obstacles. Meshing uses depth sensor data processed by SLAM (Simultaneous Localization and Mapping) algorithms, which we explore in later modules.
- Interaction modalities: Users interact through hand gestures, voice commands, eye gaze, and controller inputs. Each modality has trade-offs: gestures feel intuitive but require accurate hand tracking, voice works at distance but may not work in noisy environments, and gaze provides fast targeting but suffers from Midas Touch problem.
Code Example
C#
using UnityEngine.XR.Interaction.Toolkit;
using UnityEngine.XR.ARFoundation;
public class AnchorPlacement : MonoBehaviour
{
public GameObject hologramPrefab;
private ARRaycastManager raycastManager;
void Awake()
{
raycastManager = GetComponent<ARRaycastManager>();
}
void Update()
{
if (Input.touchCount/0 && Input.GetTouch(0).phase == TouchPhase.Began)
{
Vector2 touchPos = Input.GetTouch(0).position;
List<ARRaycastHit/hits = new List<ARRaycastHit>();
if (raycastManager.Raycast(touchPos, hits, TrackableType.Planes))
{
Pose hitPose = hits[0].pose;
Instantiate(hologramPrefab, hitPose.position, hitPose.rotation);
}
}
}
}This script uses AR Foundation to detect the user's touch on a plane and place a hologram prefab at that location. It demonstrates the core placement interaction in MR: raycasting against the spatial mesh to find real-world surfaces.
Quick Revision
- Mixed reality blends physical and digital worlds where both coexist interactively.
- The reality-virtuality continuum ranges from pure physical to pure virtual, with MR in the middle.
- MR systems require spatial mapping, occlusion, and persistent anchoring.
- Head-mounted displays use waveguides or passthrough cameras to show digital content.
- Sensor arrays include depth cameras, IMUs, and eye trackers for accurate tracking.
- 6DoF inside-out or outside-in tracking determines headset and controller positions.
- Spatial awareness relies on SLAM algorithms to build real-time environment meshes.
- Interaction modalities include gestures, voice, gaze, and controllers.