Unit
- Augmented reality (AR) technology: overlays digital content onto the real world in real time, using cameras and sensors to understand the environment. It requires precise tracking of device position and orientation relative to physical surfaces.
- SLAM (Simultaneous Localization and Mapping): a core computer vision technique that builds a map of unknown surroundings while tracking the device location within that map. It enables persistent virtual objects anchored to real-world positions.
- Plane detection: identifies horizontal and vertical surfaces like floors, tables, and walls. AR systems use point clouds and feature extraction to estimate planar geometry, providing stable placement surfaces for virtual objects.
- Light estimation: captures ambient lighting conditions to adjust virtual object appearance. By analyzing average brightness and color temperature, AR ensures digital elements blend naturally with the physical scene.
- Sensor fusion: combines data from accelerometer, gyroscope, magnetometer, and camera to improve tracking accuracy. Filtering algorithms like Kalman filters reduce drift and jitter, creating smooth motion.
- Anchor management: maintains the position and orientation of virtual objects over time. Anchors persist across sessions when stored with spatial data, allowing objects to reappear in the same location.
- Performance constraints: AR runs on battery-powered devices with limited processing power. Efficient algorithms and GPU utilization are critical to maintain frame rate and prevent overheating.
- Environmental understanding: recognizes semantic features like doorways, furniture, and open spaces. Advanced AR frameworks provide scene geometry and occlusion to enhance realism.
- Occlusion handling: enables virtual objects to be hidden behind real-world objects. AR systems use depth data or mesh reconstruction to determine which pixels belong to physical surfaces, allowing digital content to appear partially obscured by furniture or walls. This creates a more realistic sense of depth and integration with the environment.
- Hand and gesture recognition: interprets hand poses and movements as input commands. Machine learning models classify gestures like pinch, tap, or swipe, enabling touchless interaction with AR content without requiring a controller. Accurate tracking requires real-time processing of camera frames and often uses skeletal hand models for robustness.
- Spatial audio: places sound sources at specific locations in the physical space. Using head-related transfer functions and device orientation, audio appears to come from the position of virtual objects, enhancing immersion and wayfinding. Dynamic binaural rendering adjusts audio cues as the user moves, maintaining the illusion of a fixed sound source.
- Multiuser synchronization: allows multiple devices to share a common AR coordinate system. Through cloud anchors or peer-to-peer networking, players see the same virtual objects in the same real-world positions, enabling collaborative experiences. Latency compensation and conflict resolution are essential to keep all participants in sync during fast paced gameplay.
Design Principle
Principle: Spatial Consistency
Definition: Virtual objects must appear to exist at a fixed physical location, maintaining their position and scale regardless of device movement. Any drift or jitter breaks the illusion of co-presence.
Application: When placing a virtual character on a tabletop, the character should stay in place as the player walks around the table. Use SLAM and persistent anchors to ensure the character reappears in the same spot even after the app restarts.
Quick Revision
- SLAM and plane detection are foundational for AR tracking and placement.
- Light estimation and sensor fusion improve visual integration and stability.
- Anchor management preserves object location across sessions.
- Performance optimization is essential for smooth AR experiences.