Research Paper

Core Innovation

This paper introduces MINT, the first unified model that jointly estimates camera and two-hand motions in world-space coordinates directly from egocentric RGB video, enabled by a scalable supervision pipeline converting public video data into training labels, surpassing prior multi-stage approaches in accuracy and speed.

Why It Matters

MINT matters because precise, simultaneous world-space hand and camera tracking from egocentric video unlocks next-generation AR experiences and robot learning capabilities, addressing costly multi-stage pipeline inefficiencies and scaling real-time motion understanding essential for immersive and autonomous systems.

Potential Customers & Pain Points

• AR/VR device makers – Need accurate hand and camera tracking for immersive experiences
• Robotics companies – Require precise motion understanding for manipulation tasks
• Video analytics firms – Seek scalable detailed motion data extraction from egocentric footage
• Healthcare and rehabilitation centers – Need accurate hand motion capture for therapy monitoring.

Business Model

Enterprise SaaS subscription for real-time motion tracking API and SDK licensing for AR and robotics companies.

Competitive Landscape

• Qualcomm Snapdragon AR hand tracking; Meta Oculus hand tracking; Vicon motion capture systems

Implementation Challenges

• Scarcity of large-scale
• high-quality annotated egocentric motion datasets; High computational complexity in processing spatiotemporal video data; Integration challenges with existing AR and robotics hardware ecosystems

Validation Strategy

• Pilot integration with selected AR device manufacturers for real-time hand and camera pose tracking; Conduct benchmarking against existing hand and camera tracking solutions in varied environments; Engage early robotics customers to validate robot manipulation task improvements using MINT motion data