Idea
Model generating personalized driving plans from natural language instructions for adaptive autonomous vehicle control.
Research Paper
Core Innovation
This paper introduces Vega, a unified Vision-Language-World-Action model that integrates visual inputs and natural language instructions using autoregressive and diffusion paradigms. It uniquely enables joint attention across modalities for instruction-based trajectory generation, surpassing prior models that only use language for scene description or reasoning.
Why It Matters
Autonomous vehicles often lack the ability to interpret diverse user instructions, limiting personalization and adaptability. This solution enables vehicles to follow natural language commands, improving user experience and safety. It scales by integrating vision and language for flexible, context-aware driving decisions.
Market Size (TAM)
$20–50B TAM for autonomous driving software; $2–5B SAM from vehicle manufacturers and fleet operators. Driven by increasing demand for personalized and intelligent vehicle control.
Potential Customers & Pain Points
- Autonomous vehicle manufacturers – Need personalized driving control
- Ride-hailing services – Require adaptive passenger preferences
- Fleet operators – Seek efficient and safe route planning
- Automotive software developers – Demand advanced multimodal AI models.
Business Model
Licensing the Vega model and dataset to autonomous vehicle manufacturers and software developers; offering API access for integration into driving systems; providing customization and support services.
Competitive Landscape
- Waymo
- Tesla Autopilot
- Aurora Innovation
- Comma.ai
Implementation Challenges
- Complexity of real-time multimodal processing in diverse driving environments
- Regulatory and safety certification challenges for instruction-based driving systems
- Integration with existing autonomous vehicle hardware and software stacks
Validation Strategy
- Pilot deployments with autonomous vehicle partners to test instruction-following accuracy
- Benchmarking against existing autonomous driving models on public and proprietary datasets
- User studies to evaluate personalization and safety improvements in real-world scenarios
Research Paper Overview
Vega: Learning to Drive with Natural Language Instructions
Summary
Vision-language-action models have reshaped autonomous driving to incorporate languages into the decision-making process. However, most existing pipelines only utilize the language modality for scene descriptions or reasoning and lack the flexibility to follow diverse user instructions for personalized driving. To address this, we first construct a large-scale driving dataset (InstructScene) containing around 100,000 scenes annotated with diverse driving instructions with the corresponding trajectories. We then propose a unified Vision-Language-World-Action model, Vega, for instruction-based generation and planning. We employ the autoregressive paradigm to process visual inputs (vision) and language instructions (language) and the diffusion paradigm to generate future predictions (world modeling) and trajectories (action). We perform joint attention to enable interactions between the modalities and use individual projection layers for different modalities for more capabilities. Extensive experiments demonstrate that our method not only achieves superior planning performance but also exhibits strong instruction-following abilities, paving the way for more intelligent and personalized driving systems.