Idea
Platform enabling efficient large-scale pretraining of billion-parameter MoE language models on exascale supercomputers.
Research Paper
Core Innovation
This paper presents Optimus, a training library optimized for large MoE language models on the Aurora supercomputer. It achieves near-linear scaling up to 12,288 GPU tiles with custom GPU kernels and a novel EP-Aware sharded optimizer, improving training speed by up to 1.71x and ensuring fault tolerance for stable large-scale training.
Why It Matters
Training large language models requires massive compute resources and efficient scaling to reduce time and cost. This platform significantly improves training speed and stability at exascale, enabling organizations to develop state-of-the-art models faster and more reliably. It transforms AI workflows by making large-scale model training more accessible and scalable.
Market Size (TAM)
$20–50B TAM for large-scale AI training infrastructure; $2–10B SAM from cloud providers and AI enterprises. Driven by growing demand for advanced LLMs and exascale computing adoption.
Potential Customers & Pain Points
- AI research labs – Need scalable training infrastructure
- Cloud providers – Need efficient resource utilization
- Enterprises developing LLMs – Need faster model iteration and cost reduction
Business Model
Licensing the Optimus training library to cloud providers and AI enterprises; offering consulting and support for large-scale LLM training deployments.
Competitive Landscape
- NVIDIA DGX systems
- Google TPU Pods
- Microsoft Azure AI Supercomputing
Implementation Challenges
- High capital cost of exascale hardware
- Complexity of scaling and maintaining fault tolerance
- Competition from established cloud AI training platforms
Validation Strategy
- Benchmark training speed and scaling efficiency on Aurora and other supercomputers
- Pilot deployments with AI research labs and cloud providers
- Demonstrate cost and time savings compared to existing training solutions
Research Paper Overview
Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer
Summary
This work demonstrates large-scale pretraining of dense and Mixture of Experts (MoE) language models on the Aurora ExaScale supercomputer using the Optimus training library. Models up to 220 billion parameters were pretrained efficiently on thousands of GPU tiles with high scaling efficiency and improved runtime performance through custom GPU kernels and a novel optimizer. The system also includes robust fault tolerance for stable training at scale.