Idea
Multilingual ASR and AST models offering fast, accurate speech recognition and translation for European languages.
Research Paper
Core Innovation
This paper presents Canary-1B-v2, a multilingual ASR and AST model using a FastConformer encoder and Transformer decoder trained on massive data with non-speech audio to reduce hallucinations. It introduces a two-stage pre-training and fine-tuning process with dynamic data balancing and reliable timestamping via NeMo Forced Aligner. The model achieves superior speed and competitive accuracy compared to larger models and Whisper-large-v3.
Market Size (TAM)
$10–20B TAM for speech recognition and translation platforms; $2–10B SAM from enterprises and developers adopting multilingual ASR/AST. Driven by global demand for real-time multilingual communication and AI-powered transcription services.
Potential Customers & Pain Points
- Speech technology companies needing efficient multilingual ASR
- Enterprises requiring fast accurate speech-to-text translation
- Developers seeking lightweight models for diverse language support
- AI researchers focused on reducing ASR hallucinations
Business Model
Offer API and model licensing for integration into speech and translation platforms; provide custom fine-tuning services for enterprise clients.
Competitive Landscape
- OpenAI Whisper
- Meta SeamlessM4T
- Google Speech-to-Text
Implementation Challenges
- Integration complexity with existing systems
- Competition from large-scale LLM-based models
- Data privacy and compliance concerns
Validation Strategy
- Benchmark against Whisper and SeamlessM4T on multilingual datasets
- Pilot deployments with select enterprise customers
- Collect user feedback on speed and accuracy improvements
Research Paper Overview
Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST
Summary
This report introduces Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST). Built with a FastConformer encoder and Transformer decoder, it supports 25 languages primarily European. The model was trained on 1.7M hours of total data samples, including Granary and NeMo ASR Set 3.0, with non-speech audio added to reduce hallucinations for ASR and AST. We describe its two-stage pre-training and fine-tuning process with dynamic data balancing, as well as experiments with an nGPT encoder. Results show nGPT scales well with massive data, while FastConformer excels after fine-tuning. For timestamps, Canary-1B-v2 uses the NeMo Forced Aligner (NFA) with an auxiliary CTC model, providing reliable segment-level timestamps for ASR and AST. Evaluations show Canary-1B-v2 outperforms Whisper-large-v3 on English ASR while being 10x faster, and delivers competitive multilingual ASR and AST performance against larger models like Seamless-M4T-v2-large and LLM-based systems. We also release Parakeet-TDT-0.6B-v3, a successor to v2, offering multilingual ASR across the same 25 languages with just 600M parameters.