Idea
Inference runtime maximizing LLM throughput on Apple Silicon for faster, cost-effective on-device AI applications.
Research Paper
Core Innovation
This paper introduces BaseRT, a native Metal-based inference runtime that leverages Apple Silicon's unified memory and chip-specific optimizations. It outperforms existing frameworks by eliminating overhead from non-native abstractions and applying kernel fusion and custom dispatch logic, achieving best-in-class throughput across multiple LLM families and quantisation formats.
Why It Matters
As privacy, latency, and cloud cost concerns push AI inference to edge devices, BaseRT enables high-performance local LLM execution on Apple Silicon. This reduces reliance on cloud infrastructure, lowers operational costs, and improves user experience by delivering faster responses. It supports a broad range of models and quantisation formats, making it scalable across device generations and application needs.
Market Size (TAM)
$2–10B TAM for edge AI inference runtimes; $1–3B SAM from mobile and desktop AI application developers. Driven by rising demand for privacy-focused, low-latency AI and cost reduction in cloud inference.
Potential Customers & Pain Points
- AI app developers – Need efficient on-device LLM inference
- Enterprises – Require privacy-preserving AI with low latency
- Cloud providers – Seek to reduce inference costs
- Hardware OEMs – Want optimized software for Apple Silicon capabilities
Business Model
Open-source runtime with potential revenue from enterprise support, custom optimizations, and licensing for commercial deployments.
Competitive Landscape
- llama.cpp
- MLX
- Apple Core ML
- NVIDIA TensorRT
Implementation Challenges
- Adoption limited by Apple Silicon hardware penetration
- Competition from established inference runtimes and frameworks
- Need for continuous updates to support evolving LLM architectures
Validation Strategy
- Benchmark BaseRT against leading runtimes on various Apple Silicon devices
- Partner with AI app developers to integrate and test in real-world applications
- Collect performance and user feedback to guide iterative improvements
Research Paper Overview
BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal
Summary
BaseRT is a native Metal runtime optimized for large language model inference on Apple Silicon, delivering up to 1.56x higher throughput than existing solutions. It supports multiple model families and quantisation formats across all M-series devices, enabling efficient on-device LLM deployment with improved performance and lower latency.