Idea
A platform that accelerates semantic predicate queries on large document collections by combining LLMs with efficient proxy models.
Research Paper
Core Innovation
This paper introduces ScaleDoc, which separates predicate execution into offline semantic representation and online lightweight proxy filtering. It uses contrastive learning to train the proxy model for reliable decision scores and an adaptive cascade to optimize filtering policies, significantly reducing LLM calls while maintaining accuracy.
Market Size (TAM)
$2–10B TAM for AI-powered semantic search and document analysis platforms; $1–3B SAM from enterprises and analytics firms adopting scalable LLM-based solutions. Driven by growing unstructured data volumes and demand for cost-efficient AI inference.
Potential Customers & Pain Points
- Enterprises managing large unstructured document repositories needing fast semantic search
- Data analytics firms facing high LLM inference costs
- AI service providers requiring scalable semantic filtering
- Research institutions processing massive text datasets with limited compute resources
Business Model
Subscription-based SaaS platform charging per document volume and query throughput with enterprise support and customization options.
Competitive Landscape
- Pinecone
- Weaviate
- Cohere
Implementation Challenges
- High initial cost of LLM-based offline processing
- Complexity in tuning proxy models for diverse queries
- Integration challenges with existing data pipelines
Validation Strategy
- Pilot deployment with enterprise document management teams
- Benchmarking speed and cost savings against direct LLM querying
- Iterative proxy model tuning based on real-world query workloads
Research Paper Overview
ScaleDoc: Scaling LLM-based Predicates over Large Document Collections
Summary
Predicates are foundational components in data analysis systems. However, modern workloads increasingly involve unstructured documents, which demands semantic understanding, beyond traditional value-based predicates. Given enormous documents and ad-hoc queries, while Large Language Models (LLMs) demonstrate powerful zero-shot capabilities, their high inference cost leads to unacceptable overhead. Therefore, we introduce ScaleDoc, a novel system that addresses this by decoupling predicate execution into an offline representation phase and an optimized online filtering phase. In the offline phase, ScaleDoc leverages a LLM to generate semantic representations for each document. Online, for each query, it trains a lightweight proxy model on these representations to filter the majority of documents, forwarding only the ambiguous cases to the LLM for final decision. Furthermore, ScaleDoc proposes two core innovations to achieve significant efficiency: (1) a contrastive-learning-based framework that trains the proxy model to generate reliable predicating decision scores; (2) an adaptive cascade mechanism that determines the effective filtering policy while meeting specific accuracy targets. Our evaluations across three datasets demonstrate that ScaleDoc achieves over a 2x end-to-end speedup and reduces expensive LLM invocations by up to 85%, making large-scale semantic analysis practical and efficient.