Idea
A prompting framework optimizing few-shot example selection to improve large language model accuracy for software requirement classification.
Research Paper
Core Innovation
This paper identifies and addresses the over-prompting problem in LLMs, showing that excessive few-shot examples can degrade performance. It introduces a combined selection approach using TF-IDF and stratified sampling to find the optimal number of examples, improving classification accuracy with fewer prompts. This challenges the prior assumption that more examples always enhance LLM performance.
Market Size (TAM)
$2–10B TAM for AI-driven software engineering tools; $1–2B SAM from enterprises adopting LLMs for requirement analysis. Driven by increasing LLM integration in software development and demand for efficient prompt engineering.
Potential Customers & Pain Points
- AI developers struggling with prompt design inefficiencies
- Software engineers needing accurate requirement classification
- Enterprises using LLMs for software analysis facing performance drops from over-prompting
Business Model
Offer a SaaS platform or API that optimizes few-shot prompt construction for LLMs, targeting software engineering teams and AI developers with subscription and usage-based pricing.
Competitive Landscape
- OpenAI
- Cohere
- AI21 Labs
Implementation Challenges
- Complexity in generalizing optimal prompt sizes across domains
- Integration challenges with existing LLM platforms
- User education on prompt engineering best practices
Validation Strategy
- Conduct pilot studies with software development teams using real-world datasets
- Benchmark performance improvements against standard prompting methods
- Iterate on selection algorithms based on user feedback and model updates
Research Paper Overview
The Few-shot Dilemma: Over-prompting Large Language Models
Summary
This paper investigates the phenomenon of over-prompting in large language models, where too many few-shot examples in prompts reduce performance. It proposes a prompting framework using random sampling, semantic embedding, and TF-IDF selection methods, evaluated across multiple LLMs. Experiments on software requirement classification datasets reveal that an optimal number of domain-specific examples improves accuracy and avoids performance degradation, surpassing state-of-the-art results by 1%.