Idea
A fine-tuning framework that adapts activation functions in pretrained transformers for efficient, high-quality model updates.
Research Paper
Core Innovation
This paper introduces NoRA, the first PEFT method that adapts nonlinear activation functions instead of only weight matrices. It uses learnable rational functions with low-rank updates and a group-wise design to localize adaptation and improve stability. This approach achieves competitive or better performance than full fine-tuning while updating a fraction of parameters, and can be combined with existing methods like LoRA for further gains.
Market Size (TAM)
$20–50B TAM for AI model fine-tuning and adaptation; $2–10B SAM from enterprises deploying large language and vision models. Driven by growing demand for cost-efficient model updates and expanding use of pretrained transformers.
Potential Customers & Pain Points
- AI Researchers Seeking Efficient Model Adaptation
- ML Engineers Needing Parameter-Efficient Fine-Tuning
- Enterprises Deploying Large Language and Vision Models with Limited Compute
- Developers Struggling with Costly Full Fine-Tuning
- Organizations Requiring Improved Model Performance with Minimal Parameter Updates
Business Model
Offer NoRA as an open-source library with enterprise support and consulting services; license advanced features for commercial use; provide cloud-based fine-tuning APIs.
Competitive Landscape
- LoRA
- AdapterHub
- BitFit
Implementation Challenges
- Integration Complexity with Existing Frameworks
- Limited Awareness of Activation Function Tuning Benefits
- Potential Stability Challenges in Large-Scale Models
Validation Strategy
- Benchmark NoRA on diverse transformer models and datasets
- Demonstrate cost and accuracy benefits versus full fine-tuning and LoRA
- Pilot deployments with AI-focused enterprises for real-world feedback
Research Paper Overview
Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning
Summary
Existing parameter-efficient fine-tuning (PEFT) methods primarily adapt weight matrices while keeping activation functions fixed. We introduce NoRA, the first PEFT framework that directly adapts nonlinear activation functions in pretrained transformer-based models. NoRA replaces fixed activations with learnable rational functions and applies structured low-rank updates to numerator and denominator coefficients, with a group-wise design that localizes adaptation and improves stability at minimal cost. On vision transformers trained on CIFAR-10 and CIFAR-100, NoRA matches or exceeds full fine-tuning while updating only 0.4% of parameters (0.02M), achieving accuracy gains of +0.17% and +0.27%. When combined with LoRA (NoRA++), it outperforms LoRA and DoRA under matched training budgets by adding fewer trainable parameters. On LLaMA3-8B instruction tuning, NoRA++ consistently improves generation quality, yielding average MMLU gains of +0.3%–0.8%, including +1.6% on STEM (Alpaca) and +1.3% on OpenOrca. We further show that NoRA constrains adaptation to a low-dimensional functional subspace, implicitly regularizing update magnitude and direction. These results establish activation-space tuning as a complementary and highly parameter-efficient alternative to weight-based PEFT, positioning activation functions as first-class objects for model adaptation.