How fine-tuning small open models can protect your SaaS margins
TL;DR: we used Specton to fine-tune an open source model. Replacing Gemini 3.1 Flash with the fine-tuned model reduced our AI costs by 62% without quality loss.
AI offers a big chance for SaaS products to innovate. But it also brings a challenge for keeping healthy margins. Before AI, SaaS products could grow without variable costs increasing. AI-powered features can cause variable costs to rise with each new user. These costs can be unpredictable, making planning challenging. ICONIQ's State of AI report (slide 20) shows AI costs make up 23% of revenue for some B2B SaaS companies.
Fine-tuning open source models presents an opportunity to leverage AI without hurting margins. We used Specton to fine-tune a model that performed at 88% of the quality level as Gemini Flash at less than 40% of the cost.
First, the basics. What are open models?
We could do a whole post talking about this, but let's keep it simple. "Open source models" are LLMs. Their source code and parameters are available for anyone. You can view, modify, and run them locally. In theory, you could replace your calls to Sonnet 4.6, for example, with calls to a similar-sized open model. While the model would be "free to use," you'd still have to pay a fixed cost to host the model.
Why don't people simply use off-the-shelf open models? Why do I need to fine-tune?
Many teams are starting to do this, especially for software development. But this doesn't always make sense. Open models usually lag a few months behind top-tier performance. So, they might not always match the quality of Anthropic or OpenAI models. Performance of open models will vary depending on the task.
We first tried to replace Gemini with a ready-made version of Llama. We didn't do any fine-tuning. This performed 80% below baseline. That's when we decided to do fine-tuning.
What is fine-tuning and how does Specton help?
Think of fine-tuning as "teaching" the model how to do a specific task really well. In our case, we wanted to use an open model for json extraction and classification.
We used Specton's proxy to begin observing how Gemini performed in production. Specton started using that data to fine-tune the open model. This process ran in the background. Meanwhile, our web app stayed fully functional in production.
Specton was doing 3 things:
1. Collecting what "great performance" looks like
2. Fine-tuning the open model with that data. This required no additional engineering time.
3. Testing the fine-tuned open model on our benchmarks
After a few cycles of this, Specton alerted us when it had a fine-tuned model it thought could replace Gemini.
The Results
This dropped our projected model costs by 10x. Once the fine-tuned model was ready, our costs would just be fixed cloud fees. These would be about $300 per month. Our savings will scale as traffic grows.
Comments / month
Gemini cost
Fine-tuned cost
1,000,000
$803
$300
5,000,000
$4,015
$300
10,000,000
$8,030
$300
The $300 would increase at a certain point after hitting capacity, but it would not scale linearly with usage. At the 10,000,000 comment point, we'd save $60K per year on AI by using Specton to fine-tune an open model.
AI cost reduction for SaaS. You only pay when we beat your baseline.
Product
© 2026 Specton LLC
No savings. No invoice.