Self-hosting LLMs: when to buy vs run your own

We run AI features across most of our products — NLP-to-SQL chat, on-chain anomaly classification, automated caption generation, RAG for financial documents. At our scale (~1,000-5,000 inference requests per day), the decision to self-host or use an API comes down to one number: the break-even threshold.

Our analysis showed that self-hosting only beats APIs above ~1,000-1,600 requests per day AND when you need 70B+ parameter quality. Below that, APIs are cheaper and maintenance-free:

100 req/day → Gemini/Groq free tier ($0) 1K req/day → Groq Llama 8B ($~1/mo) 10K req/day → Local RTX 3090 ($~61/mo) vs DeepSeek API ($~168/mo) 100K req/day → 4x RTX 4090 ($~468/mo) vs DeepSeek API ($~1,680/mo)

We run a LiteLLM proxy with a fallback chain: local Ollama → Gemini free tier → OpenRouter free → OpenRouter paid → Claude/GPT-4o last resort. This means simple classification goes through the free tiers and only the complex reasoning queries hit the paid Claude endpoint.

For single-user and dev workloads, Ollama is fine. For production multi-user, vLLM delivers 793 TPS vs Ollama's 41 TPS — the same hardware, 19x throughput. The switch to vLLM cost one afternoon of setup and saved us from having to buy a second GPU.

The architecture lesson: don't treat inference as a binary self-host-vs-API choice. Run a gateway with a fallback chain, route each request to the cheapest tier that can handle its complexity, and only buy hardware when the math clearly pencils out.