Bill Tanker
Crazy Unicorns
When enterprises first adopt LLM technology, costs are manageable — a few hundred dollars per month for API calls during development and testing. But as usage scales from a pilot team to the entire organization, costs can grow exponentially. We’ve seen companies go from $500/month to $50,000/month in a single quarter without any corresponding increase in value. The problem isn’t the technology — it’s the lack of cost architecture. Here are the strategies we implement to keep LLM costs predictable and proportional to value delivered.
The single most effective cost optimization is semantic caching — storing LLM responses and serving cached results for semantically similar queries. Unlike exact-match caching, semantic caching uses embedding similarity to identify queries that are different in wording but identical in intent. ‘What’s our refund policy?’ and ‘How do I get a refund?’ should return the same cached response.
We implement semantic caching as a layer in front of the LLM API. Each incoming query is embedded and compared against the cache. If a sufficiently similar query exists (we typically use a cosine similarity threshold of 0.95), the cached response is returned without calling the LLM. In enterprise support and knowledge base applications, semantic caching typically reduces LLM API calls by 40-60%. The cache is invalidated when underlying data changes, ensuring responses stay current.
Not every query needs the most expensive model. A simple factual lookup doesn’t require GPT-4 — a smaller, faster, cheaper model handles it just as well. Intelligent model routing analyzes incoming queries and routes them to the most cost-effective model that can handle the task. We implement a lightweight classifier that evaluates query complexity based on length, topic, required reasoning depth, and historical accuracy data.
In practice, we find that 60-70% of enterprise queries can be handled by smaller models (GPT-4o-mini, Claude Haiku, or fine-tuned smaller models) without measurable quality degradation. Only complex reasoning, multi-step analysis, and creative generation tasks need the full-size models. This tiered approach typically reduces costs by 50-65% compared to routing everything through the most capable model.
LLM costs are directly proportional to token usage, and most prompts are far more verbose than they need to be. We systematically optimize prompts by removing redundant instructions, compressing few-shot examples, using structured output formats that reduce response length, and implementing dynamic context selection that includes only the most relevant information in the prompt.
A common pattern we see is RAG systems that stuff 10-15 retrieved documents into the context when 2-3 would suffice. We implement a relevance-based context budget: retrieved documents are ranked by relevance, and only enough documents to fill the context budget are included. The budget is calibrated per use case — some tasks need more context, others need less. This alone can reduce token usage by 30-40% in RAG applications.
Many LLM workloads don’t need real-time responses. Document classification, content moderation, data extraction from archives, and report generation can all be batched and processed during off-peak hours. Most LLM providers offer significant discounts for batch processing — OpenAI’s Batch API, for example, offers 50% cost reduction for jobs that can tolerate 24-hour turnaround.
We design systems with explicit real-time and batch processing paths. User-facing interactions go through the real-time path with full model capabilities. Background processing tasks are queued and processed in batches during low-usage periods. This not only reduces costs but also smooths out usage patterns, making monthly bills more predictable.
Cost optimization without monitoring is guesswork. We implement granular cost tracking that attributes every LLM API call to a specific team, application, and use case. Dashboards show real-time spend, cost per query, cost per user, and cost trends. Budget alerts fire when spending exceeds thresholds, and hard limits prevent runaway costs from bugs or abuse.
The most valuable metric we track is cost per successful outcome — not cost per API call. A cheaper model that produces lower-quality results and requires human correction might cost more overall than a more expensive model that gets it right the first time. We optimize for total cost of the outcome, not just the API bill.
Implement optimizations in this order for maximum impact:
LLM cost optimization is an ongoing practice, not a one-time project. If your AI infrastructure costs are growing faster than the value it delivers, let’s talk about building a sustainable cost architecture.
We build production-ready AI systems. Book a strategy call to discuss your requirements.
Hello! How can I help?
