All insightsThe Hidden Cost
LLM Cost Engineering
The Hidden Cost
of Running AI at Scale
What costs $0.002 per call at demo scale costs $200,000 a year at production scale. Most teams find out too late. Cost is a product decision, not an ops afterthought.
Cost Accumulation — Illustrative
$0.4271 accumulated this session
4,182 API calls · 6.7M tokens processed
$0.002 → $200k
Per call → per year
5 Levers
You can actually control
10–100×
Model tier cost difference
Section 01
The 4 Dimensions of LLM Cost
Every API call is a product of four variables. Miss one and your cost model is wrong before you ship.
📥
Input Tokens
System prompt + user message + conversation history + retrieved context. Often 80% of total token spend — and teams rarely measure it upfront.
System prompt
~55%
Context / RAG
~30%
User message
~15%
📤
Output Tokens
Completion text generated by the model. Costlier per token than input — typically 3–5× more expensive — and length is hard to predict.
Cost ratio: Input vs Output
Input
$1×
Output
$4×
4×
typical
multiplier
multiplier
🧠
Model Tier
Choosing the wrong model tier is the single biggest lever on your bill. Flagship vs. mini vs. fine-tuned: a 10–100× cost difference for equivalent tasks.
Flagship
100%
Mini
6%
Fine-tuned
3%
🔄
Calls per User per Session
The multiplier nobody models upfront. A feature with 3 LLM calls per interaction at 10k DAU is 30,000 calls/day — before you've thought about retry logic or fallbacks.
Cost scales superlinearly with adoption — model it early
Typical Token Breakdown — Per Single API Call
System Prompt (input)
Context / RAG (input)
User Message (input)
Completion (output · 4× cost/token)
Section 02
Latency: What Users Actually Feel
Response time isn't a single number. It's a sequence of events — and only some of them are in your control.
LLM Response Timeline
⚡ Time to First Token (TTFT)
This is the number that determines whether your product "feels instant." Users tolerate long completions — they don't tolerate a blank screen. Optimize TTFT first.
<200ms
Feels instant
>800ms
Feels slow
🌊 Streaming vs. Batch
Streaming doesn't make the model faster — it makes users perceive it as faster. Showing the first word at 200ms, even when completion takes 3s, dramatically reduces perceived latency.
3s total time → feels like 0.3s thanks to streaming
📊 p99 Is Your Real SLA
Your average latency (p50) looks great on a dashboard. But 1 in 100 users hits p99 — and that's the person writing the angry tweet. Design, monitor, and alert on p99.
🧊 Cold Starts, Rate Limits & Queues
At peak load, cold starts add 200–800ms, rate limits cause queuing (invisible until it blows up), and retry logic can compound latency 3–5× under pressure.
Cold start+200–800ms
Rate limit queue+1–5s
Retry cascade3–5× latency
Section 03
The 5 Cost Levers You Actually Control
You can't control pricing. You can control everything else. Here's where to start.
1
Prompt Compression
Trim system prompts. Remove redundant instructions. Compress retrieved context with summarization before injection. Most system prompts can be cut 40–60% with no quality loss — teams just never audit them.
2,400 tokens/call→960 tokens/call after compression audit
60%cost reduction
2
Model Routing
Route intelligently: use flagship models only where quality truly requires it — complex reasoning, high-stakes decisions. Use mini models for classification, FAQ answers, formatting tasks. Build a router classifier with a cheap model to decide.
100% flagship calls→20% flagship, 80% mini
75%cost reduction
3
Semantic Caching
Cache LLM responses by embedding similarity, not exact string match. “What are your hours?” and “When are you open?” are the same question. At scale, 20–40% of queries are near-duplicates — every cache hit is a free response.
100,000 API calls/day→60,000 real calls (40% cache hit rate)
40%on cached
4
Output Constraints
Set max_tokens strictly. Use structured outputs (JSON mode) — models generate less preamble. Avoid “explain your reasoning” unless you actually need it. Output tokens cost 4× input tokens.
~800 output tokens avg→~320 tokens with max_tokens + structured output
55%on output cost
5
Async Batching
Not every LLM call needs a real-time response. Report generation, nightly summarization, classification pipelines, and email drafts can all be batched. Batch APIs typically offer 50% cost reduction with a longer SLA (minutes vs. milliseconds).
Real-time inference pricing→Batch API pricing for async workloads
50%batch discount
Combined levers — realistic production scenario
85–92%
cost reduction achievable before scaling, vs. naive implementation with no cost engineering.
Section 04
Cost vs. Quality Trade-off Matrix
Not all tasks need the smartest model. Map your use cases to the right tier before you write a single line of prompt engineering.
Model Cost vs. Output Quality — Use Case Quadrant
The 80/20 Rule of Model Selection
80% of production LLM tasks don't require a flagship model. Classification, formatting, FAQ retrieval, and basic summarization all perform well on mini models — at a fraction of the cost.
The Overkill Tax
Defaulting to a flagship model for every call is common. It's also a 10–100× tax on tasks that a mini or fine-tuned model handles identically. The overkill zone costs teams hundreds of thousands of dollars annually.
Final Thought
Cost is the LLM metric nobody tracks until it's a problem.
Start tracking on day one. Instrument every call. Log token counts, model used, latency, and cost per call. The teams who ship AI profitably aren't smarter — they just started measuring early.
✓Log tokens + cost on every LLM call from Day 1
✓Set p99 latency budgets before writing prompt code
✓Audit system prompt tokens — cut the fat before launch
✓Default to mini models, upgrade only where justified
✓Model your cost at 10k, 100k, 1M users before building
Instrument early. Route smart. Ship AI that scales profitably.