Everyone talks about "big tasks" being expensive. The usual framing is: "tell an agent to do a large task and it burns a huge context window, so it costs a fortune and takes forever." The implication is that scaling an agent's scope means scaling its bill linearly with how much code and context you throw at it.
That's true for a naive agent. It's not true for Spectral. In steady state, on a long, multi-turn agentic session, Spectral serves roughly 97% of its prompt tokens from cache. That changes the economics of big tasks entirely: the thing that used to make them slow and expensive — a huge, re-sent context — becomes nearly free.
This post explains how that works. It's not a magic flag. It's four concrete engineering decisions that stack together.
The naive problem
An agent isn't one call. It's a loop: read your message → maybe read files → call a tool → see the result → think again → maybe call another tool → write code → check the result → repeat. Each of those turns is a separate LLM request.
On every request, the agent has to send all of the context it has accumulated so far: the system prompt, the tool definitions, the conversation history, the repo context, and every tool result. On turn 50 of a big task, that's a lot of tokens.
With no caching, every one of those repeats counts as fresh input. So cost and latency scale with turn count × context size. On a big task — hundreds of tool calls, dozens of files read, a long multi-turn session — that's exactly the worst case.
The fix is to stop paying for context you've already paid for once.
What prefix caching is
Modern LLM providers implement automatic prefix caching. The idea is simple: if the beginning of a request matches the beginning of a previous request in the same session, the provider reuses the cached representation of that prefix instead of reprocessing it.
OpenAI-style automatic prefix caching is a longest-common-prefix match against the previous request in the session, with a ~1024-token minimum granularity. You don't mark anything; the provider decides what's cacheable. Your only job is to make sure the prefix is actually byte-stable between turns — because if a single byte at the front changes, every token after it can't be reused.
So the game is: keep the beginning of every request identical, and only the tail changes.
The three turns as a diagram — the dim shared prefix stays byte-stable from turn to turn, while only the bright tail grows:
As long as the prefix is identical, the provider re-reads it from cache on each turn. Only the new tail is billed at full input rate.
The four decisions that make it work
Caching is a provider feature, but it's only useful if the agent actually produces a stable prefix. Spectral does four things to guarantee that.
1. A byte-stable system prompt
The system prompt is built deterministically: context files and skills are sorted before they're inserted, so the prompt is byte-stable across rebuilds. A stable ordering means a stable prefix — and a stable prefix means stable cache breakpoints.
This is deliberate. The whole point is that the prompt doesn't subtly reorder itself between turns and silently invalidate the cache.
2. Append-only conversation history
The conversation history is append-only and passed through verbatim. Turns are never rewritten or reordered in place; new messages are just appended. Tool output is compressed once when it's inserted, then left untouched.
That matters because the entire history up to the current turn is part of the reusable prefix. If the agent rewrote earlier turns, the prefix would shift and the cache would miss. It doesn't.
3. Session affinity (the unsung hero)
This is the one most people miss. A prefix cache only works if consecutive requests actually land on the same provider replica. If your turns get routed to different replicas, each one has an empty cache.
Spectral pins the session: it sends a stable per-session key with every request, so the provider routes every turn of a conversation to the same replica, maximising prompt-cache hits for large multi-turn agentic prompts. This is the difference between "caching that sometimes works" and "caching that reliably works."
4. The right pricing model
This is where the small percentage becomes a big number. The cache read price is a fraction of the full input price — typically 0.1× (for Codex, cache-read is 0.5 vs 5.0 for input; across the model table it's consistently 0.1×). The provider only charges full price once, on the first (uncached) turn. On every subsequent turn, the cached prefix is billed at the read rate.
So a 97% cache hit rate on prompt tokens isn't a trivial discount — it means the bulk of every request's input is effectively a tenth of its usual cost.
What this means for large tasks
Here's the counter-intuitive part: with a high cache hit rate, cost and latency stop scaling with context size. They scale with the number of turns and the size of the fresh tail, which is comparatively small.
On a big task — hundreds of tool calls, dozens of files read, a long multi-turn session — most of what the agent sends each turn is the accumulated context that's already been paid for once. Re-reading it from cache is nearly free and near-instant. The expensive part shrinks to just the new work on each turn.
That's why large tasks aren't proportionally more expensive. The replayed context is cached; the agent pays for the delta. The snowball effect inverts.
The honest caveats
Caching is not unconditional. It depends on the prefix staying stable, which means:
- Context compaction breaks it. If a session is compacted by rewriting or reframing the history (rather than trimming only the tail), the prefix shifts and you lose the cache on the next turn. Trimming the tail is safe; rewriting the head is not.
- The hit rate is a steady-state number. Turn one of a fresh session is a cold miss. The 97% figure is what you see on an established, long-running session where the prefix is large and dominant.
- Cache granularity matters. There's a minimum token threshold (~1024 tokens) before caching kicks in, so very short sessions may not benefit meaningfully.
But when you're doing the thing the agent was built for — big, long-running tasks with lots of context — these conditions are exactly the normal case, and the hit rate sits in the high nineties.
TL;DR
- An agent is a loop, and every turn resends its context.
- Naively, that makes large tasks scale with turns × context size.
- LLM providers offer automatic prefix caching — reuse the head, pay for the tail.
- Prefix caching only pays off if the prefix is byte-stable and requests hit the same replica.
- Spectral does both: a deterministic system prompt, append-only history, and per-session pinning.
- Result: in steady state ~97% of prompt tokens come from cache, billed at 0.1× — so big tasks get fast and cheap instead of slow and expensive.
The lesson for building agentic tools: don't treat the context window as a bill you pay every turn. Treat it as a stable prefix you pay for once and cache from then on. That single shift is what makes very large tasks practical.