The usual framing: "give an agent a big task and it burns a huge context window, so it costs a fortune." True for a naive agent — not for Spectral. In steady state, on a long multi-turn session, Spectral serves roughly 97% of its prompt tokens from cache. Big tasks stop scaling with context size and start scaling with the fresh work per turn. Four engineering decisions make that happen — no magic flag.
The naive problem
An agent is a loop: read → call a tool → think → repeat. Every turn is a separate LLM request that resends all accumulated context — system prompt, tool definitions, history, every tool result. With no caching, cost and latency scale with turn count × context size, which is exactly the worst case for big tasks. The fix: stop paying for context you've already paid for once.
What prefix caching is
Providers implement automatic prefix caching: if the beginning of a request matches the previous request in the session, the cached prefix is reused instead of reprocessed (longest-common-prefix match, ~1024-token minimum granularity). Your only job is keeping the prefix byte-stable — change one byte at the front and every token after it is uncacheable. Keep the head identical; only the tail grows:
The four decisions that make it work
- Byte-stable system prompt. Context files and skills are sorted before insertion, so the prompt never subtly reorders itself between turns.
- Append-only history. Turns are passed through verbatim — never rewritten or reordered — so the reusable prefix never shifts.
- Session affinity. The unsung hero: a stable per-session key pins every turn of a conversation to the same provider replica, because a prefix cache on another replica is an empty cache.
- The right pricing. Cache reads cost ~0.1× of full input (e.g. 0.5 vs 5.0 for Codex), so a 97% hit rate means the bulk of every request's input is effectively a tenth of its usual cost.
What this means for large tasks
Cost and latency stop scaling with context size. On a task with hundreds of tool calls and dozens of files read, most of each request is context that was already paid for once and is now replayed nearly free and near-instantly. The agent pays for the delta. The snowball effect inverts.
Honest caveats
- Compaction that rewrites history breaks the prefix. Trimming the tail is safe; rewriting the head is not.
- 97% is a steady-state number — turn one of a fresh session is a cold miss.
- ~1024-token granularity means very short sessions see little benefit.
For the workload agents were built for — big, long-running tasks with lots of context — these are exactly the normal case. The lesson: don't treat the context window as a bill you pay every turn; treat it as a stable prefix you pay for once.