Part 12 of How to Make Your Model Fast (Part 11: Serving and MLOps | Part 13: Conclusion)
An agent is a distributed system whose latency and cost are dominated by the number of sequential model calls, not by how fast any single call runs. This chapter builds the arithmetic: a latency model over a call chain, a token cost model that shows why naive conversation history grows quadratically, the memory and recall trades in a retrieval index, cascades and caches that cut the bill, and the critical path view that tells you what to parallelise. By the end you should be able to look at an agent graph and say where the seconds and the pounds are going, and which single change removes the most of both.
I spent a stretch of my career leading the design and implementation of LLM agent systems at a stealth startup: retrieval augmented generation, LangChain and LlamaIndex, infrastructure on Vertex AI, everything containerised with Docker and scheduled on Kubernetes. What surprised me was how little of the work was about prompts. Almost all of it was the same work as the rest of this book: counting, budgeting, and finding which term in a sum dominates.
An agent is a distributed system whose nodes happen to be model calls instead of microservices and whose currency happens to be tokens instead of FLOPs, so the discipline from Part 1 transfers intact. The headline result is that the depth of your call chain is what you are paying for, in seconds and in money, and it is the thing nobody measures.
Take an agent that makes \(N\) sequential model calls. Step \(i\) sends \(P_i\) prompt tokens and generates \(G_i\) output tokens, then possibly calls a tool. The total wall clock time is
\[T_{\text{total}} \;=\; \sum_{i=1}^{N} \left( \frac{P_i}{R_{\text{prefill}}} \;+\; G_i \, t_{\text{dec}} \;+\; T_{\text{tool},i} \;+\; T_{\text{ovh}} \right)\]Four terms, and they behave completely differently.
Prefill processes the whole prompt in parallel. It is compute bound and fast: a hosted mid-size model chews through thousands of prompt tokens per second.
Decode emits one token at a time, each token requiring a read of the entire weight set out of memory. It is bandwidth bound, and \(t_{\text{dec}}\) is essentially fixed per token regardless of how clever you are.
Tool time is whatever your search index, database or third party API takes. Often smaller than people assume, occasionally catastrophic.
Overhead is the per-call tax: connection setup, serialisation, queueing at the provider, tokenisation, framework bookkeeping. It sounds trivial. At 150 ms per call and twelve calls it is 1.8 seconds, more than most teams’ entire budget for a page load.
Notice what is missing from the sum: any term that shrinks when you buy a better model. \(N\) is set entirely by you.
All numbers below are illustrative assumptions in a plausible range for a hosted mid-size model, not measurements: \(R_{\text{prefill}} = 10{,}000\) tokens per second, \(t_{\text{dec}} = 20\) ms per token, \(T_{\text{ovh}} = 150\) ms, and 200 ms of tool time after each non-final step.
Chain A is a tight three-step pipeline: retrieve, reason, answer. Prompts of 2,000, 3,000 and 3,500 tokens; outputs of 150, 200 and 400 tokens.
Chain B is a twelve-step ReAct loop doing the same job. The prompt starts at 2,000 tokens and grows by 400 per step as observations accumulate, each of the first eleven steps emits 120 tokens of thought plus a tool call, and the twelfth writes a 400-token answer. Its prefill is an arithmetic series: eleven prompts from 2,000 to 6,000 tokens is \(11 \times 4{,}000 = 44{,}000\), plus 6,400 for the final step, so 50,400 tokens or 5.04 s. Decode is \(11 \times 120 + 400 = 1{,}720\) tokens at 20 ms, or 34.4 s.
Latency decomposition for a three-step and a twelve-step agent doing the same job, under the illustrative assumptions above.
| Configuration | Prefill (s) | Decode (s) | Tools (s) | Overhead (s) | Total (s) |
|---|---|---|---|---|---|
| 3 steps, 50 tok/s decode | 0.85 | 15.00 | 0.40 | 0.45 | 16.70 |
| 12 steps, 50 tok/s decode | 5.04 | 34.40 | 2.20 | 1.80 | 43.44 |
| 12 steps, 100 tok/s decode | 5.04 | 17.20 | 2.20 | 1.80 | 26.24 |
| 3 steps, 100 tok/s decode | 0.85 | 7.50 | 0.40 | 0.45 | 9.20 |
Read the middle two rows together. Doubling decode speed takes the twelve-step chain from 43.4 s to 26.2 s, a 1.66x win, and it is the expensive option: a different model, a different provider, or a serving stack rewrite. Cutting twelve steps to three takes it from 43.4 s to 16.7 s, a 2.60x win, and it is a control-flow change you can ship on a Tuesday.
The twelve-step chain on a model twice as fast is still slower than the three-step chain on the original model. Depth is the dominant term.
A second, quieter effect hides in that table. Prefill for chain B is 5.04 s against 0.85 s for chain A, because every step re-reads the accumulated transcript. Depth inflates prompt length, and prompt length is what you pay for. The two effects compound.
Takeaway: Total agent latency is a sum over sequential steps, and the step count is the term you control most directly. Removing four model calls from a chain almost always beats making every call twice as fast, and it is far cheaper to do.
Cost has the same shape as latency, with prices in place of rates:
\[C \;=\; \sum_{i=1}^{N} \left( P_i \, c_{\text{in}} \;+\; G_i \, c_{\text{out}} \right)\]The trap is in \(P_i\). Nearly every agent framework appends by default: each turn’s user message, assistant reply, tool call and tool result are concatenated onto the transcript and re-sent. If the system prompt and tool schemas are \(s\) tokens and each turn adds \(a\), the input tokens over \(K\) turns are
\[\sum_{k=1}^{K} \big( s + (k-1)a \big) \;=\; K s \;+\; a \,\frac{K(K-1)}{2}\]That second term is \(O(K^2)\). A conversation twice as long costs roughly four times as much.
The fix is to stop treating the transcript as the state. Three techniques, in increasing order of how much I like them:
Windowing keeps the last \(w\) turns verbatim and drops the rest. Cheap, trivial, and it forgets what the user told you at turn two.
Summarisation periodically folds older turns into a running summary with a small model call. It bounds growth, but summaries lose detail unpredictably, and a summary of a summary drifts.
Structured state is the one that works. Instead of a transcript, carry a typed object: the task, the constraints gathered, the entities resolved, the steps completed, the open questions. Each turn updates fields and the prompt renders the object, not the history. Bounded by construction, inspectable in a trace, and the difference between an agent you can debug and one you cannot.
Illustrative prices, clearly labelled as such: $3.00 per million input tokens and $15.00 per million output tokens. Round numbers in the general region of mid-tier hosted models, used only to make the arithmetic concrete. Substitute your own.
The session: a support agent with a 1,200-token system and tool schema block, running 20 turns. Each turn has an 80-token user message, a 220-token reply, and 1,500 tokens of retrieved context.
Under naive append, each turn adds \(a = 80 + 220 + 1{,}500 = 1{,}800\) tokens, and turn \(k\) sends \(1{,}200 + 1{,}580 + 1{,}800(k-1)\) input tokens. Over 20 turns that is \(20 \times 2{,}780 + 1{,}800 \times 190 = 397{,}600\) input tokens and 4,400 output tokens.
Under management, the prompt is the 1,200-token system block, a 300-token state object, the last two turns verbatim (600), this turn’s retrieval (1,500) and user message (80): 3,680 tokens, constant with \(k\). Add a summarisation call every four turns, five calls at roughly 900 input and 300 output tokens each.
Session cost under naive transcript append against managed context, at $3.00 per million input and $15.00 per million output tokens (illustrative).
| Turns | Naive input tokens | Naive total cost | Managed input tokens | Managed total cost | Ratio |
|---|---|---|---|---|---|
| 20 | 397,600 | $1.26 | 78,100 | $0.32 | 3.9x |
| 40 | 1,515,200 | $4.68 | 156,200 | $0.65 | 7.2x |
| 80 | 5,910,400 | $18.00 | 312,400 | $1.29 | 13.9x |
The ratio doubles every time session length doubles, exactly what \(O(K^2)\) against \(O(K)\) predicts. At 80 turns you are paying eighteen dollars for one conversation, and roughly 95% of those input tokens are text the model has already read.
Two things worth noticing. Output tokens cost five times input here but are under 6% of the naive bill, because agents read far more than they write: optimise input first. And the managed version pays for summarisation calls and still wins fourfold at 20 turns.
Takeaway: Naive transcript append makes session cost grow with the square of the turn count. Replacing the transcript with a bounded structured state turns that into linear growth, and the crossover where it pays off is around turn five, not turn fifty.
Most of what people call an agent is, underneath, a retrieval system with a language model stapled to the end. Getting the retrieval right is where the quality is, and it is mostly an information retrieval problem rather than a language modelling one.
Chunk size is the first decision, and the two halves of the system pull it in opposite directions.
The retriever wants small chunks. An embedding is one fixed-length vector for the whole chunk, so pushing 2,000 tokens covering four topics through an encoder gives you the average of four topics, which is close to nothing in particular. Signal is diluted and recall falls.
The reader wants large chunks. A 100-token fragment out of context may be unusable: the pronoun has no antecedent, the number has no unit. To compensate you retrieve more chunks, which inflates \(P_i\) and drags you back into the cost model above.
In practice I land between 200 and 500 tokens with 10 to 20% overlap, split on structural boundaries (headings, list items, paragraphs) rather than a character count, because splitting mid-sentence reliably produces garbage embeddings. The pattern that resolves the tension is small-to-big: search over small chunks, return the enclosing section to the reader.
Dimensionality is the embedding choice with hard consequences, because it sets your memory bill. For \(N\) vectors of dimension \(d\) at \(b\) bytes per component:
\[M_{\text{vec}} \;=\; N \times d \times b\]Take a corpus of 2 million chunks at \(d = 1024\) in fp32. That is \(2 \times 10^6 \times 1024 \times 4 = 8.19\) GB of raw vectors. Add an HNSW graph with \(m = 32\) neighbours per node, a layer-0 degree of \(2m = 64\) and 4-byte identifiers:
\[M_{\text{graph}} \;\approx\; N \times 2m \times 4 \;=\; 2 \times 10^6 \times 256 \;=\; 512 \text{ MB}\]Upper layers add a few percent more. Call the index 8.7 GB.
The levers, all of which I have used:
Index comparison at 2 million vectors, \(d = 1024\), fp32, assuming 20 GB/s of effective streaming bandwidth for the scan. Latencies and recall figures are illustrative order-of-magnitude estimates, not benchmarks.
| Index | Extra memory | Vectors scanned per query | Illustrative p50 latency | Typical recall@10 | Build cost |
|---|---|---|---|---|---|
| Flat (exact) | none | 2,000,000 | ~410 ms | 1.000 | none |
| IVF, nlist 1,414, nprobe 16 | ~6 MB of centroids | ~22,600 | ~5 ms | 0.90 to 0.97 | k-means over a sample |
| IVF-PQ, 64 x 8-bit codes | 128 MB total, replaces raw | ~22,600 | ~2 ms | 0.75 to 0.90 | k-means plus codebooks |
| HNSW, m = 32, efSearch 64 | 512 MB of graph | ~2,000 to 5,000 | ~1 to 3 ms | 0.95 to 0.99 | O(N log N) inserts, hours |
The flat number is a roofline argument in disguise. An exact scan is a dot product against every vector: 2 FLOPs per 4-byte element, an arithmetic intensity of 0.5 FLOP per byte. That is deep in the memory bound region, so query time is set not by your FLOP rate but by how fast you stream 8.19 GB. At 20 GB/s, 410 ms, and no amount of vectorisation helps. The IVF row follows: probing 16 of 1,414 cells touches 1.1% of the data, 93 MB, about 5 ms.
My default is HNSW when the index fits in memory and updates are rare, IVF-PQ when it does not. Flat is not a joke: below roughly 100,000 vectors an exact scan is a few milliseconds with perfect recall and nothing to tune.
Dense retrieval fails predictably on exact strings: error codes, part numbers, API method names, rare acronyms, surnames. The embedding of ERR_4471 sits near every other error code, because that is what a semantic space is for. BM25 nails these and fails at paraphrase, which is where dense wins.
Run both and fuse the ranked lists. Reciprocal Rank Fusion needs no score calibration between the two systems and is one line of code:
\[\text{RRF}(c) \;=\; \sum_{r \in R} \frac{1}{\kappa + \text{rank}_r(c)}\]with \(\kappa = 60\).
Then rerank. The first stage is a bi-encoder: query and document embedded separately, meeting only in a dot product. A cross-encoder puts both in one forward pass and lets attention run across them, judging relevance in a way a dot product structurally cannot. It is far too slow for the whole corpus, which is why it goes second: retrieve 50 candidates cheaply, rerank, keep 5.
Reranking is the highest leverage quality fix available in a RAG system. Here is why, with numbers.
The failure mode I have seen most often, in my own work and other people’s: answer quality is poor, so the team reaches for a bigger generator. It rarely helps, because the generator is rarely the binding constraint.
Build a labelled set: a few hundred real questions, each tagged with the chunk identifiers that contain the answer. Measure recall@k, the fraction for which at least one gold chunk lands in the top \(k\). It takes an afternoon and it is the most informative measurement in the system.
For single-hop questions recall@k is a hard ceiling on end-to-end accuracy: if the answer is not in the context, the model can only say it does not know or make something up.
Suppose end-to-end accuracy is 0.62, recall@5 is 0.68, and when handed the correct context the model answers correctly 91% of the time. Then \(0.68 \times 0.91 = 0.619\), which accounts for essentially all of your observed accuracy. Now compare the two available moves:
Six times the improvement for a fraction of the cost. The shape of the recall curve separates the two cases. If recall@50 is high (say 0.95) but recall@5 is low, the right documents are found and badly ordered: a ranking problem, which a reranker fixes. If recall@50 is also low they are not being found at all: a chunking, embedding or query formulation problem, and neither a reranker nor a larger generator rescues it.
Takeaway: Measure recall@k against a labelled set before touching the generator. Answer accuracy is bounded above by retrieval recall, and a reranker that lifts recall@5 typically buys several times the accuracy of a larger model, for a small fraction of the cost.
Traffic is not uniformly hard. Most queries are easy, a minority are not, and paying large-model prices for the easy majority is how agent bills get out of hand.
A cascade runs the small model first, computes a confidence signal, and escalates only below a threshold. With small-model cost \(c_s\), large-model cost \(c_l\) and escalation rate \(p\):
\[\mathbb{E}[\text{cost}] \;=\; c_s + p \, c_l, \qquad \mathbb{E}[\text{accuracy}] \;=\; (1-p)\,a_{s|\text{kept}} + p \, a_{l|\text{escalated}}\]| Note that $$a_{s | \text{kept}}\(exceeds the small model's overall accuracy, because the escalation rule removed the hard cases, and\)a_{l | \text{escalated}}$$ falls below the large model’s, for the same reason. The most common error in cascade analysis is using the unconditional accuracies here. |
Sweep the threshold and tabulate, with illustrative per-query costs of $0.0004 for the small model and $0.0060 for the large:
Cascade threshold sweep. Cost per query includes the small model call, which always runs. Marginal cost per accuracy point is quoted per one million queries.
| Escalation rate | Cost per query | Accuracy | Marginal gain (points) | Marginal cost per point, per 1M queries |
|---|---|---|---|---|
| 0% (small only) | $0.00040 | 0.820 | - | - |
| 10% | $0.00100 | 0.874 | +5.4 | $111 |
| 20% | $0.00160 | 0.904 | +3.0 | $200 |
| 35% | $0.00250 | 0.921 | +1.7 | $529 |
| 60% | $0.00400 | 0.929 | +0.8 | $1,875 |
| 100% | $0.00640 | 0.930 | +0.1 | $24,000 |
At 20% escalation you get 0.904 accuracy for $0.0016 per query: within 2.6 points of the large model at a quarter of the cost. The marginal column is where the decision lives. Moving from 10% to 20% escalation costs $200 per accuracy point per million queries; moving from 60% to 100% costs $24,000 per point. Somewhere between, you cross your willingness to pay, and that crossing is your threshold. Set it with a number and revisit it when prices change.
One honest detail the table exposes: at 100% escalation the cascade costs $0.0064 while always calling the large model directly costs $0.0060. Past a high enough escalation rate the cascade is pure overhead and you should delete it.
This is the on-device filter from Part 11 in a different costume: a cheap stage handles the bulk, an expensive stage the residue. The design problem is not making the cheap stage accurate, it is making it reliably aware of when it is wrong. A small model that is 82% accurate and well calibrated beats one that is 86% accurate and confidently wrong.
Exact match caching keys on a hash of the rendered prompt plus model identifier, decoding parameters and tool schema version. A hit is milliseconds instead of seconds and costs nothing. Hit rate \(h\) removes a fraction \(h\) of calls but not necessarily of the bill: repeated queries skew short and cheap, so measure cost-weighted hit rate.
Semantic caching returns a stored answer when the query’s nearest neighbour is within cosine threshold \(\tau\). It raises the hit rate and adds a failure mode: “the refund policy for EU customers” and “the refund policy for US customers” sit close in embedding space and have different answers. A 3% false hit rate on a policy assistant is a machine for generating incidents. Set \(\tau\) high, measure the false hit rate on a labelled set before shipping, and never semantically cache an answer that depends on the user, the tenant or the time.
Prefix caching is the one I reach for first. Providers cache the key-value state of a shared prefix so a repeat skips prefill entirely: the KV cache from Part 8, exposed as a billing feature. It needs the prefix byte-identical, which gives one instruction: static content at the front, volatile content at the back. System prompt, tool schemas and few-shot examples first; timestamps, user names and retrieved chunks last.
Work it. A 4,000-token static prefix, 1,600 tokens of variable content, 10 calls in a session, prices as before, and an illustrative cache shape where cached reads bill at 0.1x and the first write at 1.25x. Uncached input is \(10 \times 5{,}600 = 56{,}000\) tokens, $0.168. Cached, the first call bills \(4{,}000 \times 1.25 + 1{,}600 = 6{,}600\) token-equivalents and each of the nine others \(4{,}000 \times 0.1 + 1{,}600 = 2{,}000\), so 24,600 total: $0.074, a 56% cut. It also removes 0.4 s of prefill from nine calls, 3.6 s off the session.
The correctness hazards of caching in a stateful agent are all one hazard: the key must include everything the output depends on. Tool version, tenant identifier, user permissions, data freshness horizon, index version. Cache a tool result across a schema migration and you serve confidently formatted nonsense; cache across tenants and you have a data leak, not a performance bug. My rule: only pure functions of their inputs get cached, and any step reading mutable external state gets an explicit, short time to live.
Takeaway: A cascade with a well calibrated escalation signal lands within a few accuracy points of the large model at a quarter of the cost, and prefix caching removes half the input bill for the price of reordering your prompt. Set the escalation threshold from a marginal-cost table, not from intuition.
Draw the agent as a directed acyclic graph with durations on the nodes. Total latency is not the sum of the nodes, it is the longest weighted path from source to sink. Everything off that path is free.
An agent graph with illustrative node durations.
| Node | Kind | Duration (ms) | Depends on |
|---|---|---|---|
| A: plan | model | 1,200 | - |
| B: web search | tool | 1,800 | A |
| C: SQL query | tool | 900 | A |
| D: profile fetch | tool | 400 | A |
| E: rerank | service | 60 | B |
| F: synthesise | model | 2,500 | C, D, E |
Executed strictly sequentially, as a naive framework loop will, the total is 6,860 ms. The critical path is A to B to E to F, which is 5,560 ms. Running B, C and D concurrently saves 1,300 ms, about 19%, and costs you an asyncio.gather.
Then look again. Step A contributes 1,200 ms to the critical path and, where the tool set is fixed, usually decides to call all three tools anyway. Replace it with a deterministic fan-out and the critical path becomes B to E to F, 4,360 ms. Removing one model call saved 1,200 ms; parallelising three tools saved 1,300 ms. A model node on the critical path is usually the most expensive node on it.
The generalisation is map-reduce. With 10 documents to analyse, do not loop: fan out 10 parallel calls and reduce. Depth goes from 11 to 2 at roughly the same token cost, trading latency for concurrency and rate limit headroom, nearly always the right trade.
Speculative execution attacks the remaining serial dependency: start the likely next tool call before the model finishes deciding. With hit probability \(q\) and tool duration \(T\), the expected saving is bounded by \(q \cdot \min(T, T_{\text{decide}})\) and the expected extra cost is \((1-q)\) wasted invocations. At \(T = 600\) ms, a 900 ms deciding call, \(q = 0.75\) and $0.0002 per tool call, you buy up to 450 ms for $0.00005 per query, $50 a month at a million queries. Watch the rate limit and the side effects rather than the money: never speculate a tool call that writes.
Takeaway: Latency is the critical path, not the sum of steps. Parallelise independent tool calls, then check whether a model node sits on the critical path doing work a deterministic fan-out could do, because removing that node usually beats the parallelism.
Depth costs reliability on the same curve it costs latency. At an independent per-step failure probability of 0.01, a three-step chain succeeds 97.0% of the time and a twelve-step chain \(0.99^{12} = 88.6\%\): one session in nine fails. One retry per step drops effective per-step failure to \(10^{-4}\) and the twelve-step chain to 99.88% success.
Retries only work if the operation is safe to repeat. Reads are; writes are not, unless made idempotent with a client-generated key the downstream service deduplicates on. Use exponential backoff with full jitter, and a retry budget capping retries at some percentage of traffic, so a downstream brownout does not become a retry storm that keeps it down.
Set timeouts from the tool’s measured p99, not a framework default. A 30 s default on a tool whose p99 is 200 ms turns one blip into a half-minute stall, and twelve of those can stack.
Design for partial failure, not exceptions. An agent with six tools has six failure domains. Feed the failure to the model as data (SEARCH_UNAVAILABLE: proceed using the database result only) rather than letting the orchestration layer throw. Models degrade gracefully when told what is missing, and not at all when the process dies.
And give every tool call a budget, because an unbounded loop with a credit card is a bad combination:
@dataclass
class StepBudget:
timeout_s: float # from measured p99, not a default
max_retries: int
max_output_tokens: int
max_cost_usd: float
async def run_step(step, budget, ledger):
if ledger.spent >= ledger.session_cap:
raise BudgetExhausted(ledger)
for attempt in range(budget.max_retries + 1):
try:
async with asyncio.timeout(budget.timeout_s):
result = await step.invoke(
max_tokens=budget.max_output_tokens,
idempotency_key=step.key,
)
ledger.record(step.name, result.usage)
return result
except (TimeoutError, TransientError) as exc:
if attempt == budget.max_retries or not step.idempotent:
return Degraded(step.name, reason=str(exc))
await asyncio.sleep(backoff_with_jitter(attempt))
Session-level caps matter as much as per-step ones: maximum steps, maximum wall clock, maximum spend. A loop that fails to converge will happily burn money until a human notices, and the human notices at the end of the month.
Every model output a program consumes needs a schema and a validator. Prefer constrained decoding or the provider’s tool-calling interface, because a grammar-constrained decoder cannot emit invalid JSON, which beats detecting invalid JSON. Where you must parse free text: validate; on failure make exactly one repair call with the validation error appended; on second failure fall back to a deterministic parse or a typed error.
One repair attempt, not a loop. A second repair almost never succeeds and it doubles your worst-case latency.
Track the repair rate as a first-class metric. A schema validity rate drifting from 99.4% to 96% over a fortnight means something upstream changed, usually a prompt edit or a provider-side model update, and you want to know before users do.
Offline eval sets. A few hundred realistic queries with graded references, frozen and versioned. Evaluate components separately: retrieval recall@k, tool selection accuracy, schema validity rate, final answer quality. An end-to-end score says the system got worse; component scores say which part.
LLM as judge, with the biases stated honestly, because the technique is genuinely useful and genuinely unreliable used naively. Judges show position bias, preferring whichever candidate comes first or last; length bias, preferring longer and more confident-sounding answers regardless of correctness; self-preference for their own model family; and poor calibration on absolute scales, clustering scores around 7 out of 10. The mitigations are mechanical: pairwise comparison instead of absolute scoring, each pair run in both orders and averaged, a reference answer and explicit rubric, and length controlled for.
Regression suites from production failures. Every incident becomes a permanent test case. This one habit did more for the reliability of the systems I built than any architectural change, because it converges the eval set on the distribution that actually breaks you.
Tracing. One trace identifier per run, one span per step, each recording model identifier, prompt hash, token counts, latency, tool name, cost and, critically, the retrieved chunk identifiers. Without those you cannot separate a retrieval failure from a generation failure after the fact, and you will spend a week arguing about it. This is Part 10 applied to a distributed system: you cannot optimise what you have not attributed.
Takeaway: Every tool call needs a timeout from measured p99, a retry budget, and an idempotency key if it writes; every session needs a step, time and spend cap. Validate your LLM judge against human labels before you believe anything it tells you.
The most effective change I have ever made to an agent system was deleting steps from it. That deserves saying plainly, because the industry incentive runs the other way.
A single well-prompted call has one failure domain, one latency term, one cost term and a trace you can read in ten seconds. An agent has \(N\) of each, plus an orchestration layer, plus a class of failures where the model decides to do something else. Worth paying for when the control flow is genuinely data dependent and the space of plans is too large to enumerate. Otherwise, no.
Symptoms that you have built an agent where a program would do.
| Symptom | What it usually means | Do this instead |
|---|---|---|
| The plan is the same on almost every run | Control flow is static | Write the program, call the model at the leaves |
| One retrieval, one call, one answer | No planning is required | Single prompt with the context pasted in |
| The loop needs a step cap to terminate | The objective is underspecified | Fix the objective; make the final step deterministic |
| Most steps reformat the previous step | The model is doing string manipulation | Do it in code, for free, correctly |
| Quality is capped by recall@k | The generator was never the problem | Spend the budget on the index and the reranker |
| The trace is easier to read than the code | The logic lives in prompts | Move the invariants into the program |
A deterministic pipeline containing three model calls is not an agent, and that is a feature: testable, predictable in latency, constant in cost rather than a distribution with a tail, and when it breaks you can put a breakpoint in it. If a single call or a fixed program does the job, it is cheaper, faster and enormously easier to debug.
Problem 1. Assume prefill throughput of 12,000 tokens per second, decode at 18 ms per token, 150 ms of per-call overhead, and 250 ms of tool time after each non-final step. Chain A has 8 steps: step \(i\) sends \(3{,}000 + 500(i-1)\) prompt tokens, steps 1 to 7 each emit 100 output tokens and call a tool, and step 8 emits 500 tokens. (a) Compute the total latency. (b) You can either halve the decode time or collapse the chain to 4 steps with the same prompt growth rule, 100 output tokens for steps 1 to 3, and 500 for step 4. Which wins?
(a) Prompts are 3,000 to 6,500 in steps of 500, so the sum is \(8 \times \frac{3{,}000 + 6{,}500}{2} = 8 \times 4{,}750 = 38{,}000\) tokens. Prefill is \(38{,}000 / 12{,}000 = 3.17\) s.
Decode is \(7 \times 100 + 500 = 1{,}200\) tokens at 18 ms, so 21.60 s. Tools are \(7 \times 0.25 = 1.75\) s. Overhead is \(8 \times 0.15 = 1.20\) s.
Total: \(3.17 + 21.60 + 1.75 + 1.20 = 27.72\) s.
(b) Halving decode to 9 ms per token: decode becomes 10.80 s, total \(3.17 + 10.80 + 1.75 + 1.20 = 16.92\) s.
Collapsing to 4 steps: prompts 3,000, 3,500, 4,000, 4,500 sum to 15,000, so prefill is 1.25 s. Decode is \(3 \times 100 + 500 = 800\) tokens at 18 ms, 14.40 s. Tools \(3 \times 0.25 = 0.75\) s. Overhead \(4 \times 0.15 = 0.60\) s. Total 17.00 s.
They are a dead heat, 16.92 s against 17.00 s, and that is the instructive part. Unlike the example in the chapter, shortening the chain barely helps on total time here, because the 500-token final generation is 9.00 s all by itself and cutting steps does not touch it. Always check which term dominates before assuming which lever applies.
It is worth computing time to first token as well, since that is what a user perceives when you stream. For the 8-step chain everything before the final generation takes \(2.63 + 12.60 + 1.75 + 1.05 + 0.54 + 0.15 = 18.72\) s; for the 4-step chain, 8.00 s. So the two configurations tie on total latency and differ by more than two to one on the number that matters. Doing both changes gives a total of \(1.25 + 7.20 + 0.75 + 0.60 = 9.80\) s.
Problem 2. A corpus of 40 million chunks embedded at \(d = 1536\) in fp32, indexed with HNSW at \(m = 24\) (layer-0 degree 48, 4-byte identifiers). (a) How much memory does the index need? (b) Your nodes have 64 GB of RAM. Which of fp16, int8 scalar quantisation, or truncation to 512 dimensions makes it fit on one node? (c) At fp32, how many shards do you need at a 70% memory utilisation target?
(a) Vectors: \(40 \times 10^6 \times 1536 \times 4 = 245.76 \times 10^9\) bytes, or 245.76 GB.
Graph: \(40 \times 10^6 \times 48 \times 4 = 7.68\) GB at layer 0, plus roughly 5% for upper layers, so about 8.1 GB.
Total: about 254 GB. The graph is only 3% of it; at high dimension the vectors are everything.
(b) The graph term (8.1 GB) does not shrink under any of these, since it stores identifiers.
No single option fits. Combinations do: 512 dims at fp16 is \(40.96 + 8.1 = 49.1\) GB, and 512 dims at int8 is \(20.48 + 8.1 = 28.6\) GB with comfortable headroom. If quality at 512 dimensions is unacceptable, int8 at 1024 dimensions gives \(40.96 + 8.1 = 49.1\) GB and also fits. This is the real shape of the decision: you usually need two levers, not one.
(c) Usable memory per node is \(0.7 \times 64 = 44.8\) GB. \(254 / 44.8 = 5.67\), so 6 shards. Query each shard for its own top \(k\) and merge, which preserves recall as long as you take the full \(k\) from every shard; taking top \(k/6\) from each does not.
Problem 3. A small model costs $0.0005 per query and always runs first; the large model costs $0.0080. Threshold settings give escalation rates of 0%, 15%, 30% and 50% with accuracies 0.840, 0.892, 0.913 and 0.921. Traffic is 3 million queries per month, and you will accept at most $9,000 per month above the small-only cost. Which threshold do you pick, and where is the knee?
Cost per query is \(0.0005 + p \times 0.0080\); monthly cost is that times \(3 \times 10^6\).
| Escalation | Cost/query | Monthly | Increase over small-only | Accuracy |
|---|---|---|---|---|
| 0% | $0.00050 | $1,500 | - | 0.840 |
| 15% | $0.00170 | $5,100 | $3,600 | 0.892 |
| 30% | $0.00290 | $8,700 | $7,200 | 0.913 |
| 50% | $0.00450 | $13,500 | $12,000 | 0.921 |
Pick 30% escalation. It is +7.3 accuracy points for $7,200 per month, inside the $9,000 budget; 50% costs $12,000 and is out.
The marginal analysis says the same thing without needing the budget constraint. From 0 to 15%: $3,600 for 5.2 points, $692 per point. From 15 to 30%: $3,600 for 2.1 points, $1,714 per point. From 30 to 50%: $4,800 for 0.8 points, $6,000 per point. The knee is at 30%, where marginal cost jumps by three and a half times.
Worth noting for context: always calling the large model directly, with no cascade, is \(3 \times 10^6 \times 0.0080 = \$24{,}000\) per month, nearly three times the 30% cascade. The sweep does not give the always-large accuracy, but since the 30 to 50% band bought only 0.8 points, the escalations beyond 50% are unlikely to buy much more.
Problem 4. A 30-turn session: 900-token system prompt, 1,200 tokens retrieved per turn, 60-token user messages, 250-token replies. Prices are $2.50 per million input and $10.00 per million output tokens. (a) Cost under naive append. (b) Cost under management: a 3-turn verbatim window, a 250-token state object, plus a summarisation call every 5 turns costing 1,200 input and 250 output tokens. (c) What is the ratio at 60 turns?
(a) Each turn appends \(a = 60 + 250 + 1{,}200 = 1{,}510\) tokens. Turn \(k\) sends \(900 + 1{,}260 + 1{,}510(k-1) = 2{,}160 + 1{,}510(k-1)\).
\[\text{input} = 30 \times 2{,}160 + 1{,}510 \times \frac{29 \times 30}{2} = 64{,}800 + 656{,}850 = 721{,}650\]Output is \(30 \times 250 = 7{,}500\). Cost: \(0.72165 \times 2.50 + 0.0075 \times 10.00 = 1.804 + 0.075 = \mathbf{\$1.88}\).
(b) Per turn: \(900 + 250 + (3 \times 310) + 1{,}200 + 60 = 3{,}340\) tokens, constant. Over 30 turns, 100,200 input tokens. Six summarisation calls add 7,200 input and 1,500 output.
Input 107,400, output 9,000. Cost: \(0.1074 \times 2.50 + 0.009 \times 10.00 = 0.2685 + 0.090 = \mathbf{\$0.36}\). A 5.2x saving.
(c) At 60 turns, naive input is \(60 \times 2{,}160 + 1{,}510 \times 1{,}770 = 129{,}600 + 2{,}672{,}700 = 2{,}802{,}300\), with 15,000 output: \(7.006 + 0.150 = \$7.16\).
Managed: \(60 \times 3{,}340 = 200{,}400\) plus twelve summarisers (14,400 input, 3,000 output) gives 214,800 input and 18,000 output: \(0.537 + 0.180 = \$0.72\).
Ratio 10.0x, up from 5.2x. Doubling the session length doubled the advantage, which is the signature of quadratic against linear growth.
Problem 5. An agent graph: A (plan, model, 900 ms), then B (tool, 1,400 ms), C (tool, 600 ms) and D (tool, 300 ms) each depending on A, then F (synthesise, model, 2,200 ms) depending on B, C and D. (a) Serial total and critical path. (b) You speculatively start B at \(t = 0\). A needs B with probability 0.75; otherwise only C and D run. B costs $0.0009 per call. What is the expected latency saving and the expected extra cost at 1 million queries per month?
(a) Serial: \(900 + 1{,}400 + 600 + 300 + 2{,}200 = 5{,}400\) ms.
Critical path with the tools run concurrently: \(900 + \max(1{,}400,\, 600,\, 300) + 2{,}200 = 900 + 1{,}400 + 2{,}200 = \mathbf{4{,}500}\) ms. Parallelism alone saves 900 ms. Note this is the case where B is needed; part (b) averages over the 75/25 split, which is why the baseline there is 4,300 ms rather than 4,500 ms.
(b) Without speculation, the fan-out completes at \(900 + 1{,}400 = 2{,}300\) ms when B is needed and \(900 + 600 = 1{,}500\) ms when it is not, so in expectation \(0.75 \times 2{,}300 + 0.25 \times 1{,}500 = 1{,}725 + 375 = 2{,}100\) ms.
With B started at \(t = 0\), it completes at 1,400 ms, before C completes at 1,500 ms. So the fan-out finishes at \(\max(1{,}400,\, 1{,}500) = 1{,}500\) ms on a hit, and at 1,500 ms on a miss too. Expected completion is 1,500 ms either way.
Expected saving: 600 ms, taking the mean total from 4,300 ms to 3,700 ms.
Extra cost: B now runs on 100% of queries instead of 75%, so 0.25 wasted calls per query at $0.0009, which is $0.000225 per query, or $225 per month at 1 million queries. Six hundred milliseconds for $225 a month is a good trade. The constraints to check before shipping it are rate limits on B, since you have raised its call volume by a third, and that B must be a read: speculatively executing a write is a correctness bug, not an optimisation.
Every chapter in this book has been one argument in a different costume: write down the budget, write down the cost model, find the dominant term, attack it. In Part 1 the currency was FLOPs and bytes; here it has been tokens and sequential calls, and the answer is again that the structure of the computation matters more than the speed of any one piece. An agent with three steps and a good reranker beats one with twelve steps and a bigger model, on latency, on cost, and on whether the answer is right.