Transformers on Small Machines

Everything in the previous seven parts was building to this question: you have a decoder-only transformer and you have a device that is not a datacentre. How many tokens per second can you possibly get, and what is stopping you?

The pleasant thing about language models is that the answer is almost entirely arithmetic. A convolutional network has enough structural variety that you must profile it to know where the time goes. A transformer is four matrix multiplications and an attention operation, repeated, and the cost model fits on the back of an envelope. The unpleasant thing is that the envelope usually says you are memory bandwidth bound by a factor of fifty, and no amount of kernel work will fix that.

All hardware numbers here are illustrative: round figures in the range that real phones, laptops and small GPUs occupy, chosen so the arithmetic is easy to follow. The method transfers, not the constants.

Counting Everything in a Decoder Layer

Parameters

Fix notation for a decoder-only transformer. Let \(d\) be the model width (d_model), \(d_{ff}\) the feed-forward hidden width, \(h\) the number of attention heads, \(h_{kv}\) the number of key/value heads, \(d_h\) the per-head dimension (usually \(d/h\)), \(L\) the number of layers and \(V\) the vocabulary size.

The attention block holds four projections. The query projection maps \(d \to h \cdot d_h\). The key and value projections each map \(d \to h_{kv} \cdot d_h\), which is the entire content of grouped query attention: you keep all \(h\) query heads but only \(h_{kv}\) distinct key/value heads, and each key/value head is shared across \(h / h_{kv}\) query heads. The output projection maps \(h \cdot d_h \to d\). Taking \(d_h = d/h\):

The feed-forward block is either a plain two-matrix MLP (\(2 d \, d_{ff}\) parameters) or a gated variant such as SwiGLU with three matrices, a gate, an up and a down projection (\(3 d \, d_{ff}\)). Gated is now the default, so I will use three.When a model quotes a "hidden size" for a gated FFN, check whether it is the width of one branch or the total. Llama-style configurations quote the width of one branch, so the parameter count is three times width times model dimension, not two.

The trailing \(2d\) is the two RMSNorm gain vectors, which you can ignore for sizing and should not ignore for numerics. Outside the stack you have the token embedding table, \(V d\), and the output projection, another \(V d\) if the two are untied.

Parameter count for an illustrative 7B-class model, worked term by term. Take \(d = 4096\), \(d_{ff} = 11008\), \(h = h_{kv} = 32\), \(d_h = 128\), \(L = 32\), \(V = 32000\), untied embeddings.

That is 6.74 billion, which is where “7B” comes from. Two things are worth noticing. The FFN is two thirds of the layer, which is why FFN width is the first dial people turn when they want a smaller model. And embedding plus head is 3.9% of this model but would be 24% of a 1B model with a 128k vocabulary.

FLOPs for a forward pass

A matrix multiply of an \(S \times d_{\text{in}}\) activation by a \(d_{\text{in}} \times d_{\text{out}}\) weight costs \(2 S \, d_{\text{in}} d_{\text{out}}\) floating point operations, because each output element is a dot product of length \(d_{\text{in}}\), which is \(d_{\text{in}}\) multiplies and \(d_{\text{in}}\) adds.Strictly it is one fewer add than multiply, and fused multiply-add hardware does both in one instruction. The convention in this literature is to count a FMA as 2 FLOPs, which keeps the arithmetic consistent with how vendors quote peak throughput.

The attention operation itself has no weights, so it gets counted separately. For a prompt of \(S\) tokens, \(QK^\top\) per head is an \(S \times d_h\) by \(d_h \times S\) product, \(2 S^2 d_h\) FLOPs, and summing over \(h\) heads gives \(2 S^2 d\). Multiplying the attention weights by \(V\) costs the same again. So the naive total is \(4 S^2 d\) per layer, and with causal masking exploited properly only the lower triangle is computed, giving about \(2 S^2 d\) per layer.FlashAttention-style kernels skip fully masked blocks, so they realise close to the causal figure. A naive implementation that materialises the full score matrix and then masks it pays the full $$4 S^2 d$$ and also allocates an $$S \times S$$ buffer per head, which is the usual reason a long-prompt prefill runs out of memory.

Prefill, ignoring the output head because you only need logits for the last position:

where \(N_{\text{mat}}\) is the number of parameters in weight matrices, that is, everything except the embedding lookup table.

A single decode step processes one new token against \(S\) cached keys and values. The matmuls become vector-matrix products: \(2 N_{\text{mat}}\) FLOPs. Attention is now \(1 \times S\) rather than \(S \times S\): \(2 S d\) for the scores and \(2 S d\) for the weighted sum, per layer.

For our 7B model at \(S = 4096\), that is \(1.35 \times 10^{10}\) from the weights and \(2.1 \times 10^{9}\) from attention. Attention is 14% of decode arithmetic at 4k context, and it grows linearly while the weight term stays flat.

Where the 2N rule comes from

The “about \(2N\) FLOPs per token” rule that everyone quotes is not a fitted approximation, it is a counting argument. Every parameter that lives in a weight matrix is touched exactly once per token: it is multiplied by one activation and the product is accumulated into one output. That is one multiply and one add, two FLOPs, per parameter per token. Sum over all matrix parameters and you get \(2 N_{\text{mat}}\).

The approximation only involves what you leave out. Embedding lookup is a gather with zero arithmetic. Norms, activation functions, softmax and residual adds are \(O(d)\) or \(O(S)\) per token per layer, a few tens of FLOPs per dimension against the \(O(d^2)\) of the matmuls. Attention against the cache is the genuinely omitted term, \(4 L S d\), and the rule quietly assumes \(S \ll N_{\text{mat}} / (2 L d)\), which for our 7B model means \(S \ll 25{,}000\).

The same argument gives the training figure. Backward needs the gradient with respect to the input and the gradient with respect to the weights, one pass over the weights each, so backward is about \(4N\) and a full training step about \(6N\) per token.

Takeaway: \(2N\) FLOPs per token is a counting argument, not an empirical fit: each matrix weight does one multiply-accumulate per token. Attention against the cache adds \(4LSd\) per token, which is the term that makes long contexts expensive.

Two Regimes, Prefill and Decode

Generation has two phases with completely different bottlenecks, and conflating them is the single most common error I see in performance discussions.

Prefill consumes the prompt. All \(S\) prompt tokens go through the model at once, as a matrix-matrix multiplication. It is embarrassingly parallel and produces one token of output.

Decode produces the rest. Each token depends on the previous one, so the steps are strictly sequential, and each step is a matrix-vector multiplication.

Why prefill is compute bound

Use the roofline reasoning from Part 1. Arithmetic intensity is FLOPs divided by bytes moved from memory. During prefill you read the weights once, \(N b\) bytes where \(b\) is bytes per weight, and you do \(2 N S\) FLOPs:

With fp16 weights, intensity is simply \(S\) FLOPs per byte. A 512-token prompt gives 512 FLOP/byte. Devices sit somewhere between about 30 and 400 FLOP/byte at their ridge point, so any prompt longer than a few hundred tokens is firmly compute bound. Quantising the weights raises intensity further, since \(b\) shrinks.

An illustrative prefill. Assume a phone-class accelerator sustaining 2 TFLOP/s on this workload (illustrative). A 1024-token prompt into the 7B model costs \(2 \times 6.74{\times}10^9 \times 1024 = 1.38{\times}10^{13}\) FLOPs for the matmuls plus \(2 \times 32 \times 1024^2 \times 4096 = 2.7{\times}10^{11}\) for causal attention, so \(1.41{\times}10^{13}\) in total. That is 7.0 seconds of time to first token. Painful, and it is a genuine compute problem: a faster kernel or a wider matrix engine really does help here.

The decode latency floor

Now do the same calculation for one decode step. FLOPs are \(2N\). Bytes read are \(Nb\) for the weights, whether you are generating one token or none.

For fp16 that is 1 FLOP per byte. One. Against a ridge point of 30 to 400. Decode is memory bound by one and a half to two and a half orders of magnitude, and this is a property of the algorithm, not of your code.

That gives a hard lower bound on time per token. Let \(B_w\) be the bytes of weights, \(B_{kv}\) the bytes of KV cache you must read, and \(\beta\) the achievable memory bandwidth:

You cannot beat this without moving fewer bytes. No kernel fusion, no assembly, no vendor library beats it, because it is the time to stream the weights past the arithmetic units once. It is the equivalent of the speed of light for the decode loop.

A worked decode bound. Take an illustrative flagship phone with 68 GB/s of memory bandwidth, and the 7B model quantised to int4 with a group size of 64 and fp16 scale and zero-point per group. That is \(4 + 32/64 = 4.5\) bits per weight, so \(6.74{\times}10^9 \times 4.5/8 = 3.79\) GB of weights. At an empty cache:

Now compare the two phases on equal arithmetic. Generating 1024 tokens after that 1024-token prompt costs \(1.46{\times}10^{13}\) FLOPs, within 4% of the prefill. With the average cache over that run (1536 tokens, MHA, fp16) adding 0.81 GB per step, the bound is 14.8 tokens/s, so 69 seconds. Same arithmetic as prefill, ten times the wall clock. That ratio is the whole chapter in one number.

Takeaway: prefill has arithmetic intensity \(2S/b\) and is compute bound; decode has intensity \(2/b\) and is bandwidth bound. Decode tokens per second is bounded by bandwidth divided by (weight bytes plus cache bytes), and clever kernels do not move that bound.

The KV Cache

Bytes per token

To avoid recomputing attention over the whole prefix at every step, you cache the key and value vectors for every past token, at every layer. Per token, per layer:

The 2 is for K and V, and \(b_{kv}\) is bytes per cached element. Multiply by \(L\) for the whole model, and by \(S\) for the whole sequence.

For our 7B model with full multi-head attention (\(h_{kv} = 32\), \(d_h = 128\)) in fp16: \(2 \times 32 \times 128 \times 2 = 16{,}384\) bytes per layer per token, times 32 layers, is 512 KiB per token. At 4096 tokens that is exactly 2 GiB, which on a phone is already more than the model you were trying to run.

Multi query, grouped query and sliding windows

The cache scales with \(h_{kv}\), not \(h\), which is precisely why grouped query attention exists. Cutting \(h_{kv}\) from 32 to 8 cuts the cache by 4x and costs you a little quality; multi-query attention takes \(h_{kv} = 1\) and cuts it by 32x, with a larger quality cost that is usually recovered by uptraining.GQA also shrinks the K and V projection matrices, so it reduces parameter count slightly. For our 7B example, moving to 8 KV heads removes about 50M parameters, under 1%. The cache saving is the point; the parameter saving is a rounding error.

Sliding window attention caps the cache instead of shrinking it per token: each layer attends only to the last \(w\) positions, so the cache is \(\min(S, w)\) tokens deep and memory becomes constant in context length. Information still propagates further than \(w\), since each layer moves it \(w\) positions back for a receptive field of roughly \(L w\), but the model can no longer directly retrieve a specific distant token. The usual compromise is to interleave, keeping a minority of layers global.

KV cache size against context length, for the illustrative 7B-class model. All figures are for the cache alone; \(d_h = 128\), \(L = 32\).

The bottom row is the interesting one. Without a window, a 128k context is not a tuning problem, it is an impossibility on any edge device. With one, it is 512 MiB.

Quantising the cache

The cache is activations, not weights, so the rules from Part 5 apply differently. Keys and values are reasonably well behaved per channel but do contain outlier channels, especially in the keys, where rotary position embeddings concentrate magnitude in a few dimensions. Per-channel scaling for keys and per-token scaling for values is the combination that usually works. int8 cache is close to free in quality terms. int4 cache is workable with group-wise scales and by keeping the most recent tokens in higher precision, since recent keys carry most of the attention mass.

The payoff is double: half or a quarter of the capacity, and the same reduction in bytes read per decode step, which shows up directly in the latency bound.

Takeaway: cache bytes per token are \(2 h_{kv} d_h b_{kv} L\). The three dials are \(h_{kv}\) (GQA/MQA), \(b_{kv}\) (cache quantisation) and the effective \(S\) (sliding windows), and on a long-context edge deployment you will usually need all three.

Quantised Weights Buy Tokens per Second

Here is the clean consequence of decode being bandwidth bound: for a single-stream decode loop, the token rate is inversely proportional to bytes per weight. Not to arithmetic precision, not to kernel quality, to bytes. This is why weight-only quantisation, which looks like a compromise in a training context, is simply the correct default on the edge.

Note that it is weight-only. You dequantise to fp16 or bf16 in registers and do the arithmetic in floating point. Integer arithmetic gains you nothing here, because you are not short of arithmetic; the smaller weight footprint gains you everything. The opposite holds during prefill, where low-precision matmul throughput genuinely helps.This is why edge runtimes often use different kernels for the two phases: a low-precision matmul for prefill and a dequantise-then-fp16-dot kernel for decode. If your framework uses one kernel for both, prefill is usually the one leaving performance behind.

Decode bound against weight precision, illustrative 7B model on an illustrative 68 GB/s device. “With cache” is at 4096 tokens of fp16 MHA cache (2.15 GB).

fp16 to int4 gives 3.6x, not 4x, because the group scales cost half a bit per weight. With a 4k cache in play it is 2.6x, because the cache bytes do not shrink when the weights do. “Roughly quadruples” is the right mental model and 2.5 to 3.6x is the right number to plan with.

What does it cost? Summarising the published literature rather than any measurement of my own, and with the strong caveat that these vary by model and task:

Approximate quality cost of weight-only quantisation on a 7B-class model. Perplexity deltas are indicative, in the units WikiText-2 perplexity is usually reported in.

The decision rule that follows is worth stating plainly. If you have a memory and bandwidth budget \(B\), you almost always get a better model by taking the largest parameter count that fits at int4 than by taking a smaller model at int8 or fp16. A 7B at int4 beats a 3B at int8 at the same footprint on essentially every benchmark I have seen. The exception is when int4 pushes a specific capability off a cliff, which happens most often with code generation and multi-step arithmetic, so test those specifically.

Takeaway: because decode is bandwidth bound, token rate scales as one over bytes per weight. At a fixed byte budget, a bigger model at int4 almost always beats a smaller model at higher precision.

Speculative Decoding

The draft and verify loop

The sequential dependency in decode is what forces you to read all the weights once per token. Speculative decoding attacks exactly that.

A small draft model generates \(\gamma\) candidate tokens autoregressively. The large target model then runs a single forward pass over all \(\gamma + 1\) positions, which gives you its distribution at each one. You walk the candidates left to right, accepting each with a probability that depends on the ratio of target to draft probability, and at the first rejection you resample from a corrected distribution and stop. The acceptance rule is constructed so that the tokens emitted are distributed exactly as if they had come from the target model, one at a time. This is not an approximation; the output distribution is identical.Exactness holds for the sampling distribution, not for a particular random seed. A speculative run and a plain run with the same seed will diverge, which makes bit-exact regression tests against a non-speculative baseline impossible. Test the distribution, not the sequence.

The reason it works on bandwidth bound hardware is the crucial part. Verifying \(\gamma + 1\) tokens requires reading the target weights exactly once, the same as generating one token. The arithmetic goes up by a factor of \(\gamma+1\), but you had arithmetic to spare: intensity was 1 FLOP per byte against a ridge point of hundreds. Speculative decoding spends the surplus compute you were already wasting and buys sequential steps with it.

When it pays

Model acceptance as independent per position with probability \(\alpha\). The number of candidates accepted, \(n\), satisfies \(P(n \ge k) = \alpha^k\), so \(E[n] = \sum_{k=1}^{\gamma} \alpha^k\). You always emit one more token beyond the accepted prefix, either the corrected resample or, if all \(\gamma\) were accepted, the target’s own bonus token. Hence

Let \(c\) be the cost of one draft step divided by the cost of one target step. On bandwidth bound hardware, \(c\) is very close to the ratio of the two models’ byte footprints. A cycle costs \(\gamma\) draft steps plus one target pass, so \(\gamma c + 1\) target-step-equivalents, and

Two facts fall straight out. Since \(E[\text{tokens}] < 1/(1-\alpha)\) however large \(\gamma\) gets, you cannot do better than \(1/((1-\alpha)(\gamma c + 1))\), so acceptance rate caps the whole scheme. And at \(\gamma = 1\) the speedup is \((1+\alpha)/(1+c)\), which exceeds 1 exactly when \(\alpha > c\): the draft must be accepted more often than it is expensive.

A worked example. Illustrative 68 GB/s device. Target is the 7B at int4, 3.79 GB, 55.7 ms per token. Draft is a 1.1B at int4, 0.62 GB, 9.1 ms per token. So \(c = 0.16\). Ignoring the draft’s own cache, which is small:

At \(\alpha = 0.75\) and \(\gamma = 4\) the speedup is 1.86, taking the phone from 17.9 to 33 tokens per second, with identical output statistics. Notice the shape: low acceptance rates want short drafts or none at all, and only high acceptance rates justify long ones. A fixed \(\gamma\) is leaving something on the table, which is why production implementations adapt \(\gamma\) from the recent acceptance history.

The catch on a phone is capacity, not speed. You now hold two models and two caches in memory. If the draft pushes you over the capacity limit, the paging behaviour in the next section will destroy far more than speculation gains. Self-speculation, where early layers of the target act as the draft, and n-gram or prompt-lookup drafting, which needs no second model at all, both exist precisely to avoid that.Prompt-lookup drafting proposes continuations copied from the prompt itself. It costs no parameters and no arithmetic, and on summarisation or code-editing tasks, where much of the output is quoted from the input, acceptance rates can be very high. On open-ended generation it is close to useless. It is the cheapest thing to try first.

Takeaway: speculative decoding converts surplus arithmetic into fewer sequential weight reads, which is exactly the trade you want when bandwidth bound. It pays when \(\alpha > c\), and the optimal draft length grows with the acceptance rate.

Other Levers on the Decode Loop

Batching, where you are allowed to have one

Batching \(B\) independent decode streams reads the weights once and does \(B\) times the arithmetic, so intensity becomes \(2B/b\). To reach a device ridge point of \(R\) FLOP/byte you need \(B \ge R b / 2\). On an illustrative \(R = 100\) accelerator that is \(B = 100\) at fp16 and \(B = 25\) at int4, which is a nice illustration that quantisation makes you compute bound sooner.

On a phone, \(B = 1\) always, and there is nothing to be done. On a laptop serving a local application, or a single small GPU serving a handful of users, batching is the highest-value change available, and continuous batching, which admits and retires sequences at every step rather than running fixed batches to completion, is what makes it practical when request lengths vary. That is a serving concern and I take it up properly in Part 11.

Prompt caching and prefix reuse

If many requests share a prefix, a system prompt, a tool schema, a document being asked about, compute the prefix KV once and reuse it. The saving is the whole prefill for that prefix: about \(2 N S_{\text{prefix}}\) FLOPs per request. For a 1000-token system prompt on the 7B model that is \(1.35{\times}10^{13}\) FLOPs, roughly 7 seconds of illustrative phone compute, per request, gone.

The cost is storage: 1000 tokens of GQA-8 fp16 cache is 128 MiB, which must be resident or cheaply reloadable, so on a device this means one hot prefix rather than a general cache. Reuse requires an exact token-level prefix match, so put the variable parts of your prompt last, always. I have seen more prefix caches defeated by a timestamp at the top of the system prompt than by anything else.

The vocabulary projection

The output head is \(V d\) parameters and it produces one logit vector per step, of which you use one sampled index. For our 7B that is 131M parameters, 3.9% of the total. For a 1B model with \(d = 2048\) and a 128k vocabulary it is 262M parameters out of roughly 1.1B, nearly a quarter of everything you read per decode step, spent to produce a vector you almost entirely discard.

Three things follow. Tie the input and output embeddings on small models unless you have measured a real quality gain from untying. Consider keeping the head in a different format from the body: it is more sensitive than the FFN, so it often wants int8 while the body takes int4, but at 24% of the bytes that choice is now a throughput decision, not a rounding detail. And during prefill compute logits only for the last position; computing them for all \(S\) positions costs \(2 S d V\) FLOPs for no benefit, about 2% of prefill for a 1024-token prompt on this model, and considerably more on small models with large vocabularies.

Memory, Context and Choosing a Model

Capacity is the wall on a phone

Bandwidth sets your token rate; capacity decides whether you run at all. The budget on a device is weights, plus KV cache, plus activation working set, plus the runtime and its allocator, plus the operating system and every other application, and you get whatever is left before the low-memory killer takes an interest. On an 8 GB phone, an application can typically count on 2 to 3 GB. Our 3.79 GB int4 7B does not fit, and no amount of optimisation changes that.

The standard mitigation is to memory map the weight file read-only. Pages fault in on demand, the file backs them so the kernel can evict them under pressure without needing swap, and several processes can share one copy. It also keeps the weights out of your dirty anonymous memory, which is what most OOM heuristics actually police.

The failure mode is brutal, because decode touches every weight page every step. If a fraction \(f\) of pages has been evicted, each step pays:

With \(B_w = 2\) GB, \(\beta_{\text{mem}} = 68\) GB/s and illustrative flash at 1.5 GB/s, \(f = 0\) gives 29 ms per token and \(f = 0.1\) gives 160 ms. Losing one tenth of your pages costs you 5.4x. There is no graceful degradation here: you are either resident or you are crawling. Size the model so that the whole thing plus the maximum cache fits with headroom, and treat “it works until you open the camera” as a bug in your sizing, not in the phone.

Context length is not free

Combine the cache arithmetic with the latency bound. With \(k\) bytes of cache per token:

Token rate decays hyperbolically in context length. A useful quantity is the context at which the cache costs as much as the model:

Beyond \(S_{1/2}\) you are spending more bandwidth on history than on the model, and your throughput is more than halved. For the int4 7B with GQA-8 fp16 cache, \(S_{1/2} = 3.79{\times}10^9 / 131072 \approx 29{,}000\) tokens. With MHA it is 7,200. It is worth computing this number for any deployment, because it tells you the context length at which your product stops feeling responsive.

The quality argument points the same way. Attention cost grows strictly with sequence length; a model’s ability to use distant context does not. Retrieval accuracy typically degrades in the middle of long inputs, and effective context falls well short of advertised context, so you pay linearly in bandwidth and capacity for material the model may barely consult. The right default is the shortest context that solves the task: retrieve and insert the relevant 2,000 tokens rather than pasting 60,000 and hoping.This is also the cheapest reliability win available. Shortening a context often improves accuracy and always improves latency, which is a rare combination. When someone reports that the on-device model "got worse", the first thing I check is whether the context grew.

A decision table

Putting it together. Pick the largest model whose int4 weights fit in roughly two thirds of your usable memory, leaving the rest for cache, activations and headroom, then check the resulting token rate against your latency requirement.

Model sizing for illustrative device classes. All bandwidth and memory figures are illustrative round numbers. “Usable” is what an application can realistically hold resident. Weight budget is 65 to 70% of usable. Parameter count assumes 4.5 effective bits per weight. Token rate is the bandwidth bound at a small cache, and you should expect to realise 60 to 80% of it.

Read it as a starting point, then adjust. If you need 60 tokens per second on a phone, no 4B model will do it and you should be looking at a 1 to 2B with speculative decoding. If your context is long, move weight budget into cache budget and drop a model size. If your task is narrow, a distilled or fine-tuned small model from Part 6 will beat a general model two sizes up, and that is by far the largest single win available on constrained hardware.

Takeaway: capacity decides whether you run and bandwidth decides how fast. Size weights to about two thirds of usable memory, compute \(S_{1/2} = B_w / k\) to find where context starts hurting, and never let the working set exceed capacity, because paging costs more than everything else on this list combined.

A Few Problems to Work

1. A model has \(d = 3072\), \(d_{ff} = 8192\) (per gated branch), \(h = 24\), \(d_h = 128\), \(h_{kv} = 8\), \(L = 28\), \(V = 128256\), untied embeddings. Find the parameter count per layer, the total, the fraction in embeddings, and the fp16 KV cache size at 8,192 tokens.

2. Quantise that model to int4 with fp16 scales per group of 64 (4.25 effective bits per weight) and run it on an illustrative 120 GB/s device. Give the decode bound at empty cache and at 8,192 tokens, find \(S_{1/2}\), and say what int8 cache quantisation changes.

3. Add speculative decoding to the 8,192-token case from problem 2, using a 0.5B draft at the same 4.25 bits. Assume an acceptance rate of 0.8. Find \(c\), the optimal \(\gamma\) between 2 and 8, and the resulting token rate.

4. Deploy the problem-2 model on a device with 3.0 GB usable and 250 MB of activation and runtime overhead. How many tokens of fp16 cache fit? What happens if you force a 12,000-token context, assuming mmap’d weights and illustrative 1.5 GB/s storage?

What’s Next

The arithmetic in this chapter is deliberately narrow: one model, one stream, one device, text in and text out. That is the hardest case for latency and the easiest case for analysis. Real systems widen it in two directions. They add other modalities, so that the prefill is not a prompt but an image encoder feeding thousands of visual tokens into the same decoder, and they add a deadline that comes from the physical world rather than from a user’s patience. Both change which term in the budget you are fighting, and both are what the next chapter is about.

That’s all for Part 8! For Part 9, on perception, VLMs and robots, click here.