Part 13 of How to Make Your Model Fast (Part 12: Agents | Part 0: Introduction)
What the previous thirteen parts add up to: a single reasoning procedure that survives the change from convolutions to transformers, from a robot to a phone to a cloud service, and from FLOPs to tokens. This part collects the procedure into one page, states honestly where it stops working, and points at the books and papers that taught me most of it.
I have spent this book making one argument from thirteen different angles. This part states it plainly, compresses the method into something you can pin above a desk, and then, more usefully, tells you where it fails.
Machine learning has a strong cultural bias toward the model. The model is what gets published, what gets a name, what goes in the headline. The machine it runs on is treated as plumbing: real, necessary, somebody else’s problem.
That bias made sense when models were small relative to the hardware. It stopped making sense some years ago. Today the machine is not the plumbing, it is the constraint that decides which models are buildable at all. A phone’s memory bandwidth sets a ceiling on tokens per second that no amount of kernel engineering can lift. A robot’s control loop sets a latency deadline that a more accurate model is not allowed to miss. A cloud budget sets a cost per request that decides whether a product exists. In each case the interesting engineering happens at the boundary, and the boundary has a mathematics of its own, which is what this book has been about.
The mathematics is not complicated. It is roughly: count the operations, count the bytes, divide each by the rate the hardware provides, take the larger, and compare that against the budget. What makes it valuable is not its sophistication but that almost nobody does it before starting. The gap between a team that writes the budget down in week one and a team that discovers it in month six is the difference between a shipped product and a research demo with a sad retrospective.
Takeaway: the central skill this book tries to transfer is not any specific optimisation. It is the habit of computing the bound before writing the code, and of knowing which of the small number of hard limits you are actually against.
The second argument, quieter but I think more important, is that this reasoning transfers. I did not expect that. I learned rooflines on convolutional networks running on robot hardware, and I assumed that when I moved to language models and then to agent systems I would be starting again. I was not. The decode phase of a language model is bandwidth bound for exactly the reason a depthwise convolution is bandwidth bound: too few operations per byte moved. An agent’s latency is dominated by the depth of its sequential call chain for exactly the reason a poorly fused graph is dominated by round trips to memory. The units change from FLOPs to bytes to tokens to pounds. The shape of the argument does not.
Here is the whole book as a procedure. It fits on one page deliberately.
Before anything else, write three numbers and one sentence:
| What | Example |
|---|---|
| Latency budget for the model specifically | 12ms of a 33ms frame, because capture, preprocessing and control need the rest |
| Memory budget | 40MB of weights and activations, because the device has 128MB and the rest of the system needs it |
| Cost or power budget | 1.5W sustained, or 0.4p per thousand requests |
| The sentence | “If we miss any of these, the product does not work, because …” |
If you cannot fill in the sentence, you do not yet have a budget, you have a preference. Budgets you cannot justify get negotiated away in the first difficult meeting.
Compute the three bounds from Part 1, then measure. The decision tree from Part 10 turns the measurement into a diagnosis:
| Observation | Diagnosis | First move |
|---|---|---|
| Achieved FLOPs/s near peak | Compute bound | Fewer or cheaper operations: lower precision, smaller architecture, structured sparsity |
| Achieved bandwidth near peak | Bandwidth bound | Move fewer bytes: fusion, layout, weight-only quantisation, cache compression |
| Neither near peak | Latency bound | Dependency stalls, tiny kernels, synchronisation, dispatch overhead, accelerator fallback |
| The model is fast but the system is slow | Not the model | Preprocessing, copies, format conversion, queueing, the network |
That last row is the one I want to leave you with most firmly. In my own experience the single most common outcome of a first honest profile is that the neural network was never the problem. Resizing images, converting colour spaces, copying tensors between an accelerator and the CPU, running non maximum suppression on far too many candidate boxes, waiting on a queue: this is where the time goes in real systems, and none of it appears in a FLOP count.
The levers, in the order I would try them, cheapest and most reversible first:
| Order | Lever | Typical gain | Typical cost to you |
|---|---|---|---|
| 1 | Fix the system, not the model: preprocessing, copies, threading, dispatch | Often 1.5x to 3x end to end | Days, no accuracy risk |
| 2 | Quantise weights, and activations where the hardware rewards it | 2x to 4x, more on bandwidth bound decode | Days to weeks, small accuracy risk |
| 3 | Compiler and kernel work: fusion, layout, tuned kernels | 1.2x to 2x | Weeks, no accuracy risk |
| 4 | Architecture: a smaller or better shaped model, distilled | 2x to 10x | Weeks to months, real accuracy risk |
| 5 | Structured pruning and sparsity | 1.3x to 2x | Weeks, real accuracy risk |
| 6 | Change the problem: lower resolution, shorter context, run less often | Unbounded | Hours, and a conversation with a product owner |
Row 6 is listed last and is frequently the correct answer. Running a detector every third frame with tracking in between, or truncating a context that nobody reads, buys more than any kernel I have ever written. It requires permission rather than skill, which is precisely why engineers reach for it last. Ask early.
Optimisation has no natural end, so you need an artificial one. Mine is: stop when you are inside the budget with enough margin to survive thermal throttling and a year of feature creep, or when the next lever costs more than the gain is worth. Write down the achieved efficiency against peak so the next person knows whether there is headroom left. A kernel at 70 percent of peak bandwidth is done. A kernel at 8 percent is a message from your past self.
Takeaway: budget, diagnose, spend the cheapest lever, re-measure, stop deliberately. Everything else in this book is detail attached to that loop.
A method that never fails is a method that is not saying anything. Here is where mine does.
When the bound is not the bottleneck. Rooflines assume the machine is otherwise yours. On a shared device with a noisy neighbour, or a phone where the operating system suspends you, or a cloud instance with a throttled network, the arithmetic is right and irrelevant. Measure the distribution, not the bound.
When accuracy is the binding constraint. This whole book optimises latency, memory and cost subject to accuracy staying acceptable. Sometimes accuracy is not acceptable and no amount of efficiency work matters. Recognising that early saves months. If your model is not good enough, make it good enough first, then make it fast; the reverse order wastes the optimisation work on an architecture you are about to throw away.
When the workload is not stationary. Rooflines describe one shape of one operator. Real systems have distributions: variable image sizes, variable prompt lengths, bursty traffic. A model that is comfortably inside budget at the median can be outside it at p99, and users experience p99. I have been embarrassed by this more than once.
When the hardware lies. Peak numbers on datasheets are marketing numbers achieved under conditions you do not have: perfect data placement, no other work, unlimited thermal headroom. Achieving 60 to 70 percent of peak on a real workload is a good result. If your model says you should hit 95 percent, your model is wrong.
When the answer is organisational. Some latency lives in a handoff between two teams who each think the other owns the copy. No profiler shows you that.
In the spirit of the above, four beliefs I held confidently and had to give up.
That FLOPs were a reasonable proxy for time. They are not, and the error is not small. A depthwise separable convolution can cut FLOPs by a factor of eight and latency by a factor of two, because it has almost no arithmetic intensity. I learned this the way everyone does, by promising a speedup and delivering a rounding error.
That the accelerator was always the right target. An NPU that does not support one operator in the middle of your graph can be slower than the CPU, because the partitioning fragments the network and the fallbacks cost more than the acceleration saves. Always measure against a good CPU baseline, and always check what actually got offloaded rather than what you assumed would.
That research quality and deployed quality were the same axis. The model with the best validation score is often not the model that survives contact with a customer’s environment, a different sensor, a different season, or a class distribution nobody sampled. The evaluation set is a model of the world and it is usually an optimistic one.
That the interesting work was the clever work. The highest return project I have been part of was not a clever optimisation. It was a vision language model applied to a workflow people were doing by hand, which saved over a hundred thousand pounds and made the loop about twenty times faster. It was not novel. It was useful. Those are different things and the second one is rarer.
What I would actually hand someone, roughly in the order of this book. I have deliberately kept this short; a list of two hundred papers is a way of avoiding the recommendation.
Foundations and hardware. Hennessy and Patterson’s Computer Architecture: A Quantitative Approach remains the book on the memory hierarchy and why it dominates. The original roofline paper, Williams, Waterman and Patterson, Roofline: An Insightful Visual Performance Model for Multicore Architectures, is short and worth reading in the original. Sze, Chen, Yang and Emer’s Efficient Processing of Deep Neural Networks is the best single treatment of accelerator dataflow I know.
Performance methodology. Brendan Gregg’s Systems Performance is not about machine learning at all, which is why it is the most useful profiling book for a machine learning engineer. Raj Jain’s The Art of Computer Systems Performance Analysis will make you more careful about benchmarks than any amount of experience will.
Compilers. The TVM paper, Chen et al., TVM: An Automated End-to-End Optimizing Compiler for Deep Learning, is the clearest introduction to search based compilation. Lattner et al., MLIR: A Compiler Infrastructure for the End of Moore’s Law, explains the dialect idea that most modern stacks are now built on.
Compression. Krishnamoorthi’s quantisation whitepaper, Quantizing Deep Convolutional Networks for Efficient Inference, is still the best practical starting point, and Gholami et al.’s A Survey of Quantization Methods for Efficient Neural Network Inference is the best map of the territory. For distillation, go to the source: Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network. For pruning, Frankle and Carbin’s The Lottery Ticket Hypothesis is the paper that reframed the question, even if the practical lesson turned out to be about structure rather than tickets.
Transformers and inference. Start with Attention Is All You Need, then Shazeer’s multi-query attention, which is the single change with the largest effect on edge inference economics. FlashAttention is the clearest worked example in the literature of an algorithm designed around the memory hierarchy rather than the FLOP count, which is the whole thesis of this book applied to one kernel. For serving arithmetic at the other end of the scale, Pope et al., Efficiently Scaling Transformer Inference, and Leviathan et al. on speculative decoding.
Systems and operations. Sculley et al., Hidden Technical Debt in Machine Learning Systems, is nine pages and will describe your codebase to you. Kleppmann’s Designing Data-Intensive Applications is the reference for everything in Part 11 that is not specific to models.
The book that prompted this one. How to Scale Your Model by Austin, Douglas, Frostig, Levskaya and colleagues at Google DeepMind is the definitive treatment of the opposite direction, scaling large models up on TPUs. It is excellent, it taught me a great deal, and this site borrows its Distill-derived layout with thanks. If you have read this book and want the datacentre-scale counterpart, read theirs next.
Thank you for reading. Writing this clarified my own thinking more than I expected, which I am told is the usual experience and is still surprising when it happens to you.
If you found an error, and there will be errors, I would genuinely like to know: usamahzaheer155 [at] gmail [dot] com. If something was unclear, that is a defect too, and I would like to know about that as well. I write more regularly at usamah.me, and my code is on GitHub.
Now go and profile something before you optimise it.