Rooflines, Budgets and the Cost of a FLOP

There is a question I ask before I write a line of kernel code, and I ask it again before I agree to any latency target: how fast could this possibly go? Not how fast it goes today, and not how fast I hope it will go after a fortnight of work. The ceiling. The number that no amount of cleverness gets past, because it follows from the arithmetic the model must perform and the bytes the machine must move.

Almost every performance project I have watched go badly went badly because nobody wrote that number down first. A team spends three weeks tuning a kernel that was never the bottleneck. Someone commits to 30 frames per second on hardware whose memory bus cannot stream the weights that fast even if the arithmetic units were free. Someone ships a model that passes the bench test and misses its deadline in the field because the enclosure has no fan. All three are avoidable in fifteen minutes with a pencil. This chapter is that pencil.

Five Limits, Not Three

The standard framing gives three hard limits. At the edge there are five, and the two extra ones are the ones that get people fired.

Compute, Bandwidth and Capacity

Compute is operations per second: multiply-accumulate units times clock. It is quoted per precision, because the same silicon area packs more narrow multipliers than wide ones, so a part might do 0.5 TFLOP/s in fp32, 2 TFLOP/s in fp16 and 4 TOPS in int8. Peak means every unit issuing every cycle with no stalls, which never happens, but it is a real upper bound.By convention a fused multiply-accumulate counts as two floating point operations, one multiply and one add, so a chip advertising "1 TFLOP/s" does 500 billion MACs per second. Vendors are consistent about this for FLOPs and much less consistent for integer "OPS", so check whether a TOPS figure counts a MAC as one operation or two before comparing two parts.

Bandwidth is bytes per second between the compute units and wherever the data lives. This is not one number either: there is a bandwidth to the register file, to L1, to the shared on-chip SRAM or L2, and to external DRAM, differing by one to two orders of magnitude at each step. “Memory bandwidth” without qualification usually means the DRAM figure, because that is the slowest and therefore the one that bites.

Capacity matters in two ways that people conflate. The obvious one: does the model fit in RAM at all, weights plus activations plus the runtime’s own footprint. The important one: does the working set of the current kernel fit in on-chip memory? If your tile fits in SRAM, your effective bandwidth is the SRAM number. If it spills by one byte, it is the DRAM number, and you have just lost a factor of ten.

Energy Per Operation

Every operation costs joules. On a mains-powered server that is an accounting problem; on a battery it is a physical constraint with a hard edge, which is that the robot stops. I will show later that inference energy is dominated by data movement rather than arithmetic, so the energy-optimal and latency-optimal designs usually agree, and both say the same thing: move fewer bytes.

Thermal Headroom

Peak clocks are sustainable for a few seconds from cold. A passively cooled SoC in a sealed plastic enclosure might have a sustained envelope of three to five watts while bursting to fifteen. Then dynamic voltage and frequency scaling takes over: the clock drops until the die temperature stabilises, and your compute roof drops with it.So the roofline is not a fixed diagram for a given part. Under sustained load the compute roof slides down, and in cases where the memory clock is throttled too, so does the memory roof. Always characterise the roofs at the clocks you will actually run at.

This is the limit I have seen surprise the most people. A harness that runs a model 100 times and reports the minimum latency is measuring a machine that does not exist. Budget instead against the sustained latency after twenty minutes of the real duty cycle, in the real enclosure, at the top of the ambient temperature range in the specification. As a rule of thumb, derate peak throughput by 1.3 to 2x for sustained operation, then go and measure it, because a rule of thumb is not a measurement.

Takeaway: There are five ceilings, not three. Compute, bandwidth and capacity bound what the machine can do; energy and thermal headroom bound what it can keep doing. A design that only respects the first three passes the demo and fails the field trial.

Arithmetic Intensity and the Roofline

Deriving the Bound

Take any kernel and count two things: the arithmetic operations it must perform, \(F\), and the bytes it must move between the compute units and memory, \(M\). Define the arithmetic intensity:

in operations per byte. This ratio, and not the size of the kernel, determines which limit binds.

The machine gives you two rates: \(C\) operations per second and \(B\) bytes per second. The arithmetic cannot finish in less than \(F / C\) seconds, the data cannot arrive in less than \(M / B\) seconds, and both must happen. If the machine overlaps them perfectly, prefetching the next tile while multiplying the current one, the total time is whichever takes longer:

Divide \(F\) by that time to get achievable throughput, and substitute \(M = F / I\):

That is the roofline.Usually attributed to Williams, Waterman and Patterson, "Roofline: an insightful visual performance model for multicore architectures", Communications of the ACM, 2009. Bounding performance by the slower of two resources is a much older idea, but that paper turned it into a tool people actually draw. Plotted with intensity on the x axis and throughput on the y, both logarithmic, it is a slanted line of slope \(B\) that flattens into a horizontal line at height \(C\). Every kernel is a point underneath it.

The Ridge Point

The two pieces meet where \(B \cdot I = C\), which gives the ridge point:

This is the most useful single number about a piece of hardware: the intensity a kernel needs before arithmetic rather than memory limits it. Below it you are memory bound and adding compute units changes nothing. Above it you are compute bound and a faster memory bus changes nothing.

Ridge points of representative machines, as peak operations per second divided by peak DRAM bandwidth. Illustrative parameter sets, not measurements of any specific product.

Compute has grown far faster than bandwidth for decades, so ridge points have climbed. A machine at 160 ops per byte demands enormous data reuse before its arithmetic units are the constraint, and most real layers at batch one are nowhere near it. This is the central fact of edge inference, and much of the rest of this book is a consequence of it.

When the Maximum Is Too Optimistic

The \(\max\) is doing a lot of work. It assumes perfect overlap of computation and data movement, which is reasonable for a large regular kernel on hardware with prefetchers, double-buffered DMA or enough concurrent threads to hide latency, and a bad assumption otherwise. The honest statement brackets the true time:

The lower end is perfect overlap, the upper end none at all, and where a real kernel sits between them is a property of the code, not the algorithm. Four things break the model in practice:

1. Latency-bound regimes. With too little independent work in flight to saturate either resource, you are bound by neither rate but by the round-trip latency of the dependency chain.This is Little's law applied to memory: achieved bandwidth equals outstanding requests divided by the latency of each. A machine with 100 ns latency needing 50 GB/s must keep roughly 5 KB of requests in flight at all times, and small kernels, pointer chasing and dependent chains cannot.
2. Counting the wrong bytes. \(M\) must be the traffic that crosses the level you assigned \(B\) to. If you count each tensor as read once but the implementation re-reads a weight tile per output block because the tile does not fit in cache, the model was not wrong, your byte count was.
3. Overhead. Kernel launch, dispatch, synchronisation and allocation are not in the model at all. A network of 200 small layers on a runtime with 20 microseconds of per-layer overhead has spent 4 ms before any arithmetic happens.
4. The roof is soft. A tuned large GEMM might reach 60 to 80 percent of peak compute, a tuned streaming kernel 70 to 85 percent of peak bandwidth. Build budgets with effective roofs measured by microbenchmark, not datasheet roofs.

Takeaway: Compute arithmetic intensity first, compare it to the ridge point, and you know which resource binds before you profile anything. The roofline is a lower bound on time, so a measurement slower than the bound is expected and a measurement faster than the bound means you counted something wrong.

A Worked Example: One fp32 Matmul

Take a hypothetical edge device with stated parameters: 2 TFLOP/s peak fp32 and 50 GB/s of DRAM bandwidth. Those are plausible for an embedded SoC with a 64-bit LPDDR5 bus, and I state them as assumptions rather than quoting a product. Its ridge point is

Forty floating point operations for every byte that crosses the bus. In fp32, where every element is four bytes, that is 160 operations per element touched. Hold that number in your head.

Now a matrix multiply. Write the activation as \(A\) of shape \(m \times k\) and the weights as \(W\) of shape \(k \times n\), using lowercase for the dimensions so nothing collides with \(M\) for bytes and \(B\) for bandwidth:

taking the ideal case where each matrix is read or written exactly once.

Two matmuls on the same device, one square and one skinny. The device is 2 TFLOP/s and 50 GB/s, ridge point 40 FLOP/byte.

The square matmul sits well above the ridge at 170 FLOP/byte, so it is compute bound and the bus is idle two-thirds of the time. That is the case the datacentre optimises for, and the case in which a better kernel genuinely helps.

The skinny one is more interesting. It does a thousand times less arithmetic but reads almost as many bytes, because the 1024 x 1024 weight matrix has to come in regardless of how many rows you push through it. At 0.5 FLOP/byte, eighty times below the ridge, it runs at 1.25 percent of the machine’s advertised peak. Shown that number without the analysis, most people file a bug against the kernel. There is no bug. The kernel is at the roof; it is just a different roof.

Why Small Matmuls Are Always Memory Bound

The general case gives a rule you can apply in your head. When \(k\) and \(n\) are large and \(m\) is small, the weight matrix dominates the byte count, so

where \(s\) is the bytes per weight. The result depends only on the row count \(m\) and the precision. Everything else cancels.

Arithmetic intensity of a weight-dominated matmul depends only on the row count and the weight precision. The right-hand column is the row count needed to reach a ridge point of 40.

At batch one with fp32 weights you would need eighty rows of work per weight load before this device’s arithmetic units become the constraint. You have one. That gap is not something a compiler closes.

Two things follow. First, quantisation buys latency on a memory-bound layer directly, by shrinking the denominator: int8 weights move a quarter of the bytes and the layer gets close to four times faster with no change to the arithmetic units. That is a much bigger effect than the arithmetic speedup usually cited for it, and it is why Part 5 sits where it does in this book. Second, the ridge point moves too, because int8 peak throughput is typically two to four times the fp32 figure on the same part. Both roofs move. Bandwidth almost always wins that race, so quantisation is still a win, but redo the arithmetic rather than assume it.

Pure Bandwidth: Elementwise, Norms and Activations

Matmuls and convolutions are the layers people talk about. They are usually not where the time goes.

An elementwise addition of two fp32 tensors reads eight bytes, writes four and performs one addition: an intensity of \(1/12 \approx 0.083\) FLOP per byte, 480 times below our reference ridge point. A ReLU is 0.125, a fused layer normalisation about 1, a softmax under 1. A gather or embedding lookup does no arithmetic at all, so its intensity is exactly zero, and worse than zero in practice because random access pulls a whole cache line for every scattered element you wanted.A 4-byte gather from a random index costs a 64-byte cache line transfer on most systems, so effective bandwidth can be a sixteenth of the streaming figure and an embedding layer's real cost an order of magnitude above what a naive byte count suggests. Sort or bucket your indices where you can.

Arithmetic intensity of common operations, fp32, counting compulsory DRAM traffic only. Shapes are stated where the answer depends on them. The reference device has a ridge point of 40 FLOP/byte, so everything below about 40 is memory bound on it.

The two convolution rows hold the most instructive result in the table. The depthwise convolution does 256 times fewer FLOPs than the standard one over the same tensor shapes, and moves 73 percent as many bytes. On our device the standard convolution is compute bound at 1.85 ms and the depthwise convolution is memory bound at 129 microseconds, so the real speedup is 14x, not 256x.

A FLOP count is a proxy for latency that holds only above the ridge point, and the operators introduced to cut FLOPs are precisely the ones that fall below it. That is why architectures designed by counting FLOPs so often disappoint on real hardware.It is also why the literature moved towards latency-aware and hardware-aware architecture search rather than FLOP-constrained search. The operators people reach for to reduce FLOPs, depthwise and grouped convolutions in particular, are exactly the ones that push a layer below the ridge point.

Fusion Is the First Thing I Reach For

If most operators are bandwidth bound, the time of a region is proportional to how many times you traverse the tensor, because every unfused operator costs at least one read and one write of the whole thing to DRAM.

Take a residual block tail: a convolution, then batch normalisation, then an activation, then an addition with the skip connection. Unfused, that is the conv writing its output, batch norm reading and writing, the activation reading and writing, and the add reading two tensors and writing one. Eight full traversals. Fused into the convolution’s epilogue, with intermediates staying in registers and on-chip memory, it is one read of the skip tensor and one write of the result. Two traversals: a four times cut in DRAM traffic, and so roughly a four times cut in time, from a transformation that changes no numerics at all.Batch normalisation at inference time is better still, because it is an affine transform with constant parameters and folds directly into the preceding convolution's weights and bias, costing nothing at runtime. Any serious deployment toolchain does this automatically; if yours does not, that is a red flag about the rest of it.

I have never found a transformation with a better ratio of engineering effort to latency won. Before I write a custom kernel, before I quantise, before I touch the architecture, I check what the toolchain actually fused, because the answer is frequently “less than you think”. One unfused reshape or cast between two operators can break a fusion group and quietly cost two tensor traversals. Much of my kernel-level analysis work at Arm has been exactly this: finding the boundaries the compiler refused to cross and working out why. Part 4 is about those refusals. The same reasoning applied to attention is the whole idea behind fused attention kernels: never materialise the attention matrix in DRAM, compute it in tiles on chip, and use a numerically stable running softmax so no separate pass over the row is needed.

Takeaway: In a bandwidth-bound region, time is proportional to the number of tensor traversals, not to the number of operations. Fusing four operators into one turns eight traversals into two, and no amount of arithmetic tuning comes close to that.

The Latency Budget Is the Real Constraint

Everything above bounds the model. But nobody ships a model; they ship a system with a deadline. On a robot running a perception loop at 30 frames per second the deadline is 33.3 ms, and the model gets a slice of it.

This is the framing I learned to use at Dyson, fitting segmentation and detection networks onto robot hardware. The question was never “how fast is the network”. It was “what is left for the network after everything else the frame has to do”, and the answer was always smaller than people expected.

Where Thirty-Three Milliseconds Goes

An illustrative frame budget for a 30 fps perception loop on a mobile robot. The stage times are plausible assumptions to show the shape of the problem, not measurements from any product.

The model gets 12 ms, about 36 percent of the frame. That is for one model. Run a detector and a segmenter and each gets six. Add a depth network and you are at four each, at which point the runtime’s per-layer dispatch overhead is a material fraction of your budget.

Three refinements matter more than the exact split.

Budget against the tail, not the mean. A model with a 10 ms mean and a 25 ms 99th percentile misses one frame in a hundred, which at 30 fps is once every three seconds. A dropped frame means the control loop extrapolates from stale state, and the failure mode is not “slightly worse metrics”, it is bumping into things. Budget against p99 at sustained clocks and treat the mean as a curiosity.The usual sources of a bad tail are thermal throttling, memory allocation in the hot path, page faults on first touch of a buffer, garbage collection or Python in the loop, and contention with another process over the shared last-level cache or memory bus. The last is hardest to find, because your model is innocent and something else changed.

Latency is not throughput. Pipelining the stages across cores gives 30 fps of throughput while any individual frame takes 60 ms end to end. For a display that is fine. For a robot moving at one metre per second, 60 ms of staleness is six centimetres between what the model saw and where the machine now is. Decide which your product needs and write it down, because the two lead to different designs.

The budget is a veto, not a target. If the roofline bound alone exceeds the slice, stop: no kernel work will save you, and you must change the model, the input resolution or the precision. The earlier you discover that, the cheaper it is.

Takeaway: The model typically gets a third of the frame, measured at the 99th percentile, at sustained clocks, sharing the machine with everything else. Compare the roofline bound against that slice before committing to an architecture, not after.

Energy, Data Movement and Batch Size One

The Cost of Moving a Byte

The relative energy costs of arithmetic and data movement have been widely cited since Horowitz’s 2014 keynote on computing’s energy problem, and reproduced in the model compression literature since. The figures below are order-of-magnitude values for a 45 nm process. Absolute numbers have improved on newer nodes, but the ratios, which are what matter for design, have barely moved, because wires have not scaled the way transistors have.

Approximate energy per operation, 45 nm, normalised to an int8 multiply-accumulate. Treat these as orders of magnitude, not measurements of any particular chip.

Read the bottom row against the middle and the whole discipline of efficient inference falls out of it. Fetching one 32-bit weight from DRAM costs roughly as much energy as three thousand int8 multiply-accumulates.The 640 pJ figure for a 32-bit DRAM access is the one reproduced in the EIE and Deep Compression line of work; Horowitz's original slides give 1.3 to 2.6 nJ depending on DRAM generation and access pattern. The conclusion is the same either way and robust to that factor of four.

So if a weight is used fewer than about three thousand times after you fetch it, the energy of your inference is in the fetch, not the arithmetic. Arithmetic is free; memory is the bill. That is why energy-optimal and latency-optimal designs point the same way, and why the two big edge techniques, quantisation and fusion, both work by moving fewer bytes rather than by doing less maths.

A sanity calculation: 640 pJ per 4 bytes is about 160 pJ per byte, so a model moving 40 MB per inference spends at least \(40 \times 10^{6} \times 160 \times 10^{-12} \approx 6.4\) mJ on traffic alone, about 0.19 W at 30 inferences per second. That is a floor, excluding on-chip SRAM, control logic, clock distribution and leakage, which in a real part often add up to several times it.

Batch Size One Changes the Arithmetic

Datacentre inference batches, because batching is free money: fetch a weight once, use it across 64 rows, and intensity goes up 64-fold. Reasoning that starts from batched serving concludes that you should care about arithmetic units and FLOP efficiency, and for that workload it is correct.

At the edge there is one camera, one frame, one user, one utterance. Batch size is one for a reason you cannot engineer away: the next frame does not exist yet, and waiting for it adds 33 ms of latency to buy throughput you do not need. So every weight byte is fetched and used for exactly one multiply-accumulate per output it contributes to, nothing is amortised, and for any layer whose weights dominate its byte count, inference time collapses to one expression:

This is the most useful back-of-envelope formula in edge machine learning. A 7B-parameter model quantised to int8 is 7 GB of weights, so on a 50 GB/s device it cannot decode faster than \(7 / 50 = 140\) ms per token, about 7 tokens per second, or 5 at a realistic 70 percent of peak. Quantise to int4 and it doubles. Buy a faster processor on the same memory bus and it does not move at all.

One exception separates vision from language at the edge. Convolution reuses each weight across every spatial position: a 3x3 kernel on a 56 x 56 feature map uses each weight 3,136 times, so a CNN’s heavy layers keep high intensity even at batch one. Transformer decode has no such reuse, since each weight serves one token. That is why a CNN at batch one can be compute bound and a language model at batch one essentially never is.Prefill is the mirror image. Processing a 2,000 token prompt is a matmul with 2,000 rows, intensity roughly 2,000 times the decode case, firmly compute bound. The same model on the same machine crosses the ridge point depending on which phase it is in, which is why prefill and decode need different optimisations and sometimes different hardware. Part 8 picks this up in detail.

Takeaway: Data movement costs roughly a thousand times more energy than arithmetic, and at batch one nothing is amortised, so for weight-dominated workloads both time and joules are set by total weight bytes divided by effective bandwidth. Optimise the bytes.

A Performance Budget in Fifteen Minutes

Here is the procedure. It takes about a quarter of an hour and I run it at the start of every project, before any code gets written.

1. Write down the machine. Peak operations per second at the precision you will actually deploy, DRAM bandwidth, on-chip SRAM size, and the sustained power envelope. Start from the datasheet, then replace those with measured effective values: a large tuned GEMM for compute, a streaming copy for bandwidth. Use 0.6 of peak compute and 0.7 of peak bandwidth as placeholders until you have measured.Measuring effective bandwidth is fiddlier than it looks. A copy benchmark measures read plus write, a read-only benchmark gives a different and usually higher figure, and both change with access pattern, page size and thread count. Pick one definition, write it next to the number, and stay consistent, because a factor of 1.5 hiding in your definition of B makes every later conclusion wobble.
2. Write down the model. Parameter count, weight bytes at target precision, FLOPs per inference, and the peak activation working set. A profiler gives you the first three; the fourth needs thought about which tensors are live at once.
3. Compute both bounds and take the maximum. That is your floor.
4. Compare the floor to your slice of the latency budget. If the floor exceeds the slice, the project as specified is impossible, and you should say so now.
5. Sort the layers by arithmetic intensity. Below the ridge point is a bandwidth problem, addressed by fusion, quantisation, layout and tiling. Above it is a compute problem, addressed by better kernels. The two lists get different engineers.
6. Compute the energy floor as DRAM bytes times 160 pJ, and check it against the power envelope and the battery.

Most of this fits in a script you write once per device.

# A roofline calculator. Fill in the device numbers once per project.
PEAK_OPS    = 2.0e12   # fp32 ops/s, datasheet
PEAK_BW     = 50e9     # DRAM bytes/s, datasheet
COMPUTE_EFF = 0.60     # what a large tuned GEMM actually reaches
BW_EFF      = 0.70     # what a streaming copy actually reaches

C = PEAK_OPS * COMPUTE_EFF
B = PEAK_BW * BW_EFF
RIDGE = C / B

def bound(ops, byts):
    t_compute = ops / C
    t_memory = byts / B
    return {
        "intensity": round(ops / byts, 2),
        "t_compute_ms": round(t_compute * 1e3, 3),
        "t_memory_ms": round(t_memory * 1e3, 3),
        "floor_ms": round(max(t_compute, t_memory) * 1e3, 3),
        "limit": "compute" if t_compute > t_memory else "memory",
        "energy_floor_mj": round(byts * 160e-12 * 1e3, 3),
    }

def matmul(m, n, k, wbytes=4):
    return 2 * m * n * k, wbytes * (m * k + k * n + m * n)

print(f"effective ridge point: {RIDGE:.1f} ops/byte")
for name, shape in [("square 1024", (1024, 1024, 1024)),
                    ("batch-1 linear", (1, 1024, 1024))]:
    print(name, bound(*matmul(*shape)))

When the Measurement Disagrees with the Model

It will. That is the point of having a model: the size of the disagreement tells you what kind of problem you have. Here is how I read the ratio of measured time to the roofline floor.

Diagnosing the gap between measured latency and the roofline floor. The ratio is measured time divided by the bound from step 3.

The first row is the one people get wrong. If you measure faster than your own lower bound you have not found a fast machine, you have a bug in your measurement or your accounting, and I have wasted whole afternoons celebrating one before finding it. Suspect the timer first: on asynchronous hardware, a timer that does not synchronise measures how long it took to enqueue the work.

The discipline I have ended up with is simple. Never optimise a kernel before you can state its roofline bound and the ratio of measured to bound. If you cannot say “we are at 22 percent of the memory roof on this layer”, you do not yet know what you are doing, and any speedup you find is luck. Luck does not generalise to the next model.

Takeaway: The model is a measuring instrument, and the gap between it and reality is the diagnostic. A 1.2x gap means stop; a 3x gap means find the bytes you did not count; a 20x gap means you are paying overhead, not bandwidth.

A Few Problems to Work

Problem 1. A mobile NPU is specified at 4 TOPS int8 with 25 GB/s of DRAM bandwidth. Find its ridge point. Then take a batch-1 fully connected layer of 2048 inputs to 2048 outputs with int8 weights and int8 activations. Compute its arithmetic intensity, its bound, and the fraction of peak throughput it achieves. Assume ideal traffic and count a MAC as two operations.

Problem 2. A residual block tail produces a tensor of shape 1 x 64 x 112 x 112 in fp16, then applies batch normalisation, then ReLU, then an elementwise add with a skip connection of the same shape. Count the DRAM traffic unfused and fused, on a device with 50 GB/s of effective bandwidth. How much time does fusion save per block, and per network if there are sixteen such blocks?

Problem 3. A 3B-parameter decoder-only language model is quantised to int4 weights. It has 26 layers, a model dimension of 3072, and grouped-query attention with 8 key-value heads of dimension 96, with the KV cache kept in fp16. The device has 68 GB/s of peak bandwidth and achieves 70 percent of it. Find the decode throughput ceiling at short context, and again at a context of 4,096 tokens.

Problem 4. A compute-bound vision model measures 9.5 ms on the bench from cold. In the product enclosure at maximum ambient temperature, the sustained clock settles at 72 percent of the benchmark clock, and the p99 latency is 1.35 times the mean. The frame budget allocates the model 12 ms. Does it fit? What speedup do you actually need?

What’s Next

You can now bound a workload before writing code: count the operations and the bytes, form the intensity, compare it to the ridge point, take the maximum of the two times, and check the result against the slice of the frame budget and the energy envelope you actually have. That gets you the right question. It does not yet tell you why the machine behaves the way it does, or why the effective bandwidth is 70 percent rather than 100, or where the on-chip memory that makes fusion possible actually lives. For that we have to open the box.

That’s all for Part 1! For Part 2, on what is actually inside an edge accelerator, click here.