Profiling: Finding the Real Bottleneck

Every optimisation I have regretted started the same way. I looked at a model, decided from the shape of it what must be slow, and fixed that. Sometimes I was right. More often I spent a fortnight making a convolution 30% faster and moved end to end latency by 2%, because the convolution was 7% of the frame and the other 93% was sitting in a layout conversion, a delegate handoff and a NumPy loop nobody had touched since the prototype.

This chapter is the method for not doing that. It is deliberately not a tool tour. Tools change every eighteen months; the reasoning does not. We will build a benchmark that survives contact with a real device, descend through the levels of measurement until the bottleneck has nowhere left to hide, turn hardware counters into the only two numbers that really matter (achieved FLOPs/s and achieved bandwidth), and use those two numbers to decide what to change. Everything here assumes the roofline from Part 1 and the kernel mechanics from Part 3.

Measure Before You Optimise

The first rule is that a performance number with no error bar and no reproduction recipe is a rumour. I have watched entire sprints get spent on rumours. Before you profile anything, you need a benchmark whose output you would be willing to defend in a review.

What Makes a Benchmark You Can Trust

A benchmark is a measurement instrument, and like any instrument it has systematic and random error. Your job is to drive both down until the signal you care about, usually a 5 to 10% change, is comfortably larger than the noise.

Sources of run to run variance, and the control for each.

Here is the shape of a timing loop I actually trust. The important parts are the warmup, the synchronisation on both sides of the timed region, and the fact that it keeps every sample rather than accumulating a running mean.

import time
import torch


def bench(fn, *, warmup=50, iters=500, device="cuda"):
    """Time fn() properly. Returns median, p99 and min in milliseconds."""
    if device == "cuda":
        sync = torch.cuda.synchronize
    else:
        sync = lambda: None

    for _ in range(warmup):      # caches, autotuners, JIT, clock ramp
        fn()
    sync()

    samples = []
    for _ in range(iters):
        sync()                   # drain anything still outstanding
        t0 = time.perf_counter_ns()
        fn()
        sync()                   # the work is not done until this returns
        samples.append(time.perf_counter_ns() - t0)

    samples.sort()
    n = len(samples)
    return {
        "median_ms": samples[n // 2] / 1e6,
        "p99_ms": samples[min(n - 1, int(0.99 * n))] / 1e6,
        "min_ms": samples[0] / 1e6,
    }

Two warnings. Per-iteration synchronisation costs 5 to 20 microseconds on a GPU, so if the thing you are timing takes 30 microseconds you are measuring the synchronisation; time a batch of K iterations with one synchronisation at each end instead, accepting that this hides launch latency.Both numbers are worth having and they answer different questions. The per-iteration figure is the latency of one call, which is what a batch-1 deadline cares about. The batched figure is the throughput the hardware sustains when successive calls overlap, which is what a serving system cares about. Quoting one to an audience that wanted the other is a common way to be accidentally wrong by 3x. And benchmark the configuration you ship: same thread count, same allocator, same input distribution. A detector timed on a blank frame never hits the non-maximum suppression path that dominates its p99 in the field.

Median, p99 and Why the Mean Lies

The mean is the wrong summary for latency because latency distributions are right-skewed and the right tail is usually not your code. It is a page fault, a scheduler decision, an interrupt, a garbage collection in the surrounding application. One 200 millisecond outlier in 500 runs of an 8 millisecond kernel shifts the mean by 0.4 milliseconds, which is exactly the size of the improvement you were trying to detect.

Two versions of the same kernel, summarised four ways. All times in milliseconds.

A and B are indistinguishable in median and min. The kernel is the same. What differs is the tail, and if you only reported the mean you would conclude that B is 29% slower and go hunting inside a kernel that did not change. Report the median when you are iterating on a kernel, and the p99 when you are arguing about whether a system meets a frame budget.A p99 needs at least a hundred samples to exist at all, and in my experience closer to a thousand before it is stable enough to compare between builds. If you only ran fifty iterations, your "p99" is the second slowest run, which is a sample of the tail, not an estimate of it.

The Observer Changes the Observed

Every measurement perturbs the thing measured, and the perturbation grows sharply as you descend the hierarchy.

A sampling profiler at 1 kHz, at 1 to 2 microseconds of handler cost per sample, is about 0.1 to 0.2% overhead and is fine. The same profiler at 100 kHz is 10 to 20% overhead and pollutes the caches it is trying to observe. An instrumentation profiler that wraps every operator costs 1 to 10 microseconds per operator; on a graph of 300 operators running in 8 milliseconds that is 4 to 37% overhead, concentrated entirely on the small operators, which then look disproportionately expensive. Kernel-level GPU profilers that replay each kernel to collect full counter sets give accurate per-kernel numbers and a fictional timeline, because they serialise work that was originally concurrent.

My working rule: always keep one clean, uninstrumented wall clock number as ground truth, and at every level check that the instrumented total is within about 10% of it. If the profiler says the model takes 14 milliseconds and the stopwatch says 8, you are no longer studying the model. You are studying the profiler.

Takeaway: Fix the clocks, pin the threads, warm up, soak, keep every sample, and report median and p99. A change smaller than your measurement noise is not a change, and a profiler heavy enough to distort the total is not evidence.

The Hierarchy of Measurement

Profiling is a search, and the efficient way to search is to narrow by an order of magnitude at each step rather than jumping straight to the finest instrument. There are five levels, each answering a different question.

The five levels of measurement.

Level 1 is the only number anyone outside the team cares about, and it is the only one you are allowed to claim an improvement in. Level 2 is where most bottlenecks are actually found, and it is criminally underused: ten perf_counter calls around the stages of a pipeline will, more often than not, tell you that the model is a minority of the frame. Level 3 tells you which operator to look at, and importantly how many times each operator was called, which is how you discover that a “cheap” reshape is running 480 times per frame. Level 4 tells you the mechanism. Level 5 tells you the line.

The trap at level 2 is asynchronous execution. A timer around a GPU kernel launch or an accelerator submit measures the time to enqueue the work, not to do it, and that is usually about 5 microseconds regardless of how much work you enqueued. On a GPU you need device events; on an NPU or DSP you need the delegate’s own timestamps; on any asynchronous system, a wall clock measurement without a synchronisation is a measurement of your own optimism.

Descend one level only when the level above has narrowed the search. Going straight to instruction sampling on a 40 millisecond frame gives you a flat profile with 200 entries and no story.

Hardware Counters and the Arithmetic They Feed

Performance counters are a small set of hardware registers that count architectural events. There are usually only four to eight programmable ones per core, so asking for twelve events means the kernel time-multiplexes them and scales the results, which introduces error.`perf stat` reports the multiplexing fraction next to each event, something like `(66.67%)`. If you see anything below about 90% on an event you care about, split the run into two or three passes with fewer events each and combine the results. Scaled counters from a 30% sampling window are not worth arguing over.

The counters I actually use, and what a bad value means.

That last row is the one people forget, and it is often the fastest diagnosis in the whole chapter. Divide the algorithmic FLOP count by the retired instruction count. If a fp32 kernel on 128-bit SIMD is producing fewer than about 3 FLOPs per instruction, the compiler did not vectorise your loop and nothing else you measure matters yet.

Here is the invocation I start with on an Arm or x86 Linux target.

#!/usr/bin/env bash
# Fix the clocks, pin to one core, read the counters that diagnose.
sudo cpupower frequency-set -g performance > /dev/null

# Pass 1: issue efficiency and stalls.
perf stat -r 5 -e cycles,instructions,\
stalled-cycles-frontend,stalled-cycles-backend,branch-misses \
  -- taskset -c 3 ./bench --iters 500 --warmup 50

# Pass 2: the memory hierarchy. Split from pass 1 so that no
# event gets multiplexed onto a shared counter.
perf stat -r 5 -e L1-dcache-loads,L1-dcache-load-misses,\
LLC-loads,LLC-load-misses,dTLB-load-misses \
  -- taskset -c 3 ./bench --iters 500 --warmup 50

# What the generic names map to on this core, plus the raw
# microarchitectural events perf does not alias.
perf list hw cache
perf list | grep -i -E 'stall|mem_access|bus_cycles'

The generic event names are aliases that map to different underlying PMU events on different microarchitectures, and on some cores they do not map to anything sensible at all. When a number looks impossible, check perf list and switch to the raw architectural event.

From Counters to Achieved Bandwidth

Achieved DRAM bandwidth is the single most useful derived number in profiling, and you compute it from last-level cache misses:

Worked example. A single-threaded depthwise convolution runs 200 iterations in 0.98 seconds and reports 2.10e7 LLC load misses. With 64 byte lines:

Now compare that to what the algorithm must move. The tensor is 1x64x112x112 fp32, so 802,816 elements, 3.211 MB in and 3.211 MB out per iteration. Because the cache is write-allocate, storing the output also generates read traffic, so expected reads are about \(2 \times 3.211\ \text{MB} \times 200 = 1.284\ \text{GB}\). Measured 1.344 GB is 4.7% above compulsory. The memory system is behaving perfectly. Whatever is slow here is not the cache hierarchy.Cache-miss counters undercount real DRAM traffic. Hardware prefetchers issue requests that never appear as demand misses, write-backs of dirty lines are a separate event, and non-temporal stores bypass the cache entirely. When precision matters, read the memory controller counters directly: `uncore_imc` events on x86, or the DMC and interconnect PMUs that Arm Streamline exposes. The LLC-miss estimate is usually within 10 to 20% and is good enough to tell 15% of peak from 85% of peak, which is all the decision tree needs.

From Counters to Achieved FLOPs/s

There are two ways to get FLOPs/s. You can count floating point operations with the PMU, using events such as fp_arith_inst_retired.* on x86 weighted by vector width, or FP_SCALE_OPS_SPEC and FP_FIXED_OPS_SPEC on recent Arm cores.The `_SPEC` suffix means speculatively executed, not architecturally retired. On a deeply out-of-order core with branchy code the speculative count can exceed the useful count noticeably. For a dense tensor kernel with predictable control flow the difference is usually small, but it is another reason to prefer the analytic count.

Or you compute it analytically from the shapes, which is what I do almost every time. For a GEMM of shape \(M \times K\) by \(K \times N\) it is \(2MKN\); for a convolution it is \(2 \cdot C_{in} \cdot C_{out} \cdot K_h \cdot K_w \cdot H_{out} \cdot W_{out}\). The analytic figure counts useful work, which is exactly the numerator you want. If the hardware retired three times that many operations, that is a fact about your implementation, not about the problem.

Back to the Roofline

With achieved bandwidth and achieved FLOPs/s you can place any kernel on the roofline from Part 1. Here is the illustrative device I will use for the rest of the chapter. It is a plausible mid-range mobile class part; every number is derived from stated parameters rather than measured on anything real.

Illustrative target device.

Worked example. A decoder projection at batch 1: a 4096 by 4096 fp16 weight matrix times a vector. FLOPs are \(2 \times 4096 \times 4096 = 33.55\) MFLOP. Bytes are dominated by the weights, \(4096 \times 4096 \times 2 = 33.55\) MB. So the arithmetic intensity is exactly

which is 1/9.8 of the ridge point. This kernel is bandwidth bound by nearly an order of magnitude, and no amount of clever inner-loop work will change that. The attainable rate is \(1.0 \times 13 = 13\) GFLOP/s, which is 10% of the machine’s compute peak, and the ideal time is \(33.55\ \text{MB} / 13\ \text{GB/s} = 2.58\) ms. Measure it and you get 4.6 ms, so achieved bandwidth is \(33.55\ \text{MB} / 4.6\ \text{ms} = 7.29\) GB/s, or 56% of achievable. There is a 1.8x sitting in the memory access pattern before you change a single thing about the model.

Takeaway: Two derived numbers settle almost every argument: achieved FLOPs/s as a fraction of compute peak, and achieved bandwidth as a fraction of stream-achievable peak. Compute the first analytically from shapes and the second from cache-miss counters, then place the kernel on the roofline.

A Decision Tree for Diagnosis

Call those two fractions \(f_c\) and \(f_b\). The diagnosis is then mechanical:

• \(f_c > 0.6\): compute bound.
• \(f_b > 0.6\): bandwidth bound.
• both below 0.4: latency bound.
• both between 0.4 and 0.6: usually a kernel with two phases. Split it and measure the phases separately, because the average of a compute bound phase and a bandwidth bound phase looks like neither.

Branch One: Near Peak FLOPs/s

If you are at 70 to 85% of compute peak, congratulations, the implementation is good. Real dense kernels rarely exceed about 85% because of prologues, edges and instruction mix. The remaining 15% is typically a week of work for a 5% gain, and it is almost never the best week available to you.

The only way out is less arithmetic or cheaper arithmetic. In descending order of what has worked for me: drop precision, since int8 typically buys 2 to 4x on the same silicon (Part 5); use a cheaper algorithm, such as Winograd for small-kernel convolutions or a low-rank factorisation of a large projection; remove parameters by structured pruning or distillation (Part 6); shrink the architecture, the largest single lever and the one nobody wants to pull; and skip work at runtime with early exit, cascades or token pruning.

Branch Two: Near Peak Bandwidth

If you are at 70% or more of stream-achievable bandwidth, the kernel is running about as fast as the memory system allows, and the only fix is to move fewer bytes.

Fusion is the biggest lever and the arithmetic is simple. A chain of \(k\) elementwise operations over a tensor of \(N\) bytes, each materialised to memory, moves roughly \(2kN\) bytes. Fused into one pass it moves \(2N\). That is a saving of \(2(k-1)N\), and for a chain of four operations it is a 4x reduction in traffic on those layers. After that: change the layout so the fastest-varying axis is the one you vectorise over, usually NHWC on a CPU; drop precision, since int8 weights halve the traffic of every weight-bound layer; block for cache reuse so a tile is loaded once and used many times; and batch, the only way to amortise weight traffic when the weights dominate, as they do for every matrix-vector product in a decoder.

Branch Three: Near Neither

This is the most common branch in practice and the one people most often misdiagnose as “the model is just slow”. If you are at 9% of compute peak and 20% of bandwidth peak, you are not limited by any resource. You are limited by waiting.

Latency bound sub-causes, the tell for each, and what to do.

The reason this branch matters so much on edge and batch-1 workloads is arithmetic. A frame that issues 300 operators with a 20 microsecond dispatch cost each has spent 6 milliseconds before any arithmetic happens. If the tensors are small, dispatch can exceed compute by a factor of five and every counter will look idle, because the machine genuinely was idle.

Takeaway: If neither compute nor bandwidth is near peak, stop optimising the kernel. You are latency bound, and the fix is structural: fuse, batch, remove synchronisation, fix parallelism, or get out of the interpreter.

The Overheads That Hide Outside the Model

The cheapest test in profiling is this: sum the per-operator times your framework reports, and compare to the wall clock. If operators sum to 6.2 milliseconds and the frame takes 11.4, then 5.2 milliseconds, 46% of your budget, is in something no operator profiler will ever show you.

Overheads outside the operators. Figures are illustrative orders of magnitude from common practice on small-batch workloads, not measurements of any particular system.

Only a timeline trace shows these honestly. An operator profile is a bar chart of operators; by construction it cannot show the gaps between them. On a trace of a small model driven from Python the shape I expect is a thin spike of real work followed by a wide flat band of dispatch, and the gaps between spikes are the bottleneck.Python's per-operator cost is not mostly the global interpreter lock, which a well-behaved framework releases around the kernel call. It is argument parsing, type and device dispatch, reference counting and object allocation on the way in and out. That is why tracing or compiling the graph, which collapses hundreds of Python-level calls into one dispatch, usually helps far more than switching to more threads.

Two of these deserve special mention. Delegate handoffs are the reason an “8 millisecond NPU model” can take 21 milliseconds: if three operators in the middle of your graph are unsupported, the runtime partitions the graph around them, and you pay a handoff and a buffer copy at every boundary. Always check the partition count, not just the operator support list. And first inference cost is not a benchmarking artefact you are allowed to discard if your product does one inference when a user presses a button. In that case, the first inference is the latency, and the fix is to pre-warm at startup.

Takeaway: Sum the operator times and compare to the wall clock. The gap is your overhead budget, and at batch 1 it is routinely 30 to 60% of the frame. No kernel optimisation touches it.

Tools, and How to Read What They Show You

Four Categories of Tool

Sampling profilers, of which perf is the reference implementation, interrupt the program at a fixed rate and record where it was. They are statistical, so they answer “where does time go” with an error bar that shrinks as the square root of the sample count, and they are almost free at sensible rates. They are the right first tool for any CPU workload, they need no code changes, and with perf record -g they give you a call graph. They cannot tell you about time spent waiting on another device, because a stalled thread is not sampled.

Timeline and trace viewers record timestamped begin and end events and draw them on a shared time axis. The PyTorch profiler emitting Chrome trace JSON, loaded into Perfetto, is the workhorse here.Perfetto at ui.perfetto.dev reads the legacy Chrome Trace Event JSON format directly, so anything that emits that format, including the PyTorch profiler, TensorFlow's trace export and most hand-rolled instrumentation, will load. Perfetto's query engine also lets you sum slice durations by name, which is a much faster way to answer "how much total time is in layout conversions" than scrolling. Traces are the only tool that shows gaps, concurrency and ordering, which makes them the right tool for every latency-bound problem. They cost more than sampling, particularly with shape recording and stack capture enabled, and a trace of a thousand iterations is unusable; trace ten.

Vendor tools see hardware the generic tools cannot. Arm Streamline correlates PMU counters across cores with the interconnect and memory controller counters and the GPU, which is how you get real bandwidth rather than an LLC-miss estimate. NVIDIA Nsight Systems is the system-level timeline showing CPU, CUDA launches, copies and kernels together, and it is where you find the missing overlap. Nsight Compute is the per-kernel microscope: full counter sets, occupancy, a built-in roofline. It replays kernels to collect everything, so it perturbs the timeline heavily. Use Systems to find the kernel and Compute to understand it, never the other way round.

Framework operator profilers are built into the runtime and are the fastest way to get a ranked operator list on a device you cannot attach anything else to. TFLite’s benchmark_model with --enable_op_profiling gives per-operator times plus, critically, the delegate partitioning summary.That summary is the line I read first. It reports how many nodes the delegate accepted and how many partitions the graph was cut into. One partition covering 95% of nodes is healthy. Seven partitions covering 95% of nodes is a graph being handed back and forth across a driver boundary six times per inference, and the per-operator times will not show you that cost because it does not belong to any operator. ONNX Runtime’s enable_profiling writes JSON with per-node times and execution providers, which is how you spot nodes silently assigned to the CPU provider. Their common weakness is attributing everything to an operator, so runtime overhead is smeared across operators rather than shown as itself.

Reading a Trace

Ask a trace four questions, in order.

Where are the gaps? A gap is a period where nothing is running. Gaps are the highest-value finding in any trace, because they are time you are paying for and getting nothing. Trace the gap back to what ended just before it and what was waiting.

What is serialised that could be concurrent? Preprocessing on the CPU while the accelerator is idle, then the accelerator running while the CPU is idle, is a classic: the frame takes the sum when it could take the maximum. Pipelining across frames turns a 14 plus 9 millisecond serial chain into a 14 millisecond one.

What is on the critical path? The critical path is the longest chain of dependent work from input to output. Speeding up anything off it changes nothing at all, which is why “I made that operator 3x faster and latency did not move” happens so often. Identify the chain before you optimise anything in it.

Is this timeline busy or fast? Those are different. A timeline with no gaps and 100% utilisation can be a system recomputing the same thing four times, spinning on a lock, or converting layouts. Utilisation is a measure of activity, not of progress. The only measure of progress is useful work per second, which is exactly what the achieved FLOPs/s and achieved bandwidth numbers give you.

Two Case Studies and a Checklist

Both of these are composites. They are patterns I have seen repeatedly rather than any single incident, and every number is illustrative and internally consistent with the device table above rather than a measurement of a real product.

Case One: Fusion and Layout

A segmentation network on the four-core device above. Budget 25 milliseconds per frame, measured 41.7. The model had already been quantisation-aware trained and the team’s assumption was that the convolutions needed a better kernel.

Level 3 said otherwise:

Frame breakdown before and after, in milliseconds.

The diagnosis came from two pieces of arithmetic. First, the elementwise chain. Each of the four residual blocks operated on a 1x64x128x128 fp32 tensor, exactly 4.194 MB. Batch normalisation reads and writes it (8.39 MB), ReLU reads and writes it (8.39 MB), and the add reads two tensors and writes one (12.58 MB), giving 29.4 MB per block and 117.4 MB per frame. Measured time for those layers was 9.3 milliseconds, so achieved bandwidth was \(117.4\ \text{MB} / 9.3\ \text{ms} = 12.6\) GB/s, which is 97% of the 13 GB/s the device can stream. Those kernels were already perfect. There was no tuning to do, only traffic to delete, and fusion deletes it.

Second, the layout conversions. Six conversions of 4.194 MB, read and written, is 50.3 MB, which at 13 GB/s should take 3.9 milliseconds. It took 5.2, an achieved 9.7 GB/s, and dTLB-load-misses in those regions was roughly 40 times the rate seen in the convolutions. That is the signature of a strided transpose walking pages.

The convolutions themselves were 1.9 GFLOP per frame, so 25.4 milliseconds is \(1.9 / 0.0254 = 74.8\) GFLOP/s, 58% of the 128 GFLOP/s peak. After the layout change, 20.1 milliseconds is 94.5 GFLOP/s, or 74% of peak. That is the compute-bound branch behaving exactly as advertised: a good but not great kernel got better with a layout that suits the SIMD unit, and the remaining 26% is not worth chasing. The whole fix was compiler configuration and fusion patterns, described in Part 4. Nobody wrote a new convolution kernel.

Case Two: The Model Was Fine

A quantised object detector on a mobile SoC with an NPU delegate. The model card said 8 milliseconds. Camera to bounding box measured 61 milliseconds, 16 frames per second against a 30 fps requirement. The instinct in the room was to shrink the model.

A timeline trace disagreed, comprehensively.

End to end breakdown, in milliseconds.

Two findings mattered. The first was that the model was never running in 8 milliseconds. The delegate supported most of the graph but not a five-way concatenation, a nearest-neighbour resize and an argmax, so the runtime split the graph into seven partitions with six handoffs, each involving a driver call and a buffer copy. The 8 millisecond figure was the sum of the NPU segments; the 21 milliseconds was reality. Replacing those three operators with supported equivalents, which did not change the network’s outputs, collapsed it to one partition.

The second was that 26.5 milliseconds, 43% of the frame, was preprocessing, and 14 of those were a colour conversion written in NumPy that the pipeline inherited from the prototype. It was doing per-pixel arithmetic in an interpreted loop over a 1920x1080 frame, which is 2.07 million pixels; at even 7 nanoseconds per pixel that is 14.5 milliseconds, and the arithmetic was never in doubt once someone looked. It went to the image signal processor, where colour conversion is free.

The model weights were never touched. This is the single most common shape of performance work I do on a deployed vision system, and it is why Part 7 spends as long on the pipeline as on the network.

The Checklist

1. Write down the budget and the current end to end number, with median and p99, before touching anything.
2. Fix what you can: governor to performance, threads pinned, other tenants stopped, device soaked to thermal steady state.
3. Warm up for at least 50 iterations and run at least 500, discarding the warmup.
4. Record the clean, uninstrumented wall clock. This is ground truth for every later measurement.
5. Put coarse timers around pipeline stages: capture, preprocess, inference, postprocess, output. Find out what fraction is actually the model.
6. Run the framework operator profile. Check the total against ground truth; if it is more than 10% off, trust ground truth.
7. Sum the operator times. Wall clock minus that sum is your overhead, and it belongs on the list of things to fix.
8. Take a timeline trace of ten iterations. Find the gaps, the serialisation and the critical path.
9. If there is an accelerator involved, count the partitions and list the operators that fell back.
10. For the top one or two operators, compute achieved FLOPs/s analytically and achieved bandwidth from counters.
11. Apply the decision tree: compute bound, bandwidth bound or latency bound, and take the named action for that branch.
12. Change exactly one thing. Re-measure with the same harness. Keep it only if the median moved by more than the noise, and check that the p99 did not get worse.
13. Record the before and after numbers, the device, the clock state and the commit. Six weeks later this note is the only reason anyone will believe you.

Takeaway: The fastest wins are almost never inside the model. Check the pipeline, the partitioning and the overheads before you touch a kernel, and change one thing at a time so you can attribute the result.

A Few Problems to Work

Use the illustrative device from the table above: 4 cores at 2.0 GHz, 32 GFLOP/s fp32 per core, 128 GFLOP/s aggregate, 17.1 GB/s theoretical DRAM and about 13 GB/s stream-achievable across all cores, with a single thread reaching roughly 6 GB/s.

1. A single-threaded 3x3 depthwise convolution over a 1x64x112x112 fp32 tensor runs 200 iterations in 0.98 seconds. perf stat reports 1.960e9 cycles, 2.352e9 instructions and 2.10e7 LLC load misses with 64 byte lines. Compute IPC, achieved FLOPs/s, achieved bandwidth and FLOPs per instruction. Diagnose it and give the fix.

2. The 4096 by 4096 fp16 matrix-vector product measured 4.6 ms. Compute arithmetic intensity, the roofline attainable rate, the ideal time and the roofline efficiency. Then compare two proposals: quantise the weights to int8, or fix the access pattern to reach 85% of achievable bandwidth. Which is worth more, and what does doing both give?

3. The detector from case study two runs at 61 ms per frame with 21 ms in the model. (a) If the model became infinitely fast, what frame rate results? (b) What end to end speedup does a 2x faster model give? (c) If the model stays at 21 ms, how much must everything else shrink to hit 30 fps?

4. A kernel runs for 180 microseconds per iteration at 500 iterations per second. You profile with perf record at 4 kHz. (a) How many samples per second land in that kernel? (b) How long must you record for a 5% relative error on its share, and for 1%? (c) If the sample handler costs 1.5 microseconds, what is the overhead at 4 kHz and at 100 kHz?

5. First inference takes 340 ms; steady state is 12 ms. (a) After how many inferences does the running average fall below 15 ms? (b) The product requirement is a p99 of 20 ms and the application does exactly one inference per user session. What do you do?

What’s Next

The habit this chapter is trying to build is small: before you change anything, know which of three things is limiting you, and know it from a number rather than from an impression. Everything else in the book, the quantisation, the fusion, the pruning, the layout choices, is a menu. Profiling is how you order from it. The next chapter takes the same discipline out of a single device and into a fleet, where latency is a distribution across thousands of requests, cost is measured in currency rather than milliseconds, and being wrong has a price you can put on an invoice.

That’s all for Part 10! For Part 11, on serving, MLOps and the cost of being wrong, click here.