Pruning, Sparsity and Distillation

Part 5 shrank every number. This chapter removes numbers entirely, then asks whether you needed them at all. Those are the other two levers: pruning, which deletes weights from a trained network, and distillation, which trains a smaller network to imitate a larger one. Behind both sits a third that people rarely file under “compression” but which dominates them: the choice of architecture.

The question here is narrower than “how do I make my model smaller”. It is: which of these actually produces fewer microseconds on the device in front of me, and in what order should I apply them? The honest answer contains one large disappointment, which I want out of the way first, because I spent an embarrassing amount of time at Dyson learning it the expensive way.

Unstructured Pruning and the Speedup That Never Comes

Unstructured pruning sets individual weights to zero. Nothing about the tensor’s shape changes; you have a mask \(m \in \{0,1\}^{n}\) and an effective weight \(\tilde{w} = m \odot w\). It is the easiest form of compression to implement and, on accuracy, it works: for most convolutional networks and most transformers you can zero 80 to 90 percent of the weights and, with enough fine tuning, land within a point of the dense baseline.

Magnitude Criteria and Iterative Schedules

The criterion that refuses to die is magnitude: rank weights by \(\lvert w_i \rvert\) and zero the smallest. It is only a good proxy for importance if the loss surface is isotropic in weight space, which it emphatically is not, but it is free, needs no data, and after fine tuning the gap to a better criterion is often smaller than the run-to-run variance.

You do not do it in one shot; one-shot pruning to 90 percent destroys a network. Iterative magnitude pruning alternates: train dense, prune a slice, fine tune, repeat. The cleanest way to run it is a gradual schedule that raises sparsity smoothly inside a single training run, usually the cubic ramp

where \(s_i\) is the initial sparsity (usually 0), \(s_f\) the target, \(t_0\) the step at which pruning begins and \(n \Delta t\) the ramp length.This schedule is from Zhu and Gupta's 2017 "To prune, or not to prune" study. The cubic shape removes most of the weights early, while the network still has many steps left to recover, and barely touches it near the end. The shape does the work: aggressive early, gentle late.

def cubic_sparsity(step, start, span, s_final, s_init=0.0):
    if step < start:
        return s_init
    if step >= start + span:
        return s_final
    frac = (step - start) / span
    return s_final + (s_init - s_final) * (1.0 - frac) ** 3

def apply_global_mask(model, sparsity):
    scores = torch.cat([p.detach().abs().flatten()
                        for p in prunable_params(model)])
    k = int(sparsity * scores.numel())
    if k == 0:
        return
    threshold = torch.kthvalue(scores, k).values
    for p in prunable_params(model):
        p.data.mul_(p.detach().abs() > threshold)

Two details there matter. The threshold is global, not per layer, which lets the algorithm discover that the first convolution needs its weights and the penultimate fully connected layer does not. And the mask must be reapplied after every optimiser step, because momentum and weight decay will happily resurrect a pruned weight from zero.

Why Ninety Percent Sparse Runs at Dense Speed

Now the disappointment. You have a 90 percent sparse network, you run it, and it takes exactly as long as before. That is not a bug in your framework. Two reasons compound.

The first is that a dense GEMM kernel is already close to optimal. As Part 3 worked through, a good matrix multiply keeps the vector units fed with contiguous loads, blocks for cache reuse and hits 50 to 80 percent of peak, all of which depends on knowing exactly where the next element is. A sparse kernel gives that up: every nonzero carries an index, and indirection defeats prefetching, breaks the contiguous vector load and wastes lanes.

The second is that the sparse format costs bytes. Compressed sparse row keeps a value plus a column index per nonzero: in fp16 with int32 indices, six bytes per nonzero against two per dense element, so you need density below one third just to break even on bytes moved.Bitmask formats do better: one bit per element of the dense shape plus the packed values. At fp16 and 90 percent sparsity that is 0.325 bytes per element against two dense, a genuine 6.2x. It is the right choice when you care about the checkpoint rather than the kernel.

Make it arithmetic. Let \(P\) be the dense kernel’s achieved throughput. A sparse kernel at density \(d\) performs \(d\) of the multiply-accumulates, at some fraction \(\eta\) of that throughput:

The whole question collapses to one comparison: is the density lower than the kernel efficiency? If \(d > \eta\) the sparse version is slower, and break even is exactly \(d = \eta\).

Where the Crossover Actually Sits

So what is \(\eta\)? With a purpose-built CPU kernel for one shape, 1x1 convolutions above all, 0.15 to 0.30 is achievable, which is why published demonstrations of useful unstructured sparsity on mobile CPUs report wins starting around 70 to 80 percent sparsity. On a GPU whose dense path runs on tensor cores the baseline is so fast that a general sparse kernel lands nearer 0.02 to 0.05, and you need 95 to 99 percent sparsity to be level.

Theoretical speedup \(S = \eta / d\) from unstructured sparsity, for three kernel efficiency regimes. Anything below 1.00x means the sparse kernel is slower than just doing the dense multiply.

Treat the bottom rows as optimistic: above roughly 95 percent sparsity the kernel is limited by index traffic and by reading activations, which it must do however many weights survive, so the real curve flattens where this model keeps climbing.

At Dyson my first serious pruning experiment was unstructured, global and magnitude based, taken to high sparsity on a cubic schedule. Accuracy held up beautifully and the checkpoint got several times smaller, which mattered for over-the-air updates to a fleet of robots. The frame time did not move at all, because the runtime was still executing a dense convolution over a tensor that happened to contain a lot of zeros, and a multiply by zero costs as much as any other multiply.

Takeaway: Unstructured sparsity speeds anything up only when density falls below the sparse kernel’s efficiency ratio \(\eta\), a threshold of 85 to 95 percent sparsity or higher on dense-optimal hardware. Below it you have compressed your checkpoint and changed nothing about your latency.

Structured Pruning: The Kind That Becomes Latency

The fix is to remove weights in shapes the hardware already likes. Delete an entire output channel and the tensor is not sparse, it is smaller: the loop bound shrinks, no special kernel or index or metadata is needed, and every existing optimisation still applies, to less work.

Channels, Filters and Attention Heads

For a convolution the natural granularity is the output filter. Removing filter \(j\) from layer \(\ell\) also removes input channel \(j\) from layer \(\ell+1\), so the saving along a chain is quadratic: keep a fraction \(f\) of channels and an interior layer costs \(f^2\) of its multiply-accumulates. Take a 3x3 convolution, 256 channels in and out, at 28x28:

Prune to 192 channels on both sides and it becomes \(192 \times 192 \times 9 \times 784 = 260\) M MACs, or \((0.75)^2 = 0.5625\) of the original. A 25 percent cut in channels bought a 44 percent cut in arithmetic, and the result is an ordinary dense network any runtime can execute.

In transformers the structured targets are attention heads and the feed-forward intermediate dimension. Head pruning works because heads are demonstrably redundant: a well-known 2019 analysis found many trained heads can be removed at test time with negligible loss, and some layers tolerate being reduced to one. The saving is linear in heads removed. The feed-forward block is usually the better target, because a standard layer holds \(4d^2\) parameters in attention and \(8d^2\) in the feed-forward block: two thirds of the weight sits where pruning along one axis is trivial.

Block Sparsity and 2:4

Between “any weight anywhere” and “whole channels” sits a useful middle: constrain the zeros to a pattern the hardware can exploit cheaply.

Block sparsity zeros aligned tiles, typically 4x4 up to 32x32. Each surviving block is a small dense matrix, so the inner kernel is a dense micro-kernel, \(\eta\) climbs to 0.4 to 0.7, and one index amortises over hundreds of values. The cost is accuracy: forcing a 32x32 tile to die together is far stronger than killing 1024 individually chosen weights, so you typically reach only 50 to 80 percent sparsity for the same budget.

N:M semi-structured sparsity is the version real silicon accelerates. The 2:4 pattern requires exactly two zeros in every group of four consecutive weights along the reduction axis. That fixes sparsity at 50 percent, and the regularity lets the hardware store a selector per group and skip the zeros inside the datapath, up to 2x the math throughput.The metadata is two bits per weight of the dense shape. The doubling applies to the multiply-accumulate units only: a layer that was memory bound before is still memory bound, moving the same activations, and sees close to nothing. End-to-end wins reported on GEMM bound transformer layers are usually 1.2 to 1.6x, not 2x. The accuracy recipe is mechanical: train dense to convergence, apply the 2:4 mask by magnitude within each group, then repeat the original training schedule. That typically recovers the baseline, since 50 percent is not a demanding sparsity level.

How each pruning granularity behaves on real hardware. The last column is the only one that matters when you are chasing a frame budget.

What Structure Costs You

Structure is not free: at a fixed accuracy target, unstructured pruning removes roughly twice as many parameters as channel pruning. Pay it anyway, because half as many parameters removed with a 1.6x latency win beats twice as many removed with a 1.0x win.

There is a sneakier cost. If your kernel processes output channels in tiles of 16, pruning 256 channels to 200 gives 12 full tiles and one half used: you pay for 208 channels of compute and get 200 of capacity. Always round the surviving count up to a multiple of the tile width. It is free accuracy, and Problem 2 works an example where rounding 160 up to 192 gives 20 percent more capacity for no extra time.

On the Arm cores I work with now the vector and matrix extensions are dense engines: there is no unstructured sparse datapath to target. Structured pruning is the kind that appears in a profile; everything else is checkpoint compression under another name.

Takeaway: Prune in whatever granularity your hardware can already execute. Channel and head pruning shrink the loop bounds and need no new kernel; 2:4 works where the silicon has the path; unstructured needs 90 percent plus before it is even level.

Choosing What to Cut, and Repairing the Damage

First Order Saliency

Magnitude ignores the loss. The correction is to ask what the loss would do if a weight went to zero, so expand \(\mathcal{L}\) around the current weights:

Zeroing weight \(i\) means \(\Delta w_i = -w_i\), so keeping the first term gives a saliency of \(\lvert g_i w_i \rvert\), accumulated over a calibration batch. One backward pass, and a real improvement over magnitude alone for deciding between layers, because gradients carry the scale information magnitudes lack.

Second Order: Optimal Brain Damage and Optimal Brain Surgeon

At a converged minimum \(g \approx 0\), so the linear term is near zero by construction and the quadratic dominates. Optimal Brain Damage takes that term and assumes a diagonal Hessian:

That is magnitude pruning weighted by curvature: a large weight in a flat direction is cheap to remove, a small weight in a sharply curved one is not.

Optimal Brain Surgeon drops the diagonal assumption and asks not only what removing weight \(q\) costs, but what the optimal compensating update to every surviving weight is:

The second equation is the interesting one: after deleting a weight you shift the rest to absorb its contribution, which is why OBS-style methods can prune a layer with no gradient steps at all.The catch is the inverse Hessian: for n weights, forming and inverting H is O(n^3) time and O(n^2) memory, hopeless for anything modern. The practical descendants work per layer on the much smaller input covariance of a few hundred calibration samples, only k by k for a layer with k inputs, and apply the OBS update column by column.

The Practical Middle Ground

In production I almost never compute a Hessian. The methods that earn their place are cheap and use a little data.

Saliency criteria ranked by what they cost you. The last two are where I would start on anything large.

Reach for the activation weighted criterion first on anything large. Scaling each weight by the norm of the input feature it consumes is almost free, needs a few hundred calibration samples, and competes with far more expensive machinery, because a weight’s contribution to the output is \(w_{ij} x_j\), not \(w_{ij}\), and where activation scales vary wildly the second factor carries most of the variance. Pushed to its conclusion this becomes layerwise reconstruction, structurally identical to the GPTQ-style procedure in Part 5: pruning and quantisation are one optimisation problem with different feasible sets, which is why it is natural to do them together.

The Fine Tuning Loop and Learning Rate Rewinding

Whatever criterion you use, the pruned network needs repair: prune to the next level on the schedule, fine tune with the mask fixed and reapplied after every step, evaluate, prune again. The question is what learning rate to fine tune at, and the common instinct, a small constant rate because the network is nearly right, is wrong often enough to be worth overturning.

The alternative is learning rate rewinding: keep the pruned weights, but restore the learning rate schedule to an earlier point in the original run and replay it, warm restart and decay and all. Careful comparisons have found this matches or beats low-rate fine tuning at equal training cost across a wide range of networks and sparsity levels, and it needs no new hyperparameters, since you already have a schedule that worked.This is distinct from weight rewinding in the lottery ticket literature, where surviving weights are reset to their values at an early step. That is a scientific instrument for asking whether a sparse subnetwork was trainable from near initialisation; learning rate rewinding keeps the trained weights, costs less and works at least as well at scale. My rule of thumb is to rewind to where the original schedule had decayed to a tenth of its peak.

Knowledge Distillation

Pruning takes a trained network apart. Distillation goes the other way: keep a large, accurate teacher you never intend to deploy, and train a small student to reproduce its behaviour. The student need not be a subnetwork of the teacher, which is the point.

The Objective

Let \(z_t\) and \(z_s\) be teacher and student logits, with the temperature-softened distribution

The standard loss mixes a hard label term with a soft matching term:

Typical settings are \(T\) from 2 to 8 and \(\alpha\) from 0.7 to 0.95. The \(T^2\) is not cosmetic. Differentiating the KL term by a student logit gives

and the soft distributions themselves flatten as \(T\) grows, so without correction the gradient shrinks like \(1/T^2\) and the soft term quietly vanishes at exactly the temperatures you wanted.Hinton, Vinyals and Dean's 2015 paper introduced both the temperature and the T-squared correction, which keeps the hard and soft terms in roughly constant proportion as you sweep T, so alpha means the same thing at T = 2 and at T = 8.

import torch
import torch.nn.functional as F

def distill_step(student, teacher, x, y, opt, T=4.0, alpha=0.9):
    with torch.no_grad():
        t_logits = teacher(x)
    s_logits = student(x)

    hard = F.cross_entropy(s_logits, y)
    soft = F.kl_div(
        F.log_softmax(s_logits / T, dim=-1),
        F.log_softmax(t_logits / T, dim=-1),
        reduction="batchmean",
        log_target=True,
    ) * (T * T)

    loss = (1.0 - alpha) * hard + alpha * soft
    opt.zero_grad(set_to_none=True)
    loss.backward()
    opt.step()
    return loss.detach(), hard.detach(), soft.detach()

One trap worth naming: reduction="batchmean" is correct, while "mean" divides by batch size times class count and silently scales the soft term down a thousandfold on ImageNet.

Why Soft Targets Carry More Than Hard Labels

The usual explanation is “dark knowledge”: the teacher’s probabilities for the wrong classes encode a similarity structure the one-hot label throws away. A dog breed gets 0.9 on the right breed, 0.08 on a visually similar one and \(10^{-6}\) on “aeroplane”, and that ratio is information no hand label contains.

The gradient view is sharper. Take the high temperature limit with logits centred to zero mean per example: expanding \(\exp(z/T) \approx 1 + z/T\) for \(N\) classes gives \(\sigma(z/T)_i \approx (1 + z_i/T)/N\), and the gradient of the scaled KL term becomes

At high temperature, distillation is logit regression. That is the real reason it carries more signal: a cross-entropy gradient against a one-hot target is \(p_i - y_i\), which goes to zero as soon as the student is confident and correct, so a confidently correct example contributes nothing. The distillation gradient is a logit difference, which stays informative long after the student has the argmax right. Every example keeps teaching.

Takeaway: Soft targets keep producing gradient after the student is already correct, because the signal is a logit difference rather than a probability error. That is why distillation improves a student long past the point where supervised training on the same data has plateaued.

Feature, Attention and Self Distillation

Logits are the last thing the teacher computes, so matching them constrains the student at one point only. You can constrain it inside as well.

Feature distillation matches intermediate activations through a small learned projection \(r_\ell\) that reconciles differing widths:

Attention transfer matches spatial attention maps instead: collapse the channel dimension by summing squared activations, normalise, match. It is cheaper and more robust than matching features directly, being invariant to channel count and per-channel scale. The transformer version matches the pre-softmax attention matrices layer by layer alongside hidden states, and it is the core of the recipes that compress large encoders several-fold, consistently the component that matters most.

Self distillation uses a teacher with the same architecture as the student. This sounds pointless and is not: train a network, then train an identical one with the first as teacher, and you reliably get a slightly better model. The mechanism is regularisation, the soft targets acting as a learned label smoothing that encodes real class structure.

Choosing a Student Instead of Shrinking the Teacher

Here is the mistake I see most often: take the teacher, halve every width, call that the student. It is easy and usually the wrong shape.

The right student is chosen for the device, not derived from the teacher. On an accelerator with a large systolic array a narrow layer leaves most of the array idle, so a shallow-and-wide student beats a deep-and-narrow one at equal parameter count; on a CPU with modest vector width, depth is cheap and width is what costs. Every layer boundary also carries a fixed overhead: a dispatch, a synchronisation, an activation round trip to memory where it does not fuse. A 40-layer student with tiny layers is easily slower than a 20-layer student of twice the width and identical FLOPs.

So pick the student by measuring candidate blocks on the target, then distil into it. The teacher supplies the training signal, not the blueprint.

A Distillation Recipe That Survives Contact With Data

Teachers, Ensembles and Augmentation

The recipe that has worked for me, ordered by how much each ingredient actually contributed:

1. Augmentation, applied consistently. The dominant term; everything else is a rounding error beside it. Distillation is function matching: you want agreement across the whole input distribution, not just the training set. So feed teacher and student the exact same augmented view of each image, same crop, same flip, same mixup coefficients, and augment aggressively.Giving the teacher a clean image and the student an augmented one is the natural-looking thing to do and it is wrong: the target becomes inconsistent with the input, so the student is asked to predict the teacher's answer to a question it was never shown. Consistent teaching plus long schedules was the central finding of Beyer and colleagues' 2022 study.

2. Length. Where a supervised run converges in 90 epochs, I expect a distilled student to keep improving for 300 to 1000. That is not wasted compute: the student is learning a function, and each epoch shows it new points on that function. Budget three to ten times the supervised schedule.

3. Unlabelled data. The teacher supplies the target, so any in-domain imagery lying around becomes training data. In robotics this is close to free, since a fleet generates far more frames than anyone will ever label, and it was the cheapest accuracy improvement available to me at Dyson.

4. Teacher ensembling. Averaging the logits of two to four teachers gives a better calibrated target. The gain is real but modest, a few tenths of a point of student top-1, and it costs a full inference pass per teacher. Cache teacher outputs if your augmentation policy allows it; if not, consistent teaching is worth more, so skip the ensemble.

5. Temperature and alpha. Sweep \(T \in \{2, 4, 8\}\), take \(\alpha\) around 0.9, and move on: it matters less than any of the four above.

The Failure Mode Nobody Warns You About

The student overfits the teacher’s mistakes, and the reason is structural. Where the teacher is wrong but confident it emits a clean, low-entropy, perfectly consistent target: from the student’s point of view the easiest example in the dataset, with no label noise and the same answer every epoch. So the student learns it early and holds it hard. I have watched a student inherit a teacher’s blind spot and end up more confident in the wrong answer than the teacher ever was.

Where the teacher is wrong in an unbiased, noisy way the student can genuinely exceed it, because the noise averages out. Where it is wrong systematically, on one lighting condition or surface type or camera, the student amplifies the bias rather than diluting it. Three cheap defences:

• Keep \(\alpha < 1\). The hard label term is your only anchor to ground truth; even \(\alpha = 0.9\) leaves a tenth of the gradient pointing at reality.
• Evaluate on a held-out slice where you know the teacher is wrong. Aggregate validation hides this, because the teacher is right on most of it by construction.
• Cap teacher confidence, by clipping target probabilities or raising \(T\) on confident examples. Blunt, and it works.

Architecture as Compression

Everything so far takes the architecture as given. But the largest factor in how fast a model runs is which operations it performs, decided before a single weight exists.

Efficient Blocks

Depthwise separable convolution splits a standard convolution into a per-channel spatial filter and a 1x1 mixing convolution, with cost ratio

For \(k = 3\) and \(C_{\text{out}} = 256\) that is \(0.0039 + 0.111 = 0.115\), an 8.7x reduction in multiply-accumulates, and it is where Part 3 becomes unavoidable, because 8.7x fewer MACs is not 8.7x less time. The depthwise part does nine MACs per weight with no reuse across channels, an arithmetic intensity around 4 FLOPs per byte, firmly on the memory-bound side of almost any roofline. You cut the FLOPs 8.7x and the time perhaps 2 to 4x.

Inverted residuals expand a narrow input to \(t\) times its channel count with a 1x1, filter it depthwise, then project back, with the residual connecting the narrow ends. Why that suits edge hardware has little to do with FLOPs: it keeps the tensors crossing block boundaries small, so what lives between blocks fits in on-chip SRAM while the expanded activations stay local.

Grouped convolutions divide channels into \(g\) groups and mix only within a group, dividing parameters and MACs by \(g\). They are also a scheduling knob: choose \(g\) so channels-per-group is a multiple of the vector width, or tail handling eats the saving.

Squeeze and excitation adds a global average pool, two small fully connected layers and a per-channel sigmoid gate. Its FLOP cost is nothing: for 384 channels at reduction 16, about 18 thousand MACs among tens of millions. Its latency cost is not nothing, because a global reduction is a barrier. Nothing downstream starts until every spatial position has been pooled, so the feature map is fully materialised and then read again to apply the scale.The compiler cannot fuse across a global reduction, so an SE block turns one fused producer-consumer pair into three passes over the activation tensor. On a bandwidth-limited part I have seen that cost a double-digit percentage of a block's time for approximately zero arithmetic.

One inverted residual block with squeeze and excitation, modelled on a hypothetical device with 256 GFLOP/s fp16 and 12.8 GB/s of bandwidth (ridge point 20 FLOPs/byte). 64 channels in and out, expansion factor 6, 56x56 spatial, fp16 activations. These are modelled figures from the stated parameters, not measurements.

Read the last two columns against each other. The depthwise convolution is 6.6 percent of the arithmetic and 21 percent of the time; squeeze and excitation is 0.01 percent of the arithmetic and 10.6 percent of the time, a ratio of roughly a thousand to one. Choose blocks on a FLOP count and you are choosing almost at random.

Compound Scaling

Given a good block, how do you scale it? The axes are depth \(d\), width \(w\) and input resolution \(r\), and each saturates alone: a deep thin network is hard to train, a wide shallow one stops learning hierarchy, and resolution without capacity to use it is wasted. Compound scaling ties them to one coefficient \(\phi\):

The constraint exists because convolution FLOPs scale linearly with depth and quadratically with width and resolution, so each unit of \(\phi\) costs about twice the FLOPs. Find \(\alpha, \beta, \gamma\) with a small grid search at \(\phi = 1\), then scale. Same caveat as ever: that is a FLOP budget, so for a latency budget re-derive the constraint from measured times. On many devices resolution is cheaper than the quadratic suggests, because activations stream, while width is dearer, because weights must be resident.

Hardware Aware NAS and the Latency Lookup Table

Neural architecture search automates the block choice. The classical formulation optimises accuracy under a FLOP constraint, and produces architectures that are worse on real hardware than hand-designed ones, for exactly the reason the table above shows. Hardware aware NAS puts measured latency in the objective instead:

with \(\omega < 0\) so exceeding the target is punished. This is the MnasNet formulation; around \(\omega = -0.07\) it trades accuracy for latency smoothly, while a large negative value gives a hard cutoff, which behaves worse during search because most of the space then shares an identical zero reward and the controller learns nothing about how badly it missed.

What makes this practical is the latency lookup table. A search evaluates tens of thousands of candidates, so instead of benchmarking networks you benchmark every operator configuration once, offline, on the real hardware, and sum.

# Built once, offline, on the target device.
# Key: (op_type, in_ch, out_ch, kernel, stride, resolution)
LUT = {
    ("conv", 64, 384, 1, 1, 56): 0.61,   # milliseconds
    ("dwconv", 384, 384, 3, 1, 56): 0.38,
    ("conv", 384, 64, 1, 1, 56): 0.60,
    ("se", 384, 384, 1, 1, 56): 0.19,
}

def predict_latency(arch, lut):
    total = 0.0
    for layer in arch.layers:
        key = layer.signature()
        if key not in lut:
            raise KeyError(f"unbenchmarked op in search space: {key}")
        total += lut[key]
    return total

An additive table is a model, wrong in two known ways: it misses operator fusion, so it overestimates any network the compiler would have fused, and it misses cache state, so it underestimates networks whose working set stops fitting. For a sequential network on a simple memory hierarchy it typically lands within a few percent of measured end-to-end time, which is more than good enough to rank candidates. Validate it on a handful of complete networks before trusting it for ten thousand.

Takeaway: FLOPs are not latency, and every technique that optimises FLOPs inherits that error. Build a latency table on the real device once, then rank architectures, pruning targets and scaling decisions against measured time.

Stacking the Levers in the Right Order

Order matters, because each step changes the assumptions the next one depends on.

1. Architecture. First, because it sets the ceiling. Every later step is a percentage off whatever the architecture gives you, and none of them turns a bad block into a good one. It is also the most disruptive to change late: a new architecture invalidates your pruning masks, your quantisation calibration and your kernel tuning at once.

2. Distillation. Second, because it needs a full-precision student and a training run anyway, and because it buys accuracy headroom the later, lossy steps will spend. Entering pruning and quantisation two points above target is the difference between a comfortable project and a miserable one.

3. Structured pruning. Third, because it changes tensor shapes and both remaining steps depend on shapes: quantisation calibration measures activation distributions, and kernel tuning picks tile sizes for a shape.

4. Quantisation. Fourth, because it is the last step that changes the numerics, and doing it earlier means recalibrating after every pruning iteration. Folding it into step 3 as a quantisation-aware fine-tune usually works, but sequential is far easier to debug when accuracy falls off a cliff and you need to know which step did it.

5. Kernels, layouts and compiler work. Last, because it is shape-specific and any earlier step would throw it away, and because it is the only lossless lever, so it is the one thing you can always do more of at the end.

The exception: with a hard deadline next week, run int8 post-training quantisation first. It is a day of work for 2 to 4x and needs no training. Then come back and do it properly.

Typical compound savings in the order above, from a ResNet-50 class fp32 baseline. Ranges are what I would defend in a design review, not promises. Each row multiplies against the rows above it.

Be suspicious of that last row. Multiplying the optimistic end of every latency column gives 60x, and I have written 30x, because the factors do not compose. Each step that removes arithmetic pushes you closer to being memory bound, and once you are memory bound the next arithmetic reduction buys nothing. That is the most common way a compression plan misses: three levers worked exactly as predicted and the fourth hit a wall it was never going to cross, because the bottleneck had moved. Re-profile after every step.

Takeaway: Architecture, then distillation, then structured pruning, then quantisation, then kernels. Each step changes the shapes or numerics the next calibrates against, and compound savings are always less than the product because the bottleneck moves as you go.

A Few Problems to Work

Problem 1. A CPU core peaks at 64 GFLOP/s in fp32 and your dense GEMM kernel achieves 60 percent of that. A CSR sparse kernel on the same core sustains 6.0 GFLOP/s of useful arithmetic regardless of density. (a) At what sparsity does the sparse kernel break even? (b) What speedup do you get at 90 and at 95 percent sparsity? (c) In CSR with fp32 values and int32 indices, at what sparsity does the format break even on bytes, and how much smaller is the 90 percent sparse model?

Problem 2. A residual block has two 3x3 convolutions at 28x28: the first maps 256 channels to \(C_{\text{mid}}\), the second maps \(C_{\text{mid}}\) back to 256. Your kernel processes output channels in tiles of 64 and achieves 45 percent of peak when the channel count is a multiple of 64, dropping to 38 percent when it is not. Compare pruning to \(C_{\text{mid}} = 160\) against \(C_{\text{mid}} = 192\).

Problem 3. Teacher logits are \(z_t = [4, 2, 0]\) and student logits \(z_s = [3, 1, 1]\) for a three-class problem. Compute the gradient of the \(T^2\)-scaled KL term with respect to the student logits at \(T = 1\) and \(T = 4\), and compare with the high temperature limit.

Problem 4. A model has 25 M parameters in fp32 and 4.2 GMAC, and measures 120 ms on the target. You structurally prune to 42 percent of the MACs and 45 percent of the parameters, then quantise to int8. Predict the naive latency, then the realistic one given that the pruned shapes run at 80 percent of the original utilisation and int8 delivers 2.2x rather than 4x on this device. What is the final size?

What’s Next

Between Part 5 and this chapter you have the full compression toolkit: fewer bits, fewer weights, better blocks, and a small network trained to imitate a big one. What you should also have is a sharper instinct for which of those touches the clock and which only touches the disk. The next chapters stop being general, starting with what all of this looks like when the model is a real vision system with a camera in front of it and a frame budget that includes far more than the network.

That’s all for Part 6! For Part 7, on vision models in the real world, click here.