Quantisation

Every chapter so far has been about moving work around: onto better kernels, into fused graphs, across a memory hierarchy. This chapter is about doing less work in the first place, by representing numbers with fewer bits. It is the highest-return optimisation available to most practitioners, and the one people get wrong most often, because it looks like a compiler flag and behaves like a numerical method.

The question here is: given a trained network in fp32, how do I run it in 8-bit integer arithmetic, how much accuracy does that cost, and which parts of that cost am I allowed to refuse to pay? I spent two years at Dyson putting segmentation and detection CNNs onto robot hardware, and quantisation was never the last step; it was the constraint that shaped the architecture from the beginning. At Arm I now build the compilers and libraries that turn a quantised graph into instructions, which is a useful vantage point for seeing where the accuracy goes.

The Arithmetic of a Quantised Tensor

The affine map, the scale and the zero point

Quantisation replaces a real number \(x\) with an integer \(q\) from a small set, plus enough metadata to get approximately back. The affine (asymmetric uniform) quantiser is

and its inverse, the dequantiser, is

Two numbers define the map. The scale \(s\) is a positive real, the size of one integer step in the units of \(x\). The zero point \(z\) is the integer representing exactly zero. That second property is not cosmetic: padding, ReLU and masking all produce exact zeros, and if zero is not exactly representable then a zero-padded convolution leaks a small bias into every border pixel.This is why the affine formulation puts the zero point inside the integer domain rather than using a general $$\hat{x} = ax + b$$. With $$z$$ an integer, $$x = 0$$ maps to $$q = z$$ and dequantises to exactly $$s(z - z) = 0$$, with no residual.

Given a clipping range \([\alpha, \beta]\) and the integer range \([q_{\min}, q_{\max}]\) the format provides,

Check it: \(x = \alpha\) gives \(\text{round}(\alpha/s) + q_{\min} - \alpha/s = q_{\min}\), as it should.

Everything else in this chapter is about choosing \(\alpha\) and \(\beta\) well, choosing how many \((s, z)\) pairs a tensor gets, and keeping the integer arithmetic between them exact. Note the asymmetry in the error: inside the range it is uniform on \([-s/2, s/2]\) with variance \(s^2/12\), while outside it is the full clipping excess, which is unbounded. That asymmetry is the whole game.

Symmetric and asymmetric, signed and unsigned

Setting \(z = 0\) gives the symmetric quantiser, \(\hat{x} = s q\), with

For signed int8 the natural integer range is \([-128, 127]\), but symmetric schemes normally restrict to \([-127, 127]\) so that the representable set is genuinely symmetric and negation is exact. Giving up one code out of 256 costs 0.03 dB of signal to quantisation noise ratio and removes a whole class of edge cases.It also avoids the $$(-128) \times (-128) = 16384$$ product, which does not fit in int16 and forces saturating behaviour in kernels that accumulate in 16 bits. On Arm the accumulation is in int32 so it would be harmless, but portability across DSPs is worth one code.

Which quantiser to use for which tensor. Signedness follows the sign of the data; symmetry follows whether the runtime cost of a zero point is worth paying.

Asymmetric quantisation of a one-sided tensor is worth almost exactly one bit: a post-ReLU tensor in symmetric int8 uses codes \(0\) to \(127\) and wastes the negative half, where uint8 uses all 256. That is why the classic TensorFlow Lite scheme pairs uint8 asymmetric activations with int8 symmetric weights.

Dequantising an int8 matrix multiply

This explains why weights are symmetric and where the bias goes. Take \(X \in \mathbb{R}^{M \times K}\) (activations) and \(W \in \mathbb{R}^{K \times N}\) (weights), with per-tensor quantisers \(x_{ik} = s_x(q^x_{ik} - z_x)\) and \(w_{kj} = s_w(q^w_{kj} - z_w)\). The real output is

Expanding the product gives four terms:

Term (1) is the int8 dot product, accumulated in int32. Term (3) depends only on weights, so it is a compile-time constant per output channel \(j\), and term (4) is a scalar. Term (2) is the problem: it depends on activations, so it must be computed at run time, once per output row, and it is a whole extra reduction over \(K\).

Set \(z_w = 0\) and terms (2) and (4) vanish:

That is the entire argument for symmetric weights. It is not about accuracy, it is about deleting a reduction from the inner loop.

The bias slots in naturally: quantised with scale \(s_x s_w\) and zero point 0 into int32, \(q^b_j = \text{round}(b_j / (s_x s_w))\), so the full accumulator is

where the last two terms fold into a single precomputed int32 constant per output channel. The kernel initialises its accumulator with that constant and then does nothing but int8 dot products.

Overflow is a non-issue at int32. The worst case magnitude of term (1) is \(K \cdot 127 \cdot 127 = 16129K\), and \(2^{31} - 1 = 2147483647\), so \(K\) would have to exceed 133,000 for overflow to be even theoretically possible. The same arithmetic in an int16 accumulator overflows at \(K = 2\), which is why DSPs offering 16-bit accumulation also offer saturation and forced downshifts.

Requantisation as a fixed point multiply and shift

The accumulator \(a_{ij}\) is int32 in units of \(s_x s_w\). The next layer wants int8 in units of \(s_y\) with zero point \(z_y\):

\(M\) is real and almost always in \((0, 1)\). A float multiply here would force a round trip through the floating-point unit and make the result depend on the device’s float rounding. Instead, normalise \(M\) into a mantissa and a shift:

and evaluate

where \(\text{SQRDMULH}(a, b) = \text{round}(ab / 2^{31})\) saturated to int32. On Arm that is one instruction, SQRDMULH (“signed saturating rounding doubling multiply returning high half”), and the rounding shift is SRSHL with a negative shift. Two integer instructions per output element, fully vectorised, bit-exact across every device implementing the architecture.Bit-exactness is the underappreciated benefit. A float requantisation makes your inference results depend on whether the target rounds ties to even, flushes denormals, or contracts a multiply-add. A fixed-point requantisation gives the same integers on a phone, a microcontroller and your laptop, which means a regression test can assert on exact tensor equality rather than a tolerance.

Worked example. Suppose \(s_x = 0.047\), \(s_w = 0.0012\), \(s_y = 0.0625\), \(z_y = -14\).

Normalising: \(M \cdot 2^{10} = 0.9240576\), which lies in \([0.5, 1)\), so \(n = 10\), \(M_0 = 0.9240576\) and \(M_{\text{int}} = \text{round}(M_0 \times 2^{31}) = 1{,}984{,}398{,}586\).

Take an accumulator \(a = 100{,}000\). Then \(\text{SQRDMULH}(a, M_{\text{int}}) = \text{round}(100000 \times 1984398586 / 2^{31}) = 92{,}406\), and a rounding right shift by 10 gives \(\text{round}(92406/1024) = 90\). Adding \(z_y\) gives \(q^y = 76\). The exact answer is \(M a = 90.24\), so the fixed-point path lands on the correctly rounded integer, as it does for essentially every accumulator value: the 31-bit mantissa leaves about nine decimal digits of headroom above the output’s one-integer granularity.

Per-output-channel weight scales change nothing structurally. \(M\) becomes \(M_j\) and you load a vector of \((M_{\text{int},j}, n_j)\) pairs instead of a scalar, in a loop that already iterates over \(j\).

Takeaway: the int8 matmul is an int32 accumulation of int8 products, corrected by precomputed per-channel constants, then rescaled by a single fixed-point multiply and shift. Symmetric weights exist to delete a runtime reduction from that expression, not to improve accuracy.

Number Formats and What They Are For

The formats you will actually meet. “Dynamic range character” is the qualitative behaviour that matters in practice, not just the endpoints.

The important distinction is not bits, it is where the dynamic range lives. Floating point carries an exponent per element, so one fp16 tensor can span five orders of magnitude. Integer formats carry one exponent (the scale) per tensor, channel or block, so range is a property of the group, and any element far from its group’s typical magnitude is either clipped or drowns everything else in rounding error. Nearly every quantisation failure in this chapter restates that sentence.

The reason to endure it is throughput:

Integer multiply-accumulate throughput on Armv8-A NEON, per 128-bit instruction. Counts are architectural work per instruction, not measured throughput on any particular core.

Add to that 4x less weight memory than fp32 and 2x less than fp16, which on a part with a 1 to 4 MB last-level cache often decides whether a layer’s weights are resident at all. Those two effects compound: int8 helps when you are compute bound and helps again when you are bandwidth bound, which is unusual among optimisations.

A short honest note on fp8 and microscaling

fp8 and the OCP microscaling (MX) formats are genuinely interesting and genuinely not yet universal. fp8, in its two OCP variants E4M3 and E5M2, has hardware support in recent datacentre GPUs and is appearing in newer accelerators. MXFP4 packs an E2M1 four-bit float with an eight-bit power-of-two scale shared across a block of 32 elements, giving about 4.25 bits per value with block-local dynamic range, which is exactly the property plain int4 lacks.

My honest read: fp8 is production-real for large-model training and serving on the hardware that has it, and MX formats are moving from specification into silicon, but toolchain support lags the instruction set by a year or more and quality tooling (calibration, sensitivity analysis, debuggers) lags further still. For edge Arm hardware today, int8 has the broadest, most predictable acceleration and the most mature tooling. Plan for fp8 and MX; ship int8.

Granularity: Per Tensor, Per Channel, Per Block

Why the algebra decides the granularity

Pulling \(s_x s_w\) outside the sum in \(y_{ij} = s_x s_w \sum_k (\cdot)(\cdot)\) requires that neither scale varies with \(k\). That single fact determines legal granularity:

• Scales may vary freely along output axes. \(s_w\) may depend on \(j\) (per output channel) and \(s_x\) on \(i\) (per token). Both sit outside the reduction.
• Scales may not vary along the reduction axis \(k\). A per-input-channel activation scale would need a rescale inside the inner loop, destroying the point of integer accumulation.
• Unless you break the reduction into blocks. Split \(k\) into blocks of size \(G\), accumulate each in int32, rescale, and sum in a wider accumulator. That is group quantisation: one rescale every \(G\) multiply-accumulates.

Granularity options and their real costs.

The common sweet spot for CNNs is per output channel weights plus per tensor activations. Weights are static, so per-channel scales are computed offline and cost nothing beyond a vector load the kernel was going to do anyway. Activations are dynamic, and a per-tensor activation scale keeps requantisation to a single constant per layer, which is what lets the compiler fold the scale into the epilogue.

For transformer inference the equivalent is per output channel weights plus per token activations, since the token axis is the non-reduction axis \(i\) and a per-row scale is therefore algebraically free. It costs one max-reduction over the hidden dimension per token, cheap next to the matmul, and it gives each token its own range.

Why depthwise layers force your hand

Fold batch normalisation into a convolution and the effective weight for output channel \(c\) is scaled by \(\gamma_c / \sqrt{\sigma_c^2 + \epsilon}\). Those factors vary enormously across channels, and in depthwise separable layers, where each channel is an independent filter with no mixing, nothing averages them out. Ranges spanning two or three orders of magnitude within one weight tensor are normal.

Suppose three channels of a folded depthwise kernel have maximum absolute weights 0.031, 0.42 and 3.8. Per-tensor symmetric int8 gives \(s = 3.8/127 = 0.029921\) for all of them.

Effective resolution per channel under a shared per-tensor scale. “Levels” counts the distinct integer codes that channel can reach.

The first channel has been quantised to ternary. Per-channel scales give all three the full 8 bits. This is not a subtle effect you argue about in the fourth decimal place of an accuracy table; it is the difference between a MobileNet-style backbone that works and one that does not. Putting segmentation models onto robot hardware at Dyson, every depthwise-separable backbone we touched needed per-channel weights, and the ones that still failed after that needed cross-layer equalisation on top.

Takeaway: quantisation scales can vary along any axis except the reduction axis, which is why per-output-channel weights are free and per-input-channel activations are not. Always use per-channel weights. Depthwise layers make it mandatory.

Calibration: Choosing Where to Clip

Four estimators

Calibration picks the clipping range \([\alpha, \beta]\) for each activation tensor by running data through the network and observing. The trade-off is one line of algebra: the expected squared error is

with \(s = (\beta - \alpha)/(q_{\max} - q_{\min})\). The rounding term grows as the square of the range; the clipping term shrinks, usually fast, as the range grows. Halving the range divides rounding error by four and multiplies clipping error by whatever the tail says.

Calibration estimators, ranked by how often I reach for them. All are proxies for the metric you care about.

The KL method builds a fine histogram (2048 bins is conventional), then for each candidate threshold merges bins down to the number of quantisation levels, dumps the clipped mass into the edge bin, and picks the threshold minimising the KL divergence between reference and quantised distributions. It treats the quantiser as a communication channel and asks which threshold preserves the most information.

MSE grid search skips that framing and minimises tensor error directly. It is usually at least as good, and it has a justification beyond convenience: for a small output perturbation \(\delta\) the change in task loss is approximately \(\tfrac{1}{2}\delta^\top H \delta\), so minimising \(\|\delta\|^2\) minimises the loss change under an isotropic approximation to the Hessian. GPTQ improves on this by using a real estimate of \(H\).All of these are proxies; none minimises your actual metric. A method that minimises tensor MSE perfectly can still cost accuracy if the residual error sits in a direction the classifier is sensitive to. That is the argument for the sensitivity analysis later in the chapter, which measures the metric rather than a proxy.

A worked percentile clip

Take a synthetic activation tensor from a mixture: 99% of values from \(\mathcal{N}(0, 1^2)\) and 1% from \(\mathcal{N}(0, 6^2)\). This is a crude but realistic outlier model, and stating it explicitly makes the arithmetic reproducible. Its standard deviation is \(\sqrt{0.99 \cdot 1 + 0.01 \cdot 36} = 1.162\). Quantise symmetrically into int8 with \(q \in [-127, 127]\), so a clip threshold \(t\) gives \(s = t/127\). For a Gaussian component of standard deviation \(\sigma\), the two-sided clipping error has a closed form,

Clip threshold versus error for the stated mixture, int8 symmetric. SQNR is \(10\log_{10}(\text{signal variance} / \text{total error})\).

Read the whole curve, not just the winner. Going from min/max to the 99.99th percentile buys 3.45 dB, a factor of 2.2 reduction in error, by clipping one value in ten thousand. Going one step further to the 99.9th percentile loses 7.9 dB, because this tail is heavy enough that the clipped mass costs more than the extra resolution earns. And the 99th percentile, which sounds cautious, is a disaster at 9.36 dB.

Two more observations. The 99.99th percentile lands within 0.2 dB of the true MSE optimum, which is why a fixed percentile is such a good default: almost all the benefit of a search, for the cost of a histogram. And the ceiling for a signal that fills an 8-bit quantiser uniformly is \(6.02b = 48.2\) dB, against the 28.7 dB achieved here. That 19.5 dB gap is the price of the 1% outlier component, and it is the quantitative statement of why the outlier mitigations later in this chapter matter.The $$6.02b$$ figure is for a signal uniformly distributed over the full quantiser range; the more familiar $$6.02b + 1.76$$ dB applies to a full-scale sinusoid. Neither is achievable for a real activation tensor, whose distribution is peaked near zero and therefore uses only a fraction of the available codes.

How many samples you actually need

Hundreds, not thousands. For vision I use 100 to 500 images; for language models 128 to 512 sequences is standard and 1024 is generous.

You are estimating a handful of scalars per tensor. A feature map of 100,000 elements, over 512 calibration images, contributes \(5.12 \times 10^7\) samples, of which about 5,120 lie above the 99.99th percentile. A quantile estimated from 5,120 exceedances has a standard error far smaller than one quantisation step, so more data cannot move the scale. Weights need no data at all, since the tensor is the population.

What matters is which samples. Calibration estimates activation ranges under the deployment distribution, and if that distribution shifts, the ranges shift with it. Calibrate a language model on encyclopaedia prose and serve it code, and the statistics of the tokeniser-adjacent layers will not match. Calibrate a robot’s perception stack on well-lit captures and deploy it under furniture at dusk, and the early layers see a range they were never shown.

That second example is not hypothetical for me. Getting the calibration set right at Dyson meant deliberately over-sampling the hard frames: low light, motion blur, the near-black views the robot gets when it is somewhere it should not be. A set drawn only from clean captures produced a model that benchmarked beautifully in the lab. I now treat the calibration set as a deployment artefact deserving the same review as the test set, not as a convenience sample of whatever was on disk.

Takeaway: calibration is a bias-variance trade between clipping and rounding, and a percentile clip near 99.99 typically lands within a fraction of a decibel of the MSE optimum. A few hundred samples suffice, but they must be drawn from the deployment distribution, hard cases included.

Post Training Quantisation and Quantisation Aware Training

Post training quantisation (PTQ) takes a trained fp32 network, calibrates, and emits an integer graph. It needs no labels, no gradients and no training loop, just a few hundred unlabelled samples. For most CNNs with per-channel weights and a sane calibration, PTQ lands within a few tenths of a point of the float model, and you should always try it first.

Quantisation aware training (QAT) inserts simulated quantisation into the forward pass and fine-tunes, letting weights move to positions that survive rounding. It costs a training loop, labels and time. You reach for it when PTQ leaves more on the table than you can accept, which in my experience means aggressive bit widths (int4 weights), heavily depthwise architectures, or tasks sensitive to small logit perturbations.

The straight through estimator

Training through a quantiser fails because \(\text{round}(\cdot)\) has zero derivative almost everywhere and undefined derivative at the half-integers, so backpropagation produces exactly zero gradient and nothing learns. The straight through estimator (STE) replaces the derivative of rounding with 1. Define the fake quantisation operator

and, in the backward pass, use

Two things are going on. Inside the clipping range the gradient passes through unchanged: that is the “straight through” part, and it is the deliberate lie. Outside the range the gradient is genuinely zero, because the clamp genuinely is flat there, and keeping that zero matters. If you passed gradient through the clamp too, weights would feel no pressure to stay inside the representable range and would drift out of it.This variant is sometimes called the clipped STE, as distinct from the original formulation which passes gradient everywhere. The clipped version is what essentially every production QAT implementation uses, and the difference matters most for activations, where a runaway range directly costs resolution for every other value in the tensor.

The STE is not an approximation to the true gradient, because the true gradient is zero. It is a different and useful search direction: the gradient of the loss with respect to the unquantised latent weight, evaluated at the quantised point. The forward pass sees quantisation; the backward pass pretends it did not. That this works is empirical, and it works well enough to have been the foundation of QAT for a decade.

You can also learn the scale. Differentiating the symmetric case \(\hat{x} = s\,\text{round}(x/s)\) with the STE applied to the inner rounding gives

which is the learned step size (LSQ) formulation. Making \(s\) trainable alongside the weights usually recovers a few tenths of a point over a fixed calibrated scale at no real extra cost, and removes a hyperparameter.

Fake quantisation nodes

QAT inserts fake quantisation (“quantise then immediately dequantise”) nodes into the float graph at every tensor that will be an integer tensor at deployment: weights before each matmul or convolution, activations after each op that produces one. A minimal implementation:

import torch

class FakeQuant(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, scale, zero_point, qmin, qmax):
        q = torch.round(x / scale) + zero_point
        q_clamped = torch.clamp(q, qmin, qmax)
        # Save the clipping mask for the clipped STE.
        ctx.save_for_backward((q >= qmin) & (q <= qmax))
        return scale * (q_clamped - zero_point)

    @staticmethod
    def backward(ctx, grad_out):
        (in_range,) = ctx.saved_tensors
        # Straight through inside the range, zero outside it.
        return grad_out * in_range, None, None, None, None


def fake_quant(x, scale, zero_point, num_bits=8, signed=True):
    qmin = -(2 ** (num_bits - 1)) + 1 if signed else 0
    qmax = 2 ** (num_bits - 1) - 1 if signed else 2**num_bits - 1
    return FakeQuant.apply(x, scale, zero_point, qmin, qmax)

Three practical points. All arithmetic here is float: QAT simulates integer behaviour rather than using integer kernels, and a step typically runs 1.3 to 2x slower than the float equivalent because of the extra elementwise work. Batch normalisation must be folded into the preceding convolution before or during QAT, otherwise you train a graph you cannot deploy. And fake quant placement must match the deployment graph exactly, including which ops the backend fuses: if your runtime fuses convolution with ReLU and requantises once at the end but your QAT graph quantises between them, you have trained the wrong model.

How much fine tuning QAT actually needs

Far less than training: the weights are already in a good basin, and you are nudging them to positions that survive rounding.

Typical QAT budgets relative to the original training run. These are conventional practice, not measurements from any particular project.

My rule: budget one epoch, run it, and if it does not close most of the gap, the problem is almost never that you needed ten. It is that something structural is wrong, usually a layer that should not be quantised, a missing per-channel scale, or a fake quant placement that does not match the backend. Longer training will not fix a graph quantised in the wrong places, and spending three days discovering that is a bad trade.

Takeaway: try PTQ first; it is free and usually close. QAT works because the straight through estimator supplies a usable gradient through a function whose real gradient is zero, and it needs a few per cent of the original training budget. If one epoch does not help, debug the graph rather than extending the run.

What Breaks, and How to Fix It

Activation outliers in transformers

Transformers develop persistent, structured activation outliers: specific hidden dimensions carry values one to two orders of magnitude larger than the rest, consistently across tokens and inputs, typically emerging beyond a few billion parameters. Because they occupy fixed channels, a per-tensor activation scale is set by those channels and every other dimension is quantised into a handful of codes. This is why naive int8 activation quantisation can destroy a large language model while the same recipe on a ResNet costs 0.2 points.

Three responses, in increasing effort:

1. Per-token activation scales. Algebraically free, as established above. Helps when outliers vary by token, and does not help when they sit in fixed channels shared by all tokens, which is the common case.

2. SmoothQuant-style migration. Move the difficulty into the weights, where per-channel scales can absorb it. For \(Y = XW\) and any diagonal \(S = \text{diag}(s_1, \ldots, s_K)\), \(XW = (XS^{-1})(SW)\) exactly. Choose $$s_j = \max_i

   X_{ij}

   ^{\alpha} \big/ \max_i

   W_{ji}

   ^{1-\alpha}\(with\)\alpha \approx 0.5\(, splitting the dynamic range between the two. The\)S^{-1}$$ folds into the preceding layer normalisation’s scale parameter, so the transformation is exact and costs nothing at run time.The exactness is the appealing part: unlike QAT or bias correction, this is an algebraic identity applied to the float model before quantisation, so there is no training and no risk of changing what the model computes. The only approximation is the subsequent quantisation of the reparameterised weights, which is now easier.

3. Keep the outlier channels in higher precision. Split the matmul into an int8 part for the well-behaved channels and an fp16 part for the outliers. Correct, but it produces a mixed-precision kernel with an awkward memory layout, and the fp16 remainder often costs more than it saves.

Normalisation, softmax and residual accumulation

Three ops deserve to stay in floating point unless you have a specific reason otherwise.

Layer normalisation computes a mean and a variance. For a near-zero-mean int8 tensor the variance is a difference of similar quantities and catastrophic cancellation eats the precision. Dequantise, normalise in float, requantise; the cost is negligible because layer normalisation is memory bound anyway.

Softmax exponentiates. If logits are quantised at scale \(s\), an error of one integer step scales the resulting probability by \(e^{s}\). At \(s = 0.1\) that is a 10.5% relative probability error from a single rounding step: fine for an argmax, not fine for a calibrated confidence or for attention weights that get multiplied by values and summed. Either compute softmax in float, or use an integer approximation such as a second-order polynomial fit to \(e^x\) on a reduced range, which is what integer-only transformer implementations do.

Residual accumulation is the quiet one. A residual add takes two int8 tensors with different scales, so both must be requantised to a common scale first, and each requantisation rounds. The residual stream’s magnitude grows with depth, so a per-tensor int8 residual loses relative precision monotonically as you go deeper. Across 50 layers that compounds into a drop no single-layer sensitivity analysis will reveal, because it is distributed. Fix it by carrying the residual path in int16, or by calibrating each add’s output scale per block rather than reusing one range for the whole trunk.

First and last layers

The convention of keeping the first and last layers in higher precision is a cost-benefit calculation rather than a principle.

The first convolution’s weights are few, its compute is a small fraction of the network, and its output feeds everything downstream, so its error propagates through the entire graph. The last layer produces logits, where what matters is the ordering of values whose differences may be small relative to their magnitudes, and it too is a small fraction of total compute. Both are cheap to exempt and expensive to get wrong: in a typical vision backbone they are together 3 to 8% of the FLOPs, so keeping them in fp16 costs a few per cent of latency and often recovers several tenths of a point.

One exception: for image input the first layer’s activations are already uint8 with \(s = 1/255\) and \(z = 0\), so quantising the input is free. It is the first layer’s weights and output that want higher precision.

The mitigation toolbox

Techniques for recovering accuracy, in the order I try them.

Three of these deserve their equations.

Cross-layer equalisation exploits the positive homogeneity of ReLU: \(f(cx) = c f(x)\) for \(c > 0\). For two consecutive layers, \(W^{(2)}W^{(1)} = (W^{(2)}S)(S^{-1}W^{(1)})\) for any positive diagonal \(S\), and the ReLU between them passes the scaling through. With \(r^{(1)}_i\) and \(r^{(2)}_i\) the per-channel ranges of the two layers for the shared channel \(i\), setting the post-scaling ranges equal gives

which balances the quantisation difficulty across the pair. Like SmoothQuant, it is an exact reparameterisation of the float network applied before quantisation, requiring no data at all.

Bias correction addresses the fact that weight rounding error \(\Delta W = \hat{W} - W\) is not zero-mean in its effect on the output. The expected shift is \(\mathbb{E}[\Delta y] = \Delta W\, \mathbb{E}[x]\), so subtracting it from the bias removes it exactly, to first order. Estimate \(\mathbb{E}[x]\) from a few calibration batches. One extra forward pass, and it frequently recovers half a point on depthwise-heavy models, making it the best effort-to-reward ratio on the list after per-channel scales.

GPTQ quantises a layer’s weights one column at a time, minimising \(\|WX - \hat{W}X\|_2^2\) using the layer Hessian \(H = 2XX^\top\) estimated from calibration activations, and after fixing each column it updates the remaining columns to compensate for the error just introduced. That compensation is what separates it from round-to-nearest: error is absorbed rather than accumulated. AWQ observes instead that roughly 1% of weight channels are salient, and that saliency is predicted by the magnitude of the corresponding activation channel rather than the weight. Rather than keeping those in fp16, it scales them up before quantisation by a grid-searched per-channel factor and folds the inverse into the preceding op, so the output stays uniformly int4. Both run in hours where QAT would take weeks.

Takeaway: most quantisation damage comes from a small number of identifiable causes: channel range imbalance, fixed-channel activation outliers, precision-sensitive normalisation and softmax, and residual accumulation. Nearly all of them have a data-free or calibration-only fix that you should exhaust before considering QAT.

Where to Spend Your Precision Budget

Weight only versus weight and activation

These solve different problems, and choosing wrongly wastes your time.

Weight-only quantisation stores weights in int4 or int8 and dequantises them into registers immediately before an fp16 matmul. It reduces weight memory and bandwidth, does not reduce FLOPs, and adds the dequantisation work. It is right when you are memory bound.

Weight and activation quantisation puts both operands in int8 and uses integer matmul instructions. It reduces bandwidth and increases arithmetic throughput by 4 to 8x on Arm. It is right when you are compute bound, and it is harder, because activations are dynamic and prone to outliers.

The deciding quantity is arithmetic intensity, from Part 1. In single-token LLM decoding every weight is read once and used for two floating-point operations, an intensity of about 1 FLOP per byte in fp16. Every machine you can buy has a ridge point well above that, so decode is bandwidth bound and time is essentially (weight bytes) / (bandwidth). Cutting weights from fp16 to int4 cuts the time by close to 4x, and no amount of integer throughput would have helped.

A convolutional vision model at batch 1 is the mirror image. A 3x3 convolution with 256 input and 256 output channels on a 56x56 feature map reuses each weight \(56 \times 56 = 3136\) times. Arithmetic intensity is high, the layer is compute bound, and int8 matmul instructions deliver the 4 to 8x directly. Weight-only quantisation here buys a smaller binary and nothing else.

Choosing a quantisation scheme by regime.

A sensitivity analysis recipe

You will not have the budget to keep everything in fp16, and you should not quantise everything either. The recipe is mechanical and takes an afternoon.

1. Build a float baseline and measure the real metric on a held-out set: mAP, mIoU, word error rate, perplexity, whatever you ship against. Record per-class and tail behaviour too, not just the headline.
2. For each quantisable layer \(\ell\), quantise only that layer, leaving everything else in float, and re-measure. Call the drop \(d_\ell\).
3. Measure the latency saving \(\Delta t_\ell\) that quantising layer \(\ell\) alone actually delivers on the target device. Do not estimate it from FLOPs; memory-bound layers will not speed up 4x no matter what the instruction set says.
4. Rank layers by \(d_\ell / \Delta t_\ell\), accuracy cost per millisecond saved, ascending.
5. Quantise greedily down the ranked list until you hit your latency budget, then measure the resulting configuration end to end.

baseline = evaluate(model_fp32, val_loader)
results = []
for name, layer in quantisable_layers(model_fp32):
    with quantise_only(model_fp32, name, calib_loader) as m:
        drop = baseline - evaluate(m, val_loader)
        saving = benchmark_on_device(m) - benchmark_on_device(model_fp32)
    results.append((name, drop, -saving, drop / max(-saving, 1e-6)))

for name, drop, saving, ratio in sorted(results, key=lambda r: r[3]):
    print(f"{name:32s} drop={drop:6.3f}  saving={saving:6.2f}ms  ratio={ratio:.4f}")

Step 5 hides an assumption: the procedure treats per-layer drops as additive, which they are not. Errors compound as they propagate, so a multi-layer configuration typically measures worse than the sum of its parts. The ranking survives, because the interaction term is usually smaller than the differences between layers, but the predicted accuracy is a lower bound on the damage and you must verify the final configuration by measuring it.Occasionally the interaction goes the other way and the combination is better than the sum, because one layer's quantisation error partially cancels another's. Do not plan around this; it is not reproducible across data.

The expensive layers are predictable once you have done this a few times: the first convolution, the classifier, depthwise layers with wide per-channel range variation, and anything immediately before a softmax. The cheap ones are the wide 1x1 convolutions and the middle of the trunk, which is fortunately where most of the compute lives.

Evaluation discipline

A single top-1 accuracy number is close to useless for judging a quantised model, and I have watched more than one ship on the strength of one.

Check the tail, not the mean. Quantisation error concentrates on examples whose activations sat in the tail of the calibration distribution, which are exactly the hard, rare and safety-relevant cases. A detector can lose 0.3 mAP overall while losing 4 points on small objects, and small objects were the reason you trained a detector.

Check per class. Aggregate metrics average over classes weighted by frequency, so a rare class collapsing entirely can move top-1 by under a tenth of a point. Print the per-class delta, sorted, and read the bottom of the list.

Check the flip rate. Two models with the same accuracy drop can differ enormously in how many predictions they changed. A 0.2% drop with a 0.3% flip rate is behaving as expected; a 0.2% drop with a 6% flip rate has been substantially rewired and its errors happen to balance, which will not hold on your next data distribution.

Check on the device, not in the simulator. This is the one that has burned me. Framework simulators use float arithmetic with fake quant nodes; the device uses fixed-point requantisation. They differ in rounding mode (ties to even versus ties away from zero versus round half up), in saturation behaviour at the clamp boundaries, and sometimes in whether an intermediate is truncated or rounded. Each difference is worth at most one least-significant bit per operation, which sounds negligible and is not: those one-bit differences propagate through fifty layers and past a softmax, and a simulator-validated model can show a real accuracy delta on hardware.The practical discipline is to run device and simulator on the same fixed batch and compare intermediate tensors layer by layer, not just the final output. The first layer where the maximum absolute integer difference exceeds one tells you exactly which operator's rounding semantics disagree, and it is usually a requantisation or a pooling average. Build the on-device evaluation harness before you need it.

Takeaway: rank layers by accuracy cost per millisecond saved and spend your float budget at the top of that list. Then validate the chosen configuration on the target device, checking tail behaviour, per-class deltas and flip rate, because a single aggregate number hides exactly the failures quantisation causes.

A Few Problems to Work

Problem 1. A post-GELU activation tensor has calibrated range \([-2.5, 9.5]\) and will be stored as uint8 asymmetric (\(q \in [0, 255]\)). Compute \(s\) and \(z\). Quantise \(x = 3.3\), dequantise it, and give the error. Then state what the maximum possible dequantisation error is for any value inside the range.

Problem 2. A convolution has \(s_x = 0.0316\), \(s_w = 0.0027\) and an output quantiser with \(s_y = 0.0483\), \(z_y = -8\), producing int8. Derive the fixed-point requantisation parameters \((M_{\text{int}}, n)\) and apply them to an int32 accumulator value \(a = 35000\). Compare with the exact result.

Problem 3. A 7-billion-parameter language model is quantised weight-only to int4 with group-wise scales along the reduction axis: one fp16 scale and one int4 zero point per group of \(G\) weights. Compute the effective bits per weight and the total weight memory for \(G \in \{32, 64, 128\}\). On a device with 100 GB/s of usable memory bandwidth, estimate the single-stream decode rate in tokens per second for each, and for the fp16 baseline. State the assumption that makes this estimate valid.

Problem 4. Using the mixture distribution from the worked example (99% \(\mathcal{N}(0,1)\), 1% \(\mathcal{N}(0,36)\), total variance 1.35), suppose you apply SmoothQuant-style migration that reduces the outlier component’s standard deviation from 6 to 2 while leaving the bulk unchanged. Recompute the MSE-optimal SQNR and state how many decibels the transformation bought. You may use the fact that the optimal threshold for the new mixture is \(t = 5.63\), at which the clipping MSE is \(3.215\times10^{-5}\).

Problem 5. You have profiled a detector layer by layer on the target device. Using the table below, and assuming per-layer accuracy drops are additive, choose which layers to quantise to hit a 8.0 ms budget with the least predicted mAP loss. State the resulting latency and predicted drop, and say why the answer should not be trusted without a further measurement.

What’s Next

Quantisation reduces the cost of every number the model touches. The next lever reduces how many numbers there are: removing weights that contribute little (pruning), exploiting structure in what remains (sparsity), and training a smaller model to imitate a larger one (distillation). These compose with quantisation rather than competing with it, and the composition has its own sharp edges, because a pruned network’s surviving weights often have a wider dynamic range than the dense model you started from, which makes it harder to quantise, not easier.

That’s all for Part 5! For Part 6, on pruning, sparsity and distillation, click here.