Vision Models in the Real World

Every vision system I have shipped has been slower than its model. I spent two years at Dyson putting convolutional networks for classification, detection and segmentation onto robot hardware, and before that I built automated pipelines at the University of Leicester that chewed through high resolution satellite imagery in near real time. Different sensors, different physics, different customers. The first profile told the same story both times: the network was a minority shareholder in the latency budget, and the majority holder was a pile of unglamorous code nobody had ever looked at.

This chapter is about that gap. The previous six parts built the machinery; here I point it at a real vision workload and ask the book’s three questions. How fast can this possibly run, what is actually limiting it, and what do I change first. The answer is almost never “make the backbone smaller”, and the reason is worth several thousand words.

Three Workloads, Three Cost Structures

People say “computer vision” as though it were one thing. For a performance engineer it is at least three, and the differences live in the output tensor rather than the backbone. Classification, detection and segmentation can share an identical feature extractor and still have cost profiles that differ by two orders of magnitude at the far end.

What Each Head Actually Emits

Take a common setup: 640x640 input, 80 classes, a feature pyramid with output strides 8, 16 and 32. The number of spatial locations across the three levels is \((640/8)^2 + (640/16)^2 + (640/32)^2 = 6400 + 1600 + 400 = 8400\).This is the anchor-free count, one prediction per location. An anchor-based head with three anchors emits 25,200 predictions from the same feature maps, tripling every downstream postprocessing cost before a single extra object has been found.

Output tensor, postprocessing cost and dominant failure mode for the three workloads, at 640x640 input with 80 classes (19 for segmentation) and output strides 8, 16 and 32.

Classification feels easy because postprocessing is free, the output is 320 bytes, and the failure mode is invisible in aggregate metrics. Detection and segmentation both put real work after the network, and that work costs an amount depending on the scene rather than the tensor shapes, which makes it hostile to static budgeting.

Segmentation: Output Stride and the Upsampling Tax

Segmentation has a specific trap and almost everyone falls into it once. The network emits logits at stride 8, a [1, 19, 80, 80] tensor of 486 KB at fp32, and the application wants a label map at 640x640. The obvious implementation upsamples the logits bilinearly and then takes argmax. Count the bytes. Upsampling to [1, 19, 640, 640] at fp32 writes \(640 \times 640 \times 19 \times 4 = 31.1\) MB; the argmax reads those back and writes a 410 KB label map, so total traffic is about 62.6 MB. At 10 GB/s to DRAM that is 6.3 ms of pure memory movement with no arithmetic in it at all. Reversed, taking argmax at 80x80 first and then upsampling the labels moves under 1 MB, about 0.09 ms, roughly 70 times cheaper.The two are not identical: taking argmax first quantises every boundary to the output stride. On the datasets I have measured this costs a fraction of a point of mIoU, concentrated entirely in the boundary F-score and nothing in region level accuracy.

So keep masks at the output stride as long as you can and upsample only the region you need. A robot avoiding an obstacle wants a precise boundary in a 128x128 window around it, not across the whole frame, and upsampling a 16x16x19 crop to 128x128x19 costs about 1.3 MB, roughly 0.13 ms, for the same boundary quality where it matters.

Where the Time Actually Goes in a Detection Pipeline

Here is the thing that surprises people every time. In a naive but entirely reasonable detection pipeline, the code before and after the network costs more than the network.

The Preprocessing Chain Nobody Profiles

A camera frame does not arrive as a normalised NCHW float tensor. It arrives as NV12 from the ISP or as a compressed H.264 frame, and between the sensor and the first convolution sits a chain of memory bound operations: decode, colour conversion, resize, letterbox, normalise, and often a layout transpose. Colour conversion holds the first free win. NV12 to RGB at 1920x1080 reads 3.1 MB and writes 6.2 MB, and a scalar YUV matrix runs at a fraction of memory bandwidth because of the chroma indexing. Resize before you convert and you convert 640x360 pixels instead of 1920x1080, nine times less work.You can resize the Y and interleaved UV planes independently, since bilinear interpolation commutes with an affine colour transform to within a rounding error you will never see after int8 quantisation.

Normalisation should not exist at all. It is an affine transform, \(x' = (x - \mu)/\sigma\), and int8 quantisation is also affine, \(q = x'/s + z\). Composing the two gives an affine transform, so fold the mean and standard deviation into the input tensor’s quantisation scale and zero point and feed the network raw uint8 pixels.Every serious toolchain supports this, though rarely by default: set the input quantisation parameters so the dequantised value equals the normalised value, then delete the preprocessing op. Verify against the reference pipeline on a few hundred images first, because a sign error here produces a model that is subtly and quietly worse.

A Worked Budget

An illustrative end to end budget for a 1080p camera feeding a 640x640 int8 detector on a mid-range Arm SoC with an NPU. These figures are chosen to have realistic proportions; they are not measurements of any particular product.

In the naive column the network is 46% of the budget and everything else adds up to 13.95 ms, more than the network itself. Preprocessing alone is 5.80 ms and postprocessing alone 6.15 ms, either one half the network’s cost.

Now consider two ways to spend engineering time. Distil the backbone, retrain, requantise and revalidate to get the network from 12.00 to 8.00 ms: three weeks for a 15% cut in total latency. Or fix the plumbing above: an afternoon for a 40% cut, bit-exact outputs, no retraining, and 38.5 to 64.0 fps without touching the model. After the plumbing is fixed, the same 4 ms of distillation is worth 26% rather than 15%, so doing the cheap thing first raises the value of the expensive thing you do next.

The head decode row hides a small, general trick. The standard implementation applies a sigmoid to all \(8400 \times 80 = 672{,}000\) logits before thresholding. But the sigmoid is monotonic, so \(\sigma(z) > t\) is the same test as \(z > \ln(t/(1-t))\). At \(t = 0.30\) that is \(z > -0.847\): compare raw logits against a constant, which vectorises perfectly, and evaluate the transcendental only on the hundred or so survivors.

Takeaway: Profile the pipeline, not the model. In a naive detection pipeline the code outside the network routinely costs as much as the network, and most of it is fixable in an afternoon with no retraining and no accuracy change.

Resolution Is the Knob Everyone Abuses

Input resolution is the most powerful lever in a vision system and the one most often pulled the wrong way. Cost scales roughly with the square of the input side length, because the spatial extent of every activation does. Accuracy does not: it rises and then saturates, and where it saturates depends entirely on how large your objects are in pixels.

640 Against 1280

Cost of doubling the input side for the same detector family, with an illustrative 8.2 GMAC backbone and neck at 640 on a device sustaining 0.68 GMAC/ms.

Two rows matter more than the rest. NMS is 16x rather than 4x because it is quadratic in candidate count and the candidate count is itself quadratic in the input side: quadruple your pixels and you can sixteen-tuple your postprocessing. And 6.55 MB may not fit in on-chip SRAM when 1.64 MB did, in which case the compiler spills tiles to DRAM and the 4.0x becomes 6x or worse. Quadratic scaling is a floor, not a ceiling.

The last row is the mechanism behind small object recall. Letterboxing 1920x1080 into 640x640 applies a scale of \(\min(640/1920, 640/1080) = 0.333\), so a 48 pixel source object becomes 16 pixels at the input, two cells at the finest stride of 8. Recall against object size has a knee right about there: below roughly two cells at the shallowest head, the feature that must encode the object has a receptive field dominated by background and detection falls off a cliff. At 1280 the scale is 0.667, so the same two-cell threshold is reached by a 24 pixel source object.

So doubling the input side roughly halves the smallest object you can see, for four times the compute and up to sixteen times the postprocessing. In published scaling tables the mAP gain from 640 to 1280 is single digit points, almost all of it in the small-object bucket, so if your objects are large you are paying 4x for nothing. The better alternative is usually to keep the 640 model and run a second 640 pass on a crop: 24.0 ms against 48.2 ms, at native source resolution inside the crop. You then need a policy for choosing the crop, which is exactly what tracking gives you for free.

Tiling Large Imagery and the Overlap Tax

At Leicester I worked on pipelines over satellite scenes tens of thousands of pixels on a side. You cannot feed those to a network, so you tile them, and tiling has a tax people underestimate. Let the scene be \(W\) pixels wide, the tile \(T\) pixels and the overlap \(o\) pixels. The tile stride is \(T - o\), so tiles per side is

and the total goes as \(n^2\). For large \(W\) the ratio against non-overlapping tiling is approximately \(\left(\frac{T}{T-o}\right)^2\).

Tile counts and the overlap tax for a 10,000 x 10,000 pixel scene at a 512 pixel tile size.

The overlap is not optional: an object straddling a seam is cut in half in both tiles, and a detector that sees half a ship will miss it or emit two low confidence partials. So the overlap must be at least the largest object diameter you expect, and only that. Choosing 50% “to be safe” when your largest object is 100 pixels costs 3.80x instead of 1.69x and buys nothing; I have watched an afternoon of compute disappear into that one parameter.

Then deduplication. Every object in an overlap band is detected twice, so you need a global NMS pass in scene coordinates. With 529 tiles emitting 30 boxes each that is 15,870 boxes, and a naive global pass is \(15870^2/2 \approx 126\) million IoU tests, minutes of CPU time that dwarf the inference you just did. The fix is geometric: only boxes whose centres lie within \(o\) pixels of a seam can be duplicates, so bin them spatially and run NMS in the seam bands only.

Takeaway: Resolution costs go as the square of the side length and postprocessing costs as the fourth power. Set the input resolution from the smallest object you must detect in source pixels, and the tile overlap from the largest. Both are measurable quantities, not taste.

Backbones That Are Fast on Real Silicon

Paper FLOP counts and wall clock time have been drifting apart for a decade. A backbone with half the multiply-accumulates can easily be slower, for reasons straight out of Part 1 and Part 3: arithmetic intensity, and whether the shapes fit the machine.

Per-output-element arithmetic and data movement for common building blocks, at 56x56 spatial resolution with 128 input and 128 output channels.

The depthwise row is the important one.The "2 activation elements" accounting assumes one input read and one output write per output element, with the 9x reuse of a 3x3 window served from on-chip memory. Weights are excluded: a dense 3x3 128 to 128 convolution holds 147,456 of them, 144 KB at int8, which usually stays resident and amortises to nothing per element. A depthwise separable block does 137 MACs per output element against the dense block’s 1152, an 8.4x saving on paper, but the depthwise part has 128 times less arithmetic per byte moved. On a machine built for dense convolution that 8.4x routinely shows up as 2x to 3x wall clock, and on very wide MAC arrays as no saving at all.

What is genuinely fast on edge silicon is boring: plain 3x3 convolutions at stride 1 or 2, for which every compiler has a tuned path; ReLU and ReLU6, which fuse into the preceding convolution for free; and channel counts aligned to the machine, so that on a 16 wide MAC array 128 channels is eight full tiles while 100 channels pads to 112 and wastes 12% of every cycle. Choose 32, 64, 96, 128, 256, not 100 or 144 because an architecture search liked them.

What looks cheap on paper and is not:

• Exotic attention at high spatial resolution. Global self attention over \(H \times W\) tokens costs \(O((HW)^2 C)\). At stride 8 on a 640 input that is 6400 tokens, and with 256 channels the QK and AV products alone are \(2 \times 6400^2 \times 256 \approx 21\) GMAC, against 8.2 GMAC for the whole backbone above. One global attention layer at stride 8 costs two and a half times the entire network. At stride 32 the token count is 400 and the same layer costs 82 MMAC, which is free. This is why practical designs put attention at the deepest, smallest feature map, and it is a decision you make with arithmetic rather than taste.
• Unusual dilation rates. Dilation breaks the contiguous sliding window assumption convolution kernels are built around: at dilation 2 every other cache line is wasted, and beyond that the gather degenerates. Worse, many NPUs support dilation 1 and nothing else, so an ASPP module with rates 6, 12 and 18 silently falls back to the CPU.Read the operator support table for your target before choosing an architecture, not after. The most expensive mistake I have watched a team make was discovering, three months in, that one unsupported operator mid-graph forced a round trip to the CPU on every inference, partitioning the model into three subgraphs and destroying all the pipelining.
• Group convolutions with awkward group counts, which leave partial tiles at every group boundary, and anything that reshapes or transposes between layers, which is pure bandwidth with zero arithmetic and shows up in the profile as a mystery.

NMS, and Why Too Many Boxes Is a Latency Bug

Greedy non-maximum suppression is fifteen lines of code and, in my experience, the single most common cause of a detection pipeline blowing its latency budget. Sort the candidates by score, take the highest, emit it, discard every remaining box whose IoU with it exceeds a threshold, repeat. The sort is \(O(k \log k)\) and irrelevant; the suppression loop is the problem. In the worst case, where nothing suppresses anything, it performs \(k(k-1)/2\) IoU tests. On a clustered scene it performs perhaps a third of that, but the worst case is what your p99 sees, and it arrives when the scene is crowded, which is when you need the detection.The typical-to-worst ratio is scene dependent and cannot be bounded a priori, which is the real reason NMS is dangerous: its cost depends on the input image rather than the tensor shapes, so it is invisible to every static cost model and every compiler.

Each IoU test is four min/max operations, a few multiplies and a divide, and the divide is avoidable: instead of \(\frac{I}{A_1 + A_2 - I} > \tau\), test \(I > \tau (A_1 + A_2 - I)\). Same predicate, one multiply instead of a divide, and it vectorises.

Class Agnostic, Per Class and Batched

Class agnostic NMS runs one pass over all \(k\) candidates regardless of label. It is cheapest on paper and usually wrong: a person standing in front of a car produces two heavily overlapping boxes of different classes, and class agnostic NMS deletes one. Per class NMS partitions by label first, and if candidates spread evenly over \(C\) active classes the total is \(C \cdot (k/C)^2 / 2 = k^2/(2C)\), so it is \(C\) times cheaper as well as more correct. Candidates never spread evenly in practice, but even a skewed distribution gives a large constant factor.

The implementation trick is batched NMS: rather than launching a kernel per class, add a per class offset to every box coordinate so boxes of different classes can never overlap geometrically, then run a single class agnostic pass.The offset must exceed the largest possible coordinate, so `box + class_id * 8192` is safe for any image under 8192 pixels. The result is identical to per class NMS with one kernel launch instead of C, and the quadratic term collapses to the per class sum automatically because cross-class IoUs are all zero.

The Score Threshold Is the Real Latency Control

Illustrative NMS cost against score threshold, for an 80 class detector with 8,400 locations on a moderately busy indoor scene, at 6 ns per IoU test.

Read that again. Moving the score threshold from 0.05 to 0.25 changes NMS cost by a factor of 100. Nothing else in the pipeline has a knob with that authority, and it costs nothing to turn.

So why do people ship detectors thresholded at 0.001? Because of mAP. Average precision integrates the precision-recall curve over the full recall range, so a box emitted at 0.001 that happens to be correct adds a sliver of area: the protocol actively rewards emitting garbage. Fine for a leaderboard, ruinous for a product, because no consumer in a real system has ever acted on a 0.001 confidence detection. The threshold you evaluate at should be the threshold you ship at.

Two defences, and you want both. A score threshold picked off the precision-recall curve at the false positive rate your product tolerates, not from a config file someone copied in 2019. And a hard top-k cap before NMS: keep the best 300 regardless of threshold and the quadratic term is bounded at \(300 \times 299 / 2 = 44{,}850\) tests, about 0.27 ms, whatever the scene does. That is the difference between a latency distribution with a tail and one without.

The design-level version: a model that emits too many candidate boxes is a latency bug, not an accuracy feature. An anchor-based head with three anchors per location emits three times the candidates of an anchor-free head on identical feature maps, so up to nine times the NMS cost, for a recall difference anchor-free designs have largely closed.

Takeaway: NMS cost is quadratic in candidate count and depends on the image rather than the tensor shapes, so it hides from every static cost model. Set the score threshold from your shipping operating point, add a top-k cap, and use batched per class NMS.

Temporal Structure Is Free Accuracy

Almost every deployed vision system sees video, and almost every one throws away the fact that frame \(t\) looks very much like frame \(t-1\). That correlation is the cheapest compute you will ever find.

Detect Every N Frames, Track in Between

Run the full detector every \(N\) frames and propagate boxes with a cheap tracker in between. If the detector costs \(D\) and the tracker \(T\), the mean per frame cost is

With \(D = 16\) ms and \(T = 1.5\) ms: \(N=1\) costs 16.0 ms, \(N=4\) costs 5.13 ms (3.1x), \(N=8\) costs 3.31 ms (4.8x). The curve saturates at \(T\), so past \(N \approx 10\) you buy almost nothing.

class DetectAndTrack:
    def __init__(self, detector, stride=4, still_thresh=2.0):
        self.det, self.stride = detector, stride
        self.still_thresh, self.frame_idx = still_thresh, 0
        self.tracks, self.prev_thumb = [], None

    @torch.inference_mode()
    def step(self, frame):                  # frame: uint8 HxWx3, colour converted
        # ~0.05 ms: 32x24 grey thumbnail, mean absolute difference
        thumb = cv2.resize(cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY), (32, 24))
        still = (self.prev_thumb is not None and
                 np.abs(thumb.astype(np.int16) -
                        self.prev_thumb.astype(np.int16)).mean() < self.still_thresh)
        self.prev_thumb = thumb
        self.frame_idx += 1

        if still:
            return self.tracks              # scene unchanged, reuse everything

        if self.frame_idx % self.stride == 0:
            boxes, scores, labels = self.det(frame)          # the 16 ms path
            self.tracks = associate(self.tracks, boxes, scores, labels, iou=0.3)
        else:
            for t in self.tracks:
                t.box = t.kalman.predict()                   # microseconds
            refine_by_flow(self.tracks, frame)               # sparse LK, ~1.5 ms

        return self.tracks

Two warnings the mean cost formula hides. First, the mean is not the tail: every \(N\)th frame still costs 16 ms, so if your budget is per frame rather than per second, raising \(N\) does nothing for p99. The fix is to run the detector on a separate core and consume its result one frame late, converting a latency spike into a fixed one-frame lag.

Second, \(N\) is bounded by the physics of the scene, not by the compute saving. A newly appeared object is invisible for up to \(N-1\) frames: at 30 fps and \(N=8\) that is 233 ms of blindness, and a floor robot at 0.35 m/s travels 8.2 cm in that time, which is the difference between stopping before the cable and eating it.Constant velocity Kalman prediction also degrades badly under acceleration, which is what happens when a pet walks into frame or the robot turns. If the platform has an IMU, feeding ego-motion into the prediction step is nearly free and buys back most of the error. Part 9 picks up where perception and control stop being separable. Choose \(N\) from the consequence of a late detection, then check whether the saving is enough.

Cropping and Early Exit

Two more tricks compose with the first. Region of interest cropping: the previous frame’s tracks tell you where the action is, so on intermediate frames run the detector on a 320x320 crop around the union of tracked boxes. Cost scales with area, \((320/640)^2 = 0.25\), so 4 ms instead of 16. You lose sight of anything outside the crop, so alternate full frames every \(N\) with crops between. This is also the honest answer to the 640-against-1280 question, since the crop gives native resolution exactly where objects already are.

Early exit on an unchanged scene: downsample the current and previous frames to a 32x24 grey thumbnail, 768 pixels, and take the mean absolute difference, about 0.05 ms. Below a threshold, skip everything and reuse the previous detections.

This saving is entirely workload dependent. On a fixed inspection camera the skip rate can exceed 90% and the mean per frame cost collapses towards 0.05 ms; on a robot that is driving it is near zero and you have added 0.05 ms of overhead per frame for nothing. With skip rate \(s\) the mean is \(\bar{C} = (1-s)\left(\frac{D}{N} + \frac{N-1}{N}T\right) + 0.05s\), so measure \(s\) on real captured data before you put the saving in a slide.

Takeaway: Detecting every N frames with tracking in between typically buys 3x to 5x on mean cost. It buys nothing on the tail unless you move the detector off the critical path, and N must be chosen from how long you can afford to be blind, not from the speedup curve.

Data, Domain Shift and Metrics That Matter

Everything above is worth perhaps a 2x to 4x on latency. Data is worth more than that on accuracy, and it is almost always where the real problem is.

Take class imbalance with actual numbers. Suppose the class you care about appears in 1 frame in 500, typical for the hazards a floor robot must avoid. In a 200,000 frame set that is 400 positive frames; train for 100 epochs and the model sees the rare class 40,000 times against 20 million backgrounds. Oversample those 400 frames twenty times and it is seen 800,000 times while the epoch grows by 7,600 frames, 3.8%. Twenty times the signal for 3.8% of the compute is the best trade in this chapter, and it needs no architecture work at all.

Label noise is the sibling problem and more insidious. If 5% of true objects are unlabelled, the model is punished for 5% of its correct detections, and the loss teaches it to suppress exactly the ambiguous cases you built the system for. There is a cheap audit: run the trained model over its own training set and sort false positives by confidence. In every dataset I have done this on, the top of that list is missing labels, not model errors, and an afternoon of relabelling the top 200 moves the number more than a week of architecture search. Hard negative mining should then draw from deployment logs rather than curated data, because the confusers that matter are shoelaces that look like cables and reflections on a polished floor.

Evaluation Discipline

Domain shift is where lab numbers go to die, and the two domains I know best fail differently. The lab has clean floors, even lighting and one floor type; a customer’s home has mixed flooring, evening light, a sensor at high gain with visible noise, pets, and a robot that often works in near darkness under its own illumination. The dominant gap is not object appearance, it is illumination and sensor noise. For satellite imagery, two tiles of the same field differ by sensor, sun angle, season, atmospheric correction and ground sample distance, and a model trained on summer tiles over one region degrades on winter tiles with no error message. Per tile percentile normalisation, stretching each tile’s own intensity distribution before it reaches the network, helped me more than any architectural change I tried.

The discipline that catches this is unglamorous. Never split by random frame: frames 1001 and 1002 are the same scene 33 ms apart, so a random split puts training data in the validation set, and I have watched that inflate a headline number by around ten points in a way only visible after deployment. Split by capture session, by home, by region and by date, and for satellite work on both space and time. Keep a golden set that is never trained on and never tuned on, because the moment you use it to pick a hyperparameter it stops being a test set.

Metrics That Survive Deployment

What common vision metrics tell you and what they hide once the system is in a product.

mAP is a good research metric and a poor deployment one, for the same reason it encourages a threshold of 0.001: it averages over the thing you care about. What belongs on a dashboard is per class recall at a fixed false positive rate, measured at the exact threshold in the shipped config, because that maps onto two sentences a product owner can act on: how often does the robot miss a cable, and how often does it stop for nothing.

Latency should be a distribution. Report p99 rather than the mean, and report it from sensor timestamp to actuation decision rather than from the network’s input tensor to its output. In the budget table the forward pass was 12 ms of a 26 ms pipeline; a team reporting “12 ms inference” would be telling the truth and communicating nothing.NMS makes the latency distribution input dependent, so a mean over a quiet test set can be less than half the p99 over a busy real one. If you only ever record one number, record the 99th percentile over a replay of real captured data, not over your validation set.

Takeaway: Oversampling a rare class by 20x costs 4% more data volume and buys 20x the gradient signal, which beats any architecture change available to you. Split evaluation data by scene and by date, never by frame, and report per class recall at a fixed false positive rate alongside p99 end to end latency.

A Few Problems to Work

Problem 1. A satellite scene is 12,000 x 12,000 pixels. Your detector takes 512x512 tiles at 9.0 ms per tile, and the largest object you must detect is 96 pixels across. Choose a sensible overlap and compute the tile count and inference time. Do the same for 256x256 tiles at 2.25 ms per tile, then say which you would ship.

Problem 2. A detector emits 8,400 locations over 80 classes. At a score threshold of 0.05, 1,600 candidates survive; at 0.30, 90 survive. An IoU test costs 6 ns. Compute worst-case class agnostic NMS time at each threshold, then at 0.05 with a top-k cap of 300, then at 0.05 with per class NMS over 12 active classes.

Problem 3. A detector costs 18.0 ms, a tracker 1.6 ms, and the camera runs at 30 fps. Compute mean and peak per frame cost for \(N = 1, 4, 8\), then the worst-case delay before a new object is detected and how far a robot at 0.35 m/s travels in that time.

What’s Next

The pattern here generalises past vision. Find the whole pipeline, cost every stage in bytes and operations, and look first at the stages nobody has profiled, because those hold the easy factors of two. Vision makes this vivid because its pre and post stages sit so obviously outside the network, but the same discipline applies to the language models in the next part, where the tokeniser, the sampling loop and the KV cache play exactly the roles that colour conversion and NMS play here.

That’s all for Part 7! For Part 8, on transformers on small machines, click here.