Perception, VLMs and Robots

Every chapter so far has treated latency as a quality of service number: the model should be faster because the user waits less. Robotics is where that framing breaks. When a model sits inside a feedback loop driving an object with mass, latency becomes a term in the stability analysis. Past a certain delay the controller does not degrade gracefully. It rings, then it oscillates, then somebody unplugs it.

I spent two years as a robotics software engineer at Dyson, shipping convolutional networks for segmentation, detection and classification onto robot hardware and shrinking them with quantisation, pruning and distillation. Most of my time went on the parts that were not the network. This chapter is about the whole loop, and about where vision language models actually earn their place, which is usually not where the demos suggest.

The Robot Is a Control Loop, Not a Benchmark

A benchmark asks how good the output is and how long it took; a control loop asks when the actuator responds and how stale the information behind it is. Optimising for the first can make the second worse.

Closed loop latency and the age of information

Define the age of information as the time between the physical instant a measurement describes and the instant an actuator acts on it:

where \(t_{\text{capture}}\) is the midpoint of the sensor’s integration window, not the moment the frame lands in your process.Under a rolling shutter every row has its own capture time, and the per-row skew, often 10 to 20 ms across a frame, is itself a geometric error source on a moving camera. Then add the zero order hold: if the loop period is \(T\) and the command is held until the next update, the information behind it is up to \(T\) older again by the end of the hold.

Throughput is a different quantity: a three stage pipeline with 30 ms per stage delivers 33 frames per second at an age of 90 ms. Report the 33 and not the 90 and you have reported the wrong number.

Worked example: a 30 Hz camera, 6 ms exposure, 27 ms from end of exposure to motor command, 33.3 ms hold.

At 1.2 m/s the robot has travelled 76 mm since the photons that justified the command. Put the 76 mm on the slide, not the 63 ms: a distance is something a reviewer can argue with.

Why ten milliseconds can start an oscillation

A pure transport delay leaves a loop’s magnitude response untouched. It only rotates the phase, by \(-\omega\tau\) radians, and at gain crossover that lag comes straight out of your phase margin. In degrees:

Phase lag contributed by pure delay, as a function of loop bandwidth. This is the whole argument in one table.

Closed loop damping is roughly the phase margin in degrees over 100,From the second order prototype. The linear rule holds to a few degrees for phase margins between about 20 and 60, which covers everything you will tune in practice. so 45 degrees gives about 20 per cent overshoot and 25 degrees about 45 per cent with visible ringing.

On an 8 Hz visual servoing loop with 45 degrees of margin, a change adding 10 ms leaves 16 degrees: not slightly worse tracking, but a joint that hunts audibly. On a 1 Hz floor robot the same change costs 3.6 degrees and nobody notices. Identical software, an incident on one platform and a non-event on the other, and the only difference is \(\omega_c\).

Hence the most useful degradation strategy in this chapter: if \(\tau\) grows and you cannot shrink it, shrink \(\omega_c\). Slow the robot down. Their product is what matters.

Takeaway: Delay is phase and phase is stability. Budget the age of information from exposure midpoint to actuator command, not inference time alone.

Budgeting the Whole Perception Stack

I start every perception system by writing down each stage between a photon and a torque with a number next to it. Most teams never do this, which is why most teams optimise the wrong stage.

Where the time actually goes

An illustrative sense-to-act budget at two loop rates. These are allocations rather than measurements from any product, but they are the shape I have repeatedly seen once a stack is instrumented. Preprocessing covers debayer, undistort, resize and normalise; postprocessing covers decode, non maximum suppression and mask upsampling.

The network is usually a minority of the budget

In the 33 ms column the network is 27 per cent of the loop. Halving inference time through heroic quantisation work buys 4.5 ms, a 14 per cent improvement in what matters. Noticing that you block on a depth frame you could have predicted forward, and that preprocessing runs a colour conversion the ISP would do for free, can buy the same 4.5 ms in an afternoon.

This is the most common mistake I have watched teams make, including teams I was on: the network is the part with a leaderboard attached. Exposure is the largest single line in the 33 ms column and almost nobody treats it as tunable, yet halving it from 6 ms to 3 ms costs a stop of light, paid for in gain, noise and accuracy. That trade belongs in the same conversation as pruning.Auto exposure makes this worse by varying exposure with scene brightness, so loop delay changes when the robot drives from a bright room into a dark corridor. If you are serious about determinism, cap the exposure and let the image get noisy rather than letting phase margin depend on the lighting.

The network’s share grows as the loop slows: at 100 ms it is 40 per cent and model work is the right lever. Know which regime you are in first. Part 10 is about getting these numbers honestly rather than guessing them.

Takeaway: Write the whole chain down with a millisecond next to every stage before optimising anything. In a fast loop the network is often a quarter of the budget, and the cheapest milliseconds sit elsewhere.

Time Is a First Class Citizen

Clock domains and timestamps

A robot has several clocks and they disagree: a pixel clock in the sensor, a crystal in the motor MCU, a monotonic clock and an NTP-stepped wall clock in the application processor. Treat “now” as one global concept and you build a system whose errors scale with velocity and that nobody can debug.

Four rules, simple to state and tedious to enforce. Every measurement carries a timestamp for the physical event, not for when your code saw it. All timestamps refer to one monotonic reference, with offsets to the other domains estimated continuously, not assumed constant.Crystal frequency offsets of 20 to 50 parts per million are ordinary, which is 72 to 180 ms of drift per hour. Estimate both offset and rate, typically with a two state filter over hardware-triggered or PTP-synchronised events, or cross-sensor alignment degrades over a shift. Nothing downstream asks the clock what time it is; it uses the timestamps that arrived with the data, which is also what makes deterministic replay possible later. And where the budget is tight, synchronise by hardware trigger, because software alignment costs a wait.

Getting this wrong has a signature: the system works on the bench and degrades in proportion to speed in the field, because a fixed timing error becomes a growing spatial error. If accuracy scales with velocity, suspect time before you suspect the model.

Jitter, not the mean

Here is the rule benchmark culture teaches you backwards: a predictable 40 ms beats an average of 25 ms with a 90 ms tail.

A constant delay is compensable: feed it forward, propagate the state estimate to the moment the command will land, retune around it. A delay that varies unpredictably can only be tolerated, and you must design for its worst case anyway. Jitter also injects noise into the effective phase of the feedback path, so a controller tuned at the mean is under-damped at the tail.

Design against a high percentile, and be honest about how high. At 30 Hz you process 108,000 frames per hour, so p99.9 means about 108 overruns per hour, close to two a minute. A dropped frame is fine. But if an overrun is a potential safety event and you want fewer than one per thousand operating hours, you need a miss rate below \(10^{-8}\), a percentile no realistic test campaign can establish. Safety comes not from a tight latency distribution but from a fallback that is correct when the distribution misbehaves.

Determinism, Watchdogs and Graceful Degradation

What is hostile to a real time loop

Dynamic memory allocation. A malloc is usually a few hundred nanoseconds and occasionally a few hundred microseconds, when the arena is contended or first touch of a page faults. Occasionally is the problem. Allocate every buffer at startup, size for the worst case, reuse, and call mlockall so nothing is paged out. Most runtimes manage this given static shapes and almost none do given dynamic ones.

Garbage collection. A managed runtime in the control path introduces pauses you do not control, whose length depends on allocation history. Keep the loop in C++ or Rust, and Python for configuration, tooling and tests.

Opportunistic batching. Serving stacks batch because it raises arithmetic intensity and therefore throughput, as Part 1 describes. Inside a control loop the objective is inverted: filling a batch of four at 30 Hz adds up to 100 ms to the oldest frame’s age so a throughput number can improve by 50 per cent. A dynamic batch size is worse still, because latency then depends on what the other sensors were doing. Use a fixed batch equal to the number of synchronised sensors, even when it is not full.

Anything that adapts at runtime. Autotuned kernel selection, recompilation on a new shape, frequency governors, thermal throttling and NUMA page migration all give a fast steady state with an unpredictable transient. Warm every code path before leaving the workshop, pin threads with SCHED_FIFO on isolated cores, fix the frequency, and steer interrupts off the working cores.Thermal throttling correlates with the conditions you care about: a robot that has run hard for an hour in a warm room is both the most loaded and the most likely to slow down. Measure p99.9 after a sustained thermal soak, never from a cold start.

Missing the budget safely

You will miss the budget, so plan the miss: a deadline-aware wrapper with a tier ladder, plus a watchdog independent of what it watches.

// Preallocated tensors, fixed shapes, no allocation in this function.
Percept run_perception(const Frame& f, Clock::time_point deadline) {
  wd_.kick();                             // hardware watchdog, independent timer
  if (Clock::now() + tiers_[0].wcet < deadline) {
    return tiers_[0].run(f, buf_full_);   // full model, int8, 640x384
  }
  if (Clock::now() + tiers_[1].wcet < deadline) {
    degraded_frames_++;
    return tiers_[1].run(f, buf_small_);  // distilled model, 320x192
  }
  missed_deadlines_++;                    // no tier fits; do not start work
  return Percept::extrapolated(last_good_, Clock::now() - last_good_.stamp);
}

Tiers are chosen on worst case execution time measured offline, so the decision happens before the work starts rather than being discovered halfway through. Nothing allocates, and the degraded path runs regularly rather than only in emergencies.Forcing the degraded tier on one frame in a thousand in production is cheap insurance: it keeps the path warm in instruction cache, keeps its latency in your telemetry, and means the first time you exercise it is not during an incident. And the last branch returns an extrapolated percept with its age attached, so consumers widen their uncertainty instead of receiving a confident stale answer.

Above that sits the safety layer, whose important property is that it does not depend on perception at all. A learned model is not a safety function and cannot be argued to be one. What keeps the robot safe when perception fails is a certified proximity sensor, a bumper, a torque limit and a speed limit chosen so the machine stops within the distance the non-learned sensors can see. Perception makes the robot useful; the independent layer makes it safe, and separating those roles is what lets you ship a neural network at all.

The degradation ladder, in increasing severity: slow the robot, restoring phase margin through the \(\omega_c \tau\) product; hold the last good estimate with growing uncertainty and a hard timeout; execute a controlled stop. A hardware timer the application kicks decides the third, because a software watchdog shares the fate of what it watches.

Takeaway: Report p99.9 and the maximum, never the mean alone. Preallocate everything, refuse dynamic batching, pick tiers on worst case execution time before starting work, and keep a safety layer that does not depend on the model.

Vision Language Models On and Around the Robot

Vision language models are the most interesting thing to happen to robotics perception in years, and the subject of the largest gap between demonstration and deployment. Separating two deployment patterns, and being precise about which you are proposing, helps enormously.

Pattern one: in the loop, on the device

The VLM runs on the robot and its output influences behaviour. This is rare, expensive, and almost always uses a small distilled model. The arithmetic shows why.

Take an illustrative edge part: 2 TFLOP/s of dense int8 peak at 40 per cent achieved utilisation, so 0.8 TFLOP/s effective, with 50 GB/s of bandwidth. Transformer compute is about \(2NP\) operations for \(N\) matmul parameters over \(P\) tokens, and decode is bound by reading the weights once per token, as Part 8 sets out.

On-device VLM latency on that illustrative hardware.

Even the distilled model, at 2.8 Hz, is eleven control periods away from a 30 Hz loop. A VLM is not a perception front end for a fast loop. It works as a supervisory layer in a rate hierarchy: a VLM near 1 Hz emitting goals or a chosen behaviour, a policy at 30 Hz turning that into velocity commands, a motor loop at 1 kHz. Nobody needs the category of the object ahead re-evaluated 30 times a second; they need the distance to it, and those are different models.Make the slow layer's output sticky: require the VLM to agree with itself across two or three consecutive queries before the fast layer changes behaviour. That costs a second of reaction time on genuine changes and removes most of the flapping caused by borderline classifications.

Note which term dominates. For the 2B model, prefill over 576 image tokens is 71 per cent of the total, so the best optimisation is not quantising harder but cutting image tokens by pooling or a coarser patch grid. For the distilled model, decode is 56 per cent, so the lever is fewer output tokens: return a class index or a short tag, never a sentence.

Pattern two: out of the loop, around the programme

Here the VLM never touches the control loop. It runs offline at whatever rate is convenient and automates work humans were doing by hand: auto labelling the images that feed the fast models; triaging fleet logs by clustering interventions and describing what each cluster shares; turning “confidence dropped on 400 frames” into “the robot keeps failing on dark, high pile rugs in low light”; asserting that a scripted test run behaved as intended.

I want to be direct, because the field’s incentives push the other way. The highest return use of a VLM in a robotics programme today is almost always pattern two. Pattern one is a research programme with an uncertain schedule; pattern two is an automation project with a computable payback period, and it usually removes a bottleneck throttling everything else, because what gates your fast models is labelled data and what gates your releases is failure triage.

That is exactly the shape of the VLM project I built at Dyson. It was not a model in the control loop. It took a workflow people were doing by hand, automated the bulk of it with a vision language model under human review, and made the remaining human work an order of magnitude cheaper. It saved over one hundred thousand pounds and improved the workflow by twenty times, and I presented it to the CEO and senior leadership. What made it approvable was not the model. It was that the economics were written down in a form a finance-literate person could check.

Takeaway: Be explicit about which pattern you are proposing. In the loop on device is rare, needs a distilled model and lives in a rate hierarchy above 1 Hz. Out of the loop is where the defensible return is, because it automates human work at a computable payback.

Using a VLM to Build Datasets

Auto labelling with human review

The system is a pipeline, not a model: a VLM or open-vocabulary detector proposes labels; a rule layer rejects proposals violating known constraints, such as impossible sizes or objects floating above the floor plane; a tracker propagates accepted labels across adjacent frames so one decision covers many; and a reviewer sees what remains in an interface where accepting is one keystroke.

The economics live in the difference between drawing and checking. Drawing a polygon mask is slow; checking one and nudging two vertices is fast. The quantity that governs everything is the acceptance rate \(p\), the fraction of proposals accepted without correction. With labour rate \(r\) per second, manual draw time \(t_m\), verify time \(t_v\), extra fix time \(t_f\) and model cost \(c_{\text{model}}\):

Illustrative cost per labelled frame at £25 per hour (£0.00694 per second) and £0.002 of model compute per frame. These are worked assumptions showing the method, not figures from any employer.

Moving from the weak regime to the tuned pipeline is a 4x change in cost, driven mostly by acceptance rate rather than raw model accuracy. Acceptance rate is a product property: it responds to the rule layer, the review interface, sensible class definitions, and a threshold set so the model offers fewer, better proposals. Precision matters more than recall, because a false positive costs a correction while a miss costs a draw.

Active learning: choosing what to label next

Once labelling is cheap per frame, the constraint is choosing which frames deserve a human. Four signals, in rough order of usefulness:

1. Disagreement between independent sources, such as the VLM against the deployed detector. Disagreement across model families beats one model’s own uncertainty.
2. Embedding novelty. Greedily select frames far from everything already in the set; this stops you labelling the same corridor five thousand times.
3. Failure adjacency. Frames near a logged intervention are, by construction, where the system is wrong.
4. Predictive uncertainty. Alone it selects motion-blurred junk no label can fix, so pair it with a diversity term.

The savings compound: if random sampling needs 50,000 frames to hit a target and disagreement-plus-diversity reaches it at 18,000, that 2.8x multiplies the 11.7x per-frame saving.

Are the labels good enough?

Answer this before you start, because it is what kills auto-labelling projects.

Measure the human ceiling first. Have two or three annotators independently label the same 500 frames and compute agreement with the downstream metric. If humans agree at 0.86 mIoU, a pipeline at 0.84 is at parity, and demanding 0.95 from the machine is demanding something humans never delivered. Hold out a gold set the pipeline never touches, stratified by lighting, surface and class, and report per stratum, because an aggregate hides the failures that hurt.

Measure downstream accuracy. Label quality is instrumental. Train identical models on \(n\) gold labels, \(n\) auto labels and \(4n\) auto labels and compare on the gold set; more noisy labels often beats fewer clean ones, up to a threshold you can only find by measuring.Label noise is not uniformly harmful. Random boundary jitter on masks is largely averaged out in training, while systematic errors, such as omitting a class under one lighting condition, are learned faithfully and become a permanent blind spot. Weight auditing towards correlated errors.

Do not train on your own output, or you teach the student the teacher’s blind spots. The source should differ meaningfully: a larger model, another modality, temporal information the student lacks at inference, or a sensor the robot does not carry. Then audit forever, sampling 2 per cent of accepted labels for blind review, because acceptance rate drifts when the fleet meets new environments and that drift warns you the deployed model is about to degrade too.

The economics, and how to get it approved

A technically excellent automation project nobody funds has produced zero value. The bridge is a payback calculation. With build cost \(C_0\), monthly running cost \(C_{\text{run}}\), monthly volume \(V\) and per-frame saving \(\Delta = C_{\text{manual}} - C_{\text{auto}}\):

Worked example on the “good model” row, all figures illustrative: \(C_0 = £24{,}000\) for three engineer-months at £8,000 fully loaded, \(C_{\text{run}} = £1{,}500\), \(V = 20{,}000\) frames per month, and \(\Delta = 0.833 - 0.071 = £0.762\) per frame.

Sanity check the volume, because a reviewer will: 20,000 frames at 120 seconds each is 667 hours, about four full time people. If your programme has no such operation, scale down and expect a longer payback.

Three points of honesty do more for your credibility than any accuracy figure. Say whether the saving is cash or capacity: outsourced labelling produces cash finance can see in the ledger, while internal staff produce capacity, which is only real if those hours go somewhere valuable. Include the system you have not built: the model is a third of the work, and the review interface, queueing, auditing and pipeline integration are the rest. State the sensitivity: at 60 per cent acceptance, cost per frame rises to £0.113, monthly net falls to £12,904 and payback stretches to 1.9 months. A proposal that survives its own pessimistic case is far easier to approve.

Takeaway: Write the per-frame cost equation, measure the acceptance rate because it dominates everything, and compute the payback explicitly. Distinguish cash savings from freed capacity and show the pessimistic case; that is how the work gets approved.

Sim to Real, Replay and Failure Analysis

Domain randomisation and the calibration gap

Simulation is excellent for covering situations you cannot safely stage, and poor at being correct. Domain randomisation addresses the first by training over a distribution of worlds: textures, lighting, geometry, sensor noise, intrinsics and extrinsics. The item everyone forgets is to randomise the latency. Your simulator very likely delivers perception with zero delay and perfect timestamps, which makes it the least realistic thing in the setup with respect to everything else in this chapter, and a policy trained that way depends on information it will never have in time.

The calibration gap is quieter. In simulation the camera is exactly where the model says it is; on a real robot the extrinsics carry residual error and drift with temperature, vibration and the first knock to the mount. One degree of error at 3 metres is \(3000 \tan(1°) = 52\) mm of lateral displacement, enough to miss a grasp or put an obstacle on the wrong side of a boundary. Treat calibration as a monitored runtime quantity with an alarm, not a commissioning step, and report the sim-to-real gap as one number: the same metric on both domains, tracked over time.

Logging the inputs that produced a bad action

When the robot does something wrong you must be able to see exactly what it saw. Logging the output is nearly useless, because the output is the part you already know was wrong. Use a ring buffer of the last few seconds of raw inputs, flushed on a trigger: an intervention, an emergency stop, a planner failure, a deadline miss. Size it with arithmetic. Three cameras at 1920x1080, 10-bit Bayer, 30 fps:

A 10 second buffer is 2.3 GB of RAM, which most robot compute does not have spare, so the realistic answers are a hardware-encoded stream for the long window with a second or two of raw frames around the trigger, or reduced resolution and harder debugging.Keep raw Bayer for the short window rather than the debayered image: preprocessing is part of what you may need to debug, and a compressed RGB log has already destroyed the evidence of a demosaic or white balance problem. Compression artefacts also shift model outputs enough to make a replay diverge from the original run. Log alongside it whatever makes replay possible: timestamps, model version hashes, calibration, configuration and seeds. A clip you cannot replay deterministically is an anecdote.

Regression suites built from real failures

Replay harness. Feed a recorded clip through the exact production perception and planning code and get bit-identical outputs. That works only if you enforced the earlier rules: no wall-clock calls, timestamps from the data, fixed seeds, deterministic reductions. Non-determinism is not a philosophical inconvenience, it is what stops you ever proving you fixed something.

Promote failures to tests. Every clip gets assertions about behaviour, not tensors. Not “the logit for class 4 exceeds 0.8”, which breaks on every retrain, but “the obstacle is detected beyond 1.5 m” or “commanded speed reaches zero within 300 ms of the bumper event”. Behavioural assertions survive model changes, which is the entire point.

Gate releases on the suite. Pass rate over the accumulated corpus becomes the release criterion, with no fixed failure allowed to regress. That corpus grows monotonically and is the most valuable artefact the team owns, because it encodes every mistake the programme has already paid for. Track mean distance between interventions as the top-level metric; a programme that only tracks offline accuracy optimises a number uncorrelated with whether the robot is useful.

This is where pattern two pays off a second time: given thousands of failure clips, a VLM can cluster them and describe each cluster in a sentence, turning a week of scrubbing video into an afternoon of reading summaries. The output is not a better model. It is a faster loop, and in robotics the speed of the failure analysis loop sets the rate of progress.

Takeaway: Log the inputs, not the outputs; enforce enough determinism that clips replay bit-exactly; and turn every real failure into a behavioural assertion in a suite that gates releases. The corpus of past failures outlives every model you ship.

A Few Problems to Work

Problem 1. A heading controller has gain crossover at 6 Hz and 50 degrees of phase margin measured with no perception delay. Safety review requires at least 30 degrees in operation. (a) What total sense-to-act delay can you afford? (b) Your non-inference stages sum to 14 ms. What is the inference budget? (c) You must ship a model taking 9 ms. What crossover preserves 30 degrees?

Problem 2. Manual labelling takes 150 s per frame at £22 per hour. A VLM pipeline verifies in 5 s with 85 per cent acceptance, rejected proposals take a further 25 s to fix, and model compute is £0.003 per frame. Build cost £30,000, running cost £1,200 per month, volume 25,000 frames per month. (a) Cost per frame before and after, and the speedup. (b) Net monthly saving. (c) Payback period. (d) What if acceptance falls to 70 per cent, and what if volume were only 2,000 frames per month?

Problem 3. An edge device delivers 2 TFLOP/s int8 peak at 40 per cent achieved utilisation and 60 GB/s of bandwidth. You want a distilled VLM on the robot: a ViT-B/16 encoder at 224 px (197 tokens, 86M matmul parameters), image tokens pooled to 49, a 1.1B parameter int8 language model, a 20 token prompt and 24 output tokens. (a) Encoder time. (b) Prefill time. (c) Decode time. (d) Achievable rate, how many 30 Hz control periods that is, and which term to attack first.

What’s Next

Almost everything here depends on trustworthy numbers: the per-stage budget, the p99.9 rather than the mean, the worst case execution time the tier ladder rests on, the real utilisation of the accelerator. I have written them down as illustrative allocations, but on your system they are measurements, and measuring them correctly on a device that is thermally throttling, sharing memory bandwidth and running a scheduler you do not fully control is its own discipline.

That’s all for Part 9! For Part 10, on profiling and finding the real bottleneck, click here.