Part 11 of How to Make Your Model Fast (Part 10: Profiling | Part 12: Agents)
A model that is fast on a benchmark still has to survive a load balancer, a queue, a container registry and a business. This chapter works the arithmetic that turns a latency objective into a machine count, shows why utilisation above about seventy percent destroys your tail, and treats batching as the trade between throughput and p99. It then covers placement across device, edge and cloud, packaging and versioning, batch pipelines against online services, autoscaling on a signal that is not CPU, ML-specific observability and drift. It ends with the thing that should come first: pricing your model's two kinds of error and choosing an operating point from that price.
Every chapter so far has been about making one inference cheap. This one is about what happens when ten thousand people want that inference at once, at three in the morning, while somebody merges a minor version bump to a kernel library.
Three questions. How many machines does a latency objective actually cost, and why can you not run them all at ninety five percent busy? Where should the model physically live? And, last, the question that should come first: what is a wrong prediction worth in pounds, and what operating point does that price imply? A model with 94 percent accuracy is not a fact about the world. It is a fact about a threshold somebody chose, usually without doing the sum.
Four numbers: peak request rate \(\lambda\), per-replica service rate \(\mu\), target utilisation \(\rho\), and a latency objective. The machine count is the boring part:
\[N = \left\lceil \frac{\lambda}{\rho\,\mu} \right\rceil\]The interesting part is that \(\rho\) is not a free parameter you set to 0.95 because idle hardware offends you. It is determined by the SLO, and the relationship is violently non-linear.
The running example is a detector service: a 640x640 int8 object detector of the kind built in Part 7. Profiling gives a clean batch-latency line,
\[T(B) = T_0 + t\,B = 4\ \text{ms} + 0.6\ \text{ms} \times B\]where \(T_0\) is the fixed cost of launching the graph and streaming the weights and \(t\) is the marginal per-image work. The figures are illustrative; the shape is what you measure once you have done the work in Part 3. At a maximum batch of 16 a replica finishes a batch in 13.6 ms, so \(\mu = 1176\) per second and the mean service time is \(S = 0.85\) ms.
Treat one replica as a single server with Poisson arrivals and exponential service. It is a caricature, but the right one: time in the system is then exponentially distributed with rate \(\mu - \lambda\), so
\[\mathbb{E}[T] = \frac{S}{1-\rho}, \qquad T_{p} = \frac{S}{1-\rho}\,\ln\!\left(\frac{1}{1-p}\right)\]so the p99 is \(\ln(100) \approx 4.605\) times the mean.
Sojourn time as a multiple of service time, and in milliseconds for the detector at \(S = 0.85\) ms. The last column is the share of a 120 ms model-tier budget spent on queueing and service alone.
| Utilisation \(\rho\) | Mean (\(\times S\)) | p99 (\(\times S\)) | p99 at \(S = 0.85\) ms | Budget used |
|---|---|---|---|---|
| 0.50 | 2.0 | 9.2 | 7.8 ms | 7% |
| 0.70 | 3.3 | 15.4 | 13.0 ms | 11% |
| 0.80 | 5.0 | 23.0 | 19.6 ms | 16% |
| 0.90 | 10.0 | 46.1 | 39.1 ms | 33% |
| 0.95 | 20.0 | 92.1 | 78.3 ms | 65% |
| 0.98 | 50.0 | 230.3 | 195.7 ms | 163% |
| 0.99 | 100.0 | 460.5 | 391.4 ms | 326% |
Between \(\rho = 0.7\) and 0.9 you pay a factor of three in tail latency for a 22 percent saving in machines; between 0.9 and 0.98, a factor of five for an 8 percent saving. The \(1/(1-\rho)\) term does not care about your budget review.
Real arrivals are burstier than Poisson. Kingman’s approximation generalises the queueing wait to
\[W_q \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right) S\]with \(c_a\) and \(c_s\) the coefficients of variation of inter-arrival and service times; M/M/1 is the case \(c_a^2 = c_s^2 = 1\). Traffic driven by human behaviour, or by a retry storm, easily reaches \(c_a^2 = 3\) and doubles the queueing wait at every utilisation.
Peak load 20,000 requests per second, p99 SLO 150 ms end to end, with 30 ms of measured non-model overhead (TLS, load balancer hop, decode, serialisation), leaving 120 ms for the model tier.
At \(\rho = 0.95\): \(N = \lceil 20000/(0.95 \times 1176) \rceil = 18\) replicas, actual \(\rho = 0.944\), p99 sojourn 70 ms. It fits, and the finance spreadsheet is happy. At \(\rho = 0.70\): \(N = 25\), actual \(\rho = 0.680\), p99 sojourn 12 ms. Seven more machines, 39 percent more money.
Both meet the SLO on a good day, which is why the argument is hard to win. The difference shows up on a bad day.
The two fleets under a lost node and under a spike. “Unstable” means \(\rho \geq 1\): the queue grows without bound and latency is capped only by your timeout.
| Scenario | 18 replicas | 25 replicas |
|---|---|---|
| Nominal 20,000 rps | \(\rho = 0.944\), p99 70 ms | \(\rho = 0.680\), p99 12 ms |
| Lose 1 replica | \(\rho = 1.000\), unstable | \(\rho = 0.709\), p99 13 ms |
| Lose 2 replicas | \(\rho = 1.063\), unstable | \(\rho = 0.739\), p99 15 ms |
| Spike to 25,000 rps | \(\rho = 1.181\), unstable | \(\rho = 0.850\), p99 26 ms |
| Rolling deploy, 20% drained | \(\rho = 1.181\), unstable | \(\rho = 0.850\), p99 26 ms |
The 18-replica fleet cannot survive one node failure, a routine rolling deployment, or a modest spike. It is not running at 95 percent utilisation; it is running at 100 percent with a 5 percent apology. The seven extra machines are the failure budget, the deployment budget and the spike budget, and they are almost always cheaper than the incident they prevent.
Takeaway: Tail latency scales as \(1/(1-\rho)\), so utilisation is a latency decision, not a cost decision. Plan for about 0.7, because the headroom you think you are wasting is what absorbs node failures, rolling deploys and spikes.
Batching is the biggest server-side lever, and the one that most directly trades throughput against tail latency. With \(T(B) = T_0 + tB\) throughput is \(B/T(B)\), rising towards the asymptote \(1/t\).
Throughput and per-batch latency for the detector, \(T_0 = 4\) ms, \(t = 0.6\) ms, per replica.
| Batch \(B\) | \(T(B)\) | Throughput | Speedup vs B=1 | Relative cost per inference |
|---|---|---|---|---|
| 1 | 4.6 ms | 217/s | 1.00x | 1.00 |
| 2 | 5.2 ms | 385/s | 1.77x | 0.56 |
| 4 | 6.4 ms | 625/s | 2.88x | 0.35 |
| 8 | 8.8 ms | 909/s | 4.18x | 0.24 |
| 16 | 13.6 ms | 1176/s | 5.41x | 0.18 |
| 32 | 23.2 ms | 1379/s | 6.35x | 0.16 |
| 64 | 42.4 ms | 1509/s | 6.94x | 0.14 |
Batch 1 to 16 cuts cost per inference by 82 percent. Batch 16 to 64 cuts it by a further 22 percent and triples the time a request spends inside the model. That is the whole trade, in two sentences.
A dynamic batcher does not run at its configured maximum. It runs at the size arrivals produce: while one batch executes, requests accumulate, and the next batch is whatever accumulated. In steady state that is self-consistent:
\[B^{*} = \lambda_r\,T(B^{*}) = \lambda_r (T_0 + t B^{*}) \quad \Longrightarrow \quad B^{*} = \frac{\lambda_r T_0}{1 - \lambda_r t}\]with \(\lambda_r\) the per-replica arrival rate. Note the pole at \(\lambda_r = 1/t\): the same utilisation cliff in different clothing.
For the 25-replica fleet \(\lambda_r = 0.8\) per ms, so \(B^{*} = 3.2/0.52 = 6.2\) and the model takes 7.7 ms. For the 18-replica fleet \(\lambda_r = 1.111\) per ms, so \(B^{*} = 4.44/0.333 = 13.3\) and the model takes 12 ms. The batcher self-tunes: heavier load makes bigger batches, which make more throughput. That is why a fleet degrades gracefully right up until it does not.
The maximum queue delay is how long the scheduler waits for more requests before firing a partial batch.dynamic_batching { max_queue_delay_microseconds } alongside preferred_batch_size; TensorFlow Serving calls it batch_timeout_micros. Same semantics: an upper bound on how long the first request in a forming batch may be made to wait.
p99 budget 150 ms
minus non-model overhead 30 ms
minus queueing p99 at rho = 0.7 12 ms
minus compute at max batch 16 13.6 ms
= headroom available for waiting 94.4 ms
So you could afford 90 ms. You should not use it. The timeout only binds when arrivals are too slow to fill the batch, and the fill time is \(B_{max}/\lambda_r = 16/0.8 = 20\) ms. Anything above 20 ms buys nothing at nominal load and costs everything at 3 am, when \(\lambda_r\) falls to 20 per second, 16 requests take 800 ms to arrive, and every overnight request pays the full timeout to be batched with one other.
The rule I use: set the timeout to the smaller of the leftover budget and the fill time at your lowest normal load, then fix the overnight case by scaling replicas down rather than waiting longer. Scaling down raises \(\lambda_r\), which refills batches. Batch efficiency and autoscaling are one knob seen from two ends.
Takeaway: Dynamic batch size is set by load, not configuration: \(B^{*} = \lambda_r T_0 / (1 - \lambda_r t)\). The maximum batch caps the tail; the maximum wait should be the smaller of your leftover latency budget and the batch fill time.
Placement options and what each costs you. Latency figures are typical network round trips, not compute, and all figures are illustrative orders of magnitude.
| Property | On device | Edge (site or PoP) | Cloud region | Split |
|---|---|---|---|---|
| Network latency added | 0 ms | 1 to 10 ms | 20 to 100 ms | 0 ms for the filtered majority |
| Marginal cost per inference | ~0, hardware is sunk | low, fixed capacity | metered, the dominant line item | a small fraction of full cloud |
| Compute available | tens of GOPS to a few TOPS | tens of TOPS | effectively unbounded | tiered |
| Privacy | best, data never leaves | good, stays on site | weakest, raw data egresses | good, only escalations leave |
| Offline capability | full | partial, local network only | none | degraded but functional |
| Update velocity | weeks, fleet OTA or app review | days | minutes | minutes for the cloud half |
| Observability | poor, you see what you ship | moderate | complete | complete on escalations only |
| Failure blast radius | one device | one site | everyone | partial |
The row that settles most arguments is update velocity against observability. On-device inference is cheap and private and leaves you nearly blind: you learn about a regression from a support ticket six weeks later. Cloud inference costs real money per call and lets you canary in minutes and roll back in seconds. Most systems that survive end up split, because the halves have complementary weaknesses.
A small model on the device looks at everything and decides what to escalate; a large cloud model handles the escalations. That is a gate, and gates have arithmetic. Let \(\Lambda\) be inputs per day, \(C\) the cloud cost per inference, \(p\) the fraction forwarded, \(c_d\) the daily device cost of the gate, \(E\) the genuine events per day, \(R\) the gate’s recall on them and \(L\) the loss per miss. The gate pays when
\[\underbrace{C \Lambda (1-p) - c_d}_{\text{saved}} \;>\; \underbrace{(1-R)\,E\,L}_{\text{cost of misses}}\]Take a camera estate: 2,000 cameras at 5 analysed frames per second, so \(\Lambda = 864\) million frames a day. A cloud vision-language call at an illustrative £0.0015 per frame makes the all-cloud bill £1,296,000 a day, which settles that question. An on-device detector forwarding \(p = 0.4\%\) cuts it to £5,184, and the gate itself draws perhaps 1.5 W extra per camera, 72 kWh a day, about £18. Savings: roughly £1,290,800 a day. Rearranging for the required recall,
\[R > 1 - \frac{C \Lambda (1-p) - c_d}{E\,L}\]Required gate recall against the price of a miss, holding savings at £1,290,800 a day and 2,000 events a day.
| Loss per missed event \(L\) | \(E \cdot L\) per day | Required recall |
|---|---|---|
| £50 | £100,000 | any recall pays |
| £500 | £1,000,000 | any recall pays |
| £5,000 | £10,000,000 | 0.871 |
| £50,000 | £100,000,000 | 0.987 |
| £500,000 | £1,000,000,000 | 0.9987 |
That table is the lesson. When misses are cheap the cost argument is so slack that nobody needs to think. When misses are expensive the cloud bill stops mattering entirely and the only question is the gate’s recall, measured on hard cases and not on a convenient test set. I have watched a team spend a month pushing the escalation rate from 0.5 to 0.3 percent while the gate’s recall on night-time frames sat unmeasured.
Takeaway: In a split system, cost decides whether you need a gate; recall decides whether the gate is acceptable. Compute the recall your loss-per-miss demands before tuning the escalation rate.
A served model is not a file. It is a tuple: the weights, the preprocessing (resize, letterbox pad value, channel order, mean and standard deviation, colour space), the postprocessing (NMS thresholds, anchor decoding, top-k), the label map, and the runtime and kernel library versions. Ship any one of those independently and you will eventually ship an outage. The classic failures are a new model paired with the previous normalisation constants, which quietly costs several accuracy points and looks like drift, and a retrained model whose label map gained a class in alphabetical order, so every subsequent class is off by one and the confusion matrix looks fine because it used the new map.
Pinning the runtime matters for a less obvious reason: a minor version bump in a kernel library can change numerics. Not because it is buggy, but because a new heuristic picks a different tiling, which changes the order of floating-point accumulation, which changes the last bits, which changes the argmax on near-ties.
# Pin by digest, not by tag. Tags move; digests do not.
FROM nvcr.io/nvidia/tritonserver@sha256:9f3c1d... AS runtime
# Pin every layer of the numerics stack explicitly.
RUN pip install --no-deps --require-hashes -r /tmp/requirements.lock
COPY manifest.json /srv/manifest.json
ENV MODEL_URI=s3://models/detector/sha256-4a7b9e2c.../
Model artefacts belong in object storage under content-addressed keys, never a mutable name like latest.onnx. The digest is the version; a human-readable tag is a pointer file containing a digest, so promoting a model means writing a new pointer and rolling back means writing the old one.
{
"model_version": "detector-2026.02.17",
"weights_sha256": "4a7b9e2c8d1f...",
"preprocess": {
"resize": "letterbox", "size": [640, 640], "pad_value": 114,
"channel_order": "RGB",
"mean": [0.485, 0.456, 0.406], "std": [0.229, 0.224, 0.225]
},
"postprocess": { "score_threshold": 0.0148, "nms_iou": 0.65 },
"labels_sha256": "e1d5a90b77c3...",
"runtime": { "onnxruntime": "1.20.1", "cuda": "12.4.1", "cudnn": "9.1.0" }
}
The service asserts this manifest at startup, before accepting traffic: hash the weights it just downloaded, hash the label map, compare, refuse to become ready on any mismatch. That check has caught more of my mistakes than any test suite, because it catches the one tests cannot see: the right code with the wrong artefact.
At the University of Leicester I built automated pipelines that processed high-resolution satellite imagery in near real time, on Docker and Kubernetes. It is a useful case study because almost nothing about it resembles the service above.
A scene is 30,000 x 30,000 pixels, four bands, 16-bit, so 7.2 GB raw. The segmentation model takes 512 x 512 tiles with 64 pixels of overlap so that objects at tile boundaries get context, giving a stride of 448 and \(\lceil (30000-512)/448 \rceil + 1 = 67\) tiles per side, 4,489 tiles per scene. At an amortised 6 ms per tile that is 26.9 seconds of accelerator time.
That is not the bottleneck. Reading 7.2 GB at 800 MB/s takes 9 seconds, but decoding a compressed multi-band raster is CPU work and can take 40 seconds on eight cores, so the accelerator idles for a third of the wall clock while a general-purpose CPU inflates pixels. In batch pipelines the accelerator is very often the cheap part, and the discipline of Part 10 applies to the whole pipeline, not just the model.
Cost per scene under three configurations. Illustrative prices: £1.80 per accelerator-hour, £0.30 per 8-vCPU-hour, 70 percent spot discount.
| Configuration | Accelerator utilisation | Accelerator cost | CPU cost | Cost per scene |
|---|---|---|---|---|
| One fat pod, decode and infer together | 0.60 | £0.0225 | included | £0.0225 |
| Decode pool feeding a shared inference service | 0.90 | £0.0150 | £0.0033 | £0.0183 |
| As above, on spot instances | 0.90 | £0.0045 | £0.0010 | £0.0055 |
At 5,000 scenes a day that is £41,000 a year against £10,000. The ratio is the point, not the absolute figures. Three properties make the spot row possible at all.
Idempotent stages. Each stage is keyed by a hash of scene identifier, stage name, code version and parameters, writes to a content-addressed path, and is a no-op if its completion marker exists. Retrying the whole pipeline then costs nothing and is always safe, so the retry policy can be aggressive and stupid, which is what you want at 3 am.
def stage(name, version):
def deco(fn):
def wrapped(scene_id, **params):
key = sha256(f"{scene_id}|{name}|{version}|{canonical(params)}")
marker = f"{PREFIX}/{key}/_COMPLETE"
if store.exists(marker):
return f"{PREFIX}/{key}/"
tmp = f"{PREFIX}/{key}.partial-{uuid4()}/"
fn(scene_id, out=tmp, **params)
store.rename(tmp, f"{PREFIX}/{key}/") # atomic publish
store.put(marker, b"")
return f"{PREFIX}/{key}/"
return wrapped
return deco
Checkpoint granularity below the interruption interval. Spot capacity is reclaimed with about 30 seconds of notice. A 45-second unit of work loses half a scene on average; a 2-hour mosaic job loses the job. Keep the checkpoint unit an order of magnitude smaller than the mean time between interruptions.
Parallelism by tile shard, not by scene. Sharding by scene gives coarse, unbalanced units and a long tail when one scene is cloudier. Sharding by tile range balances load and lets a scene finish faster than one pod can process it, which is what “near real time” requires.
How the two pipeline shapes differ, and why a team good at one is usually bad at the other.
| Property | Batch pipeline | Online service |
|---|---|---|
| Objective | cost per scene | p99 latency under an SLO |
| Target utilisation | as near 1.0 as the scheduler allows | about 0.7 |
| Batch size | as large as memory permits | bounded by the latency budget |
| Instance type | spot or preemptible | on-demand or reserved |
| Failure response | retry the idempotent stage | fail fast, client retries |
| Cold start | amortised over hours, ignore it | on the critical path |
| Scaling signal | backlog of work remaining | in-flight requests per replica |
| Worst failure mode | silently reprocessing nothing | queue collapse at \(\rho \to 1\) |
Scaling an accelerator-bound service on CPU utilisation is the commonest autoscaling mistake I see, and it fails both ways. If the host thread blocks on a synchronisation primitive while the accelerator works, you see 5 percent CPU on a saturated replica and never scale up. If the runtime busy-polls, you see 100 percent CPU on an idle replica and scale up for nothing. Accelerator utilisation counters are little better: most report the fraction of time any kernel was resident, so a kernel using three percent of the compute units reads as 100 percent busy.
Scale on the quantity that predicts SLO violation: concurrency. Load test to find \(C^{*}\), the in-flight requests per replica at which p99 first crosses the objective, and target a fraction of it:
\[N_{\text{desired}} = \left\lceil \frac{\text{in-flight requests}}{0.7 \times C^{*}} \right\rceil\]Queue depth per replica and mean batch wait work equally well. All three are causally upstream of the SLO; CPU percentage is not.
Cold start is where large models embarrass autoscalers: pull 18 GB of image layers at 400 MB/s on a cold node (45 s, or 5 s if cached), download 14 GB of weights at 1.5 GB/s (9 s), initialise the runtime and capture or autotune the graph (30 s), run warm-up inferences (5 s). Call it 50 to 90 seconds warm and 90 to 150 cold. Add a 30-second metric window and you cannot react to a two-minute ramp, only anticipate it.
A warm pool is that anticipation, and it has arithmetic of its own. Seventeen idle replicas at £1.80 an hour is £734 a day, roughly £22,000 a month, to insure against an incident costing a few thousand pounds, so permanent full warm pools are usually wrong. What works, in order of cost effectiveness: scheduled pre-warming against known traffic shapes, since most load is boringly diurnal; a small standing pool sized to the ramp you can absorb in one metric window; and a cheap fallback, typically the quantised small model from Part 5, serving degraded results during the cold window. Losing two accuracy points for 90 seconds beats shedding a million requests.
Takeaway: Autoscale on in-flight requests per replica against a load-tested saturation concurrency, never on CPU. Cold start for a large model is one to three minutes, so anticipate load with pre-warming and a degraded fallback rather than reacting to it.
Rate, errors and duration tell you the server is alive. They say nothing about whether the model is right. For that you need the input distribution, the output distribution, the confidence histogram and whatever ground truth you can get.
You cannot store everything: the camera estate produces 864 million frames a day at 200 kB each, 173 TB a day. Log metadata exhaustively and raw inputs selectively.
A sampling plan for the camera estate. “Near-threshold” is the confidence band where a small perturbation flips the decision.
| Stream | Volume per day | Bytes each | Daily storage |
|---|---|---|---|
| Everything (reference) | 864M | 200 kB | 173 TB |
| Structured metadata, 100% | 864M | 60 B | 52 GB |
| Uniform raw sample, 0.01% | 86,400 | 200 kB | 17 GB |
| Near-threshold raw, 20% of a 0.2% band | 345,600 | 200 kB | 69 GB |
| Errors and user reports, 100% | ~5,000 | 200 kB | 1 GB |
| Total retained | ~139 GB |
That is 0.08 percent of the raw volume, and it supports everything you actually do: metadata gives unbiased population statistics and the confidence histogram, the uniform sample reconstructs the input distribution, the near-threshold sample is your next training batch, and the error and complaint streams are how you find the failure mode you did not anticipate.
Store the sampling probability \(\pi_i\) with each record and estimate population quantities by inverse-probability weighting, \(\hat{\theta} = \frac{1}{\Lambda}\sum_i g(x_i)/\pi_i\).
For input change, the population stability index over binned features is the workhorse,
\[\text{PSI} = \sum_i (a_i - e_i)\,\ln\!\frac{a_i}{e_i}\]comparing actual bin proportions \(a_i\) against a reference window \(e_i\). Below 0.1 is conventionally stable, 0.1 to 0.25 deserves attention, above 0.25 warrants investigation.
Covariate shift is a change in \(P(x)\) with \(P(y \mid x)\) unchanged: a new sensor, a new customer segment, a new season. The monitors above detect it without any labels, and it is often fixable without retraining, because recalibrating the threshold against a small freshly labelled sample recovers much of the loss for almost nothing.
Concept shift is a change in \(P(y \mid x)\): the same input now has a different correct answer. Fraud tactics adapt, spam adapts, “relevant” moves when the product changes. It is invisible to input monitoring by construction, so detecting it needs labels: delayed ground truth, human review of a stratified sample, or a proxy outcome such as a click, a return or a complaint.
Label latency governs the whole loop. If chargebacks arrive 45 days after a transaction, your detector runs 45 days behind reality and no engineering makes it faster. A noisy proxy at 2 days beats a clean signal at 45.
Whether to retrain is a payback calculation. Say error cost has drifted from £185,500 to £232,000 per million units on 2.5 million units a month, so excess cost is £116,250 a month. A retrain costs labelling 20,000 samples at £0.40 (£8,000), 300 accelerator-hours at £1.80 (£540), ten engineer-days (£6,000) and a two-week shadow and canary (£3,000): about £17,500. Payback is four and a half days, so retrain. Run the same sum at £2,000 a month of excess cost, find a payback of nine months, and do not: retune the threshold, which is free.
Shadow mirrors live traffic to the new model while still serving the old one’s output. It costs a second copy of the fleet for the window and catches, at zero user risk, crashes, memory growth, latency regressions, shape surprises and mismatches between your evaluation set and reality. It cannot catch anything with a feedback loop, since shadow predictions never influence behaviour.
Canary serves the new model to a small traffic fraction behind automated guardrails on latency, error rate and at least one business metric, ramping 1 to 5 to 25 to 100 percent. The trap is statistical power: a two-proportion comparison at 80 percent power and 5 percent significance needs roughly \(n \approx 16\,p(1-p)/\delta^2\) per arm. At a 2 percent base rate and a 10 percent relative effect (\(\delta = 0.002\)) that is 78,400 per arm, about 7 minutes at a 1 percent canary on 20,000 rps. Tighten to a 1 percent relative effect and it is 7.84 million per arm, nearly 11 hours.
Rollback is the step people skip. It must be one configuration change, must not need a rebuild, the previous artefact must still be warm somewhere, and somebody must have run the procedure in the last quarter. Keep version N-1 in the warm pool throughout a rollout. A rollback that needs a cold start is not a rollback, it is an outage with a plan attached.
Takeaway: Monitor inputs and outputs, not just latency and errors, and sample raw data by stratum with the sampling probability stored alongside. Covariate shift needs no labels and often yields to a threshold retune; concept shift needs labels, and label latency is the hard floor on your loop.
All of the above exists to serve predictions, and whether it was worth building depends on a quantity most teams never write down: the price of each kind of error.
Let \(C_{FP}\) be the cost of a false positive and \(C_{FN}\) that of a false negative, correct decisions free. For a calibrated probability \(p = P(y=1 \mid x)\), predicting positive costs \((1-p)C_{FP}\) in expectation and predicting negative costs \(p\,C_{FN}\). Predict positive when the first is smaller:
\[p > \frac{C_{FP}}{C_{FP} + C_{FN}}\]That is the entire theory. The threshold is a ratio of two business numbers and contains no machine learning at all.
Defect detection on a production line: prevalence 0.5 percent, a shipped defect costs £400 in warranty and returns, pulling a good unit for manual inspection costs £6 of inspector time. The threshold is \(6/(6+400) = 0.0148\), so flag anything with more than a 1.5 percent chance of being defective. That feels absurdly aggressive until you price it. Take two points on the precision-recall curve over a million units containing 5,000 defects, with \(TP = 5000R\), \(FN = 5000(1-R)\) and \(FP = 5000R(1-P)/P\).
Expected cost per million units at two operating points, plus the trivial baselines. \(C_{FN} = £400\), \(C_{FP} = £6\).
| Policy | TP | FN | FP | Miss cost | Inspection cost | Total |
|---|---|---|---|---|---|---|
| Ship everything | 0 | 5,000 | 0 | £2,000,000 | £0 | £2,000,000 |
| Inspect everything | 5,000 | 0 | 995,000 | £0 | £6,000,000 | £6,000,000 |
| Model A: P=0.80, R=0.60 | 3,000 | 2,000 | 750 | £800,000 | £4,500 | £804,500 |
| Model B: P=0.25, R=0.95 | 4,750 | 250 | 14,250 | £100,000 | £85,500 | £185,500 |
Model B, at a precision of 0.25, beats model A by £619,000 per million units. Three of every four things it flags are fine and it is still correct by a wide margin, because the cost ratio is 67 to 1 and recall is what you are buying. If you have ever sat in a review where a model was rejected because “precision is only 25 percent”, this table is the reply.
Two caveats turn it from a slide into a decision. Capacity: model B needs 19,000 inspections per million units, and if the line can only do 8,000 you must slide back along the curve to where \(5000R/P \leq 8000\). Volume: these are per-decision costs, so at 30 million units a year the A-to-B gap is £18.6 million, which buys a great deal of labelling.
So put the currency figure on the dashboard beside the F1 score, at the same freshness. Once error cost is denominated in pounds, threshold tuning stops being an argument about taste, retraining becomes a payback calculation, and the split gate’s required recall falls out of the same sum.
Takeaway: The decision threshold is \(C_{FP}/(C_{FP}+C_{FN})\), a ratio of business costs, not a model hyperparameter. Price both error types, compute expected cost per million decisions, and let that pick your operating point.
1. Fleet sizing under a spike. A speech model replica has \(T(B) = 12\ \text{ms} + 2.5\ \text{ms} \times B\) at a maximum batch of 8. Peak traffic is 9,000 requests per second, the p99 SLO is 400 ms end to end and non-model overhead is 60 ms. How many replicas at \(\rho = 0.7\), what is the p99 sojourn, and can that fleet absorb a 30 percent spike while losing one replica?
\(T(8) = 12 + 20 = 32\) ms, so \(\mu = 8/0.032 = 250\) per second and \(S = 4\) ms.
\(N = \lceil 9000/(0.7 \times 250) \rceil = 52\), actual \(\rho = 9000/13000 = 0.692\).
p99 sojourn \(= \frac{4}{0.308} \times 4.605 = 59.8\) ms against a model-tier budget of \(400 - 60 = 340\) ms, so 18 percent of budget.
Spike plus failure: 11,700 rps on 51 replicas of capacity 12,750, so \(\rho = 0.918\) and p99 \(= \frac{4}{0.0824} \times 4.605 = 223.7\) ms. Inside 340 ms, but nothing is left for a second failure, and if bursty arrivals push \(c_a^2\) to 3 then Kingman roughly doubles the queueing term and you are over budget.
2. The batch you actually get. For the same replica, what is the equilibrium batch size at the 52-replica fleet, and what maximum queue delay should you configure given that overnight traffic falls to 400 requests per second across the whole fleet?
\(\lambda_r = 9000/52 = 173.1\) per second \(= 0.1731\) per ms.
\[B^{*} = \frac{0.1731 \times 12}{1 - 0.1731 \times 2.5} = \frac{2.077}{0.5673} = 3.66\]Check: \(T(3.66) = 21.15\) ms and \(3.66/21.15 = 0.173\) per ms. Consistent. The batcher naturally runs at about 3.7 against a maximum of 8, so the maximum only binds during spikes.
Fill time for a batch of 8 is \(8/0.1731 = 46\) ms, and the leftover budget is \(340 - 60 - 32 = 248\) ms, so the budget is not the constraint. Take the smaller: about 45 ms.
Overnight, if the fleet does not scale down, \(\lambda_r = 7.7\) per second, a batch of 8 takes over a second to fill, and every request pays the full 45 ms to be batched with roughly one other. Scale to about \(400/(0.7 \times 250) = 3\) replicas instead, restoring \(\lambda_r\) to 133 per second and refilling the batches.
3. Does the gate pay? A document product runs a cloud model at £0.004 per page over 12 million pages a day. An on-device classifier forwards the 6 percent of pages it believes contain a signature block, and costs £400 a day in extra fleet compute. There are 90,000 signature blocks a day, classifier recall is 0.94, and a miss costs £3 of manual handling. Does the gate pay? What if a miss instead triggers a compliance incident costing £2,500?
Savings: \(0.004 \times 12{,}000{,}000 \times 0.94 - 400 = 45{,}120 - 400 = £44{,}720\) a day.
Misses at £3: \(0.06 \times 90{,}000 \times 3 = 5{,}400 \times 3 = £16{,}200\) a day. Net benefit £28,520 a day, so the gate pays by a factor of 2.8.
At £2,500 per miss: \(5{,}400 \times 2{,}500 = £13{,}500{,}000\) a day. The gate loses catastrophically. Required recall:
\[R > 1 - \frac{44{,}720}{90{,}000 \times 2{,}500} = 1 - 0.000199 = 0.9998\]99.98 percent recall is not realistic for a small on-device classifier on documents. At that loss per miss the correct architecture is to send everything to the cloud model and spend the £48,000 a day, or to use the gate only to route between cloud tiers so no page goes unexamined.
4. An operating point under a capacity constraint. A moderation system reviews 4 million items a month at a 0.8 percent violation rate. A false negative costs £150 in expected harm and remediation, a false positive costs £1.20 of reviewer time, and review capacity is 60,000 items a month. The available points are (A) P=0.60, R=0.55; (B) P=0.30, R=0.80; (C) P=0.12, R=0.93. Which do you choose?
Violations per month: \(4{,}000{,}000 \times 0.008 = 32{,}000\). For each point \(TP = 32{,}000R\), \(FN = 32{,}000(1-R)\), \(FP = TP(1-P)/P\) and review load \(= TP + FP\).
A: \(TP = 17{,}600\), \(FN = 14{,}400\), \(FP = 11{,}733\), load 29,333. Cost \(= 2{,}160{,}000 + 14{,}080 = £2{,}174{,}080\).
B: \(TP = 25{,}600\), \(FN = 6{,}400\), \(FP = 59{,}733\), load 85,333. Cost \(= 960{,}000 + 71{,}680 = £1{,}031{,}680\).
C: \(TP = 29{,}760\), \(FN = 2{,}240\), \(FP = 218{,}240\), load 248,000. Cost \(= 336{,}000 + 261{,}888 = £597{,}888\).
On expected cost C wins, and the unconstrained threshold \(1.20/(1.20+150) = 0.0079\) agrees. But C needs 248,000 reviews and B needs 85,333 against a capacity of 60,000, so only A fits today.
The useful output is not “choose A”, it is the price of capacity. A to B costs 56,000 extra reviews and saves £1,142,400: £20.40 of avoided harm per extra review, against £1.20 of reviewer cost. B to C costs a further 162,667 reviews and saves £433,792, or £2.67 per review, still above £1.20. Every review seat is worth several times its cost, so choose A today, present the £20.40 figure, and go and buy or automate more capacity.
Serving one model is tractable: one queue, one batch size, one threshold, and arithmetic that closes. Agent systems break every one of those assumptions. A single request becomes a chain of model calls whose length you cannot predict, latency compounds along the chain instead of being amortised across a batch, batching fights with sequential dependencies, and cost per request has a heavy tail because one query decided to call the retriever eleven times. The queueing intuition here still holds, but you have to apply it to a graph.