Adding the arithmetic side, because the three-tier split only pays off once each tier has a number attached to it.
Lead time should be decomposed rather than measured as one number. Sum the four stages: image pull (image size over the pull bandwidth you actually get), weight load (weight bytes over storage read bandwidth, plus any conversion step), warmup (kernel/compilation caches, allocator warmup, first-batch latency), and registration/readiness checks. Measuring the total tells you what to expect; measuring the split tells you where the money goes. On most clusters the pull and the weight load dominate, which is why caching images node-local and keeping a warm node pool beats trying to make the scheduler faster.
Backlog sizing is a multiplication. During a spike the pool absorbs lambda_peak - lambda_capacity requests per second, over the lead time T. That product is the number of requests that must sit in the queue without timing out, and it should be compared against the queue limit you have configured, not against a feeling. With token-based traffic, convert it: requests x average tokens in plus out gives the memory the backlog will hold. If that number exceeds what the queue can hold, the honest options are to shed load early with a priority policy or to pre-warm more capacity — not to raise the queue limit and hope.
Needed capacity per instance is also a quotient, and it needs a stated SLA. Measure the per-instance stable throughput at the SLA (p99 latency at that arrival rate), not the throughput where latency is still "acceptable". Then peak instances = lambda_peak / per-instance throughput, plus a redundancy term for failures and a term for the gap while a new instance warms. Writing this down forces the conversation about what p99 you are actually promising.
Scaling signal: utilization alone lags for LLM serving. The useful composite is arrival rate measured over a short window, queue wait, and KV-cache occupancy — but each needs a threshold expressed in the same unit as the SLA. If the SLA is on time-to-first-token, the signal should be queue wait plus prefill time, not GPU utilization. If the SLA is on tokens per second, decode occupancy matters more. Mixing a utilization threshold with a latency SLA is the usual source of "we scale but the p99 does not improve".
Oscillation is the failure mode nobody budgets for. Scale-out on a short window, scale-in on a long one, with hysteresis and a minimum hold time. Without a minimum hold time the pool fights the load: instances come up, the queue drains, instances go down, the queue rebuilds. Log scale events per hour; a healthy cluster shows a low count with clear correlation to real spikes, a sick one shows a sawtooth.
Acceptance table for the autoscaling policy:
- Replay the top-N historical spikes, not the average day; record queue wait p99, shed rate, and time to recover after the spike ends.
- Per stage lead time (pull / weights / warmup / readiness), with the dominant stage named.
- Backlog capacity in requests and in tokens, compared against the configured queue limit.
- Scale events per hour and the scale-in threshold; confirm no sawtooth on a quiet day.
- Idle GPU minutes per day versus the violation count they bought; this is the actual cost/benefit ratio.
- Failure test: kill a node during a spike and confirm the redundancy term was sized for it, not assumed.
One point worth stating explicitly: raising the request timeout is not capacity. It converts a visible violation into a slow one, and it moves the cost from the error budget to the user's patience.