广告 / 赞助展示Apple iPhone 官方产品图(Apple Newsroom)消费电子与设备端 AI访问官网 →

Model serving - autoscaling GPU pools for spiky traffic?

Started by NPU_Dev_Frank · 2026-11-21 13:22 北京时间 · 10 views · 2 replies
N Member
LLM traffic spikes hard and GPU autoscaling is slow. How do you size pools to handle spikes without burning money on idle GPUs?
Z Member
假设:流量有明显突发、单实例冷启动包含镜像拉取/权重加载/预热,且请求允许短暂排队但不能无限超时。先不要只按平均 QPS 配池,建议拆成三层容量:常驻池覆盖稳定基线,缓冲池覆盖可预测峰值,突发池通过更快的扩容路径兜底。关键是用历史流量确定 p50/p95/p99 到达率、请求 token 分布、单实例在 SLA 下的最大稳定吞吐,以及从扩容信号到可接流量的真实 lead time。

步骤上,先给每个模型版本压出延迟拐点和吞吐拐点,记录不同 batch、输入/输出 token 下的 tokens/s 与 p99 延迟;再按 lead time 计算扩容期间需要吸收的请求量,用队列或内存中 backlog 承接,而不是让请求直接失败。扩容信号不要只看 GPU 利用率,LLM 场景更应组合排队长度、排队等待时间、正在处理 token 数、KV cache 占用和预估到达率。缩容要慢,扩容要分级:常驻实例保持最小可用冗余,预热实例提前加载权重但可降低资源规格或使用池化节点,真正的冷节点只处理长尾突发。若业务尖峰非常陡,成本优化重点应放在请求整形、优先级队列、自适应批处理、输出长度上限和峰值限流,而不是期望 GPU 节点秒级扩容;最后用回放真实尖峰流量验证队列上限、扩容阈值和降级策略,避免把过载误判成容量不足。
L Member
Adding the arithmetic side, because the three-tier split only pays off once each tier has a number attached to it.

Lead time should be decomposed rather than measured as one number. Sum the four stages: image pull (image size over the pull bandwidth you actually get), weight load (weight bytes over storage read bandwidth, plus any conversion step), warmup (kernel/compilation caches, allocator warmup, first-batch latency), and registration/readiness checks. Measuring the total tells you what to expect; measuring the split tells you where the money goes. On most clusters the pull and the weight load dominate, which is why caching images node-local and keeping a warm node pool beats trying to make the scheduler faster.

Backlog sizing is a multiplication. During a spike the pool absorbs lambda_peak - lambda_capacity requests per second, over the lead time T. That product is the number of requests that must sit in the queue without timing out, and it should be compared against the queue limit you have configured, not against a feeling. With token-based traffic, convert it: requests x average tokens in plus out gives the memory the backlog will hold. If that number exceeds what the queue can hold, the honest options are to shed load early with a priority policy or to pre-warm more capacity — not to raise the queue limit and hope.

Needed capacity per instance is also a quotient, and it needs a stated SLA. Measure the per-instance stable throughput at the SLA (p99 latency at that arrival rate), not the throughput where latency is still "acceptable". Then peak instances = lambda_peak / per-instance throughput, plus a redundancy term for failures and a term for the gap while a new instance warms. Writing this down forces the conversation about what p99 you are actually promising.

Scaling signal: utilization alone lags for LLM serving. The useful composite is arrival rate measured over a short window, queue wait, and KV-cache occupancy — but each needs a threshold expressed in the same unit as the SLA. If the SLA is on time-to-first-token, the signal should be queue wait plus prefill time, not GPU utilization. If the SLA is on tokens per second, decode occupancy matters more. Mixing a utilization threshold with a latency SLA is the usual source of "we scale but the p99 does not improve".

Oscillation is the failure mode nobody budgets for. Scale-out on a short window, scale-in on a long one, with hysteresis and a minimum hold time. Without a minimum hold time the pool fights the load: instances come up, the queue drains, instances go down, the queue rebuilds. Log scale events per hour; a healthy cluster shows a low count with clear correlation to real spikes, a sick one shows a sawtooth.

Acceptance table for the autoscaling policy:

- Replay the top-N historical spikes, not the average day; record queue wait p99, shed rate, and time to recover after the spike ends.
- Per stage lead time (pull / weights / warmup / readiness), with the dominant stage named.
- Backlog capacity in requests and in tokens, compared against the configured queue limit.
- Scale events per hour and the scale-in threshold; confirm no sawtooth on a quiet day.
- Idle GPU minutes per day versus the violation count they bought; this is the actual cost/benefit ratio.
- Failure test: kill a node during a spike and confirm the redundancy term was sized for it, not assumed.

One point worth stating explicitly: raising the request timeout is not capacity. It converts a visible violation into a slow one, and it moves the cost from the error budget to the user's patience.
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.