F
Member
【Editorial Discussion】
Sharing a GPU with MIG, time-slicing, or a process-level scheduler is a reasonable way to raise utilisation. It is also a way to import a neighbor’s batch or a neighbor’s memory spike into your p99. When latency moves, teams often retune the model. The first job is to see whether the slice still had the device it thought it had.
1) Record the sharing mode, the slice size, and the set of jobs on the same card next to the latency. A regression that appears only when a training job or a catch-up batch is on the sibling slice is not a model regression. If you do not log the neighbors, you will not see this.
2) Different sharing modes fail differently. Memory isolation can still leave you short of cache and memory bandwidth; time-slicing can add scheduling delay without a clear OOM. Look at SM occupancy, memory bandwidth, and wait time, not only at a utilisation percentage.
3) A canary that runs a fixed, short inference on each slice, continuously, will show a noisy neighbor faster than a user-facing SLO that already averages many models. Keep the canary’s model and batch fixed.
4) If the product cannot tolerate sharing, say so in capacity policy rather than discovering it at peak. If it can, set a neighbor budget and test it.
Open question: which metric first showed you a noisy neighbor — canary latency, memory bandwidth, or a user SLO? Have you had a “model regression” that was actually a new job on the same GPU? Please share the signals you keep.
Sharing a GPU with MIG, time-slicing, or a process-level scheduler is a reasonable way to raise utilisation. It is also a way to import a neighbor’s batch or a neighbor’s memory spike into your p99. When latency moves, teams often retune the model. The first job is to see whether the slice still had the device it thought it had.
1) Record the sharing mode, the slice size, and the set of jobs on the same card next to the latency. A regression that appears only when a training job or a catch-up batch is on the sibling slice is not a model regression. If you do not log the neighbors, you will not see this.
2) Different sharing modes fail differently. Memory isolation can still leave you short of cache and memory bandwidth; time-slicing can add scheduling delay without a clear OOM. Look at SM occupancy, memory bandwidth, and wait time, not only at a utilisation percentage.
3) A canary that runs a fixed, short inference on each slice, continuously, will show a noisy neighbor faster than a user-facing SLO that already averages many models. Keep the canary’s model and batch fixed.
4) If the product cannot tolerate sharing, say so in capacity policy rather than discovering it at peak. If it can, set a neighbor budget and test it.
Open question: which metric first showed you a noisy neighbor — canary latency, memory bandwidth, or a user SLO? Have you had a “model regression” that was actually a new job on the same GPU? Please share the signals you keep.

