TurboServe serves interactive Streaming video generation workloads by jointly rebalancing long-lived sessions across GPUs and elastically scaling the GPU pool from one closed loop, cutting worst-case per-chunk latency by 37.5% and GPU cost by 37.2% on production traces.
Imagine you’re running an interactive video product where each user opens a session, types a prompt, watches a few seconds of video appear, tweaks the prompt, goes idle for a minute, comes back. Every session must keep emitting a new video chunk under a hard latency budget, and each session carries a persistent KV cache plus temporal state that can’t just be thrown away between chunks. Existing diffusion serving stacks like xDiT and FastVideo, and even LLM-serving systems like vLLM, assume stateless one-shot jobs. They have no story for a session that lives 20 minutes, goes idle, comes back on a different GPU, and still has to hit a per-chunk deadline.
The core object is a session, not a request. A session has three states: executing on a GPU, suspended with its state offloaded to host RAM, or terminated. Multiple active sessions on the same GPU are batched together each step via coalesced chunk processing, which runs the model once and writes each session’s new chunk back into its own state slot.
On top of that runtime sit two controllers wired into a closed loop. The placement controller watches the load on each GPU and, whenever a session arrives, activates, or leaves, runs a local search: find the GPU with the worst per-chunk latency, try moving one of its sessions elsewhere, keep the move if the latency drop exceeds the migration cost. Migrations happen at chunk boundaries and copy only the per-session state (not the model) between GPUs over NCCL using RDMA-style one-sided reads. The autoscaling controller watches the max per-GPU load ρ_max and compares it to a target utilization ρ̂ with a hysteresis band. If load is too high, add GPUs; too low, drain and release GPUs. Crucially, scale-out expands first then rebalances onto the new GPUs, while scale-in rebalances first (consolidating sessions) then removes GPUs.
The target utilization ρ̂ itself adapts to workload burstiness. The system measures recent activation volatility (standard deviation of new-session arrivals over a sliding window), quantizes it into ~10 levels, and looks up an offline-profiled ρ̂ for that level. Bursty workloads get a lower ρ̂ (more headroom); stable workloads get a higher one (denser packing). This offline-profiled lookup borrows the workload-classification idea from Quasar.
for event in stream: # arrival, departure, active/idle flip
placement, load = place(sessions, prev_placement, budget)
target_budget = scale(load, budget)
if target_budget < budget: # scale-in
placement, load = place(sessions, placement, target_budget)
budget = target_budget
elif target_budget > budget: # scale-out
budget = target_budget
placement, load = place(sessions, placement, budget)
The default reflex for elastic serving is to treat autoscaling and load balancing as independent layers: an autoscaler decides how many GPUs, then a placer fills them. This paper argues the opposite. For long-lived stateful sessions, placement and provisioning have to be one closed loop, because the GPU budget bounds which placements are feasible while the current placement determines whether that budget is enough. The load-bearing evidence is the three-way ablation on real traces: migration alone, autoscaling alone, or both together.
The ablation is the finding that carries the thesis. Removing autoscaling raises GPU cost by 42.9% on average (up to 80.4%) at matched latency, because the system has to statically provision for peak demand. Removing migration raises cost by 15.0% on average (up to 28.0%), because sessions pile up unevenly and the worst-loaded GPU sets the latency ceiling. Only the combined system hits both targets.
Other results support this:
•
End-to-end on production traces from Shengshu Technology, spanning LongLive-style models from 1.3B up, on clusters of 16 NVIDIA H20 and 64 NVIDIA B300 GPUs: 37.5% lower worst-case per-chunk latency at matched cost, 37.2% lower GPU cost at matched latency, versus baselines that do round-robin, load-aware-greedy, or memory-aware-greedy placement without migration or scaling.
•
The min-max rebalancing search comes within 3.6% of an exhaustive oracle placement on average while running >10× faster; on 64 GPUs it finishes in under 15 ms, well below per-chunk generation time.
•
The volatility-keyed autoscaler comes within 6.1% on average (max 8.3%) of an offline oracle that sees the entire future trace.
•
GPU-to-GPU migration costs 23–30 ms, roughly 2–3% of per-chunk latency, cheap enough to trigger routinely.
Reach for this when you’re building an interactive generative product with sticky sessions: streaming video, long-form voice, agentic sessions with warm context, anything where per-user state is expensive to rebuild and requests come in bursts. The lesson to steal even if you don’t adopt the system: register your per-session state buffers with the network stack up front, migrate only that region between GPUs at natural step boundaries, and drive both your autoscaler and your placer off the same load signal instead of running them as separate services.
Code is released at GitHub. The production traces from Shengshu Technology aren’t released as a public dataset, but the paper documents their arrival, departure, and active-session profiles across six traces in enough detail to synthesize similar workloads. The offline volatility-to-parameter table is described procedurally, so you can rebuild it against your own traces.
When your workload is long-lived stateful sessions, autoscaling and load balancing stop being two problems and become one. The GPU budget you pick constrains where sessions can live, and where sessions currently live determines whether that budget is enough. Wiring them together as a closed loop, driven by the same load signal, is what unlocks both the latency and the cost win. Split them and you leave roughly a third of each on the table.
•
The evaluation is entirely on one company’s traces and one family of streaming models. Workloads with very different session-length distributions, or with tighter per-chunk deadlines than ~1 second, may not see the same gains.
•
The adaptive ρ̂ depends on offline profiling per volatility level. If your production distribution drifts away from what you profiled, you’re back to a hand-tuned utilization target.
•
Cheap migration (2–3% of a chunk) assumes fast interconnects: NVLink within a node and InfiniBand across nodes. On commodity PCIe-only clusters, per-session state transfers get much more expensive and the migration-versus-latency tradeoff shifts.