SAO replaces GRPO’s group-of-N sampling with one rollout per prompt in asynchronous LLM RL, keeping training stable past 1000 steps where vanilla Group Relative Policy Optimization (GRPO) collapses around step 160, by pairing a value model with strict token-level clipping against rollout log-probs.
You’re training a coding agent with RL. Each rollout is a multi-turn trajectory: some finish in 30 seconds, some grind through 300 tool calls over minutes. In a synchronous pipeline your GPUs sit idle waiting for the slow ones, and if you use Group Relative Policy Optimization (GRPO) you also have to wait for all 8 samples of the same prompt before you can compute an advantage. Asynchronous RL fixes the idle time by training on rollouts as they arrive, but now the model that generated a trajectory is several updates behind the model you’re training. That drift (“policy lag”) is what usually blows training up.
Prior asynchronous LLM-RL systems like AReaL focused on the throughput side of this problem. SAO focuses on what the loss function should look like once you accept that policy lag is unavoidable.
Three moves, all aimed at making single-trajectory updates survive under lag.
First, drop group sampling. Group Relative Policy Optimization (GRPO) estimates advantage by averaging rewards across N rollouts of the same prompt, so every prompt is a synchronization barrier. SAO uses one rollout per prompt and gets its baseline from a learned value network instead. This is closer to classic PPO but in a fully async loop.
Second, simplify importance sampling. Standard off-policy correction needs the exact old-policy probabilities, but in async training a trajectory may have been generated across many intermediate model versions, so “the old policy” isn’t a single well-defined thing. SAO just uses the log-probs the rollout engine already logged as the behavior policy, ratio = current / rollout. Then it applies double-sided token-level clipping: if a token’s probability ratio falls outside the trust window, its gradient is masked to zero entirely (not just clipped to the boundary as in PPO). The paper calls this Direct Double-Sided Importance Sampling.
Third, make the value model trustworthy enough to be the sole baseline. Three tricks: update the critic K=2 times per policy update so it tracks the moving policy; freeze the attention layers of the value model and only train the Mixture of Experts projections, because full-attention critic training produced large unstable gradient norms; and use a skip-observation GAE that computes advantages from one action token to the next action token, jumping over environment-feedback tokens the model didn’t generate.
while training:
traj, rollout_logprobs = async_queue.get() # arrives when ready
for _ in range(K): # K=2 critic updates per policy update
update_value(traj, freeze_attention=True)
adv = skip_obs_gae(traj, V) # skip env-feedback tokens
ratio = exp(logpi_theta(traj) - rollout_logprobs)
mask = (1 - eps_lo < ratio) & (ratio < 1 + eps_hi)
loss = -(mask * ratio * adv * logpi_theta(traj)).mean()
update_policy(loss)
The prevailing move for stable RL on LLMs has been Group Relative Policy Optimization (GRPO): skip the value model and get your baseline from a group of same-prompt rollouts. This paper argues the opposite for async and agentic settings. When rollouts are long, variable-length, and arrive one at a time from a real environment, the group-baseline is a liability. Invest in a well-behaved value model instead, and the async pipeline stops fighting you. The load-bearing evidence isn’t the headline benchmark, it’s that vanilla Group Relative Policy Optimization (GRPO) and vanilla VAPO collapse within a few hundred steps while SAO trains stably to ~1000.
The stability result is the main finding. Standard Group Relative Policy Optimization (GRPO) collapses at roughly step 160; standard VAPO collapses around step 90. Adding SAO’s Direct Double-Sided Importance Sampling clipping to GRPO fixes the collapse, showing the clip strategy alone carries the stability. SAO and GRPO-with-DIS then track each other for the first ~400 steps and diverge after that, which is where the single-rollout plus value-model design starts paying off.
On benchmarks, using a Qwen3-30B-A3B base fine-tuned for tool-integrated math reasoning:
•
AIME2025: 84.2 (GRPO) → 97.3 (SAO)
•
HMMT Nov 2025 Nov 2025: 76.0 → 88.3
•
IMOAnswerBench: 55.8 → 74.0
•
SWE-Bench Verified (coding, OpenHands scaffold, 300 turns): 23.0 base → 27.0 GRPO+DIS → 29.8 SAO
Ablations: dropping the K=2 critic updates costs ~2\u20135 points; unfreezing critic attention costs ~7 points on AIME2025; a running-mean reward baseline instead of a value model drops AIME2025 to 79.8. In an online-learning simulation where the reward preference flips between writing styles (cute, chuunibyou, classical), SAO’s value critic re-aligns within a phase while a running-mean baseline lags because its window is still averaging over the old reward regime. SAO was used in production for training the GLM-5.2 750B-A40B model.
Reach for this when you’re doing RL on an agent whose rollouts are long, uneven, and only produce one trajectory per prompt (a coding agent against a test suite, a GUI agent, an online-feedback system). The recipe: run rollouts async, log the rollout engine’s per-token log-probs, keep a value model that you update twice per policy step with attention frozen, and clip token ratios hard against those logged log-probs rather than trying to reconstruct “the” old policy.
The paper doesn’t advertise a code release; the method was deployed inside the GLM-5.2 training stack but no repo URL is given in the text. The math training data comes from distilling GPT-OSS-120B into tool-integrated reasoning traces on Qwen3-30B-A3B. Benchmarks used (SWE-Bench Verified, AIME2025, BeyondAIME, HMMT Nov 2025, IMOAnswerBench) are all public.
In async RL for long agentic rollouts, a well-trained value model beats a group-of-samples baseline. GRPO’s group trick was a good deal when rollouts were short and synchronous. Once you have stragglers, environment feedback, and single-trajectory reward signals, the value model earns its keep, provided you update it faster than the policy and stop it from wrecking its own attention weights.
•
Everything is shown on one backbone (Qwen3-30B-A3B, MoE). The frozen-attention critic trick is explicitly motivated by MoE-vs-attention gradient behavior, so it may not port to dense models unchanged.
•
SAO needs the rollout engine to preserve exact per-token log-probs through async generation. If your serving stack doesn’t emit those, the Direct Double-Sided Importance Sampling clipping loses its anchor.
•
The online-learning result is on a simulated reward-shift over writing styles judged by an LLM, not real user feedback. Generalization to real non-stationary user preferences is asserted, not demonstrated.