StepAudio 3 Realtime is a full-duplex speech model that runs private reasoning in parallel with spoken output via Think-While-Speaking, so it can deliberate on hard turns without stalling the voice channel, and hand off long tasks to async tools while the conversation continues.
If you build a voice assistant, you hit three problems that a text chatbot never sees. First, you don’t know when the user is done talking. A 400ms silence could be a breath or a turn ending. Second, if the user interrupts you, you need to figure out whether “right, right” means “keep going” or “stop, you’re wrong.” Third, if the request is hard enough to need chain-of-thought reasoning, spending 4 seconds thinking silently before opening your mouth feels broken. And if the request needs a tool call that takes 20 seconds (book a flight, search a database), you can’t just go mute.
The usual baseline stack solves these piecewise: a VAD module decides turn boundaries, a separate reasoning LLM handles hard queries with visible latency, and tool calls block the dialogue. Recent full-duplex models like GPT-realtime-2 and Qwen Audio 3.0 Realtime Plus fold some of this into one model but still tend to trade reasoning depth for responsiveness. StepAudio 3 Realtime is StepFun’s attempt to keep both.
The model is a roughly 196B-parameter MoE (about 11B active per token) with Step 3.7 Flash as the language backbone and the Audio Transformer (AuT) encoder from Qwen3-Omni as the audio frontend. Two audio streams feed the decoder at once: what the user is saying and what the model itself is currently saying. That second stream matters because it lets the model interpret an overlapping user utterance in light of what the user just heard.
Time is chopped into 320ms audio blocks, and after each block the model emits a small state token: keep listening, start speaking, keep speaking, or yield the floor. This is how Seamless Duplex turn-taking works, it’s a token-level decision on a shared timeline rather than an external VAD.
The headline mechanism is Think-While-Speaking, derived from Mind-Paced Speaking. Two concurrent calls to the same model act as a “Formulation Brain” (writes a private reasoning trace) and an “Articulation Brain” (emits short spoken segments conditioned on whatever reasoning is available so far, plus what’s already been said aloud). By default the model speaks first and reasons in parallel; a later continuation can correct an answer that started from incomplete reasoning.
Two helpers make this affordable. Adaptive Thinking decides per-turn whether explicit reasoning is even needed. Training data is built by giving a probe model an empty think block, then having a blind judge compare the no-think answer against the original reasoned answer; turns where reasoning didn’t help get relabeled as no-think. And Multi-Token Prediction (MTP) uses three draft heads with Medusa-style typical acceptance to speed up the private reasoning stream, while keeping strict speculative-decoding verification for the spoken output.
for block in audio_stream_320ms():
state = model.step(user_audio=block, model_audio=own_output)
if state == "speak":
if adaptive_thinking.needs_reasoning(context):
spawn(formulation_brain.think()) # private, MTP-accelerated
segment = articulation_brain.emit(reasoning_so_far, spoken_so_far)
play(segment)
elif state == "yield":
break
A Voice Agent layer routes requests: direct answer, lightweight tool (weather, search), or async backend task. Long tasks run in the background, and the user can ask about progress mid-execution, following the interleaved reasoning-plus-action pattern of ReAct.
On StepAudioChat in pure reasoning mode, the model hits a 73.0 macro average, above Doubao 2.0 Lite (70.5) and DeepSeek-V4-Flash (71.4), below Kimi K3 (77.1). In realtime mode with Think-While-Speaking active, the macro slips to 70.4, roughly matching the non-realtime dialogue baselines. So the claim is that speaking in realtime costs about 2.6 points versus pure reasoning mode on their own benchmark.
On the Artificial Analysis Full-Duplex Bench, StepAudio 3 Realtime scores 98.9 overall, top of the reported table, with 100.0 on turn-taking and 99.0 on interruption handling. On MMSU (spoken-language understanding) it reaches 90.6, versus 83.6 for Gemini 3.1 Pro. It leads four of eight audio-understanding benchmarks but trails Gemini 3.1 Pro by 17.7 points on AudioMultiChallenge (multi-turn constraint following).
On \u03c4-Voice, a tool-grounded customer-service benchmark scored by whether the final database state matches the target, it gets 56.0% macro success, essentially tied with Grok Voice Think Fast 2.0 High at 56.5% and above the GPT and Qwen entries. Domain breakdown is uneven: telecom 70.2% (best of the group), airline 60.0%, retail 37.7% (well behind Grok’s 49.7%).
Two ablations are worth flagging. A “less is more” SFT ablation compares ~2M random examples against ~100K quality-filtered ones; the smaller set lifts MMSU from 78.78 to 89.70. And the Adaptive Thinking table shows the selection policy is imperfect: the Reasoning category invokes explicit thinking only 59.5% of the time even though full thinking helps Reasoning the most (+11.37 pts over forced no-think), and Adaptive Thinking actually underperforms Direct SFT on Reasoning (66.80 vs 71.89). So the router misallocates on the category that most needs deliberation.
If you’re building a voice product, the design pattern here is the takeaway even if you don’t use this specific model. The interesting move is splitting reasoning from articulation as two parallel decode streams on the same weights, rather than either (a) blocking speech on a full CoT or (b) skipping reasoning to stay snappy. If you control your own inference stack, this is worth prototyping: it needs shared KV state between the two decode passes, which is hard through a hosted API.
For evaluating voice agents, the Full Duplex Bench category breakdown (pauses vs turn-taking vs interruption vs backchannel) is a more useful mental model than a single “latency” number. A system that scores 100 on turn-taking but 60 on backchannel handling will feel jumpy in a way a mean score hides.
The Adaptive Thinking result is a cautionary datapoint: a learned think-or-not-think router trained by comparing reasoned-vs-unreasoned answers with a blind judge did not, in their setup, actually route the reasoning-heavy categories toward more thinking. If you’re planning to build a similar dispatcher, budget for iteration on the labeling signal, not just the policy.
The retail gap on \u03c4-Voice is worth heeding before pitching this class of model for e-commerce workflows. The paper doesn’t explain why retail lags, so treat that as an open question rather than a solved capability. Code and weights: the paper links a project page but doesn’t explicitly state a model-weights release in the excerpt.
Most baseline comparisons are against vendor systems with their own version drift (GPT-realtime-2 for duplex, GPT-Realtime-2.1 for agentic), and the dialogue benchmark, StepAudioChat, is the authors’ own construction. It’s designed carefully with paired valid/flawed responses and blind judging, but it is not a third-party benchmark, so the 73.0 reasoning-mode number should be read as “comparable on our own eval,” not as an independent ranking.
The wall-clock speedup numbers for MTP (1.49x to 2.05x) are, per the authors, not a controlled comparison across draft depths, since they use different timing configs. Don’t quote those as apples-to-apples.
Adaptive Thinking’s own ablation shows it reduces accuracy on the Reasoning category versus always thinking. The realtime system’s dialogue macro (70.4) is measurably below the same model’s reasoning-mode macro (73.0), so “comparable to dedicated reasoning models while speaking in realtime” holds against Doubao and DeepSeek but not against Kimi K3. And the paper is transparent that multi-turn constraint following (AudioMultiChallenge) and retail tool use are unresolved.