HybridAgent lets an LLM agent switch mid-generation between normal text (for tool calls and final answers) and a latent mode where it feeds its own hidden state back as the next input, keeping the full mixed history in one KV cache across turns. The result: fewer tokens generated, faster inference, and latent steps that are still readable by nearest-token lookup.
If you’ve built an agent on top of an LLM, every reasoning step is text. The model writes “Let me think about X,” writes out X, writes a tool call, waits for the result, writes more reasoning, and so on. Each of those words costs a decode step. For a GAIA-style question that chains search, Python, and multi-hop reasoning, you can easily spend hundreds of tokens on inner monologue that no human ever reads.
A parallel line of work called Latent reasoning (notably Coconut from Hao and colleagues) skips the decoding step: the model’s last hidden state is fed straight back in as the next input embedding, so “thinking” happens in continuous vectors instead of tokens. That’s faster, but it breaks the two things agents actually need. You can’t call a tool from a hidden vector (tools expect text), and you can’t inspect what the model is thinking, which matters for debugging and safety.
The authors want both: latent speed when the model is just thinking to itself, and text when it needs to act on the world or expose its reasoning.
The core trick is a small adapter called HybridLink. In plain terms: a hidden state and a token embedding live in related but not identical spaces, so you can’t just feed one in place of the other. HybridLink is a tiny two-layer residual network (~10M params for a 4B backbone) that maps a last-layer hidden state into something the input embedding layer would accept. When the model emits a special <latent> token, generation switches to latent mode: each step’s hidden state goes through HybridLink and becomes the next input, no sampling, no decoding. The model exits latent mode when its hidden state’s nearest token embedding is </latent>.
Across turns, the paper keeps everything (text, latent vectors, tool outputs) as entries in the Transformer’s KV cache, which they call continuous working memory. There is no re-tokenization step between turns, because latent vectors have no tokens to recover. An appendix proves this cached continuation gives the same hidden states as re-running the full prefix.
Training has two stages. HybridSFT teaches coordination between the backbone and HybridLink. A naive approach would train HybridLink to match ground-truth embeddings at latent positions, but at inference time the backbone sees HybridLink’s own (imperfect) outputs, not ground truth. This gap is a continuous version of Exposure bias. Their fix is self-conditioned training: run two forward passes, where the second pass replaces ground-truth latent embeddings with HybridLink’s actual predictions from pass one, then compute loss on that.
HybridRL adapts Group Relative Policy Optimization (GRPO) to hybrid trajectories. The problem: standard policy-gradient methods compute an importance ratio per token, but latent positions have no sampled token. The paper proves a hybrid policy factorization: because latent steps are deterministic given the backbone and HybridLink, the trajectory likelihood factorizes over just the sampled text positions. So you only need importance ratios on text tokens, and gradients still flow through the backbone that produced the latent inputs.
# One HybridRL step, roughly
for prompt in batch:
rollouts = []
for _ in range(group_size):
tau = agent.generate(prompt) # text, latent vecs, tool obs
rollouts.append(tau)
advantages = group_normalize([reward(t) for t in rollouts])
for tau, A in zip(rollouts, advantages):
# Replay with recorded latent vecs fixed; recompute text logprobs
logp = agent.replay_text_positions(tau)
loss = -A * clipped_ratio(logp, tau.old_logp)
loss.backward()
Reward has three terms: correctness, format validity, and a bonus for calling multiple tools. There is no explicit reward for using latent vs text mode, so the balance is learned.
They train on Qwen3-Base at 4B, 8B, and 14B. Tools are a Python sandbox and a search API, capped at 10 calls per trajectory. Evaluation covers 11 benchmarks: math (AIME 25/26, MATH500), knowledge (GPQA-Diamond, MedQA, MuSiQue), and agentic tasks (GAIA, Humanity’s Last Exam (HLE), WebWalker, SealQA, DSQA).
Headline numbers from the paper’s own framing:
•
Averaged across benchmarks, HybridAgent improves over matched text/latent baselines by 10.4% at the RL stage and 18.5% at the SFT stage.
•
Token usage drops by 35.5%–54.7%, inference is 1.8× to 2.6× faster, and estimated decoding cost is 2.7× to 3.7× cheaper than text-only agents on the same backbones.
•
At 14B, HybridAgent beats the strongest agentic framework baseline by 17.4% on average across agentic reasoning tasks.
•
Pure latent baselines like Coconut collapse on agentic tasks (often under 5% accuracy on GAIA/WebWalker), consistent with the authors’ claim that pure latent reasoners can’t do tool use well.
Two more interesting results. First, causal interventions on MATH500: replacing latent vectors with ones from a different problem drops 14B accuracy from 88.2 to 48.6, while nearest-token replacement barely hurts (87.2). So the latent trace carries real problem-specific information, not just filler. Second, an auditability study on 560 math problems shows HybridAgent exposes 43–49% fewer text tokens than matched text agents, but a weak monitor’s ability to predict rollout correctness stays within 0.022 AUROC of the text agent. First-error localization and corrective-hint fix rates are comparable or better. The authors interpret this as: the exposed text, while shorter, is denser in load-bearing reasoning steps.
As a limit: on the hardest category of HLE (chemistry/engineering/“CE”), HybridAgent 14B scores 13.3%, which beats small-model baselines but is still well under frontier closed models (Kimi-K3 at 32.0).
For an engineer building agents, the practical takeaways:
•
The latency and cost numbers are the clearest selling point. If you’re serving an agent that produces long chain-of-thought you never show the user, moving that compute into latent steps saves tokens without obviously hurting accuracy. The paper does this on open Qwen3 backbones, so you’d need to be in a position to train (or fine-tune with full-parameter access), not just prompt a hosted API. There’s no reported way to do this through OpenAI or Anthropic.
•
The auditability finding is worth sitting with before you decide latent-mode agents are opaque by default. The exposed text here is enough for a weaker model to flag likely-wrong rollouts and even to localize the first erroneous step. If you’re designing monitoring for agent outputs, that suggests you can keep oversight on tool calls and visible checkpoints without insisting every reasoning step be verbalized.
•
The safety angle the authors raise is specific: because latent thoughts exist only as hidden states readable via the provider’s own HybridLink, a model provider could in principle monitor the full reasoning while giving users only the text. That’s a different threat model than hiding chain-of-thought as encrypted text, which prior work (Panfilov et al. 2026) showed can be replayed into a sibling model.
•
Not evaluated: whether a trained HybridLink transfers across backbones, what happens with much longer latent spans than the 256-step cap used here, or how this interacts with speculative decoding and other inference optimizations.
The paper does not link a public code or model release in the supplied text.
•
The 10.4% average improvement is against matched same-size baselines, not against frontier closed models. On the hardest agentic splits, frontier agents like Kimi-K3 and GPT-5.6-Luna still lead or are competitive.
•
“Interpretable latent mode” means each latent step has a nearest token in the vocabulary; it does not mean the resulting pseudo-text is a faithful account of what the model is computing. The causal-intervention results show the latent content matters, but readability is measured by cosine similarity to token embeddings, not by human verification of meaning.
•
The architecture requires modifying the model, the training stack, and the inference server (vLLM and verl needed extensions). This is not a drop-in prompting technique. If you can’t retrain and can’t modify serving, none of this applies to you yet.
•
Numbers for cost and speedup assume specific utilization and pricing (10% of peak BF16 throughput, $2.50/H100-hour) and exclude prefill and tool execution time. Real deployments will differ.
•
Benchmarks like AIME26 and HLE are recent; some baseline numbers come from the paper’s own runs rather than published leaderboards, and some frontier model identifiers in the paper (GPT-5.6-Luna, Kimi-K3, DeepSeek-V4.1-Flash) suggest an evaluation window where exact model versions matter.