LatentStream handles streaming video by folding retrieved history into a small pool of trainable Latent Memory Tokens (LMTs) instead of stuffing retrieved frames back into the prompt, using entropy-driven test-time updates to shrink KV cache memory from 30.8 GB to 22 GB while lifting OVO-Bench accuracy by +10.2 pp.
You’re building a live assistant that watches a video feed (a security camera, a robot’s egocentric stream, a smart-glasses recording) and answers questions on demand. The naive pipeline keeps every frame’s tokens in context, which blows up KV cache memory and slows every new token. The prevailing fix is store-and-retrieve: compress old frames into an external bank, and when a question arrives, pull back the most relevant frames and paste them into the prompt as extra visual tokens. Systems like FluxMem and OASIS follow this pattern.
The problem: retrieved evidence still enters the model as variable-length visual context. The model re-reads it fresh every query, and there’s no persistent working state that accumulates what’s been figured out. LatentStream targets that gap. It’s training-free (the Video-LLM backbone is frozen), so you can bolt it onto an existing Qwen2.5-VL deployment.
The pipeline has three parts. First, a hierarchical memory bank: incoming frames go into a short-term buffer at full detail, older stuff gets compressed into mid-term memory, oldest stuff gets aggressively spatially merged into long-term. The split points aren’t hand-tuned thresholds. They come from Jenks Natural Breaks on per-token importance scores, so consolidation adapts to how redundant the current stream actually is.
Second, and this is the contribution, the model maintains a handful of learnable Latent Memory Tokens (LMTs) (called LMTs). They’re split into three groups. The “short” group can only look at short-term memory. The “mid” group sees short plus mid. The “long” group sees everything. When a query arrives, each group retrieves its top-B most-similar frame tokens from its allowed scope, and those retrieved tokens get inserted next to the LMTs in the input.
Third, the LMTs themselves get optimized at test time. No labels. The reward is that the model’s next-token distribution should get more confident (lower top-δ entropy) as the LMT group sees more history — so entropy for the “long” group should be lower than “mid”, which should be lower than “short”. Optimization is REINFORCE-style: perturb the LMT embeddings with Gaussian noise, keep the direction that improves the reward.
for r in range(R): # R=4 evolution rounds
for g in [short, mid, long]:
evidence[g] = topk_similar(LMT[g], memory_bank[g], B=8)
U = concat(LMT, evidence)
for step in range(inner_steps):
noise = gaussian(sigma=0.1)
reward = confidence_progression(U + noise)
U += lr * reward * noise / sigma**2
# keep retrieved tokens only if reward improved
# at decode time: drop evidence, keep only evolved LMTs
Crucially, before generating the final answer, the retrieved visual tokens are thrown away. Only the evolved LMTs plus the compressed memory bank plus the query go into the decoder.
The prevailing approach to query-aware video memory is to retrieve relevant past frames and paste them into the prompt as extra visual context. This paper argues the opposite. Retrieved evidence should be absorbed into a small, fixed-size set of learnable tokens and then discarded before decoding, so the model reasons from an evolving latent state rather than re-reading raw frames. The cleanest evidence is Table 4: pasting retrieved frames directly gives +2.8 pp on OVO-Bench, adding un-optimized LMTs gives nothing, but LMTs optimized by the confidence reward give +7.3 pp — and the retrieved frames are deleted before the answer is generated.
The load-bearing ablation is the retrieve-and-internalize breakdown on OVO-Bench. Starting from a baseline of 56.9%: adding retrieved frames as extra context reaches 59.7%, adding LMTs without optimization drops slightly to 56.5%, but adding LMTs with the entropy-progression optimization jumps to 64.2%. The latent tokens do the work, not the retrieved frames themselves.
•
On OVO-Bench, the 7B model goes from 54.0% to 64.2% (+10.2 pp) overall. The Backward Tracing subscore jumps from 44.7% to 60.0%, driven mainly by the HLD (Hallucination Detection) category which more than doubles (23.7 → 68.3).
•
On StreamingBench, 73.9% → 76.9%, beating training-free peers FluxMem and OASIS.
•
On three offline long-video suites (Video-MME, MLVU, LongVideoBench), it also improves the frozen Qwen2.5-VL backbone by 1.4 to 6.1 pp — the mechanism generalizes past the streaming setting.
•
Ablating the hierarchical progression reward (using only absolute confidence, without requiring the longer-scope group to be more confident than the shorter-scope one) costs 1.7 pp. Ordering matters, not just confidence.
•
Efficiency: peak GPU memory drops from 30.8 GB to 22.0 GB and per-output-token latency from 6.45 ms to 3.16 ms, though time-to-first-token grows from 7.63 s to 8.41 s because of the test-time optimization loop.
Reach for this when you’re building a streaming video agent on top of a frozen Qwen2.5-VL-class backbone and hitting KV-cache pressure. Instead of a retrieval layer that pastes past frame tokens back into the prompt on every query, you keep a small LMT pool per session, run a few rounds of the entropy-driven update when a question arrives, then decode from a much shorter context. You trade higher first-token latency (about +0.8 s here) for lower per-token cost and much better answers on questions that need long-range history.
The paper reports results on Qwen2.5-VL 3B and 7B and compares against several concurrent training-free memory systems (FluxMem, OASIS, QueryStream, ReKV). It doesn’t provide a repository link in the text supplied; the release status isn’t stated. Implementation is well-specified: 8 short-term frames, 64 mid-term, 2048-token long-term budget, 2 LMTs per group, B=8 candidates per round, R=4 rounds, σ=10%, lr=1e-3, all on 8× H20 GPUs.
•
The test-time optimization adds ~0.8 s to time-to-first-token. For low-latency interactive use (voice agents, live captioning), that’s a real cost, and the paper doesn’t explore reducing R below 4 for latency-critical deployments.
•
The confidence-progression reward assumes broader context should make the model more confident. On adversarial or genuinely ambiguous questions where more context reveals more uncertainty, this signal could push the LMTs in the wrong direction. Not tested.
•
Gains rely on a capable frozen backbone (Qwen2.5-VL 3B/7B). The paper doesn’t show whether the retrieve-and-internalize mechanism helps smaller or weaker Video-LLM backbones, where the model may not have a useful entropy signal to optimize against.
•
The HLD (Hallucination Detection) subscore of OVO-Bench jumps from 23.7 to 68.3, disproportionately driving the headline number. If that category has quirks (few examples, format sensitivity), the average lift may look smaller in practice.
Stop pasting retrieved frames back into the prompt. Distill them into a few learnable tokens, tune those tokens at inference against the model’s own confidence signal, then throw the frames away before decoding.