DeltaStreamer answers questions about live video by writing per-frame change deltas into a hierarchical text memory, then letting a planner replay the original frames when the stored notes miss a visual detail.
Imagine a security review agent watching a camera feed. It cannot store every pixel, so it summarizes what it sees. An hour later you ask “which hand did the person use to pick up the red block?” The summary noted a person and a block, but not the hand. The frame is gone, and the agent either guesses or refuses.
Most prior streaming-video systems hit exactly this failure. They compress history through KV cache reuse, token pruning, or rolling dialogue summaries, which capture what is present in a scene but blur what changed between frames and discard the raw pixels. Agent-style approaches like StreamAgent and EventMemAgent added planning and tool use, but still answer only from whatever was written down during observation. If the needed detail was never encoded, it is gone.
DeltaStreamer treats streaming video QA as active investigation rather than passive encoding. Four pieces fit together:
•
Delta-augmented observation. For each sampled frame (1 fps), a VLM looks at the current and previous frame together and writes two things: a structured scene description, and a delta describing what changed (object appeared, state flipped, action began). Making change explicit matters because consecutive frames often read nearly identical in prose even when something important happened.
•
Traceable hierarchical memory. Observations go into a three-layer Directed Acyclic Graph (DAG): working events, episode summaries, and long-term pins. Reverse pointers link summaries back to the specific observations and frame indices that support them. An entity index maps named actors and objects to the timestamps where they appear.
•
Query-driven planner. When a question arrives, a planning call produces a plan specifying the target, temporal anchor, evidence type, where to look (past, current, incoming frames), and a search strategy. This plan is persisted and re-injected into every later prompt, so the agent does not drift from the original information need as tool results accumulate.
•
Look-back replay. If retrieved text lacks a decisive detail, or a rule-based verifier flags conflicting claims across memory layers, the agent follows source pointers back to the original frame and re-examines it with the VLM. Replay can also trigger zoom crops or SAM2-based object tracking on that frame.
Control flow at query time, in pseudo-code:
plan = vlm_plan(question, current_frame, memory_snapshot)
while not done:
evidence = backtrack(memory_graph, plan.retrieval_query)
answer = vlm_answer(question, plan, evidence, current_frame)
conflicts = verifier(evidence)
if answer.needs_detail or conflicts.severe:
frame = replay(pick_target(conflicts, plan))
memory.update(reinspect(frame))
else:
done = True
Episode boundaries are decided online by Bayesian Online Changepoint Detection over hashed-token embeddings of each observation, with a content-continuity check to avoid splitting a scene that merely drifted stylistically.
All results use a single shared Qwen3.6-27B backbone for observation, planning, and answering.
•
On OVO-Bench, DeltaStreamer scores 66.07 overall, +5.32 over EventMemAgent and +2.05 over Gemini 3 Flash. Gains concentrate on tasks that reward tracking transitions: action-sequence identification (79.05) and hallucination detection (69.89).
•
On StreamingBench, it scores 59.53 overall, +1.95 over the next-best full-open-source system (STRIDE). Contextual-understanding tasks improve notably, but Proactive Output drops to 10.40, so retrospective evidence recovery does not fix when to speak.
•
Ablations isolate the three mechanisms. With identical two-frame input, writing an explicit visual delta lifts OVO-Bench from 61.12 to 66.07; dropping the delta field is the equivalent of 2F-NoDelta. Flat memory scores 61.25 vs hierarchical 66.07. Removing replay is the largest tool-removal drop (66.07 \u2192 63.28), bigger than removing zoom or tracking.
•
Cross-component ablation shows delta and planning compound: adding delta on top of hierarchical memory gives +0.93 without the planner but +4.95 with it. The authors read this as the planner being what lets the agent actually use the recorded transitions.
•
If you’re building a streaming-video agent and your current failure mode is “we summarized the frame but not the detail that mattered,” the DeltaStreamer recipe suggests two cheap additions worth testing: write an explicit inter-frame change field alongside each scene caption, and keep raw frames addressable so a later call can re-inspect them. Both are implementable on top of any VLM you already use.
•
If you cannot store raw frames (bandwidth, privacy, retention limits), the replay mechanism is unavailable and the paper’s largest single ablation drop (replay removal) applies to you. Change-aware captions and hierarchical memory still help, but expect a smaller gain.
•
For QA tasks where the answer depends on precise timing of events (the Proactive Output result), this architecture does not help. The paper is explicit that recovering evidence after the fact cannot substitute for responding at the right moment.
•
The persistent-plan trick is reusable outside video: generate a structured plan once at query time, re-inject it into every subsequent tool-use turn, so the agent does not drift as context grows. The HLD gain from plan generation alone (45.28 \u2192 65.83) in their planner ablation is the clearest signal here.
•
A project page is linked; the paper does not state whether code or weights are released.
•
One backbone, two benchmarks. The authors acknowledge they have not tested other VLMs or measured accuracy-latency-storage trade-offs.
•
Replay and multi-round tool use add inference calls. The paper notes latency overhead but does not report wall-clock numbers against the streaming baselines it beats.
•
The verifier is rule-based (fixed patterns for states, locations, counts). Agreement across memory layers is used as a correctness proxy, which the authors note does not guarantee the answer is visually correct.
•
Several comparison rows in the main tables are partial (dashes for backward-tracing and forward-responding columns on limited-open-source systems), so some head-to-head comparisons are only on a subset of OVO-Bench tasks.
•
Continuous observation with retained frames raises obvious privacy exposure. The authors flag this and recommend consent, retention limits, and access controls, but did not evaluate those safeguards.