LaMem-VLA gives a robot policy two memory banks (recent visuals, longer-horizon action state), retrieves relevant entries, and compresses them into a fixed number of tokens that sit inside the Vision-Language-Action model input sequence alongside the current image and instruction. On the memory-free ablation, this lifts SimplerEnv-Bridge success from 57.3% to 73.9%.
Imagine you’re building an agent that executes a multi-step recipe: open drawer, pick up spoon, place it on a towel, then stack a cube on top. Most current Vision-Language-Action model models re-decide each action from just the current camera frame plus the instruction, which is fine for one-shot picks but forgets which subtasks are already done. The dominant fix, exemplified by MemoryVLA, keeps a separate memory bank and hands retrieved history to the action head as an extra input, after the vision-language model has already done its reasoning. LaMem-VLA argues that’s the wrong seam: memory should be a token in the same sequence the model is already reasoning over.
The backbone is a 7B Prismatic VLM fine-tuned on Open X-Embodiment, driving a diffusion action head based on Diffusion Policy. At every timestep, four modules cooperate:
•
Curator writes two banks. Short-term stores compressed current-episode visual tokens as key/value pairs. Long-term stores the hidden states of the action-query tokens (a proxy for “what the model thought it was doing”). When either bank exceeds capacity (L=16), the two temporally adjacent entries with highest cosine similarity get averaged together.
•
Seeker takes the current visual+instruction hidden states, appends learnable query slots, runs a small transformer, and mean-pools the result into one retrieval vector. It picks top-K=8 entries from each bank by cosine similarity.
•
Condenser doesn’t paste retrieved entries verbatim. It initializes fixed slot tokens (8 for short-term, 4 for long-term) and lets them attend over the retrieved evidence plus the query, keeping only the final slot outputs. The output length is now constant regardless of how much history was retrieved.
•
Weaver prepends those 12 memory tokens to the visual tokens, instruction tokens, and learnable action queries, then runs the whole thing through the VLM. Action-query outputs condition the diffusion head via DDIM sampling sampling.
for t in episode:
x_t, i_t = vlm_encode(obs_t, instr)
q = seeker(x_t, i_t) # context-aware query
Zs = topk(short_bank, q, K=8) # retrieved visual evidence
Zl = topk(long_bank, q, K=8) # retrieved action-state evidence
Ms = condense_short(q, Zs) # 8 tokens
Ml = condense_long(q, Zl) # 4 tokens
seq = [Ms, Ml, x_t, i_t, Q_action]
z_action = vlm(seq)[-N_a:]
actions = diffusion_expert(z_action) # DDIM, 10 steps
short_bank.append(compress(x_t)); long_bank.append(z_action)
The prevailing approach to memory in robot policies is to keep history in an external bank and inject retrieved chunks as side conditioning to the action head, after the Vision-Language-Action model has already reasoned. LaMem-VLA’s claim is the opposite: put memory tokens inside the same sequence the VLM attends over, so history participates in perception and instruction-grounding, not just action decoding. The evidence that carries this claim is the latent-native vs policy-side ablation, not the headline benchmark score.
The load-bearing ablation swaps where memory enters the model while holding memory content roughly fixed. On SimplerEnv-Bridge, the memory-free baseline sits at 57.3%. Feeding memory as policy-side conditioning (the MemoryVLA-style path) lifts it to 71.9%. Feeding raw retrieved evidence directly hits 69.8%. Putting condensed memory tokens inside the VLM sequence reaches 73.9%. So latent-native integration adds ~2 points over the same memory used as external context, and compression matters (raw retrieval underperforms condensed).
Secondary numbers as corroboration:
•
On SimplerEnv-Bridge overall, LaMem-VLA hits 73.9% average vs CogACT (its baseline) at 57.3% and π0 at 69.2%.
•
On LIBERO, it reaches 97.6% averaged across five suites, edging MemoryVLA by 1.1 points; on the long-horizon Long-10 and Long-90 suites specifically, it leads MemoryVLA by 2.4 and 1.4 points.
•
Removing both memory streams costs ~16 points on SimplerEnv; removing either stream alone costs ~8, suggesting short-term (visual) and long-term (action-state) memory carry non-overlapping signal.
•
Retrieval budget K peaks at 8; K=12 mildly regresses, consistent with the condenser being overloaded by redundant hits.
Reach for this design when you’re building a manipulation policy that repeatedly loses track of “which subtask am I on” across a long trajectory. The concrete change vs a standard CogACT-style setup: keep two small ring buffers keyed by pooled visual features and by action-token hidden states, retrieve top-K by cosine at each step, and route them through a small transformer that outputs a fixed 12-token memory prefix. That prefix goes at the front of the VLM input, not into the policy head. The number of memory tokens stays constant, so per-step compute doesn’t drift as the episode grows.
Code is promised at LaMem-VLA though the linked repo state at the time of writing isn’t described in the paper. Training used 8 H800 GPUs; the recipe fine-tunes on LIBERO demos (50 per task) and Bridge v2 separately. All evaluation is in simulation (SimplerEnv-Bridge, LIBERO), no released real-robot artifacts.
If memory should influence what the model perceives, not just what it decides, put it inside the token sequence rather than bolted onto the policy head. The gap between policy-side conditioning and latent-native injection is small in absolute terms (~2 points here), but it’s the difference between memory as an afterthought and memory as context. The compression step matters as much as the placement: raw retrieved history hurts, condensed fixed-length tokens help.
•
All results are simulated. The authors flag real-robot experiments as future work, so the claim that “native memory helps long-horizon manipulation” is not yet validated on physical hardware where perceptual noise and control latency change the picture.
•
The gap between latent-native (73.9%) and policy-side conditioning (71.9%) on SimplerEnv-Bridge is 2 points; on LIBERO-90 it’s 2.2 points. That’s meaningful but not decisive, and the paper doesn’t report variance across seeds, so how much of that gap is architectural vs noise is unclear.
•
Memory-bank capacity is fixed at L=16 with a naive “merge most similar adjacent pair” eviction. On genuinely long horizons where task-critical evidence appears far outside a 16-slot window, this heuristic could quietly discard the exact frame the seeker needs, and the current evaluation suites may not stress that regime hard enough to expose it.