Metis is a foundation model that stores conversation history as dynamic parameters inside the backbone instead of as retrieved text, updating that state with a single forward pass and reading it through a second attention branch, keeping per-session storage flat at ~17 MB even as history grows to 32K tokens.
You’ve shipped a chat assistant that needs to remember what a user told it three sessions ago: their dog’s name, that they moved to Berlin, that they no longer eat dairy. Today you probably run Retrieval-Augmented Generation over a vector store, stuff the top-k chunks back into the prompt, and pay the prefill cost every turn. That pipeline has known pain: retrieval can miss, the prompt grows, and gradients don’t flow through the discrete retrieve step so you can’t fine-tune the memory behavior end-to-end. Metis proposes moving memory inside the model, so remembering, forgetting, and updating happen as part of forward computation on a fixed-size parametric state.
Each transformer block gets a companion Metis block with two parts. A local memory block holds a fixed-size matrix M (roughly a key-to-value associative store) that acts as the persistent state across turns. A hyper memory block holds the trainable machinery that decides what to write into M given the current turn’s hidden states.
Writing works like this: after a turn finishes, a learned scorer picks the most informative token positions (a top-\u03c1 selection over a softmax importance distribution), projects those hidden states into memory keys and values, and folds them into M with a discounted update. In plain terms: “score tokens, keep the important ones, project them, add them to the running memory matrix with decay.” The authors found a Gated Delta Network (GDN)-style update beats a plain linear one and use it as the default. Reading works through a separate memory attention branch: a learned query projection queries M, and its output is blended with normal self-attention via a mixing weight \u03b3.
# per layer, per turn t
scores = softmax(H_t @ w_agg / tau) # importance over tokens
sel = top_rho(scores) # keep smallest prefix >= rho
K_t, V_t = H_t[sel] @ W_K, H_t[sel] @ W_V # project to memory k/v
M[t+1] = lam*M[t] + (1-lam)/L * K_t.T @ V_t # write (discounted)
# read during next turn:
A_mem = normalize(Q_tilde @ M[t])
A_out = gamma*self_attn(...) + (1-gamma)*A_mem
The backbone (Qwen3.5 at 4B/9B/27B) is frozen; only the memory-related projections are trained. Training data is synthesized from 27 public benchmarks like LoCoMo and LongMemEval into four operation classes: remember, update, forget, and reflect (multi-hop composition). Three loss objectives run jointly: memory reconstruction (regenerate a stored passage verbatim, sets a lossless-storage upper bound), memory operation (produce the right answer after a sequence of remembers/updates/forgets), and regularization (multi-entity binding so similar facts don’t blur, plus dialogue turns that shouldn’t touch memory at all).
The prevailing way to give an LLM long-term memory is to keep the model stateless and bolt on an external store: embed chunks, retrieve, paste into prompt. This paper argues the opposite. Memory should live inside the backbone as dynamic parameters that are written and read by ordinary forward computation, so remembering, forgetting, and updating can be trained end-to-end instead of hand-coded as retrieval rules. The load-bearing evidence is not the headline QA lift but the no-context setting: with the original conversation removed from the prompt, Metis still answers, while the same backbone scores near zero.
The key result is that without any conversation in the prompt, Metis extracts real information from its parametric state that the base model cannot. On LoCoMo (Gold) with no context, Qwen3.5-27B scores 0.07 average; Metis-27B on the same backbone scores 26.74. Full-context Qwen sits at 65.03, so parametric memory is not yet a replacement for stuffing evidence in the prompt, but it is clearly retaining something.
Secondary findings that support the mechanism:
•
On MemOps operation tasks (no context), Metis-27B averages 24.76 vs 1.69 for the raw backbone, and beats Temp-LoRA and \u03b4-Mem, the closest parametric-memory baselines.
•
Ablations: removing adaptive token selection (using the last-token hidden state instead) collapses overall performance by ~61%. Removing query-key normalization drops it ~28%. These are the two components the theoretical error analysis flags as suppressing noise from irrelevant past steps.
•
Scaling from 4B\u21929B is modest; 27B jumps noticeably, suggesting native memory needs a strong-enough backbone to be useful.
•
Storage stays flat at ~16.8 MB per session regardless of history length, vs a KV cache that grows to ~1.1 GB at 32K tokens for full context. A rank-64 Singular Value Decomposition (SVD) compression of the memory state keeps ~99.9% of quality at ~2.1 MB.
•
Honest failure modes the paper reports: performance degrades on long trajectories (accumulated compression error), similar facts get confused in latent space, and general-task accuracy drops sharply once the memory state is loaded with irrelevant chatter (IFEval falls from 79.85 to 54.53).
Reach for this direction when you’re building a personal assistant or long-running agent where a growing prompt is your latency and cost ceiling, and where you’d rather train memory behavior than tune a retriever. The scenario: instead of running vector search plus re-ranker plus prompt assembly every turn, you commit each turn to a fixed-size state with one forward pass, and subsequent turns read from it through an added attention branch. Storage per session stops growing with history.
The authors release code and checkpoints (Qwen3.5-based 4B/9B/27B, plus a Llama-3.1-8B port that transfers reasonably) at GitHub and HuggingFace. License is CC BY-NC-SA 4.0, so non-commercial only. Practical caveats they surface: memory-write is a separate forward pass (~0.2s for a chunk on their setup), and the full-context baseline still wins in absolute quality below ~64K tokens on their hardware.
Memory in an LLM stack can be a trained parameter update instead of a retrieved string, and that reframing lets you fine-tune remember, forget, and update as first-class model behaviors. The catch is that fixed-size state means fixed-size capacity: strong on short and medium histories, degrades on genuinely long ones, and can pollute unrelated tasks when the state fills with noise. Treat it as complementary to retrieval, not a drop-in replacement.
•
Long histories still degrade. The state is fixed-size, so compression error accumulates across many updates. The paper’s own capacity study shows accuracy on early facts falling steadily as the trajectory grows.
•
The memory state can pollute unrelated tasks. After irrelevant turns are stored, IFEval strict-mode accuracy drops from 79.85 to 54.53 on Metis-4B. If your product mixes memory-heavy and memory-irrelevant queries, you need a gate.
•
Full-context prompting still wins on quality when it fits: Metis closes the no-context gap dramatically but does not match full-context Qwen on LoCoMo. The win is efficiency and scalability, not raw accuracy at short lengths.