LatentPress trains a tiny adapter that rewrites long chat history into continuous “soft tokens” a frozen LLM reads directly through its embedding layer, hitting 7.7× compression with no accuracy loss versus feeding the raw evidence.
You’re building a long-running assistant that accumulates months of user chats, tool calls, and observations. Today you either stuff a giant transcript into context (slow, expensive) or run an LLM summarizer over it (loses precise facts like dates and names) or render it to images and DeepSeek-OCR the text back (adds an autoregressive decoding stage before the reader even sees the input). All three routes force the stored context back into human-readable text before the model can use it. LatentPress asks why: if the consumer is a language model, the compressed form doesn’t have to be text at all. Prior soft-token work like ICAE (In-Context AutoEncoder) pointed here but still fine-tuned an LLM-scale encoder; this paper keeps the reader entirely frozen and trains roughly 0.1% of decoder parameters.
Split context handling into two operations: Write turns text segments into continuous vectors; Read hands those vectors to a frozen decoder that answers the question. The writer borrows the frozen decoder’s bottom two transformer layers as an encoder (deep-copied, never updated), then a small trainable linear adapter (initialized to identity) projects the pooled hidden states into the reader’s embedding space. At inference, the decoder receives [soft_tokens; embed(question)] through its normal input-embedding interface. No text is ever reconstructed.
Two design choices matter. First, variable pooling by role: user turns pass through at 1 token per token (lossless), assistant turns get pooled 8, 16, or 32:1. User turns carry the answer-bearing facts, so preserving them verbatim is worth spending the budget on. Second, training signal: the adapter learns from teacher-forced reconstruction of target tokens plus a forward-KL term that matches the frozen reader’s next-token distribution under full context versus compressed context. Roughly, train the writer so the reader behaves the same either way.
def write(segments, reader, adapter, k_per_role):
hidden = reader.bottom_layers(embed(segments)) # frozen
pooled = []
for seg in segments:
k = k_per_role[seg.role] # user=1, assistant in {8,16,32}
pooled.append(mean_pool(hidden[seg.span], k))
return adapter(concat(pooled)) # soft tokens
def read(soft_tokens, question, reader):
return reader.generate(inputs_embeds=cat(soft_tokens, embed(question)))
The prevailing assumption is that compressed context must round-trip through a human-readable form: a text summary, or OCR’d pixels, before the LLM can consume it. This paper shows the opposite. The consumer is a language model, so the compressed form should live in the model’s own embedding space and enter through the input-embedding interface unchanged. The cleanest evidence is not the headline benchmark but the role-allocation ablation on LongMemEval: preserving user turns and pooling assistant turns holds accuracy where a symmetric pooling collapses it.
The load-bearing finding is the ablation isolating what the mechanism does. At the same 4.62× compression on LongMemEval with a frozen Qwen2.5-7B reader, the full LatentPress writer scores 0.476; replacing the learned writer with plain mean-pooling of the same embeddings drops to 0.325; swapping the role allocation so user turns get compressed instead of assistant turns collapses to 0.087. So the win is the combination of learned writer plus keeping user turns lossless, not either alone.
•
On LongMemEval (500 oracle-evidence questions), LatentPress reaches 0.504 at 7.70× compression versus 0.490 for uncompressed evidence, 0.184 for LLM text summaries, and 0.426→0.312 for DeepSeek-OCR as its compression tightens. ICAE (In-Context AutoEncoder) at comparable rates lands at 0.452 (4.12×) and drops to 0.174 at 17.28×.
•
Generalizes across three frozen readers (Qwen2.5-7B-Instruct, qwen3-8b, Qwen3-1.7B); role-aware beats uniform pooling by +0.34 to +0.45 accuracy at matched compression.
•
On LongBench-QA long documents (no role structure), in-domain training lifts compressed readers above their raw-context baselines at 4× and often 8×, but 16× falls below raw on all three readers. Zero-shot cross-domain transfer only matches raw at 4×.
•
Efficiency: writing takes 43 ms per conversation (roughly 22× faster than the OCR route, 9–15× faster than text summarization); reading from soft tokens is 5–9× faster than reading raw context.
Reach for this when you’re shipping a chat assistant that has to remember months of conversation and you’re currently either truncating history or running a summarizer that loses precise facts. The recipe: train one small adapter per frozen reader on generic dialogue data (the paper uses 2,000 UltraChat conversations, no QA labels), keep user turns lossless, pool assistant turns 8–32:1. You get roughly the accuracy of feeding the full evidence at a fraction of the read latency, and the writer runs in a single forward pass rather than autoregressive generation.
Code is at context_softtoken_compress. The interface is orthogonal to retrieval, memory-update policies, and conflict resolution, so it can drop inside existing memory systems (the paper points to MemGPT, MemoryBank, Mem0 as complementary). One writer must be trained per reader, since soft tokens are tied to that reader’s embedding space.
If the consumer is a language model, don’t force the compressed context back through text. Write it directly into the reader’s embedding space, spend the compression budget where the facts actually live (short user turns, not long assistant turns), and leave the expensive decoder untouched. The gains only hold up to moderate compression, at 16× and beyond, verbatim detail starts to matter and raw context wins back.
•
Every frozen reader needs its own trained writer, so this doesn’t help if you swap decoders often or serve many models behind one memory store.
•
LongMemEval uses oracle evidence: retrieval is assumed solved. Pairing LatentPress with an actual retriever over a full haystack is left to future work, and it’s where real deployments live.
•
The role-based schedule is hand-specified and exploits the fact that answer-bearing facts sit in user turns. On unstructured documents (LongBench-QA at 16×) the interface degrades below raw, and Qwen3 readers show format pathologies (blank outputs, repetition loops, </think> leaks) as compression tightens.