SpeakerMem-R1 handles memory in group chats by keeping two parallel stores: raw speaker-tagged messages, and derived person-level and group-level state records with source/owner tags. A Qwen2.5-3B Writer trained via Group Relative Policy Optimization (GRPO) converts messages into structured records, closing most of the gap to a much larger LLM writer.
Generic long-term memory for chatbots was built for one-on-one conversations. Compress the history, embed chunks, retrieve by similarity. That falls apart in group chats where five people discuss overlapping topics, reference each other’s opinions, and revise decisions over time. If Alice says “Bob agreed to X” and Bob later says “actually no,” a summary-based memory tends to collapse this into one confused fact. If you ask “what does every member think about X,” top-k retrieval returns the loudest three members and misses the quiet ones.
The paper frames two concrete failures. Message attribution: who said something versus who it’s about (Alice’s belief about Bob is not Bob’s self-report). State reconstruction: rebuilding the current or historical state of a person or group from clues scattered across time. Existing systems like Mem0, A-MEM, and HippoRAG compress or graph the history but don’t preserve these speaker relations, and on the multi-party benchmarks GroupMemBench, SocialMemBench, and EverMemBench, simple BM25 retrieval sometimes beats them.
Memory is split into two tracks. System 1 stores every message verbatim with speaker, time, and channel. System 2 stores four kinds of derived records: per-person stable facts (Core), per-person observations including one person’s view of another (Profile), group-level events and decisions (Interaction), and group norms (Insight). Every derived record carries a source (who said it), an owner (who it’s about), a scope (PERSON or GROUP), and a from_ids pointer back to the raw messages that support it. Updates never overwrite: they append a new node and link the old one as superseded, so the history stays queryable.
At query time, the system runs a four-step routine the authors call Anchor–Separate–Resolve–Compose: figure out which people/groups the question is about, pull records per row from both tracks, resolve which version of the state applies (current vs. historical), and hand a composed evidence bundle to a frozen answerer LLM. Retrieval budgets for the two tracks are independent.
The interesting training piece is the Writer. Instead of always calling a big model to produce structured records, they train a small local Qwen2.5-3B with reinforcement learning. The reward has three parts: (1) structural validity of the JSON action, (2) a novel SpeakerLevenshtein score that matches predicted records against reference records within owner buckets (so a fact about Alice can’t cancel out a missing fact about Bob), and (3) terminal QA gain measured as the accuracy lift from adding System 2 on top of System 1 alone. That last term isolates credit: questions already answerable from raw messages don’t reward the Writer.
for segment in conversation:
system1.append_verbatim(segment)
heads = system2.current_heads()
actions = writer(segment, roster, heads) # Add/Update/Noop
for a in validate(actions):
system2.apply(a, provenance=segment.ids)
# query time
Q = project(question, roster) # rows, issue, head/full
E1 = system1.retrieve(Q); E2 = system2.retrieve_by_row(Q)
answer = frozen_llm(question, compose(E1, E2))
On the three multi-party benchmarks, SpeakerMem-R1 reports binary accuracies of 47.9%, 69.2%, and 61.9% on GroupMemBench, SocialMemBench, and EverMemBench, beating the best mainstream baseline on each by 3.3, 12.4, and 9.4 points respectively. On SocialMemBench the score is essentially tied with feeding the entire conversation as context (69.4%), which is the practical ceiling. On EverMemBench’s public leaderboard the system reports 62.33%, ahead of EverOS (~60.08%) and RippleMem (~54.75%).
The Writer training result is the more targeted claim. On a held-out 305-question subset with the query and answer modules frozen, the RL-trained Qwen2.5-3B Writer reaches 68.20%, up from 57.38% for the same model trained by supervised fine-tuning, and within 3.28 points of a DeepSeek-based LLM writer at 71.48%. Ablations show removing either track hurts on all three benchmarks, and removing just the person layers or just the group layers also hurts, supporting the claim that both views contribute.
On the two-person LoCoMo benchmark, the system gets 70.85% overall but is weak on multi-hop (41.1%) and open-domain (40.6%) questions, which the authors flag as remaining gaps. Also worth noting: binary accuracy and token-F1 don’t always move together, because adding an extra wrong person to an answer can invalidate it under the judge while still overlapping tokens with the reference.
If you’re building conversational memory for multi-user settings (team assistants, group planning bots, social simulations), the concrete takeaway is that flat retrieval plus summaries is the wrong shape. Storing source and owner as separate fields, and keeping a verbatim track alongside derived records with provenance pointers, is a cheap architectural change worth testing against your current setup. The GitHub repo is linked.
The Writer training story is useful if you already run a memory pipeline with a big model doing structured extraction and you want to swap in a local 3B. The recipe (SFT on teacher trajectories, then RL with owner-bucketed matching plus terminal QA gain) closed 95% of the gap on this task. Whether that transfers to your domain is worth testing rather than assumed; the paper trains on only 15 networks and explicitly disclaims broad RL generalization.
If you’re primarily doing two-person long chats, the multi-party machinery is probably overkill. The LoCoMo numbers show LightRAG-style approaches are competitive there, and SpeakerMem-R1 doesn’t dominate.
One prerequisite the authors call out: the system assumes reliable rosters and clean speaker attribution. If your input has aliases, anonymous participants, or shifting membership, the owner/source machinery degrades.
The reported benchmark deltas come from comparing SpeakerMem-R1 (using its full pipeline including a trained Writer) against baselines running under their own official configurations, with the data, judge, and interface standardized. That’s a reasonable protocol but not a clean head-to-head on identical Writer budgets. The Writer RL result is on a 305-question held-out slice from the same benchmark family used for training-signal QA, so it’s an in-distribution result rather than evidence of general RL-for-memory transfer. Multi-hop reasoning and open-domain questions remain weak across benchmarks, so this is not a general memory solution, it’s a specifically multi-party attribution-and-state solution. Finally, some of the numbers here (e.g. GPT-5.6-luna, DeepSeek-V4-Flash) reference model names that suggest this is a future-dated preprint; treat cross-model robustness claims as tied to whatever those specific endpoints actually are.