Safin-1 turns a recurrent language model’s own hidden state history into an addressable memory bank, then slots a learned Safety State into that same bank so safety behavior can be attached or detached without touching the frozen backbone, cutting jailbreak attack success by 42–52%.
You’ve shipped an LLM assistant on a linear-attention or hybrid recurrent backbone because you needed cheap long-context decoding. Two things now hurt you. First, once the fixed-size recurrent state overwrites an earlier fact, that fact is gone. Second, when you want to add safety behavior, your options are heavy: full fine-tune, LoRA adapters that mix into every weight matrix, or brittle inference-time steering. Both problems have the same root cause: the recurrent state is treated as a disposable compression of the past, not as a first-class place where capabilities can live. This paper argues state should be the substrate for both memory and capabilities, and it builds a concrete architecture that treats it that way.
The backbone is a Gated DeltaNet-style recurrence that keeps a single evolving matrix state per layer. The core idea, called MARCH, is to periodically snapshot that state (default every 512 tokens) and keep the snapshots around as an addressable bank of state anchors. Each anchor gets a small routing key derived from the hidden state at the snapshot point, so anchors are addressed by their contents, not by position. For every new token, a router computes a query, scores it against all causally visible anchor keys plus a learned null option, softmaxes, and additively fuses the weighted anchor readouts into the normal recurrent output. If nothing historical is relevant, the null wins and the layer behaves like plain Gated DeltaNet. Dense routing gets expensive at long context, so at inference they keep only the top-4 anchors per token, which they show is a state-level analogue of sparse attention over compressed memories rather than over tokens.
The second move is the interesting one. Because the bank is just “a set of matrix-valued states plus routing keys,” you can drop in states that were never checkpointed from any sequence. These persistent capability states are learned parameters that sit in the same bank at every layer and compete for routing probability against the context-derived anchors. Instantiated for safety, this becomes the Safety State: freeze the whole language model, train only these persistent states on harmful+benign examples, and you get a detachable module.
for layer in state_bearing_layers:
S = recurrent_update(S, x_t) # normal GDN step
if t in anchor_boundaries:
anchors.append(snapshot(S)) # checkpoint state
cands = visible(anchors) + persistent_states + [NULL]
pi = softmax([route_q(x_t) @ k for k in keys(cands)])
o_t = S @ q_t + sum(pi[j] * cands[j] @ q_t for j in cands)
Training the Safety State uses a trick worth noting: they take the same target response and prepend 0, 512, or 2048 tokens of benign context, so the state is forced to work whether the bank contains zero, one, or many context-derived anchors.
The usual way to add safety to a base model is to bake it into the weights (SFT, RLHF) or bolt on adapters like LoRA that co-mingle with every matmul. This paper argues the recurrent state itself is the right place to put reusable capabilities: keep the backbone frozen, put safety in a persistent state that the model’s own router decides when to attend to, and it becomes as attachable as a USB stick. The load-bearing evidence is not the jailbreak numbers in isolation, but the comparison to a training-matched LoRA control that sees the same data and updates.
The key result is the safety-vs-over-refusal comparison against the LoRA control at both scales. The persistent Safety State cuts average jailbreak attack success rate by 42.3% at 4B and 52.3% at 35B-A3B, while producing “substantially less over-refusal” than the rank-8 LoRA trained on identical data. That’s the finding that makes the thesis work: if LoRA had matched it, there’d be no reason to invent a new mechanism.
Supporting evidence, in order of how much they lean on it:
•
Architecture generality at 0.8B. Adding MARCH to three different recurrent backbones (Gated DeltaNet, Kimi Delta Attention, Gated DeltaNet-2) improves RULER needle-in-a-haystack scores in 11 of 12 backbone-length settings. On Gated DeltaNet specifically, six-task NIAH average jumps from 31.58 to 46.43, with the largest gains at 32K, beyond the 16K training length.
•
Scaling to real backbones. Applied to Qwen3.5-4B and Qwen3.5-35B-A3B via matched continual pretraining and SFT, the ten-benchmark macro-average moves from 66.79 → 69.20 at 4B and 76.25 → 78.35 at 35B-A3B. Gains concentrate on hard reasoning: MMLU-Pro jumps 9.28 points at 4B, GPQA-Diamond 6.56 points at 35B, AIME 2025 gains roughly 8 points at both scales.
•
The null option matters. Removing the “don’t retrieve anything” candidate consistently degrades every aggregate metric, so letting the router opt out is doing real work.
•
Sparse routing is nearly free. Top-4 routing at 128K tokens more than doubles end-to-end throughput vs dense routing and cuts core forward-backward runtime by roughly 10x, with modest NIAH loss.
Reach for this when you’re building on a linear-attention or hybrid recurrent backbone and want a safety layer you can ship, audit, and revoke independently of the base weights. The mental model: instead of retraining or attaching LoRA to add a safety policy, you train one small tensor per layer, drop it into a routing bank the model already uses for its own historical states, and let the router decide per-token whether to consult it. Removing safety later is deleting a file, not merging weights. The same mechanism generalizes to any capability you’d want as a hot-swappable module.
Artifacts are pointed to via an “artifact button” in the paper but no repo URL is included in the text provided here; the SFT data is Nemotron-Cascade-2-SFT-Data, pretraining used the Intern-S2 mixture, and safety training used STAR-1. Backbones are Qwen3.5-4B and Qwen3.5-35B-A3B. If you don’t already run a hybrid recurrent stack, most of this doesn’t apply directly, though the routing-bank idea could inspire similar module designs on other architectures.
If your backbone already keeps a recurrent state, treat that state as a first-class place to plug in swappable capabilities, not just as a rolling summary of the past. The safety win here isn’t fundamentally about safety. It’s that a frozen backbone plus a small learned state in the model’s own memory-routing pathway beats a same-budget LoRA on both robustness and over-refusal, because the router gets to decide per-token whether the capability is relevant.
•
The mechanism only exists for recurrent or hybrid recurrent architectures with an addressable state. On a pure attention Transformer there’s no equivalent “bank of prior states” to route over, so the Safety State idea doesn’t transplant directly.
•
The LoRA baseline is rank-8 with a specific coverage pattern (no expert or shared-expert MLPs at 35B-A3B). A more aggressive LoRA configuration might narrow the gap; the paper doesn’t sweep this.
•
Safety numbers are attack-success rates on five jailbreak benchmarks plus one over-refusal set. This says nothing about subtler failure modes (deceptive compliance, capability elicitation via long multi-turn context) that the Safety State was never trained against, and the paper doesn’t claim otherwise.