SynthSelf teaches a small model new skills without forgetting old ones by having a slightly larger same-family sibling rewrite each training example into the small model’s own distribution, then filtering rewrites that got the answer wrong before standard fine-tuning.
You’ve fine-tuned a 0.5B or 1.5B model to add a new domain skill (say, medical QA) because latency or on-device constraints rule out a bigger model. Two things go wrong: the new skill lands weakly, and benchmarks the model used to pass (coding, commonsense, instruction following) regress. This is Catastrophic forgetting, and plain Supervised Fine-Tuning on gold answers is the usual culprit because the gold text is written in a style the model doesn’t natively produce.
A recent fix, Self-Distilled Fine-Tuning (SDFT), has the model rewrite each gold demonstration in its own words (using in-context learning) and trains on that rewrite. The rewrite stays close to the model’s existing distribution, so updates are gentler. The problem the paper identifies: SDFT needs the model to be good at following in-context instructions. Small models aren’t. The paper reports SDFT’s self-rewrites in a tool-use setting average around 60% accuracy and drop chain-of-thought traces 8.2% of the time, so the “teacher” is feeding the student garbage.
The core observation, backed by their own measurements: how much a model’s output distribution shifts during fine-tuning (measured by KL divergence between the before and after policies) correlates strongly with both poor acquisition and forgetting (Spearman -0.85 against forgetting across 23 math datasets). Staying in-distribution is the lever.
So instead of asking the small model to rewrite its own training data, outsource the rewrite to a minimally larger model from the same family, for example use Qwen2.5-7B or 14B as the aligner for a Qwen2.5-0.5B student. Same family means the larger sibling’s natural phrasing is close to the student’s natural phrasing, so its rewrites land in-distribution for the student. Being larger means its rewrites are more often correct than what the student would produce itself.
Two further design choices matter:
•
Go off-policy: generate all the rewrites once as a static dataset, then do normal SFT on them. This is what lets you apply rejection sampling, keeping only rewrites whose final extracted answer matches the gold answer. The small-model student is very sensitive to label noise, so this filter is doing real work.
•
Formalize the aligner choice as: pick the aligner with the smallest negative log-likelihood of its rewrites under the student (closest in distribution), subject to its rewrite accuracy exceeding the student’s own and its distributional distance beating plain gold SFT.
D_hat = []
for (x, y_gold) in D_train:
y_hat = aligner.greedy_decode(rewrite_prompt(x, y_gold))
if extract_answer(y_hat) == extract_answer(y_gold):
D_hat.append((x, y_hat)) # keep in-distribution rewrite
# else: drop; noisy supervision hurts small students
student.sft(D_hat) # plain next-token SFT
No new hyperparameters, no on-policy training loop, no KL regularizer. Just: rewrite, filter, SFT.
Setup: Qwen2.5-Instruct students at 0.5B, 1.5B, 3B; four acquisition tasks (chemistry via SciKnowEval, a 4-subject science mixture, ToolAlpaca, and MedMCQA); retention measured across 14 benchmarks spanning instruction-following, commonsense, knowledge, code, and math.
•
SynthSelf wins acquisition on all four tasks across all three student sizes while matching the best baseline on retention. SFT, SDFT, DFT (Density Functional Theory), and Anchored SFT (ASFT) all trail on acquisition at these small scales.
•
Controlling for the fact that SynthSelf uses a larger model to generate data, the authors compare against off-policy and on-policy distillation that use the same aligner but condition only on the question (no rewrite prompt that includes the gold answer). SynthSelf still beats these by +5.7 to +16.9 percentage points on acquisition averaged across settings and student sizes. The authors read this as evidence that the in-distribution rewrite itself (not just the stronger teacher) drives gains.
•
Bigger isn’t better if it means out-of-family. Using Gemma 4-31B or a Qwen3.5-27B as aligner produces higher rewrite accuracy but larger distributional gap to the Qwen2.5 student, and acquisition goes down or flat. Same-family 7B/14B aligners use 62.9% to 76.9% fewer FLOPs for data generation than those bigger cross-family teachers.
•
Prompt-matched ablation: when SDFT is given the exact same rewrite prompt as SynthSelf, SynthSelf still beats it by +18.96 pp on acquisition and +1.68 pp on retention. The gain is from the aligner being a larger sibling, not from prompt differences.
•
Removing rejection sampling degrades both SynthSelf and the off-policy distillation baseline, confirming the filter matters.
•
One honest null: in the medical domain, SynthSelf barely beats plain off-policy distillation with the same aligner. The authors’ explanation is that the aligner already knows MedMCQA well enough that rewriting adds little.
•
If you are continually fine-tuning a small open-weight model (sub-3B) from a family that ships larger siblings (Qwen, Llama 3, Gemma, OLMo 2), it is worth testing a 2x-5x larger same-family model as a one-shot rewriter for your SFT data, filtering by answer correctness, and then doing ordinary SFT. The paper’s evidence supports this on three single domains plus one mixture, not on arbitrary tasks.
•
Do not reach for a stronger cross-family teacher by default. The paper’s Table 2 shows that even when a cross-family model produces more accurate answers, the resulting SFT data pushes the student further from its original distribution and hurts both acquisition and retention. The “smaller same-family” choice is doing real work, not just saving compute.
•
Use the KL / NLL gap between candidate teachers as a selection signal, not just teacher accuracy. The paper proposes picking the aligner that minimizes NLL of its rewrites under your student, subject to it being at least as accurate as the student. This is cheap to measure and matches what predicted downstream performance in their experiments.
•
If you already have an SDFT pipeline that works for your 7B+ models and are porting it down to a 1B class model, the paper gives a concrete reason it may silently stop working (self-rewrite quality collapses) and a drop-in replacement.
•
All students are Qwen2.5-Instruct at 0.5B/1.5B/3B. Behavior on other families or on larger students (where SDFT already works) is not evaluated here, and the authors position SynthSelf specifically against SDFT’s small-scale failure.
•
The method requires a larger model in the same family. If you are using a one-off or in-house model with no larger sibling, the paper suggests but does not demonstrate workarounds (for example, giving an out-of-family model the base model’s own answer in-context).
•
Gains over same-aligner off-policy distillation are domain-dependent. On MedMCQA the rewrite step adds little, likely because the aligner already masters the data.
•
Retention is measured on 14 fixed benchmarks; “no forgetting” means no measured regression on those, not a guarantee for arbitrary deployed capabilities.
•
Rewrites are generated greedily and filtered by exact-answer match against gold. Tasks without a clean verifiable answer (open generation, long-form writing) would need a different filter, which the paper does not study.