A separate “weight-writer” network reads a batch of facts and emits a LoRA adapter for a frozen LLM, teaching it those facts without touching its weights. Across width, depth, and target-model size, this beats fine-tuning on out-of-distribution generalization, and the OOD gap widens as the target model scales.
You run a support-bot on top of a base LLM and every quarter your company ships 3,000 new policy facts. Today you either stuff them into a Retrieval-Augmented Generation context window at inference, or you fine-tune a LoRA adapter. Fine-tuning tends to make the model forget other things and it generalizes poorly when a user phrases a question differently from your training data. This paper studies a third option: train a Hypernetwork whose job is to produce the adapter, given the facts as input. The frozen base model never changes; the adapter is generated fresh from whichever facts you feed in. The dominant baseline for this comparison is standard LoRA fine-tuning of the base model on the same fact corpus.
The setup: freeze a target LLM (they use Qwen2.5 sizes from 0.5B up to 14B). Train a transformer, from random init, that takes a small set of natural-language facts and outputs the A and B low-rank matrices of a LoRA adapter applied to the later half of the target’s layers. During training, each example gives the hypernetwork 4 facts (1 relevant to the question, 3 random distractors), plus a question; loss is cross-entropy on the answer, backpropagated through the frozen target into the hypernetwork’s weights. At eval time the model has no access to the fact corpus except through the generated adapter. Because the hypernetwork is initialized from scratch (not from a pretrained LM), the authors can cleanly vary its depth, width, and the target size, and read off scaling exponents. They build MegaWikiQA to have enough labeled examples for this: ~10M multi-hop QA pairs derived deterministically from Wikidata5M via random walks and hand-curated question templates, with three held-out domains (philosophy, linguistics, civil engineering) forming the OOD split.
# train step
facts = sample_fact_set(corpus, k=4) # 1 relevant + 3 negatives
q, a = example.question, example.answer
A, B = hypernet(facts) # generate LoRA factors
adapter = (alpha / r) * B @ A # low-rank delta
logits = frozen_llm(q, lora=adapter) # target weights untouched
loss = cross_entropy(logits, a)
loss.backward() # updates only hypernet params
The usual move when you want an LLM to internalize a fact corpus is to write those facts into the base weights via fine-tuning or LoRA. This paper shows the opposite is a better bet as models grow. Keep the base model frozen and learn a function that generates the adapter from the facts on demand; the injection then generalizes to unseen entities, rephrased questions, and new formats far better than baking facts into weights. The load-bearing evidence is not a single benchmark score but the slope of the log-log fits: the hypernetwork’s OOD scaling exponents get steeper than fine-tuning’s as target size grows.
The headline finding is a fitted Power-law scaling fit comparison across three adaptation methods at matched target sizes (0.5B to 14B). On in-distribution validation, plain fine-tuning wins slightly, with exponents -0.250 (LoRA FT) and -0.249 (full FT) vs -0.226 (hypernetwork). On out-of-distribution splits the ranking flips and the gap grows with target size:
•
OOD non-rephrased (held-out domains): hypernet -0.184 vs LoRA -0.151, full FT -0.183.
•
OOD rephrased (GPT-4.1 paraphrases): hypernet -0.107 vs LoRA -0.083, full FT -0.069.
•
OOD MCQ evaluation split (multiple-choice reformat): hypernet -0.171 vs LoRA -0.119, full FT -0.101.
Secondary results: scaling hypernetwork width and depth both give validation exponents near -0.09, roughly interchangeable; scaling the target model is worth about 2.5x more per unit compute (-0.226 vs -0.09). Number of injected facts per example helps mildly (-0.080). One capacity floor: the 0.5B target sits noticeably above its own fit, suggesting a minimum target size below which generated adapters just don’t land. And LoRA rank scaling saturates with an additive floor, so you can’t close the OOD gap by simply widening the adapter.
Reach for this when you have a large, evolving fact corpus (product docs, policies, medical or legal records) and you’re currently choosing between fine-tuning quarterly or piping everything through Retrieval-Augmented Generation. The pitch: train one hypernetwork on your domain once, then at inference feed it the currently-relevant facts and get an adapter back; the base model stays untouched, so you don’t accumulate forgetting across updates, and it holds up better when users paraphrase or when new entities appear.
Artifacts: MegaWikiQA dataset and code, including the deterministic multi-hop QA generator over Wikidata5M with domain-level OOD splits. This is directly reusable as a scaling-laws or knowledge-injection benchmark; the paper doesn’t ship trained hypernetwork checkpoints.
When you need a model to internalize a corpus that will keep changing, generate the adapter from the facts instead of baking the facts into weights. The in-distribution numbers slightly favor fine-tuning, so if your queries look exactly like your training data this isn’t the win. The payoff is generalization headroom, and it compounds with target model size, so the case gets stronger the bigger the base model you deploy on.
•
The hypernetwork itself gets large. At their top setting it was ~2.5B params generating adapters for a 1.5B target, which erodes the deployment story unless follow-up work shrinks it (weight sharing, distillation).
•
Held-out OOD domains happened to have higher base-model prior accuracy than in-distribution ones, so OOD non-rephrased loss is close to ID loss. The rephrased and MCQ splits are the real generalization tests; a truly novel domain the base model has never seen isn’t evaluated here.
•
Target sizes cap at 14B Qwen2.5 and questions are up to 4 hops from Wikidata5M. Whether the OOD-scaling advantage keeps widening at 70B+, or on deep compositional reasoning, is stated as open by the authors.