How large language models are built: data curation and mixtures, pretraining objectives, scaling laws, instruction tuning, preference learning, and the full post-training stack.
LLM pretraining and post-training: a mechanism-first reading guideBuilding an LLM is a sequence of coupled choices rather than one training recipe. Pretraining determines how parameters, compute, data, and architectural depth are used; post-training then converts demonstrations, rewards, teacher feedback, or interaction traces into behavior. A useful comparison starts with the bottleneck: compute, supervision density, executable environments, or behavioral discrimination.The readings focus on mechanisms that change one of those bottlenecks. They show how to compare architectures under matched budgets, make small-scale pretraining reproducible, extract denser learning signals from outcomes or teachers, reduce redundant post-training data, recover agent environments, feed evaluation results into later training mixtures, and prevent safety supervision from teaching superficial refusal cues.
DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent TrainingStart here for a detailed account of DRACO's dynamic rubrics, cited-turn attribution, and closed-form GRPO reweighting. It also explains the key ablation: step credit helped substantially with rollout-specific rubrics, but little or negatively with fixed rubrics.Tested on AppWorld with transfer to τ-bench; citation correctness was not human-validated.
What are you trying to solve?
Spend pretraining compute more deliberatelyTraining efficiency can come from three different levers: reuse depth while holding major budgets fixed, make layers optional during training and inference, or optimize the whole hardware-and-data recipe for an accessible platform. Read these separately: they answer architectural, sparsity, and reproducibility questions rather than forming a single recipe.
SMELT: Scaling Laws for Compute-Matched MoE Looped TransformersRead for a controlled architecture comparison: SMELT matches per-token FLOPs, non-embedding parameters, and KV cache before testing shared-layer looping. It supports claims about sparse MoE looped Transformers, not dense architectures or every kind of parameter sharing.Four sparse-MoE scales were studied, up to 54B non-embedding parameters.
Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and InferenceRead for evidence that tuned layer-dropout distribution, scheduling, and optimizer choices can lower training cost and enable inference shortcuts. The abstract reports broad experiments but not the exact tuning settings needed for reproduction.Models span 271M–8.2B and up to 160B tokens; all pretraining ran on Cerebras CS-3.
Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090Read when planning accessible from-scratch pretraining: it combines FP8, RTX 5090 hardware, data choices, and optimization in an open Puro-2B pipeline. Its cost law and performance comparisons apply to the released model family and evaluation protocol.Puro-2B used up to 1.4T tokens; the best reported run cost under $6.9K.
Does reusing Transformer layers still help when compute, parameters, and KV cache are matched?SMELT reports that a sparse MoE model which revisits its middle layers twice improves compute scaling relative to an unlooped baseline matched on per-token FLOPs, non-embedding parameters, and KV cache. The reported advantage is strongest for code and grows with sample length and in-context examples.SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Extract more supervision from outcomes, teachers, and tracesPost-training can be limited by a coarse outcome score, an imperfect teacher, too few useful rollout states, or long demonstrations full of redundant steps. These readings address different parts of that problem: assign credit to cited agent turns, turn checkpoint movement into token targets, use weak teachers only to steer verifier-supported updates, and test how much prompt or trajectory data is actually needed.
DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent TrainingStart here for a detailed account of DRACO's dynamic rubrics, cited-turn attribution, and closed-form GRPO reweighting. It also explains the key ablation: step credit helped substantially with rollout-specific rubrics, but little or negatively with fixed rubrics.Tested on AppWorld with transfer to τ-bench; citation correctness was not human-validated.
RISE: Recursive Improvement via Self-Extrapolating Policy DistillationRead for a route from sparse reward to dense supervision: RISE creates a synthetic teacher by extrapolating policy changes induced by RLVR. It reports cross-domain gains, but the abstract does not give settings for parameter- versus logit-space extrapolation.Requires verifiable outcomes, successive checkpoints, and a trailing anchor.
Eliciting Weak-to-Strong Generalization with On-Policy Reverse DistillationRead when a weaker or earlier model is available but should not cap the student. OPRD uses the teacher's policy shift to rescale verifier-supported student updates, aiming to accelerate the student's own optimization rather than imitate the teacher as a fixed target.Reported settings include successive-model, multi-teacher, and strong-to-weak language-model transfer.
Rethinking On-Policy Distillation of Large Language Models II: One Training ExampleRead to question the assumption that OPD needs many prompts. Its state-coverage measurements separate prompt count from optimization speed and motivate semantically diverse query selection. The reported query counts are empirical findings, not universal budgets.Results span the studied task domains, model families, and multi-teacher OPD configurations.
Revisiting Complete Reasoning Traces for Post-TrainingRead before assuming that every reasoning token must be retained. The paper compares complete, truncated, and endpoint-based traces and reports benefits across SFT, RL, and OPD, motivating controlled tests of trace redundancy rather than automatic full-trace collection.The abstract does not identify which tasks, models, or truncation levels generalize best.
How can an agent learn which steps mattered when no programmatic verifier exists?DRACO generates rollout-specific criteria, scores them once per trajectory, and asks the judge to cite responsible turns. A closed-form rule then redistributes the GRPO advantage toward cited good steps on winning trajectories and cited bad steps on losing ones, without training an attribution model. Its ablations show that step credit depends on sufficiently specific, dynamic rubrics.DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training
How can training get dense guidance without treating a teacher as the final target?RISE extrapolates the change between a current checkpoint and a trailing anchor to create a synthetic teacher, combining verifiable outcomes with on-policy token supervision. OPRD instead measures a teacher's policy shift and amplifies only the student's verifier-supported updates aligned with it, so a weaker teacher guides direction without defining the endpoint.RISE: Recursive Improvement via Self-Extrapolating Policy DistillationEliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Can post-training use fewer prompts or shorter reasoning traces?The OPD study reports that semantically varied rollouts can cover many of the states reached by full-data training, while optimization remains slow; 16 queries reached 98.9% measured coverage in its experiments. A separate trace study reports limited benefit from full trajectories and gains from partial traces or endpoints across SFT, RL, and OPD. Together they motivate measuring state coverage and testing truncation separately.Rethinking On-Policy Distillation of Large Language Models II: One Training ExampleRevisiting Complete Reasoning Traces for Post-Training
Build reusable agent training systems from interaction evidenceAgent post-training needs more than static demonstrations. Terminal-Universe reconstructs executable workspaces from tool traces; Feyospace filters trajectories through execution checks in resettable cyber environments; NeoHorse records routing and interaction context, then uses evaluation feedback to choose later training data. These are distinct approaches to environment recovery, data validation, and iterative allocation.
Terminal-Universe: Turning Agent Trajectories into Scalable Terminal EnvironmentsRead for a concrete mechanism that turns terminal traces into partial workspaces, completes missing state, and synthesizes breadth- and depth-scaled tasks. It is most useful when recorded file operations expose enough workspace structure for reconstruction.Built from public terminal-agent traces; SFT used Qwen3.5-27B and terminal/code-agent evaluations.
Feyospace-v1: How the Cyber Mercury Seven Trained Frontier Cyber ModelsRead for a cyber-agent pipeline that treats executable environments, teacher access, and trajectory verification as the main constraints. Its useful contribution is combining resettable environments with execution-verified, audited long-context supervision.Reported results cover three checkpoints on CyberGym and pooled CTF suites.
NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing HarnessRead for a concrete evaluation-selection-update loop: a routing harness records capability demand and interactions, validated traces feed staged training, and benchmark feedback changes the next data mixture. It demonstrates an initial iterative loop, not unrestricted autonomous self-improvement.Evaluated with 4B and 9B models across eleven agent, tool-use, coding, and instruction benchmarks.
How can recorded terminal sessions become executable post-training tasks?Terminal-Universe replays file operations to restore pre-edit files, uses a completion agent to supply missing files and dependencies, then reconstructs the original intent and synthesizes new tasks. It expands breadth with cross-workspace dependency queries and depth with multi-round requirement refinement.Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments
Teach safety discrimination instead of refusal styleSafety tuning can accidentally make risky vocabulary a shortcut for refusal. Separating the explanation of why a request is unsafe from boilerplate refusal language exposes a data-curation lever: supervise the underlying distinction without repeatedly reinforcing the surface form that appears in every refusal.
How can safety tuning reduce refusals of benign prompts that contain risky words?The rationale-only study removes boilerplate refusal statements while retaining explanations of why harmful examples are unsafe. In its evaluated supervised-training setup, this reduces false refusals while maintaining comparable safety; related benefits also appear in its in-context-learning and inference-time mitigation evaluations.Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models