Puro-2B is a 2B-parameter dense LLM pretrained from scratch on 1.4T tokens for about $6.9K of rental-equivalent compute, using consumer NVIDIA RTX 5090 GPUs plus blockwise FP8 (blockwise), the MuonH optimizer, and Curriculum Model Averaging to approach Qwen2.5-1.5B quality.
If you want to test a pretraining idea (a new optimizer, a data curriculum, a precision format) you have two options today: rent an H100 pod for six figures, or study someone else’s released weights and hope you can guess what they did. Even “open-recipe” 3B models like SmolLM3-3B cost an estimated $719K to reproduce, and Llama-3.2-3B costs over $1.5M by the authors’ accounting. That’s out of reach for most academic groups and small labs.
Puro-2B is a full pretraining stack, data manifests, training code, checkpoints, and cost ledger, that gets a competitive 2B base model for under $7K on gaming GPUs. The point isn’t the leaderboard; it’s that the full pipeline is now cheap enough to actually rerun and modify.
The recipe is a co-design across five layers, each attacking a different bottleneck. The reader-facing intuition per layer:
•
Hardware. RTX 5090 has ~40% the raw FP8 throughput of an H200 but costs roughly a tenth as much to rent, giving ~2.7x compute-per-dollar. The tradeoff is no NVLink and only 32GB per GPU, so the authors modified NVIDIA’s driver to enable PCIe peer-to-peer and GPUDirect RDMA, recovering most of the intra-node and inter-node bandwidth.
•
Precision. All Transformer linear layers run in blockwise FP8 (blockwise) (128-value groups for activations, 128x128 blocks for weights) from step 0, no BF16 warmup. Master weights and optimizer state stay in higher precision. Cost: ~0.003 validation-loss penalty vs BF16. Benefit: 1.36x throughput.
•
Optimizer. MuonH wraps the Muon optimizer with a Frobenius-norm projection that pins each matrix to its initial radius. This makes the effective learning rate (update size relative to weight size) an explicit knob rather than an emergent quantity, which the authors argue matters more than the raw LR schedule.
•
Data curriculum + averaging. In Phase 2, examples within each source are sorted low-to-high by that source’s own quality score, then interleaved so preferred content appears late in training. Because late-training LR is small, the authors then run a constant-LR continuation and average the last 6 checkpoints (Curriculum Model Averaging).
•
Data selection via proxy models. Instead of one global quality score, they continue-train a Qwen3-0.6B proxy on each candidate data slice and read off a 15-benchmark capability vector to decide what to keep.
The curriculum-to-training handoff, in pseudocode:
# Phase 2 curriculum construction and training
for src in sources:
if src.has_quality_score:
src.examples.sort(key=score) # low -> high within source
else:
src.examples.shuffle(seed)
buckets = align_by_within_source_rank(sources, n=376)
for bucket in buckets: # linear LR decay
train_step(bucket, lr=schedule(step))
# Late stage: freeze LR, average
resume_from(step_218000, lr=constant)
final_weights = mean([ckpt_222100, ..., ckpt_222569])
The prevailing story about affordable pretraining is that you pick one efficiency lever, a better optimizer, or FP8, or cheaper hardware, and hope the gains compound. This paper argues the opposite. Cost-effective pretraining is a co-design problem: hardware choice constrains precision, precision constrains kernel shapes, optimizer constrains LR schedule, and curriculum constrains checkpoint averaging. No single lever gets you to $7K; the stack does. The cleanest evidence is the fitted Puro Cost Scaling Law, which shows the canonical run reaching a target performance at ~2.4x the cost-efficiency of the uniform-data recipe on the same hardware.
The load-bearing result is the cost-efficiency decomposition, not the benchmark table. Under matched-quality scaling-ladder fits, each layer contributes independently: hardware gives ~2.7x compute-per-dollar over H200, blockwise FP8 gives a quality-adjusted 1.34x throughput gain, MuonH gives a 1.19x compute-equivalent shift, and the Phase 2 curriculum-plus-CMA gives a 2.40x cost-efficiency multiplier vs a uniform-data run on the same fitted curve. Because each factor uses its own reference, they don’t multiply into a single end-to-end number, but they attack different bottlenecks.
Secondary evidence that the mechanism transfers:
•
Puro-2B (canonical, $6.9K) reaches an average of 43.5% on 4 math/code benchmarks and 63.0% on 11 reasoning/knowledge benchmarks, beating Qwen2-1.5B and Gemma-2-2B and closing most of the gap to Qwen2.5-1.5B.
•
The scale-down variant at $4.4K already exceeds Qwen2-1.5B on the aggregate.
•
The curriculum-vs-uniform difference survives supervised finetuning: on GSM8K, curriculum-initialized checkpoints beat uniform-initialized ones by +1.77pp (focused SFT) and +2.02pp (scaled SFT), and the broad-instruction 15-task macro-average improves by +1.59pp.
That last result matters because it rules out the boring explanation that curriculum-late-training just memorized eval-adjacent tokens; the gap persists after further training on different data.
Reach for this if you’re a small lab or an infra team that wants to rerun pretraining rather than just consume weights. The most transferable pieces: the FP8 blockwise-scaling recipe adapted for Blackwell’s MXFP8 path, the RTX 5090 driver tweaks for P2P and GDR (which are broadly useful for any consumer-GPU cluster), the MuonH effective-LR analysis (a lens on why some optimizers overtake others late in training), and the proxy-benchmarking protocol for data selection (train a 0.6B model for 2K steps on each candidate slice, read the capability vector, decide).
Everything is released Apache 2.0: model weights and 10 intermediate checkpoints on Hugging Face, the training code (a Megatron-Core fork), and the Kaiyuan-Spark data-processing framework. Some upstream datasets carry non-permissive terms (notably NVIDIA’s Nemotron data agreement, which forbids raw redistribution), so the release is data manifests plus reconstruction code, not always bulk downloads.
Affordable pretraining isn’t one clever trick; it’s five ordinary decisions that only pay off when you make them together. Cheap hardware needs a precision format that survives on it, which needs an optimizer whose LR schedule you actually control, which needs a data curriculum whose late tokens don’t get thrown away by decay. Pick three and the recipe breaks.
•
The cost number excludes almost everything except the final training run. Data acquisition, proxy experiments, failed runs, evaluation, and researcher time are all outside the $6.9K boundary. It’s a marginal-compute figure, not the actual project budget.
•
Contamination isn’t fully audited. The curriculum places higher-scored data late in training, and some of the reported gain over uniform ordering might come from benchmark-adjacent content appearing near the end. The post-training persistence weakens but doesn’t eliminate this concern.
•
The recipe is validated at 2B parameters and a scale-down curve. The authors explicitly don’t claim it extrapolates upward; RTX 5090’s 32GB memory would bite hard at larger scales, and the communication tricks assume a specific PCIe topology.