Compact language and vision models designed to run on phones, laptops, and edge hardware — usually under a few billion parameters, often with aggressive distillation or quantization.
Small and on-device models: a guide to training, adapting, and running themA small model is useful only when the whole system meets its constraints. Parameter count affects storage and compute, but deployment also depends on memory traffic, training data, adaptation quality, latency, and whether the model can recover information omitted during compression or supervision.This guide separates four decisions: whether to pretrain or specialize a compact model, how to reduce reasoning-trace supervision, how to fit inference into constrained memory, and when to vary computation by input or hardware reliability. Start with the bottleneck you can measure rather than treating every efficiency technique as interchangeable.
Revisiting Complete Reasoning Traces for Post-TrainingStart here for a detailed account of endpoint-based reasoning supervision, including middle trimming, comparisons with full traces, extensions to GRPO and OPD, and the observed competence threshold for student models.Results center on Qwen-family models up to 32B and math or science reasoning; code tasks matched rather than beat standard SFT.
What are you trying to solve?
Choose between compact pretraining and task-specific specializationTraining a small general model from scratch and building a compact specialist solve different problems. Puro-2B focuses on accessible pretraining with consumer GPUs and low-precision training. KaLM-Embedding-V2 instead changes attention, pooling, objectives, and data curation for retrieval embeddings. Read them together to decide whether you need broad generative capability or a narrow representation model.
Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090Read this when the bottleneck is access to pretraining rather than inference. It reports a consumer-GPU FP8 recipe, cost measurements, curriculum variants, and released code, data, and weights.Performance and cost claims use the authors’ Puro-2B collection and evaluation protocol.
When is training a compact language model from scratch a plausible research option?Consider it when you need control over data, curricula, and the full training recipe—not merely inexpensive inference. Puro-2B reports training from scratch on up to 1.4 trillion tokens with FP8 on RTX 5090 GPUs, with its best model costing under $6.9K under the paper’s accounting.Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090
How can a 0.5B embedding model improve without simply adding parameters?KaLM-Embedding-V2 combines bidirectional attention and mean pooling with weakly supervised pretraining, supervised fine-tuning, and contrastive distillation. Curated task categories, hard-negative mining, and reweighting concentrate capacity on difficult retrieval distinctions.KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model
Trim redundant supervision from reasoning tracesDistilling long reasoning traces into smaller students can consume context and training compute on detours, self-checks, and backtracking. Endpoint-based training keeps the problem setup and conclusion while dropping part of the middle. The key test is whether the student is capable enough to reconstruct omitted reasoning rather than losing necessary scaffolding.
Revisiting Complete Reasoning Traces for Post-TrainingStart here for a detailed account of endpoint-based reasoning supervision, including middle trimming, comparisons with full traces, extensions to GRPO and OPD, and the observed competence threshold for student models.Results center on Qwen-family models up to 32B and math or science reasoning; code tasks matched rather than beat standard SFT.
Can reasoning traces be shortened without discarding their useful structure?The featured overview reports that retaining the beginning and end while dropping roughly the middle 20% improved average results over full-trace SFT in tested Qwen settings. Prefix-only and suffix-only variants were worse, suggesting that setup and conclusion play complementary roles.Revisiting Complete Reasoning Traces for Post-Training
When is middle trimming a poor choice for a small reasoning model?Avoid assuming redundancy when the base model is weak, traces are unusually short, or essential information appears in the middle. The reported gains emerged only above a competence threshold, and aggressive trimming degraded results. Test the cut rule on held-out tasks before applying it broadly.Revisiting Complete Reasoning Traces for Post-Training
Fit inference to real memory and device constraintsA nominally small model can still fail deployment when weights exceed DRAM or compression damages the adapted task. LeanStream addresses weight movement by refining load and cache priorities from partial GPU results. The factory-agent study treats deployment as measured sub-network selection after retrieval-grounded adaptation, rather than assuming the smallest network is best.
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM InferenceUse this when model weights exceed device DRAM and storage traffic dominates generation. It presents a speculate-and-refine mechanism for overlapping GPU work with SSD or flash I/O.Reported comparisons are against prior on-device systems on the implemented mobile and embedded platforms.
How can an on-device LLM overlap storage I/O with computation when weights exceed DRAM?LeanStream makes early sparse-execution predictions, then refines computation, loading, and cache-retention priorities as partial GPU results arrive. This is designed to avoid serialized execution and redundant transfers. It reports lower memory use and higher token throughput than prior systems on mobile and embedded implementations.LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
How should one model family be assigned across heterogeneous edge devices?Measure after adaptation. In the manufacturing-manual case, general capability declined with parameter count while judged retrieval-augmented answer quality did not follow size cleanly. The selector chooses one sub-network per device using answer quality, on-device throughput, a capability floor, and a memory budget.Measurement-Driven Sub-Network Selection for On-Premise Retrieval-Augmented Factory Agents
Spend extra computation only where it helpsFixed computation can waste resources on easy inputs, while unreliable hardware can make a cheap answer unsafe to accept. RecurTrace learns when a looped language model should add latent depth. RACE-AIMC instead accepts or defers outputs from an imperfect analog accelerator under a certified conditional error bound. These are separate control problems, not a combined architecture.
RecurTrace: Adaptive Latent Reasoning with Loop-Time MemoryUse this to study adaptive latent computation: shared layers revisit earlier loop states, while a learned halting head allocates more depth to inputs predicted to benefit.The matched-compute result is on MathQA with a shared looped backbone; other comparisons span 0.6B to 8B models.
How can a compact model vary reasoning depth without generating more reasoning tokens?RecurTrace repeats a shared block, lets each iteration attend to earlier loop states, and uses a learned halting head to decide whether to continue. On a shared MathQA backbone it averaged 2.0 loops and exceeded the best fixed-depth result by 2.2 points at matched compute.RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory
How can edge inference save energy without blindly trusting noisy analog hardware?RACE-AIMC selects one accelerator offline, computes an upper bound on its error rate when it answers, and applies a lightweight online accept-or-defer check. In simulations, certified bounds stayed below a 10% target while about 70.9% of inputs were answered directly.RACE-AIMC: Selective Inference for Heterogeneous Analog In-Memory Accelerators at the Edge