Get Started
Home
Topics
Search
Library
Research questionHow can large language models cut training and inference costs without materially harming accuracy?Large language models require substantial computation both to train and to serve. Reducing the layers used during training or inference can lower cost but may degrade the model’s predictive accuracy.
AI
Inference Optimization
LLM Pretraining & Post-training
Machine Learning
Natural Language Processing
Research Paper
Statistical Machine Learning
Technology
Latest papersRecent research connected to this question, newest first.Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and InferenceThe evidence concerns layer dropout during transformer LLM pretraining and its use with early exit, intermediate-layer skipping, and self-speculative decoding. Experiments covered models from 271M to 8.2B parameters and datasets of up to 160B tokens; the source reports up to 25% lower training FLOPs and up to 1.5× faster inference with negligible reported accuracy loss. All pretraining experiments ran on Cerebras CS-3 systems, so hardware generality is not established here.research paper · Sep 4, 2026Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090The report studies Puro-2B models trained from scratch on up to 1.4 trillion tokens using FP8 precision on RTX 5090 GPUs. It releases the data, code, recipe, and weights under Apache 2.0, and reports costs below $6.9K, with a fitted estimate of about $4.4K to reach Qwen2-1.5B performance under its evaluation protocol; evidence is limited to this model collection and protocol.research paper · Sep 3, 2026UE5M3 FP4 Block Scaling for Stable Language Model PretrainingThe evidence comes from software-emulated FP4 pretraining of a Nemotron-H 8B model for nearly 190 billion tokens. It compares E2M1 payloads with E5M3 block scales against a Transformer Engine recipe, reporting training and validation loss, quantized-inference downstream results, and a throughput ablation; the results do not establish performance across other model sizes, hardware, or native FP4 implementations.research paper · Sep 2, 2026Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models CompressionThe source studies a Kronecker-based approximation of cross-layer Fisher structure in billion-parameter language models. Evidence covers quantization, sparsification, inter-layer corruption, and post-corruption fine-tuning across multiple model families; reported sensitivity correlates with performance degradation and recovery, with value projection layers often showing high sensitivity while other behaviors vary by architecture.research paper · Sep 2, 2026QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column OptimizationThe source studies weight-only post-training quantization using ternary weights, salient residuals with semi-structured 1:4 sparsity, and column-wise refinement. It reports results for Qwen3-14B and Llama3-8B on average accuracy and WikiText/C4 perplexity, plus a lookup-table kernel's per-token generation speed relative to FP16; broader models and deployment conditions are not established.research paper · Sep 2, 2026
Related questions
How can large language models allocate reasoning computation to preserve accuracy under limited training and inference budgets?How can idle inference resources reduce scarce-GPU training cost without biasing gradient estimates?How can LLM pretraining avoid sudden gradient explosions when scaling to larger models?How can autoregressive LLM decoding generate multiple tokens in parallel at large batch sizes without sacrificing quality?