NCP-ArchPreview adds a Next Concept Prediction (NCP) objective alongside next-token prediction: the model compresses groups of hidden states into discrete concepts via Product Quantization, predicts the next concept, then feeds it back to guide token generation. At 8.9B parameters, it matches OLMo-3-7B’s final pretraining loss using only 51.3% of the tokens.
When you pretrain a standard LLM, supervision only ever fires on individual tokens. Any higher-level structure the model builds (semantic concepts, plans, world state) shows up as a side effect of getting the next word right. There’s no gradient that says “you should have known this paragraph was heading toward a conclusion.”
Other groups have tried to fix this. Multi-Token Prediction (MTP) adds auxiliary heads that predict several future tokens, but the loss is still measured on surface tokens. Large Concept Model predicts sentence embeddings from a separately trained encoder. JEPA (Joint Embedding Predictive Architecture) predicts latent representations for images and video. What’s been missing is a language model where a learned discrete concept space is a first-class prediction target during pretraining at real scale. This paper is the biggest attempt so far: 8.9B parameters, 5.73T tokens, built on the OLMo-3-7B backbone so the comparison is apples-to-apples.
The direct predecessor is ConceptLM, which introduced the NCP objective at up to 1.5B params from scratch (and as continual pretraining on an 8B model). NCP-ArchPreview scales that idea to a full 8.9B pretraining run.
The backbone is split into three pieces: a Token Encoder (16 layers), a Concept Module (8 layers), and a Token Decoder (16 layers). Tokens flow through the encoder normally. Then every group of k=4 consecutive hidden states gets mean-pooled into one continuous concept vector.
Those concept vectors get quantized against a learned codebook using Product Quantization: each vector is split into 32 segments, each segment is snapped to its nearest entry in a small per-segment codebook of 128 entries. That gives an effective vocabulary of 128^32 combinations without a giant monolithic embedding table. The Concept Module then autoregressively predicts the next concept as a weighted mix of codebook entries (differentiable, no argmax). The predicted concept is repeated k times, causally shifted, and added into the Token Decoder’s hidden states. The decoder then predicts the next token as usual.
On top of this the authors add hierarchical residual connections adapted from MUDDFormer: within each module, layers can pull from any earlier layer with input-dependent weights (IRC); across modules, encoder/concept/decoder streams can inject into each other (CRC).
Three losses are optimized jointly: standard next-token cross-entropy, a mean-squared-error next-concept loss, and a VQ codebook-fitting loss with stop-gradient so the codebook tracks the encoder’s distribution without pulling on it.
h = TokenEncoder(tokens) # token hidden states
c = mean_pool(h, k=4) # continuous concepts
d = product_quantize(c, codebooks) # nearest codewords (for VQ loss)
u = ConceptModule(c[:-1]) # predict next concept
c_hat = sum(softmax(head_s(u)) * E_s # differentiable expected codeword
for s in segments)
h_fused = h + causal_shift_repeat(c_hat, k)
logits = TokenDecoder(h_fused)
loss = NTP(logits, tokens) + a*MSE(c_hat, c) + b*VQ(c, d)
Optimization uses Muon optimizer for matrix parameters and AdamW for the rest.
Faster convergence on the same data. Trained on identical Dolma-3 tokens as OLMo-3-7B, NCP-ArchPreview hits OLMo-3’s final Stage-1 loss after only 51.3% of the tokens (1.95x speedup) and ends 0.091 lower. Stage-2 shows a smaller but consistent 1.51x speedup.
Downstream, after Stage-1, the macro-average across ~26 benchmarks is +2.45 points over OLMo-3-7B, with the biggest lifts on math (+3.75 avg, +5.99 on GSM8K), non-STEM multiple choice (+4.63), and code (+2.64). After Stage-2 the aggregate gain shrinks to +0.59 points and some code benchmarks regress; the authors attribute this to Stage-2’s data mixture containing only ~10% code.
Ablations isolate the contribution. Compared against a parameter-aligned 40-layer vanilla OLMo-3 baseline and a compute-aligned 34-layer baseline, NCP-ArchPreview beats the compute-matched one clearly and approaches the parameter-matched one while using only 85% of its compute. Progressively adding the Concept Module, then the hierarchical residuals, then the NCP loss each lowers training loss. Scaling-law fits across FLOPs budgets suggest a 1.74x compute efficiency gain over OLMo-3.
The concept space is reusable. Freezing the 8.9B backbone and updating only the 17M VQ parameters (codebooks + prediction heads) gives a domain-adaptation interface competitive with LoRA at matched parameter count: on math adaptation, VQ-only training beats LoRA on target performance (+4.27 vs +3.15) and also improves the general average (+0.39) where LoRA regresses (-0.42). On code adaptation, VQ is the only method that doesn’t tank MBPP+. On knowledge (TriviaQA), VQ underperforms full/LoRA fine-tuning because factual associations live in the FFN weights VQ doesn’t touch.
Speculative decoding. Injecting the concept representation into a block-parallel drafter based on DFlash2 raises mean accepted length by +4.17% on average (up to +7.59% on HumanEval) with only 0.04M extra drafter parameters.
A training-stability note. Combining Muon optimizer with OLMo-3’s layer-wise Q/K normalization produces attention-logit blowups concentrated in a few heads. Per-head Q/K normalization fixes it, but the main runs kept layer-wise normalization for controlled comparison.
•
If you’re evaluating architectural alternatives to plain NTP at scale, this is currently the largest data point that a discrete-concept auxiliary objective actually converts into pretraining efficiency and downstream gains, not just a loss curve trick. The compute-aligned and parameter-aligned baselines are the ablations to look at, not the headline vs. OLMo-3 number.
•
If you fine-tune open-weight models for a domain, the VQ-only adaptation result is worth testing: 17M trainable params, no added parameters, and less catastrophic forgetting than LoRA in their math and code experiments. The prerequisite is that you’re using a model with this kind of concept module in the first place, which today means their released checkpoints. The authors release weights, drafters, and intermediate checkpoints on Hugging Face.
•
If you’re building speculative decoders, the pattern of feeding a coarser “where is this going” signal (here, the last completed concept) into a block-parallel drafter is a cheap add-on worth trying. The gain is modest (~4%) but the parameter cost is negligible.
•
Treat the Stage-2 downstream numbers cautiously. The paper itself flags that lower pretraining loss doesn’t cleanly convert to downstream gains when the training data mixture is skewed away from the eval distribution. Don’t extrapolate the Stage-1 macro-average gain to what you’d see after your own post-training.
•
Watch for optimizer/normalization interactions if you build on this. The Muon + layer-wise Q/K norm instability is real and reproduces in their setting; per-head normalization is the fix.
•
No long-context training yet. Everything is at 8,192 tokens. The authors note this is a natural next step (the concept sequence is 4x shorter, so long context should benefit) but it’s not evaluated.
•
Stage-2 gains are small and uneven. The +0.59 macro-average hides regressions on several code and multiple-choice tasks. The pretraining-loss advantage doesn’t translate uniformly to downstream.
•
Comparison scope. The primary baseline is OLMo-3-7B and same-family variants. Broader comparisons against other recent 7-9B open models aren’t part of the report.
•
The direct concept feedback path bypasses discretization. The predicted concept fed to the decoder is a differentiable weighted sum of codebook entries, not a sampled discrete concept. So “discrete concept prediction” describes the training target, not the runtime information bottleneck.
•
The VQ domain-adaptation story is strong on math and code, weaker on factual knowledge, exactly where you’d expect (FFN memories aren’t being updated). Pick the adaptation method to the task.