DCVLM shows that for vision-language pretraining on already-curated data pools, adjusting the ratio of instruction-tuning to image-caption data matters far more than adding quality filters, with instruction-heavy mixes scaling from worst at 1B×6.25B tokens to best at 4B×25B tokens.
You’re building a VLM product. Your team has scraped and licensed a bunch of image-caption pairs, some interleaved web documents, a pile of visual instruction data, and text-only corpora. The default move borrowed from CLIP and language-model work is: run a quality filter (CLIP-score filtering, a text classifier, a perplexity model) and keep the top X%. This paper says that habit is mostly wasted effort in the VLM setting, because your source datasets already went through filtering upstream. The dominant baseline for open VLM data is FineVision, which unifies public sources but doesn’t systematically ablate what actually drives quality. DCVLM is the first controlled testbed that isolates filtering versus mixing versus scale for autoregressive VLMs.
The authors freeze the model recipe (InternViT-300M vision encoder, MLP projector, Qwen2.5 language backbone, joint training) and vary only the data. They assemble a pool of 160 public datasets across four types: image-caption pairs, interleaved multimodal documents, text-only corpora, and multimodal instruction-tuning data, totaling 6T tokens. They define a scaling ladder with four points (1B/2B/4B/8B models trained on 6.25B/25B/100B/200B tokens), keeping the pool 30× larger than the training budget at every scale. Evaluation is 52 benchmarks across 9 domains, filtered for stability and monotonic improvement from small to medium scale, and split into Validation (13), Core (33), and Extended (52) tiers.
They then run two axes of experiments. First, filtering: they try CLIP-score filtering with three encoders, text-quality classifiers (DCLM fasttext, Nemotron, Mixtral educational), and multimodal filters including UniFilter and Conditional Mutual Information filter. They distinguish local filtering (percentile within each source, preserving mixture) from global filtering (single threshold across the pool, implicitly reshaping the mixture). Second, mixing: they sweep three ratios along the image-caption to instruction-tuning axis (Caption-heavy 65/15, Balanced 40/40, Instruction-heavy 10/70) across a 3×3 grid of model size and token budget.
for mix in [caption_heavy, balanced, instruction_heavy]:
for model_size in [1B, 2B, 4B]:
for tokens in [6.25B, 12.5B, 25B]:
data = sample_length_proportional(pool, mix)
model = train(vision_encoder, projector, qwen_base, data)
score = eval_core_33(model)
# finding: ranking of mixes flips as (model_size, tokens) grows
The prevailing move imported from CLIP and text-LM work is to bolt a quality filter onto the pool and expect a few points of lift. This paper shows the opposite for VLMs. When your sources are already pre-filtered (as public VLM datasets are), extra filters give diminishing or negative returns; the real lever is the ratio between instruction-tuning data and image-caption pairs, and that ratio’s optimum shifts with scale. The load-bearing evidence isn’t the headline 8B number but the sub-pool ablation showing filter gains shrink from +2.4 pp to +0.6 pp as upstream pre-filtering rises from 25% to 100%.
•
Filtering is nearly a no-op on pre-filtered pools. Across 60+ filter configurations at small and medium scale, the best result was SigLIP-2 global image-text filtering at +0.8 pp over no filter. The controlled sub-pool experiment nails the mechanism: gains drop from +2.4 pp (25% pre-filtered) to +1.3 pp (65%) to +0.6 pp (100%). Local vs global filtering diverge because global cuts silently reshape the mixture, and the mixture is what actually moves the needle.
•
Mixture rankings flip with scale. At 1B × 6.25B tokens, Caption-heavy wins and Instruction-heavy is worst. At 2B × 25B and 4B × 25B, Instruction-heavy is best. Picking the mix at small scale would systematically choose the wrong recipe.
•
Instruction data tolerates repetition. At the medium scale, 2× repeats of instruction data drop performance ~1.5 pp; 4× repeats still beat the base mix. The mix advantage outweighs moderate repetition cost, breaking down around 8×.
•
Pretraining ranking survives SFT. Across 54 supervised-fine-tuning runs on LLaVA-665K and Mammoth-VL-12M, pretraining and post-SFT scores correlate at Pearson r = 0.99, so pretraining-only evaluation is a valid proxy. Switching from Qwen2.5-Base to Qwen2.5-Instruct preserves mix rankings (r = 0.97).
•
Headline scaling. DCVLM-Baseline (10% caption / 5% docs / 15% text / 70% instruction) beats FineVision on the 33-task Core suite by +0.3 / +1.1 / +4.7 / +5.4 pp at the four scales. At 8B × 200B tokens, it reaches 63.6% vs 58.2% for FineVision, and a 4B model at 100B tokens beats an 8B FineVision model at 200B, a 4× compute reduction.
Reach for this when you’re assembling a VLM pretraining mix from public sources and your instinct is to tune a CLIP-score threshold. The paper’s guidance: skip the extra filter, spend that engineering budget on sweeping the caption-to-instruction ratio at the scale you’ll actually ship, and validate at more than one point on the ladder because small-scale winners lie. Instruction-heavy (~70%) is the right default at 2B+ models with 25B+ tokens; caption-heavy is right only at very small scales. Moderate repetition of your instruction data (up to ~4×) is fine.
The authors release the full 160-dataset pool, the 52-benchmark eval suite, checkpoints at all four scales, and training infrastructure at GitHub and the project site. The overall release is CC BY-NC-SA 4.0 (non-commercial), and individual source datasets carry their own licenses (a long table in the appendix lists them; several are non-commercial or unspecified, so downstream commercial use requires per-source auditing).
On already-curated multimodal pools, mixture ratios are the lever and filters are theater, but only if you tune the mix at the scale you’ll deploy at. Small-scale sweeps will hand you the wrong recipe because instruction-heavy mixtures scale steeper than caption-heavy ones. Budget your ablations across the ladder, not just at the cheap end.
•
The “filters don’t help” claim is conditional on starting from pre-filtered public sources. The sub-pool experiment shows filters do help (+2.4 pp) when the input is mostly raw. Teams curating directly from raw web crawls are outside this paper’s regime.
•
Mixing was studied along one axis (caption vs instruction) with text-only and multimodal documents pinned at 15% and 5%. Whether those pinned components are also scale-sensitive is untested here.
•
All results use the InternVL3-style architecture and Qwen2.5 backbones. Robustness across two Qwen variants is shown, but a genuinely different family (e.g. a non-Qwen LM or a different vision encoder) could shift the ranking, and post-training beyond simple SFT (RLHF, reasoning-style tuning) isn’t evaluated.