DataPrep-Bench evaluates how well LLM-based pipelines both build SFT data from raw domain sources and score candidate datasets before training, and finds that piling synthetic domain data on top of a general instruction corpus often hurts downstream accuracy across most generators.
You’re building a domain assistant (finance, medical, legal). You point Claude Opus or GPT-5.2 at a stack of textbooks, ask it to generate QA pairs, and mix them into your Supervised Fine-Tuning run on top of a general instruction set like Dolly-15k. You expect a lift. This paper shows that, more often than not, you get a regression, and no existing per-sample quality scorer would have warned you. The dominant baseline in this space is per-example scoring (QuRating, Superfiltering, perplexity) validated on one backbone in one domain; the authors argue that framing misses the real question, which is whether a whole candidate dataset will actually improve a fine-tuned model.
The benchmark has two tracks that share the same base models, the same downstream tests, and the same training recipe, so results are comparable end-to-end.
•
Construction track. Every method gets the same folder of Markdown textbooks (converted via MinerU) across six domains: General, Math, Science, Medical, Finance, Law. Each method emits a QA dataset of whatever size it wants. That dataset is fine-tuned jointly with Dolly-15k on either Qwen2.5-7B or Llama-3.1-8B, and the resulting model is scored on held-out domain benchmarks. Methods span three families: pipeline-based (DataFlow), single-shot LLM prompting, and agent-based generation.
•
Quality-evaluation track. For each domain the authors assemble a pool of 8–14 public SFT datasets (mix of in-domain and out-of-domain), fine-tune the base model on each one, and record the true downstream score. A quality metric takes each candidate dataset and emits a scalar; the metric is judged by Pearson correlation between its scalars and the actual post-fine-tune scores. This tests whether a metric can rank datasets before you pay the fine-tuning cost.
•
Their construction baseline, Data-Construction-Skill, is an agent with a reusable “skill” layer instead of a one-off prompt. The skill fixes output schema, filtering rules, coverage tracking, and validation, while the agent still plans over the corpus.
for chunk in structure_aware_chunk(book):
if not chunk_has_reusable_knowledge(chunk):
continue
# skill defines schemas + validity rules for each type
qa += gen_concept_qa(chunk) # definitions, rules
qa += gen_reason_qa(chunk) # short grounded reasoning
qa += gen_case_qa(chunk) # applied scenarios
qa = validate_and_dedupe(qa) # skill's rubric
log_progress(chunk) # resumable
•
Their evaluation baseline, DAS (Distributional Alignment Score), encodes each candidate dataset with Qwen3-Embedding-8B, samples 5000 rows, and computes MMD (Maximum Mean Discrepancy) against a fixed in-domain proxy dataset (e.g., ODA-Math-460k for Math). The score is the negative MMD: closer distribution to the proxy means higher predicted utility. The justification comes from a domain-adaptation bound where MMD between source and target upper-bounds the target risk. Using a proxy rather than the test set avoids benchmark contamination.
The prevailing habit in data curation is to score each training example with a quality classifier or perplexity model and aggregate. This paper shows the opposite. Dataset-level distributional distance to a domain proxy predicts fine-tuning outcomes better than any per-example quality score the authors tested, and per-example scorers frequently flip sign across (domain, model) cells and would actively recommend worse data. The load-bearing evidence is not the headline construction lift; it is that DAS is the only metric clearing r>0.70 in Math, Science, and Medical simultaneously while every per-sample scorer specializes to one or two domains.
•
The load-bearing finding: DAS is the only metric that clears r>0.70 in Math, Science, and Medical simultaneously, and leads or ties for the lead in four of six domains. On Math it reaches r>0.93 on two of three base models. Competing quality scorers each have a domain where they look fine and others where correlation flips negative — meaning they would recommend the worse dataset. That sign-inconsistency, not average correlation, is what makes them unsafe.
•
Synthetic domain data often hurts. On Llama-3.1-8B, most generators drag Math and Science scores below the Dolly-only baseline. Since the only variable is the domain data, that data is the cause. This is the finding surface-level scorers would miss.
•
No construction family wins everywhere. DataFlow variants lead in structured domains (Finance, Law on Qwen); agent-based generators lead in reasoning-heavy domains (Math, Medical); Data-Construction-Skill is strongest where knowledge extraction dominates. On Llama-3.1-8B Finance, Skill lifts the Dolly-only baseline by nearly 20 absolute points (15.1 → 34.2 avg).
•
Finance and Law are hard for everyone. No metric reaches strong average correlation there. The candidate pools are structurally imbalanced (one in-domain vs. seven out-of-domain) and the benchmarks test professional conventions that generic embeddings miss.
•
DAS is not the cheapest option, but it is the cheapest option that works, roughly 1.4–1.5× faster than BERTVendi and Deita-Quality, both of which score worse.
Reach for this when you’re deciding whether to fine-tune on a candidate SFT corpus for a domain assistant. Instead of averaging a per-sample quality classifier, encode 5000 rows of the candidate and 5000 rows of a trusted in-domain reference with a strong embedding model, compute MMD with a Gaussian kernel, and rank candidates by negative MMD. On Math and Medical this actually predicts post-fine-tune accuracy; on Finance and Law, don’t trust any single metric yet.
The authors release the raw Markdown source corpus, the candidate pools with ground-truth fine-tune scores, and Data-Construction-Skill itself as a downloadable skill package for Claude Code-style agents. The DAS pipeline needs only an embedding model and a proxy dataset. The paper doesn’t specify a code repo URL for DataPrep-Bench beyond the skill artifact.
Score the whole dataset against a trusted in-domain reference, not each example against a quality classifier. Per-sample scorers are trained on one backbone in one domain and silently flip sign elsewhere; distributional distance to a proxy is a cheaper, more honest predictor of what fine-tuning will actually do, at least when the target domain has a decent public reference corpus.
•
DAS collapses in Finance and Law, exactly the specialized domains where you might most want a dataset ranker. Generic embeddings don’t capture professional conventions, and imbalanced pools give the metric little to rank against.
•
Every construction method decides its own output volume, so the construction leaderboard mixes “quality per sample” with “how much the agent decided to emit.” A method that writes 300K samples has more fine-tuning signal than one that writes 5K, and the paper does not disentangle these.
•
All construction results use two 7–8B base models. Whether the “synthetic data hurts” finding survives at larger scales, or with continued pre-training rather than SFT-on-top-of-Dolly, is untested.