DataFlow-Harness wraps a coding agent so that instead of writing a throwaway Python script, it mutates a live, typed pipeline graph through validated tool calls, cutting measured cost 72.5% versus vanilla Claude Code while keeping pass rates near the script baseline.
You’ve shipped an internal agent that turns product-manager requests like “generate 10k synthetic QA pairs from these PDFs, dedupe, score, filter” into runnable data pipelines. Today the agent hands back a one-off Python file. Nobody on the data team can open it in your workflow UI, tweak the filter threshold, or audit which operators ran. If a step is wrong, someone re-prompts the agent and gets a different script back.
The dominant baseline here is repository-grounded Claude Code: dump the platform’s source into context and let the agent mimic its operators in free-form Python. It works, but the artifact is still a disposable script. This paper’s move is to constrain the agent to edit a persistent graph object that the platform already knows how to render, validate, and re-execute.
The pipeline lives in a backend as a typed graph: data sources, operator instances, edges between them, and per-operator input/output schemas. The agent never writes Python that is the pipeline. Instead, it issues structured mutations, add this operator, connect this edge, change this parameter, through an MCP (Model Context Protocol) tool layer. Every mutation goes through a request-validate-commit loop: the backend checks the graph is still acyclic and that adjacent operator schemas line up. If either check fails, the mutation is rejected before it ever lands.
Two pieces sit on top of the mutation API. First, at the start of each turn, the current pipeline state and the live operator registry are injected into the agent’s context, so it plans against what actually exists rather than hallucinated operator names. Second, DataFlow-Skills injects procedural playbooks, recommended construction sequences (schema inference, then operator selection, then parameter config) and compositional rules like modality matching. The MCP (Model Context Protocol) layer tells the agent what operators exist; Skills tell it how to compose them. A synchronized web UI renders the same graph, and any manual edit in the UI is committed to the same backend, so the agent’s next turn sees human changes without special handoff.
for user_turn in session:
state = backend.get_pipeline() # includes manual UI edits
registry = backend.list_operators()
ctx = inject(state, registry, skills) # MCP + procedural guidance
mutation = agent.propose(user_turn, ctx) # typed, not free-form code
if backend.validate(mutation): # DAG + schema check
backend.commit(mutation)
broadcast_to_ui(backend.get_pipeline())
The prevailing move for “agent builds a data pipeline” is to give the agent more context (the full repo, better prompts) and let it emit a Python script. This paper shows the opposite. Constrain the action space to typed mutations on a live, validated graph object, and the artifact you get back is the pipeline itself, not a description of one. The load-bearing evidence isn’t the headline pass rate; it’s the token and cost drop, structured mutations are dramatically more compact than the scripts they replace.
Moving from free-form scripts to structured graph mutation is what drives the efficiency story. Against repo-grounded Claude Code, which is the strong baseline, DataFlow-Harness cuts monetary cost by 42.8% and latency by 17.6% at essentially matched task success (93.3% vs 94.2% end-to-end pass on the 12-task benchmark, 10 trials each). Against vanilla Claude Code with no platform context, cost falls 72.5% and latency 49.9%. Total tokens drop 25.5% versus an MCP (Model Context Protocol)-only variant that has the tool layer but no Skills.
•
Stripping Skills (the MCP (Model Context Protocol)-only ablation) drops pass rate to 83.3%. The per-task breakdown shows Skills help most on procedurally ambiguous tasks like QA generation (18/30 → 29/30 runs), and barely matter on trivial rename/filter tasks where both hit 10/10.
•
On a harder textbook-to-VQA extraction workflow, the full system reaches 97.2% precision and 87.3% coverage, versus 89.3/80.1 for repo-grounded Claude Code. The gain concentrates on coverage, suggesting the agent assembles more of the specialized document-processing operators rather than filtering more aggressively.
•
Two downstream training case studies (single runs, not repeated): a math synthesis pipeline authored under the harness lifts fine-tuned Qwen2.5-32B-Instruct average accuracy from 54.5 to 55.7 at two epochs, with AIME gains from 25.1 to 35.9 (AIME24) and 21.6 to 34.5 (AIME25). A from-scratch general SFT pipeline lifts a nine-benchmark average from 61.5 to 63.8, mostly from code (MBPP 64.6 → 75.4).
Reach for this pattern when you’re shipping an agent that authors artifacts your team needs to keep: data pipelines, ETL DAGs, evaluation harnesses, RAG ingestion flows. The recipe is: define a typed graph schema for the artifact, expose CRUD-style mutations through MCP (Model Context Protocol) instead of letting the agent write raw code, add a validator that rejects mutations breaking structural or schema invariants, and layer procedural guidance separately from the tool spec. The Skills-vs-MCP (Model Context Protocol)-only split is the useful design lesson: exposing operators isn’t enough, you also need to encode how to compose them, and those two concerns are worth keeping in separate layers.
The paper describes the platform (backend, web UI, MCP (Model Context Protocol) tools, Skills, DataFlow operator ecosystem) but doesn’t in this text link a public repo, license, or the 12-task benchmark for external use. It builds on the existing DataFlow pipeline framework. If you want to replicate the approach, you’d rebuild the harness against your own platform; the paper is a design blueprint more than a drop-in tool.
If you want an agent’s output to be a first-class artifact in your platform, don’t ask it to write code that produces the artifact. Give it typed mutations on the artifact itself. Free-form code generation optimizes for the wrong thing: it maximizes what the model can express, when what you actually need is what your platform can validate, render, and edit tomorrow.
•
One model family (Claude Code on Claude Opus 4.7), one platform (DataFlow), 12 benchmark tasks. The pass-rate near-tie with the repo-grounded baseline isn’t a statistical equivalence claim, the authors are explicit about that.
•
Structural validation catches DAG cycles and schema mismatches. It cannot catch semantically wrong pipelines that happen to type-check, so “validated” is weaker than it sounds.
•
The downstream training results (math and general SFT) are single-shot case studies, one authored pipeline per condition, no seed variation. Treat the AIME and MBPP deltas as suggestive, not as causal estimates of pipeline quality.
•
The ablation compares MCP (Model Context Protocol)-only against the full system, so it can’t separately quantify how much of the win comes from Skills versus from the validation layer.