DCP audits AI research agents by testing whether a matched agent, given the same starting information but not the target run’s history, can independently reach the same numerical result, and separately whether truthful feedback beats a neutral placebo channel from a shared checkpoint.
Suppose an AI agent runs 87 experiments and reports a program that scores well on your held-out test. That score tells you the artifact is useful. It does NOT tell you that the agent’s research process deserves credit. Maybe a fresh agent given just the problem statement and public docs, with no experiment history, would land on the same answer through routine search. Maybe the experimental feedback the agent claimed to use didn’t actually help.
Existing evaluations of AI research agents like AlphaEvolve or AI Scientist mostly check the final score, or replicate published papers (PaperBench, MLE-bench). They don’t cleanly separate three questions about one result: is it useful, is it recoverable without the run’s private history, and does truthful feedback actually cause the gain? DCP (Discovery Certification Protocol) is a protocol for auditing those three questions with pre-registered rules and a deterministic offline verifier.
The cost of conflating them: overclaiming discovery. A benchmark win can be re-derivable in an afternoon by any competent agent, or driven by prior knowledge in the base model rather than by the reported experimental loop.
DCP frames every audit around five registered objects: background knowledge K (model, tools, starter code, public methods), initial task observations E_0, the target run’s private research history L* (its hypotheses, experiments, scores), the final artifact A*, and a machine-checkable recovery predicate P (validity rules plus a score threshold). The boundary between E_0 and L* follows provenance: anything present before the run starts is E_0, anything the agent generated by taking an action is L*.
The audit runs up to three gates:
•
Gate 1: on sealed evaluation, does A* beat the baseline by at least a pre-registered margin δ_min? Standard held-out validation.
•
Gate 2: give a fresh challenger agent the same K, E_0, tools, budget, and even the exact Web bytes the target agent observed (W_obs), but withhold L*. Run many independent episodes. Any valid artifact scoring within tolerance ε of the target counts as a recovery witness and refutes the discovery claim. Zero recoveries across n episodes yields a Clopper-Pearson interval-style upper bound on the fresh-episode recovery probability.
•
Gate 3 (optional): from a shared checkpoint after Web reads but before experiments, run paired branches. One arm gets truthful feedback from its own experiments. The other gets a neutral policy that matches message timing, schema, and length but strips the directional information. Estimate the average effect of truthful-vs-neutral feedback. Separately calibrate the neutral channel against a second non-informative channel on tasks whose answer is fixed by E_0, to prove the neutral channel isn’t itself helping or hurting.
Two decisions come out. Core = Gate 1 passed, zero recovery witnesses, adequate positive controls, and the recovery-probability upper bound is under a pre-registered threshold ρ. Evidence = Core plus a Gate 3 lower confidence bound on the feedback effect that clears both the effect threshold δ_evidence and the calibration margin δ_sham.
def dcp_audit(target_run, registration):
if not gate1_useful_improvement(target_run.A_star, registration):
return "fail"
hits = []
for episode in range(registration.n_episodes):
a = challenger_agent(K, E_0, W_obs, tools, budget) # no L*
if valid(a) and score(a) >= target_run.x - registration.epsilon:
hits.append(a)
if hits: return "recovered (Core refuted)"
if not controls_adequate(): return "audit incomplete"
if upper_bound(hits=0, n=registration.n_episodes) > registration.rho:
return "statistical uncertainty"
core = True
if gate3_registered:
delta = paired_feedback_effect(checkpoint, truthful, neutral)
if null_calibration_passes() and lcb(delta) >= d_evidence + d_sham:
return "Core + Evidence"
return "Core"
A separate deterministic verifier, dcp-audit, recomputes the decision from the frozen evidence bundle with no LLM in the loop.
The authors ran two full three-gate audits and three diagnostic cases. Scores are only comparable within a task.
•
SQLite index selection (agent: DeepSeek-V4-flash). Task: pick 4 partial indexes from 16 query families to minimize traffic-weighted SQLite virtual-machine work. Target agent hit 0.8855 (an 88.55% work reduction vs no secondary indexes). In Gate 2, 0 of 96 challenger episodes recovered within ε of the target; best challenger reached 0.6734 against a recovery line of 0.8805. Positive controls passed 45/45. Zero-hit upper bound on fresh-episode recovery: 0.0468.
•
Virtual catalyst optimization (agent: DeepSeek-V4-Pro). Task: pick 5 experimental controls with 8 levels each (32,768 recipes), score 1.0 means hitting all five hidden optima. Target scored 1.0 vs baseline 0.599. Again 0/96 recoveries, best challenger 0.8146 against recovery line 0.95.
•
Gate 3 in both audits: 30/30 truthful-feedback recoveries vs 0/30 neutral-feedback recoveries per audit. Estimated binary policy effect 1.0 with a 99% paired interval of [0.638, 1.0]. The separate 60-pair null calibration returned a difference of 0 with interval [-0.095, 0.095], comfortably inside the registered ±0.17 equivalence band. Both cleared Evidence.
•
Diagnostic knapsack case: one challenger episode produced two legal solutions scoring 0.9363 and 0.9356, both above the recovery line of 0.9329. That’s a qualified recovery witness: Core is refuted, and the audit is closed as “recovered” rather than certified. The authors use this to show the recovery rule fires when it should.
•
Affine calibration case: intentionally under-sampled; positive-control recall was too weak, so the verifier returned “audit incomplete.” This exercises the adequacy check that prevents thin controls from producing false Core certifications.
The two full audits cost $56.40 and $61.17 in provider fees and used 435 and 507 recorded model sessions respectively. Replay of the frozen bundles is offline and free.
An important reading note: Core + Evidence is a claim about this registered scope (this model, this budget, this information packet). It says a fresh matched challenger under the same setup rarely recovers the target, and truthful feedback caused a measurable lift from the shared checkpoint. It does NOT claim historical priority or that no cleverer challenger under a different budget could recover it.
•
If you’re evaluating an agent that claims to have discovered something, the operational question DCP formalizes is: can a matched challenger without the run’s history reach the same result? Even without adopting the full protocol, running a few no-history challenger episodes with the same Web access and starter code is a cheap sanity check. A single recovery witness is enough to walk back a discovery claim.
•
If you’re designing an ablation for an agent framework that claims “the feedback loop matters,” the Gate 3 shape (paired branches from a shared checkpoint, one with real feedback and one with a length-and-timing-matched neutral placebo, plus a separate null-task calibration of the placebo channel) is a directly reusable template. The null calibration is what rules out “the placebo itself changed behavior.”
•
If you’re a reviewer, the useful decomposition is: utility ≠ recoverability ≠ feedback-caused improvement. A paper reporting a leaderboard gain has shown utility. It has usually not shown either of the other two.
•
The released dcp-audit verifier and dcp-harness are worth looking at if you want a concrete pattern for separating evidence collection from decision checking so that an outside party can replay a decision without rerunning any LLM.
The protocol is heaviest when you actually need Evidence, since Gate 3 requires paired runs, a checkpoint, and a null-calibration family. For internal red-teaming, Gate 2 alone (challenger recovery) delivers most of the epistemic value at a fraction of the setup.
•
Only two full audits are reported, both on tasks the authors constructed and both against DeepSeek models. Generalization to other agent architectures, harder open-ended tasks, or agents with substantially larger budgets is unestablished.
•
The recovery bound is tied to the registered challenger distribution Q_B. A more capable challenger, a bigger budget, or an unblocked Web could recover a target that a smaller registered episode did not. Core is a scope-conditional statement, not a universal impossibility claim.
•
Gate 3’s neutral policy is a design artifact. The authors say development cases guided its design, but a poorly constructed neutral channel could either leak information or handicap the neutral arm; the null-calibration study is what tries to catch this, and its ±0.17 band is itself a registered choice.
•
The knapsack and affine “diagnostic” cases are constructed to exercise specific decision branches. They demonstrate that the verifier fires correctly on planted inputs; they are not independent evidence that real-world discovery claims will frequently be refuted or certified.
•
Gate 2 conditions on W_obs, the Web bytes the target agent actually saw. If a target agent avoids reading a helpful page, the challenger doesn’t get to read it either. This is deliberate (fair recovery under the same information) but means DCP is not measuring recoverability under “any competent Web search,” only under the target’s realized information packet.