EvoOntology gives a data agent a queryable ontology layer it edits from its own failure traces, keeping only changes that beat the prior version on a held-out fold, cutting agent turns per task by roughly 40% while raising accuracy.
Suppose you point an LLM agent at a company’s data: a warehouse of SQL tables, a folder of CSVs, some 10-K filings. A user asks, “which cards are banned in Standard?” The agent doesn’t know your schema. Today it does what a ReAct loop does: reads schemas, samples rows, guesses column meanings, and burns turns rediscovering the same facts every task. The paper calls this the agent–data gap: the data lives outside the model, and the only bridge is generic tools like a SQL runner or file reader.
Two standard fixes exist. Raw querying (what most Text-to-SQL agents do) lets the agent probe freely, which drowns in wide schemas. Semantic layers (things like dbt semantic/metric layer’s metric layer or OWL ontologies) hand-write a metadata document and paste it into the prompt. That document is static, competes with the rest of the prompt for attention, and blows past context limits on real workloads. Neither adapts to how a specific agent actually behaves, which is the gap this paper targets.
EvoOntology treats the semantic layer as a living, tool-accessible object rather than a prompt blob. It has three parts. A content layer is a typed graph with Terms (domain concepts), Mappings (which columns/join paths back the term), Constraints (rules for valid use), and Evidence (probe queries that verify the term exists in the data). A schema layer defines what kinds of nodes and edges are allowed. A tool layer exposes two MCP (Model Context Protocol) endpoints: browse (semantic search over the ontology) and resolve (fetch full records for an ID). Only a small manifest goes into the prompt; details are pulled on demand.
Construction happens in two phases. First, a builder agent reads a training workload of tasks, guesses concepts the tasks will need, and issues probe queries to confirm each concept maps to real fields. Only verified concepts are committed. Second, an evolution agent watches interaction trajectories, clusters failure patterns, and proposes typed edits. Each edit touches exactly one level (content, tool, or schema) and states a hypothesis about what should improve. The edit is accepted only if it beats the current ontology on a held-out validation fold by a margin, using the same backbone model that will deploy it. Different backbones therefore evolve different ontologies from the same start.
state = builder.construct(workload, raw_sources)
for round in range(max_rounds):
signatures = analyze(trajectories, state) # cluster failures
for sig in signatures:
level = attribute(sig) # Content | Tool | Schema
candidate = patch(state, sig, level) # single-level edit
if score(candidate, val, backbone) - score(state, val, backbone) >= tau:
state = candidate # accept, else log and skip
On DDR-Bench (10-K financial research), EvoOntology lifts trajectory-wise accuracy by an average of +17.8 points over a ReAct baseline across six backbones, with per-backbone gains ranging from +4.8 (Qwen3.5-Flash) to +26.7 (GPT-5.5). The static-semantic-layer baseline (same content, pasted into the prompt) is not just neutral, it hurts on strong models: −15.0 points on Claude-Sonnet-5. The authors attribute this to the prompt fragment competing with the agent’s own instructions instead of being pulled selectively.
On InsightBench (business-intelligence flags on CSVs), gains are smaller (+1.9 overall on average), which the authors explain by the metric saturating once the agent’s finding matches the short reference. On BIRD (text-to-SQL), execution accuracy rises +7.4 points and efficiency +8.6 on average. The static semantic layer improves efficiency but lowers execution accuracy on four of six backbones, suggesting it helps well-formedness while distracting from correctness.
Ablations pin down what matters. Removing the validation gate loses −11.2 points, the largest single drop, because unfiltered edits admit regressions later rounds cannot undo. Removing attribution (the level tag) loses −6.3. Restricting to a single edit level caps gains at +13.2 (tool-only) versus +20.0 for the full three-level loop. Masking Mappings from the content graph loses −13.4, confirming that grounding to concrete columns is the load-bearing object. On cost: evolved ontology adds ~1.4K input tokens per turn but cuts trajectories from 14.6 to 8.4 turns, netting a ~20% token reduction per task.
Cross-backbone transfer is negative. Serving an ontology evolved for one backbone to another loses at least 6.6 points versus the same-backbone store, and pairwise term overlap between backbones tops out at 0.62 Jaccard. Different models genuinely want different ontologies.
If you run a ReAct-style agent against a stable heterogeneous data source (databases plus files) and see it re-discovering the same schema facts every session, the mechanism to borrow is: expose learned semantics through a retrievable tool rather than growing the system prompt. The paper’s evidence is specifically that a static prompt-time semantic layer can underperform no layer at all on capable models, so pushing metadata into prompts as a first move is worth reconsidering.
If you build an evolution loop over agent traces, the ablations argue for two guardrails worth testing in your own setup: a paired validation gate that only accepts edits beating the prior version on a held-out fold, and a level or type tag on each proposed edit so the loop doesn’t rewrite the wrong thing. These two together account for most of the reported gain.
Budget for backbone-specific tuning. The transfer results suggest an ontology evolved on GPT should not be assumed to help Claude; if you change model, plan to re-run evolution. The paper’s protocol also requires a training workload of representative tasks to seed both the builder and the trajectory pool, so this is not a zero-shot drop-in for a brand-new data source.
The headline gains are strongest on DDR-Bench, which is open-ended multi-source research where schema rediscovery is the main cost; on InsightBench the metric saturates and gains shrink to under two points. The evaluation uses a reciprocal two-fold split within each benchmark, so “generalization” here means held-out tasks against the same data source, not new data sources. The evolution loop needs a workload of tasks and a scorer to run paired validation; the paper doesn’t characterize what happens when the workload is small or the scorer is noisy. Finally, the reported cost numbers cover token usage only. LLM calls during builder probing and evolution rounds are training-time overhead the paper doesn’t itemize.