Get Started
Home
Topics
Search
Library
Research questionHow can coding agents reliably implement systems-level requirements and detect the defects they introduce?Systems-level tasks combine interacting requirements such as schema design, asynchronous orchestration, configuration, and retrieval behavior. Defects can violate these constraints while remaining undetected, and claimed fixes may go unverified without regression measurement.
AI Agents
Code Generation & Program Synthesis
Evaluation & Benchmarks
Information Retrieval
Latest papersRecent research connected to this question, newest first.When Agents Implement Systems: A Case Study in Defects, Detection, and Evaluation RigorEvidence comes from a single coding agent implementing a fixed multi-component data system during one session, with five cataloged defects. The retrieval analysis uses 100 HotpotQA questions, a pooled corpus of 2,994 paragraphs, gold evidence labels in place of entity identification, and standard recall across retrieval budgets of 1–10; it does not directly evaluate the agent performing entity identification.research paper · Sep 2, 2026
Related questions
How can teams make coding-agent output reliable enough for production?How can coding agents report defective test infrastructure instead of exploiting it to pass?How can repository-level coding-agent benchmarks detect review-constraint failures beyond passing functional tests?How can agent skills support reliable procedural execution while making failures easy to diagnose and repair?