Get Started
Research questionHow can we establish the source of authority behind an LLM-based software test oracle’s verdict?An LLM oracle may ground an assertion in a written specification or in patterns learned during training, and its production mechanism does not reveal that distinction. Without a specification, a challenged verdict has less explicit support, while agreement with a reference oracle may not show whether faults are detected.
AI
Code Generation & Program Synthesis
Evaluation & Benchmarks
Latest papersRecent research connected to this question, newest first.LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature ReviewThis PRISMA 2020 systematic literature review analyzes 83 studies—54 identified through screening and 29 added by citation searching—across source of authority, oracle form, and adjudication mechanism. It reports that just over half of the corpus reaches verdicts without a specification and that source and mechanism do not coincide; the reviewed studies most often assess quality by resemblance to a known oracle rather than fault detection.research paper · Sep 2, 2026
Related questions
How can we assess whether LLM moral decisions are defensible when no ground truth exists?How can we tell whether agreement among LLM judges reflects human alignment or shared blind spots?How can human reviewers reliably detect LLM errors when verification reasoning is hard to retrieve?How can high-stakes LLM systems distinguish unsupported claims from novel ones and prioritize expert verification?
Home
Topics
Search
Library