Get Started
Home
Topics
Search
Library
Research questionHow can binary LLM calibration be audited when APIs hide probabilities but expose logit bias?Standard calibration metrics require continuous output probabilities, but many commercial LLM APIs expose only limited output information. This makes it difficult to determine whether binary predictions reflect reliable confidence estimates.
AI
Alignment & Safety
Evaluation & Benchmarks
Machine Learning
Statistical Machine Learning
Latest papersRecent research connected to this question, newest first.Single-Query Black-Box Calibration Auditing via Logit BiasThis applies to black-box LLM APIs that conceal token probabilities while exposing a logit_bias parameter. The source addresses binary tasks, exact probability-threshold evaluation, one query per sample, and estimation of True Calibration Error; it does not establish coverage of APIs without logit_bias or multiclass calibration.research paper · Sep 4, 2026
Related questions
How can black-box LLM APIs detect hallucinations without trusted context while limiting false alarms?How can fairness audits of multi-step clinical LLM agents separate demographic disparity from stochastic action instability?Can black-box LLM judges provide reproducible measurements on shared endpoints?How can LLMs produce reliable confidence estimates for deciding when to defer outputs to humans?