Statistical method gives reliable confidence bounds for AI agent quality

Who: Authored by Yuxuan Gao, Megan Wang, and Yi Ling Yu. No institutional affiliations are listed in the metadata. The paper was accepted as a poster at the ICML 2026 Workshop on Agentic Uncertainty Quantification.

What's new: Researchers have built a framework for putting honest, statistically guaranteed error bars around scores used to rank AI agents — think of it as a system that tells you not just "Agent A scored 82" but "Agent A's true score is almost certainly between 76 and 88." The core insight is that these confidence ranges should automatically grow wider whenever something changes, such as a new agent being released, and then shrink back once the situation stabilizes. That adaptability is the novel piece.

How it works: The framework adapts two existing statistical tools — split conformal prediction and ACI (adaptive conformal inference) — to the specific problem of tracking agent quality over time using live data. The team also builds rules for when a head-to-head ranking between two agents is too uncertain to call, using a technique called FDR (false discovery rate) correction to keep mistakes rare even when comparing many agents at once. They track 50 agents using 18 real-time signals, collected every hour.

The numbers: The system's calibration error stays below 0.02 across all tested confidence levels. After a new agent is released, the framework correctly widens its uncertainty ranges by 35% before reconverging. Per-agent coverage clusters tightly around the target 80% level, with 90% of agents falling between 72% and 90%. A separate validation confirms the signals captured go beyond what standard benchmarks measure.

Caveats: The paper is a six-page workshop contribution, so the scope is deliberately narrow. The agents and signals evaluated are not named, making it hard to assess real-world generalizability. The framework also assumes continuous, hourly data streams, which many evaluation settings do not have.