AI Model Evaluation
Swap the question set and the same model moves several places — a leaderboard is never a report card, it is one slice taken from the examiner's angle. This tag follows how benchmarks are built, how graders are designed and how scores stop being inflated.
The hard part of evaluation is not writing questions, it is making sure the score actually measures ability. UNO-Bench unifies 44 task types across five modality combinations and ships a general grader that keeps automated scoring around 95% consistent. AMO-Bench goes further and commissions 50 fresh IMO-level problems from human experts, specifically so memorized training data cannot inflate results; most models still score below 40%.
Single-turn evaluation breaks down on agents. Anthropic's engineering team suggests starting from 20 to 50 real failure cases, giving every task a passing reference solution and an explicit rubric, covering both "should do" and "should not do" examples, and running each trial in an isolated environment so shared state and caches cannot raise the number. The metric matters too: pass@k asks whether one of several attempts succeeds, while pass^k asks whether all of them do — and only the second matches a product that has to work every single time.
