The Capability Ladder
Status: draft v0, October 2026. The rungs are this wiki’s proposal. The placements are judgments from public documentation and will be wrong wherever a system measures more internally than it publishes.
Driving automation has graded levels, and laboratories and educational tests have standards (below); judgment-based evaluation in general has no shared way to say how far a system has got. This page proposes one. Each rung is defined by something the system must be able to measure or do, so a grade can be checked rather than asserted. The unit being graded is the standing system, not a single evaluation (Evaluation as a System).
The rungs
Section titled “The rungs”| Rung | Name | The system can… | Test |
|---|---|---|---|
| 0 | Ad hoc | produce judgments | none: each judgment is made its own way |
| 1 | Standardized | apply one written procedure and output scale to every item, and keep records | two items judged a year apart sit on the same scale and trace to the rubric version used |
| 2 | Measured | state its error on a defined population, with an interval | it publishes chance-corrected agreement with independent raters, alongside the rater–rater baseline, or a proper-score track record against resolved outcomes |
| 3 | Monitored | re-measure that error in operation and notice when it moves | a standing audit stream on random or hidden items; every accuracy figure carries a date and version; re-validation triggers are declared in advance |
| 4 | Gaming-tested | state its error under deliberate attempts to fool it | an attack suite or red team has been run against it, success rates are reported, and detected gaming is penalized |
| 5 | Accountable | be checked and contested by parties it does not control | independent audit of the whole system; disclosed payer and conflicts; an appeal route for the evaluated whose outcomes feed back into measurement |
A system’s grade is the highest rung whose test it passes together with every rung below. The ordering is a claim about dependence: drift (3) cannot be detected without a measured baseline (2), and the cost of an attack (4) cannot be stated without the unattacked error. Rung 5 is institutional rather than technical, and real systems often acquire it out of order. A jagged profile is information, not a grading error: credit ratings now have strong external oversight but failed under issuer pressure. “Error” at rungs 2–4 means disagreement with a reference (raters or resolved outcomes), which is validity only to the extent that the reference is what the system is meant to track (Reliability is not validity).
The ladder does not grade whether the question is worth asking, or cost. A rung-4 system can be too expensive to run (the design space). The methods behind rungs 2–4 are in Measuring an Evaluator; the operational steps are in the Launch Checklist.
Eight systems graded
Section titled “Eight systems graded”✓ passes, ~ partial, ✗ fails or absent, ? not publicly documented.
| System | 1 | 2 | 3 | 4 | 5 | Grade |
|---|---|---|---|---|---|---|
| Journal and grant peer review | ✓ | ~ | ✗ | ✗ | ~ | 1 |
| Cochrane systematic reviews | ✓ | ~ | ? | ? | ~ | 1 |
| Credit ratings (US NRSROs) | ✓ | ✓ | ~ | ✗ | ✓ | 2 |
| LMArena | ✓ | ✓ | ? | ~ | ~ | 2 |
| Community Notes (X) | ✓ | ~ | ? | ✓ | ~ | 1–2 |
| METR capability evaluations | ✓ | ✓ | ? | ~ | ~ | 2–3 |
| Metaculus | ✓ | ✓ | ✓ | ? | ~ | 3 |
| GRE essay scoring (e-rater plus human) | ✓ | ✓ | ✓ | ✓ | ~ | 4 |
Peer review. Journals and funders use fixed criteria and score scales, but reliability is mostly measured by outside researchers. A meta-analysis of 48 studies found mean inter-rater κ ≈ 0.17 (Bornmann, Mutz & Daniel 2010). In a mock NIH study section, 43 reviewers scoring the same 25 funded oncology applications showed little agreement in scores or written critiques (Pier et al. 2018). The partial mark is for NeurIPS, which sent 10% of submissions to two independent committees in 2014 and again in 2021; both times about half the accepted list would have changed on a rerun (Beygelzimer et al. 2023). Gaming tests come from outside, as stings: a fabricated paper with obvious flaws was accepted by 157 of 304 open-access journals, with no subscription journals tested (Bohannon 2013), and hidden prompts aimed at AI reviewers (LLMs as Evaluators) show a newer exposure.
Cochrane. The Handbook and the RoB 2 risk-of-bias tool standardize the procedure. Their reliability is again measured externally, and poorly: four experienced raters applying RoB 2 to 70 trials reached Fleiss’ κ 0.16 on the overall judgment, at about 28 minutes per outcome (Minozzi et al. 2020). Cochrane’s strength is institutional: commercial organizations with a financial interest in a review’s outcome may not fund it (commercial sponsorship policy). Grade 1 with rung-5 features, which is the opposite profile to credit ratings.
Credit ratings. Registered agencies must publish standardized performance statistics (transition and default rates) as Exhibit 1 to Form NRSRO, and SEC staff must examine each agency at least annually (SEC Office of Credit Ratings staff report, Jan. 2025). That is rungs 2 and 5. Rung 4 failed in practice: the official inquiry called the agencies’ failures “essential cogs in the wheel of financial destruction” (FCIC 2011). Most of the examination regime postdates the failure, and nobody has shown that it tests resistance to issuer pressure rather than compliance.
LMArena. Pairwise votes, a Bradley–Terry model and published confidence intervals give rungs 1–2. Its builders checked crowd votes against expert raters and found good agreement (Chiang et al. 2024). Rung 4 is split. Vote manipulation was tested with the operators: in an offline simulation about a thousand votes could move a target model’s rank, and mitigations were deployed (Huang et al. 2025). The process failed: a few providers tested many private variants and published only the best (Singh et al. 2025).
Community Notes. The bridging algorithm and the rating data are public so that outsiders can audit them (X help centre). Its developers validated note selection with a survey experiment and a reshare effect in deployment (Wojcik et al. 2022), but no routine error rate on a random sample of displayed notes has been published. Rung 4 has a recent pass of the useful kind: a study co-authored with the Community Notes team found that up to 10.7% of lower-quality notes in historical data could be pushed over the display threshold with fewer than 10 coordinated ratings, and mitigations were deployed (Selvam et al. 2026). An independent simulation found 5–20% bad raters could suppress targeted helpful notes (Truong et al. 2025). The profile is the reverse of most systems’: attacked and patched before its unattacked error is measured.
METR. Task suites with timed human baselines, and capability estimates with hierarchical-bootstrap 95% intervals (Kwa et al. 2025), give rungs 1–2. Here the evaluated party is the model, and it can game in either direction. METR checks for inflation: o3 reward-hacked in 30.4% of RE-Bench runs and 0.7% of HCAST runs, and successful hacks are excluded from reported results, though METR says it may miss some (METR 2025). For a dangerous-capability evaluation the worse direction is deflation. Models can be prompted or fine-tuned to underperform selectively on such evaluations (van der Weij et al. 2024), and frontier models tell evaluation transcripts from deployment well above chance (Needham et al. 2025). METR’s GPT-5 report found no significant evidence of sandbagging but noted that its monitors miss many misbehaviours and that a capable enough model could evade them (METR, Aug. 2025). That is a partial pass on rung 4. Whether error is re-measured across releases in a way that would meet rung 3 is not documented publicly, hence the range.
Metaculus. Every resolved question is scored with proper scoring rules, and the platform publishes a track record with calibration curves. Because most questions resolve (some are annulled), the audit stream is nearly the whole population rather than a sample. We know of no published test of how the community forecast responds to deliberate manipulation.
GRE essay scoring. Each Analytical Writing essay is scored by a trained human reader and by e-rater; if they disagree, a second human scores it and the human scores are averaged (ETS). ETS published machine–human agreement for the GRE prompts (Ramineni et al. 2012), including by subgroup: e-rater matched human raters for most groups but scored essays from mainland China much higher (Bridgeman, Trapani & Attali 2012). The engine alone fails gaming tests. Outsiders invited by ETS to fool it inflated scores more often than they deflated them, and the authors concluded it should not be the sole scorer in high-stakes use (Powers et al. 2002); later, gibberish from the BABEL generator got the top score in 16 of 38 submissions to ETS’s practice service (Perelman 2020). The system passes rung 4 because a human reads every essay and an off-topic essay scores zero; test takers can pay for a score review, a partial rung 5. The top grade thus goes to a design in which the machine checks a human on every item and saves at most the second read: evidence that the cheap layer can check, not that the human can be removed.
What the grading suggests
Section titled “What the grading suggests”These are eight hand-picked systems, so the observations are hypotheses. The most trusted systems by reputation (peer review, Cochrane) grade lowest on measurement, which suggests trust tracks institutional standing more than measured error. The highest grades go to systems whose reference arrives cheaply (questions resolve; a human reads every essay). The top grade also goes to the one system built under a mature measurement standard: educational testing’s Standards (AERA, APA & NCME 2014) require validity and reliability evidence, quality control of scoring, and a route for test takers to challenge results. ISO/IEC 17025 does the same for testing laboratories: accredited labs must evaluate measurement uncertainty and monitor the validity of results, including by proficiency testing against other labs where available (clauses 7.6–7.7), under independent accreditation. Other fields can borrow these rather than start from scratch. And every system that has been seriously attacked, whether by issuers, model providers, coordinated raters or the model itself, was attacked through the incentive the score created. That is the Goodhart pattern, and it is why rung 4 cannot be skipped by anything that gates value.
The sibling RRP wiki’s Reliability Ladder grades applications of strong reasoners by how much verified trust they need; mapping its tiers to these rungs is open work.
Open questions
Section titled “Open questions”- Do the rungs discriminate? Grading the 161 catalogue systems would show whether most cluster at 1, and whether grade predicts documented failure better than funding model does.
- Should rung 5 be a separate axis rather than a rung? The Cochrane and credit-rating profiles argue for two axes, technical and institutional.
- What is the minimal rung-4 evidence for a judge facing unknown attacks, given that a suite only covers known ones?