Evaluation Methods
Status: draft, October 2026. Costs are given as structure (fixed vs. marginal) unless a source prices them. Measured reliabilities come from the cited studies and do not transfer automatically to a new domain.
The evaluation component needs a method. Each method below is a point in the design space of accuracy, unit cost, robustness and legitimacy. Evaluation Systems in the Wild shows each deployed; the capability ladder grades systems built from them.
| Method | What it measures | Known failure modes | Cost structure | Evidence |
|---|---|---|---|---|
| Expert panel / peer review | expert judgment of quality or merit | low inter-rater agreement (mean κ ≈ 0.17 across 48 studies); outcome depends on which reviewers are drawn | high marginal: reviewer time, over 100M hours globally in 2020, valued at over $1.5B for US-based reviewers alone | Bornmann et al. 2010; Pier et al. 2018; Aczel et al. 2021 |
| Structured expert elicitation (Cooke’s classical model; Delphi, IDEA) | calibrated quantities and uncertainty from experts | under a third of experts were statistically accurate across 33 classical-model studies; performance weighting beat equal weighting on combined score in 31 of 33 (in-sample) | high fixed (seed questions, facilitation) per study | Colson & Cooke 2018; see the sibling elicitation guide |
| Structured appraisal tools (GRADE, Cochrane RoB 2) | conformance of a study or evidence body to a checklist | checklist judgments still diverge: four expert raters reached κ 0.16 on RoB 2’s overall judgment | about 28 min per trial outcome per rater | Minozzi et al. 2020 |
| Survey | opinion of a defined population, e.g. a forecast’s actual readers | nonresponse bias, which response rate predicts poorly; question wording | moderate marginal per respondent | Groves & Peytcheva 2008 |
| Crowd ratings / reviews | self-selected users’ satisfaction | J-shaped self-selection; herding on early votes; fake reviews, reduced but not removed by verified-purchase rules | high fixed (platform, moderation), near-zero marginal | Patterns §2, §4 |
| Statistical indicator | a measurable proxy | Goodhart and surrogation once the indicator carries stakes; measures what is convenient | low marginal once data exist | Patterns §1 |
| Composite index | a broad construct from weighted indicators | arbitrary weights; manipulation of inputs: the World Bank’s Doing Business was discontinued in 2021 after data irregularities | moderate fixed, low marginal | World Bank 2021 |
| Inspection / independent testing | compliance or performance against a standard | snapshot validity; works when the metric is hard to game and demand responds (LA hygiene grade cards) | high marginal per inspection | Jin & Leslie 2003 |
| Impact evaluation (RCTs, quasi-experiments) | an intervention’s causal effect on chosen outcomes | effects vary widely across contexts, so results transfer poorly; across 600+ development studies, government-run programs showed smaller effects than academic- or NGO-run ones | high fixed per study | Vivalt 2020 |
| Forecasting resolved by an evaluator (markets, polls) | predicted outcome of an expensive evaluation | thin incentives when few items resolve; accuracy must be read against the base rate (markets called 73% of 104 replication outcomes) | low marginal per item, plus a resolution budget | Liu et al. 2020; Techniques |
| Pairwise comparison (Bradley–Terry, arenas) | relative preference | order effects in online Elo; cyclic preferences; selective disclosure of privately tested variants | low marginal per vote | Singh et al. 2025; Patterns §5 |
| LLM judge | predicted human judgment | agreement with humans varies by task and item difficulty; position, verbosity and self-preference bias; prompt sensitivity; correlated errors across models; null responses and adversarial phrases inflate scores | very low marginal; fixed cost of validation and re-validation dominates | LLMs as Evaluators; Zheng et al. 2024 |
Reading the table
Section titled “Reading the table”Two regularities stand out. First, the methods with the highest standing (peer review, appraisal tools) have some of the lowest measured agreement; their legitimacy comes from who does them, not from measured accuracy. These are single-rater figures, though. Averaging helps: by the Spearman–Brown formula, peer review’s mean single-rater ICC of .34 implies about .61 for the mean of three interchangeable reviewers. And reviewers are often chosen for different expertise, so part of the disagreement may be by design. Second, the cheap methods (indicators, indices, crowd ratings, LLM judges) fail mainly under optimization pressure, so their unattacked accuracy overstates their deployed accuracy. A system usually pairs one of each: a cheap method on every item, audited by an expensive one on a sample (the prediction–evaluation pattern). High-stakes essay scoring runs a more cautious pairing, with a human on every item and the machine as a check (capability ladder).
Not every method answers the whole question. Surveys, indicators, appraisal checklists and impact evaluations are often partial evaluations: inputs to a fuller judgment rather than substitutes for it. Designing a system means deciding which items get full evaluation and which ride on partial measures plus prediction.
Open territory
Section titled “Open territory”- Elicitation. How to phrase questions so evaluators answer the intended one, and how to elicit value judgments. QURI’s relative value functions and the sibling RRP wiki’s Constructing Utility Functions are concrete attempts.
- Pricing. Few methods have published cost per item; most rows above give structure, not numbers. A system cannot trade accuracy against cost without them.
- Deployed LLM evaluators. RRP’s LLM Epistemics in Production documents RoastMyPost and Squiggle AI, including their false-positive problems.