Measuring an Evaluator

Status: draft, October 2026. This chapter collects the measurement methods the rest of the wiki leans on. LLMs as Evaluators argues for auditing a cheap judge against an expensive one; the worked example applies the arithmetic to one case. Every number below that is not cited is computed in the text or in the snippet, and the simulated population in the snippet is illustrative.

An evaluator is a measuring instrument, and the questions are the ones metrology asks of any instrument: does it give the same answer twice (reliability), does it measure the intended thing (validity), are its stated confidences honest (calibration), and how much does a measurement of all this cost in labelled items (sample size)? For an evaluator facing people who benefit from its verdicts there is a fifth: does it still work when they try to fool it?

Reliability is not validity

Reliability is agreement of the evaluator with itself, or with interchangeable copies of itself: two runs of the same judge, two prompt phrasings, two reviewers drawn from the same pool. Validity is agreement with the thing the evaluation is meant to track. Reliability caps validity. In classical test theory the correlation of a score with any criterion is at most the square root of the score’s reliability, so a judge whose repeated runs correlate at 0.5 cannot correlate above 0.71 with the truth, however good it is on average. Re-running an LLM judge under fuzzed prompts is therefore a cheap first measurement: it gives an upper bound on validity before a single expert label is bought. The bound is only as good as the perturbations. A judge run greedily on one prompt is close to perfectly reliable by construction; reliability has to be measured across the variations (phrasing, model version, item order) that deployment will actually introduce.

The converse fails. A judge can be perfectly reliable and wrong every time in the same way, and agreement between two models is mostly reliability: models share errors (correlated error). Agreement with a human panel is a validity measure only to the extent the panel is the target. Usually it is a stand-in for something the funder actually cares about, such as whether the grant achieved anything. The panel can itself be unreliable or systematically wrong, as can benchmarks: a review of 445 LLM benchmarks found recurring validity weaknesses in what they measured, the tasks and the scoring (Bean et al. 2025), and Salaudeen et al. (2025) match each kind of evaluation claim to the validity evidence it needs. The practical rule: write down what the panel stands in for, then decide which disagreements between panel and target the system can tolerate.

Agreement, corrected for chance

Raw agreement $p_o$ is the share of items on which two raters give the same label. It is inflated by base rates: two raters who each independently say “yes” to 80% of items agree $0.8^2 + 0.2^2 = 68\%$ of the time without looking. Cohen’s kappa subtracts the agreement expected from the raters’ own marginal rates, $p_e$, and rescales:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

so that chance scores 0 and perfect agreement 1 (Cohen 1960).

The same raw agreement means very different things at different base rates. Each curve is the raw agreement needed for a given κ when both raters say “yes” at rate π; the bottom one is chance, π² + (1 − π)². At 70% agreement, κ is 0.40 when raters split 50/50 but only 0.06 in the worked example, where experts renew 80% of grants. Fifty audits put about ±13 points on that 70% (the bar), and the κ estimated from them could fall anywhere from about −0.20 to 0.37: it cannot tell this pair of raters from chance. The picture is symmetric for π below 50%.

The relatives:

• Fleiss’ κ extends the idea to many raters: each item gets the same number of ratings, possibly from different raters (Fleiss 1971).
• Weighted κ gives partial credit for near-misses on an ordinal scale. The quadratic-weighted version is the standard in automated essay scoring, whose acceptance test (LLMs as Evaluators) compares machine–human against human–human weighted κ.
• Krippendorff’s α works for any number of raters, any level of measurement (nominal, ordinal, interval, ratio), and missing ratings (Hayes & Krippendorff 2007). That makes it the default when the design is messy, as audit designs usually are.
• ICC (the intraclass correlation) plays the same role for continuous scores. Use the absolute-agreement form; the consistency form ignores a judge that scores every item a point high (McGraw & Wong 1996).

Kappa has two well-known paradoxes (Feinstein & Cicchetti 1990). Both follow from $p_e$ depending on the marginals:

2×2 table (yes-yes, yes-no, no-yes, no-no)	$p_o$	$p_e$	κ
40, 9, 6, 45 (balanced)	0.85	0.50	0.70
80, 10, 5, 5 (raters’ “yes” rates 90% and 85%)	0.85	0.78	0.32
45, 15, 25, 15 (raters’ “yes” rates 60% and 70%)	0.60	0.54	0.13
25, 35, 5, 35 (raters’ “yes” rates 60% and 30%)	0.60	0.46	0.26

• Prevalence. The first two rows have identical raw agreement. Kappa falls by more than half when both raters say “yes” to most items, because chance agreement is already high.
• Bias. In the last two rows, raters who disagree more about the base rate get the higher kappa, because their marginals predict less chance agreement.

Neither paradox is an error in kappa. Each is a reason not to report kappa alone. Byrt, Bishop & Carlin (1993) recommend reporting prevalence and bias indices next to κ; the prevalence- and bias-adjusted kappa, PABAK $= 2p_o - 1$, is usually credited to them. Gwet’s AC1 is an alternative coefficient that is more stable at extreme prevalence. The minimum honest report for a binary judge is the full 2×2 table, or equivalently $p_o$, each rater’s “yes” rate, and κ. A judge that says “yes” more often than the experts do is lenient. That is a bias worth knowing about even when κ looks acceptable. Leniency is among the failure modes that Thakur et al. (2024) found in LLM judges, and their judges with high percent agreement still assigned scores far from the human ones.

The human-panel baseline

Agreement with a human is only interpretable next to agreement between humans. If two experts reach κ = 0.2 on the same items, a judge at 0.2 against one expert is as good a stand-in for “an expert” as a second expert is. A judge can legitimately score somewhat above the human–human figure: by the reliability cap above, a judge that tracked the experts’ shared signal perfectly would correlate with one expert at about $\sqrt{r}$, where $r$ is the expert–expert correlation (on the correlation scale, not κ). Agreement beyond $\sqrt{r}$ has to come from something else the judge shares with that expert, such as a taste for polished prose, and deserves suspicion. Journal peer review averages κ ≈ 0.17 (Patterns §2), so in judgment-bound domains this ceiling is low. It has to be measured per task, by having at least some items rated independently by two or more humans.

Aggregate figures also hide where agreement comes from. When many items have genuinely split human labels, a judge can match or beat human–human agreement with the majority label while falling well below it on the items humans agree on; stratifying the comparison by human disagreement exposes this (Elangovan et al. 2024). Forcing raters to pick one label on such items makes it worse: across 11 rating tasks, judges chosen by agreement with forced-choice labels performed up to 31% worse, by the authors’ measure, than judges chosen against the set of ratings each rater considered reasonable (Guerdan et al. 2025). Where items admit more than one defensible label, collect every label each rater would accept.

Two details make the comparison fair:

• Compare like with like. A judge scored against a panel’s majority vote is being compared with an aggregate that is less noisy than any single member. The matching human baseline is one held-out panelist against the rest of the panel, not panelist against panelist. The alternative annotator test formalizes this (Calderon, Reichart & Dror 2025). It drops each human annotator in turn and asks whether the LLM agrees with the remaining annotators at least as well as the dropped human does. It allows the LLM a margin ε for being cheaper (0.2 against experts, 0.1 against crowd workers) and passes it if it beats at least half the annotators; the authors recommend at least three annotators and 50–100 items.
• Measure per task, not per model. The same judge can be reliable on one property and poor on another, and agreement measured on easy items does not transfer to hard ones (evidence).

HealthBench’s grader validation is a large-scale instance: on consensus rubric criteria graded by several physicians, the GPT-4.1 grader was scored against the physicians as each physician was scored against the others. Its macro-F1 beat the average physician’s in five of seven themes and the 33rd percentile of physicians in all seven (Arora et al. 2025).

Calibration

A judge that outputs a probability (“the panel would renew this with probability 0.7”) gives more information per audited item than one that outputs a label, and its probabilities can be scored. Use a strictly proper scoring rule, one whose expected score is optimized only by reporting one’s true belief: the Brier score (squared error) or the log score (Gneiting & Raftery 2007). The Brier score decomposes into a reliability term (forecasters’ name for calibration error, unrelated to the reliability above), a resolution term, which measures how much the forecasts separate items with different outcomes, and an uncertainty term fixed by the base rate (Murphy 1973). Plot calibration by binning items by stated probability and comparing each bin’s mean to its observed rate. Resolution is what makes a calibrated judge useful rather than merely honest: a judge that says 0.8 for every item is perfectly calibrated at an 80% base rate and useless.

LLMs’ verbalized confidences tend to be overconfident (Xiong et al. 2023), so a judge’s raw probabilities should be recalibrated on audited items before use, and the calibration then checked on audited items not used to fit it. At audit scale, use Platt scaling: its two parameters beat isotonic regression on calibration sets below roughly 200–1,000 items (Niculescu-Mizil & Caruana 2005).

Calibration matters operationally because escalation rules run on it. “Send items below 70% confidence to a human” means something only if 70% means 70%. The sibling wiki’s calibration chapter covers the case where calibration must hold in the tail.

Error bars and sample sizes

Every number above is an estimate from a sample of items, and the sample is usually small because labels are expensive. Miller (2024) sets out the practice for LLM evals, and it applies unchanged to judges:

• treat the items as a draw from a larger population and report standard errors;
• cluster the standard errors when items come in related groups (several criteria from one document, several questions from one passage); in Miller’s example clustered errors were up to three times the naive ones;
• compare two judges on the same items using paired differences;
• run a power analysis before collecting labels.

For a proportion near $p$, a 95% interval of half-width $h$ needs $n \approx 1.96^2\, p(1-p)/h^2$ items. That is 246 items for ±5 points at $p = 0.8$, and 62 for ±10. Kappa is far noisier than the proportion it is built from. With both raters at an 80% “yes” rate, a κ estimated from 50 items has a 95% sampling spread of about ±0.3 whether the true value is 0.17 or 0.4. At the true κ ≈ 0.17 typical of peer review, about 300 double-rated items are needed for 80% power to distinguish the judge from chance with the standard test of κ = 0. These figures come from simulating the 2×2 table and are reproduced in the worked example. Bootstrap the interval for κ or α rather than trusting a normal approximation at these sizes.

Prediction-powered inference

The audit design in LLMs as Evaluators produces two datasets: the judge’s score $f$ on all $N$ items, and gold labels $y$ on a random $n$ of them. The naive estimator of a population quantity, such as the share of grants experts would renew, uses only the $n$ gold labels. Plugging in the judge’s scores on all $N$ items is precise but biased by however wrong the judge is. Prediction-powered inference (PPI) uses both. It takes the judge’s average over all $N$ items and corrects it by the judge’s average error on the $n$ audited ones:

$$\hat\theta_{\text{PP}} = \frac{1}{N}\sum_{i=1}^{N} f_i \;-\; \frac{1}{n}\sum_{j=1}^{n} (f_j - y_j)$$

The result is unbiased whatever the judge’s quality, provided the audited items are a uniformly random draw (or have known inclusion probabilities) and the judge is frozen before the audit labels are seen. Its variance is roughly $\mathrm{Var}(f)/N + \mathrm{Var}(f - y)/n$, so the interval narrows as the judge’s errors shrink (Angelopoulos et al. 2023). PPI++ adds a tuning weight on $f$, chosen from the data, so that, asymptotically, the interval is never wider than the gold-labels-only one (Angelopoulos, Duchi & Zrnic 2023). Several refinements fit audit design directly:

• Stratification. Stratified PPI helps when the judge is better on some kinds of item than others (Fisch et al. 2024). This is also the principled form of “oversample near the decision boundary, then reweight”.
• Active selection. Active inference spends the label budget on the items where the model is least certain, while keeping valid intervals (Zrnic & Candès 2024).
• Model evaluation. Applied to evaluating ML models with GPT-4 labels, these methods raised the effective human-labelled sample size by up to 50% (Boyeau et al. 2024).

The gain is easy to state for a mean. With $N \gg n$ and the PPI++ weight, the variance is the gold-only variance times $1 - \rho^2$, where $\rho$ is the correlation between the judge’s score and the gold label, so $n$ audits are worth $n/(1-\rho^2)$ gold labels. Doubling them takes $\rho \approx 0.71$; a tenfold saving takes $\rho \approx 0.95$. When the judge is no more accurate than the model being evaluated, no debiasing method can cut the gold labels needed by more than half (Dorner, Nastl & Hardt 2025), and in their experiments the savings were smaller. In the illustrative simulation below, the judge’s score has $\rho \approx 0.74$ (κ ≈ 0.8 at a 0.5 threshold) and narrows the interval on the renewal rate by about 30%; a noisier judge with $\rho \approx 0.5$ (κ ≈ 0.4) narrows it by about an eighth. PPI makes a weak judge safe to use, because the estimate stays valid; it does not make a weak judge useful.

import numpy as np
rng = np.random.default_rng(0)

def kappa(a, b):
    """Cohen's kappa for two raters' labels on the same items."""
    po = np.mean(a == b)
    pe = sum(np.mean(a == k) * np.mean(b == k) for k in np.union1d(a, b))
    return (po - pe) / (1 - pe)

def bootstrap_ci(a, b, stat=kappa, reps=5000):
    idx = rng.integers(0, len(a), (reps, len(a)))
    return np.nanpercentile([stat(a[i], b[i]) for i in idx], [2.5, 97.5])

def n_for_halfwidth(p, h, z=1.96):
    """Items needed for a 95% interval of +/-h on a proportion near p."""
    return int(np.ceil(z**2 * p * (1 - p) / h**2))

def ppi_mean(y_lab, f_lab, f_all, z=1.96):
    """PPI++ estimate of mean(y): model scores f on all N items, gold y on n."""
    n, N = len(y_lab), len(f_all)
    lam = np.cov(y_lab, f_lab)[0, 1] / ((1 + n / N) * f_lab.var(ddof=1))
    est = lam * f_all.mean() + (y_lab - lam * f_lab).mean()
    se = np.sqrt(lam**2 * f_all.var(ddof=1) / N + (y_lab - lam * f_lab).var(ddof=1) / n)
    return est, est - z * se, est + z * se

# Illustrative population: 1,000 items, experts say "yes" to 80%;
# the model outputs a probability that is informative but noisy.
N, n = 1000, 50
y = rng.random(N) < 0.8
f = np.clip(np.where(y, 0.75, 0.35) + rng.normal(0, 0.15, N), 0, 1)
lab = rng.choice(N, n, replace=False)          # the random audit
y_lab, f_lab = y[lab].astype(float), f[lab]

print("kappa (model at 0.5 vs expert):", round(kappa(f_lab > 0.5, y[lab]), 2),
      "95% CI", np.round(bootstrap_ci(f_lab > 0.5, y[lab]), 2))
m, se = y_lab.mean(), np.sqrt(y_lab.var(ddof=1) / n)
print("audit only:", np.round([m, m - 1.96 * se, m + 1.96 * se], 3))
print("PPI++     :", np.round(ppi_mean(y_lab, f_lab, f), 3))
print("audits for +/-5 points at 80%:", n_for_halfwidth(0.8, 0.05))

Output (NumPy 2.5):

kappa (model at 0.5 vs expert): 0.82 95% CI [0.56 1.  ]
audit only: [0.8   0.688 0.912]
PPI++     : [0.791 0.712 0.87 ]
audits for +/-5 points at 80%: 246

Over 2,000 repetitions of this simulated audit, the gold-only and PPI++ intervals covered the true renewal rate 93% and 94% of the time, near the nominal 95%. Their average widths were 0.22 and 0.15. Note the bootstrap interval on κ: even for a strong judge, 50 items pin κ down only to somewhere between “moderate” and “near perfect”.

Gaming tests

Every measurement above is taken at zero optimization pressure. A judge that will gate anything of value should also be measured against inputs built to fool it, and the attacks in the literature double as a test suite. The evidence for each is in LLMs as Evaluators; the tests are:

Test	What it measures	Pass condition
Null model. Submit a constant response unrelated to the instruction, including crafted ones (Zheng et al. 2024)	whether the judge can be won without doing the task	null submissions score at the floor
Master keys. Append generic openers or symbols (“Thought process:”, ”:”) to wrong answers (Zhao et al. 2025), and universal adversarial phrases transferred from a surrogate model (Raina, Liusie & Gales 2024)	false-positive rate on known-bad inputs	false positives no higher than on unmodified bad inputs
Injection. Embed instructions to the judge in the evaluated text, as found in 2025 arXiv preprints aimed at AI reviewers	whether evaluated content can instruct the evaluator	verdict unchanged by the injected text
Swaps and padding. Reverse answer order, add length without content	position and verbosity bias	verdict invariant to order; score not increased by padding
Best-of-n pressure. Have a model rewrite each test item $n$ ways, keep the judge’s favourite, and have experts grade originals and winners blind	how far judge score and expert grade diverge under selection, the curve Gao, Schulman & Hilton (2022) measured for reward models	the winners’ expert grade rises with the judge’s score, or at least does not fall, as $n$ grows
Selective disclosure. Let one party submit many private variants and keep the best	the process, not the judge: best-of-k inflates scores even against an honest judge (Singh et al. 2025)	variants are counted, or all scores published

Report each result with an error bar, as for any other measurement. Re-run the suite whenever the judge’s model or prompt changes. Passing it is necessary, not sufficient: these are the known attacks, and an evaluated party with query access can search for new ones. The best-of-$n$ test simulates that search only at small $n$. Only random audits with a penalty for detected gaming address unknown attacks (LLMs as Evaluators, step 3).

What to report

For any evaluator, human or model, before its scores are used: the task and population it was measured on; its self-agreement under re-runs or fuzzed prompts; its chance-corrected agreement with humans, with the confusion table and each rater’s base rate; the human–human agreement on the same items, with the same statistic; bootstrap intervals on all of these; its calibration, if it outputs probabilities; its results on the gaming tests; and the date and model version, since every one of these numbers expires when either changes.

Open questions

• How small can the double-rated set be in practice, given that the human–human baseline has the same sampling error as the judge–human number it is compared to?
• PPI is unbiased for what the panel would say, panel noise included. When the target is the latent quality the panel measures badly (κ ≈ 0.2), what estimator, or what second source of signal, recovers it?
• Does a judge’s best-of-$n$ curve at small $n$ predict how fast it degrades when real evaluated parties optimize against it for months?