LLMs as Evaluators

Status: early draft, September 2026; evidence sections revised October 2026. How to measure a judge (chance-corrected agreement, baselines, error bars, prediction-powered inference, gaming tests) is in Measuring an Evaluator. This chapter develops the claim the whole wiki rests on: that LLMs are the cheap, general executor the 2021–22 program lacked (see Lineage). Background citations are in the LLM-evaluation section of Adjacent Fields.

The claim, and what would falsify it

The founding program stalled on labor. Its architecture needed someone to draft evaluations, structure ontologies, and run calculations for thousands of items, and paying analysts for that made the whole system too expensive to run at scale. The “why now” of this wiki is that language models do much of that labor at a small fraction of the cost.

Stated as a testable claim: for a given class of evaluation questions, an LLM-based evaluator agrees with a trusted panel about as often as a second, independent panel would, at a cost per item at least ten times lower, and keeps that agreement on audited items after months of deployment, when the evaluated parties have had time to optimize against it. The claim is falsified for a question class if post-deployment LLM–panel agreement on randomly audited items falls clearly below panel–panel agreement.

The evidence is uneven across the three parts. Accuracy and cost are well supported for checkable or chat-style tasks: code that runs, forecasts that resolve, preferences between chat responses. For judgment without ground truth, the closest measurements come from automated essay scoring and LLM review of research papers; normative, long-horizon judgment is largely unmeasured. On the third part the direct evidence is negative: undefended LLM judges have been gamed in the lab and in the wild (below). That part decides whether LLM evaluation is engineering or a demo.

What LLMs cheapen, component by component

Mapped onto the four components:

Component	What an LLM does	State of the evidence
Ontology	drafts structured question sets, definitions, resolution criteria	cheap to generate; quality control is the bottleneck. LLM-authored questions skew toward ambiguous entities and timeframes
Calculation	writes and debugs estimation code	deployed: Squiggle AI generates probabilistic models at $0.10–0.35 per workflow, with documented systematic overconfidence
Prediction	forecasts directly, or predicts an expensive evaluator	on ForecastBench the best LLM systems trailed superforecasters by 0.017 difficulty-adjusted Brier points in January 2026, about a year of LLM progress at the recent rate (Karger et al. 2024; FRI 2026)
Evaluation	judges quality, scores against rubrics, critiques documents	the most-studied use: the LLM-as-judge literature below; deployed as RoastMyPost, whose authors report a “significant” false-positive rate

The pattern across the column: the cost per call is similar everywhere, but LLM output is cheapest to validate where it can be checked. Code runs or fails; forecasts resolve; a rubric score does not check itself. So the accuracy-vs-cost question has to be asked separately for each component, and it is hardest for evaluation.

What the judge literature measures

The research on LLMs as judges has moved from “can they judge?” to cataloguing how they fail:

• Agreement. On open-ended chat responses, a strong LLM judge agreed with human preferences about as often as humans agreed with each other: over 80% in the landmark study (Zheng et al. 2023). That is the accuracy half of the claim, for one task class.
• Agreement depends on the task and on the item’s difficulty. Across 20 annotated NLP datasets, the same judges were reliable for some properties and poor for others (Bavaresco et al. 2025). Where humans agree closely with each other, only the largest judges aligned reasonably with them, still well short of human–human agreement, and judges tended to be lenient (Thakur et al. 2024). On expert-graded finance questions, judges without a reference answer agreed with experts mainly on questions the judges could answer themselves; an expert-written reference answer largely closed the gap (Krumdick et al. 2025). A judge validated on easy items is not validated on hard ones.
• Biases that are truth-independent. The same study names position bias (favoring whichever answer appears first), verbosity bias (favoring longer answers), and self-enhancement (favoring answers the judge generated itself; later work finds judges also favor models similar to themselves, Goel et al. 2025). Each has a known partial fix: judge both orderings and average (Wang et al. 2023); regress out length (Dubois et al. 2024); use a judge from a different family.
• Sensitivity to phrasing. Meaning-preserving formatting changes alone can swing few-shot task accuracy by up to 76 points (Sclar et al. 2024). That result is for classification tasks, not judging, but judges run on the same prompt machinery. QURI’s opinion fuzzing proposal treats this as something to measure: sample across prompts, models, and personas, and report the spread as the judgment’s uncertainty.
• Teams match the model and hallucinate less. On catching bugs in real assistant code, trained critic models’ critiques were preferred over human critiques 63% of the time, and human–model teams caught about as many bugs as the model alone while hallucinating fewer (McAleese et al. 2024). The cheapest reliable deployment may be making each expensive evaluator faster, not removing them. That is the use-case page’s augmentation framing, measured.
• Judgment closer to evaluation. Automated essay scoring has used machine–human against human–human agreement as its acceptance test for over a decade. A commonly applied criterion, attributed to Williamson, Xi & Breyer (2012), requires machine–human quadratic-weighted kappa to fall no more than 0.10 below human–human. On research papers, the points GPT-4 raised overlapped with human reviewers’ points about as much as two human reviewers’ overlapped with each other: 30.85% vs. 28.58% for Nature-family journals, 39.23% vs. 35.25% for ICLR (Liang et al. 2024). HealthBench scored its rubric grader against physicians as physicians are scored against each other, and it beat the average physician in five of seven themes (Arora et al. 2025; details).
• Deployed in review. At AAAI-26 every one of 22,977 main-track papers received one labelled AI review, without scores or a recommendation, alongside the human reviews (Biswas et al. 2026). In a survey with 5,834 responses, authors and committee members preferred the AI reviews on six of nine criteria, including identifying technical errors, but also rated them somewhat more likely to contain technical errors themselves. About half of committee respondents (49.4%) said the AI review overlooked points a human would likely have caught. This measures perceived usefulness, not agreement with a panel’s verdict.

None of these measures the normative, long-horizon judgments on the evaluation side of the field’s core distinction, such as “how much did this organization reduce risk?”, and none was measured while the evaluated parties optimized against the judge.

Evidence under optimization pressure

The direct evidence is that undefended judges fail:

• Null models win benchmarks. A constant response, unrelated to the instruction, reached an 86.5% length-controlled win rate on AlpacaEval 2.0, 83.0 on Arena-Hard-Auto and 9.55 on MT-Bench (Zheng et al. 2024).
• Short strings inflate scores. Universal adversarial phrases appended to a text push LLM judges toward maximum scores, transfer from a surrogate model to unseen judges, and work better against absolute scoring than against pairwise comparison (Raina, Liusie & Gales 2024). In reference-based grading, bare symbols or openers such as “Thought process:” elicited false-positive rewards from many judges, including frontier models (Zhao et al. 2025). Nobody designed these: they surfaced when a policy trained by reinforcement learning against a Qwen2.5-72B judge collapsed to short reasoning openers that the judge kept rewarding.
• In the wild. In July 2025 Nikkei Asia found hidden instructions, in white or tiny text, in 17 arXiv preprints from 14 institutions, telling AI reviewers to give positive reviews only. This is the evaluated party optimizing against the judge in a judgment-bound domain. It followed the spread of LLMs into reviewing: an estimated 6.5–16.9% of review text at four 2023–24 AI conferences was substantially LLM-modified (Liang et al. 2024).
• Human judges degrade the same way. RLHF made models better at convincing time-limited human evaluators without making them more correct, raising the evaluators’ false-positive rate by 24.1% on a question-answering task and 18.3% on a programming task (Wen et al. 2024).
• The process gets gamed, not only the judge. On Chatbot Arena, a few providers privately tested many variants and could retract the scores they did not want; Meta tested 27 before the Llama 4 release. Best-of-k selection inflates scores even against an honest judge (Singh et al. 2025).

Apart from training runs, these are proofs of concept against public or known judges. How fast a judge deployed on people degrades, and what defenses buy, is unmeasured.

The two system-level failures

Per-item biases can be engineered around. Two failures show up only at the level of the whole system, which makes them this wiki’s concern.

1. Correlated error. The standard move with a noisy judge is to average several. That works only if their errors are independent, and LLMs’ are not: when two models are both wrong they tend to agree on the same wrong answer, far above chance (Kim et al. 2025), and error similarity rises with capability (Goel et al. 2025). So a panel of five models can deliver the confidence of five judges with the accuracy of fewer; how many fewer depends on correlation that has to be measured per task. Human expert panels are correlated too, but through different channels (shared training, groupthink). That makes mixing human and model judges a plausible way to buy independence that adding models cannot, though it is untested (see Open questions). The sibling RRP wiki calls correlated error the field’s largest undefended threat, and the Delegation Risk wiki recomputes Kim et al.’s data on the scale needed to price stacked checks.

2. Goodhart under optimization. An LLM judge validated on a fixed test set is validated at zero optimization pressure. Deploy it as a gate that anything valuable depends on, and whoever is being evaluated starts optimizing against it. The evidence above shows that this happens; the reward-modeling literature shows the shape of what follows: as optimization against a proxy reward model increases, the proxy score keeps rising while the true objective (there, a larger “gold” reward model) first rises and then falls, both under reinforcement learning and under best-of-$n$ selection (Gao, Schulman & Hilton 2022). Best-of-$n$ is the closer analogue to an applicant drafting several versions of a report and submitting the one a public judge scores highest. The numbers don’t transfer to report-writing; the pattern, proxy up and target eventually down, is Goodhart’s law, and the same selection can be run as a test before launch. The design consequence: a judge’s validity is indexed to the optimization pressure it was validated under. Re-validate when the stakes attached to its scores rise, when the evaluated population changes, and whenever the underlying model is updated.

The design pattern: LLMs as the cheap layer

Put together, these findings suggest one architecture, which is the prediction–evaluation system with the cheap predictor now a model:

1. Keep an expensive, trusted evaluator. Humans, or a slow, heavily checked process. It is the reference the rest of the system is scored against.
2. Have the LLM predict that evaluator on every item, not judge in the abstract. “What would the panel say?” can be scored; “what is true?” often cannot. This makes the question scorable, not the answer true: the LLM inherits the panel’s errors, and the panel is itself a target for the same optimization.
3. Audit a hidden random subset with the expensive evaluator. Random audits measure how much gaming is happening; by themselves they don’t deter it. In Techniques, random resolution disciplines predictors because they are paid on the audited items. Here the gamer is the evaluated party, and if the model is public or queryable it can search for a winning report offline for almost nothing. Deterrence needs a penalty when an audit exposes gaming (disqualification, a forfeited deposit), which is the inspection-game bound in the sibling wiki. Keeping the model and prompt private raises the cost of the offline search. The statistics for this step exist: prediction-powered inference combines the model’s labels on every item with the audited subset to give confidence intervals that are valid whatever the model’s quality, and narrower the better the model is (Angelopoulos et al. 2023). The saving is bounded: a judge no more accurate than what it evaluates can at most halve the audits needed (Dorner, Nastl & Hardt 2025).
4. Report agreement against a second panel, not against perfection or a single rater, and chance-corrected: two raters who both approve 80% of items agree 68% of the time by chance. A model predicting a panel’s consensus can beat an individual rater just by averaging, so the fair bar is a held-out panelist against the rest (Measuring an Evaluator). Two further caveats: where panels rarely agree, parity is easy and means little; and parity says nothing about where the errors fall. Weight agreement by decision impact (errors near the funding line matter most), and measure it on audited items after deployment, not only on a fixed pre-deployment set.
5. Escalate disagreement. Items where fuzzed prompts disagree, or where the model’s confidence is low, go to the expensive layer first. Trust or Escalate formalizes this as a cascade: cheaper judges escalate to stronger ones when unsure, with a provable guarantee of a user-specified level of human agreement on the items the cascade does judge (Jung, Brahman & Choi 2024). The guarantee is statistical, tuned on a labelled calibration set, and holds for items that resemble that set. This catches honest uncertainty, not attacks: an input optimized against the model tends to produce a confident, consistent verdict, and a known escalation rule becomes one more target. Random audits (step 3) remain the only check on gaming.
6. Re-validate on drift. New model version, new stakes, or new population means a fresh audit sample before trusting old agreement numbers.

The worked example puts numbers on this pattern for a grantmaking use case.

What LLMs do not fix

Evaluation’s defining requirement is that the audience trusts the result. Cheap judgments nobody believes move no decisions. A model can match a panel’s accuracy and still carry none of its standing, because trust in an evaluator comes partly from accountability: a named panel can be questioned, blamed, and replaced. This is crux 4, and nothing in the literature above resolves it. Nor do LLMs change the cultural constraint: a cheaper evaluator of people and organizations meets the same pushback, faster.

Open questions

• Beyond essay scoring, paper review and clinical rubrics, for which judgment-bound questions has chance-corrected LLM–panel agreement been measured against panel–panel agreement?
• How fast does an LLM judge’s accuracy decay as the evaluated parties start optimizing against it, and how large a penalty on audit failure is needed to slow the decay?
• How much independence does a mixed human–model panel buy over a model-only panel, per dollar?
• What audit fraction keeps a deployed LLM evaluator honest, as a function of the stakes attached to its scores? The sibling wiki’s inspection-game bound is one starting model.