Skip to content
Evaluation Engineering

Evaluation Engineering

Evaluation engineering is the design of systems that produce large numbers of estimates and evaluations, such as grant reviews, charity assessments and research triage, at known cost and under incentives that keep them honest.

We know how to produce one careful evaluation of a grant or a charity. The bet here is that the hard part is now the ten-thousandth: producing it cheaply and reliably enough that the results get used. The core pattern is old, with precedents well outside this project. A cheap layer scores every item, a hidden random sample also goes to an expensive, trusted evaluator, and that sample measures how far the cheap layer can be trusted.

Who pays is part of the design. In the wiki’s coding of about 160 standing systems, an evaluator’s main source of funding says little about whether it failed. What does matter is whether the rated party pays it anything at all, through fees, advertising or logo licences: those systems had about 3.5 times the odds of a documented failure.

0% 20% 40% 60% all 154: 27% Share of systems with an established failure, with 90% interval Who is the evaluator’s main funder? The rated party 12 of 35 Mixed 11 of 39 A third party 9 of 35 Public money 4 of 19 Users 5 of 26 Does the rated party pay it anything? No 10 of 62 Yes, as main funder 12 of 35 Yes, on the side 15 of 32 0% 20% 40% 60% all 154: 27% Share with an established failure Who is the evaluator’s main funder? The rated party 12 of 35 Mixed 11 of 39 A third party 9 of 35 Public money 4 of 19 Users 5 of 26 Does the rated party pay it anything? No 10 of 62 Yes, as main funder 12 of 35 Yes, on the side 15 of 32
Share of 154 rated systems with an established failure: one found by a regulator, court, audit or named investigation. Funding coded from public sources by one coder; “none found” is not “did not fail”. Where the rated party pays anything, 27 of 67 failed, against 10 of 62 where it pays nothing: an odds ratio of 3.5 (90% interval 1.6–7.8). The lower three rows leave out 25 systems whose side payments were not checked. Method and caveats.

This is not the AI-model “evals” field, though the two overlap wherever an LLM is the judge. The pages are working notes, not settled positions. They were drafted largely with Claude Code, and each states its own status.

Start with Chapter 1, or see the ideas applied in the worked example: a funder reviewing 1,000 grants a year under four designs, with illustrative costs.

Contents

Section titled “Contents”

Start here

Section titled “Start here”
  1. Evaluation Engineering — what the field is, and what it is not
  2. Estimation vs. Evaluation — checkable numbers versus judgment
  3. Why It Matters — where audited evaluation would change decisions
  4. Cruxes — the open questions that would most change the field

Part I — The Systems View

Section titled “Part I — The Systems View”
  1. Evaluation as a System — the design space and its four properties
  2. The Four Components — prediction, calculation, ontology, evaluation

Part II — Methods & Techniques

Section titled “Part II — Methods & Techniques”
  1. Evaluation Methods — expert panels to LLM judges, and how each fails
  2. Techniques — patterns that wire the parts together, with their evidence
  3. LLMs as Evaluators — what they cheapen, and how judges get gamed
  4. Measuring an Evaluator — agreement, calibration and error bars
  1. The Capability Ladder — five rungs for grading a system
  2. Launch Checklist — what to have in place before scores are used

Part III — Evidence from the Wild

Section titled “Part III — Evidence from the Wild”
  1. Evaluation Systems in the Wild — about 160 systems, and how they failed
  2. Patterns & Failure Modes — reactivity, Goodhart effects, rating inflation

Part IV — The Environment

Section titled “Part IV — The Environment”
  1. Epistemic Culture — what happens when the evaluated push back

Reference

Section titled “Reference”

Glossary, Objections & FAQ, Lineage, Related Work, Adjacent Fields, Open Problems. The whole wiki is also one text file, llms-full.txt.

Other CAIRN sites: Delegation Risk, Robust Reasoning Processes, CAIRN Handbook.

Maintained by the Quantified Uncertainty Research Institute. Source on GitHub.