Patterns & Failure Modes

Status: early draft, synthesized from a June 2026 literature sweep. This is the analytical companion to Evaluation Systems in the Wild: the catalogue lists about 160 systems; this page asks what is repeatedly found across them. Each pattern names a mechanism, the key papers, and the real systems that illustrate it.

The findings cluster into five families. The first is the most consequential for this field: a public evaluation changes what it measures.

1. The measure reshapes the measured

Deploying a public evaluation is an intervention as well as an observation. Espeland & Sauder call this reactivity — “how public measures recreate social worlds” (AJS 2007; book-length in Engines of Anxiety, 2016) — and identify two mechanisms:

Self-fulfilling prophecy. Once a measure is authoritative, audiences act on it, making it true. A law school that drops a US News tier gets weaker applicants, less alumni giving, and worse placement — becoming actually worse, confirming the rank.
Commensuration (Espeland & Stevens, Ann. Rev. Sociology 1998). Collapsing diverse qualities into one number erases information and reorganizes how the rated parties think and allocate effort. Power flows to whoever defines the metric — its categories, inputs, and weights.

Reactivity’s destructive twin is Goodhart’s law. Goodhart (1975, monetary policy) and Campbell (1979, from a 1976 paper; social indicators) stated it independently, and the Lucas critique (1976) made the related point that policy changes break the econometric relationships they rely on. Campbell adds a dose–response claim: corruption scales with the stakes attached. The popular phrasing, “when a measure becomes a target, it ceases to be a good measure,” is Strathern’s gloss (1997), often misattributed to Goodhart.

Manheim & Garrabrant (2018) give the precise decomposition. When proxy M is optimized for hidden goal G, four distinct things go wrong, and three require no bad actor:

Variant	Mechanism	Adversary needed?
Regressional	M = G + noise; the top of M selects partly for noise (winner’s curse)	No
Extremal	the M–G correlation breaks down in the extreme region optimization reaches	No
Causal	M correlates with G but isn’t causally upstream — intervening doesn’t move G	No
Adversarial	an agent who knows you optimize M manipulates it (incl. the “cobra effect” where the metric’s own incentive backfires)	Yes

Two further mechanisms matter. Surrogation (Choi, Hecht & Tayler 2012): people don’t just game the proxy cynically — they psychologically substitute it for the goal, strongest when a single measure is tied to pay. And a behavioral severity ladder (Bevan & Hood 2006, on NHS targets): effort substitution (“hitting the target and missing the point”) → gaming (meeting the letter, subverting the purpose) → outright fabrication. Effort substitution has a formal model: when an agent does several tasks and only some are measured, strong incentives on the measured ones pull effort from the rest, so the best contract may offer weak incentives or a flat wage (Holmström & Milgrom 1991).

The same law governs optimization against automated evaluators, from learned reward models (Gao, Schulman & Hilton 2022) to monitors: when a chain-of-thought monitor’s verdict was added to a reasoning model’s training reward, strong optimization taught the model to hide its intent while still reward hacking (Baker et al. 2025). LLM Evaluators takes up what this means for model judges.

Seen in the wild: US News rankings (reactivity — schools restructure around the formula; a 2022–23 revolt saw many top law/medical schools withdraw); Journal Impact Factor (adversarial — coercive self-citation, Wilhite & Fong, Science 2012; citation cartels); credit ratings (adversarial — issuer “ratings shopping”); NHS waiting-time gaming; cardiac-surgery report cards (adversarial, by selecting on inputs — surgeons avoid sick patients, below).

Design implication. Gaming rises with the stakes on a measure (Campbell) and with its predictability. The levers are to lower the stakes on any single measure, to measure the neglected tasks or weaken incentives on the measured ones, to keep the metric definition out of the rated parties’ hands, and to add randomness to assessment (random audits, rotating or held-out items), at some cost to transparency (Bevan & Hood 2004). The sibling RRP wiki’s hardening chapters develop the versions for automated evaluators.

2. The ratings you collect are a biased sample

Even setting aside gaming, the raw ratings entering a system are not a clean sample of quality.

Distributions are J-shaped / bimodal, from self-selection. Online ratings pile up at 5 stars with a 1-star spike and little middle (Hu, Pavlou & Zhang, CACM 2009; MISQ 2017). Two mechanisms: acquisition bias (buyers already liked the product) and under-reporting / “brag-and-moan” bias (only extreme experiences bother to review). The mean is therefore a biased signal; the full distribution predicts behavior better.
Social influence herds, asymmetrically. In a randomized experiment on >100,000 comments, a single seeded positive vote made the next viewer ~32% more likely to up-vote and, through accumulating herding, raised the comment’s final mean rating by ~25%; a seeded negative vote was corrected by the crowd (Muchnik, Aral & Taylor, Science 2013). Positive herding accumulates; negative does not.
Expert review has low inter-rater reliability. A meta-analysis of peer review (48 studies, ~19,443 manuscripts) found mean agreement ICC ≈ 0.34, κ ≈ 0.17 (Bornmann, Mutz & Daniel, PLoS ONE 2010); the classic NSF re-review found funding “depends to a significant extent on chance” — i.e., on which reviewers are drawn (Cole, Cole & Simon, Science 1981).
Scales drift and compress. Grade inflation is the canonical case: the A-share rose from ~15% of grades (1940) to over 40% (2008), making A the most common grade (Rojstaczer & Healy 2012). The ceiling compresses and the signal degrades — the same shape as reputation inflation (§4).
Fakes are detectable but persistent. Crowdsourced fake reviews have linguistic signatures a classifier catches at ~90% where humans are near chance (Ott et al., ACL 2011), but on reviews Yelp’s filter had flagged, linguistic features did much worse than behavioural ones such as posting patterns (Mukherjee et al., ICWSM 2013). Requiring a verified purchase raises the cost of faking and measurably reduces — but doesn’t eliminate — manipulation (Mayzlin, Dover & Chevalier, AER 2014).

Seen in the wild: Amazon’s J-shape (and its Verified-Purchase badge as the Mayzlin fix); IMDb’s 1/10 polarization (and its Bayesian weighting as the §5 fix); Reddit/HN early-vote snowball (and vote-fuzzing / “controversial” sort as countermeasures); peer review and grant panels (low reliability → motivates more reviewers + calibration).

Design implication. Don’t treat the mean of self-selected ratings as the quality signal. Verify who is evaluating, model the selection process, and prefer distribution-aware metrics.

3. Incentives and funding shape trustworthiness

The economics of quality disclosure predicts that who pays a certifier shapes what it certifies. A coding of the catalogue gives partial support: more documented failures where the rated party pays, but more of them gaming than lenient certification.

Quality information is valuable because of adverse selection. When buyers can’t tell quality, good sellers exit and markets collapse to “lemons” (Akerlof, QJE 1970). Evaluation systems exist to restore those lost gains from trade.
A signal works only if it’s differentially costly to fake for low-quality types (Spence, QJE 1973). The design question is always: can a bad type cheaply mimic this?
Voluntary disclosure should “unravel” to full disclosure — but often doesn’t, because receivers aren’t fully skeptical, disclosure is costly, the sender may not know its own quality, and quality is multidimensional (Grossman 1981; Milgrom 1981; survey: Dranove & Jin, JEL 2010). This is the case for mandatory disclosure.
When the rated party pays, the certifier prefers a coarse, lenient signal. A profit-maximizing monopoly certifier optimally reveals only whether quality clears a low threshold (Lizzeri, RAND 1999); issuer-pays adds ratings shopping and inflation, worse in booms (Bolton, Freixas & Shapiro, J. Finance 2012). Reputation disciplines certifiers only under strong conditions, and price competition can erode honesty rather than improve it (Strausz 2005). The evidence is contested: Fitch’s growing market share coincided with higher, less informative incumbent ratings (Becker & Milbourn, JFE 2011), but the association vanished once unobserved industry effects were controlled (Bae, Kang & Wang, JFQA 2015). Adding competitors is not a reliable fix.
Disclosure backfires when the rated party games the input mix instead of improving. Risk-adjusted cardiac-surgery report cards led surgeons to avoid sick patients, worsening outcomes for the sickest, because providers know more about severity than any adjustment can capture (Dranove, Kessler, McClellan & Satterthwaite, JPE 2003). It works when the metric resists gaming and demand responds: LA restaurant hygiene grade cards raised scores, shifted demand, and cut foodborne-illness hospitalizations partly via real improvement (Jin & Leslie, QJE 2003).
Too many labels destroy a label’s value — the “Groucho effect”: small uncertainty about what a label means makes consumers infer the labeled product is marginal (Harbaugh, Maxwell & Roussillon, Mgmt Sci 2011).
Gatekeepers fail in correlated, predictable ways when they rent their reputation to the issuer who pays them (Coffee, Gatekeepers 2006) — Enron, WorldCom, the 2008 ratings.

Seen in the wild: credit ratings (issuer-pays → inflation); LEED (fee-for-cert; certified New York City offices showed no collective energy savings: Scofield, Energy & Buildings 2013); hospital report cards (cream-skimming); restaurant hygiene (independent inspector + hard-to-game metric → LA grade cards worked, though NYC’s inspections proved inconsistent: Ho, Yale LJ 2012).

Design implication. Credible evaluation needs (a) a signal that is differentially costly to fake, (b) an independent payer or a strong reputational stake, (c) skeptical receivers, and (d) a clear standard. Mandatory, standardized disclosure has the better record than voluntary, issuer-paid coarse certification, provided the rated party cannot improve its score by choosing its inputs; risk adjustment alone did not prevent that.

4. Reputation systems converge on the same arms race

Reputation is real but modestly priced. A matched-item eBay field experiment found established reputation raised willingness-to-pay ~8% (Resnick et al., Exp. Econ. 2006).
Reputation inflation is the central pathology. eBay feedback runs ~99% positive, so even high-90s scores barely discriminate between sellers; dividing positives by all transactions, so that silence counts against a seller, gives a measure that predicts whether buyers return (Nosko & Tadelis, NBER 2015). On a large online labour market the share of workers given a perfect rating rose from 33% to 85% in six years, while private ratings of the same jobs, never shown to the workers, fell (Filippas, Horton & Golden, Marketing Science 2022). The authors attribute this to raters’ reluctance to harm the person they rate; every market they studied was person-to-person.
Bilateral feedback causes retaliation, so platforms moved to blind / simultaneous-reveal or one-sided feedback (Bolton, Greiner & Ockenfels, Mgmt Sci 2013). That fixes retaliation, not inflation: the labour market above already used simultaneous reveal.
But selection bias often dominates retaliation: on Airbnb, who chooses to review biases scores more than retaliation, with “socially induced reciprocity” from face-to-face contact suppressing negatives (Fradkin et al., EC 2015).
Cheap pseudonyms enable whitewashing — abandon a bad identity, re-enter clean — so cooperation survives only via a costly “newcomers pay dues” convention (Friedman & Resnick 2001); and without a trusted identity authority one party can mint many fake identities (the Sybil attack, Douceur 2002) — the engine of fake reviews and ballot-stuffing.
Cold start is an efficiency loss, not just unfairness: newcomers can’t get the first transaction that would build reputation; subsidizing it and publishing detailed evaluations raised later earnings enough to indicate they had been inefficiently excluded (Pallais, AER 2014). (Foundational survey: Tadelis, Ann. Rev. Econ. 2016.)

Seen in the wild: eBay (the most-studied — inflation, retaliation→one-sided ratings, EPP search ranking); Airbnb (double-blind reveal; selection bias; “New listing” badges); Uber (ceiling compression ~4.6+; face-to-face suppression; driver re-registration); Amazon/Yelp (Sybil fakes and the filters that fight them).

Design implication. Verifying who evaluates, by tying identity to a scarce resource, is the recurring fix for fakes, and information-flow design (blind, one-sided, recency-weighted) is the fix for retaliation. Neither stops inflation; expect consequential scores to drift up, and recalibrate them against signals that cost the rater less to give, such as silence or private feedback. This is the trust-network problem in miniature.

5. The scale and the aggregation rule are not neutral

Naive averages mis-rank low-volume items. A 100%-positive item with 2 votes outranks a 95%-positive item with 500 under mean sorting. Two principled fixes: the Wilson score lower bound for binary votes (Evan Miller, 2009; Reddit’s “best” sort) and Bayesian shrinkage toward a global prior weighted by volume (IMDb’s Top-250 formula; Trustpilot’s TrustScore). Both regularize the cold-start problem.
Scale granularity depends on who aggregates. For one respondent, reliability, validity and discriminating power rose with the number of response categories up to about 7, test–retest reliability tended to fall beyond 10, and respondents preferred 10 points (Preston & Colman, Acta Psychologica 2000). For ranking items across many raters, an up/down vote removes differences in how raters use a scale and volume restores the resolution; 5-star scales behave nearly bimodally anyway (§2).
Pairwise comparison often beats absolute scores. Humans judge “is A better than B?” more reliably than “rate A 1–5”; Bradley–Terry (1952) / Elo model this and sidestep scale-anchoring — now used to rank LLMs in preference arenas, where selective disclosure of privately tested variants has biased the rankings (Singh et al. 2025). (Caveat: online Elo is order-sensitive; the static MLE is more robust; cyclic preferences break the model.)
The aggregation rule materially changes the ranking. Mean vs. median vs. Bayesian vs. positional Borda vs. Kemeny (the maximum-likelihood, Condorcet-consistent — but NP-hard — rule) give genuinely different winners on the same votes (Dwork et al., WWW 2001). Arrow-style impossibility lurks underneath: there is no canonical “correct” aggregator.

Design implication. Choosing the output format and aggregation rule is choosing your failure mode and partly choosing your answer. Make it explicit and defensible; for sparse data, regularize (Wilson or Bayesian); when raters are noisy, prefer pairwise.

What this means for evaluation engineering

Pulling the five families together:

Deploying an evaluation changes what it measures. Rated parties optimize the proxy, more so as the stakes rise, and a system that ignores this feedback loop will be gamed. Moderate stakes per measure, randomness in assessment and control of the metric definition are the levers (§1).
Whether the rated party pays is the funding variable with some evidence behind it. Theory predicts lenient certification when the rated party pays. The catalogue coding finds about 3.5 times the odds of documented failure in such systems, more often through gaming than leniency, and no signal in the broader payer typology. It cannot separate payment from stakes, so treat the payer as a design variable worth controlling, not a proven lever; Evaluation as a System takes up the consequences.
Raw ratings are a biased, gameable sample. Verifying who evaluates and aggregating with the whole distribution in view are what separate signal from noise.
Every output format and aggregation rule embeds a choice with predictable, different failure modes. None is neutral.
The hard problems are old and partly solved. Adverse selection, signalling, unravelling, reputation inflation, shrinkage estimators and the reactivity of public measures have decades of theory and evidence behind them; an automated evaluation system should start from this literature, not rediscover it.

Core references by theme

Reactivity & quantification: Espeland & Sauder (AJS 2007); Espeland & Stevens (1998, 2008); Porter, Trust in Numbers (1995); Power, The Audit Society (1997); Merry, The Seductions of Quantification (2016); Muller, The Tyranny of Metrics (2018); Davis, Kingsbury & Merry, Governance by Indicators (2012).
Goodhart & gaming: Goodhart (1975); Lucas (1976); Campbell (1979); Strathern (1997); Manheim & Garrabrant (2018); Holmström & Milgrom (1991); Bevan & Hood (2004, 2006); Choi, Hecht & Tayler (2012/2013); Krakovna (specification gaming); Gao, Schulman & Hilton (2022); Baker et al. (2025).
Rating bias: Muchnik, Aral & Taylor (2013); Hu, Pavlou & Zhang (2009, 2017); Godes & Silva (2012); Bornmann et al. (2010); Cole et al. (1981); Ott et al. (2011); Mukherjee et al. (2013); Mayzlin et al. (2014); Rojstaczer & Healy (2012).
Certification & disclosure: Akerlof (1970); Spence (1973); Grossman (1981); Milgrom (1981); Lizzeri (1999); Strausz (2005); Becker & Milbourn (2011); Bae, Kang & Wang (2015); Dranove & Jin (2010); Dranove et al. (2003); Jin & Leslie (2003); Bolton, Freixas & Shapiro (2012); Harbaugh et al. (2011); Coffee (2006).
Reputation systems: Resnick et al. (2000); Dellarocas (2003); Resnick & Zeckhauser (2002); Resnick et al. (2006); Friedman & Resnick (2001); Douceur (2002); Bolton, Greiner & Ockenfels (2013); Nosko & Tadelis (2015); Filippas, Horton & Golden (2022); Fradkin et al. (2015); Pallais (2014); Tadelis (2016); Jøsang, Ismail & Boyd (2007).
Scales & aggregation: Evan Miller (2009); Wilson (1927); Preston & Colman (2000); Bradley & Terry (1952); Elo (1978); Dwork et al. (2001); Hunter (2004).

See Adjacent Fields & Literature for the broader disciplines and Evaluation Systems in the Wild for the systems these patterns describe.