Evaluation Systems in the Wild

Status: curated catalogue, assembled from a June 2026 sweep; coded by payer, method and documented failure in October 2026 (below). This is the descriptive companion to the conceptual pages: the world is already full of standing systems that produce many evaluations of a repeated type. They are the field’s case record: its documented successes, failures and capture stories. The founding program called for exactly this survey (the proto “process catalogue”).

How to read this

Every entry names what it evaluates, its output format, its method, and one notable weakness or failure. Before the catalogue, four cross-cutting lenses that the systems below keep illustrating:

Method archetypes. Independent lab testing · professional anonymous inspection · expert panel/committee · critic aggregation · crowd reviews (often Bayesian/credibility-weighted) · two-sided/reciprocal rating · statistical/algorithmic model · composite index · market-based · regulatory review. These are the menu on Evaluation Methods, seen in the wild.

Output formats. Stars (1–5), points (0–100), letter grades (A–F, AAA–D), rankings, pass/fail certification, tiers, probabilities, and dollar estimates. The format is an editorial choice with consequences — binary “fresh/rotten” discards intensity; 100-point wine scales compress to 88–100.

Who pays. Three funding archetypes recur:

Independent / nonprofit, buys its own units, no ads (Consumer Reports, Which?, Stiftung Warentest, IIHS). Even here, Which? and Stiftung Warentest sell logo licences to makers whose products score well.
Affiliate / ad-funded editorial (Wirecutter, RTINGS, CNET) — recommendation-revenue conflict.
Rated party pays — issuer-pays ratings, fee-for-certification, award-licensing (Moody’s/S&P, UL, ISO, LEED, J.D. Power, DXOMARK).

How much this matters is tested below; the answer is weaker than the hand-picked contrasts suggested.

Recurring failure modes. Fake reviews / astroturfing / review bombing · gaming and Goodharting (citations, ratings) · capture (credit ratings in 2008; the World Bank’s Doing Business scandal) · pay-to-play · grade inflation · self-declaration abuse · snapshot-in-time validity · reciprocal retaliation in two-sided systems · opaque weighting.

Consumer products & testing

Consumer Reports — consumer products & services. 0–100 scores + “Recommended”/“Best Buy”. Independent lab testing + member reliability surveys; nonprofit, buys all units at retail, no ads. Weakness: affiliate-link revenue creates a perceived conflict.
Which? (UK) — products & services. Scores + “Best Buy”/“Don’t Buy”. Independent lab testing; nonprofit.
Stiftung Warentest (Germany) — products & services. German school-grade scale, printed on packaging. Undercover purchasing + outsourced scientific testing; ad-free, government-seeded foundation. Weakness: sued by manufacturers ~10×/year.
CHOICE (Australia) — products & services. Scores + “CHOICE Recommended”. Accredited in-house labs; nonprofit.
Wirecutter (NYT) — consumer gear. Narrative “top pick”/“budget pick”, no scores. Hands-on reviewer testing; affiliate revenue, no on-site ads.
RTINGS — TVs, monitors, headphones, etc. 0–100 overall + per-use-case scores. Standardized in-house bench measurements. Weakness: 2025 scoring overhaul drew backlash over weighting; since March 2026 full test results are members-only.
DXOMARK — camera/phone image, audio, display. Open-scale scores + sub-scores. Lab + structured perceptual testing. Weakness: core conflict — sells consulting to the firms it scores.
Tom’s Hardware / PCMag — PC components/devices. Stars + “Editors’ Choice” + hierarchy charts. Standardized benchmarking labs; affiliate + ads.
J.D. Power — vehicle quality/satisfaction. PP100 (problems per 100) + segment awards. Large owner surveys. Weakness: clients are the automakers; winners license the awards to advertise.
Kelley Blue Book / Edmunds — vehicle valuation & reviews. Dollar values (TMV / Blue Book Value). Statistical models on transaction data. Weakness: dealer-referral revenue; values can diverge from actual sales.
Robert Parker / Wine Advocate, Wine Spectator — wine. 100-point scale. Professional critics, often blind. Weakness: “Parker palate” homogenization; score compression into 88–100.
Untappd, BeerAdvocate, RateBeer — beer. Crowd star/score averages. Weakness: hype/novelty bias; RateBeer is owned by AB InBev (BeerAdvocate/Untappd by Next Glass) — big-brewer ownership of the rater.
Coffee Review — coffee. 100-point scale. Expert blind cupping. Weakness: pay-to-submit service; mostly 90+ published.
America’s Test Kitchen / Cook’s Illustrated — kitchen gear, ingredients, recipes. Tiered verdicts. Expert panels, blind taste tests, heavy repeated testing; no ads.

Media, entertainment & content

IMDb — films, TV, people. 1–10 Bayesian-weighted average; “Top 250”. Crowd votes. Weakness: vote brigading; demographic skew; polarized 1/10 voting.
Rotten Tomatoes — film/TV. % “fresh” critics + audience score. Binary critic aggregation (discards intensity). Weakness: review bombing of audience scores; binary loses nuance.
Metacritic — film/TV/games/music. 0–100 Metascore. Weighted critic average. Weakness: undisclosed weights; user-score review bombing.
OpenCritic — games. Top Critic Average (transparent, unweighted mean). Weakness: no user component; small samples for niche titles.
Steam user reviews — games. Positive/negative tiers, “Recent” vs “All-time”. Owner-gated binary. Weakness: protest review bombing.
Goodreads — books. 1–5 simple average. Crowd, minimal verification. Weakness: sockpuppet scandals; pre-publication bombing of unreleased books.
Letterboxd — film. 0.5–5 stars, weighted average. Cinephile crowd.
RateYourMusic — music. Credibility-weighted crowd charts. Weakness: opaque user-weighting; canon/obscurity skew.
MyAnimeList / AniList — anime/manga. Bayesian-weighted scores. Weakness: score inflation; seasonal brigading.
Billboard charts — songs/albums. Weekly ranking. Statistical blend of streams + sales + airplay. Weakness: bundling/stream-campaign manipulation; opaque weights.
Pitchfork — albums. Single critic 0.0–10.0. Weakness: single-reviewer subjectivity.
Nielsen — TV/streaming audience. Ratings/share. Panel + (since 2025) big-data hybrid. Weakness: panel sampling error for niche audiences; clients are the rated networks.
Common Sense Media — media for kids. Age (2–18) + 5-star quality. Expert reviewers on child-development criteria; nonprofit.
Age/content boards — MPA (G–NC-17, anonymous parent panel), ESRB (games), PEGI (games). Self-regulatory; rely on publisher disclosure (hidden content can slip).

Finance, credit, insurance & risk

FICO / VantageScore — consumer credit. 300–850. Proprietary statistical model. Weakness: opacity; thin-file exclusion; entrenched gatekeeper.
Credit bureaus — Experian, Equifax, TransUnion. Full credit reports. Data aggregation. Weakness: common data errors hard to dispute; the 2017 Equifax breach (~147M people).
Bond/sovereign ratings — Moody’s, S&P Global, Fitch. AAA–D letter scales. Analyst committee + models, issuer-pays. Weakness: the canonical capture story — inflated AAA on mortgage securities; S&P settled with the US Justice Department and states for $1.375B (2015) and Moody’s for about $864M (2017).
Morningstar — funds/stocks. 1–5 stars (quant, backward-looking), Medalist (forward-looking), Economic Moat. Weakness: star ratings weakly predict future performance; “star chasing”.
ESG ratings — MSCI (AAA–CCC), Sustainalytics (0–100 risk), S&P Global ESG. Weakness: ratings divergence — correlations between six ESG raters average 0.54 (range 0.38–0.71), against 0.99 for credit ratings (Berg, Kölbel & Rigobon, Review of Finance 2022).
A.M. Best — insurer financial strength. A++–F. Insurance-specialist analysis; largely issuer-pays.
Credit-based insurance scores — LexisNexis, FICO. Risk scores for underwriting. Weakness: fairness/proxy-discrimination concerns; restricted or banned in several US states.
Zillow Zestimate — home value. Dollar estimate + range. ML automated valuation. Weakness: off-market median error ~7%; ignores condition; “not an appraisal”.
Cyber risk — BitSight (250–900), SecurityScorecard (A–F), CVSS (0–10, open standard). Weakness: external-only signals; CVSS severity routinely conflated with risk → “everything is Critical”.
Dun & Bradstreet PAYDEX — business payment reliability. 1–100. Vendor-reported trade data.

Academia, science & education

Scholarly peer review — manuscripts. Accept/revise/reject. Expert review, mostly unpaid. Weakness: low inter-rater reliability (“lottery”); slow; weak fraud screening.
Journal Impact Factor (Clarivate) — journals. Citation ratio. Weakness: heavily gamed (coercive/self-citation, cartels); DORA condemns its use to judge individuals.
h-index — authors. Single integer (productivity × impact). Weakness: field-dependent; gameable via self-citation; can’t decrease.
Citation databases — Web of Science, Scopus (Elsevier — also a publisher), Google Scholar (widest, least curated).
Altmetric — online attention. Weighted “donut” score. Weakness: measures attention, not quality; gameable.
University rankings — QS, THE (reputation-survey heavy), ARWU/Shanghai (objective, prize-weighted), US News (self-reported data enabled the Columbia fraud; 2023 boycott), Leiden (bibliometric, deliberately no composite).
REF (UK Research Excellence Framework) — university research. 4*–1* profiles. Expert panel review; allocates ~£2B/yr. Weakness: very high administrative cost.
GRADE / Cochrane RoB 2 — evidence quality / trial bias. Tiered ratings. Structured expert rating. Weakness: domain judgments still subjective.
Standardized tests — SAT/ACT, GRE, PISA, TIMSS. Weakness: scores track family income; teaching-to-the-test.
School ratings — GreatSchools (1–10; historically correlated with race/affluence), Ofsted (England; replaced single-word grades with report cards in 2025 after criticism).
Accreditation — ABET (engineering/computing), AACSB (business schools), US institutional accreditors (Title IV gatekeepers). Weakness: peers accredit peers (conflict); slow on failing schools.

Health, safety, standards & certification

Hospital ratings — CMS star ratings (1–5, federal), Leapfrog (A–F safety), US News Best Hospitals, Healthgrades. Weakness: CMS criticized for penalizing complex/teaching hospitals; Healthgrades and Leapfrog charge hospitals to license their grades for marketing. In March 2026 a federal judge ruled that Leapfrog’s revised method, which gave hospitals that skipped its voluntary survey the lowest possible score on some measures it lacked data for, violated Florida consumer-protection law, and ordered five such hospitals’ grades removed; Leapfrog is appealing (WUSF).
Restaurant hygiene — NYC letter grades (A/B/C), UK Food Hygiene Rating Scheme (0–5). Unannounced inspections; municipal, no fee-for-grade. Weakness: snapshot validity; inspection inconsistency.
Drug/device — FDA, EMA, NICE (cost-per-QALY HTA). Approve/not. Expert regulatory review. Weakness: user-fee funding criticized as “cozy”; QALY thresholds called arbitrary.
Crash tests — IIHS (Good–Poor + Top Safety Pick; insurer-funded, independent of makers), Euro NCAP (0–5 stars), NHTSA 5-Star (most cluster at 4–5★). Weakness: limited scenario set; “test to the test”.
Product safety — UL (lab testing + factory audits, fee-for-cert), CE marking (mostly self-declared). Weakness: UL cost barrier + counterfeit marks; CE self-declaration is gameable.
Energy — Energy Star (a 2010 GAO sting certified a gas-powered “alarm clock” → triggered third-party testing), EU energy label (A–G; rescaled 2021).
ISO 9001 certification — quality-management systems. Pass/fail + surveillance audits. Third-party audit, client pays the auditor. Weakness: “audit shopping”; certifies process not outcome.
LEED — green buildings. Certified–Platinum, points-based; fee-for-cert. Weakness: design- not performance-based — certified buildings don’t reliably use less energy.
B Corp — whole-company social/environmental. Pass/fail seal (≥80/200); fee-for-cert. Weakness: bar seen as low; Dr. Bronner’s dropped the cert in 2025 over multinational dilution.
Food/agriculture — USDA Organic, Fairtrade, MSC seafood (logo-royalty conflict), Rainforest Alliance. Weakness: royalty/fee models create incentives to certify generously.

Hospitality, travel & local business

Michelin Guide — restaurants/hotels. 1–3 stars. Professional anonymous inspectors, multiple visits. Weakness: tourism boards increasingly pay for regional entry (conflict); fine-dining/Eurocentric bias.
AAA Diamonds — N. American hotels/restaurants. 1–5 Diamonds. Anonymous inspectors; nonprofit. Forbes Travel Guide — luxury. 4–5 Star. Inspectors on ~900 standards. Weakness: Forbes also sells training on how to earn its ratings.
Hotel star systems — accommodations. 1–5 stars. Hotelstars Union standardizes 21 European countries; the US has no government system (self-declared “5-star” is meaningless).
Yelp — local businesses. 1–5 stars + automated review filter. Weakness: long-running extortion / pay-to-play allegations.
TripAdvisor — travel. 1–5 bubbles. Crowd, no proof of stay. Weakness: a 2018 investigation alleged ~1 in 3 reviews fake; 200k+ AI-generated reviews removed in 2024.
Google Reviews — places. 1–5 stars + AI moderation. Weakness: ~240M fake reviews removed in 2024; extortion scams at scale.
Booking.com / Hotels.com — accommodations. Score /10, verified guests only, recency-weighted. Weakness: commission model is a structural conflict.
Trustpilot — businesses. TrustScore (Bayesian-weighted). Weakness: paying businesses get more tools (two-tier criticism).
BBB grades — business trustworthiness. A+–F. Composite + accreditation fees. Weakness: in 2010 Los Angeles business owners got a fictitious business named “Hamas” accredited with an A− by paying $425, as ABC’s 20/20 reported (pay-for-grade); the Los Angeles chapter was expelled in 2013.
Glassdoor — employers. 1–5 stars, anonymous, “give to get”. Weakness: anonymity enables fakes; the rated employer pays the host.

Online platforms & reputation systems

eBay feedback — sellers. % positive + detailed star ratings. Transaction-linked. Weakness: extreme grade inflation; seller retaliation led eBay to bar negative buyer feedback.
Amazon reviews — products/sellers. 1–5 stars + Verified Purchase. ML-weighted crowd. Weakness: persistent fake/incentivized reviews; the FTC’s 2024 fake-review rule targets this.
Airbnb — hosts/guests. 1–5 stars, double-blind reveal, Superhost badge. Weakness: retaliation/extortion via review leverage; strong inflation (~4.8+ norm).
Uber / Lyft — drivers/riders. 1–5 reciprocal rolling average. Weakness: drivers deactivated below ~4.6; a 2020 suit alleged aggregating biased customer ratings is discriminatory.
DoorDash — Dashers. 1–5 (last 100) + completion %. Weakness: low deactivation thresholds; ratings reflect restaurant/app delays outside the driver’s control.
Stack Overflow reputation — Q&A expertise. Points + privilege tiers. Weakness: voting rings / sockpuppets (study).
GitHub stars — repo popularity. Integer count. Weakness: a fake-star economy — millions of bought stars, often promoting malware (study).
Reddit karma — contribution. Numeric. Weakness: karma farming via reposts/bots.
App store ratings — Apple (legacy ratings persist), Google Play (recency-weighted). Weakness: bought reviews + review bombing.
Wikipedia pending-changes / editor trust — editor trustworthiness. Permission flags + edit counts. Automated thresholds + admin grants. Weakness: edit count is a shallow, gameable proxy.

Sports & competition rankings

Elo / Glicko-2 — player skill. Numeric rating (Glicko adds a confidence/deviation term). Zero-sum statistical update. Weakness: single K-factor models uncertainty crudely; pool-wide inflation.
FIDE — chess. Elo with tiered K-factors. Weakness: decades-long inflation debates.
ATP / WTA — tennis. Rolling 52-week points. Weakness: no opponent-strength weighting.
OWGR — golf. Strength-of-field-weighted points. Weakness: the LIV Golf exclusion controversy.
FIFA rankings — national football teams. Elo-based “SUM” model (since 2018, fixing the gameable old system).
Sabermetrics / WAR — baseball player value, in wins. Statistical composite. Weakness: the two main versions (bWAR, fWAR) disagree — “which WAR?”.
College Football Playoff committee, AP Poll, Coaches Poll — top-25 rankings. Expert/voter judgment. Weakness: opacity, reputation bias, and (Coaches Poll) direct conflicts of interest.

Corruption Perceptions Index (Transparency International) — public-sector corruption. 0–100. Composite of expert/business surveys. Weakness: measures perceptions, not corruption.
Freedom in the World (Freedom House) — political rights/civil liberties. 0–100 + Free/Partly/Not Free. Expert assessment. Weakness: majority US-government funded (independence critique).
V-Dem — democracy (5 dimensions). 0–1 indices. ~3,500 expert coders → Bayesian IRT with explicit uncertainty bounds. Weakness: expert-coding subjectivity; complex to audit.
EIU Democracy Index — democracy. 0–10 + regime type. Weakness: opaque, proprietary, anonymous experts.
Human Development Index (UNDP) — health/education/income. 0–1. Geometric mean of three indicators. Weakness: only three crude dimensions; arbitrary weighting.
World Press Freedom Index (RSF) — press freedom. 0–100. Abuse tally + expert survey.
Worldwide Governance Indicators (World Bank) — six governance dimensions, with standard errors.
World Bank Doing Business (DISCONTINUED) — ease of doing business. Killed in September 2021 after audits found deliberate data manipulation favoring certain countries under leadership pressure — the cleanest documented case of index capture.
Gallup World Poll / World Happiness Report — wellbeing. Survey means (Cantril Ladder 0–10). Large-N self-report (not expert perception). Weakness: translation/cultural bias; over-reading a single question.
Also: Global Peace Index, WJP Rule of Law Index, Environmental Performance Index, and the ideologically-framed economic-freedom indices (Heritage, Fraser).

Charity & nonprofit evaluation

GiveWell — global health/development charities. Short “Top Charities” list + cost-per-life-saved estimates. Deep in-house CEA, publishes full models. Weakness: very narrow, evidence-rich cause focus.
Charity Navigator — US 501(c)(3)s. 0–4 stars / 0–100. Largely automated from Form 990s + impact “beacons”. Weakness: historic overhead-ratio reliance is a poor, gameable impact proxy.
Candid / GuideStar — nonprofit profiles. Bronze–Platinum transparency seals. Self-reported data. Weakness: seals measure disclosure, not effectiveness.
Animal Charity Evaluators, Founders Pledge — impact-focused evaluation in harder-to-measure causes. ImpactMatters (cost-per-impact) was folded into Charity Navigator (2020).
Also: CharityWatch (A+–F), BBB Wise Giving / Give.org (pass/fail accreditation), Giving What We Can (meta-evaluation of evaluators).

Forecasting & prediction platforms

Metaculus — many event types. Community-prediction probability; forecasters scored by proper rules. Crowd aggregation, no betting. Weakness: aggregate accuracy largely self-reported; money enters only through prize tournaments (e.g. the AI forecasting benchmark series).
Good Judgment / GJ Open — geopolitics/economics. Probabilities scored by Brier; curated Superforecasters. Weakness: premium forecasts paywalled; small expert panel.
Polymarket — real-world events. Market price = probability; real-money crypto. Weakness: left the US after a 2022 CFTC settlement and returned via its 2025 purchase of the CFTC-licensed exchange QCEX; thin-market manipulation.
Kalshi — US event contracts. Binary $0–$1; CFTC-regulated exchange. Weakness: much volume is sports, not forecasting.
Manifold — user-created markets. Play-money market maker. Weakness: play money weakens incentives; creator-resolved miscalibration.
PredictIt — US politics. Real-money, academic project. Weakness: position/withdrawal caps distort prices.

AI models & public information

Arena (Chatbot Arena, then LMArena; renamed January 2026) — LLMs. Bradley–Terry/Elo leaderboard from crowd pairwise votes. Venture-funded; since September 2025 it also sells evaluations to model labs and enterprises. Weakness: labs can test private variants and publish the best (Singh et al. 2025 count 27 private variants from Meta before Llama 4). In April 2025 Meta’s leaderboard entry was a non-public, chat-tuned variant; the operator apologised and changed its policy (TechCrunch).
X Community Notes (formerly Birdwatch) — context notes on posts. A matrix-factorisation “bridging” algorithm favours notes rated helpful across groups of raters who usually disagree, so one faction cannot carry a note alone (Wojcik et al. 2022). Meta replaced US third-party fact-checking with the same model in 2025 (Meta). Since mid-2025 LLM “AI writers” may propose notes, which human raters still score; in a three-month field test one team’s LLM notes were rated more helpful than human notes on the same posts (Li & Bakker 2026).
The Unjournal — economics, policy and social-science research. Commissions paid, public expert evaluations with ratings. Grant-funded nonprofit; charges authors nothing.
Also (added after the coding): Epoch AI’s FrontierMath benchmark was funded by OpenAI, which owns its 300 core problems and has their solutions apart from a 50-problem holdout. The funding was disclosed only around the December 2024 announcement of o3’s score on it, and Epoch conceded it had not communicated the relationship clearly. With Arena’s sales to model labs, it brings the rated-party-pays pattern into AI evaluation.

Does who pays predict failure?

This page and Patterns §3 used to say that funding structure predicts trustworthiness better than method does. In October 2026 the catalogue was coded to test that. The data and script are in the repository (data/systems.csv, data/analyze.py).

Coding. Each named system is one row (161; “Also” mentions excluded). Each row records:

the primary payer: the evaluated party, users, a third party (advertisers, affiliate retailers, insurers, donors), the public, or mixed;
whether the evaluated party pays the evaluator anything material, such as fees, commissions, advertising, or logo or award licences;
the method family;
any documented major failure. A failure counts as established when a regulator, court, official audit, peer-reviewed study, the operator itself or a named investigation found the outputs materially distorted. It counts as alleged when the system has only been criticised. Fakes that a platform merely removed do not count, and neither do single disputed ratings.

The result was 42 established failures, each sourced in the CSV, 19 alleged, and 100 with none found.

Results. 154 systems have a known payer, and 41 of them have an established failure.

The payer typology predicts nothing detectable. Failure rates run from 19% (user-paid) to 34% (paid by the evaluated party), well within noise (permutation test p = 0.72). In leave-one-out prediction, knowing the payer category does slightly worse than ignoring it.
Method does little better. Among the larger method families, testing and inspection have the lowest failure rate (12%) and crowd systems the highest (39%), but p = 0.23 and the predictive gain is nil. Across bootstrap resamples of catalogue entries, payer came out ahead of method in 27%: no clear winner either way.
One contrast does hold up: does the rated party pay at all? Where it does, 27 of 67 systems (40%) have an established failure; where it does not, 10 of 62 (16%). That is an odds ratio of 3.5 (90% CI 1.6–7.8, Fisher p = 0.003). The odds ratio stays between 2.6 and 4.9 within method families, after dropping low-confidence codings, with each catalogue bullet counted once, and whichever way the 25 unknowns are assigned.
Those failures are more often gaming than capture. Of the 27, 6 are capture in the strict sense, meaning a lenient rating of the payer: the three agencies’ 2008 credit ratings, the BBB, the ACICS accreditor and the FDA’s Aduhelm approval. Eleven are manipulation by the rated party or outsiders (fake reviews, falsified rankings data, leaked tests); the other ten concern validity or data quality. Capture and evaluator misconduct together are rare (7 of 154), too rare to test on their own.

Caveats. Treat this as weak evidence.

Selection. A failure is counted only if someone documented it. Firms paid by the parties they rate attract regulators, litigants and journalists, so “none found” means “not found in a targeted search”.
Coder bias. One coder, not blind: payer and failure were coded from the same reading.
Multiple tests. Three codings were tested and none was pre-registered; correcting for three tests puts the binary contrast at p ≈ 0.01.
A broad variable. “Pays anything” lumps issuer fees with advertising, marketplace commissions and products sold to the rated party (the three credit bureaus qualify because consumers buy monitoring from them). The contrast is not concentrated where certification theory puts it: where the rated party is the main payer, 12 of 35 systems have an established failure (34%); where it pays only on the side, 15 of 32 (47%; Fisher p = 0.33). This split was examined after the main results.
Stakes as a confounder. A rated party pays for a rating when the rating is worth a lot to it, and high stakes are what drive gaming (Patterns §1). Because gaming outnumbers leniency almost two to one in the paying group, the data fit “payment marks high-stakes ratings” at least as well as “payment buys the evaluator”.

The claim as stated is not supported. What survives is narrower: when the rated party pays the evaluator, the odds of a documented failure are about 3.5 times higher, and the mechanism is unresolved.

Lessons for evaluation engineering

The literature behind each lesson is on Patterns & Failure Modes.

Whether the rated party pays matters; the finer funding story did not survive coding. See above: higher odds of documented failure, mostly through gaming; the leniency that certification economics predicts accounts for only 6 of the 27 failures.
Every output format is gameable, differently. Binary fresh/rotten invites bombing; 100-point scales inflate and compress; reciprocal two-sided ratings breed retaliation and inflation; self-declared certifications get faked. Choosing the output format is choosing your failure mode.
Crowd systems converge on the same arms race (fakes, astroturfing, review bombing) and the same defences: verified purchase or stay, credibility and recency weighting, and ML fraud detection.
Composite indices live or die on weighting, which is inherently contested (HDI’s three dimensions, ESG divergence, ranking methodology churn). The OECD composite-indicators handbook exists precisely because this is hard.
Cheap, standardized ratings can change behaviour at scale. Los Angeles’s hygiene grade cards cut foodborne-illness hospitalizations (Patterns §3), though New York’s inspections proved inconsistent and crash ratings invite testing to the test. That is the systems view’s trade of accuracy for unit cost, made many times already.

Meta-lists & further reading

Curated catalogues of evaluation systems (the “good lists” that already exist):

Wikipedia — List of international rankings — the best single index of country rankings by domain.
Wikipedia categories — Review websites, International rankings, Credit rating agencies, Certification marks.
Wikipedia overviews — Review aggregator, Reputation system, List of academic databases and search engines, Sustainability standards and certification, List of freedom indices.
Ecolabel Index — a directory of ~450+ ecolabels across ~200 countries.
Academic — Davis, Kingsbury & Merry, Governance by Indicators (Oxford, 2012) — the scholarly catalogue + critique of global indicators; Jøsang et al., A survey of trust and reputation systems (2007); Tadelis, Reputation and Feedback Systems in Online Platform Markets (2016).

See also Adjacent Fields & Literature for the academic disciplines behind these systems, and Related Work for QURI’s own evaluation tools.