Techniques

Status: draft, adapted from the 2021–22 estimation-theory notes. Each technique carries a status: tested (measured in a controlled or deployed setting), deployed elsewhere (in use, but not as proposed here), or untested.

The components are the parts; these are ways to wire them into a system. Most answer one question: how can a small amount of expensive, trusted judgment discipline a large amount of cheap judgment?

Prediction–evaluation systems

Status: tested.

Ask predictors to forecast a large set of items, say 10,000, and announce that a random subset, say 50, will be resolved by an expensive, trusted evaluation. Reward accuracy on that subset. Because no predictor knows which items will be graded, each must treat every item as if it might be. A small evaluation budget then calibrates forecasts across the whole set. What gets calibrated is agreement with the evaluator, errors included: predictors are paid to model the evaluator, not the truth.

The worked example runs this at a smaller scale: 1,000 grants, a hidden random 50 audited by two experts.

Fifty audits measure the model, but not precisely enough to judge it when experts agree as weakly as in peer review. The model scores all 1,000 grants; a random 50, hidden from everyone, go to two experts, and the model–expert κ is compared with the expert–expert κ on the same 50. Each κ carries about ±0.3 of sampling error (bars, simulated with both raters renewing 80%), so 50 audits cannot tell a model that guesses (κ 0) from one as good as a second expert (κ 0.17). About 300 audits, $180,000 per expert at $600 a review, mostly separate them.

The incentive thins as the set grows: with 50 of 10,000 resolved, each item has a 1-in-200 chance of counting. A proper scoring rule stays proper under random resolution, but the reward for effort on any one item falls by that factor.

The pattern can chain. Calibrated human predictions become training labels for cheaper ML predictors, which are audited against a fresh subset, and so on. Whether each step loses little accuracy has not been measured, and errors compound across steps.

The evidence runs in both directions. The idea was written up as Prediction-Augmented Evaluation Systems (2018) and tested in Amplifying generalist research via forecasting (2019). There, 19 forecasters predicting a trusted evaluator recovered 87% of the value of her direct evaluation at 120% of her cost; cheaper crowdworkers, at 52% of the cost, recovered negative value. The authors framed it as an existence proof, not a cost-effectiveness result.

The largest tests use replication as the expensive evaluator. Across four forecasting studies run alongside replication projects, prediction markets called 73% of 104 replication outcomes correctly, and surveys of the same participants did somewhat worse; accuracy has to be read against each project’s replication base rate (Liu et al. 2020, summarizing Dreber et al. 2015 and three later studies). DARPA’s SCORE program then had forecasters price about 3,000 social- and behavioural-science publications, of which about 250 would be replicated, using decision markets and surrogate scoring rules to keep incentives intact when most forecasts would never resolve (same source). When no item will ever resolve, peer-prediction mechanisms score each report against the others; the Bayesian truth serum rewards answers that are more common than respondents collectively predicted, which makes truth-telling optimal even for a minority view (Prelec 2004). The guarantee needs a large population of Bayesian respondents with a common prior, and truth-telling is an equilibrium, not the only one. The statistics for turning many cheap predictions plus a random audited subset into valid estimates is prediction-powered inference.

Open: whether the incentive survives gaming by the evaluated party rather than the predictors (LLMs as Evaluators, step 3), and what happens when participants are deceptive (see Cruxes and the sibling RRP wiki’s oversight protocols and untrustworthy sources).

Forecasting over structured ontologies

Status: deployed elsewhere, at benchmark scale.

Many questions worth forecasting are structured: “for each country, each month for 20 years, what will each of 20 metrics be?” Judgmental platforms, built around hand-written questions, handle this poorly. Automated generation from data sources now produces question sets in the thousands (ForecastBench), but continuous, high-dimensional forecasting at that scale remains a tooling problem that leans on the ontology component.

Estimation functions

Status: deployed elsewhere; value at scale untested.

A function that returns estimates over large parameter sets, usually with caching: the unit of reuse for the estimation layer. It needs tooling for uncertainty, composition and dependency tracking. Squiggle is one attempt, and Squiggle AI is a deployed LLM front-end that writes such models, with documented overconfidence. The Scorable Functions (2024) proposal should be read with the author’s later comment that he had “become much less enamored by this broad idea”, expecting people to ask LLMs for estimates on demand instead.

Their systems value is propagation: if estimates are composable functions of shared inputs, an updated input flows through to everything downstream instead of leaving stale reports.

Automated trust networks

Status: untested in this form; simpler trust networks are deployed.

Central evaluators are single points of capture: issuer-paid credit ratings before 2008, and the World Bank’s Doing Business index, discontinued in 2021 after data irregularities (Patterns §3). The proposed alternative is many agencies that evaluate the large ones and each other. A stronger version lets an agency publish declared corrections to another’s outputs: X accepts Y’s economic forecasts but shades Y’s steel-industry forecasts for overconfidence. “Whom do you trust” becomes inspectable structure rather than a binary. The simpler form, where trust propagates through a graph of ratings, has long been deployed: EigenTrust computes global trust in peer-to-peer networks from local ratings and resists simulated collusion among malicious peers (Kamvar, Schlosser & Garcia-Molina 2003). Declared corrections are what has not been tried. The sibling RRP wiki develops the identity layer this needs in identity and track records.

Robust aggregation

Status: deployed elsewhere.

How raw judgments become a score is never neutral (Patterns §5). Defaults with evidence behind them:

• Regularize sparse items: a Wilson lower bound for binary votes, Bayesian shrinkage for scores.
• Prefer pairwise comparison when raters are noisy; QURI’s relative value functions add uncertainty to the idea.
• For probability forecasts, average log-odds rather than probabilities, and consider extremizing the result: averaging probabilities pulls toward 0.5, and each forecaster’s confidence ignores evidence others hold (Baron et al. 2014).
• Pool with a median or trimmed mean, which one bad input cannot drag far; RRP develops this as breakdown-point ratings.
• Tie each rating to a verified purchase, stay or identity, the recurring fix for fake and brigaded ratings (Patterns §4).

Candidness, which the 2021–22 notes listed as a technique, is a precondition for deployment and has its own chapter. For the operational sequence of standing up a system, see the Launch Checklist.