Related Work

Status: working bibliography, links checked October 2026. This page maps QURI’s published work to the parts of the field it bears on. See the EA Forum QURI topic for the broader list and Adjacent Fields & Literature for the academic literatures.

QURI’s work is one strand of a wider literature. The closest outside work on the same core pattern, a cheap layer predicting an audited expensive evaluator, is larger than anything below: automated essay scoring’s validation standard, DARPA SCORE and Replication Markets (whose human forecasts beat chance on 37 audited replications while AI imitators did not), prediction-powered inference and selective escalation. These are summarized in Lineage. What QURI’s corpus adds is a small set of dated experiments and deployed tools with published failure admissions.

Foundational framing

Prediction-Augmented Evaluation Systems (Ozzie Gooen, LessWrong, 2018). The original “predict the evaluation” idea — the direct ancestor of prediction–evaluation systems. The wiki’s whole estimation/evaluation-bridging move is implicit here.
(Highly Optimized) Evaluations Are All You Need and the earlier Advanced / Symbolic Evaluation Systems drafts. The cause-area statement this wiki is built from. See Lineage.

Empirical results (the scarce, valuable part)

These are quotable, dated experiments — exactly the “concrete case studies” the field is short on (see Objections & FAQ).

Amplifying generalist research via forecasting, Part 1 (models/challenges) and Part 2 (results) (Gooen, Sempere, et al., 2019). The flagship test of prediction–evaluation, and one of very few real experiments in this space. Network-adjacent forecasters predicting a trusted evaluator recovered 87% of her value at 120% of her cost (benefit/cost 72%); online crowdworkers were cheaper (52% of cost) but had negative value. Framed by its authors as an existence proof, not a cost-effectiveness result.
An experiment to evaluate the value of one researcher’s work (EA Forum, 2019). Elicitation of value estimates over research outputs.
Predicting the value of small altruistic projects (Nuño Sempere, EA Forum, 2020). Ten forecasters predicted the upvotes (a weak proxy for value) of 20 small projects before they were done; the forecasts separated more- from less-upvoted projects but were too optimistic. The author calls the experiment underpowered.
Relative-value elicitation experiments (Sempere: valuing research works, March 2022; Open Phil’s 2018 AI-safety grants, September 2022). Six researchers took part in each. In the first, estimates of the same work differed by orders of magnitude and individual raters were internally inconsistent. Real data on inter-rater disagreement and how it aggregates.

Estimation & calculation tooling

Squiggle (squiggle-language.com; GitHub). A small language for probabilistic estimation — the working instance of estimation functions.
Squiggle AI (QURI, October 2024; EA Forum release post, January 2025). An LLM front-end that generates Squiggle models. A deployed estimation system: 30 outside users had run 168 workflows at $0.10–0.35 each by the release post, which reports that outputs are “often highly overconfident”.
Scorable Functions (LessWrong, May 2024). The estimator-as-program object. In a later comment the author wrote that he had “become much less enamored by this broad idea”, expecting people to ask LLMs for forecasts when they need them.
Guesstimate (2016). The early spreadsheet-style tool that motivated much of this; see Use Cases.

Ontology & aggregation

Metaforecast (metaforecast.org). Aggregates and searches forecasts across platforms — infrastructure for the ontology layer.
Foretold.io (EA Forum, 2019). An open-source prediction registry; early structured-forecasting plumbing.

Evaluation methods & utility elicitation

Relative Value Functions: A Flexible New Format for Value Estimation (EA Forum, 2023), plus the Utility Function Extractor and comparison-polling tools. The closest thing to a methods stack behind the evaluation-methods page’s open “elicitation” questions.
RoastMyPost (QURI, December 2025). A deployed LLM-plus-code tool that checks posts and research documents for errors, fallacies, and inaccuracies with several evaluator types. Its authors report that “the false positive rate for error detection is significant”. The sibling wiki’s case study covers it with Squiggle AI and an LLM-judged Fermi competition.
Shallow evaluations of longtermist organizations (Sempere, EA Forum, 2021). A real, scaled-down charity-evaluation effort, qualitative rather than numeric; the kind of “shallow but useful” output skeptics have found valuable.
Quantifying Uncertainty in GiveWell’s GiveDirectly Cost-Effectiveness Analysis (Sam Nolan, EA Forum, 2022). Putting distributions on a real CEA — estimation in the charity-evaluation domain.

Incentives, trust & failure modes

Incentive Problems / Alignment Problems with Current Forecasting Platforms (Sempere, 2020 blog post; Sempere & Lawsen, arXiv 2021). The concrete catalogue of reward-specification failures — directly relevant to whether prediction–evaluation incentives survive gaming.
Prediction Markets in the Corporate Setting (Sempere, Yagudin & Lifland, EA Forum, 2021). An honest negative result on why organizations reject internal markets (tooling, question-writing cost, social disruption) — feeds Epistemic Culture and Objections.
Opinion Fuzzing (QURI, December 2025). A proposal, not a measurement: sample an LLM judgment across prompts, models and personas, and report the spread as its uncertainty. It reports no measurements of its own; for evidence that phrasing matters, see Sclar et al. 2024. Bears on evaluation reliability.
Accuracy Agreements (Gooen, EA Forum, 2023). Contracts that pay a forecaster according to a proper scoring rule — a trust-network-adjacent incentive design. A proposal.

Resolution & oversight

Can We Place Trust in Post-AGI Forecasting Evaluations? (LessWrong, 2019) → AI for Resolving Forecasting Questions, which introduces Epistemic Selection Protocols (LessWrong, 2025). The deferred-resolution thread: how to ground evaluations when the resolver is itself an AI. Overlaps heavily with the sibling RRP wiki’s What Grounds an Oversight Protocol?.

A note on sourcing. Specific figures above are quoted from QURI’s published posts; check them against the linked originals before relying on them. The amplification figures were re-checked against the Part 2 results table in September 2026 — an earlier version of this wiki misreported them as “~73%, far cheaper”. Links, dates and quotes for the other entries were checked in October 2026. This list is not exhaustive — additions welcome.