Workshop Details

We propose and test a method for out-of-population prediction termed model-assisted judgmental bootstrapping, which leverages a predictive model from one domain combined with expert judgment to generate training data and subsequently a predictive model for a new domain. In a preregistered experiment (=1440), we assessed the predictive accuracy of this method in increasingly challenging environments. We also analyzed the individual contributions of two techniques that underlie the method: model-assisted estimation and judgmental bootstrapping. Our findings revealed that both techniques significantly improved predictive accuracy. Furthermore, their impacts were complementary: model-assisted estimation provided the largest accuracy gains in the least demanding environment, while judgmental bootstrapping did so in the most challenging environment. Our results suggest that model-assisted judgmental bootstrapping is a promising technique for creating predictive models in domains in which outcome data are not available.

Bio: Daniel G. Goldstein is a Senior Principal Research Manager at Microsoft Research in New York City. Dan's research focuses on decision making in business, economics, and statistics, drawing on methods from computer science and cognitive psychology. Before joining Microsoft, he held research and professorial roles at the Max Planck Institute, Columbia University, London Business School, and Yahoo Research. Dan has authored award-winning research on forecasting and served as president of the Society for Judgment and Decision Making.

Prediction markets elicit and aggregate beliefs by paying agents based on how close their predictions are to a verifiable future outcome. However, many important questions involve outcomes that are difficult or impossible to verify. This includes questions about causal effects where running randomized trials is infeasible or unethical, as well as those posed over long time horizons where delayed outcomes distort agents’ incentives to report truthfully. We present a result showing that it is possible to run an ε−incentive compatible prediction market to elicit and efficiently aggregate information from a pool of agents without observing the outcome by paying agents the negative cross-entropy between their prediction and that of a carefully chosen reference agent. Our key insight is that a reference agent with access to more information can serve as a reasonable proxy for the ground truth. We use this insight to propose self-resolving prediction markets that terminate with some probability after every report and pay all but a few agents based on the final prediction. We show that it is an ε−Perfect Bayesian Equilibrium for all agents to report truthfully in our mechanism.

This talk is based on joint work with Siddarth Srinivasan and Ezra Karger. 

Bio: 
Yiling Chen is a Gordon McKay Professor of Computer Science at Harvard University. Her research sits at the intersection of computer science, economics, and other social sciences, with a focus on the social dimensions of computational systems. Her work has earned best paper awards at conferences, including ACM EC, AAMAS, ACM FAT* (now ACM FAccT), and ACM CSCW. She has co-chaired several major conferences, including WINE’13, EC’16, HCOMP’18, and AAAI-23, and served as an associate editor for multiple journals.

We introduce “full inference cycle forecasting”: a structured process for eliciting and communicating decision-relevant judgmental forecasts to policymakers. This process involves six phases: (1) identifying an expert population; (2) producing measurably informative forecasting questions; (3) eliciting forecasts from experts; (4) generating a menu of policies designed to affect a key outcome; (5) eliciting forecasts of the causal effects of those policies; and (6) communicating those forecasts to policymakers. We apply this process to the study of nuclear risk, surveying 110 domain experts and 41 expert forecasters to identify key policies that experts believe would, upon implementation, substantially reduce the likelihood of a nuclear catastrophe by 2045. This work identifies critical gaps between the theory and practice of forecasting, so we present new experimental evidence on how to elicit forecasts of unresolvable events, how to elicit forecasts in low-probability domains, and how to use a battery of cognitive tasks to accurately measure forecasting skill in a general population. We conclude with a discussion of open research questions that would help to improve the policy-relevance of forecasting.

Bio: Ezra Karger is a research economist in the microeconomics group at the Federal Reserve Bank of Chicago and the Research Director at the Forecasting Research Institute, where he works with academic and non-academic coauthors to develop and experimentally test methods for forecasting unresolvable questions, forecasting in low-probability domains, and forecasting causal policy effects. In his role as an economist, he also uses large datasets to construct high-frequency indices that track policy-relevant economic indicators.