Title: Evaluating Probability Forecasts
Speaker: Shulamith Gross, Baruch College of The City University of New York
Date: Monday, April 30, 2012 11:00am - 12:00pm
Location: DIMACS Center, CoRE Bldg, Room 431, Rutgers University, Busch Campus, Piscataway, NJ
Abstract:
This work was done in collaboration with Tze Leung Lai of Stanford University, and Catherine Huber of Paris V.
Probability forecasts of events are routinely used in climate predictions, in forecasting default probabilities on bank loans, or in estimating the probability of a patient's positive response to treatment. Scoring rules have long been used to assess the efficacy of the forecast probabilities after observing the occurrence, or non-occurrence, of the predicted events. We develop a statistical theory for scoring rules and propose an alternative approach to the evaluation of probability forecasts. This approach uses loss functions relating the predicted to the actual probabilities of the events, and applies martingale theory to exploit the temporal structure between the forecast and the subsequent occurrence or non-occurrence of the event.
In Epidemiology, a variety of indices, such as IDI (Integrated Discrimination Improvement), NRI (Net Reclassification Improvement), predictiveness curve, the area under the ROC curve and difference in PEV (Proportion Explained Variation), between models are routinely used, often without adequate inferential tools. We provide such tools for the IDI and the Brier Score difference when models are estimated and indices are computed on the same data.
DIMACS/CCICADA Interdisciplinary Series, Complete Spring Calendar 2012