by Paul Stephenson
The success of ESPN, sports periodicals and local sports radio stations has dramatically increased the number of “experts” that present their predictions for games, tournaments and seasons. Further, the fact that these proclaimed experts present their speculations on television, radio and the internet legitimizes their opinions. But to what degree has the accuracy of these experts been evaluated and what metrics are appropriate to evaluate their accuracy. As might be expected, the most straightforward metric for quantifying accuracy is the percent of predictions that are actually correct. However, some of these correct predictions could have been expected due to chance error. The kappa statistic is a chance-corrected metric for agreement, and this statistic can be used to quantify the accuracy of the predictions. In some cases, the predictions are numeric data, and the correlation between the predictions and the outcomes is a reasonable metric for quantifying the accuracy of a set of predictions.In this presentation the authors will discuss these metrics for quantifying the accuracy of predictions. In addition, we will demonstrate the application of these metrics on a variety of sports, and we will use these metrics to compare the accuracy of statistical experts and umpires.