The signal and the noise: why most predictions fail-- but some don't

This lively, informative, multivalent romp through the history of data accumulation and its uses, leading to today's refined ability to predict the outcome of a future event with considerable accuracy, appeared in the closing months of the 2012 presidential election campaign. During the campaign, statistician/blogger Silver (The New York Times FiveThirtyEight ), in articles and interviews, proffered increasingly confident opinions about its outcome, opinions at odds with those of most of the pundits, poll takers, and statisticians examining the same data. His vindication by the actual results of the election lends his book that much more interest. Though it does not trace new ground, its auspicious message is, in an era of bigger and bigger "Big Data," that it is possible, with objective judgment and sophisticated statistical tools, to make ever more reliable predictions on which the future of the species and the security of the country may depend. Silver pays particular homage to the well-known Bayes's theorem as fundamental to the proper application of statistical theory to actual situations. Individual chapters deal with the 2008 financial meltdown, earthquakes, baseball, chess, poker, the weather, climate change, and terrorism. Abundant endnotes and references. Summing Up: Highly recommended. Academic and general audiences, all levels. M. Schiff CUNY College of Staten Island

Despite the fact that there is more information about everything from finance to professional sports available than ever before, predictions "may be more prone to failure" in this "era of Big Data." Balancing technical detail and thoughtful analysis with fluid prose, statistician Silver (FiveThirtyEight ) picks apart the many ways in which predictions in various fields have been flawed, while suggesting approaches that could improve the practice. The catastrophic miscalculations on the part of financial lending agencies that led to the recession of 2008 arose for the same types of reasons that caused baseball scouts to undervalue Boston Red Sox all-star player Dustin Pedroia or feed into a political pundit's flawed forecast: overconfidence in models based on oversimplified principles and unrealistic initial assumptions. Though there is no simple solution, a Bayesian methodology, in which prior beliefs are taken into account and initial assumptions constantly revised, would lead to more accurate predictive models. Effective prediction requires, according to Silver, "the serenity to accept the things we cannot predict, the courage to predict the things we can, and the wisdom to know the difference." Agent: Sydelle Kramer, The Susan Rabiner Literary Agency. (Oct.) © Copyright PWxyz, LLC. All rights reserved.

An anointed wunderkind explains his own success as a prognosticator and explains why so many self-anointed "experts" are often wrong about winners in politics, sports and other realms. New York Times blogger Silver initially gained attention by developing a computer-based system meant to predict performances of Major League baseball players. Eventually, the author turned his talents to nonsports topics, including trying to figure out who would win the U.S. presidency during 2008. In 49 of 50 states, Silver correctly chose the presidential vote winner. In the 35 races for the U.S. Senate, he called every one accurately. In the 2012 election, he accurately called the presidential vote in all 50 states. Silver emphasizes that predictions are ultimately a human endeavor and that computers are programmed by humans. Meteorologists, for example, predict the weather incorrectly more than anybody would like. They have, however, used computer-based data analysis to improve accuracy. In the financial sphere, economists and other professional predictors failed to grasp the coming recession in 2008 despite sophisticated computer modeling. However, Silver writes, "nobody saw it coming" is an unacceptable excuse. The financial collapse was foreseeable with the proper underlying assumptions about economic behavior programmed into the computers. Too many underlying assumptions were misguided. Even more significant, 9/11 could have been predicted as well. Intelligence-agency analysts, however, could not grasp that religious zealots would plot their own deaths in order to kill Americans. No amount of computerized information can rectify a blind spot of that nature, Silver writes. Predicting the future performance of baseball players with well-documented pasts is more conducive to predictive accuracy than trying to understand previously anonymous fanatics. Some of the sections of the book are best understood by readers with mathematical reasoning skills, but the author is mostly accessible and enlightening.]] Copyright Kirkus Reviews, used with permission.

Despite the fact that there is more information about everything from finance to professional sports available than ever before, predictions "may be more prone to failure" in this "era of Big Data." Balancing technical detail and thoughtful analysis with fluid prose, statistician Silver (FiveThirtyEight ) picks apart the many ways in which predictions in various fields have been flawed, while suggesting approaches that could improve the practice. The catastrophic miscalculations on the part of financial lending agencies that led to the recession of 2008 arose for the same types of reasons that caused baseball scouts to undervalue Boston Red Sox all-star player Dustin Pedroia or feed into a political pundit's flawed forecast: overconfidence in models based on oversimplified principles and unrealistic initial assumptions. Though there is no simple solution, a Bayesian methodology, in which prior beliefs are taken into account and initial assumptions constantly revised, would lead to more accurate predictive models. Effective prediction requires, according to Silver, "the serenity to accept the things we cannot predict, the courage to predict the things we can, and the wisdom to know the difference." Agent: Sydelle Kramer, The Susan Rabiner Literary Agency. (Oct.)

Despite the fact that there is more information about everything from finance to professional sports available than ever before, predictions "may be more prone to failure" in this "era of Big Data." Balancing technical detail and thoughtful analysis with fluid prose, statistician Silver (FiveThirtyEight ) picks apart the many ways in which predictions in various fields have been flawed, while suggesting approaches that could improve the practice. The catastrophic miscalculations on the part of financial lending agencies that led to the recession of 2008 arose for the same types of reasons that caused baseball scouts to undervalue Boston Red Sox all-star player Dustin Pedroia or feed into a political pundit's flawed forecast: overconfidence in models based on oversimplified principles and unrealistic initial assumptions. Though there is no simple solution, a Bayesian methodology, in which prior beliefs are taken into account and initial assumptions constantly revised, would lead to more accurate predictive models. Effective prediction requires, according to Silver, "the serenity to accept the things we cannot predict, the courage to predict the things we can, and the wisdom to know the difference." Agent: Sydelle Kramer, The Susan Rabiner Literary Agency. (Oct.)