Before I started to read Nate Silver's book, I had formed expectations as to its style and content. I expected it to be a "gung-ho" manifesto for statistical methods, big data and prediction. However I was pleasantly surprised, this is a cautious, self-depreciating and wise book. Of course, the book is shaped by the author's expertise, which is prediction, but it does take quite a wide approach to this topic covering much that will be of interest to social simulators.
The main point that this book makes (from our point of view) is that social prediction is possible. The author correctly predicted the Obama election results in all 50 states weeks before the election (who would probably win each state) and, what is more important, has established an impressive track record at this kind of prediction. However, what is much more interesting is what he says about doing prediction. I summarise these points and briefly discuss their relevance for social simulation.
- He is talking only about real prediction - predicting what will happen before it does. He rules out any equivocation in this regard, fitting out-of-sample data or claiming that one had predicted something after the fact do not count. The distinction between models for prediction and models for exploring explanation is explicit here - I wish more simulators were as clear.
- He emphasises that all such predictions should be probabilistic. What he produces is a spread of possibilities each of which is given a probability of occurring. He, rightly, rails against those who do not admit the uncertainty in their predictions - indeed he suggests the principle that prediction confidence is anti-correlated to its success. He documents cases where not declaring uncertainty has entailed considerable cost, even lives. Many social simulation papers do not come clean about the levels of variance in their outcomes if they happen to fit data well.
- He talks about how difficult prediction is to do. In particular that there is no "silver bullet" to achieve good prediction, but it is the result of a lot of hard work, adding detail, and building models. There is a lot of trial and error involved, looking to see which added details improve the prediction. He hypothesises that one needs to think like a "fox" rather than like a "hedgehog", by which he means knowing a lot about many different things rather than having one big idea. He thus eschews grand theory preferring a bricolage of more mundane but more reliable bits of knowledge. He is thus agnostic on the simple-complex modelling debate, adding complexity if this is justified by evidence and discarding parts if they do not help predictive accuracy.
- As a consequence, Nate puts a lot of emphasis on choosing a problem where prediction is feasible. He looks at the case of macro-economics and concludes that prediction is infeasible there (in case you are still in any doubt he comprehensively demolishes any claim they might have in this regard). The criteria for the feasibility of prediction include having enough quality data and having a good reason that one's target is actually predictable. Economics fails in these regards. He does not think many things are very predictable, and thus advises that one should be very careful in choosing the field in which you are attempting it. Most papers in JASSS are explanatory in purpose rather than predictive, and so are less restricted in this regard (explanation being easier than prediction). However, more care in evaluating whether a simulation model provides any added value (as discussed in Gilbert and Troitsch 2005) would not go amiss .
- The most difficult part of prediction is the honesty it requires of oneself. He describes many examples of how everybody thinks that they know more than they actually do, including himself in this description. It is only by, repeatedly, making predictions before the answer is known, not making excuses when wrong and then re-evaluating the model that progress is made. However carefully one makes a model, even when basing it scrupulously on evidence, one can not rely on it until it has established a track-record of predictions. On the other hand one should expect to be wrong a percentage of the time - a percentage that is consistent with your estimations of your own accuracy.
- Mind you, he also cautions against the opposite extreme - claiming that things are 100% unpredictable. The archetypal cases here are earthquakes and terrorist incidents. Almost no progress has been made at predicting when significant earthquakes will occur, but this does not mean absolutely no prediction is possible. For example, one can confidently predict that there will be more large earthquakes in Japan than in the UK. Similarly terrorist incidents follow a power-law distribution and so one should not be surprised when large incidents occasionally occur. Prediction is a matter, according to this book, of progressively (and sometimes competitively) improving our predictions, adding knowledge in order to reduce uncertainty. In this regard he takes a thoroughly Bayesian attitude, seeing progress as a matter of iteratively improving the estimation of probabilities even if one starts from raw guesses.
- He does discuss the difficulty of missing factors one has not even imagined, stressing how often we are constrained by our experience of what has happened before - the "unknown unknowns" popularised by Rumsfeld[1]. Here he is sympathetic to those whose job it is to predict in such situations of uncertainty, noting that what might be obvious after the fact is often not obvious before it. Over the history of social simulation new factors to be included in models have repeatedly arisen, factors that were not seriously considered preciously. The myriad factors and structures we presently do not even consider for our simulations may turn out to be crucial - the trouble is we do not know which.
- I have often propounded the principle that "scientists should not ignore evidence without a very, VERY good reason" and it would seem that this author agrees with this principle. He values both qualitative and quantitative evidence, using them both in his predictive models. This does not mean any particular bit remains there, of course. If they do not turn out to help then they are dropped[2]. Many times in the book he emphasises the importance of basing predictive models upon good understanding - he does not believe in generic techniques but sees statistical models (made well) as vehicles for harnessing such understanding.
He briefly touches upon some agent-based models and is broadly sympathetic to the approach, seeing how expressive they are and the range of evidence they can address. However he values them for the increased understanding they can bring and not as predictive vehicles. He rightly points out their brittleness and the amount of data that they need, makes then unsuited to prediction.
My only difference in emphasis I would make is to switch attention from the accuracy of predictions for a single target towards finding out the scope of model applicability - the conditions under which a model is reliable. However this is understandable as he does expend a lot of thought as to choosing his targets (and then wishes to simply compete on that target). Achieving generality in his predictive models is not his aim, indeed I suspect he would not claim any beyond each specific target.
All in all this is a revealing and highly readable book. A lot of it will be familiar to those who know Bayesian statistics, but a lot of the examples and lessons he draws from these are highly instructive. If you are even thinking of going into the business of predicting social phenomena, this is a must-read, and even if you are not this suggests lots of useful lessons.
Notes
References
Return to Contents
of this issue