« Return to Contents of this issue
Computer Simulation Validation. Fundamental Concepts, Methodological Frameworks, and Philosophical Perspectives
Beisbart, Claus & Saam, Nicole J. (Eds.)Springer-Verlag: Berlin, 2019
ISBN 9783319707655 (hb)
Order this book
Reviewed by
Martin Neumann
Johannes Gutenberg University Mainz
In 2009, a survey of agent-based modelling practices had been published in JASSS (Heat et al. 2009) which concluded that 65% of models in the survey had not or only incompletely been validated. Still validation remains an open issue and much debated topic in the ABM community. Therefore, the book “Computer simulation validation: fundamental concepts, methodological frameworks, and philosophical perspectives”, edited by Claus Beisbart and Nicole Saam is of high relevance for JASSS readers. On more than 1000 pages, the 43 chapters of this book provide a comprehensive overview of the multi-facetted field of simulation validation. The book is a must-to-have for libraries in departments in which simulation research is undertaken.
It needs to be emphasized that the book is highly interdisciplinary, not only addressing ABM or social sciences. It includes contributions from earth sciences, physics and engineering, mathematics, as well as methodological and philosophical discussions. Only few chapters specifically address social sciences. Likewise, the book does not attempt to provide a unique definition of validation. Rather, it highlights the ambiguity of the at first sight quite straightforward concept of validation. In the introduction, Beisbart and Saam emphasize that already in 1994 Oreskes et al. (1994) argued that even in the seemingly “hard” sciences the very term validation is misleading. Since then, their thesis has been controversially debated. Many different approaches are represented in this book. The aim of the book is bringing together these different viewpoints from different disciplines to enable mutual learning from each other and fostering an open mindset.
The book is organized in 9 sections: The first part consists of 4 chapter clarifying the very concept of validation. The second part of again 4 chapter takes a more philosophical stance and discusses issues such as falsification or Baysian epistemology and touches even upon a hermeneutic perspective on simulation validation. The next parts dig more deeply into different validation practices in different sciences and the various stages involved in the process of examining the credibility of a simulation model. The 5 chapters of Part 3 examine preparatory steps such as verification or choosing a validation matrix. In Part 4, 4 chapters look at different reference points for validation such as data, stylized facts, or the users’ judgement, whereas the 4 chapters of part 5 introduce mathematical techniques for validation such as Baysian methods or the quantification of uncertainty. Part 6 contains 3 chapters on the organization and management of validation procedures in large scale projects and Part 7 provides an overview of best practices in various disciplines ranging from physics via whether forecasting and climate science to economics. This part contains of 6 chapters. Finally, parts 8 and 9 widen the perspective by discussing challenges in 5 chapters of part 7 and providing 7 chapters with philosophical reflections in part 8.
Mostly, the individual chapters provide a handbook-like introduction into their topics and can be read also by non-specialists or researchers who want to familiarize themselves with a new field. In consequence the chapters vary in their degree of technical specialization. While some are rather sophisticated, others remain more on the surface but offer a lot of hints for further reading. In sum, the book provides reference material for working scientists or doctoral students in various fields, ranging from the natural to the social sciences or even philosophy. Indeed, the book demonstrates that philosophical expertise is needed also for working scientists by drawing attention to the philosophical underpinnings of the very term “validation”. The presumably most straightforward understanding of validation is claiming that a model to some degree sufficiently represents the real world. However, the very term “representation” is controversially debated in the philosophy of models (see Knuuttila 2011), as it implies a philosophical position that objective knowledge of an objective world can be attained. As highlighted in several chapters, this positivist position is predominant in the natural sciences but rather an exception in hermeneutical social sciences. For such reasons, some modellers perceive simulation models as an epistemic tool as it can actively be manipulated by the researcher rather than a representation of the world (Secchi forthcoming).
The fact that such philosophical exercise is of practical relevance can be illustrated by comparing two chapters of the book which are also of particular interest for JASSS readers: The chapter “The users’ judgments – the stakeholder approach to simulation validation” by Nicole Saam und the chapter “Challenges to simulation validation in the social sciences. A critical rationalist perspective” by Michael Mäs. Saam’s chapter is on participatory modelling, in which modellers develop a simulation model together with people who are affected by a problem that should be solved (or managed) by the model. This approach can often be found in environmental modelling. Often such models apply a constructivist approach. Saam classifies these models in the domain of action research. Objective of action research is to empower the research subjects. Therefore, the validation of a participatory model is its active use by the community that is modelled. On the other hand, Mäs discusses the challenges of validation at the example of social influence modelling, often known as opinion dynamics modelling. In this field the challenge for validation is the exact description of the real world because of measurement problems, the interplay of multiple processes in the field and other issues that make a comparison between simulation results and data problematic. In the two chapters validation has a completely different meaning. The importance of being aware of these subtle differences can be illustrated by the current attempts of simulating the Covid-19 crisis: Such models need to include various elements ranging from physical processes to behavioural aspects. For validating physical aspects, presumably a post-positivist framework is useful, whereas validating behavioural aspects might benefit from applying a constructivist framework. While the book demonstrates that much work still needs to be done, it is highly timely endeavour.
References
-
Heath, B., Hill, R., Ciarallo, F. (2009). A survey of agent-based modelling practices (Januar 1998 to July 2008). Journal of Artificial Societies and Social Simulation, 12(4), 9: https://www.jasss.org/12/4/9.html
Knuttilla, T. (2011). Modelling and representing. An artefactual approach to model-based representation. Studies in history and philosophy of science Part A, 42(2): 262-271.