Engineering Simulations as Scientific Instruments: A Pattern Language

Order this book

Reviewed by Bruce Edmonds
Manchester Metropolitan University

This book is a summary of the CoSMoS (Complex Systems Modelling and Simulation) approach, resulting from the project^[1] and meetings under the same name. It aims to describe and formalise the steps needed to develop simulations that are fit for purpose for the serious understanding of complex systems. That is, simulations we can rely upon – whose assumptions and purposes are open and clear, and for which the adequacy of the simulation (for its purpose) is supported by clear arguments based on evidence. The vision is very much that of simulations as a tool – we have engineered clocks to measure time; this is how to engineer simulations that we can rely upon to understand complex systems.

Although written as a general guide to developing simulations, this book has the style and goals of a software engineer. It thus aims for the greatest possible generality (being applicable to a wide range of simulation contexts) but also for formality (being as unambiguously specific about the concepts and processes involved). Generality and formality are difficult to achieve together, so there is inevitably some compromise involved. Here the authors have decided to utilise Design Patterns to describe their approach (Alexander 1977; Gamma & al. 1995). These are concise, semi-formalised descriptions that adhere to the same structure, but which are ultimately described in terms of other patterns and natural language. The book starts with a very high-level pattern: a “CoSMoS Simulation Project”, which is then defined in terms of four stages described in other patterns. These stages are then described in terms of other patterns etc. so that one gradually gets to increasingly specific patterns. Thus patterns are rather like procedures in simulation code, and this gives a computer-sciency style to the book, which, to those used to a more social-science, discursive, style can feel a bit alien. Like all code this is not easy to read (unless you have been involved in its development) and will take some effort to get into – it is probably best to try and work through these during the development of a simulation, so that all the detail makes more sense as you get to it. On the plus side, if you did follow this it would help you avoid many potential simulation traps and would result in a more reliable and better documented simulation.

These prescriptions do not result from mere “armchair” theorising but from extensive experience with designing, developing and checking simulations for such scientific purposes. Whilst it is true that these were in the robot engineering and biological domains, this book has the “ring of truth” about it – reflecting much that I have experienced as a simulation modeller of social phenomena. Thus, even though not specifically designed to guide social simulation, this book has much that we can usefully learn from. In particular, I would like to highlight some of the important distinctions that they make, which are important in terms of improving our simulation methodology.

• Firstly, they distinguish four different roles involved in a simulation project. These are (a) Domain Scientist – who is an expert in what is being modelled, (b) Domain Modeller – who formalises the relevant knowledge about the modelling domain, (c) Simulation Engineer – who implements the domain model into code, and checks it etc. and the (d) Argument Modeller – who takes a meta-vies of the model and its ultimate justification. In much social simulation these roles are conflated, so that (b), (c), and (d) are all done by the same person, and sometimes they do (a) as well! Also stage (b) and (c) tend to be merged together into a single task. Separating out these roles, so they are done by different people, is very useful as it provides a better check on the others and helps draw out assumptions concerning translation between kinds of representations. However, at least separating out the different stages and making their accompanying representations explicit is a good idea.
• Secondly, they put a lot more emphasis on the construction of a formal argument to justify the fitness for purpose of the simulation, managed by the Argument Modeller. This person: gets agreement on the assumptions behinds the model and documents these, gets agreement as to what the crucial claims about the simulation are (in terms of its fitness for purpose) and how the arguments to back these up will be formalised, and then formulates and checks these arguments leading to conclusions about the simulation claims. In much social simulation work this is done informally and often in a frankly slap-dash manner, so that the reader is not clear as to what is claimed, and how that is established. I suspect that there are two reasons why this is not done in social simulation: (1) that many authors are entirely unclear about their aims and how they justify these and (2) their arguments to support their claims would be revealed as extremely weak if they did this.
• Thirdly, they distinguish four different contexts involved (in addition to that of the domain itself), that is: (a) the Domain model – an explicit scientific description of the domain (b) the Platform model – a formal design for the simulation (c) the Simulation platform – which implements the platform model in terms of specific software and (d) the Results model – a description of the results of the simulation experiments in terms of the domain model. Again these are very useful conceptual distinctions and correspond to different kinds of activity. Often, in social simulation (a), (b) and (c) are conflated or poorly distinguished with (d) being omitted or only implied.
• Lastly, the authors stress the essential collaborative nature of developing a good simulation – there is a lot of discussion and agreement involved. Thus, for example, the Domain Modeller is responsible for the domain model, but this is in conjunction with a deep involvement in the domain (as a source), the platform model (conversations about how these will be formalised as a simulation design), and the results model (conversations about what the results mean in terms of the domain model and agreeing the argument for model fitness for purpose). The other roles have a similar spread of vision and the simulation results from the agreements between them. This constant discussion makes for a much more reliable simulation, forcing otherwise hidden steps to be made explicit and ensuring that independent minds are cross-checking everything.

I suggest that, even if you do not follow the full detail of this book, these more general lessons and distinctions would result in a much higher quality of simulation, where the epistemic contribution was much clearer. Doing this does require a lot more work, documentation and time but if we are serious about social simulation – that is, we want to go beyond playing with simulations as vague and unreliable analogies – this kind of care is essential.

The “Anti Patterns” are short descriptions of some of the things that often go wrong in simulation work, and are fun to read. I think I can say with certainty that I have been guilty of most of them at some stage or other. Thus, for a good laugh skip to pages 114, 137, 169 and 182. I will not spoil this by describing them all here, but merely mention a few. Analysis Paralysis is when one is spending too much time analysing and modelling the Domain trying to get everything perfect and never getting to the implementation^[2]. A Bespoke Too Far, when one invents a new modelling approach for each simulation. One Size Fits All, is where a particular approach is used regardless of research questions, scientific context or resource limitations. I also liked the use of the text “!! Future” to indicate intentions and characteristics that have not been done yet, but are left to future work – in some social simulation work I suspect many authors are not so clear about the difference between what they hope to be and what has presently been achieved (e.g. thinking of their simulation as predictive when this is far from being established).

The book does not merely present its approach in its raw, complete detail but gives a good overview of the main ideas, as well as giving a simple example to help explain it, and a very detailed example where the approach is applied. The core of this book – the full specification of the approach – is not meant to be read cover-to-cover as a narrative, but rather as a reference source to be consulted when appropriate.

To sum up, this book contains many good suggestions for tightening up our approach to simulation, distinguishing many things that are often conflated or confused, and is flexible in its approach (it does not suggest you have to do all of this, but should select what is appropriate in a reasoned manner). It has been developed with very different domains to that of social science in mind – in particular, in domains where most of the micro-details about the phenomena of interest are well understood, and the general level of domain-level uncertainty much lower. Thus, in parts it does assume one is working upwards from the micro-level towards emergent macro-level properties and has fewer prescriptions for some of the particular difficulties of modelling social phenomena. However, with a only a little adaption, this is highly relevant to social simulation and I heartily recommend taking to heart its key distinctions and approach.

Notes

1. See http://cosmos-research.org.

2. Not one of my failings ;-) but this has been true of some of my PhD students.

References

Alexander, C. (1977). A Pattern Language: Towns, Buildings, Construction. Oxford, NY: Oxford University Press.

Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design Patterns: Elements of Reusable Object-Oriented Software, Addison-Wesley. Reading, MA.

Return to Contents of this issue

© Copyright JASSS, 2020