Juliette Rouchier, Claudio Cioffi-Revilla, J. Gary Polhill and Keiki Takadama (2008)

Progress in Model-To-Model Analysis

Journal of Artificial Societies and Social Simulation vol. 11, no. 2 8
<https://www.jasss.org/11/2/8.html>

For information about citing this article, click here

Received: 03-Aug-2007    Accepted: 03-Aug-2007    Published: 31-Mar-2008

Abstract

Keywords:

Social Simulation, Agent-Based Modelling, Comparative Computational Methodology, Validation, Replication

1.1

The model-to-model series of workshops was set up with a view to gathering work on comparative analysis of social simulations. The first workshop was held in Marseille, March 2003, to counter a perceived dearth of comparison and transfer of knowledge among a burgeoning number of models in the area (Hales, Rouchier and Edmonds 2003). Since then, a second workshop was held alongside ESSA 2004 in Valladolid, the forum of JASSS has been dedicated to model comparison work, and now, after a relatively long interval, a third workshop was held in Marseille, March 2007.

1.2

Comparative analysis of social simulations can draw on the rich and distinguished tradition of comparative social research (Bartolini 1993; Dion 2003; Przeworksi and Teune 1970; Saberwal 1987; Sartori 1991). Despite a growing interest in model-to-model analysis, there is arguably still not enough of it being done. It is not difficult to suggest reasons for this. Developing one's own model is much more fun than studying or developing syntheses of others' work. Model-to-model work also generally tends to be very time-consuming. Comparative modelling research can be broadly categorised into a number of areas, each of which has its own challenges:

• Comparison, replication, reimplementation, and docking (or alignment). Here, in various ways, new or existing social simulation models have their structure and behaviour compared, with a view to verifying, confirming, or criticising a reported result or conclusions from it. Replication, for example, is sometimes seen as an activity for students learning about social simulation, rather than something for innovative professors to trouble themselves with. Whilst it may quite reasonably be countered that replication is currently not going to attract much in the way of funds, the situation is somewhat strange, as there is a case for suggesting that replication is a more challenging activity than developing the original model. Various authors who have tried replicating the results of others (e.g. Bigbee, Cioffi-Revilla and Luke 2005; Axtell, Axelrod, Epstein and Cohen 1995), have had to contact the original authors, and have even then sometimes not been successful (Rouchier 2003). Indeed, there is anecdotal evidence of a case where the original authors, when contacted by a researcher struggling to replicate their result, had considerable difficulty in doing so themselves! (Anonymous pers. comm.) Such tales further serve to emphasise the point made by Edmonds and Hales (2003) at the first workshop, that simulation models and their results should not be trusted until they have been independently replicated — a common practice in empirical science. This is the recommendation followed in this issue by Merlone and colleagues.
• Validation. Simulation results can be sensitive to parameter settings of the corresponding model and to the algorithm used to model the agents' behaviour. In order to validate computational models and simulation results, both sensitivity analysis on the parameter setting and on agent modelling should ideally be conducted. Izquierdo and collaborators study sensitivity in their social dilemma model in this issue. For the agents in particular, it is desirable to compare simulation results from different approaches to modelling their behaviour. Such a direction is pursued in this issue by Takadama and colleagues.
• Taxonomy and classification. Taxonomy and classification are often known as "systematics" in other fields, such as biology. Here models are grouped into common classes. This is a potentially fruitful line of enquiry, as yet little explored in social simulation, particularly if certain classes of models can be shown to have specific expected results. Perhaps if more researchers would work with frameworks such as Kahn's (2007) (paper in preparation), more standard results might be obtained for model components, submodels, and classes of model, such as the TRAP ² class defined by Cioffi-Revilla and Gotts (2003) at the first Model-to-Model workshop. However, the systematics of complex models such as most social simulations (which are dynamic, depend on initial conditions, and usually have a large number of parameters) is difficult to achieve through intuitive reasoning alone. The tools to systematically establish equivalence among models are not yet sufficiently well understood and much research lies ahead.
• Multi-scale analysis, abstraction, and models of models. Models are compared at various spatial, organisational or temporal scales, sometimes using a simple model as an abstraction of a more complex one. Abstraction is important to the social sciences, particularly where different case studies can be abstracted to grow models and meta models that can be exploited to develop more general theories (Przeworksi and Teune 1970; Cioffi-Revilla 2002). Huet and Deffuant in this issue explore the use of mathematical models of an agent based model, with an approach they term 'double-modelling'.
• Cross-paradigm comparison. Social simulation models are compared with models developed in alternate paradigms, e.g. equilibrium models, or social theoretical models. The comparison to mathematical results is often developed to relate to literatures such as economics and game theory, where the development of formal models is widespread. However, this strategy may not be fruitful in general if the researcher has to make a choice between building a very simple agent model that can be compared to a formal analysis but contributes little understanding to empirically observed social phenomena, and a more applicable agent-based model that includes a lot of heterogeneity and learning but is far from tractable analytically. Where such limitations do not apply, or the horns of the dilemma are not too large, such work can nevertheless prove interesting, as illustrated in this issue by Izquierdo and colleagues, and Vilà.
• Reuse. Models or components of models are reused within other models. Outside standard simulation libraries, such as Swarm, RePast or MASON, very little of this is done. This is despite the reusability of code being one of the main advantages of object-oriented programming (OOP). At the M2M workshop, however, Kahn demonstrated an architecture that might encourage greater reuse and sharing of code.
• Standards, protocols, and methodologies. Although it is perhaps too early in the field to discuss standards, developing protocols and methodologies for working with social simulations makes sharing, understanding, replicating, and reusing each others' work easier. Some hints about ways forward are proposed in this issue by Janssen and co-authors, and Polhill and colleagues.

1.3

The third Model-to-Model workshop covered some of the latest developments in these areas. Fifteen papers were presented from twenty-seven peer-reviewed submissions, and there was one invited presentation. Of the sixteen papers in the workshop, eight reflecting the breadth of discussions have been selected for inclusion in this special issue. We introduce them below in no particular order. The full proceedings of the workshop are available at http://m2m2007.macaulay.ac.uk/.

1.4

Izquierdo, Izquierdo, and Gotts replicate Macy and Flache's (2002) work with 2×2 social dilemma games, using mathematical analysis to understand the sensitivities of the model to different learning rates and the introduction of stochasticity.

1.5

Vilà develops an analytical model of Bertrand competition alongside a simulation, comparing the effects of using the simulation to relax the restrictive assumptions of the mathematical analysis on the results obtained. In a challenge to what is sometimes claimed by practitioners in social simulation, Vilà finds that relaxing the assumptions in the simulation model does not change the conclusions from the mathematical analysis.

1.6

Huet and Deffuant study the primacy effect (the effect of the order in which information is presented on perceptions), at the individual and population levels. They also complement their simulation work with mathematical analysis and conclude that the latter assisted them with understanding their model.

1.7

Polhill, Parker, Brown, and Grimm discuss the application of a proposed document structure for describing individual based models in ecology (Grimm et al. 2006) to three agent-based models of land use change. They find Grimm's protocol, which was intended to be applicable to social as well as ecological models, is indeed useful for structuring social simulation model descriptions in journal articles. However, some refinements are needed to capture all the agent-based simulation work in the social sciences.

1.8

Merlone, Sonnessa, and Terna, taking Edmonds and Hales' (2003) recommendations to heart, use three separate implementations in radically different simulation architectures to study population changes in industrial districts. They find differences in the floating point and pseudo-random number environments of each architecture prevent exact replication of their results, but are able to generate qualitatively similar results, that, through the multiple implementation strategy, are arguably more trustworthy.

1.9

Takadama, Kawai, and Koyama apply validation at both the micro and macro level to model agents who can reproduce not only human-like behaviours externally but also human-like thinking internally. Such agent modelling is investigated on reinforcement learning in an agent-based simulation of a sequential bargaining game. Their validation is based on experiments with human subjects, and they find that a certain configuration of the reinforcement learning algorithm is able, within the context of the game, to reproduce both the observed human behaviour and thinking.

1.10

Janssen, Alessa, Barton, Bergin, and Lee report on the establishment of the Open Agent-Based Modelling Consortium, a community forum in which to develop best practice, protocols, and standards. This is a significant initiative, that should be supported along with others like it.

1.11

As in earlier workshops, the timetable was scheduled to allow plenty of time for presentation and discussion of each paper. We, and all who attended, believe the event was a success, and express the hope that in the future, model-to-model workshops will be held more regularly than in the past.

Acknowledgements

The programme committee thanks the Centre National de la Recherche Scientifique (CNRS), Groupement de Recherche en Econome Quantitative d'Aix Marseille (GREQAM) and the European Social Simulation Association (ESSA) for their financial and organisational support for the workshop.

References

AXTELL R, Axelrod R, Epstein J, and Cohen M D (1995). Aligning simulation models: A case study and results. Computational and Mathematical Organization Theory 1 (1), 123-141.

BARTOLINI S (1993). On time and comparative research. Journal of Theoretical Politics 5 (2):131-67.

BIGBEE A, Cioffi-Revilla C and Luke S (2005). Replication of Sugarscape using MASON. In Troitzsch K G (Ed.) Representing Social Reality: Pre-proceedings of the Third Conference of the European Social Simulation Association, Koblenz, September 5-9, 2005. Koblenz: Verlag Dietmar Fölbach. pp. 6-15.

CIOFFI-REVILLA C (2002). Invariance and universality in social agent-based simulations. Proceedings of the National Academy of Science of the U.S.A. 99 (Supp. 3) (14):7314-7316.

CIOFFI-REVILLA C and Gotts N (2003). Comparative analysis of agent-based social simulations: GeoSim and FEARLUS models. Journal of Artificial Societies and Social Simulation 6 (4), 10. https://www.jasss.org/6/4/10.html

DION D (2003). Evidence and Inference in the Comparative Case Study. In Necessary Conditions: Theory, Methodology, and Applications, edited by G. Goertz and H. Starr. Lanham: Rowman & Littlefield Publishers, Inc.

EDMONDS B and Hales D (2003) Replication, replication and replication: Some hard lessons from model alignment. Journal of Artificial Societies and Social Simulation 6 (4), 11. https://www.jasss.org/6/4/11.html

GRIMM V, Berger U, Bastiansen F, Eliassen S, Ginot V, Giske J, Goss-Custard J, Grand T, Heinz S K, Huse G, Huth A, Jepsen J U, Jørgensen C, Mooij W M, Müller B, Pe'er G, Piou C, Railsback S F, Robbins A M, Robbins M M, Rossmanith E, Rüger N, Strand E, Souissi S, Stillman R A, Vabø R, Visser U and DeAngelis D L (2006). A standard protocol for describing individual-based and agent-based models. Ecological Modelling 198 (1-2), 115-126.

HALES D, Rouchier J and Edmonds B (2003) Model-to-model analysis. Journal of Artificial Societies and Social Simulation 6 (4), 5. https://www.jasss.org/6/4/5.html

KAHN K (2007) Comparing multi-agent models composed from micro-behaviours. In Rouchier, J., Cioffi-Revilla, C., Polhill G. and Takadama, K. (eds.) M2M 2007: Third International Model-to-Model Workshop, Marseille, France, 15-16 March 2007, pp. 165-177. http://m2m2007.macaulay.ac.uk/M2M2007-Kahn.pdf

MACY M W and Flache A (2002) Learning dynamics in social dilemmas. Proceedings of the National Academy of Science, USA 99 (3), 7229-7236.

PRZEWORSKI A and Teune H. (1970). The Logic of Comparative Social Inquiry. New York: Wiley.

ROUCHIER J (2003) Re-implementation of a multi-agent model aimed at sustaining experimental economic research: The case of simulations with emerging speculation. Journal of Artificial Societies and Social Simulation 6 (4), 7. https://www.jasss.org/6/4/7.html

SABERWAL S (1987). A comparative analysis of very long-term processes. International Political Science Review 8:307-318.

SARTORI G (1991). Comparing and Miscomparing. Journal of Theoretical Politics 3 (3):243-57.

Return to Contents of this issue

© Copyright Journal of Artificial Societies and Social Simulation, [2008]