Ana Maria Ramanath and Nigel Gilbert (2004)

The Design of Participatory Agent-Based Social Simulations

Journal of Artificial Societies and Social Simulation vol. 7, no. 4
<https://www.jasss.org/7/4/1.html>

To cite articles published in the Journal of Artificial Societies and Social Simulation, reference the above information and include paragraph numbers if necessary

Received: 13-Jul-2003    Accepted: 21-Jul-2004    Published: 31-Oct-2004

Abstract

It is becoming widely accepted that applied social simulation research is more effective if potential users and stakeholders are closely involved in model specification, design, testing and use, using the principles of participatory research. In this paper, a review of software engineering principles and accounts of the development of simulation models are used as the basis for recommendations about some useful techniques that can aid in the development of agent-based social simulation models in conjunction with users. The authors' experience with scenario analysis, joint analysis of design workshops, prototyping and user panels in a collaborative participatory project is described and, in combination with reviews from other participatory projects, is used to suggest how these techniques might be used in simulation-based research.

Keywords:

Participatory Simulation, Software Methodologies, Development Techniques

Introduction

1.1

While agent-based social simulation as an approach to the understanding of social processes and as a technique for creating tools to assist policy formation and evaluation has grown rapidly over the last decade, there has been insufficient effort as yet to formalize, systematize and communicate methods for developing useful social simulations (but see Gilbert and Troitzsch 1999; Suleiman, Troitzsch and Gilbert 2000; Barreteau and others 2003). This is especially true of research that aims to involve stakeholders in the development of simulation models, using a 'participatory' approach in which users are involved in the design of the models from the earliest stage (see Gilbert et al. 2002). A growing number of descriptions of such models are available, some of which include reflections on how the models were designed and used, but much of this work has proceeded by trial and error and the accumulated experience is somewhat fragmented and anecdotal.

1.2

This paper begins to augment such methodological experience by examining the literature on the development of complex software systems. There is a great deal more that can be learned from the literature in software engineering about how to manage such participatory processes and, in particular, about techniques such as rapid iterative development and user workshops that deserve to be more widely known within the social simulation community.

1.3

Drawing from methodological traditions in several research fields, several of these techniques have been applied to a current European project called Simweb (http://www.simdigital.com). The project uses agent-based simulation for exploring innovative e-business models in the digital contents sector. The effect of the growth of e-business on the publishing and music retailing sectors in Europe has been to open up new ways of doing business for publishers and retailers. On-line delivery of news and music products to consumers has become possible, but new questions about managing new classes of consumer, the appropriate legal and financial infrastructures, intellectual property rights and discouraging fraud have to be tackled. Firms entering this market have to create new business models, while the existing providers often find themselves facing competitors whose activities are as yet little understood.

1.4

Section 2 of this paper reviews a sample of participatory simulation studies to extract common methodological lessons that may be used as a starting point for future participatory simulation projects. Section 3 examines the similarities between techniques for building participatory simulations and other software and systems. In section 4, initial results from the Simweb project are used to illustrate these techniques and section 5 offers some recommendations.

Lessons from Agent-based Participatory Social Simulation Studies

2.1

The more traditional way of doing social simulation (that is, researcher-centred) gives stakeholders an essentially passive role. Stakeholders are those with a direct or indirect stake in the outcomes of research projects (see Hare and Pahl-Wostl 2002; Estrella 2000). Stakeholders may be researchers, management and decision-makers in organisations, policymakers (or their representatives), etc. Without a greater involvement of clients (or end-users) in particular, as Gilbert et al. (2002:2) argue: "...it is frequently the case that policy-makers dismiss academic research as too theoretical, unrelated to the actual problems they are wrestling with, or in other ways irrelevant to their concerns..."

2.2

User-centred social simulation allows stakeholders to rerun decisions made under differing scenarios and environmental and strategic conditions (see for example, Bonabeau and Meyer 2001; Thesen and Travis 1995; Hannon and Ruth 2001). This can provide support for decision making in a relatively cost effective way (Wickenberg and Davidsson 2002). The use of scenarios as a technique that enhances understanding of tasks and communication among stakeholders is also receiving increasing attention in the neighbouring field of systems/software engineering (see for example, Sampaio et al. 2000; Zhu and Jin 2000). For the purposes of this paper, user-centred simulation and participatory simulation can be considered synonymous. To achieve more end-user involvement, participatory simulation draws on ideas from two areas. First, it borrows techniques from approaches variously labelled interactive social science, participatory methods, integrated assessment, and action research (see Gilbert et al. 2002). Secondly, it makes use of games as a strategic technique to achieve requirement and knowledge elicitation from end-users, model testing by users and even end-user training (Colella et al. 1998).

2.3

A number of previous studies, including d'Aquino et al. (2002), Gilbert et al. 2002), Barreteau et al. (2001); Lynam et al. (2002), Bousquet el al. (2002) and Purnomo et al. (2002), that have adopted a participatory simulation approach have been reviewed to identify methodological lessons. Other literature that examines the methodology of participatory related simulations was also reviewed, including Parunak et al. (1998); Terna (1998); Parker and Swatman (1999); Asakawa and Gilbert (2002) and Hare et al. (2001). The main conclusion that can be drawn from these studies is that participatory approaches need to allow for iteration, collaboration and exploration of alternative scenarios. Table 1 brings together some of the comments made in these studies about critical aspects of the simulation development process.

Table 1: Advice from previous studies on practice in developing participatory simulations

Stakeholders

Participatory agent-based simulations are useful for the exploration of scenarios involving multiple stakeholders (see Parunak et al. 1998; Purnomo et al. 2002; Hare et al. 2001; Gilbert et al. 2002; d'Aquino et al. 2002; Lynam et al. 2002; Barreteau et al. 2001). Effective communication between stakeholders and putting decision-making into the hands of stakeholders are key for collaboration throughout all of the stages of participatory projects. However, stakeholders need to be aware of the tangible and intangible costs and benefits of cooperating (see Purnomo et al. 2002; d'Aquino 2002; Lynam et al. 2002) Lack of motivation can result in stakeholders losing interest or even dropping out of a project. A good structure, organisation and a certain degree of formality of activities contribute to maintaining stakeholder motivation throughout projects. As Barreteau et al. (2001:11) argue, during role-playing, for example, the focus needs to be on achieving a balance "between the extremes of a 'stage' play (if the sessions are too slow, or the players too neutral), and too much uncontrolled drift". Other factors favouring motivation reported in the literature (Asakawa and Gilbert, 2002; Gilbert et al. 2002; Hare et al. 2001; d'Aquino et al. 2002; Barreteau et al. 2001) include good time management, a desire to learn among stakeholders, and user-friendly software interfaces (where applicable). Achieving the 'right mix' of stakeholders for these projects is important. Stakeholder/player homogeneity needs to be taken into account too, as the mixing of superiors and subordinates may at times be problematic (Parker and Swatman 1999; Hare et al. 2001; Asakawa and Gilbert 2002; d'Aquino et al. 2002; Bousquet et al. 2002; Barreteau et al. 2001).

Design

There can be a number of sources of bias and data problems in the design process. Irvine et al. (1998), for example, list 29 possible sources of what the authors call "judgemental biases" grouped into three main categories: biases related to data (e.g. adjusting and anchoring; availability; conservatism; data saturation; spurious cues etc.); biases related to decision-makers (e.g. desire for self-fulfilling prophecies, expectations, fact value confusion; habit; hindsight, wishful thinking etc.); and decision-maker use of data (data presentation context, law of small numbers, representativeness etc.). In the design of Internet-mediated simulations (e.g. games) aspects such as: synchronicity (of play), support for technical or training problems, security (copyright, passwords, login names) and communication (sensitivity to cultural and language differences, sharing of sensitive data/information), are important for designers to consider. With regards to synchronicity, decisions about, for instance, how participants in different locations and time schedules will play with each other (e.g. over the Internet, or via video-conferencing, or person-to-person, etc.) need to be made early (see Asakawa and Gilbert 2002).

Tools

The use of high level, structured programming tools assists in publishing simulation results and aids replication. These high level tools (for a review, see Tobias and Hofman 2004), can help to make writing the software easier and less prone to error (Terna 1998). Using programming languages such as Java and running the resulting simulation software as applets on client machines may place great demands on these machines and can make maintenance of the code tricky. With recent Web-enabled languages (such as PHP) that make it easier to generate web pages dynamically, all the code remains on the server. This can also have advantages for inter-player communication during gaming simulations, for example, because all players' actions are stored on the server for later analysis (see Gilbert et al. 2002). Another tool that has been used for building multi-agent based applications is XML (Extensible Mark-up Language). XML is an open standard for the representation of information that can be processed by machines and read by people. It supports data representations in a way that is generic enough and yet sufficiently feature-rich to allow for a range of concepts to be modeled. It also allows information to be stored in structured files and ported across different platforms (see Erl 2004).

Evaluation

There can be significant differences between simulation results and reality. This can be due to software errors, but also to omission of key (social) aspects during the initial conceptualisation and modeling stages (see Purnomo et al. 2002; Edmonds and Hales 2003). The potential for undesirable as well as desirable results in multi-agent based simulations can be a side effect of emergent behaviour that arises as a result of interactions between the system components (see Bousquet et al. 2002; Kearney and Merlat 1999). This can pose a problem for the more engineering-oriented simulation researchers. As Edmonds and Hales (2003) point out: "If we are to be able to trust the simulations we use, we must independently replicate them. An unreplicated simulation is an untrustworthy simulation". Parker and Swatman (1999) suggest that simplified documentation which places emphasis on a step-by-step, rather than an explanatory approach, can reduce the time and costs associated with activities such as training, briefing/debriefing, role-playing, and dissemination of results; and, that use of email as a tool for communication and as substitute for certain types of detailed documents has been effective. However, those authors also conclude that, despite the benefits of using the Web, stakeholders continue to favour face-to-face communication, particularly for briefing and debriefing purposes.

2.4

The stages of an iterative research process (Conceptualisation, Design, Construction, and Evaluation) are carried out in most published simulation models. There is, however, another stage of the process that is less often performed although it also needs to be considered, that is, replication (see for example, Gilbert and Troitzsch 1999; Axelrod 1997). Replication means confirming that the claimed results of any given simulation are reliable, in the sense that they can be reproduced by anyone. However, replication that involves literal reconstruction of the same simulation using the same programming infrastructure has the disadvantage that unacknowledged assumptions built into the original version can be reproduced in the replication and are not therefore tested. A better technique may involve docking, the construction of a conceptually identical simulation using different tools and even different methodology (Hales, Rouchier and Edmonds 2003).

Bringing together Techniques from the fields of Social Simulation and Software Engineering

3.1

The steps involved in participatory simulation projects (that is, conceptualisation, design, construction and evaluation) are not dissimilar to the development stages of software (or systems) engineering projects. Some software engineering techniques which may be valuable in simulation research include: structured workshops with users; formal or informal stakeholder interviewing and prototyping.

3.2

However, although software engineering methods have evolved to increase stakeholder participation, some of the methods probably originated from the social sciences, where structured group processes in which participants play an active role and articulate their knowledge, values and preferences have been evolving for nearly a century (see for example, van Asselt et al. 2001; Estrella 2000; Vennix 1998; Chambers 2002). A review of simulation literature revealed some useful and detailed advice for exploration, involving stakeholders in situations where the target reality is messy or complex. As such, the software engineering community may benefit from new developments taking place in this area of social science research. The use of scenarios as a technique that enhances understanding of tasks and communication among stakeholders, for example, is receiving increasing attention in the software engineering field (see Sampaio et al. 2000; Zhu and Jin 2000).

3.3

From a review of software and simulation literature, four techniques were found to have value in helping to understand and refine user requirements and in developing a foundation for the simulation model(s) during the Simweb project. These techniques, namely, scenario analysis, joint application design workshops, prototyping and user panels are described next.

Scenario Analysis

3.4

Scenario analysis is an interactive process in which a group is engaged in a process of identifying key issues and creating and exploring scenarios, in order to learn about the external environment and to integrate insights into the decision-making of the organisation. The free-format approach enables the exchange and synthesis of ideas and encourages creative thinking (van Asselt et al. 2001). 'Scenarios' are connected statements that paint a broad but relatively shallow picture of possible futures (Gilbert et al. 1998). According to van Asselt et al. (2001), the first step in the process of scenario analysis is the identification of key issues or questions relevant to the organisation(s), or project and the time frame associated with the focal issue(s). This is followed by brainstorming exercises to expose ideas associated with the relevant issues. Also, the involvement of decision-makers in the scenario creation and exploration process may be limited to a single interview, but it can also involve participating in workshops that may run for several days at a time. These scenario workshops may involve the use of software (e.g. prototypes) as a facility for group support. As a follow up to the workshops (or interviews), there should be a period of reflection, writing up the scenarios and exploring their implications (van Asselt et al. 2001:15,16).

3.5

Foresight (Gilbert et al. 1998) is one type of strategic scenario exploration process that could be useful at the outset of a participatory project. Scenario analysis could also be used during the evaluation phase of participatory simulations in the form of round-table discussions with users in order to help them elucidate plausible, innovative alternative futures based on the results of many simulation runs.

Joint Application Design Workshops

3.6

JAD workshops (or sessions) are one type of user-oriented participatory workshop aimed at the initial phases of a software development project. Martin and Odell (1996) describe in considerable detail the various types of workshops that are possible (for example: 'Joint Enterprise Modeling', or JEM; 'Joint Requirements Planning', or JRP; etc.). The workshops vary in duration and scope depending upon the complexity of the requirements, the availability of key stakeholders, etc. (Martin 1991). The authors offer suggestions on how to organise them (location, equipment, participant selection, training), control them (conflict management, documentation), and moderate them (duration, dynamics).

3.7

JAD is very useful for the discussion of more technical issues aided by the use of prototypes and also as a side effect contributes to user computer literacy (Martin 1991). JAD sessions typically last three to five days depending upon the complexity and novelty of the requirements. Martin and Odell (1996) offer guidelines on how to set-up and conduct JAD sessions, including sample agendas and questions for such meetings. When issues come up during the workshops that cannot be resolved, they recommend that such problems are declared as "open issue(s)", and that a scribe (or note-taker) should note the issue number, issue name, person assigned to resolve the issue, date for resolution and description. JAD may therefore be a useful technique during model design in participatory simulations. The literature stresses the importance of a skilled facilitator for these sessions. Facilitators are expected to have excellent communication skills, be impartial, unbiased and neutral, be good at conducting meetings, have negotiating skills and be diplomatic (Martin and Odell 1996).

Prototyping

3.8

Prototyping is a firmly established technique in software engineering, although going under a variety of different names. For example, in the Extreme Programming methodology (or XP) it is referred to as a 'spike' (see Wells 2001). Also, methodologies employ prototyping to different degrees. In the Rational Unified Process methodology (or RUP) an 'executable architectural prototype' is considered an important output of the construction phase (see Jacobson et al. 1999). Prototyping can be employed in several phases of the software life cycle for building a quick and (if need be) rough version of a desired software product or parts of a system (see for example, Agresti 1986, Martin 1991). During design it can be used to develop screens, dialogs and reports to conform to user requirements (for discussion during workshops with end-users, for example); and during construction prototyping may speed up the iterative cycles of software enhancements (particularly in situations where users are unsure of what they want) (Martin and Odell 1996). Depending on the chosen purpose, prototypes can either evolve continuously until eventually they become the final software product, or in a step-by-step fashion in which they are discarded after every step (Pressman 1994; Brooks 1995>; Martin and Odell 1996). In line with the advice in the software engineering literature, it is recommended that prototyping may be used often and from the start in participatory simulation projects.

User Panels

3.9

The Freshwater Integrated Resource Management with Agents (FIRMA) project (seehttp://www.cpm.mmu.ac.uk/firma) has used panels of stakeholder representatives to evaluate design proposals and this is a technique that may also be useful for many participatory simulation projects. Potential users of the simulation software are gathered around a table and the software is demonstrated to them. The panel is then asked to discuss it, and the resulting interaction is videotaped for later analysis. The approach is similar to using a focus group (and the procedures for running the panel are similar), but differs in that the group is responding to the software as a 'stimulus' for their discussion. A similar format can be used for de-briefing sessions when several users have tried the software - it may be better to have group de-briefing sessions than individual ones. It is envisaged that this technique may be useful during the evaluation phase of participatory simulation projects because the comments from users are likely to be much more revealing and helpful than could be obtained from, for example, a structured evaluation form.

Modelling the European Digital Contents Market

4.1

The Simweb project has used the four techniques outlined in the previous section in the development of two participatory simulations, one modelling the development of online news and the other, online music. In the paragraphs that follow, some results of using such techniques during the initial iterative cycles of simulation build and evaluation are presented.

Model Conceptualisation: Scenario Analysis(es)

Goal:	Workshop(s) to explore future possibilities by developing scenarios
Structure:	Half-day workshop with agenda and minutes
Inputs:	Literature review about the state-of-the-art of the online news and music sectors
Participants:	Researchers and Board representatives from end-user organisations (between 10 and 12 people per session)
Outputs:	Summary report; video and audio recording of the meeting

4.2

The Scenario workshops contributed to a better understanding of the social, technological, economic, environmental and political factors affecting the sectors in which the end-user organisations operate (e.g. newspapers in the online news sector, music retailers, etc.). However, the prior commitments of some of the Board members prevented them from participating for the full duration of the workshop. While they found the exercise interesting, immediate operational issues were of much higher priority.

Model Design: JAD Workshop(s)

Goal:	Workshop for software conceptualisation and design that involves both developers and end-users
Structure:	Day workshops with agenda, minutes, presentations by participants (as starting points for some of the topics for group discussion) and use of visual aids (e.g. flipcharts) for team design/brainstorming
Inputs:	Executive summaries or reports resulting from the previous scenario analysis. Brief, initial descriptions of the online news and music markets derived from literature reviews, face-to-face or telephone interviews with market players, survey data analysis, etc.
	Rough (prototype) software design. The prototype was presented to the JAD audience for feedback as a presentation from a projected laptop
Participants:	End-users representing the sectors and developers (between 7 and 10 people)
Outputs:	An initial list of the main entities (whether animate or inanimate) which will make up the model structure (e.g. software agents: key attributes and properties, their interactions, etc.)

4.3

The JAD workshops were useful for the learning and exchange of ideas between partners with different technical expertise (for example, software developers and news media). As the project evolved, participants engaged in additional face-to-face meetings between research partners and increased telephone, email and web-based communications among research team members. One problem was the learning required of those who needed to grasp specialist aspects of the development starting with very little previous knowledge (e.g. about agent-based simulation, industry sectors, IT platform and tools, etc.).

Model Construction: Prototyping

Goal:	Building a quick and rough version of a desired software/system involving designers and end-users
Structure:	The number of software developers involved in prototyping tends to vary depending upon the size and needs of each project. In Simweb's case, for example, the prototypes were built by small teams (3 to 5 software developers) based at different locations.
Inputs:	Reports and software releases derived from previous activities (e.g. scenario analysis, JAD meetings, emails, surveys, web site documentation, etc.), but mainly the initial model structure, draft simulation software requirements, and initial software prototype releases.
Participants:	Simulation designers and developers with input from relevant stakeholders (researchers, end-users)
Outputs:	Up-to-date prototype software release. Prototypes are functional software but with an incomplete model and reduced features. They may or may not evolve into the final simulation software product as required by stakeholders. The software may consist of, for example, of a graphical input/output interface plus a core background application.

4.4

With regards to prototyping, 'quick' software releases can provide a tangible basis for discussion about requirements and features among team members. This, combined with the less formalised, rapid development environment of a small team of developers can encourage frequent face-to-face communications. In Simweb, the proximity of developers at times resulted in the use of simpler, low tech design tools (e.g. pen-and-paper, blackboard, etc.), but email, instant messaging and video conferencing were also used.

4.5

The rapid prototyping cycles in Simweb each lasted approximately 12 weeks (including software design, coding and testing). But it is expected that as the requirements become clearer and the software more robust, the design/build cycles will take longer to complete (around six months). That is, the software build activity then becomes a more 'top-down' activity, because the previous prototypes provide a more useful tool for discussion among developers and end-users.

4.6

Partitioning the prototyping tasks amongst partners in different locations led to some software integration issues. The speed at which the initial prototypes are built (combined with the learning curve with technical aspects and difficulties in pinning down software requirements) also led to lack of time for thorough software testing and more need for software 'debugging'. Also, the focus of rapid prototyping on programming activities left developers little time to produce software documentation. In Simweb's case, for example, given the rapid iterative approach, it became difficult for the small team of developers to produce relevant (software and end-user) documentation (e.g. detailed requirements and design specifications, test specifications, tutorials, etc.). However, some UML (Unified Modelling Language, Jacobson et al. 1999) descriptions are being used by prototype developers to represent objects/agents in more recent software iterations.

Evaluation: User Panel

Goal:	Demonstration and discussion among developers and users
Structure:	A full day meeting with agenda and minutes. Given time constraints and a need to capture as much feedback from end-users as possible during these panel meetings, roles were assigned to participants prior to these meetings (e.g. chair/time keeping, facilitator, minute-taker).
Inputs:	Prototype software presentation(s) by relevant stakeholders
	A list of unresolved issues usually provided by the prototype developers (e.g. behavioural rules for the agent-based software; software implementation issues, data related issues, etc.)
Participants:	Between 10 and 12 people including representatives from all the research partners, and representatives from end-user organisations
Outputs:	Hand-written (or electronic) notes about software refinements required based mainly on user feedback

4.7

During User Panel meetings, prototype presentations by technology partners (and the discussions which follow) assisted participants to understand better what to expect as an outcome from the simulation modelling process. Hands-on demonstrations (with real data) of prototype software can help users assess the usability and functionality aspects of the software more effectively. Following the User panel, additional face-to-face meetings between partners may be required in order to refine software requirements prior to further iterative simulation development cycles.

Conclusion

5.1

This paper has offered an initial set of techniques and ideas for the development of participatory agent-based social simulation. It has done this by drawing on the extensive literature on methodologies to be found within the field of software engineering and on the much rarer accounts of how existing social simulation models have been developed. An illustration of the use of these techniques in the Simweb project on business strategies for digital online media were presented. Overall, the participatory simulation approach has proved useful as a research design strategy for Simweb. In particular, it has contributed to the detailed planning and conduct of tasks and as such it has been a useful guide for the project management and researchers. However, it is not possible to compare the progress of the project with what might have happened if a less participatory approach had been taken, or if the methods that were used had not been adopted, since we only have the one history of the project and no control case study. It is our impression that it was easier to motivate all the partners to engage with the project because their participation is so clearly a central part of the project work. But we cannot tell whether, without the participation, the development might have been more straightforward or efficient.

5.2

It is for this reason that the methods and techniques suggested in this paper should be regarded at this stage as hypotheses: it seems plausible that adopting some of these techniques could enhance the quality and effectiveness of social simulation models, but the community does not yet have enough experience and has not yet devoted sufficient effort to recording and synthesizing that experience to be sure that these techniques do work as one might expect. In this respect, this paper is merely a start and much remains to be done if we are to understand clearly how to build participatory agent-based models so that we are able to pass on our expertise to those who come after.

Acknowledgements

The study has been conducted as part of a European Union supported project, Simweb, contract IST-2001-34651.

The authors are indebted to the anonymous reviewers of the article for their suggestions and to all the members of the project, who not only provided the inspiration for using rapid prototyping, but also consented to be guinea pigs, being observed as subjects for our study while wrestling with the trials of gathering data in a fast moving business sector which changed as we studied it, and also producing complex software to schedule. The following in particular contributed to this study: Kornelia van der Beek, Mary Ebeling, Cornelia Krüger, Barbara Llacay, Maite López, Xavier Noria, Gilbert Peffer, Stephan Schuster, Paula Swatman and Miguel Varela.

References

AGRESTI, W. (1986). What are the New Paradigms? In Agresti W. (Ed.). New Paradigms for Software Development, Tutorial, IEEE Computer Society Press, pp. 6-10.

ASAKAWA, T. and Gilbert, N. (2002). Synthesizing Experiences: Lessons to be Learned from Internet-Mediated Simulation Games (forthcoming in Simulation and Gaming).

AXELROD, R. (1997). Advancing the Art of Simulation in the Social Sciences. In Conte, R., Hegselmann, R. and Terna, P. (Eds.). Simulating Social Phenomena, Lecture Notes in Economics and Mathematical Systems, pp. 21-40.

BARRETEAU, O., Bousquet, F. and Attonaty, J-M. (2001). Role-playing Games for Opening the Black Box of Multi-agent Systems: Method and Lessons of its Application to Senegal River Valley Irrigated Systems. Journal of Artificial Societies and Social Simulation, Vol. 4, No. 2,https://www.jasss.org/4/2/5.html.

BARRETEAU and others (2003). Our Companion Modelling Approach. Journal of Artificial Societies and Social Simulation, Vol. 6, No. 1, https://www.jasss.org/6/2/1.html

BONABEAU E. and Meyer, C. (2001). Swarm Intelligence: A Whole New Way to Think about Business. Harvard Business Review, Vol. 79, No. 5, pp. 107-114.

BOUSQUET, F., Barreteau, O., d'Aquino, P., Etienne, M., Boissau, S., Aubert, S., Le Page, C., Babin, D. and Castella, J.-C. (2002). Multi-agent Systems and Role Games: Collective Learning Processes for Ecosystem Management. In Janssen, M. (Ed.). Complexity and Ecosystem Management: The Theory and Practice of Multi-agent Approaches, Edward Elgar Publishers.

BROOKS, F. (1995). The Mythical Man-Month: Essays on Software Engineering. Anniversary Edition, Addison-Wesley, MA.

CHAMBERS, R. (2002). Participatory Workshops: A Sourcebook of 21 Sets of Ideas and Activities. Earthscan Publications, U.K.

COLELLA, V., Borovoy, R. and Resnick, M. (1998). Participatory Simulations: Using Computational Objects to Learn about Dynamic Systems, at: http://web.media.mit.edu/~vanessa/part-sims/chi.final.html

D'AQUINO, P., Barreteau, O., Etienne, M., Boissau, S., Bousquet, F., Le Page, C., Aubert, S. and Daré, W. (2002). The Role Playing Games in an ABM Participatory Modeling Process: Outcomes from Five Different Experiments carried out in the last Five Years. In: Rizzoli, A.E. and Jakeman, A.J. (Eds.). Integrated Assessment and Decision Support,1st Biennial Meeting of the International Environmental Modelling and Software Society, Lugano, Suisse, pp. 275-280.

EDMONDS, Bruce and David Hales (2003) Replication, Replication and Replication: Some Hard Lessons from Model Alignment Journal of Artificial Societies and Social Simulation vol. 6, no. 4 https://www.jasss.org/6/4/11.html

ERL, Thomas (2004) Service-oriented Architecture: A Field Guide to Integrating XML and Web Services. Prentice-Hall.

ESTRELLA, M. (Ed.) (2000). Learning from Change: Issues and Experiences in Participatory Monitoring and Evaluation, Intermediate Technology Publications, Institute for Development Studies, University of Sussex, UK.

GILBERT, N., Bowen, A., Collins, P., Gregory, M., Ormerod, P. and Saynor, D. (1998). Foresight: A Guide to Future Markets - Future Business. Department of Trade and Industry, UK.

GILBERT, N., & Troitzsch, K. G. (1999). Simulation for the social scientist. Milton Keynes: Open University Press.

GILBERT, N., Maltby, S., & Asakawa, T. (2002). Participatory simulations for developing scenarios in environmental resource management. In C. Urban (Ed.), 3rd workshop on Agent-based simulation (pp. 67 - 72). Passau, Germany: SCS-Europe.

HALES, David, Juliette Rouchier and Bruce Edmonds (2003) Model-to-Model Analysis. Journal of Artificial Societies and Social Simulation vol. 6, no. 4https://www.jasss.org/6/4/5.html.

HANNON, B. and Ruth M. (2001). Dynamic Modelling, Second Edition. Springer-Verlag, New York.

HARE, M., Gilbert, N., Medugno, D., Asakawa, T., Heeb, J. and Pahl-Wostl, C. (2001). The Development of an Internet Forum for Long-term Participatory Group Learning about Problems and Solutions to Sustainable Urban Water Supply Management In L. M. Hilty and P. W. Gilgen (Eds.), Sustainability in the Information Society, pp. 743-750. Marburg: Metropolis Verlag.

HARE, M. and Pahl-Wostl, C. (2002). Stakeholder Categorisation in Participatory Integrated Assessment Processes. Integrated Assessment, Vol. 3, No. 1, pp. 50-62.

JACOBSON,I., Booch, G., and Rumbaugh, J. (1999) The Unified Software Development Process. Addison-Wesley Professional, MA.

KEARNEY, P J and W. Merlat (1999) Modelling Market-Based Decentralised Management Systems. BT Technology Journal, 17(4) pp. 145-56.

LYNAM, T., Bousquet, F., Le Page, C., d'Aquino, P., Barreteau, O., Chinembiri, F. and Mombeshora, B. (2002). Adapting Science to Adaptive Managers: Spidergrams, Belief Models and Multi-agent Systems Modeling. Conservation Ecology 5(2): [online] URL:http://www.ecologyandsociety.org/vol5/iss2/art24/.

MARTIN, J.(1991). Rapid Application Development, Macmillan, New York.

MARTIN, J. and Odell, J. (1996). Object-oriented Methods: Pragmatic Considerations. Prentice Hall, New Jersey, U.S.

PARKER, C. and Swatman, P. (1999). An Internet-mediated Business Simulation: Developing and using TRECS. Simulation and Gaming, Vol. 30, No. 1, pp. 51-69.

PARUNAK, H., Savit, R. and Riolo, R. (1998). Agent-Based Modeling vs. Equation-Based Modeling: A Case Study and User's Guide. In Sichman, J., Conte, R. and Gilbert, N. (Eds.). Multi-Agent-Based Simulation, First International Workshop, MABS 1998, Paris, France, pp. 10-25.

PRESSMAN, R. (1994). Software Engineering: A Practitioner's Approach. European Adaptation. Third Edition. McGraw-Hill International, Maidenhead, UK.

PURNOMO, H., Priyadi, H., Yasmi, Y., Yuliani, L. and Prabhu, R. (2002). Development of Multi-stakeholder Scenarios of Forest Management: A Multiagent System Simulation Approach. Participatory Modeling Symposium, Harare, Zimbabwe.

SAMPAIO, J., Hadad, G., Doom, J. and Kaplan, G. (2000). A Scenario Construction Process. Requirements Engineering, Vol. 5, No. 1, pp. 38-61, Springer-Verlag, London.

SULEIMAN,R., Troitzsch, K.G. and Gilbert, N (2000) Tools and Techniques for Social Science Simulation. Heidelberg: Physica-verlag.

TERNA, P. (1998). ABCDE: Agent Based Chaotic Dynamic Emergence. In Sichman, J., Conte, R. and Gilbert, N. (Eds.). Multi-Agent-Based Simulation, First International Workshop, MABS 1998, Paris, France, pp. 79-94.

THESEN, A. and Travis, L.E. (1995). Simulation for Decision Making. PWS Publishing Company, Boston. M.A.

TOBIAS, R. and Hofmann, C. (2004) Evaluation of free Java-libraries for social-scientific agent based simulation Journal of Artificial Societies and Social Simulation vol. 7, no. 1 https://www.jasss.org/7/1/6.html.

VAN Asselt, M., Mellors, J., Rijkens-Klomp, N., Greeuw, S., Molendijk, K., Beers, P. and van Notten, P. (2001). Building Blocks for Participation in Integrated assessment: A Review of Participatory Methods. ICIS working paper I01-E003 ICIS, Maastricht, The Netherlands.

VENNIX, J. (1998). Group Model Building: Facilitating Team Learning using System Dynamics, John Wiley & Sons, Chichester, UK.

WELLS, D. (2001). Extreme Programming: A Gentle Introduction at: http://www.extremeprogramming.org.

WICKENBERG, T. and Davidsson, P. (2002). On Multi Agent Based Simulation of Software Development Processes. In Sichman, J., Bousquet, F. and Davidsson, P. (Eds.). Multi-Agent-Based Simulation, MABS 2002, Bologna, Italy, pp. 104-113.

ZHU, H. and Jin, L. (2000). Scenario Analysis in an Automated Tool for Requirements Engineering. Requirements Engineering, Vol. 5, No. 1, pp. 2-22, Springer-Verlag, London.

Return to Contents of this issue

© Copyright Journal of Artificial Societies and Social Simulation, [2004]