Abstract

The potential of agent-based modeling (ABM) has been demonstrated in various research fields. However, three major concerns limit the full exploitation of ABM; (i) agents are too simple and behave unrealistically without any empirical basis, (ii) 'proof of concept' applications are too theoretical and (iii) too much value placed on operational validity instead of conceptual validity. This paper presents an operationalization approach to determine the key system agents, their interaction, decision-making and behavior for context specific ABM, thus addressing the above-mentioned shortcomings. The approach is embedded in the framework of Giddens' structuration theory and the structural agent analysis (SAA). The agents' individual decision-making (i.e. reflected decisions) is operationalized by adapting the analytical hierarchy process (AHP). The approach is supported by empirical system knowledge, allowing us to test empirically the presumed decision-making and behavioral assumptions. The output is an array of sample agents with realistic (i.e. empirically quantified) decision-making and behavior. Results from a Swiss mineral construction material case study illustrate the information which can be derived by applying the proposed approach and demonstrate its practicability for context specific agent-based model development.

Keywords:

Agent Operationalization, Decision-Making, Analytical Hierarchy Process (AHP), Agent-Based Modeling, Conceptual Validation

Introduction

1.1

During the last decade, agent-based modeling (ABM) has been regarded as a promising methodology for quantitative modeling in the social sciences (Axelrod 1997; Epstein and Axtell 1996; Gilbert and Troitzsch 2005; Janssen 2002; Tesfatsion and Judd 2006), but not without contradictory trends. Although ABM's potential for modeling a variety of phenomena in different research fields has been repeatedly demonstrated (e.g. Bousquet and Le Page 2004; Macy and Willer 2002), its effectiveness in solving problems more relevant to the real world is increasingly being questioned (Louie and Carley 2008; Parker et al. 2003). The three central questions being raised are: (i) How to go beyond a "proof of concept" (e.g. Janssen and Ostrom 2006) (ii) How realistic are agents with simple behavioral rules? (e.g. Jager and Janssen 2002, Mosler and Tobias 2005) (iii) How could or should agent-based models be validated? (e.g. Axelrod 1997, Windrum et al. 2007, Louie and Carley 2008).

1.2

Beyond "proof of concept": While the potential of ABM for addressing a wide range of research question in social sciences is undoubted, there is a growing appreciation that there is a need for addressing problems more relevant to the real world (Matthews et al. 2007). Janssen and Ostrom (2006) claim that ABM has mostly been applied to the modeling of theoretical issues, whereas its application to empirically measurable phenomena is quite rare, and models therefore often do not go beyond a "proof of concept". These authors distinguish four ways (stylized facts, laboratory experiments, role games and case studies) of how empirical data can be included into ABM depending on the number of subjects and the degree of contextualization or generalization. In addition, Boero and Squazzoni (2005) highlight the importance of ABM's empirical embeddedness. They argue that empirical knowledge needs to be integrated into modeling practice and used for micro specification as well as macro validation by integrating ABM with qualitative, quantitative, experimental and participatory methods. Although these studies make a significant contribution to the development and classification of empirically-based ABM, they conclude that new approaches are still needed, in particular regarding the empirical validation of ABM and the formalization of empirical knowledge integration into ABM.

1.3

Behaviorally realistic agents: Most of the recent applications in ABM implement rather simple behavioral rules. The underlying decision-making process, however, is usually not included (Macy and Willer 2002), despite the fact that one of the specific advantages of ABM is its ability to model individual decision-making entities and their interactions (Matthews et al. 2007). This may have two reasons. First, simple behavioral rules are easily implementable, whereas the underlying decision-making is often regarded as a rather complex process (Mintzberg et al. 1976). Second, behavior itself can be better observed than the underlying decision-making processes (Keeney 1982). To overcome these issues, Mosler et al. (2001) highlight the need for a theoretical and empirical (Mosler and Tobias 2005) basis for collective action simulation. Following this line, Jager and Janssen (2002) propose a general theoretical decision-making framework, based on the six decision rules applicable to different situations, and propose basing the agent architecture on a solid empirical ground (Janssen 2002). That is, in order to achieve more behaviorally realistic agents, the agents' architecture needs to shift from simple behavioral rules to more complex decision making processes with a solid basis in theory and empiricism.

1.4

Model validation: According to Gilbert and Troitzsch (2005) "a model which can be relied on to reflect the behavior of the target is valid".^[1] Operational validation as the most widely accepted way to perform model validation (Sargent 2008) is difficult to perform in ABM (Louie and Carley 2008; Schutte 2010; Windrum et al. 2007). Typically, operational validation is performed by comparing the simulation output with the system (i.e. problem entity, target) data (Gilbert and Troitzsch 2005; Sargent 2008). This is impossible to perform for the future development of a system, and it is rather difficult, if emergent phenomena are modeled. First, per definition, emergent phenomena patterns as aggregated outcomes cannot be predicted by examining the system's elements in isolation (Parker et al. 2003). Second, the empirical detection of emergent phenomena in a real system is difficult, because they are described as patterns rather than as numerical values (Grimm et al. 2005) and are often not recognized. The difficulty with or even impossibility of operational validation for ABM increases the importance of the other ways of model validation, in particular, conceptual model validation. Conceptual model validation is defined as "determining that the theories and assumptions underlying the conceptual model are correct and that the model representation of the problem entity is 'reasonable' for the intended purpose of the model" (Sargent 2008). Consequently, to increase the validity of ABM it is necessary to focus on conceptual model validation rather than on comparing model performance with system data (i.e. operational validation).

1.5

Significant contributions in ABM have been made to overcome the three mentioned methodological shortcomings. However, none of them explicitly addresses all three issues. Thus new approaches are still needed to include more behaviorally realistic agents, in particular regarding agents' decision-making and behavior, and empirical data, with more emphasis on conceptual validation.

1.6

Our paper therefore aims at contributing to filling this gap, by presenting an approach for empirically operationalizing agents, their interaction, decision-making and behavior for ABM. The approach was developed for highly context specific ABM applications where high-stakes and/or reflected decisions are involved. As a participatory approach it requires direct contact with the actors. We exemplify the approach by presenting operationalized agents for an ABM of the Swiss construction stakeholders' material selections case study. In the following we use the term operationalization as "the transformation of an abstract, theoretical concept into something concrete, observable, and measurable" (Scott and Marshall 2005). Furthermore, we define agents as the model representatives of real world social actors, such as construction stakeholders in this case study.

1.7

The paper is structured as follows: We start with a short introduction of our case study. Second, we provide an approach for the operationalization of agents' identification, interactions and decision-making for ABM, based on structural agent analysis (SAA) and the analytical hierarchy process (AHP). We support each step of the approach by presenting results from the recycling construction material case study and elaborate the potential and limits of the methods used. Third, we discuss the contribution of the approach to the above mentioned shortcomings. Finally we draw conclusions from our findings and propose further research.

Case study introduction: Demand for recycling mineral construction materials (RMCM) in Switzerland

1.8

Increasing amounts of construction and demolition (C&D) waste have been observed worldwide (Bergsdal et al. 2007; Brunner 2004; Hao et al. 2007; Hashimoto et al. 2007; Moser et al. 2004; Muller 2006; Wang et al. 2004). So far, C&D waste has been deposited or reused for low-grade applications (Moser et al. 2004; Tam and Tam 2006). Limited landfill and low-grade application capacities led to the development of high performance applications (e.g. Hoffmann and Leemann 2006). However, due to a lack of construction stakeholders' recycled mineral construction materials (RMCM) acceptance, information and training (Hoffmann 2004; Spoerri et al. 2009), RMCM are still deposited or down-cycled and not reused at the same application level. This study aims at developing strategies for aligning the demand for RMCM and the increasing C&D waste amounts by analyzing and modeling stakeholders' decisions and interaction influencing the demand for RMCM.

The agent operationalization approach for ABM

Conceptual framework for the operationalization approach

2.1

Our operationalization approach is based on the conceptual framework presented in Figure 1. The theoretical foundations are Giddens' structuration theory (Giddens 1984) and the theory of planned behavior (Ajzen 1991). Material and energy flows on the aggregate level are affected by micro level agents' decisions and interactions, which in turn are influenced in their decision-making by the social and physical environment (Axtell et al. 2001). This dualism between the micro and macro level (Andrews 2001) relates to key system features modeled with ABM, namely emergence of social structure based on micro behavior and feedback of the new structure on the behavior itself. The structural agent analysis (SAA) uses Giddens' structuration theory (Giddens 1984) for a heuristic aimed at analyzing this micro-macro relationship, more specifically, for coupling social science approaches to material flow analysis (MFA) (Binder 2007b, 2007a). That is, it provides a conceptual basis for the modeling socio-ecological as well as socio-technical systems with ABM.

2.2

This conceptual framework consists of the agents (decision-making and behavior), social structures (rules and resources), and the agents' environment. It includes the consequences of the agents' behavior on social structures, environment (e.g. material flows) and other agents' decisions (Figure 1). The outcome from the decision-making process (i.e. decision preference) can be seen as the intention, according to the theory of planned behavior (Ajzen and Fishbein 1977; Ajzen 1991), determining to a large extent the agents' behavior (Ajzen and Madden 1986). The decision-making itself can be directly affected by past individual behavior and the behavior of other agents, through the perceived intended and unintended consequences (Feola and Binder 2009; Triandis 1980). Furthermore, decision-making can be influenced by the rules and resources of the social structure and the perceived environmental consequences. The behaviors of agents affect the environment synchronically (e.g. the disposal of construction waste that is not reused) and/or the decision-making of other agents (e.g. material recommendations from structural engineers) and with a certain time delay the social structure (e.g. development of law and standards for emergent technologies).

Figure 1. Conceptual framework of the interaction between social structure, agents and the environment (continuous arrows indicate synchronic, dotted arrows indicate diachronic impacts) (adapted from Binder 2007a, Giddens 1984 and Nikolic 2009)

2.3

In ABM, system behavior (e.g. social structure and environment) emerges from the agents' behaviors and interactions (Axelrod 1997; Gilbert and Troitzsch 2005; Janssen 2002; Tesfatsion and Judd 2006). Therefore, knowing the relevant agents affecting the problem addressed (step 1), determining their interaction (step 2), analyzing their decision-making process including its determinants (step 3), is sufficient for agent operationalization for ABM (Figure 1). In addition, one must analyze how consistent decision preference (intention) and behavior are (step 4) to conceptually validate the model. For each of the four steps a sound theoretical background and empirical methods are required (Table 1).

2.4

In this paper, we use Giddens' structuration theory (Giddens 1984) as a guideline for the assumptions-in-design of the ABM. Giddens' structuration theory is only one among several social process theories (Cedermann 2005) and the issue of how different social process theories could possibly be implemented in ABM and what theory is best suited for each particular model's purposes is still being debated. Nevertheless, the suitability of Giddens' structuration theory for ABM operationalization is highlighted by its focus on how social structure emerges from human action (Binder 2007a). Further, Cedermann (2005) has concluded that the agent-based paradigm is fundamentally compatible with process-theoretical foundations. Finally, because our approach explicitly aims at the agent operationalization, the macro level analysis (i.e. social, technical and natural environment) is not explicitly addressed in this paper.

Table 1: The four steps of the agent operationalization approach
Step	Description	Theoretical background (exemplified)	Methods (exemplified)
Prerequisite step: Problem definition (Precise definition of the problem addressed and the purpose of the model)
Step 1	Identification of the relevant agents	Social network theory (Wasserman and Faust 1994)	Agent-impact analysis
Step 2	Analysis of agents' interaction chain	Economic action and social structure (Granovetter 1985) and theory of embeddedness (Uzzi 1997)	Expert interviews Expert workshops
Step 3	Quantification of agents' decision-making process	Multi criteria decision analysis (MCDA) (Belton and Stewart 2002) / Analytical hierarchy process (AHP) (Saaty 1980)	Expert interviews Expert workshops Survey methods
Step 4	Behavioral consistency analysis and conceptual validation	Theory of planned behavior (Ajzen 1991) and interpersonal behavior (Triandis 1980)	Survey methods

Prerequisite step: Problem definition

2.5

As a clear purpose and problem definition are considered indispensable for modeling (Costanza et al. 1993), they set the stage for all subsequent steps of the agent operationalization. This is particularly important when one is using participatory approaches for gathering empirical knowledge as is proposed here (Cornwall and Jewkes 1995). Shifting model purposes or problem misunderstandings may otherwise increase the so called "error of the third kind" defined as "the probability of having solved the wrong problem when one should have solved the right problem" (Mitroff and Featheringham 1974).

Step 1: Identification of the relevant agents:

2.6

The goal of this step is to identify the key system actors to be included as agents in the ABM. According to social network theory (e.g. Wasserman and Faust 1994) key system actors within a network are active, able to connect to each other through efficient paths, have the potential to mediate flows between other actors and are tied to other central actors (Faust 1997). In other words, key system actors are actors which strongly affect the system and are themselves strongly affected by the system. In order to identify the key system actors, we propose the actor impact analysis (AIA) adapted from qualitative cross-impact analysis (Godet 1994; Gordon and Hayward 1968; Götze 1991; Scholz and Tietje 2002; Vester 2007; von Reibnitz 1992), which performs an analysis of the actors' activity, revealing their connectedness and impact on other possible actors.

2.7

In doing so, first all relevant actors affecting the system are identified. This can be done either by analyzing the actors' interaction with the system along the production-consumption chain (Maier Bergé and Hirsch Hadorn 2002), their functional relationships (Hermans 2005) or by studying which actors interact with each other through, for example, information, money or resource flows (Hirsch Hadorn et al. 2002; Knoeri 2007). The indicator for defining the actor interaction shall be chosen according to the predefined problem definition and model purpose. If multiple indicators are possible several interaction matrixes might be constructed and compared. We propose doing all this by considering literature, expert interviews (Mieg and Näf 2006) or consensus building expert workshops (Susskind et al. 1999).

2.8

Second, all potential direct impacts between the actors are set up in a cross-impact matrix and their strengths are assessed on predefined scales (e.g. from 0 to 2; 0 means no influence, and 2 strong influence). This can be done through expert interviews (Knoeri et al. 2011) or workshops. The sums of the row entries in the matrix reflect the influence values (activity sum) and the sums of the column entries the dependence values (passivity sums) (Godet 1994,Lang et al. 2006). Thus, depending on their activity or passivity the actors can be classified into disconnected, indicating, driving and key connected actors referring to their roles in the system (Table 2).

Table 2: Actor types in AIA
actor type (role)	influence value (activity sum)	dependence value (passivity sum)
driving / active	high	low
key connected / ambivalent	high	high
indicating / passive	low	high
disconnected / buffering	low	low

2.9

The results of the cross-impact matrix can be visualized in a system grid (Scholz and Tietje 2002; Tietje 2005). Figure 2 illustrates a system grid for the case of RMCM showing the various actor types involved. The key connected actors were the awarding authorities, architects and engineers and contractors (i.e. prime, masonry and concrete, roadwork contractors). They were key in the sense of strongly influencing other actors and being strongly influenced by others. The construction material production actors, deconstruction and disposal actors were passive system actors. They were medium linked with other actors, whereas their strong relations were mainly unidirectional (i.e. they were strongly influenced by other actors). Therefore, they served as indicators for system behavior. In a manner similar to passive actors, active actors (i.e. regulation authorities) had mainly unidirectional relations, although with reversed signs (i.e. they strongly influenced other actors) and acted as drivers in the system. Media (i.e. daily press and journals) as well as academia (research institutes) were considered disconnected or buffering system actors being loosely linked with the system (i.e. fewer and less important relationships).

2.10

Finally, the key system actors to be included as agents in the model are selected. The ambivalent or key connected actors are considered most important for agent operationalization for ABM, as any change in their behavior has large impacts on the system (Asan and Asan 2007,Scholz and Tietje 2002). Consequently the awarding authorities, architects, engineers and contractors were selected for inclusion in the case of RMCM.

Figure 2. System grid of the actor groups (dependence and influence values; means of the two system experts)

2.11

Selecting the key connected actors to be included in ABM ensures that those system actors that are most affected and have the most impact will be included in the model (Faust 1997; Schlange and Juttner 1997; Wasserman and Faust 1994). Nevertheless, other actor groups, especially active actors due to their driving role, may be additionally considered for being operationalized as agents in ABM. However, since these groups are only weakly influenced by the system, they can also be included as external parameters affecting the system. This is the way regulation authorities were included in the RMCM case study, which allows to simulate the effect of regionally-different regulation practices on agents behavior and thus, on the RMCM demand. If the research focus lays on changing regulation practices, regulation authorities might become key connected actors and might be included as agents in the model.

Step 2: Analysis of agents' interaction chain

2.12

This step determines both parts of the agents' interaction: How agents interact with each other (i.e. agents' interaction chain) and how they select each other (i.e. agents' embeddedness). This is considered to be a key step for ABM, because of its focus on agent interaction (Macy and Willer 2002; Reynolds 1987). Furthermore the graph of the agents' interaction chain provides the first conceptual model. Social structure (i.e. agents' interaction chain) and embeddedness in the network (i.e. strength of the ties) are important for agent interaction. It is acknowledged that economic action is embedded in social structure, in contrast to neoclassical atomized-agent approaches (Granovetter 1985). In particular in interfirm networks embeddedness in social structure has beneficial effects on performance (Uzzi 1997). We therefore propose to analyze the agents' local interconnections and embeddedness in two steps.

2.13

First, agents' local interconnections and feedbacks (i.e. agents' interaction chain) determining system behavior are identified. In this way we analyze how agents are linked to other agents (e.g. awarding authorities specify project to the architects). This determines which agents potentially interact. Furthermore, possible interaction options (i.e. behavioral alternatives) are identified. We propose doing this step as a combination of literature review and participatory approaches (e.g. expert interviews or workshops) (Cornwall and Jewkes 1995; Mieg 2000).

2.14

Second, agents' embeddedness in the network or the strength of their ties is analyzed. We propose doing this by analyzing the importance of network factors among the criteria which agents consider when selecting each other for the particular economic interaction (i.e. individual selection decision). According to Ling (2002) this depends on the criteria task performance, contextual performance, network and price factors. Therefore, for each selection decision, the particular decision criteria are defined and their impact is quantified. We suggest defining the criteria with a literature/theory review and weighting their importance on the individual selection decision with survey methods.

2.15

Agent interaction chain: Figure 3 shows the conceptual model we have developed for the case of RMCM, illustrating chronologically the agents' interaction chain with multiple involvements of the awarding authorities. In the project specification (1) awarding authorities specify the project requirements, dictating the use of RMCM, claiming sustainable construction in general or making no specification about sustainable construction. Receiving the project specifications via the architects, structural engineers make material design specifications (2). They recommend conventional or recycled materials or give the option to choose one of the two, by specifying material properties. Architects project design (3) aims at recommending a project to the awarding authorities, meeting awarding authorities' requirements, engineers' recommendations as well as the architects' personal ambitions. In the project confirmation (4) the awarding authorities confirm or set the materials to be specified in the tender documents. Contractors submit their tender (5) to the awarding authorities in order to win the contract, submitting conventional and recycling materials. Again, awarding authorities commission the project to one of the tendering contractors (i.e. tender selection (6)) which finally determines the material demand.

2.16

The agents' interaction chain is highly context dependent and, therefore, not generalizable to nearby or associated decisions. All the more, there should be a consensus about agents' behavioral options when interacting, which can be achieved through expert interviews and workshops. Note that, for highly formalized interaction models, like those in the case study presented here, concentrating on the interaction decision affecting the problem studied might already be sufficient. For more informal social interaction various additional aspects (e.g. interdependence and relationship aspects) may gain importance (Rusbult and Van Lange 2003).

2.17

Agents' embeddedness / Individual agent selection: According to (Ling 2002), the key criteria for the individual selection decision in the building sector were job experience (task performance factor), reputation and personal contact (network factors) and economic considerations (price factor). Personal contact was the decisive network factor for most agents when selecting construction partners. The exception was public awarding authorities, who basically considered job performance and price factors, and architects who selected contractors mainly based on price considerations.

2.18

In the agent operationalization approach, the individual selection decisions were defined on a theoretical (e.g. Ling 2002) and an empirical basis (e.g. expert interviews), in contrast to many ABM applications where interaction mechanisms are defined on theoretical assumptions only. However, quantifying agents' embeddedness by analyzing how important agents' network criteria are when they select each other for an economic interaction might be limited when criteria have threshold utility functions (e.g. trust) (Uzzi 1997). In this case using hierarchical decision heuristics might be more appropriate. What types of networks emerge from the operationalized selection decisions and how they affect the system output will be addressed in the model evaluation.

2.19

The resulting conceptual model of the agents' interaction chain is the first step for ABM. Besides enhancing the understanding of agents' interaction, this approach increases the acceptance of the model through the participatory procedure. For the model implementation, it not only provides the qualitative agent interaction chain but also empirically quantifies the agents' selection decisions reducing the degrees of freedom of the model.

Figure 3. Agent interaction chain (blue boxes indicate the agents, light green boxes their decisions and green arrows their interaction).

Step 3: Quantification of the agents' decision-making process

2.20

The goal of this step is to quantify the agents' decision-making process. Thus, the decision criteria and their relevance to the choice of one of the behavioral alternatives determined in step 2 are specified.

2.21

Decision-making depends on the cognitive effort in the decision-making process (Jager and Janssen 2002; Jungermann et al. 1998; Svenson 1990,1996) and ranges from simple decision heuristics (requiring little cognitive effort) to homo economicus (a lot of cognitive effort and rational actors). Referring to Svenson (1990, 1996), Jungermann et al. (1998) distinguish routinized, stereotyped, reflected and constructed decisions with increasing cognitive effort involved. Because of the large investment sums involved in strategic economic decisions in general and construction decisions in particular, we propose to quantify reflected decisions according to Svenson (1990, 1996). Thus, decision makers know the options and actively strike a balance among the options regarding different criteria.

2.22

Analyzing the relevance of weighted criteria to agents' decision-making is the field of multi criteria decision analysis (MCDA) (Belton and Stewart 2002; Mendoza and Martins 2006). We based our MCDA analysis on the analytical hierarchy process (AHP) proposed by Saaty (1980), because it allowed us to structure complex decision-making processes (Saaty 1990) and to measure ratio scales on all hierarchical levels (Forman and Gass 2001).

2.23

Figure 4 illustrates the procedure of the AHP. In the decomposition phase, decision goal and alternatives are defined and the decision problem is decomposed into a hierarchy of decision criteria and sub-criteria clusters. Subsequently, the alternatives are compared with respect to each criterion and sub-criterion, and the relevance of the criteria and sub-criteria is assessed, in comparative judgments on pairs. In the hierarchical composition or synthesis, local criteria and sub-criteria priorities are multiplied to yield an overall alternative ranking. Finally, the consistency of the comparisons of pairs is assessed. (Please see Saaty (1980, Saaty 1994) for details and calculations.)

2.24

In the decomposition phase local system knowledge is important. We therefore propose decomposing the decision problem with participatory approaches such as system expert interviews (Mieg and Näf 2006). We propose using survey methods for quantifying the relevance of criteria and alternatives with comparative judgments. This enables one to achieve a reasonable numerical representation of the agents' decision making distribution in the population. (The AHP elicitation protocol used in the RMCM case study survey is reported in Appendix 1 (Table A.1).) Finally, the synthesis can be carried out through a matrix multiplication of the criteria weight vector with the alternative weight matrix leading to a performance vector of the alternatives (Saaty 1980).

Figure 4. Phases of the analytical hierarchy process according to Saaty (1980, 1994) illustrated with the Brunswikian lens model adapted from Scholz and Tietje (2002).

2.25

In the RMCM case study each decision of the agent interaction chain (Figure 3) was quantified according to the AHP procedure. Figure 5 shows as an example the decision-making process, criteria and the resulting alternative weights for the design specification decisions of structural engineers (i.e. decision (2) in Figure 3) for external concrete applications in our case study. From the column Alternative weights per criterion it can be seen, that the ranking of the alternatives was almost stable among the criteria with conventional concrete (CC) as the outperforming option, although their mean differed significantly. Regarding the expected tender price the three options were more balanced, while for the criterion project specification engineers experienced a performance of CC that was almost three times better than the recycling (RC) or property specification (PS) option. In Decision criteria weights, law and standards was the most important criterion followed by experience, whereas expected tender price and awarding authorities' project specification were less important. In addition, the comparably high standard deviations (i.e. up to more than half of the actual value) highlighted the existence of individual agents with different ranking preferences.

Figure 5. Alternatives, decision criteria, mean alternative weights per criterion, mean criteria weights and mean preferences in structural engineers' design specification for external concrete applications [Mean / StD, N = 70, CC: conventional concrete, RC: recycling concrete: PS: property specification].

2.26

AHP as an MCDA approach presupposes that agents fully process their decision information (Mendoza and Martins 2006), decide rationally and don't use simple decision heuristics (Johnson et al. 1988). AHP allows one to do a consistency check of the judgments of pairs, thus providing information on whether the methodical prerequisites are fulfilled. If the decision maker uses simple decision heuristics (e.g. repetition, imitation or normative behavior) MCDA approaches may not be adequate (Johnson et al. 1988; Jungermann et al. 1998; Svenson 1990, 1996). This may limit the applicability of the AHP for decision-making quantification when ordinary and more repetitive decisions are addressed (e.g. everyday consumer behavior). In such cases using other methods for the quantification of agents' decision making or specifying simpler decision rules based on agents behavior might by more adequate. Whenever decision makers decide consciously we consider AHP to be a good starting point, even though the great effort required for making the comparison of pairs in AHP may cause higher rates of survey drop out and lower response compared with behavior reporting studies.

2.27

For ABM this quantification procedure has two main advantages. First, it provides not only decision-making data reasonably representing the real population, but it also provides an array of sample agents to set up the model population. This allows one to skip the resource intensive step of deriving mathematical distribution functions from survey data and implementing agent populations based on these distributions. Second, the quantification based on the AHP provides data about all levels of each agent's decision-making process (e.g. criteria and alternative weight matrixes). The procedural structure of AHP further simplifies the decision-making implementation.

Step 4: Behavioral consistency analysis and conceptual validation

2.28

The goal of the last step is to analyze agents' behavioral consistency by comparing their behavior with the preferred alternative from the decision-making process, and to conceptually validate the presumed decision-making concept.

2.29

Behavioral consistency: Knowing to what extent the implemented decision-making process or behavioral rule explains actual behavior is fundamental in any behavioral modeling, and particularly in ABM. This is the operational validation of the decision-making model. Although we determined agents' decision-making process (step 3), the preferred alternative (i.e. intention (Ajzen 1991)) may differ from the subsequent behavior, because of external (i.e. contextual factors) and internal drivers (i.e. habit and psychological arousal) (Feola and Binder 2009; Triandis 1980). In addition, perceived behavioral control may influence behavior directly and via intention (Ajzen 1991, Armitage and Conner 2001).

2.30

Assuming rational stakeholders, the best performing alternative, derived by the AHP synthesis (Figure 4), is preferred. Comparing the best performing alternative for every individual agent with his actual behavior allows one to assess whether the intended behavior (i.e. decision preference) differs from the reported one. We propose determining the behavior of the key agent groups with survey methods (e.g. according to Diekmann 2007) in combination with the survey conducted for analyzing the decision-making process (Step 3).

2.31

In the case of RMCM, structural engineers' preferred option was highly consistent (77%) with reported behavior. They decided for the conventional alternative (i.e. best performing) in 80% of the structural concrete application cases (60% for lean concrete applications).

2.32

The high behavioral consistency confirmed that in reflected decisions the effect of perceived behavioral control (Armitage and Conner 2001) as well as the effect of habit and psychological arousal is minimized. Although the high behavioral consistency demonstrates the usefulness of the decision-making model, potential differences between actual and reported behavior may limit the usefulness of our approach. This is because of more frequently reported socially desirable answers or biases in survey participation. This difference can be quantified and the limitations minimized by analyzing how the sample represents the basic population studied regarding socio-demographic and behavioral variables.

2.33

Conceptual validation: Conceptual validation requires assuring that theories and assumptions underlying the decision-making model are correct. This goes beyond providing a decision-making model simply mirroring behavior.

2.34

According to Svenson (1990, 1996), the assumption behind quantifying decision-making with AHP is a reflected decision, where decision-makers consciously strike a balance between known alternatives and decision criteria. In reflected decisions we expect to derive consistent judgments in the AHP comparisons of pairs. In other words, comparing options of pairs reveals absolute options' values, which mirror the relative judgments. The AHP consistency analysis gives insight into how consistently the comparisons were made and therefore how high the cognitive effort in the decision was. A certain inconsistency (10%) is hereby accepted in the standard AHP (Saaty 1980). In the adapted procedure presented here, alternatives and criteria were predefined and therefore higher inconsistencies were expected.

2.35

In the case of RMCM, structural engineers showed slightly higher inconsistencies (i.e. 44% for weighting the criteria, 24% for weighting alternatives) in their comparative judgments compared with the other agent groups. In other words, they may use simpler decision heuristics.

2.36

The steps of comparing decision-making preferences and behavior as well as empirically validating underlying decision-making assumptions are key for ABM. Analyzing behavioral consistency allows one to assess the operational validity of the decision-making model. The conceptual validity of the decision-making process further increases the overall conceptual validity of the ABM.

Discussion

3.1

This paper addressed three major shortcomings limiting a full exploitation of ABM's potential; (i) applications "proof of concept" that is too theoretical, (ii) agents that are too simple and not behaviorally realistic and lack a basis on empirical data and (iii) too much value placed on operational validity instead of conceptual validity. Furthermore, the agent operationalization approach was presented as a specific procedure that links theoretical concepts and empirical methods addressing the above mentioned shortcomings. This approach provides guidance to identify the relevant agents, analyze their interaction, quantify their decision-making and conceptually validate agents' decision-making.

3.2

In the following we discuss how the agent operationalization approach contributes to each of the shortcomings highlighted in the introduction.

Beyond proof of concept

3.3

Janssen and Ostrom (2006) argue that "although most models are inspired by observations of real biological and social systems, many of them have not been rigorously tested using empirical data and therefore do not go beyond a 'proof of concept'." Including empirical system knowledge regarding ABM is referred to as participatory or collaborative modeling (Voinov and Bousquet 2010). According to Moss (2008), the agent operationalization approach lies between the "economic modeling" and the "companion modeling" approach. Like the economic modelers we presume the existence of a real data generating process (e.g. decision-making process) (Windrum et al. 2007), but we aim at observing and quantifying it directly by including local system knowledge as in the companion modeling approach (Barreteau 2003; Bousquet and Le Page 2004). Integrating empirical system knowledge has been found to be important for case studies in general (Scholz and Tietje 2002) and resource management in particular (Pahl-Wostl 2007). Furthermore it generates trust in the model through participant identification (Berger et al. 2007) and promotes ownership through stakeholder involvement (Nikolic 2009).

3.4

The contribution of the here presented agent operationalization approach consists in providing a specific strategy for embedding empirical knowledge into modeling practice as called for by Boero and Squazzoni (2005). Therefore, empirical knowledge is gathered at each step of the approach. However, the proposed approach to operationalizing agents for ABM was developed as a case-based model. The price for higher model realism achieved by this context dependency is less generality (Costanza et al. 1993). We acknowledge the broad range of ABM application from highly context specific "case-based models" to generalizable "theoretical abstractions", influencing the type of empirical data and validation methods required (Boero and Squazzoni 2005; Janssen and Ostrom 2006). The adaptation of the proposed approach for operationalizing agents to "typifications" or "theoretical abstractions" will therefore be the subject for further research.

Behaviorally realistic agents

3.5

Focusing on individual decision-making rather than on simple behavioral rules (Macy and Willer 2002) is the first step required towards more behaviorally realistic agents (Janssen 2002). The agent operationalization approach contributes to that by obtaining an array of sample agents (including their decision-making and behavior) as well as allowing one to operationally validate the individual decision-making model by comparing decision-making preferences and behavior (i.e. behavioral consistencies):

3.6

Array of sample agents: The array of sample agents is obtained by operationalizing the agents' decision-making process through the AHP. AHP allows one to indirectly gather data about agents decisions by weighting criteria and alternatives per criterion, while the final alternative decision is derived by a simple matrix calculation (Saaty 1990). The ratio-scale weighting method we have included (Jia et al. 1998) simplifies transfer of the derived information into ABM. In other words, deriving an array of sample agents' decision-making based on AHP provides not only a set of directly implementable decision-making data but also the procedure for its processing. This significantly reduces the models' degree of freedom and decreases the parameters space to scan. However, there will still be remaining assumptions-in-design which have to be specified (e.g. agents' time horizon for their retrospective memory) and whose effects on the system output have to be analyzed.

3.7

Behavioral consistencies: Decision preferences (i.e. intention) and their consistency with real behavior are central parameters for operationalizing more behaviorally realistic agents for ABM. Comparing a decision-making outcome with actual behavior allows one to assess how well a particular decision-making model mirrors behavior. A further advantage of the combined quantification of decision-making and behavior for ABM agent operationalization is that simple decision heuristics (e.g. based on socio-demographic variables and behavior) can be implemented instead of the complex AHP decision-making process, whenever operational validation fails.

Conceptual validation

3.8

We have argued that "ensuring that the theories and assumptions underlying the conceptual model are correct" (i.e. conceptual validation) should be given more importance in the validation process of ABM, instead of concentrating on operational validation. The need for a "micro-level validation" (i.e. ensuring that micro-level behavior adequately represents actors' activity (Gilbert 2004)) in order to reproduce human-like behavior and thinking, is highlighted by Takadama et al. (2008). The agent operationalization approach contributes to that by providing a specific procedure with which to assess the conceptual validity of the models.

3.9

Each step of the agent operationalization approach - from the selection of the agents to the inclusion to their individual decision-making and behavior - draws upon local system knowledge, either qualitatively through expert interviews or quantitatively through surveys. This allows us to test the assumptions made in each step of the model development procedure leading to the conceptual model and, therefore, to ensure the validity of the conceptual model (Sargent 2008).

3.10

However, the approach was developed for a contextual, case-based model purpose. Validation may have different meanings for different model purposes (Küppers and Lenhard 2005) which is why different validation techniques and procedures exist (Louie and Carley 2008; Moss 2008). Even though in our approach the focus is on conceptual validation, we acknowledge the importance of verification (e.g. computerized model validation) and operational validation for ABM development and validation (Louie and Carley 2008; Sargent 2008; Takadama et al. 2008). Louie and Carley (2008) have proposed a framework for how models ought to be validated based on their purpose. However, how to exactly balance verification, conceptual and operational validation depending on the model purpose is still an open question.

Conclusion

4.1

This paper presented an agent operationalization approach, with the aim of providing a comprehensive framework to operationalize key system agents, their interaction, decision-making and behavior for ABM, exemplified by means of the Swiss mineral construction material case study.

4.2

The approach addresses three major concerns limiting ABMs' full potential:

1. Going beyond a "proof of concept": The approach gives a specific strategy for embedding empirical knowledge into modeling practices. It provides a step-by-step procedure for identifying the relevant agents to be included in the ABM and for analyzing their interaction chain in participatory approaches (e.g. expert interviews and workshops), thus enhancing the credibility of models implemented consequently.
2. Behaviorally realistic agents: The approach provides an array of sample agents with realistic (i.e. empirically quantified) decision-making and behavior, reducing the parameter space to scan. Quantifying agents' decisions with AHP provides not only a set of directly implementable decision-making data but also an opportunity to test decision-making assumptions empirically. In addition, checking the consistency of the decision-making outcome with behavior allows one to further validate/falsify the implemented decision-making theory.
3. Conceptual validity: The approach enhances the importance of conceptual model validity by providing a way to empirically test one's theoretical assumptions.

4.3

The comprehensive framework embedded in social process theory and decision making theory leads to more behavioral realistic agents and increases the conceptual model validity. The credibility of ABM is increased by the use of participatory processes. The example of the Swiss construction material case has demonstrated the practicability of the approach. The approach thus provides a transparent and well founded procedure applicable to a broad field of socio-ecological and socio-technical system modeling problems with ABM to the degree possible within the limits of the constituent theory and method. Further research should deal with, highlighting the added value of the approach by modeling the agents' interaction and adapting the approach for more generalizable ABM applications and cases with more informal social interaction and less cognizant decisions.

Appendix 1: Questionnaire AHP elicitation protocol

Table 1: AHP elicitation protocol in the questionnaire exemplified by the structural engineers' design specification decision

Acknowledgements

The authors thank all the experts for their support, time and knowledge; the three anonymous reviewers, Giuseppe Feola and Bernhard Steubing for their valuable comments; Thomas Ruddy and Andrea Delduce for editing the text; and FEDRO (Swiss Federal Roads Office), FOEN (Federal Office for the Environment), AWEL (Amt für Abfall, Wasser, Energie und Luft; Environmental Agency) of the Canton of Zurich, AHB (Amt für Hochbauten; Office for structural engineering) of the City of Zurich and Eberhard Recycling AG for funding the study as part of a cooperative project with the TSL (Technology and Society Laboratory) at EMPA and the SIE (Social and Industrial Ecology) group at University of Zurich.

Notes

¹ A detailed discussion of the ongoing controversy about verification and validation of simulation models in general is outside the scope of this paper. For further information we refer to e.g. Küppers and Lenhard (2005); Oreskes et al. (1994); Rykiel (1996); and Sargent (2008).

References

AJZEN, I. (1991), 'The theory of planned behavior', Organizational Behavior and Human Decision Processes, 50 (2), 179-211. [doi:10.1016/0749-5978(91)90020-T]

AJZEN, I. and Fishbein, M. (1977), 'Attitude-behaviour relations - theoretical analysis and review of empirical research', Psychological Bulletin, 84 (5), 888-918. [doi:10.1037/0033-2909.84.5.888]

AJZEN, I. and Madden, T. J. (1986), 'Prediction of goal-directed behavior - attitudes, intentions, and perceived behavioral-control', Journal of Experimental Social Psychology, 22 (5), 453-74. [doi:10.1016/0022-1031(86)90045-4]

ANDREWS, Clinton J. (2001), 'Building a Micro Foundation for Industrial Ecology', Journal of Industrial Ecology, 4 (3), 35-51. [doi:10.1162/108819800300106375]

ARMITAGE, C. J. and Conner, M. (2001), 'Efficacy of the theory of planned behaviour: A meta-analytic review', British Journal of Social Psychology, 40, 471-99. [doi:10.1348/014466601164939]

ASAN, S. S. and Asan, U. (2007), 'Qualitative cross-impact analysis with time consideration', Technological Forecasting and Social Change, 74 (5), 627-44. [doi:10.1016/j.techfore.2006.05.011]

AXELROD, R. (1997), The complexity of cooperation: Agent-based models of competition and collaboration (Princeton: Princeton University Press).

AXTELL, Robert L., Andrews, Clinton J., and Small, Mitchell J. (2001), 'Agent-Based Modeling and Industrial Ecology', Journal of Industrial Ecology, 5 (4), 10-13. [doi:10.1162/10881980160084006]

BARRETEAU, O. (2003), 'Our companion modelling approach', Journal of Artificial Societies and Social Simulation, 6 (2) 1 https://www.jasss.org/6/2/1.html.

BELTON, V. and Stewart, T. S. (2002), Multi criteria decision analysis. An integrated approach (Boston: Kluwer Academic Press). [doi:10.1007/978-1-4615-1495-4]

BERGER, T., et al. (2007), 'Capturing the complexity of water uses and water users within a multi-agent framework', Water Resources Management, 21 (1), 129-48. [doi:10.1007/s11269-006-9045-z]

BERGSDAL, H., et al. (2007), 'Dynamic material flow analysis for Norway's dwelling stock', Building Research and Information, 35 (5), 557-70. [doi:10.1080/09613210701287588]

BINDER, C.R. (2007a), 'From material flow analysis to material flow management Part II: The role of structural agent analysis', Journal of Cleaner Production, 15, 1605-17. [doi:10.1016/j.jclepro.2006.08.017]

BINDER, C.R. (2007b), 'From material flow analysis to material flow management Part I: social sciences modeling approaches coupled to MFA', Journal of Cleaner Production, 15 (17), 1596-604. [doi:10.1016/j.jclepro.2006.08.006]

BOERO, Riccardo and Squazzoni, Flaminio (2005), 'Does Empirical Embeddedness Matter? Methodological Issues on Agent-Based Models for Analytical Social Science', Journal of Artificial Societies and Social Simulation, 8 (4) 6, https://www.jasss.org/8/4/6.html.

BOUSQUET, F. and Le Page, C. (2004), 'Multi-agent simulations and ecosystem management: a review', Ecological Modelling, 176, 313-32. [doi:10.1016/j.ecolmodel.2004.01.011]

BRUNNER, P.H (2004), 'Material flow analysis and the ultimate sink', Journal of Industrial Ecology, 8 (3), 4-7. [doi:10.1162/1088198042442333]

CEDERMANN, Lars-Eric (2005), 'Computational models of social forms; advancing generative process theory', American Journal of Sociology, 110 (4), 864-93. [doi:10.1086/426412]

CORNWALL, A. and Jewkes, R. (1995), 'What is participatory research?', Social Science & Medicine, 41 (12), 1667-76. [doi:10.1016/0277-9536(95)00127-S]

COSTANZA, R., et al. (1993), 'Modeling complex ecological economic-systems - toward an evolutionary, dynamic understanding of people and nature', Bioscience, 43 (8), 545-55. [doi:10.2307/1311949]

DIEKMANN, A. (2007), Empirische Sozialforschung; Grundlagen, Methoden, Anwendungen [Empirical social research. foundations, methods, applications] (17; Reinbek: Rowohlt ).

EPSTEIN, Joshua M. and Axtell, Robert (1996), Growing artificial societies: social science from the bottom up (Washington, D.C: Brookings Institution Press).

FAUST, K. (1997), 'Centrality in affiliation networks', Social Networks, 19 (2), 157-91. [doi:10.1016/S0378-8733(96)00300-0]

FEOLA, G. and Binder, C. (2009), 'The integrative agent-centered (IAC) framework as a conceptual tool to investigate transition processes in local agricultural systems.', First European Conference on Sustainability Transitions: Dynamics and Governance of Transitions to to Sustainability (Amsterdam, The Netherlands).

FORMAN, E. H. and Gass, S. I. (2001), 'The analytic hierarchy process - an exposition', Operations Research, 49 (4), 469-86. [doi:10.1287/opre.49.4.469.11231]

GIDDENS, A. (1984), The constitution of the society (Berkeley, CA: University of California Press).

GILBERT, N. (2004), 'Open problems in using agent-based models in industrial and labor dynamics', in Roberto Leombruni and M. Richardi (eds.), Industry and Labor Dynamics: the agent-based computational approach (World Scientific), 401-05. [doi:10.1142/9789812702258_0019]

GILBERT, N. and Troitzsch, K. G. (2005), Simulation for the social scientist. (2 edn.; Berkshire, England: Open University Press, McGraw-Hill).

GODET, M. (1994), From Anticipation to Action: A Handbook of Strategic Prospective (Paris: UNESCO).

GORDON, T. J. and Hayward, H. (1968), 'Initial experiments with cross impact matrix method of forecasting ', Futures, 1 (2), 100-16. [doi:10.1016/S0016-3287(68)80003-5]

GÖTZE, U. (1991), Szenario-Technik in der strategischen Unternehmungsplanung [Scenorio-method in strategic corporate planning] (Wiesbaden: Deutscher Universitätsverlag). [doi:10.1007/978-3-663-06736-8]

GRANOVETTER, M. (1985), 'ECONOMIC-ACTION AND SOCIAL-STRUCTURE - THE PROBLEM OF EMBEDDEDNESS', American Journal of Sociology, 91 (3), 481-510. [doi:10.1086/228311]

GRIMM, V., et al. (2005), 'Pattern-oriented modeling of agent-based complex systems: Lessons from ecology', Science, 310 (5750), 987-91. [doi:10.1126/science.1116681]

HAO, J.L., Hills, M.J., and Huang, T. (2007), 'A simulation model using system dynamic method for construction and demolition waste management in Hong Kong', Construction Innovation, 7 (1), 7-21. [doi:10.1108/14714170710721269]

HASHIMOTO, Seiji , Tanikawa, Hiroki , and Moriguchi, Yuichi (2007), 'Where will large amounts of materials accumulated within the economy go? - A material flow analysis of construction minerals for Japan', Waste Management, 27, 1725-38. [doi:10.1016/j.wasman.2006.10.009]

HERMANS, L. M. (2005), 'Dynamic actor network analysis for diffuse pollution in the province of North-Holland', Water Science and Technology, 49 (2), 205-12.

HIRSCH Hadorn, G. , Wölfling Kast, S., and Maier, S. (2002), 'Restrictions and options: A heuristic tool to integrate knowledge for strategies towards a sustainable development.', International Journal of Sustainable Development and World Ecology, 18 (9), 193-207. [doi:10.1080/13504500209470116]

HOFFMANN, Cathleen (2004), 'Materialkenngrö§en von Beton aus Mischabbruch [Material properties of recycling concrete with mixed rubble aggregates]', (Zürich: Amt für Hochbauten der Stadt Zürich, Material Science and Technology (EMPA)), 37.

HOFFMANN, Cathleen and Leemann, Andreas (2006), 'Recycled concrete and mixed rubble as aggregates: Influence of variations in composition on the concrete properties', Advances in Cement and Concrete X: Sustainability (Davos, Switzerland), 211-15.

JAGER, W. and Janssen, M. (2002), 'The need for and development of behaviourally realistic agents', in J. S. Sichman, F. Bousquet, and P. Davidsson (eds.), 3rd International Workshop on Multi-Agent-Based Simulation II (MABS 2002) (Bologna, Italy: Springer-Verlag Berlin), 36-49.

JANSSEN, M. A. (2002), Complexity and ecosystem management: the theory and practice of multi-agent systems (Cheltenham: Elgar).

JANSSEN, M. A. and Ostrom, E. (2006), 'Empirically based, agent-based models', Ecology and Society, 11 (2) http://www.ecologyandsociety.org/vol11/iss2/art37/.

JIA, J., Fischer, G. W., and Dyer, J. S. (1998), 'Attribute weighting methods and decision quality in the presence of response error: A simulation study', Journal of Behavioral Decision Making, 11 (2), 85-105. [doi:10.1002/(SICI)1099-0771(199806)11:2<85::AID-BDM282>3.0.CO;2-K]

JOHNSON, E. J., Payne, J. W., and Bettman, J. R. (1988), 'Information displays and preference reversals', Organizational Behavior and Human Decision Processes, 42 (1), 1-21. [doi:10.1016/0749-5978(88)90017-9]

JUNGERMANN, Helmut , Pfister, Hans-Rüdiger , and Fischer, Katrin (1998), Die Psychologie der Entscheidung: eine Einführung [Decision psychology - an introduction] (Spektrum Lehrbuch; Heidelberg: Spektrum Akademischer Verlag).

KEENEY, R. L. (1982), 'Decision-analysis - an overview', Operations Research, 30 (5), 803-38. [doi:10.1287/opre.30.5.803]

KNOERI, C. (2007), 'Agent based modeling (ABM) for analyzing demand for recycled mineral construction material ', Diploma Thesis (Swiss Federal Institute of Technology ETH).

KNOERI, C., Binder, C. R., and Althaus, H.J. (2010), 'Decisions on recycling: construction stakeholders' decisions regarding recycled mineral construction materials', Resources, Conservation and Recycling.

KNOERI, C., Binder, C. R., and Althaus, H.J. (2011), 'Decisions on recycling: construction stakeholders' decisions regarding recycled mineral construction materials', submitted to Resources, Conservation and Recycling.

KÜPPERS, G. and Lenhard, J. (2005), 'Validation of Simulation: Pattern in the social and Natural Science', Journal of Artificial Societies and Social Simulation, 8 (4) 3 https://www.jasss.org/8/4/3.html.

LANG, D. J., et al. (2006), 'Impact factors and regulatory mechanisms for material flow management: Integrating stakeholder and scientific perspectives - The case of bio-waste delivery', Resources Conservation and Recycling, 47 (2), 101-32. [doi:10.1016/j.resconrec.2005.08.008]

LING, Y. Y. (2002), 'A conceptual model for selection of architects by project managers in Singapore', International Journal of Project Management, 21 (2003), 135-44. [doi:10.1016/S0263-7863(02)00014-5]

LOUIE, M. A. and Carley, K. M. (2008), 'Balancing the criticisms: Validating multi-agent models of social systems', Simulation Modelling Practice and Theory, 16 (2), 242-56. [doi:10.1016/j.simpat.2007.11.011]

MACY, M. W. and Willer, R. (2002), 'From factors to actors: Computational sociology and agent-based modeling', Annual Review of Sociology, 28, 143-66. [doi:10.1146/annurev.soc.28.110601.141117]

MAIER Bergé, S. and Hirsch Hadorn, G. (2002), 'Options & Restrictions: A heuristic tool in transdiciplinary research for an effective implementation of sustainable practices', in F. Biermann, S. Campe, and Jacob K. (eds.), 2002 Berlin Conference on the Human Dimensions of Global Environmental Change ãKnowledge for the Sustainability Transition. The Challenge for Social Science" Global Governance Project (19; Amsterdam, Berlin, Potsdam (Germany) and Oldenburg (Germany) ), 322-36.

MATTHEWS, R. B., et al. (2007), 'Agent-based land-use models: a review of applications', Landscape Ecology, 22 (10), 1447-59. [doi:10.1007/s10980-007-9135-1]

MENDOZA, G. A. and Martins, H. (2006), 'Multi-criteria decision analysis in natural resource management: A critical review of methods and new modelling paradigms', Forest Ecology and Management, 230 (1-3), 1-22. [doi:10.1016/j.foreco.2006.03.023]

MIEG, HA. (2000), 'University-based projects for local sustainable development: Designing expert roles and collective reasoning', International Journal of Sustainability in Higher Education 1, 67-82. [doi:10.1108/1467630010307110]

MIEG, Harald A. and Näf, Matthias (2006), Experteninterviews in den Umwelt- und Planungswissenschaften [Expert interviews in environmental and management research] (Lengerich: Pabst Science Publishers).

MINTZBERG, H., Raisinghani, D., and Theoret, A. (1976), 'Structure of unstructured decision-processes', Administrative Science Quarterly, 21 (2), 246-75. [doi:10.2307/2392045]

MITROFF, II and Featheringham, T. R. (1974), 'Systemic Problem-Solving and Error of 3rd Kind', Behavioral Science, 19 (6), 383-93. [doi:10.1002/bs.3830190605]

MOSER, K., et al. (2004), 'Baustoffmanagement 21 an der EMPA, Stand des Wissens und Forschungsbedarf [Management of construction materials at EMPA: standard of knowledge and need for further research]', (Material Science and Technology (EMPA)).

MOSLER, H.-J. and Tobias, R. (2005), 'Simulation of collective action based on empirical data', 2nd International Conference of the European Social Simulation Association (Valladolid, Spain).

MOSLER, H. J., et al. (2001), 'Computer simulation as a method of further developing a theory: Simulating the elaboration likelihood model', Personality and Social Psychology Review, 5 (3), 201-15. [doi:10.1207/S15327957PSPR0503_2]

MOSS, S. (2008), 'Alternative Approaches to the Empirical Validation of Agent-Based Models', Journal of Artificial Societies and Social Simulation, 11 (1) 5 https://www.jasss.org/11/1/5.html.

MULLER, D. B. (2006), 'Stock dynamics for forecasting material flows - Case study for housing in The Netherlands', Ecological Economics, 59 (1), 142-56. [doi:10.1016/j.ecolecon.2005.09.025]

NIKOLIC, Igor (2009), Co-evolutionary method for modelling large scale socio-technical systems evolution (Delft, The Netherlands: Next Generation Infrastructures Foundation).

ORESKES, Naomi, Shrader-Frechette, Kristin, and Belitz, Kenneth (1994), 'Verification, validation and confirmation of numerical models in the earth sciences', Science, 263 (5147), 641-46. [doi:10.1126/science.263.5147.641]

PAHL-WOSTL, C. (2007), 'The implications of complexity for integrated resources management', Environmental Modelling & Software, 22 (5), 561-69. [doi:10.1016/j.envsoft.2005.12.024]

PARKER, C., et al. (2003), 'Multi-agent systems for the simulation of land-use and land-cover change; a review', Annals of the Association of American Geographers, 93 (2), 314-37. [doi:10.1111/1467-8306.9302004]

REYNOLDS, C. W. (1987), 'Flocks, herds, and schools: a distributed behavioral model', Computer Graphics (SIGGRAPH '87 Conference Proceedings), 21 (4), 25-34. [doi:10.1145/37402.37406]

RUSBULT, C. E. and Van Lange, P. A. M. (2003), 'Interdependence, interaction, and relationships', Annual Review of Psychology, 54, 351-75. [doi:10.1146/annurev.psych.54.101601.145059]

RYKIEL, E. J. (1996), 'Testing ecological models: The meaning of validation', Ecological Modelling, 90 (3), 229-44. [doi:10.1016/0304-3800(95)00152-2]

SAATY, T. L. (1980), The analytical hierarchy process: planning, priority setting, resource allocation (New York: McGraw-Hill).

SAATY, T. L. (1990), 'How to Make a Decision - the Analytic Hierarchy Process', European Journal of Operational Research, 48 (1), 9-26. [doi:10.1016/0377-2217(90)90057-I]

SAATY, T. L. (1994), Fundamentals of decision making (Pittsburgh, PA: RWS Publication).

SARGENT, R. G. (2008), 'Verification and validation of simulation models', 2008 Winter Simulation Conference (Washington, DC), 112-25. [doi:10.1109/WSC.2008.4736065]

SCHLANGE, L. E. and Juttner, U. (1997), 'Helping managers to identify the key strategic issues', Long Range Planning, 30 (5), 777-86. [doi:10.1016/S0024-6301(97)00054-X]

SCHOLZ, R.W. and Tietje, O. (2002), Embedded case study methods; Integrating quantitative and qualitative knowledge (Thousand Oaks: Sage Publications).

SCHUTTE, Sebastian (2010), 'Optimization and Falsification in Empirical Agent-Based Models', Journal of Artificial Societies and Social Simulation, 13 (1) 2 https://www.jasss.org/13/1/2.html.

SCOTT, John and Marshall, Gordon (2010), '"operational definition."', Oxford Dictionary of Sociology (updated 2006) <http://www.enotes.com/oxsoc-encyclopedia/operational-definition>, accessed 5 May.

SPOERRI, Andy, et al. (2009), 'Expert-based scenarios for strategic waste and resource management planning - C&D waste recycling in the Canton of Zurich, Switzerland', Resources, Conservation and Recycling, 53 (10), 592-600. [doi:10.1016/j.resconrec.2009.04.011]

SUSSKIND, L., McKearnnan, S., and Thomas-Larmer, J. (1999), The consensus building handbook - a comprehensive guide to reach agreement (Thousand Oaks, CA: Sage).

SVENSON, O. (1990), Some Propositions for the Classification of Decision Situations, eds K. Borcherding, O. I. Larichev, and D. M. Messick (Contemporary Issues in Decision Making; Amsterdam: Elsevier Science Publ B V) 17-31.

SVENSON, O. (1996), 'Decision making and the search for fundamental psychological regularities: What can be learned from a process perspective?', Organizational Behavior and Human Decision Processes, 65 (3), 252-67. [doi:10.1006/obhd.1996.0026]

TAKADAMA, K. , Kawai, T , and Koyama, Y (2008), 'Micro- and Macro-Level Validation in Agent-Based Simulation: Reproduction of Human-Like Behaviors and Thinking in a Sequential Bargaining Game', Journal of Artificial Societies and Social Simulation, 11 (2) 9 https://www.jasss.org/11/2/9.html.

TAM, V. W. and Tam, C. M. (2006), 'A review on the viable technology for construction waste recycling', Resources Conservation and Recycling, 47 (3), 209-21. [doi:10.1016/j.resconrec.2005.12.002]

TESFATSION, L. and Judd, K. L. (2006), Handbook of computational economics: agent-based computational economics (Handbooks in economics, 2; Amsterdam: Elsevier).

TIETJE, O. (2005), 'Identification of a small reliable and efficient set of consistent scenarios', European Journal of Operational Research, 162 (2), 418-32. [doi:10.1016/j.ejor.2003.08.054]

TRIANDIS, H.C. (1980), Values, Attitudes and Interpersonal Behaviour, Nebraska Symposium on Motivation (Lincoln/London: University of Nebraska Press).

UZZI, B. (1997), 'Social structure and competition in interfirm networks: The paradox of embeddedness', Administrative Science Quarterly, 42 (1), 35-67. [doi:10.2307/2393808]

VESTER, F. (2007), The art of interconnected thinking: Tools and concepts for a new approach to tackling complexity (Munich, Germany: MCB).

VOINOV, Alexey and Bousquet, Francois (2010), 'Modelling with stakeholders', Environmental Modelling & Software, 25 (11), 1268-81. [doi:10.1016/j.envsoft.2010.03.007]

VON Reibnitz, U. (1992), Szenario-Technik. Instrumente für die unternehmerische und persönliche Erfolgsplanung [Scenario-methods; Instruments for corporate and personal planning] (Wiesbaden, Germany: Gabler).

WANG, J. Y., et al. (2004), 'A systems analysis tool for construction and demolition wastes management', Waste Management, 24 (10), 989-97. [doi:10.1016/j.wasman.2004.07.010]

WASSERMAN, S. and Faust, K. (1994), Social Network Analysis: Methods and Applications (Cambridge Univ. Press). [doi:10.1017/CBO9780511815478]

WINDRUM, P., Fagiolo, G., and Moneta, A. (2007), 'Empirical Validation of Agent-Based Models: Alternatives and Prospects', Journal of Artificial Societies and Social Simulation, 10 (2) 8 <https://www.jasss.org/10/2/8.html>.