Abstract
- Agent-based models are more likely to generate accurate
outputs if they
incorporate valid representations of human agents than if they don't.
The present article outlines three research methodologies commonly used
for explicating the cognitive processes and motivational orientations
of human judgment and decision making: policy capturing,
information seeking, and social choice.
Examples are given to demonstrate how each methodology might be
employed to supplement more traditional qualitative methods such as
interviews and content analyses. Suggestions for encoding results of
the three methodologies in agent-based models are also given, as are
caveats about methodological practicalities.
- Keywords:
- Research Methodology, Cognition, Motivation, Judgement, Decision Making
Introduction
- 1.1
- Most agent-based models
include programming code to simulate how agents make judgments about
their situation and decisions about how to act on their judgments. It
seems reasonable to assume that the validity of agent-based models
depends in part on the realism of this code; code that closely mimics
how people make judgments and decisions is likely to produce more
realistic outputs than code that does not. This prompts two important
questions: What methodologies exist for explicating people's judgment
and decision processes? How can we employ these methodologies to
improve the realism of agent-based models? The present article
addresses these questions.
- 1.2
- Literature on
agent-based models rarely includes references to research on the
processes people employ to make judgments or decisions. Indeed, the
choice of a judgment or decision algorithm often seems rather casual,
relying more on intuition, tradition or programming ease than on
empirical research about people's cognitive and motivational processes.
This has two drawbacks. First, the algorithms are likely to be wrong.
Second, the simulations are likely to ignore the consequences human
variability in the judgments and decision processes people employ. Both
errors can generate simulation outputs that wander far from what really
happens in the model's domain. Economic simulations based on
assumptions of economically rational agents, for example, frequently
produce outputs that wander far from relevant economic observations (Thorngate & Tavakoli 2005).
- 1.3
- How do people make
judgments and decisions? Sixty years of research and thousands of
studies in psychology, political science, marketing, behavioural
economics and elsewhere have given us increasingly detailed answers to
the question. Among the details is an important conclusion: People make
judgments and decisions in hundreds of different ways, and these ways
frequently lead to different choices (for reviews, see Gigerenzer & Gaissmaier 2011;
Griffin et al. 2012).
Most judgment and decision processes incorporate mental short-cuts
called heuristics, some of which have been documented in laboratory
studies under labels such as satisficing, recognition,
elimination by aspects, temporal
discounting, representativeness, anchoring
and adjustment.
Dozens of additional heuristics await labels and laboratory scrutiny,
but they can still be seen in daily life. Here is a small sample:
- Choose what was successful in previous, similar situations;
- Ask for advice; follow the advice of the most convincing advisor;
- Watch what others are choosing; if they seem satisfied, mimic their choice;
- Wait; postpone decision in hopes of better, future alternatives;
- Flip a coin;
- Delegate the decision to someone else; if outcome is good, reward yourself; if outcome is bad, blame others;
- Vote;
- Do what tradition, the law or moral codes pre/proscribe;
- Pray for divine guidance;
- Choose the first alternative that meets minimal standards
- Lower your standards;
- Change your desires to those satisfied by current alternatives;
- Use a spreadsheet or mathematical formula.
- 1.4
- The choice of a decision process likely depends on dozens
of variables (Gigerenzer
& Gaissmaier 2011; Thorngate
1975).
Some of the variables are psychological, including a person's judged
importance of the decision, decision making habits and training, memory
and attention requirements, social tradition and pressure (see, for
example, Knox & Inkster 1969).
Other
variables are situational, including time available to make a decision,
the number of alternatives available, and opportunities for reversing a
decision made (see, for example, Chowdhury
& Thorngate 2013). Different decision processes can
often lead to the same decisions (Thorngate
1980),
but no one has yet found simple rules for determining when this will or
will not occur. It is thus prudent to assume that the decision
processes employed at any time will depend on salient features of the
decision maker and the situation.
- 1.5
- This assumption
suggests that agent-based models can be improved by learning from the
people to be modeled how they perceive the situation to be modeled and
how they convert their perceptions into a choice of actions. To do so
requires collecting three kinds of data: (1) data about which features
of the situation people judge to be salient; (2) data about the
alternative actions they consider; and (3) data about people's criteria
for judging these alternatives.
- 1.6
- Words seem to be
the best medium for capturing the first two kinds of data. As other
articles in this issue document, verbal descriptions are highly
efficient for conveying useful information to researchers trying to
recreate the situations decision makers face when making their choices.
Words, for example, can efficiently convey information about the
time-fame, timing and history of a decision, the features of
alternatives considered, the number and roles of people involved in a
decision, and the sequences of events culminating in a choice. Skillful
use of interview and questionnaire techniques usually extract enough
situational information to render an accurate synopsis for a simulation
to mimic in code.
- 1.7
- Alas, however,
people's abilities to verbalize their situation definition and choice
criteria are rarely equaled by their ability to verbalize what happens
in their head and heart when processing the information that leads to a
decision (see, for example, Kahneman
2011; Nisbett &
Wilson 1977).
Introspective and retrospective accounts of the choice process tend to
rationalize as much as to describe what occurred. Reasons range from
selective attention to bad memory to saving face. The resulting
accounts tend to be biased towards greater consistency and
sophistication than a person's choices reveal. Often they do not
reproduce the choice made (Hoffman
et al. 1968; Nisbett
& Wilson 1977).
- 1.8
- What else can be done beyond introspection to capture the cognitive and motivational processes (often called policies) of decision makers without asking them to explicate these processes in words? Below I describe three methods for inducing characteristics of these processes, not by analysing what decision makers say but by analysing what they do. The methods have their own limitations, discussed towards the end of this article. They offer, however, what I believe are good supplements to the analyses of verbal protocols, and give useful information for writing the lines of computer code representing the heads or hearts of agents making choices.
Policy capturing and paramorphic representation
- 2.1
- Over 50 years ago, Ken Hammond and his colleagues (for
example, see Brehmer &
Joyce 1988; Hammond 1955;
Hammond et al. 1964; Hammond & Summers 1972)
began to use multiple regression equations to represent Egon Brunswik's
ideas about how brains use information. Labeled the paramorphic
representation of judgment (Hoffman
1960), the equations
summarized how variations in features, called cues,
of stimuli correlated with judgments made. The strength of each
cue-judgment relationship was measured by its beta weight in the
multiple regression equation. The larger the beta weight, the more
important a cue was in determining the judgments.
- 2.2
- Dozens of
studies were soon conducted to assess which cues were important to
individual judges, often experts, when performing tasks associated with
their profession (e.g., see Dawes
& Corrigan 1974; Goldberg
1959, 1968; Hoffman et al. 1968; Slovic 1966, 1969).
In a typical study, research participants are first asked to write down
the names of features of situations they face in their profession.
Researchers then construct 10-100 hypothetical cases described by
varying the presence or absence, or the values, of the named features.
The descriptions are then shown to each participant who is asked to
assess each description on a relevant numerical scale. A multiple
regression or an analysis of variance is then performed on the
collected data to determine how much of each participant's judgment
variance can be accounted for by variations in each feature and
combination of features. The procedure is known as policy
capturing (Brehmer
& Joyce 1988).
- 2.3
- Hoffman, Slovic and Rorer's (1968)
study of nine radiologists judging the chances of malignant gastric
ulcers from x-ray reports nicely illustrates use of the
policy-capturing procedure. Interviews with the radiologists revealed
that all had been taught to look for seven signs of malignancy – signs
with names such as filling defect and small
crater.
Hoffman et al. constructed 96 tabular summaries of the presence (yes)
or absence (no) of these seven signs, each summary showing a different,
plausible yes-no combination. The radiologists independently examined
these 96 summaries twice, each time generating 96 ratings of the
chances of malignancy. The resulting fractionally-replicated, factorial
design allowed the Hoffman, Slovic, and Rorer (1968)
to conduct an Analysis of Variance on each radiologist's judgments. The
percentage of variance accounted for by each of the resulting main
effects and interactions served, like beta weights, as an indicator of
the salience of each sign and sign combination.
- 2.4
- Several surprising results emerged from Hoffman, Slovic and
Rorer's (1968)
study. Only two of the nine radiologists, for example, showed
statistical evidence of using all cues in making their judgments. Three
of the radiologists showed evidence of using only three cues, and one
radiologist used only two. Variations in which cues were used likely
accounted for the modest correlations among radiologists; the median
correlation of their malignancy assessments was r = +0.44. In short,
most of the radiologists did not reflect in their judgments what they
reported they did. The paramorphic representation of their judgment
process was more accurate than their verbal description of it.
- 2.5
- The policy
capturing procedure can be easily adapted to explicate a paramorphic
representation of judgment or decision processes employed by people
modeled in agent-based simulations. Suppose, for example, that we are
interested in creating an agent-based model examining the consequences
of people's decisions about using private versus public transit. Here
is one way to explicate features of the cognitive processes people
would use to make the decision.
- 2.6
- Step 1.
Suppose we interview a sample of, say, 30 citizens about which factors
they would consider if they had to choose between commuting to work by
car versus bus. Analyses of their qualitative responses reveals 13
features, three of which are mentioned by more than 20 of the citizens,
and three more of which are mentioned by at least ten. To make the
policy capturing procedure more manageable, we choose these six
most-popular features to construct a policy-capturing task. Suppose
these six features are as follows.
- weekly cost of driving and parking car
- car commuting time
- weekly cost of riding bus
- bus commuting time
- minutes of walking to bus stop
- chances of standing room only
- 2.7
- Step 2. We
then create a range of plausible values to describe variations of the
six factors. For example, we might vary weekly costs of driving an
parking from $60 to $120 in $20 increments, and chances of standing
room only from 20% to 100% in 20% increments. It is also possible to
employ words to express feature values. We might, for example, add a
seventh feature called bus cleanliness with values
such as "very clean," "moderately clean," "moderately dirty," and "very
dirty."
- 2.8
- Step 3.
Using a 10-to-1 examplar-to-feature rule of thumb, we next create a set
of 60 examplars of car-bus situations. Each situation is defined by a
random combination of values of the seven features. To reduce
confounds, each person is given a fresh, random set of 60 different
situations. Table 1 illustrates three such exemplars.
Table 1: Three examples of car-bus situations. Feature Situation A Situation B Situation C weekly cost of driving + parking $80 $120 $100 car commuting time 20 minutes 25 minutes 40 minutes weekly cost of riding bus $20 $25 $15 bus commuting time 30 minutes 30 minutes 20 minutes minutes walking to bus 2 5 5 chances of standing room only 80% 40% 60% - 2.9
- Step 4.
Once the 60 situations are created, we present them to a sample of
perhaps 20-50 participants of our choosing and ask them to judge each
situation, one at a time, on some meaningful scale(s). Selection and
wording of the scales is, as always, as much a matter of art as
science. For our purposes, let us use this one:
definitely take my car 0 1 2 3 4 5 6 7 8 9 definitely take the bus
Several computer programmes are available to generate the 60 random combinations of feature values defining the 60 situations, present these situations to research participants, and record their ratings. Examples include Excel®, LiveCode® and Fluid Survey®. - 2.10
- Step 5.
Once we gather 60 ratings of 60 situations from each participant, we
can undertake statistical analyses each participant's data to distill
her/his results into a paramorphic representation. There are several
ways to proceed. One way is to deploy multiple regression software to
estimate the best-fitting linear equation of each citizen's data,
extracting regressing weights and function shapes (linear? parabolic?)
for each of the seven variables and for whatever combination of
variables, such as car-expense minus bus-expense, seem interesting. If
the software requires some error variance for its computations, we can
generate it by copying a person's data, adding a bit of random error to
each judgment in the copy, and using the original and copy as a sample
size of two. Alternatively, we follow the lead of Hoffman, Slovic and
Rorer (1968) and ask
participants to complete the rating task twice.
- 2.11
- The statistical
analyses of each participant's ratings should generate regression
weights reflecting how important he/she believes each of the seven
features of the situation to be. We can expect these weights to vary
from person to person; one participant, for example, might vary her
car/bus ratings mostly according to driving costs and bus commuting
time, while another might vary his ratings according to the difference
between car and bus driving costs and the chances of standing room
only.
- 2.12
- Step 6. Once each person's policy has been captured, it can easily be implemented in an agent-based model; the best-fitting regression equation for each participant simply becomes a line of programming code. Outputs of the model related to the seven features of the situation (Table 1) at time t are given as inputs to the regression equations of people a researcher wishes to include in the model. The result is a probability of each person taking the bus or car at time t+1. From these probabilities, the expected number of people taking each form of transport can be calculated and fed into parts of the model related to costs, commuting times, crowding, etc. In short, rather than employing one policy equation with arbitrary weights of average or typical rules that agents might use to make transportation decisions, the model can employ several rules, grounded in empirical observations, to more accurately reflect the nature and variety of human judgment processes.
Information seeking and conditional processing
- 3.1
- The policy capturing procedure
described above is useful when a list of plausible criteria for making
judgments and decisions can be obtained, when plausible and realistic
alternatives can be constructed, and when people to be modeled have the
time and motivation to judge a large set of these alternatives. When
one or more of these conditions are not met, however, the policy
capturing procedure begins to lose its charm. Another means of
gathering data to explicate the judgment or decision processes might
then be employed, one that simply requires persons being modeled to ask
questions.
- 3.2
- One alternative
method is founded on a simple premise: When people must seek
information before making a judgment or decision, the information they
seek, and the order in which they seek it, reflects their cognitive
processes. Consider, for example, a modified game of 20 Questions in
which two voters are allowed to ask a researcher up to 20 yes/no
questions about three candidates for mayor in a local election before
voting for the one they prefer. Suppose the dialogue with Voter 1
proceeds as follows:
- Q1: Will Candidate A lower property taxes?
- A1: No.
- Q2: Will Candidate B lower property taxes?
- A2: Yes.
- Q3: Will Candidate C lower property taxes?
- A3: No.
- Decision: I'll vote for B.
- 3.3
- It appears that
Voter 1 considered only one campaign issue in making a choice: lowering
taxes, rejecting A and C from further consideration because of their
position on taxes, leaving B as the only acceptable candidate. In the
jargon of decision heuristics, Voter 1's trajectory of information
seeking is associated with a simple, non-compensatory,
elimination-by-aspects, lexicographic choice heuristic in which a
minimum standard (lower taxes) for selection is set and the first
candidate meeting this standard is chosen. One way of coding the rule
in an agent-based model would be as follows:
- If candidate does not promise lower taxes, then reject;
- else if candidate promises lower taxes, then continue;
- If no more candidates, chose one from the set of those not rejected.
- 3.4
- Now suppose a dialogue with Voter 2 proceeds as follows:
- Q1: Will Candidate C give more money for schools?
- A1: No.
- Q2: Will Candidate B give more money for schools?
- A2: Yes.
- Q3: Will Candidate B build more social housing?
- A3: No.
- Q4: Will Candidate A build more social housing?
- A4: Yes.
- Q5: Will Candidate A give more money for schools?
- A5: Yes.
- Decision: I'll vote for A
- 3.5
- In contrast to
Voter 1, Voter 2 seems to consider two campaign issues in making a
choice: schools and social housing, looking for a candidate who is
favourable to both. Voter 2's rule might be coded in an agent-based
model as follows:
- If candidate does not promise more money for schools, then reject;
- Else if candidate does not promise more money for social housing, then reject;
- Else accept.
- 3.6
- The 20 Questions
procedure illustrated above is only one of several information seeking
methods that might be useful for explicating people's decision rules
and coding them in simulation algorithms. Another method is eye
tracking, recording the trajectory of people's eye movements as they
scan information on a screen before making a choice (see, for example, Orquin & Mueller Loose 2013).
Eye tracking is sophisticated but rather expensive and tricky to employ
(see Duchowski 2007).
- 3.7
- For researchers on
a budget, an inexpensive alternative to eye tracking is the information
board: a matrix of alternatives in rows and features in columns with
information available about each alternative-feature pair by clicking,
flipping or otherwise opening its matrix cell (for example, see Bettman & Kakkar 1977;
Hofacker 1984; Payne 1976; Thorngate 1974). Table 2 illustrates a typical information
board as it might appear on a computer screen.
Table 2: A partially-examined information board [order of examination in square brackets] Alternatives
(candidates)Features (issue) Property taxes Schools Public housing Pollution A [6] increase taxes by 10% [1] hire teachers [7] double amount not a problem B [5] lower taxes by 30% [2] more homework click for quote click for quote C click for quote [3] more computers click for quote click for quote D click for quote [4] no change needed click for quote more bike paths - 3.8
- Records of the
cell-by-cell examination order and the decision made provide useful
information for deducing major aspects of a person's decision process.
In one early experiment (Thorngate
& Maki 1976),
for example, students sat opposite the researcher who arranged,
face-down, a matrix of 3x5 cards, each card containing on its face side
a quote from 2, 4 or 8 candidates for city council about 2, 4 or 8
local issues chosen by a city newspaper. Research participants were
asked to choose their most preferred candidate by flipping over as many
cards as they wanted to read, in any order, until they decided on their
preferred candidate. Results showed a highly popular tendency for
participants first to examine all candidates on a favoured campaign
issue (such as Schools in Table 2),
then examine a subset of these candidates on a second issue (such as
taxes in Table 2),
examine a further subset on a third issue, etc. until only 2-3
finalists remained. One of these finalists was then chosen by comparing
their quotes about a remaining issue, often an issue of minor
importance.
- 3.9
- A few caveats
about using an information-seeking method are warranted. To be useful,
methods such as the information board require the set of features
describing alternatives to be relevant to most people whose decision
processes will be simulated. So, as with the policy capturing method,
it is prudent to interview people in advance about which features are
relevant to their judgments. Expect that 10-30% of participants will
sometimes be inconsistent in their selection of information, diverting
a predictable trajectory out of curiosity, boredom, or whimsy. As a
result, it is advisable when possible to ask each participant to engage
in more than one choice situation so the most common trajectories can
be detected. In the case of choosing candidates for political office
(Table 2), for example, it
would be worthwhile
for a participant to seek information about Candidates A-D, make a
choice among these four, then seek information about Candidates E-H,
make a choice among these new four, then seek information about
Candidates I-L, and so forth. Modal trajectories of this participant
could then be calculated to represent her/his decision process.
- 3.10
- The judgment and decision processes people employ are known to vary according to the amount and layout of information available prior to a decision (Thorngate & Maki 1976). Participants with inchoate decision processes are likely to alter their process depending on how many alternatives are available and how many features describe these alternatives, in part to meet constraints of their attention and working memory. The number of alternatives and features should thus be selected to approximate the situations represented in an agent-based model. Decision processes are also influenced by the order in which the features are displayed. There is, for example, a tendency for people to examine information presented at the top-left corner of information boards more often than the information presented at the bottom-right. Such tendencies can be mitigated by shuffling matrix rows and columns for each presentation of an information board.
Social choice and motivational inference
- 4.1
- Just as it is desirable to
employ accurate representations of people's judgment and decision
processes in the code of agent-based simulations, it is also desirable
to render their motives for making these judgments and decisions. Most
economists and some cynics believe that people are self-centred or individualistic,
motivated to maximize their own gain. However, social psychologists
long ago demonstrated that people frequently choose to pursue other
motives (see Grzelak 1982;
Messick & McClintock
1968). High on the list of alternatives are the motives of competition
(maximizing relative gain), cooperation (maximizing
group gain) altruism (maximizing other's gain) and aggression
(minimizing others gain).
- 4.2
- These five motives
are often seen as points along two continua defined by the weights a
person gives to (1) her/his own gain and (2) the gain of one or more
others (see, for example, Griesinger
& Livingstone 1973; MacCrimmon
& Messick 1976).
To illustrate, suppose Mary is asked to make a choice between two pay
packages, A and B, each giving herself and John different amounts of
money. Choosing A would give Mary $4 and John $3. Choosing B would give
Mary $5 and John $6. To make her choice, Mary assigns an overall value
to A and an overall value to B, then chooses the one with the higher
value. The values depend on how much Mary weighs her own outcome and
John's outcome, and can be rendered in the following altorithm:
- value(A) (Wown x $Aown) + (Wother x $Aother);
- value(B) (Wown x $Bown) + (Wother x $Bother);
- if value(A) > value(B) then choose A;
- if value(B) > value(A) then choose B;
- if value(A) = value(B) then choose A with probability = 0.5.
- 4.3
- Different values of Wown and Wother define different
motives, for example:
- individualism: Wown = 1.0, Wother = 0.0.
- competition: Wown = 1.0, Wother = -1.0.
- cooperation: Wown = 1.0, Wother = 1.0.
- altruism: Wown = 0.0, Wother = 1.0.
- aggression: Wown = 0.0, Wother = -1.0.
- value(A) = (1.0 x $4) + (-1.0 x $3) = $4 - $3 = +1.0
- value(B) = (1.0 x $5) + (-1.0 x $6) = $5 - $6 = -1.0
- 4.4
- Other values of
Wown and Wother can be used to represent combinations of motives. A
combination of individualism and cooperation, for example, can be
represented as
Wown = +1.0 and Wother = +0.5. - 4.5
- Different
motives can lead to different choices. Table 3 shows a set of five
situations, each requiring Mary to choose between pay packages A and B.
Below each situation are dots (•) indicating which alternative would be
preferred given the five possible motives listed above.
Table 3: Different motives lead to different choice patterns. Situation# 1 1 2 2 3 3 4 4 5 5 If Mary chooses: A B A B A B A B A B Mary gets $6 $5 $4 $3 $4 $0 $5 $4 $1 $7 John gets $2 $5 $5 $0 $3 $5 $9 $2 $0 $2 Motive and choice Individualism • • • • • Competition • • • • • Cooperation • • • • • Altruism • • • • • Aggression • • • • - 4.6
- In addition to
deducing choices from motivational orientations, we can also induce the
motivational orientations from the choices made. One of the easiest
induction procedures employs a simple choice task and logistic
regression. We begin by presenting a person with dozens of situations,
such as the five shown in Table 3,
varying the
four payoffs randomly and independently. We then ask the person to make
choices in these situations, recording for each situation the four
payoffs and choice made. Finally, we calculate a logistic regression of
own gain and other's gain on the choices made to estimate the odds
ratios for own gain and for others gain. The logistic regression
equation becomes a paramorphic representation of a person's
motivational orientation; the odds ratios become the best estimates of
Wown and Wother.
- 4.7
- Though Table 3
shows situations with two people, the situations need not be restricted
only to two. Situations can have two, three, or more others affected by
a person's choices, each other represented as a predictor variable in
the logistic regression equation. As the number of others increases, so
too should the number of situations shown to a person in order to
obtain stable estimates of his/her beta weights. It is traditional to
ask a research participant to make choices in 50-100 two-person
situations. The task goes quickly; it normally takes no more than 30
minutes for a person to make choices in 100 situations. Once a logistic
regression equation capturing a person's motivational orientation is in
hand, it is straightforward to insert the equation as code for a
relevant part of an agent based model.
- 4.8
- As with
cognitive processes, it is safe to assume that motivational
orientations will vary from person to person, context to context, time
to time (see, for example, Brewer
& Kramer 1986; Messick
& Brewer 1983; van
Lange et al. 2007).
For example, people are more likely to be motivated to reduce the
difference between their own gain and other's gain when they receive
less than the other than when they receive more, demonstrating more
competition or aggression when "losing" (McClintock
et al. 1973).
Motivations are also likely to be influenced by a chooser's
relationship with the others. It is reasonable to assume, for example,
that people are more altruistic in situations where their choices
affect their children than in situations where their choices affect
their enemies. Including such motivational variations in the code of
agent-based models is likely to improve their accuracy.
- 4.9
- The policy-capturing and motive-assessment methodologies discussed above both employ statistical analyses of stimulus-response pairs to generate paramorphic representations of underlying cognitive and motivational processes. The information-seeking methodology can be adapted to explicate motivational processes as well. Research participants could, for example, be required to ask for payoff information before making choices in situation such as those shown in Table 3. A participant who repeatedly asked only "How much for me in Alternative A? How much for me in Alternative B?" then chose the higher of the two numbers might reasonably be assumed to be individualistic. Another participant who repeatedly asked only "How much would the other receive if I chose A? If I chose B?" and always chose the higher payoff, might reasonably be assumed to be altruistic. Such an information seeking methodology might explicate motives faster than one requiring choices across dozens of situations. The methodology, however, remains unproven.
Examples
- 5.1
- Suppose we are interested in
the consequences of employing new rules for choosing winners of a
research funding competition. The new rules prescribe that each funding
application must be assessed according to three criteria: (1) the merit
of its research ideas; (2) the merit of its research methodology; and
(3) the practical implications of its findings. The new rules also
prescribe that applications should be adjudicated by a committee of
three, and that final decisions should be reached by consensus.
Procedures for accomplishing these goals are left to the committee. How
might different procedures affect the outcomes of the adjudications?
- 5.2
- One sensible way
to construct an answer is to determine the cognitive rules members of
the committee are likely to use in assessing and combining the three
merit criteria. Assume the committee members are known, and they agree
to participate in relevant research about their own judgment and
decision processes. We may then construct a task using either or both
of the policy-capturing or information-seeking methods above to
estimate how each of the committee members maps the words of funding
proposals into assessments of merit. We might, for example, find or
construct dozens of short proposals that vary in their merit on each of
the three criteria, ask each committee member to give each proposal an
overall merit rating, then conduct regression analyses to determine the
beta weights each member gives each criterion in her/his assessment.
- 5.3
- Alternatively,
we might ask each committee member to examine a set of, say, 20
proposals and observe how he/she accomplishes the task. Does the member
always begin by reading the practical implications paragraphs,
continuing to the method paragraphs only if the practical implications
are judged good or better? Does the member stop reading the practical
implications paragraphs after proposal 3 and concentrate only on the
ideas paragraphs, commenting that the proposals' practical implications
are all speculative? Such information seeking trajectories and comments
can help explicate the committee member's judgment process.
- 5.4
- Suppose that,
after the three committee members undergo our research tasks, we
distill from their judgments the following representations of their
cognitive rules.
- Member A starts each proposal assessment by reading the practical implications paragraphs of the proposal. If A judges the practical implications to be lower than 4 on a 6-point scale (0 = horrible to 5 = wonderful), then A stops reading and rejects the proposal; otherwise, A reads rates the ideas section and the method section on the same 6-point scale, and averages the two ratings for an overall assessment. After all proposals are thus assessed, A rank orders the average ratings to bring to the committee.
- Member B first reads the ideas section of each proposal and eliminates the proposals with ideas she rates lower than 3 on the 6-point. She then reads the method section of each remaining proposal and eliminates all those with unfathomable methods. Finally, she ranks the practical implications sections of her finalists.
- Member C creates a spreadsheet with one row for each proposal and three columns for the three criteria She systematically reads each proposal and assigns a rating on the 6-point scale indicating her assessment. After rating all the applications on the three scales, member C assigns weights of 0.5 to ideas, 0.3 to methods, and 0.2 to practical implications, then calculates the weighted average of the ratings for each proposal. Finally, she rank orders these averages.
- 5.5
- We are now able to
begin writing a simulation incorporating algorithms that represent the
cognitive processes of the three committee members. Translation from
judgment processes to algorithms is usually straightforward. For
example, A's judgment process could be captured thusly,
- If practical implications rating < 4, then
reject; else
º Rating = (ideas rating + method rating) / 2.
- Judgment = 0.5*ideas + 0.3*methods + 0.2*practical
- If practical implications rating < 4, then
reject; else
- 5.6
- Thus constructed,
the simulation could be used to estimate how much agreement we might
expect among the three committee members when assessing any
hypothetical set of applications. Hundreds of hypothetical applications
– each described by a vector of three numbers representing ratings of
ideas, method, and practical implications of one or all committee
members – could serve as input to the simulation. Analysis of
variations in the outputs of committee members' judgments would
indicate how susceptible the assessments of applications are to
variations in the committee members' judgment processes. If these
processes generated a high proportion of disagreements, adjudication
administrators might then seek either to change members of the
committee or train its members to use a common judgment process.
- 5.7
- Short of training
or replacing committee members to increase their inter-judge
consistency, can anything else be done? Committee discussion remains an
obvious alternative; members meet in a spirit of cooperation, talk
about their differences, and modify their assessments in light of each
other's arguments. Consensus follows.
- 5.8
- Yet, what if one
or more committee members lacks a spirit of cooperation? How might
other motives influence the outcome of committee deliberations? We
might try to guesstimate an answer by assessing each committee member's
social motive using the motivational inference technique previously
presented. I am unaware of any research linking social motivations
inferred by this technique to the behavior of adjudication committee
members. Still, for illustrative purposes it is reasonable to speculate
that cooperators are more likely than competitors to reach consensus
quickly because, by definition, competitors are more likely to view
compromise as a personal loss ("Winning is the only thing!" and all
that).
- 5.9
- If the social motives of committee members can predict aspects of the members' group dynamics, then a simulation of this relationship is possible. A simulation of three cooperative adjudicators, for example, might indicate rapid consensus in 90% of discussions to resolve their disagreements. A simulation of two cooperative adjudicators and one competitive adjudicator might show rapid consensus in only 10% of the discussions, and no consensus in another 25%. Such simulation results would be useful to administrators seeking rapid consensus, perhaps leading them to conclude that cooperativeness is just as important as expertise in selecting future adjudicators.
Discussion
- 6.1
- Policy capturing, information
seeking, and motivational inference are three of several methodologies
for explicating salient features of people's cognitive processes and
motivational orientations. The incorporation of their results in
agent-based models has the potential to increase the realism and
utility of these models. Still, all methodologies have limitations, and
the three outlined here are no exception. The three methodologies are
all reactive, requiring that people engage in tasks
while
knowing their judgments and decisions are being recorded. Even with
assurances of anonymity, some people might fake good during these
tasks, trying to conceal sloppy thinking or sinister motives. The
procedures themselves are somewhat artificial, so the resulting
explications might not generalize to procedural variations in the world
being simulated by an agent-based model.
- 6.2
- These and other
possible limitations, however, are no greater than those of alternative
methodologies. Interviews, for example, are equally reactive and
produce responses equally likely to change from time to time. Content
analyses of relevant documents or other archival data might not reveal
a judgment or decision process relevant to the situation being
simulated. Despite their limitations, most of these empirical
methodologies still appear more worthy of a try than the traditional
alternative: relying on a researcher's academic introspections or a
programmer's expediencies to portray a typical agent.
- 6.3
- Who should be
chosen to model? If a simulation can validly assume that an agent or a
group of agents all use the same judgment and decision processes, then
any one of the people modeled by the agent or agent group is fair game.
If, for example, we wish to model air traffic controllers in a
simulation of how increases in air traffic affects the chances of an
accident, and if we know these professionals do their job in pretty
much the same way, then explicating the process of one controller would
suffice. Alas, as Hoffman, Slovic and Rorer (1968)
discovered among radiologists, this interpersonal consistency is rare.
Final choices about the number and variety of cognitive and
motivational processes to be coded in an agent-based model are as much
art as science. But the choices are likely to be improved by trying one
or more of the methodologies noted here.
- 6.4
- Groups of people,
rather than individuals, often make the decisions being simulated by an
agent-based model. When they do, modeling becomes more complex. Group
judgment and decision processes are often quite different the processes
of individual group members. Social interactions and group pressures,
for example, often lead people to abandon their preferred processes in
order to avoid interpersonal conflicts, to assert their dominance in a
group, to reduce the time taken in boring group discussions, etc. (for
example, see Esser 1998; Janis 1972, 1982).
As a result, it is worthwhile to consider modeling the group itself
rather than modeling its individual members. To my knowledge, no
research has yet been done to capture a group policy, trace a group
information-seeking trajectory, or estimate the motivational
orientation of a group. It might be possible to do so if a group is
small and its members cooperative. On the other hand, it is little more
than amusing to think of turning a House of Commons into a lab.
- 6.5
- Finally, people
are known to change their hearts and minds on occasion, causing them to
change the cognitive processes or motivational orientations they employ
in making judgments and decisions. Indeed, many of these changes form
the goals of higher education; consider, for example, management or
leadership programmes offering courses on how to be a better decision
maker by breaking old habits of thought and feeling. Yet people do not
need a diploma to make such changes. The outcomes of current decisions
often affect the ends or means of making future decisions. When
decisions are repeated and outcomes are known, learning can occur, with
its concomitant changes in cognition or motivation. When decisions
proliferate or become routine, stress, burnout or boredom can occur,
motivating cognitive shortcuts. Interpersonal conflicts can poison
group decision making routines. The same routines can also change as
group members come and go. These and other changes in judgment and
decision processes present even more challenges to researchers trying
to model real situations. Assessing such moving targets is likely to
require repeated measurements.
- 6.6
- Unfortunately, it takes time, effort and often money to design, construct, conduct and analyse the results of studies based on the three methodologies discussed above. They can be, and sometimes will be, a bother. It is thus worthwhile to learn in advance if the likely decision processes to be explicated by these methodologies will make much of a difference in the outcomes of subsequent simulations. Sensitivity analyses can help. By varying the lines of code representing agents' judgment or decision processes during trial runs of the simulation, it should be possible to determine how much variations in this code influence the simulation outcomes. If the influence is small, there is little reason to deploy the methodologies noted here. If they influence is large, the methodologies are worthy of deployment.
References
- BETTMAN, J., & Kakkar,
P. (1977). Effects of information presentation format on consumer
information acquisition strategies. Journal of Consumer
Research, 3(4), 233–240. [doi:10.1086/208672]
BREHMER, B., & Joyce, C. (1988) (Eds.). Human judgment: The Social Judgment Theory view. Amsterdam: Elsevier Science Publishers.
BREWER, M. B., & Kramer, R. M. (1986). Choice behavior in social dilemmas: Effects of social identity, group size and decision framing. Journal of Personality and Social Psychology, 50, 543–549. [doi:10.1037/0022-3514.50.3.543]
CHOWDHURY, W., & Thorngate, W. (2013). Take it or leave it: Cognitive rules for satisfying choices. In R. West & T. Stewart (eds.), Proceedings of the 12th International Conference on Cognitive Modeling, Ottawa: Carleton University.
DAWES, R., & Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81(2), 95–106. [doi:10.1037/h0037613]
DUCHOWSKI, A. (2007). Eye tracking methodology: Theory and practice (2nd Ed.). New York: Springer.
ESSER, J. (1998). Alive and well after 25 years: A review of groupthink research. Organizational Behavior and Human Decision Processes, 73(2/3), 116–141. [doi:10.1006/obhd.1998.2758]
GIGERENZER, G., & Gaissmaier, W. (2011). Heuristic decision making. Annual Review of Psychology, 62, 451–482. [doi:10.1146/annurev-psych-120709-145346]
GOLDBERG, L. (1968). Simple models or simple processes? Some research on clinical judgments. American Psychologist, 23, 483–496. [doi:10.1037/h0026206]
GOLDBERG, L. (1959). The effectiveness of clinicians' judgments: The diagnosis of organic brain damage from the Bender-Gestalt Test. Journal of Consulting Psychology, 23(1), 25–33. [doi:10.1037/h0048736]
GRIESINGER, D. W. , & Livingstone, J. W. (1973). Toward a model of interpersonal motivation in experimental games. Behavioral Science, 18, 73–78. [doi:10.1002/bs.3830180305]
GRIFFIN, D., Gonzalez, R., Koehler, D., & Gilovich, T. (2012). Judgmental heuristics: A historical overview. In K. Holyoak and R. Morrison (Eds.), Oxford Handbook of Thinking and Reasoning. Oxford: Oxford University Press, 322–345. [doi:10.1093/oxfordhb/9780199734689.013.0017]
GRZELAK, J. L. (1982). Preferences and cognitive processes in social interdependence situations. In V. Derlega & J. Grzelak (Eds), Cooperation and helping behavior: Theory and research. New York: Academic Press. [doi:10.1016/B978-0-12-210820-4.50010-1]
HAMMOND, K. (1955). Probabilistic functioning and the clinical method. Psychological Review, 62(4), 255–262. [doi:10.1037/h0046845]
HAMMOND, K., Hursch, C., & Todd, F. (1964). Analyzing the components of clinical inference. Psychological Review, 71(6), 438–456 [doi:10.1037/h0040736]
HAMMOND, K., & Summers, D. (1972). Cognitive control. Psychological Review, 79(1), 58–67. [doi:10.1037/h0031851]
HOFACKER, T. (1984). Identifying consumer information processing strategies: New methods of analyzing information display board data. In T Kinnear (Ed.), Advances in Consumer Research Volume 11. Provo, UT : Association for Consumer Research, pp. 579–584.
HOFFMAN, P.(1960). The paramorphic representation of clinical judgment. Psychological Bulletin, 57(2), 116–131. [doi:10.1037/h0047807]
HOFFMAN, P., Slovic, P. & Rorer, L. (1968). An analysis-of-variance model for the assessment of configural cue utliization in clinical judgment. Psychological Bulletin, 69(5), 338–349. [doi:10.1037/h0025665]
JANIS, I. (1972). Victims of groupthink. Boston: Houghton-Mifflin.
JANIS, I. (1982). Groupthink (2nd ed.). Boston: Houghton-Mifflin.
KAHNEMAN, D. (2011). Thinking, fast and slow. New York: Farrar, Strauss and Giroux.
KNOX, R., & Inkster, J. (1969). Postdecision dissonance at post time. Journal of Personality and Social Psychology, 8(4), 319–323. [doi:10.1037/h0025528]
MCCLINTOCK, C., Messick, D., Kuhlmn, M., & Campos, F. (1973). Motivational bases of choice in three-choice decomposed games. Journal of Experimental Social Psychology, 9(6), 572–590. [doi:10.1016/0022-1031(73)90039-5]
MACCRIMMON, K., & Messick, D. (1976). A framework for social motives. Behavioral Science, 21(2), 86–100. [doi:10.1002/bs.3830210203]
MESSICK, D., & Brewer, M. (1983). Solving social dilemmas: A review. In L. Wheeler & P. Shaver (Eds.), Review of Personality and Social Psychology (Vol 4, pp 11–44). Beverly Hills: Sage.
MESSICK & McClintock, C. (1968). Motivational bases of choice in experimental games. Journal of Experimental Social Psychology, 4(1), 1–25. [doi:10.1016/0022-1031(68)90046-2]
NISBETT, R., & Wilson, T. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259. [doi:10.1037/0033-295X.84.3.231]
ORQUIN, J., & Mueller Loose, S. (2013). Attention and choice: A review of eye movements in decision making. Acta Psychologica, 144(1), 190–206. [doi:10.1016/j.actpsy.2013.06.003]
PAYNE, J. (1976). Heuristic search processes in decision making. in B. Anderson (Ed.), Advances in Consumer Research, Volume 3. Cincinnati, OH: Association for Consumer Research, pp. 321–327.
SLOVIC, P. (1966). Cue-Consistency and Cue-Utilization in Judgment. American Journal of Psychology, 79(3), 427–434. [doi:10.2307/1420883]
SLOVIC, P. (1969). Analyzing the expert judge: A descriptive study of a stockbroker's decision process. Journal of Applied Psychology, 53(4), 255–263. [doi:10.1037/h0027773]
THORNGATE, W. (1974). Information seeking: An alternative methodology for the investigation of social judgements. Report 74-6, Social Psychology Labs, University of Alberta, 14 pages.
THORNGATE, W. (1975). Process invariance: Another red herring. Personality and Social Psychology Bulletin, 1, 485–488. [doi:10.1177/014616727500100304]
THORNGATE, W. (1980). Efficient decision heuristics. Behavioral Science, 25, 219–225. [doi:10.1002/bs.3830250306]
THORNGATE, W. & Maki, J. Decision heuristics and the choice of political candidates. Report 76-1, Social Psychology Labs, University of Alberta, 37 pages.
THORNGATE, W., & Tavakoli, M. (2005). In the long run: Biological versus economic rationality. Simulation & Gaming, 36(1), 9–26. [doi:10.1177/1046878104270471]
VAN LANGE, P., de Cremer, D., van Dijt, E., & van Vugt, M. (2007). Self-interest and beyond: Basic principles of social interaction. In A. Kruglanski & E.T. Higgins (eds.), Social Psychology: Handbook of Basic Principles. New York: Guilford.