Microsimulation in Government Policy and Forecasting

Order this book

Reviewed by
Douglas A. Wolf
Centre for Policy Research, Syracuse University, Syracuse, New York, USA.

Close Enough for Government Work? The Current State of Public Policy Microsimulation

Most of the chapters in this volume were papers originally presented at a special session (bearing the same name) held in conjunction with the 1997 International Conference on Combinatorics, Information Theory and Statistics. They are organised into four areas - "Microsimulation in Tax Policy", "Microsimulation Modelling: Issues and Developments", "Recent Advances in Dynamic Modelling" and "Data Issues", each of which is preceded by an introductory overview that summarises and expands upon the themes raised by the respective chapters. A final section of the book ("Practical Microsimulation Models") presents brief (6 pages on average) treatments of 12 microsimulation efforts in current development and use (mostly at the national level) from a number of countries.

The editors portray the book as an attempt to capture the developments in microsimulation since a 1993 international conference held in Canberra, which also led to a published volume edited by Ann Harding (1996). Thus, this book joins a progression of edited volumes that collectively represent a large part of the publicly available literature about this type of simulation activity. Among several earlier examples in this genre are Haveman and Hollenbeck's two volume set (1980), Orcutt, Merz and Quinke (1986), and Lewis and Michel (1990), although the present volume may have the broadest coverage to date of modelling efforts around the globe. Among the books in the long line of edited volumes that it joins, this one, more than the others, tells the story of public policy microsimulation activity from the perspective of government employees engaged in in-house model building efforts and their applications. By my count, 21 of the 39 authors whose biographical information is summarised at the end of the book have governmental affiliations, compared to a total of 14 from the academic or "think tank" worlds. (Another four come from the private sector or consulting arena.) Governments are, of course, the principal users of results from public policy microsimulation efforts. Furthermore, government agencies seem to have a comparative advantage in creating the organisational setting and mobilising and sustaining the human and financial resources necessary to undertake efforts on the obviously large scale revealed by many of the chapters in this book. Government agencies certainly tend to have an advantage with respect to the kind of data access upon which these types of modelling efforts depend.

According to the editors, the focus of the 1997 conference was upon "practical uses of microsimulation in government policy" and this focus is certainly visible in the book: much of the material presented deals with the production of microsimulation programs, and of the data files that feed them; somewhat less time is spent on applications of the programs to concrete policy issues; and (to my thinking a noteworthy weakness of the book) almost none to the consequences of those applications with respect to policy formulation and program design. Furthermore, the editors appear to have pursued a strategy of broad coverage - the book contains 29 chapters reporting on nearly that many distinct modelling enterprises, plus the 12 additional mini-chapters that summarise still more programs - leading inevitably to a shortage of detail on the individual models represented. While no reader should expect to see extensive details on what are, after all, extraordinarily complex programs, the general tendency of these chapters to dwell on what their programs produce at the expense of how they produce it limits the usefulness of the book. (For example, Caldwell's CORSIM program, the subject of chapters 20 and 21, depends upon more than 5000 numerical parameters, not a single one of which is reproduced.) Bureaucrats in agencies not already engaged in microsimulation may find the book useful, inasmuch as it documents the continued spread of the technique to additional countries, agencies and topics, providing ammunition with which to make the case for new resources, and points them towards practitioners whose experiences (and model documentation or reports) will doubtless prove helpful.

The types of simulation programs depicted in this book fall into just one of the seven categories of social simulation, each given its own chapter in the excellent Simulation for the Social Scientist by Gilbert and Troitzsch (1999). Placed in the broader context of social simulation, public policy microsimulation is distinctive in its emphasis on applications - an emphasis well depicted in the current book - as well as its immediacy, complexity and scale. Government policy planners and analysts deal with tax and benefit programs in which small programmatic changes can produce large changes in the circumstances and well being of many citizens, at great aggregate cost. As many of the chapters of this book point out, the cost of even the most complex microsimulation enterprise, one that entails many person-years of effort to develop, is dwarfed by the potential costs of the programmatic changes that these models attempt to forecast. Thus, these large-scale activities seem well justified.

Unsurprisingly, nearly every chapter in this book deals narrowly with a single microsimulation program, operating within and on behalf of a single government ministry or agency, often focused on a narrow policy area (e.g. corporate tax, income tax or social assistance benefits). Thus the broadly oriented chapters by Michael Wolfson (chapter 9, which introduces the "Microsimulation Modelling: Issues and Developments" section) and Ann Harding (chapter 15, which opens the section on "Recent Advances in Dynamic Modelling") are particularly welcome. Wolfson moves from a broad and somewhat philosophical portrayal of historical developments to some musings on possible - and promising - future areas of progress, areas that will require the practitioners of microsimulation to think in new ways about the form, structure, accessibility and scope of their work. Harding's overview of dynamic models, while brief, covers much territory (historically, geographically and in subject matter) and she cites numerous microsimulation researchers and models not otherwise represented in the book.

Another standout chapter is chapter 13 by Gary Bagley, Joseph E. Burpee, and Stéphane Jetté. This offers, in a candid (and often entertaining) way, some experience based advice to would-be microsimulators, advice intended to enhance the usefulness, effectiveness and impact of their efforts. As these authors point out, although we generally "... want our models to give the 'right' answers ... a more fundamental issue ... is whether we can defend those answers" (p. 248). Defensibility, as they go on to say, entails consistency with available data sources, but there are often no such data sources. At any rate, defensibility may also be a matter of consistency with strongly held beliefs on the part of higher-level officials, or contrary political agendas and ideologies, a battle in which simulated numbers have an intrinsic disadvantage. On the other hand, ex post defensibility is rarely investigated: the policy environments in which the authors of these chapters operate rarely afford an opportunity to check how well their model predicted the response to policy changes later adopted.

While few researchers are likely to read this book in its entirety, doing so prompts a number of reactions on the current state of public policy microsimulation, as least as represented here. Some of these reactions are criticisms of what this book does (or does not) contain, while others reflect more broadly on the world of public policy microsimulation as a whole.

First, the long-standing distinction between "static" and "dynamic" models, further perpetuated here, fosters compartmentalisation and inhibits developments in both areas. Roughly speaking, static models take a cross-sectional database and add contemporaneous but unrecorded data elements to it, while dynamic models take a cross-sectional database and simulate subsequent life events for the individuals and households represented in it, often including self-replenishing population dynamics. But the static simulations are often preceded by a data "aging" exercise, in which a sample taken (necessarily) from a population that existed in the past is adjusted to represent the characteristics of the present (or even future) population. Given the lags that exist in any population sampling or monitoring system, we never know (or have conventional statistical estimates of) the characteristics of today's population and we certainly don't know the characteristics of a future population. Thus, in practice, these "aging" exercises typically involve adjustments of sampling weights so that means on selected attribute variables match an assumed value of their population counterparts. The latter data, in turn, probably comes from another type of model produced in some other government agency. Aging should, however, be viewed as a simulation exercise rather than a prelude to a simulation exercise. Moreover, "aging" is precisely what dynamic models are designed to do. Dynamic microsimulation, in other words, should be viewed as the input into so-called static microsimulation. At the same time, static microsimulation can be viewed as a useful postscript to dynamic microsimulation. One example of a successful effort along these lines can be found in Zedlewski et al. (1990), who adopt a two-stage modelling procedure: they first use the well-known DYNASIM program to project the size and composition of the older population over a 40-year period, and then impute additional characteristics to the simulated individuals contained in that population, using cross-sectional (i.e. "static") methods applied to selected future years. As ill-founded as the implicit assumptions regarding causality might be in such a case, they are no more ill-founded than those routinely encountered in the world of static microsimulation analyses.

The first eight chapters in this book (collectively, the section on "Microsimulation in Tax Policy") deal exclusively with static models, while chapters 15 through 21 deal exclusively with dynamic models. The former chapters are substantially less detailed with respect to model contents, formal structure and procedures than are the latter. I believe thisreflects the fact that static microsimulation requires relatively little of the sort of effort normally viewed as "modelling" - i.e. creating a formal analytic apparatus that abstracts, in some way, from the complexities of the observed world. Instead, much of what takes place in a static microsimulation is the assignment, through imputation, matching or calculation, of values that might have been (but weren't) solicited from the respondents to a large-scale household survey. The imputation or matching, in turn, is not based upon an explicit "model" - i.e. a conscious depiction of the underlying data generation process. One of the chapters (chapter 4 by Tove Birgitte Pedersen) stretches the notion of microsimulation even farther. This chapter reports results from analyses of income inequality and tax progressivity, based on an unusually rich database taken from Danish population-registry information. None of the data elements analysed appears to have been unrecorded in the database. It seems more accurate to classify this work as "data analysis" than as "microsimulation."

Dynamic microsimulation is more ambitious, more complex, more demanding but also (in my opinion) more interesting than static microsimulation. The chapters on dynamic microsimulation included in this book give the reader more to think about than do the chapters on static microsimulation, partly because as already mentioned, they contain a good deal more detail on the contents and structure of the models discussed. Statistics Canada's LifePaths model (described in chapter 19 by Steve Gribble) appears to be the most elegant and most highly developed dynamic microsimulation currently in existence. It is built almost exclusively of continuous-time competing-risks event modules and therefore enjoys a good deal more internal consistency and coherence than most other dynamic models. Interestingly, it runs off a completely synthetic database, freeing it from the need to adjust, impute or otherwise tinker with a starting population, yet seems to track observed population characteristics rather well. Yet its advantages are somewhat oversold. Gribble points out that using computed waiting times for each competing event type (making the next simulated event that with the shortest waiting time) reduces Monte Carlo variability resulting from the conventional procedure of comparing an event probability with a draw from the random number generator to assign outcomes. This is true, but the advantageous reduction in Monte Carlo variability is accompanied by a disadvantageous reduction in real world variability. To see why this is so, simply consider that in a simulation in which all outcomes are determined by evaluation of competing-risks waiting time distributions, two simulated individuals with identical histories will also have identical futures, undoubtedly understating the heterogeneity found in real world data.

Second, missing from this book is material that depicts the impacts that microsimulation analyses have in actual policy deliberations. Several of the chapters allude to real world policy initiatives, or proposed initiatives, that prompted model developments and produced findings. In some cases those findings are excerpted, but in no case is the reader told what happened next. How warmly were the findings embraced? How strongly were they attacked? Many chapters begin with a ritual invocation of the notion that the policy process demands high quality information, an unassailable notion. But the policy process demands (or at least makes use of) many other inputs as well. These include the interplay of legislative and executive manoeuvring, partisan conflict, interest-group advocacy, personal ambition, journalistic coverage - and possibly even the public interest - in addition to policy research and forecasts derived from sources other than official microsimulations. How do microsimulation results hold up in this rough-and-tumble environment? The modellers themselves may not be able to provide a comprehensive account of this broader set of issues; it would have added an interesting dimension to the book to have included the perspectives of consumers - and possibly even of opponents - of microsimulation analysis.

Third, there is substantial tension between the apparent goal of the policy oriented microsimulator - i.e. to reflect accurately and in considerable detail the measurable world of hours worked, wages paid, taxes imposed, funds deposited, benefits received and so on - and the world of inaccurate perceptions and incomplete knowledge occupied by the human informants whose responses to survey (or census) questions provide many of the data elements that serve as inputs to the computer programs. Analysts routinely perform adjustments to bring survey data on incomes, benefits and taxes into agreement with aggregate totals that are known (or believed) to be true, based on "external" sources such as national income accounts, the ledgers of taxing authorities, social insurance administrators and so on. A telling example is seen in chapter 21 (by Lisa A. Keister and Steven B. Caldwell) which reports on an ambitious effort to model trajectories of wealth holdings among US households. Speaking of the inadequacies of surveys in which questions on assets are asked, Keister and Caldwell write that "... even if top wealth holders are represented in a survey, they are unlikely to be fully informed about their wealth holdings" (p. 432). The authors are no doubt correct in this assessment, but is it not the misinformation on wealth holdings that underlies the daily lives of the very wealthy, and shapes many of their routine actions? And what of the many dimensions of human behaviour that are reported in surveys, and represented in microsimulation models, but for which there are no "external" sources of "control totals" to which to adjust the survey responses? One good example of the latter is cohabitation and another is time use. What we are left with are models in which the "micro" units are in some respects like people - possibly ill-informed individuals, or otherwise content to report inaccurately, or incompletely, the details of their lives to interviewers - and in other respects like perfectly informed, law-abiding and rule-adhering robots.

Finally, the chapters in this book collectively reveal the continued failure of practitioners of public policy microsimulation to appreciate the parallels (and potentially fruitful interconnections) between statistics (including probability theory) and microsimulation. For example, simulations inevitably depend on sampling of several kinds. Most (with LifePaths as a rare exception) take as input data a sample from a real population. Virtually all assign outcome values using one or more statistically estimated parameters, themselves based on sample data and virtually all sample from a set of hypothetical futures (or otherwise unobserved variable domains) using some combination of deterministic and random elements. The pervasive presence of sampling gives rise to uncertainty about the "true" value of population characteristics derived from simulated microdata. Many of the authors of these chapters mention techniques to reduce the uncertainty associated with Monte Carlo methods - i.e. the use of random numbers to assign outcomes - but my experience (and that found in several other published articles) suggests that Monte Carlo variation is inconsequential (or can be rendered inconsequential). Far more problematic is the uncertainty associated with the use of statistically estimated model parameters. Little effort has gone into addressing the latter type of uncertainty. Interestingly, the volume editors mention the work on this topic by Pudney and Sutherland (1996) in their opening chapter, but there is no other evidence of concern about this problem in the book.

The uncertainty inherent in microsimulations, but largely unrecognised by microsimulators, is closely tied to another issue of much evident concern to the community, namely the need to validate models through comparison to out-of-model "known" quantities, or to calibrate ("align") the models to reproduce out-of-model quantities, including forecast totals produced by other models or other agencies (often actuarial analyses or macroeconomic forecasts). In particular, a microsimulation may be judged to give a "wrong" answer if the answer it gives disagrees with some other known quantity - that is, if the point estimate produced by a microsimulation differs from an exogenously given point value. Once we realise that microsimulations embody a degree of statistical uncertainty, some inherent in the sampling that produced the starting population and some propagated from parameter errors, then we should be willing to compute interval estimates of the population characteristics associated with microsimulation output. Having done this, we can see whether these intervals cover the exogenous quantities. Similarly, if a microsimulation of, say, future retirement income obligations is required to agree with an actuarial forecast, then we should compare an interval estimate of the mean (or total) future obligations based on microsimulation to the point estimated produced by the actuaries. This is especially the case since the actuarial estimate is itself certainly "wrong" albeit, perhaps, more widely accepted or simply better established among policy analysts and the consumers of their findings. In other words, the "defensibility" discussed by Bagley et al. might improve if modellers adopted a broader, yet rigorously and statistically based, conception of what constitutes the "right" answer. But little work has been done on this problem, and it remains an area of formidable challenges and, I would contend, much importance.

Overall, the picture of the world of public policy microsimulation presented in this book is one of slow development. Because government officials, generally under great pressure to produce quick answers, are the principal consumers of microsimulation output, and because government agencies are in a special position to mobilise human and financial resources, and - especially - to sustain them over a long period of model development and ongoing refinement, the public sector is likely to remain the main venue for this type of model building. The enterprise continues to develop slowly, notwithstanding the presence of some particularly bright spots here and there within it. More cross-fertilisation - of academics and public employees, of static and dynamic simulators, of model builders and statisticians, and especially of public policy microsimulators and those working in other branches of the social simulation family tree - would surely contribute to more, and faster, progress. I look forward to a day on which the actuaries (and perhaps even the macroeconomic forecasters) are required by their agency chiefs to calibrate their findings to those of the microsimulators.

References

GILBERT N. and K. G. Troitzsch 1999. Simulation for the Social Scientist. Open University Press, Buckingham. [JASSS REVIEW]

HARDING A., editor, 1996. Microsimulation and Public Policy. North Holland, Amsterdam. [JASSS REVIEW]

HAVEMAN R. H. and K. Hollenbeck 1980. Microeconomic Simulation Models for Public Policy Analysis. Academic Press, New York, NY.

LEWIS G. H. and R. C. Michel 1990. Microsimulation Techniques for Tax and Transfer Analysis. The Urban Institute, Washington, DC.

ORCUTT G., J. Merz and H. Quinke, editors, 1986. Microanalytic Simulation Models to Support Social and Financial Policy. Elsevier, Amsterdam.

PUDNEY S. and H. Sutherland 1996. Statistical reliability in microsimulation models with econometrically based behavioural responses. In A. Harding, editor, Microsimulation and Public Policy. North Holland, Amsterdam.

ZEDLEWSKI S. R., R. O. Barnes, M. R. Burt, T. D. McBride and J. A. Meyer 1990. The needs of the elderly in the 21st century. Report 90-5, The Urban Institute, Washington, DC.

Return to Contents of this issue

© Copyright Journal of Artificial Societies and Social Simulation, 2003