Nils B. Weidmann and Luc Girardin (2005)

Technical Note: Evaluating Java Development Kits for Agent-Based Modeling

Journal of Artificial Societies and Social Simulation vol. 8, no. 2
<https://www.jasss.org/8/2/8.html>

To cite articles published in the Journal of Artificial Societies and Social Simulation, reference the above information and include paragraph numbers if necessary

Received: 23-Feb-2005    Accepted: 24-Feb-2005    Published: 31-Mar-2005

Abstract

We present experiments assessing the performance of different Java Development Kits (JDKs) with respect to agent-based models. In our experiments, we measure the execution time of two prototypical agent-based models written in Java under different JDKs and operating systems. Our findings suggest that whereas the impact of the operating system is fairly small, the JDK in use affects performance considerably, with the most recent JDKs performing best. The results of a numerical Java benchmark computed for the same JDKs and operating systems display only a small correlation with these results. In a series of scaling experiments we examine the JDK performance along an increasing model size, again finding a competitive advantage of recent JDKs.

Keywords:

Model Performance, Java, RePast

Introduction

1.1

Agent-based models can serve as an important tool for the social scientist. A lot of those models are developed in Java, not least because of the system-independence of the programming language and the availability of advanced modeling toolkits for Java. Particularly for large models, the performance drawback of Java's interpreted execution is likely to be significant. However, a range of different Java Virtual Machines (JVM) is available to the developer, each employing different techniques to speed up execution.

1.2

During the installation of our research group's new cluster computing system for agent-based models the question was raised which JDK and JVM to choose. In order to get a satisfying answer, we performed a systematic evaluation of currently available JDKs with respect to agent-based simulation models. This paper presents a summary of our method and results.

1.3

It needs to be emphasized that the focus of this technical note are agent-based simulations without graphical output generation, as typically required during model validation and model evaluation phases. Here, a large number of model runs is performed in order to assess model robustness or generate numerical output later used for statistical analysis. Though graphical visualization is an important tool for the exploration of an agent-based model, we focus on settings where a large number of model runs is required. In those cases one will typically disable graphics feature to make computation feasible.

1.4

The JDKs taken into consideration include the Sun JDK (Sun Microsystems Inc. 2005), both version 1.4.2 and 1.5.0 (referred to as 5.0 by Sun), the JRockit JDK (BEA Systems Inc. 2005) by BEA Systems Inc., both version 1.4.2 and 1.5.0, and the IBM JDK (IBM Corporation 2005) version 1.4.2. We have evaluated the selected JDKs on two different operating systems, MS Windows XP Professional^TM (WinXP) and Linux Fedora Core 2.

1.5

Two well-known agent-based models implemented in Java by the authors were used for the performance evaluation^[1]. Schelling's Segregation model (Schelling 1978) is one of the first agent-based models in the social sciences and aims at explaining the segregation of neighborhoods. The Iterated Prisoner's Dilemma (IPD) by Cohen, Riolo and Axelrod (1999) extends the two-player Prisoner's Dilemma game to a multiplayer setting on different topologies. The Schelling model was implemented using only the Java core classes, whereas the IPD is based on the RePast modeling toolkit (RePast Developers Group 2005). Further details of the models are given in the next section.

1.6

In a second series of experiments, we compared the results obtained in the first series to the scores of the numerical benchmark SciMark 2.0 (Pozo and Miller 2000). The goal is to examine the degree to which JDK performance with respect to agent-based models can be measured by a numerical benchmark.

1.7

A third series of performance tests was run to assess the scaling behavior of the selected JDKs. To this end, model runs were performed with stepwise increments of the model size, again measuring the execution time of the model.

Method

2.1

Evaluating JDKs with respect to agent-based models first requires a definition of the class of models we are dealing with. Although we do not discuss a conceptual definition of an agent-based model here, we claim that there are, from a technical point of view, certain characteristic features of such a model. Since agents are typically represented by objects, a lot of execution time is spent on object operations such as object creation, modification and garbage collection. On the other hand, there are parts of an agent-based model where numerical computation is needed. Instead of determining the computational characteristics of agent-based models in general and then evaluating JDKs with respect to those characteristics, our approach was to pick two prototypical agent-based models as examples and empirically measure the JDK performance on those models.

2.2

The example models are well known in the literature. The Schelling segregation model (Schelling 1978) describes a way segregation patterns arise among different kinds of agents by actions of individuals. The Iterated Prisoner's Dilemma game consists of a collection of agents placed on a grid or network, with the agents engaging in repeated Prisoner's dilemma games with their neighbors (Cohen et al. 1999). Specifications of both models are given in the next paragraphs. The implementations were not tuned to maximum performance; we rather followed standard object orientation programming guidelines.

2.3

Schelling model specifications. The Schelling model consists of a 100 × 100 grid where each cell is populated with probability 0.9 and empty otherwise. Agents belong to one out of two identities, and the identity distribution of the agent population is equal. In each time step, agents are activated in random order. The currently active agent examines the agents in its Moore neighborhood and moves to another convenient location if it is not satisfied at its current position. The satisfaction criterion set by Schelling requires about a third of an agent's neighbors to be of the same kind. The model is run for 100 time steps. Note that because (Schelling 1978) does not fully specify the model, the grid size, population and identity ratio were set to reasonable values by the authors. Furthermore, the implementation is based solely on Java core classes, and no additional library was used.

2.4

IPD model specifications. In the Cohen et al. (1999) IPD model, there is a grid of 16 × 16 agents. The agents play repeated Prisoner's Dilemma games with each of their four Von-Neumann neighbors, each episode consisting of four games. An agent is acting according to a strategy it has been assigned. The set of possible strategies is ALLD (always defecting), ALLC (always cooperating), TFT (Tit-for-Tat, starts with cooperation and then reciprocates the opponent's previous action) and ATFT (Anti-Tit-for-Tat, does exactly the opposite of what TFT would have done). The strategies are equally distributed on initializing the model. After each agent has played against its four neighbors, it adapts the strategy of its most successful neighbor, if that neighbor has received a higher outcome than the agent itself. This model setting corresponds to the "2DK" interaction topology and the "ImitS" adaptation process introduced in Cohen et al. (1999). The model is run for 2500 iterations (only 500 in the third experiment series, see below), each one consisting of a game phase with mutual interactions of the agents, and an adaptation phase where agents copy their most successful neighbor's strategy as explained above.

2.5

The IPD model was implemented using the RePast modeling toolkit (RePast Developers Group 2005), version 2.2. RePast facilitates scheduling of the model and has a predefined grid structure for the agents. In order to perform multiple runs without graphics output, we used the so-called "batch" mode of RePast. In a batch run, the model reads a parameter file specifying the initial parameter settings and the runs to be performed. This is a convenient way to perform parameter sweeps during model exploration and validation phases.

2.6

Three series of experiments were run. In the first series, we measure the execution time of both models, run under different JDKs and both Windows XP and Linux. These experiments enable a performance comparison across different JDKs and operating systems. A second series of experiments aims at the comparison of our results with the numerical Java benchmark "Scimark 2.0" to answer the question of how a numerical benchmark can predict JDK performance for agent-based models. Finally, in a third series of experiments we examine the scaling behavior of the JDKs, that is, how they can cope with models of increasing size.

2.7

The hardware for the experiments was kept constant. All models were run on an Intel Pentium IV 2.4 GHz computer with 1 GB RAM and Hyperthreading enabled, and both under Windows XP Professional for Intel x86, Service Pack 2 (Microsoft Corporation 2005) and Linux Fedora Core 2 for Intel x86, Kernel 2.4.20^[2] (Red Hat Inc. 2004). The following JDKs were selected for evaluation.

Sun JDK.

Sun is the main promoter of the Java technology. Each JDK comes with a "client" and a "server" virtual machine, and the one to be used is specified by a command line switch for the "java" command ("-client" or "-server", respectively, where the former is the default setting). These two solutions share the Java HotSpot runtime environment code base, but use a different compilation inlining policy and heap defaults that are suited to the distinct tradeoffs between startup time and peak operating speed of client and server applications (Sun Microsystems Inc. 2002). We evaluated version 1.4.2 release 07 and version 1.5.0 release 01, which are available free of charge to end-users. Complete source code can also be obtained under the Sun Community Source License.

BEA JRockit.

BEA's JDK is designed for enterprise applications. Because of its special server-side focus, JRockit compiles methods upon their first execution. It then dynamically collects statistics to perform subsequent optimizations while the application keeps running (BEA Systems Inc. 2003), following the dynamic compilation strategy of the HotSpot JVM. We evaluated version 1.4.2 Release 04 and version 1.5.0, both available at no cost for evaluation and production use.

IBM JDK.

IBM has produced their own version of the JDKs, with implementations for a fair number of hardware architectures and operating systems. The embedded IBM JVM Mixed Mode Interpreter (MMI) serves the same purpose as the HotSpot JVM, which is to spread the compilation of methods out over the lifetime of the JVM (IBM Corporation 2004). IBM's JDKs can be downloaded free of charge from their website.

2.8

While it is obvious that each vendor uses some sort of dynamic compilation strategy, it is unclear to what extent their dynamic compiler is able to generate code that is optimized to the specific behavior of the program, such as the memory allocated by arrays and the time spent in different parts of the code. This is a major shift from previous generations of JVMs, allowing for aggressive method inlining that can surpass static and JIT compilation strategies. Sun's Java HotSpot is known to produce code tailored to the dynamic properties of the program, but this is unclear for the BEA and IBM products. Other factors that can significantly affect the performance of a JVM are the garbage collection facility and the threading scheme.

2.9

The models were compiled and executed with the compiler and the virtual machine taken from the same JDK, although we do not expect a significant impact of different compilers. However, this might be an issue requiring closer examination.

Results

Series 1: JVM and Operating System Comparison

3.1

The results for the first series of experiments are reported as model execution times in milliseconds and their standard deviations, averaged over 30 runs with different seeds for the random number generator. The time recorded does not include the startup time of the JVM. All results are rounded to full milliseconds.

Table 1: Execution times for the Schelling model in milliseconds
	Sun 1.4.2 Client	Sun 1.4.2 Server	Sun 1.5.0 Client	Sun 1.5.0 Server	JRockit 1.4.2	JRockit 1.5.0	IBM 1.4.2
Win XP	4748 [103]	3348 [111]	3044 [68]	2315 [92]	4997 [221]	3360 [470]	3375 [87]
Linux FC 2	5108 [131]	3772 [118]	3038 [69]	2393 [90]	5072 [240]	3384 [441]	3564 [92]

Table 2: Execution times for the IPD model in milliseconds
	Sun 1.4.2 Client	Sun 1.4.2 Server	Sun 1.5.0 Client	Sun 1.5.0 Server	JRockit 1.4.2	JRockit 1.5.0	IBM 1.4.2
Win XP	4955 [45]	3199 [102]	4089 [86]	3011 [103]	2417 [240]	2277 [441]	2183 [109]
Linux FC 2	5253 [67]	3250 [153]	4229 [69]	3108 [141]	2668 [329]	2466 [458]	2245 [152]

3.2

An important observation we made is that the influence of the operating system on model performance is rather small. In general, execution times under Linux tend to be higher, the difference is, however, negligible in most cases.

3.3

The choice of the JVM seems to be crucial for model performance. A speedup by a factor of 2 can be achieved by exchanging the worst performing JVM for the best one - a result consistent across operating systems and tested models. In general, we can state that the latest versions of the JDKs (Sun 1.5.0 Server and BEA 1.5.0) perform well both under Windows XP and Linux, and with both models.

3.4

An important point that needs to be mentioned is the impact of the performance optimization built into all but the Client virtual machines. This optimization takes place during the first run of the model, thus causing considerably longer execution times for this run. The standard deviations of the runs reflect this phenomenon, especially for the BEA JRockit JVMs. On the other hand, Sun's Client virtual machines do not show this behavior, but instead require constantly long execution times.

Series 2: Comparison with SciMark 2.0 scores

3.5

For a comparison of the performance results described above with a numerical benchmark, we computed benchmark scores with the same JDK/operating system combinations, using the "SciMark 2.0" Java benchmark. SciMark 2.0 is "a Java benchmark for scientific and numerical computing" (Pozo and Miller 2000) and computes an approximate score in Mflops. It is composed of a range of numerical kernels (Fast Fourier Transform, Jacobi Successive Over-relaxation, Monte Carlo integration, Sparse matrix multiply, and dense LU matrix factorization) and produces the composite score by taking the arithmetic mean of them. Each benchmark computation was repeated five times, and the results reported are average composite scores and their standard deviations. The higher the score, the better is the JVM's performance in numerical computation. All results were rounded to the second decimal.

Table 3: SciMark 2.0 benchmark scores
	Sun 1.4.2 Client	Sun 1.4.2 Server	Sun 1.5.0 Client	Sun 1.5.0 Server	JRockit 1.4.2	JRockit 1.5.0	IBM 1.4.2
Win XP	184.50 [0.28]	428.71 [0.98]	187.59 [0.23]	437.46 [1.81]	299.37 [0.84]	307.26 [0.53]	384.04 [0.45]
Linux FC 2	191.19 [0.87]	423.94 [0.99]	184.39 [0.85]	426.10 [2.45]	306.37 [1.93]	297.80 [3.23]	380.69 [2.71]

3.6

The correlation between the SciMark scores and execution times achieved for the Schelling model across operating systems is -0.45, for the IPD model it is -0.66. Split by operating system, the correlation coefficients are -0.5 (Schelling, WinXP), -0.64 (IPD, WinXP), -0.39 (Schelling, Linux) and -0.67 (IPD, Linux).

3.7

We can conclude that the evaluation based only on this numerical benchmark does not correspond to the performance results obtained in the first series of experiments, and therefore cannot serve as a good indicator to infer the performance of an agent-based model.

Series 3: Scaling behavior

3.8

A third series of model runs was performed to examine the scaling behavior of the JDKs, i.e. their performance along an increasing model size. To this end, we increased the grid size in the IPD model by increments of 5, starting with small models of 5 × 5 agents up to a model size of 100 × 100. For each grid size, we performed five runs with different random number seeds, each run with 500 time steps. The first five runs with a 5 × 5 model size were executed but not included in the chart below in order to eliminate JDKs performance optimization impacts as described above. The scaling behavior is given in the charts below.

Figure 1. Scaling behavior of the IPD under WinXP

Figure 2. Scaling behavior of the IPD under Linux FC2

3.9

In summary, one can observe the quadratic scaling behavior of almost all JVM's, caused by the number of agents being quadratic in the grid dimension. However, as we would have expected, scaling differences exist between JVMs, and those differences increase with the model size. Sun's Server JVMs and BEA's JRockit 1.5.0 show a good scaling behavior across operating systems, whereas IBM and JRockit 1.4.2 perform well under WinXP, but cannot keep up under Linux.

Conclusions and Further Work

4.1

Although there is no operating system / JDK combination an agent-based modeler should favor in general, there are some lessons to be learned from our experiments.

4.2

An important point is that the impact of the operating system seems to be negligible. Apart from the IBM JDK, whose performance under Linux does not seem to scale well with the model size, differences between runs on different operating systems are fairly small and not consistent. The model developer can thus safely run his experiments under Linux without having to accept performance degradations, and at the same time enjoying low software costs and an open software architecture.

4.3

However, the choice of the JDK (more precisely, the Java virtual machine) has quite an impact on model performance. We would tend to say that more recent JDKs seem to perform better, though a general recommendation cannot be made. What becomes obvious is that in agent-based models with a large number of runs, one should try to avoid Sun's Client virtual machines, because the impact of the Server virtual machine's optimization clearly speeds up performance in those settings.

4.4

It needs to be mentioned that the dynamical optimization only pays off during the lifetime of the JVM. In our experiments, model runs with different random seeds require only one startup of the JVM, so the optimization typically executed during the first run speeds up the subsequent ones (this holds for all models using RePast's batch mode). If, however, a new JVM startup is required for each run of the model, the performance difference between optimized and non-optimized JVMs is likely to be smaller.

4.5

The results achieved with the SciMark 2.0 benchmark suggest that the performance required in agent-based models is mainly not a numerical one. It might therefore be of little use to an agent-based modeler to select the best JDK by its score in a numerical benchmark. Furthermore, it is important to keep in mind that such synthetic micro-benchmarks, while not completely meaningless, can never replace the analysis of real-world applications. We are convinced that benchmarking is useful and feasible, but we would like to recommend moving away from the quest of the ultimate single benchmark to instead develop specific benchmarks that capture the behavior of a single field or application area.

4.6

For that reason, there is general recommendation we would like to give based on the experiments described above. In the context of large agent-based models and/or a large number of iterations, it might be worthwhile doing a short systematic evaluation of different JDKs. To this end, one could execute a small instance of the model with different virtual machines, thus determining the best runtime configuration for the current model.

4.7

In order to achieve more general conclusions about JDK performance, further work would have to be done. There is quite a number of virtual machine parameters that would have to be examined, and one could even go down to operating system parameters. Keeping experiments within feasible boundaries, we restricted ourselves to default operating system and JDK configurations. Moreover, one could select a broader range of models for evaluation, thus possibly enabling more general statements. The relation of other existing benchmarks with respect to agent-based models is one more issue that would have to be examined in more detail. There is a wide range of Java benchmarks available, some of which might be applicable to agent-based model performance.

4.8

However, despite the lack of a clear recommendation, we hope to have drawn the attention of the agent-based modeler to the impact of different JVMs on model performance and to the fact that merely by exchanging the JDK or JVM in use, one can achieve a significant execution speedup.

Notes

¹The Iterated Prisoner's Dilemma implementation is based on an earlier version by Lars-Erik Cederman and Laszlo Gulyas. The Java code of the models is available at http://www.icr.ethz.ch/publications/javaabm/.

²A 2.4 kernel was chosen because of stability considerations with our hardware configuration. Running the experiments on a 2.6 kernel could speed up execution; however, we do not expect major differences.

References

BEA SYSTEMS INC. (2003) 'BEA Weblogic JRockit Technical Whitepaper', http://www.bea.com/content/news_events/white_papers/BEA_JRockit_wp.pdf.

BEA SYSTEMS INC. (2005) 'BEA JRockit Website', http://www.bea.com/framework.jsp?CNT=index.htm&FP=/content/products/jrockit/.

COHEN, MICHAEL D, RIOLO, RICK L, and AXELROD, ROBERT (1999), The Emergence of Social Organization in the Prisoner's Dilemma: How Context-Preservation and Other Factors Promote Cooperation, SFI Working Papers, http://www.santafe.edu/research/publications/wpabstract/199901002.

IBM CORPORATION (2004) 'Java 2 Diagnostics Guide, Version 1.4.2', http://www.ibm.com/developerworks/java/jdk/diagnosis/diag142.pdf.

IBM CORPORATION (2005) 'IBM Java Website', http://www.ibm.com/developerworks/java/jdk/.

MICROSOFT CORPORATION (2005) 'Windows XP Home Page', http://www.microsoft.com/windowsxp/default.mspx.

POZO, ROLDAN and MILLER, BRUCE (2000) 'SciMark 2.0 Benchmark', http://math.nist.gov/scimark2/.

RED HAT INC. (2004) 'Fedora Project Website', http://fedora.redhat.com/.

REPAST DEVELOPERS GROUP (2005) 'RePast Home Page', http://repast.sourceforge.net.

SCHELLING, THOMAS C. (1978), Micromotives and Macrobehavior (New York: Norton).

SUN MICROSYSTEMS INC. (2002) 'The Java HotSpot(TM) Virtual Machine', http://java.sun.com/products/hotspot/docs/whitepaper/Java_Hotspot_v1.4.1/JHS_141_WP_d2a.pdf.

SUN MICROSYSTEMS INC. (2005) 'Java Technology Website', http://java.sun.com.

Return to Contents of this issue

© Copyright Journal of Artificial Societies and Social Simulation, [2005]