Introduction
Computational models have been influential in the COVID-19 policy response of national governments (Adam 2020). Specifically, analysis of modelled scenarios indicated that mitigation strategies would not work once significant community transmission had occurred (Ferguson et al. 2020; Holmdahl & Buckee 2020). Instead, most models suggested that whole-of-population social distancing would be required to substantially reduce the number of cases to a manageable level, with periodic suppression to maintain control until effective treatment or vaccination is available. Governments around the world have accordingly implemented a wide range of more or less stringent social distancing measures (Flaxman et al. 2020; Hale et al. 2020).
Policy oriented models of COVID-19 can be organised by general approach (Holmdahl & Buckee 2020). Classic statistical models (including GIT 2020; IHME 2020; Jombart et al. 2020) normally provide short term forecasts with uncertainty based on data series such as cases or hospital admissions. In contrast, mathematical models represent the mechanisms of disease transmission in the model rules and generate simulated epidemics based on those rules. Methods such as differential equations and system dynamics implement rules at the aggregate level (including Hill 2020; Noll et al. 2020; Struben 2020). In the epidemic context, these are referred to as compartmental models as they calculate the number of people in different epidemic states or compartments. Such models are deterministic as the mechanisms represent flows from one compartment to another, or rates of change in the epidemic as a whole. Microsimulation and agent-based models (including Davies et al. 2020; Ferguson et al. 2020; Gaudou et al. 2020; Mahmood et al. 2020) instead apply rules at the individual level and are stochastic. Each simulated individual is exposed to risk of transmission based on their interactions with simulated infectious people and progress through epidemic states probabilistically. Activities that give rise to such interactions are represented with various levels of fidelity, with detailed simulations incorporating transport routes and commuting behaviour or the location of different types of retail and leisure facilities.
Despite this abundance of models, as Squazzoni et al. (2020) highlighted in their call to action, the policy response to the COVID-19 pandemic revealed significant challenges in how computational modelling was used to address the public health crisis. Scientific modelling tools are well established in policy processes (Gilbert et al. 2018), but those practices bear little similarity to how these tools are being used in the COVID-19 response, by whom and with what goals.
In terms of how they are used, COVID-19 models were being quickly developed and employed in contrast to the "normal path of scientific progress, which is based on complex and delicate practices of peer review, testing, and replicability" (Squazzoni et al. 2020, 1.4). This conflict between scientific practice and the urgent need for inputs into a policy decision is not restricted to modelling, but also affects treatment research and broader social concerns such as the effect of school closures on the development and mental health of children. Further exacerbating potential concerns over the quality of models, individual case records and similar detailed information necessary for calibration (or for recognising poor models) are not generally available due to the extensive privacy regulations that apply to health data.
Experts during the early months of the COVID-19 pandemic "built simulation models to estimate disease trajectories ... to guide decision-makers to formulate policy measures to limit the epidemic, they [did not] cover the full behavioural and social complexity of societies under pandemic crisis" (Squazzoni et al. 2020). That is, the goals of these models were to motivate and guide blunt policy choices at the national level. As governments gain some control over the epidemic and lockdowns are eased, local actions and finer adjustments are more prominent in the policy debate. Model goals must be reconsidered to meet these needs.
Models for local planning
We believe that there is a gap for models that focus less on high-resolution representations of reality and more on the importance of justified stories – that is, models that organise empirical, theoretical and expert knowledge about the target system, and can then be used as a tool for thinking (Pidd 2009) about the social planning measures variously employed or argued for at the local level (e.g., shelter-in-place, exit-strategies, social distancing). Such a focus is grounded in the realities of a paucity of timely detailed data about both transmission and behaviour. For example, there is little benefit in building detailed transport models if the risk of transmission for different modes of transport is essentially unknown. There is also a limit to the value of national level models to local planners where the level of resolution or aggregation means relevant local dynamics cannot be observed.
In the United Kingdom (UK), national policies in an emergency situation are complemented by local actions coordinated through Local Resilience Forums or their equivalents. These Forums are formal partnerships between various public sector agencies including police and other emergency services, National Health Service trusts that provide health care, and local government (councils). Local Resilience Forums have a legal responsibility for emergency planning, response and recovery. The UK government collects and provides national data about tests, hospital admissions, deaths and similar key measures. Local Resilience Forums typically have limited access to data analysis support to extract information for the specific region of interest and prepare reports. As the epidemic has progressed, this is being partially alleviated with some publicly available models and analyses that present specific interpretations of such data, some of which include regional level reporting (for example, see Birrell et al. 2020; Gov.UK 2020a; Mishra et al. 2020).
Through our discussions with local planners, we realised that they sought models that could:
- provide them a suite of decision making options;
- help them think through the potential consequences of these actions;
- do this quickly;
- explore combinations of social distancing interventions that could be applied at different times;
- afford them richer scenarios that helped them answer the more locally or regionally focused questions they were concerned with;
- provide them a platform to evolve their planning as they learned from their previous efforts, successes and mistakes; and, finally,
- give them outputs and interpretations which are easy to understand and explore to support iterative exploration.
Implicit in all of these needs is the fact that local planners need models which can address their questions, rather than the questions that others, working at different scales or with different interests, are asking. In short, they wanted models that could help them understand their situation and the consequence of potential actions during a crisis period when the need for rapid decision making in the face of untrustworthy data was a significant concern. This paper documents our development of JuSt-Social, a model for local planners that deliberately emphasises justified stories as the output and goal of the model, rather than detailed predictions. In its name, ‘JuSt’ is short for ‘justified stories’, and ‘social’ denotes the focus on social policy interventions. Further, as local information becomes available, it can replace the national and international based knowledge encoded in the model.
The function of JuSt-Social is description (as defined in Edmonds et al. 2019), capturing what is known about the COVID-19 epidemic and the effect of social intervention policy options. While such a model does not include any knowledge that is not already available to the model user, the model provides access to a coherent and internally consistent summary of that knowledge. This coherence provides the stories aspect, the model generates sequences of events that are logically connected to each other through appropriate representation of causal mechanisms and theoretical understanding. These stories are justified through connection to empirical features of the real world and external consistency with empirical data in many forms.
Descriptive models such as JuSt-Social are particularly suitable for local scenario planning, where plausible futures are required to understand the potential consequences of policy options for regional stakeholders and their communities. Further interrogation of the model can also reveal aspects of the encoded knowledge such as how specific scenarios arise, highlighting gaps and inconsistencies in the represented knowledge and facilitating estimates of uncertainty.
We believe the model thus sits neatly between the needs of local planners, and the range of modelling options available (from simple aggregate curve-fitting of cases, hospitalisation and death numbers, through to high-resolution spatially-explicit agent-based models). There are, of course, many other plausible and sensible model goals, and other models; we hope this model gives readers a clear example of how an agent-based model can be developed quickly and purposefully to meet planners’ and policy makers’ needs, and contributes to the call made by Squazzoni et al. (2020) in this journal.
Sections 2 and 3 present the model construction, focusing on the way in which it incorporates knowledge and describes the relevant aspects of COVID-19 epidemics and policy options. In Section 4, we present two simple scenarios that highlight the benefits of the justified stories orientation. Section 5 discusses strengths and weaknesses of the model, and how these are appropriate given model goals, before Section 6 concludes.
JuSt-Social Model Design
Making description the primary focus of JuSt-Social has consequences for its design, including "direct and immediate connection with observation, data or experience" (Edmonds et al. 2019, 4.4). This includes knowledge of both the mechanisms by which relevant processes occur and the parameters that match the generic process model to the specific case being modelled.
Consistent with the descriptive purpose, the amount that a policy reduces contact is explicitly part of the scenario rather than emerging from economic, social or other endogenous factors. This allows the policy user to include their judgement of the effect of a policy intervention (and variability in that effect) as part of the evidence in the description. It also avoids confounding directly relevant knowledge about transmission with more contested theoretical frameworks about how these potential factors are related to personal behaviour (Jager 2017), which is particularly important when the situation is novel and any framework is both untested and difficult to parameterise.
Given our focus on describing the epidemic situation and potential responses, we also considered the consequences of the modelled policy scenarios on outcomes other than COVID-19 transmission to be out of scope. These are diverse, including economic contraction, unemployment, financial hardship, impact on delivery of planned health services, disruption of education for children, mental health issues such as anxiety and isolation, and strain on personal relationships (OECD 2020; ONS 2020a). Ongoing care for the proportion of people who acquire chronic health problems from COVID-19 is also excluded. While such considerations are clearly part of the broader decision making process, they are better examined with specific models that, for example, include details of the relevant social theory or local implementation of support policies.
JuSt-Social is therefore restricted to two distinct processes: transmission of SARS-CoV-2 coronavirus from infectious to susceptible people, and disease progression through a series of epidemic states once a susceptible person is exposed. These are the processes most directly relevant to the Local Resilience Forum as the number of infections and the translation of infection into hospital care and deaths are the drivers of health system and local planning stresses.
There are four broad policy options available in JuSt-Social (described in more detail below): bed blocking, social distancing, limiting movement, and symptomatic isolation. The first of these modifies the state transition process, and the others modify the transmission from infectious to susceptible people. Implementation details include intensity and timing, allowing complex scenarios that combine multiple policy initiatives. For scenario exploration, these details are managed through the interface (see Figure 1), with the user adjusting the intensity of the policies or adding or deleting strands as the simulated situation changes.
The remainder of this section describes how these processes are implemented in JuSt-Social, and how social interventions modify the model rules. Setting particular values to represent knowledge about COVID-19 is deferred to the Calibration section, so that the interlinked calibration processes can be described together. The only agents in the model are simulated persons, who progress through epidemic states, take action in response to social interventions, and infect each other. Internally, each time step (‘tick’ in NetLogo terminology) represents one day.
Epidemic state transition
COVID-19 follows the general SEIR epidemic transitions through susceptible (S), exposed (E), infectious (I), and removed (R) states. The final removed state comprises two mutually exclusive sub-states: recovered or dead. The model is constructed in such a way that the recovered agent can fail to acquire immunity with some probability and therefore return to the susceptible state but this probability is set to 0 for all scenarios presented in this article and the recovered state is referred to as immune for simplicity.
The infectious state is composed of sequential sub-states: infectious within the community, hospitalisation and critical care. An agent can transition from any one of these infectious states to either removed sub-state (immune or dead) without passing through more severe infectious states. However, there is no path from the more severe to less severe states. Whether an agent is symptomatic is modelled as its own state, but dependent on the infectious states in disease progression. See Figure 2 for a summary of the state progression process.
When JuSt-Social is initialised, one randomly selected agent is set to the exposed state and all others are susceptible. As an agent enters any state, there are two random draws. The first draw is to choose the next state. For example, as an agent enters the Community Infectious state, the agent draws for whether its next state is Immune, Hospital, or Dead. The second draw is for the duration of the current state, given the subsequent state, allowing different durations for different paths. For example, the length of stay in hospital differs between those who eventually survive and those who do not. The durations are drawn from weighted distributions that return a number of days or ticks. The default values for these probabilities and durations are set from published literature about the characteristics of COVID-19 and its treatment, as described in the Calibration section.
The bed blocking option assumes that a person who needs hospital admission but is unable to be admitted has a higher probability of death. This requires two settings: the number of beds available and the specific probability of death to apply if the beds are all occupied, that is, if the patient isn’t hospitalised when necessary. If switched on, an agent who is scheduled to be admitted to hospital first checks whether there is a bed available (comparing the number of beds to the total agents either in the hospital or critical care states) and, if not, draws against the bed blocked probability of death. If the draw is for death, then the number of days to death will be drawn from the distribution used for the number of days in hospital before transferring to critical care. If the draw is for survival, then the number of days to immunity is drawn from the same distribution as would have been used if the agent had been admitted to hospital (and survived). In either case, the agent’s epidemic state does not change (it remains infectious in the community), only the next state and when that transition is to occur.
The selection of the state following community infection differs according to risk status. At initialisation, a proportion of the agents (\(H\)) are assigned to the high-risk group. With \(R\) as the relative risk of those agents compared to the low-risk population and \(p\) as the combined probability of hospitalisation or death (\(p = P_H + P_{ID}\) in Figure 2), the risk adjusted probability of hospitalisation or death for each subpopulation is:
| \[\begin{aligned} \begin{split} \frac{p}{HR + 1 - H} & \quad \text{Low risk population} \\ \frac{Rp}{HR + 1 - H} & \quad \text{High risk population} \end{split}\end{aligned}\] | \[(1)\] |
Epidemic transmission
JuSt-Social uses NetLogo’s embedded spatial structure to transmit the epidemic from infectious to susceptible people. The world is composed of a grid of \(41 \times 41\) square patches with a side length of one unit of distance. At initialisation, twelve people are located on each patch. Each tick, some people not in hospital (and not dead) move one unit distance in a random direction and some other people move three units distance. The probabilities of moving are two of the user controls.
Each tick, each infectious agent has a probability of exposing every susceptible agent (assessed individually) on the same patch. The baseline transmission probability excluding interventions is identical for all pairs of agents and within all patches. Interventions either modify the transmission probability or influence the transmission process less directly.
For those interventions that include isolation, each isolated agent is excluded with some probability from the infectious and susceptible agents considered in the transmission process. The same probability applies regardless of why the agent is isolated. Agents in the hospitalised state are automatically considered to be in isolation and treated by the model rules in the same way as agents isolated through policy.
The social distancing options modify either the group of agents involved in the transmission process or the probability of transmission. Shielding of the high-risk population is implemented by setting the isolation flag for the proportion of agents assigned as high risk during initialisation, and treating them in the same way as any other isolated agent. The other three options include an additional parameter (\(r\)) for the reduction induced by social distancing, which operates slightly differently for each option. The ‘AllOrNone’ and ‘AllPeople’ options represent homogeneity and extreme heterogeneity in agent-centric transmission calculations. Each agent has a multiplier in the range \([0,1]\) that is applied to the transmission probability. Usually this multiplier is set to one and has no effect. However, the ‘AllPeople’ option sets the multiplier to \(1-r\) for all agents. In contrast, the ‘AllOrNone’ option sets the multiplier to 0 for some agents (\(r\) interpreted as a proportion), effectively precluding them from the transmission process, with one (no effect) for other agents. If high-risk shielding is also in effect, the high-risk population are in the group with multiplier set to zero. The ‘ByContact’ option uses a contact-centric mechanism to reduce transmission; each agent has their multiplier set to one, but the probability of transmission is instead reduced by \(r\).
Summarising these different social distancing approaches mathematically, given two agents on the same patch with epidemic status of infectious (in the community) and susceptible and baseline transmission probability \(\tau\), the susceptible agent becomes exposed with probability given by:
| \[\begin{aligned} P(\text{transmission}) = \begin{cases} \tau \times M_S \times M_I \quad &M_S, M_I \in \{0,1\} \quad\text{ for AllOrNone} \\ \tau \times M_S \times M_I \quad &M_S = M_I = 1 - r \quad\text{ for AllPeople} \\ \tau(1 - r) \times M_S \times M_I &M_S = M_I = 1 \quad\text{ for ByContact} \\ \end{cases} \\ \notag\end{aligned}\] | \[(2)\] |
The limiting movement option simply reduces the proportion of agents that move one or three units of distance. This intervention does not directly modify the transmission process, but nevertheless reduces epidemic spread as there is less mixing between infectious and susceptible agents.
The final social intervention included in the model concerns the extent to which agents isolate when they or one of their recent contacts becomes symptomatic. All infected people who are eventually admitted to hospital and some of those who remain in the community develop symptoms. With the specified probabilities, the symptomatic agent can choose to isolate and inform recent and future contacts of their symptomatic status. If the agent chooses to inform, there is a separate probability of being successful in identifying the contact and that agent choosing to isolate. Within the model, this is implemented by each agent keeping a list of all agents they infect and also the agent that infected them. Previously infected agents are informed (and potentially isolate) at the time at which their infecting agent becomes symptomatic and future exposures are informed at the time of exposure. Note that a full contact tracing programme may have a larger impact as such programmes require people to isolate even where transmission does not occur, at least until they have been tested.
Calibration
The calibration approaches for the two modelled processes of JuSt-Social substantially differed. The main parameters to calibrate regarding the progression of the disease fall into two broad groups: number of days in each state, and transition probabilities between states. As the epidemiological characteristics of COVID-19 are a major area of research, these parameters were mainly set from published literature. In contrast, there is little information available about transmission, such as the risk associated with different activities. The transmission parameters were therefore set to generate realistic epidemic curves. Model validation compared macro-phenomena generated in the model from the micro-level mechanisms and parameters to the equivalent macro-phenomena in the target system (that is, empirical UK epidemic measures). Note that the parameter values in the model are updated regularly as further data becomes available. While the calibration for JuSt-Social version 1.0 was conducted with UK level data as described here (and is unchanged for version 1.1), some parameter values have been updated in later versions of the model to reflect local data.
Disease characteristics
Five epidemic state transition probabilities are required in JuSt-Social (labels of the form \(P_x\) for arrows in Figure 2). These were derived from a mix of official statistics and research reports from administrative data. As the relevant parameters are all proportions, we used values from the most complete UK data available, ignoring the differences in periods and patients included in the sources (summarised at Table 2). This is equivalent to assuming that the transition probabilities are constant throughout the epidemic. Distributions are also required to allow each agent to draw a duration (number of days or ticks) for each state before progressing to the subsequent state (summarised at Table 1).
Length of stay (hospital duration) distributions are easily constructed from individual hospital records. While such data are not publicly available, summaries have been published in a variety of reports by researchers with access. The distributions for length of stay in critical care were constructed by digitising a figure of the cumulative proportion of patients discharged or surviving by days of care (ICNARC 2020 Figure 11). The same digitisation approach was used for length of stay for those not requiring critical care, from a report concerning COVID-19 admissions in three London hospitals (Perez-Guzman et al. 2020 Figure 2A). Note, however, that the report did not separate patients by critical care status, which is expected to particularly affect the duration for hospitalisation leading to death as death rates are higher in the critical care population.
Supplementing these four transition paths, the final distribution required for days in hospital is the number of days before transfer to critical care. This is reported as a median of 1 day (IQR 0,3) and mean of 2.4 days (standard deviation of 6.7 days), indicating an extremely skewed distribution (ICNARC 2020 Figure 11). In the model, we created a distribution with the same median and inter-quartile range, but it is much less skewed, with mean of 1.95 days (SD = 2.1). Distribution metrics are presented at Table 1, together with reported summary measures from the (unavailable) data underlying the used figures.
| Derived | ||||
| Distribution | Median | IQR | Reported | Source |
| Exposed to Community | Lognormal \(\mu = 1.621, \sigma = 0.418\) | Lauer et al. (2020 Table A1) | ||
| Community to Immune | 7-12 days (assume uniform) | ECDC (2020b) | ||
| Community to Hospital | 6 | 3,8 | Not available | 1 day + ECDC (2020a fig 4.1.1(a), UK) |
| Community to Dead | 7 | 4,9 | Not available | 2 days + ECDC (2020a fig 4.1.1(a), UK) |
| Hospital to Immune | 7 | 4,13 | Median stay 7 (IQR 6,8) | Perez-Guzman et al. (2020 fig 2A) |
| Hospital to Dead | 7 | 4,11 | To clinical deterioration 9 (IQR 6,17) | Perez-Guzman et al. (2020 fig 2A) |
| Hospital to Critical | 1 | 0,3 | Median 1 (IQR 0,3) but more skewed | ICNARC (2020 Table 2) |
| Critical to Immune | 12 | 5,24 | Median 11 (IQR 4,22) | ICNARC (2020 fig 11) |
| Critical to Dead | 10 | 5,18 | Median 9 (IQR 5,15) | ICNARC (2020 fig 11) |
There are fewer data available concerning duration in epidemic states in the community. For those agents admitted to hospital, we estimated their pre-admission infectious duration by digitising a plot of days from symptom onset to hospitalisation extracted from the European COVID-19 Surveillance Report (ECDC 2020a Figure 4.1.1(a), UK) and added one pre-symptomatic day. The same distribution was used for the infectious duration for those agents who died without admission, with an extra day to represent dying in the absence of admission to hospital.
The final two durations were modelled with standard random distributions. The European Centre for Disease Prevention and Control reported an infectious period of 7-12 days for moderate cases (ECDC 2020b) with no further information, so a uniform distribution was used. The largest available study of exposure periods fit a lognormal model with median 5.1 days (Lauer et al. 2020).
For transition probabilities, the ISARIC study (Docherty et al. 2020) provided the default values for the proportion of those hospitalised that require critical care (\(P_C = 0.17\)) and the proportion who died in hospital without transfer to critical care (\(P_{ID} = 0.30\)). It described the clinical characteristics of 20 133 patients admitted to hospital in the UK with COVID-19, approximately 34% of the initial admissions. While this study also reports the proportion who died in critical care, we used the more complete administrative data collected by the Intensive Care National Audit & Research Centre (ICNARC 2020).
As for durations, information about disease progression in the community is much more limited than for those patients requiring hospital care. We use reported community infection level (ONS 2020b) to estimate the proportion of infected people who are admitted to hospital (\(P_H\)) and, with death statistics (ONS 2020c), this allows estimates of those who die without admission (\(P_{ID}\)). This approach takes advantage of the consistency requirements between interdependent parameters so that the plausible range of each parameter can constrain and inform the range of the other. They are related by:
| \[\begin{aligned} P_{ID} &= F_H \frac{ \frac{\text{all deaths}}{\text{deaths in hospital}} - 1 }{\frac{1}{P_H} - 1} \\ \notag\end{aligned}\] | \[(3)\] |
From the probabilities calculated so far, the case fatality rate for COVID-19 for those hospitalised is 36.7%. Deaths in hospital account for 64.4% of deaths with COVID-19 listed on the death certificate (ONS 2020c, Figure 9 dataset for 29 May 2020 Bulletin). These parameter values therefore suggest \(P_{ID}\) and \(P_H\) are related to each other with: \(P_{ID} \approx 0.20 \times \frac{P_H}{(1 - P_H)}\). Further, the total fatality rate is given by:
| \[\begin{aligned} \begin{split} F &= P_{ID} + P_H \times F_H \\ &\approx 0.20 \, \frac{P_H}{(1 - P_H)} + 0.367 \, P_H \end{split}\end{aligned}\] | \[(4)\] |
The value for \(P_H\) (or any of the related parameters with which it can be constrained) is more speculative than other parameters in the model because number of people infected is highly uncertain. We matched the regular infection survey (ONS 2020b) estimates of the average infection numbers in two week periods in England with average daily admissions in England (Gov.UK 2020a). The admission period was lagged by 6 days to adjust for the duration in the infectious state prior to hospitalisation (Table 1). This procedure estimated \(P_H\) at 4.6% based on admissions from 10-23 May 2020, with 7.2%, 8.8% and 7.8% for the three following fortnights. While the earliest estimate is substantially different than the other three, it is also based on higher positive test rates than later surveys and we therefore retained it. The calculated average is \(P_H = 0.071\). This also gives the probability of death without admission as \(P_{ID} = 0.015\), with overall fatality rate of 4.1%.
| Transition | Value | Derivation summary | Source |
|---|---|---|---|
| \(P_H\) | 0.071 | Mean of: 0.046, 0.072, 0.088, 0.078 | ONS (2020b) and Gov.UK (2020a) |
| \(P_{ID}\) | 0.015 | \(0.2 \times \frac{0.071}{(1-0.071)}\) | |
| \(P_C\) | 0.17 | \(\frac{3001}{18183}\) | Docherty et al. (2020 pg 1) |
| \(P_{HD}\) | 0.30 | \((1-P_C) \times \frac{3954}{(3954+7203)}\) | Docherty et al. (2020 pg 7) |
| \(P_{CD}\) | 0.43 | \(\frac{3483}{8062}\) | ICNARC (2020 Table 9) |
Fitting transmission parameters
As transmission opportunities are completely defined by patch boundaries in JuSt-Social, the number of agents on a patch influences the speed of transmission. Two studies have examined peoples’ daily contact patterns from the specific perspective of sufficiency for respiratory disease spread. The POLYMOD (Mossong et al. 2008 Table 1) and BBC Pandemic (Klepac et al. 2020) studies reported 11.74 and 10.47 relevant contacts per day on average for the UK. The number of agents per patch was set to twelve so that the average number of contacts is eleven to match these studies. In practice, movement introduces variation in the number of contacts, with approximately 80% of the patches containing 8-16 people at any point in time.
Three other parameters are involved in the transmission process. The most direct of these is the probability of transmission by an infectious agent to each susceptible agent on the same patch. The other parameters control the proportion of agents moving one or three units of distance each day. Higher values of these movement parameters facilitate mixing and allow infected agents to access new susceptible agents. These three parameter values were set jointly so as to best replicate target epidemic behaviour.
The target epidemic is generated with an online deterministic model (Hill 2020) rather than empirical data concerning epidemic spread. This approach avoids the need for JuSt-Social to include epidemic management policies and changes in personal behaviour that may have different representation in the two models. That is, we avoid confounding calibration for the three parameters of interest with uncertainty in the representation and parameter values of other elements of the model. For similar reasons, we simplify the epidemic progression for both models and set our model with fixed durations to match the deterministic model. The simplest progression able to be modelled is to remove the possibility of death and minimise hospitalisation. The model settings are summarised at Table 3
| Target model parameter | JuSt-Social parameter | Value | Comment |
|---|---|---|---|
| Population size | people per patch, world size | 20172 | |
| Duration of incubation period | Duration of E state | 6 days | |
| Duration of mild infections | Duration of I in community | 9 days | |
| % of infections that are severe | Probability of hospitalisation | 1% | lowest setting |
| Duration of severe infection | Duration of I in hospital | 1 day | |
| % of infections that are severe | Probability of critical care | 0 | |
| Death rate for critical infections | Probability of death (all) | 0 | |
| Transmission rate: mild infections | 0.33/day | to be calibrated |
We used three measures that together characterise an epidemic curve to compare the epidemics modelled by JuSt-Social and the target deterministic model. The first concerns the width of the curve, or the duration of the epidemic. As stochastic variation is (relatively) larger when there are few events, we discard the early part of the curve and measure the days between five infections and peak prevalence. The second measure is the height of the curve, in this case peak prevalence. Finally, the curvature or the area under the curve is measured as the number of people ever infected. With the parameter values reported at Table 3, the deterministic model generated an epidemic with \(R_0 = 3.0\). Five people had been infected at day 16, peaking on day 118 with 3,518 currently infected. The final size was 18,929 people immune once the epidemic had run its course.
We used a full factorial design to generate epidemic curves with JuSt-Social throughout the parameter space for the three parameters being calibrated. Twenty values were sampled for probability of transmission: from 0.005 to 0.1 by 0.005. The proportion of the population moving the short distance (one unit of distance) was tested with 10 values: 0.05 to 0.5 by 0.05. For long movement (three units), seven values were sampled: 0, 0.025, 0.05, 0.075, 0.1, 0.15 and 0.2. While there are 1400 combinations of parameter values, those with a larger value for long distance population proportion than short distance were removed. Fifty simulations were run for each of the remaining 1180 parameter sets. Following the removal of any runs that did not achieve five infections, averages were constructed for each parameter set combination for each of three target measures.
To choose the combination of parameter values that provides the best match, we first identified those combinations on the three dimensional Pareto frontier and then selected subjectively among those candidates. The candidates on the Pareto frontier are objectively the best choices in the sense that improving the match for one measure can only be done by reducing the fit for another. This approach allows multiple criteria to be explicitly offset with knowledge of what is being gained and lost (Badham et al. 2017).
The three matching criteria are:
- the absolute difference between the target of 102 days and the average number of days between five cases and peak prevalence over the (up to) 50 JuSt-Social generated epidemics;
- the absolute difference between the target of 0.1744 and the average maximum prevalence over the (up to) 50 JuSt-Social generated epidemics; and
- the absolute difference between the target of 0.9384 and the average proportion of the immune proportion at the end of the simulation over the (up to) 50 JuSt-Social generated epidemics.
Each parameter value combination was assigned a level using the rPref package for R (Roocks 2016). Level 1 corresponds to the Pareto frontier, with level 2 assigned to those that would be on the Pareto frontier if the level 1 sets were removed, and so on. Nineteen parameter combinations were identified as level 1. As shown in Figure 3, JuSt-Social is unable to achieve a good match on both the peak prevalence and the number of days to reach that peak. We prioritised timing and excluded eleven candidates with a difference of at least 5 days in duration to peak. The remaining eight parameter sets are displayed at Table 4, together with their performance on the criteria. Of these, we selected parameter set 866 as it balanced the criteria and the parameter values are moderately placed within the ranges of the candidate parameter sets.
| Parameters | Criteria (difference) | |||||
| ID | \(\tau\) | Short | Long | Duration | Max prevalence | Final size |
| 326 | 0.030 | 0.20 | 0.15 | 0.5 | 0.052 | 0.004 |
| 745 | 0.025 | 0.35 | 0.20 | 0.7 | 0.039 | 0.042 |
| 846 | 0.030 | 0.40 | 0.10 | 0.6 | 0.051 | 0.002 |
| 866 | 0.030 | 0.40 | 0.15 | 3.1 | 0.040 | 0.001 |
| 986 | 0.030 | 0.45 | 0.10 | 2.5 | 0.050 | 0.002 |
| 1006 | 0.030 | 0.45 | 0.15 | 4.4 | 0.034 | 0.000 |
| 1126 | 0.030 | 0.50 | 0.10 | 0.1 | 0.055 | 0.001 |
| 1165 | 0.025 | 0.50 | 0.20 | 2.0 | 0.038 | 0.041 |
Justified Stories (Scenarios)
With the model constructed to best represent what is known about epidemic behaviour and the plausible mechanisms of social interventions, JuSt-Social can be used to generate justified stories. Two examples are presented here to demonstrate different types of justified story questions that were developed with local planners in the north-east of England. The first example recreates history so as to understand the current situation. This situation is then extended into the future by adding two judgements about people’s behaviour as examples of a different type of knowledge to be represented in the model.
Scenario description
While there is evidence of reduced contacts between people from about 9 March 2020 (Google 2020), we constructed the scenario to reflect key legislative dates about social distancing and advice on high risk shielding and what can be understood about the current situation following those restrictions. The timeline is lockdown and shielding from 23 March 2020, with increased activity level on 5 May, and again on 15 June then 4 July (including end of shielding). Easing of social distancing is accompanied by introduction of a contact tracing programme with monitored compliance. The date of interest for ‘now’ is 13 July, the date at which almost all general restrictions were lifted (Dunn et al. 2020). This should not be interpreted as ending social distancing as various protective measures remained in place including contact tracing, local restrictions in response to outbreaks, post travel isolation, and cleaning and distancing procedures in public places.
As JuSt-Social is stochastic, the number of days varies between runs for the epidemic to take hold in the population. Hospital admissions are the most reliable data for guidance; cases are not observable, and deaths are both less frequent than admissions and later in the disease process. There had been 6,794 COVID-19 admissions in the UK by 23 March 2020 (Gov.UK 2020a), equivalent to 2.02 admissions for the model population of 20,172 agents. However, using two admissions as the trigger would have been very volatile, with substantial variation in the number of cases generating two admissions. Over 1000 simulation runs with no intervention, we found the second admission occurred with an average of 113 cases and the third with an average of 167 cases. While this suggests that an appropriate trigger would be between 113 and 167 cases, we found that the simulations triggered in this range did not provide sufficient pre-intervention epidemic spread and consistently underestimated both admissions and the slowing of growth with the introduction of social distancing. The scenario used 200 cases as a trigger for the start of the interventions and therefore as equivalent to 23 March 2020.
Additional scenario settings balanced representation of changes in the target system with scenario simplicity. In the first week following the Prime Minister’s announcement of the lockdown on 23 March (officially referred to as business closure and stay-at-home orders), activity associated with essential retail dropped by about 30% and work, transport and leisure activities by about 70%, with a concomitant increase in time spent at home (Google 2020). Rather than distinct jumps with regulation changes, these activity levels gradually moderated over the following months, to only 10% reduction for essential retail and 40% for the other activity types by mid July. The government contact tracing scheme was introduced on 28 May (supplementing informal notification by symptomatic people to their friends). For the week of 13 July, 80% of people with positive tests who were managed by the contact tracing scheme were reached, and of these 80% provided information about at least one contact and 78% of these nominees were reached (Gov.UK 2020b). The scenario capturing these changes is described at Table 5.
| Date | Distancing | Shielding | Self-Isolators | Traced | Comment |
|---|---|---|---|---|---|
| 23 Mar 20 | 0.50 | Yes | 0 | 0 | Initialised at 200 cases |
| 04 May 20 | 0.45 | Yes | 0.5 | 0.2 | |
| 15 Jun 20 | 0.35 | Yes | 0.5 | 0.5 | Official tracing programme |
| 03 Jul 20 | 0.25 | No | 0.5 | 0.5 | End shielding |
| 10 Jul 20 | 0.25 | No | 0.5 | 0.4 or 0.6 | Future scenarios |
There is increasing official emphasis on the need for an effective contact tracing system to manage the epidemic as schools, offices and the hospitality industry reopen. We therefore focused on this aspect for the projection element of the scenarios, comparing the effect of 40% and 60% traced and isolating. These represent two plausible paths for the the future success of contact tracing: increase through experience and process improvements, or decrease as continued outbreaks lead to distrust.
Results
We ran 2000 simulations with the scenario settings specified at Table 5. Of these, 181 were excluded as the epidemic never reached the scenario trigger of 200 cases. A further 208 ended before the ‘now’ date of 13 July and were therefore excluded as implausible simulations of the real-world situation. Of the remaining 1611 runs, 821 had 60% successfully tracked and self-isolating when exposed to a symptomatic case from 10 July and 790 were set to 40%.
From Figure 5, the scenario is able to generate a good qualitative match with UK hospital admissions, with a sharp (albeit low) peak on the correct date and a curved decline to flat admissions in July 2020. Overall, the model underestimates admissions slightly, as is evident from cumulative admissions at Figure 5.
Nevertheless, the match is close enough to use it to understand uncertainty in the current situation. The mean prevalence (including exposed but not yet infectious) in the simulations at 13 July 2020 is 0.067% of the population, and median is 0.053% with interquartile range of 0.025% to 0.094%. Restricting to the 874 simulations within 15% of the target cumulative admissions increases these estimates slightly to 0.070% for mean prevalence with interquartile range of 0.03% to 0.99%.
This example uses dynamic information about hospital admissions (Figure 5), dynamic intervention detail (Table 5) and clinical characteristics (Tables 1 and 2) to support prevalence estimates at 13 July 2020, extrapolating from what is known about the current situation to make estimates of something unknown but important. Prevalence is of particular concern when considering different policy options, and local policy makers had no information available due to the limited testing at the time. While there is substantial uncertainty in these prevalence estimates, the range can be used as assumptions for broader planning.
Both scenarios for future contact tracing effectiveness result in several months of slowly increasing hospital admissions. The simulated epidemic for the 40% traced scenario peaks in March 2021 at a level about 50% higher than the first wave, while the peak for the 60% scenario is both lower and later (not shown). Using the model has forced the planner to specify their assumptions, in this case about the realistic capability of contact tracing and compliance with isolation orders. Even the scenario with improved contact identification and compliance is unable to control the epidemic in the simulations, although growth would be slow. The model thereby supports a motivational purpose, and stimulated a planning discussion about supplementary policies such as what support would be required to allow people to isolate if requested.
Unlike the good match in admissions, the model consistently overestimates deaths (see Figure 6). Note that this cannot be interpreted as simple under-reporting of real-world deaths because the definition of deaths used in the calibration process is the same as for the comparison. That is, any reporting differences should cancel out.
Finally, we also reported newly exposed agents each day (see Figure 7). The effect of the simulated lockdown at 23 March 2020 is clearly visible as a dramatic drop in the exposures. While there is another week of increases in exposure following that drop, the growth is slower (change in slope) and the exposures then start reducing. The slight easing on 4 May is visible as a small spike.
Discussion
JuSt-Social integrates many different types of knowledge, and such organisation of knowledge is one aspect of a descriptive model (Edmonds et al. 2019). All models simplify the target system, excluding some real-world features completely and replacing others with a less detailed alternative that is intended to reproduce relevant aspects of the real-world behaviour. The simplified description encoded in JuSt-Social is used for decision support, by generating justified stories that assist policy planners to understand the COVID-19 situation and the plausible outcomes of policy options.
Clinical characteristics of COVID-19 such as the epidemic state changes and the distribution of number of days in each state are represented realistically in model processes and parameters. Many of these characteristics are measurable and the corresponding parameters are set to the appropriate values. Others are not included in the model at all, such as variability in infectiousness throughout the infectious period.
While still empirically based, the transmission process in JuSt-Social is idealised, reflecting the limited knowledge about how transmission relates to time since infection, proximity, activity and contact patterns. It is technically feasible to map the location of schools, workplaces, transport infrastructure and other relevant public places, and also to assign simulated people to household and activities. However, such a design would be under-determined. That is, the resolution used to describe mechanisms of transmission would need to be matched with detail about the associated risk of transmission in each of these environments to appropriately connect patterns of activity to the process of epidemic spread. Such hazard information is currently unknown for COVID-19 and the data available are insufficient to constrain the large number of parameters that would need to be calibrated to distinguish between different transmission paths.
Instead, we used a simplified transmission process and calibrated the three parameters to match a standard SEIR compartment model that had been tuned to the available COVID-19 epidemiological knowledge. That reference compartmental model captures the infectivity, contact patterns and population behaviour of COVID-19 in a real population. The parameter values are therefore empirically justified indirectly, the macro-behaviour is realistic but specific processes and parameters do not correspond to features of the real-world epidemic. Further, this transmission process neglects features of real world epidemics such as age specific mixing patterns, and instead implicitly assumes that the population being modelled has a similar pattern as the population over which the reference SEIR model had been fitted.
In contrast, the social intervention mechanisms are theoretically based. This reflects their hypothetical nature, responding to questions such as ‘how would epidemic spread change if people reduced their contact with each other?’. While the theory is not deep (reduced contact leads to reduced transmission), this association reflects theoretical knowledge about how airborne pathogens spread between people. That theoretical understanding is then applied in combination with expert knowledge in the form of judgements about behavioural responses to specific policy options. Observational data such as hospital admissions over time provides feedback on that judgement.
Once the knowledge has been organised in JuSt-Social, it can be used to apply that knowledge for various purposes. While the model does not include any information not already known to the modeller, the synthesis and organisation makes the knowledge more accessible and allows us to explore in detail some of its consequences.
Two examples were presented in the results. The model was used to estimate a plausible range for prevalence at a specific date, which could then be included in planning assumptions. In the second example, the simulations were extrapolated into the future under two different scenarios about the effectiveness of contact tracing, a key policy tool for future epidemic management. The failure of the optimistic scenario to control the epidemic stimulated a discussion about additional measures that could be taken.
One outcome of organising knowledge in a model is that all the pieces of knowledge are connected. Consequently, inconsistencies in that knowledge are more visible. Our model underestimates deaths (Figure 6) despite providing reasonable values for hospital admissions (Figure 5). There is little information to estimate the proportion of infections leading to admission (\(P_H\)), but increasing this so as to better estimate deaths would simultaneously lead to overestimation of admissions. One potential explanation is that the parameter values are derived from the initial COVID-19 cases in the UK, and therefore do not reflect any changes in treatment protocol or characteristics of the infected population such as age profile. However, what is important here is not the source of the discrepancy, but that the model highlights such discrepancies.
Connected knowledge also encourages complex systems thinking, particularly when presented dynamically as in an agent-based model. Consider exposures over time (Figure 7), there is a clear and substantial decrease when lockdown starts but exposures continue to increase before decreasing about a week later. Why? At the time of lockdown, there are many people who have been exposed but are not yet infectious. They become infectious and start exposing others, but the new exposures are at a slower rate than before the lockdown (as can be seen by the change in slope). Each day, the number of newly infectious increases because the epidemic had been growing prior to lockdown, so the number of people they are able to expose also increases initially. Hospital admissions also continue to rise after lockdown (see Figure 4) but there is no dramatic decline because the variation in durations smooths the curves. The connection between amounts, rates of change and delays can be difficult to understand even for people with a strong mathematical grounding (Sweeney & Sterman 2000). The model allows users to develop experience with potentially counterintuitive behaviour of the represented system.
Conclusion
There remains tremendous need for planning tools that respond to local needs, to assist the National Health Service Trusts, County Councils and other regional organisations as the UK government shifts focus from interrupting the epidemic crisis to longer term management. We presented JuSt-Social: a descriptive model of the COVID-19 epidemic, calibrated to the UK situation. The model organises and synthesises various types of knowledge, including official statistics, observed clinical characteristics, theoretical understanding of the ways in which interventions interfere with transmission, and expert assessment of responses to interventions.
While ‘description’ is an appropriate category for the proximate model purpose (Edmonds et al. 2019), the value of the model arises from the way in which it allows the organised knowledge to be accessed. The model generates justified stories, or sequences of events that are both internally coherent and consistent with available evidence. While ‘what if’ scenarios have long been a mainstay of simulation use (Gilbert et al. 2018; Mahmoud et al. 2009), our use of justified stories terminology is to emphasise the two principles that make such scenarios useful in the context of local epidemic planning. Stories are not mere sequences of events, they must be internally consistent. In a good descriptive simulation, model rules summarise the available knowledge and coherence follows from the model rules. Similarly, the generated stories are justified by the evidence that contributes to the knowledge that is described by the model.
Our implementation of a model for local COVID-19 planning demonstrates how the concept of justified stories supports computational models that are useful during a global pandemic. The justified stories support the ultimate purpose of the model from the perspective of local planners, examining plausible outcomes of policy options. This decision support is direct, helping planners to compare options (Mahmoud et al. 2009). However, justified stories also contribute to other aspects of the policy process, including understanding of uncertainty about the current situation, highlighting inconsistencies in the knowledge, and gaining experience with complex system behaviour (Barbrook-Johnson et al. 2016; Gilbert et al. 2018). COVID-19 decisions may need to be made quickly with incomplete information because delaying the decision is effectively a decision to take no action. Such contributions therefore support decisions indirectly, by helping planners to better understand their policy environment and therefore deal with these limitations.
The local planners have become more deeply engaged with the model over time. Initially we were providing standalone reports on policy questions such as the examples provided in this article. More recently, questions are developed cooperatively, including regular discussions about new information and scenario assumptions, and the model output is integrated into the planning dashboard available to decision makers. With substantial time pressures on local planners, this high level of engagement demonstrates the value of the justified stories approach in their decision making.
Model Documentation
JuSt-Social code and documentation is available from CoMSES OpenABM at https://doi.org/10.25937/119s-yx54 (version 1.1 used in this study).Acknowledgements
We would like to acknowledge the important assistance of Dr Ruth Meyer (Centre for Policy Modelling, Manchester Metropolitan University) and Dr Corinna Elsenbroich and Dr Kavin Narasimhan (both from Centre for Research in Social Simulation, University of Surrey) who tested various draft versions of the model and made valuable development suggestions. We would also like to thank the reviewers for insightful questions and comments that helped us to improve the article. This work was supported by the Medical Research Council [grant number MC_PC_18045] and the Economic and Social Research Council [grant title "Agent-based modelling of COVID-19 in support of NHS Trusts and Councils in the North East of England"]. PBJ is supported by a UKRI Innovation Fellowship funded by the Economic and Social Research Council [grant number ES/S000402/1]. Financial support for the publishing of this article came, in part, from an IBM grant for supporting open access COVID-19 research.References
ADAM, D. (2020). Special report: The simulations driving the world’s response to COVID-19. Nature, 580, 316–318. [doi:10.1038/d41586-020-01003-6]
BADHAM, J., Jansen, C., Shardlow, N., & French, T. (2017). Calibrating with multiple criteria: A demonstration of dominance. Journal of Artificial Societies and Social Simulation, 20(2), 11: https://www.jasss.org/20/2/11.html. [doi:10.18564/jasss.3212]
BARBROOK-JOHNSON, P., Badham, J., & Gilbert, N. (2016). Uses of agent-based modeling for health communication: The TELL ME case study. Health Communication, 32(8), 939–944. [doi:10.1080/10410236.2016.1196414]
BIRRELL, P., Blake, J., van Leeuwen, E., MRC Biostatistics Unit COVID-19 Working Group, & Angelis, D. de. (2020). COVID-19: Nowcast and forecast: https://www.mrc-bsu.cam.ac.uk/now-casting/.
DAVIES, N. G., Kucharski, A. J., Eggo, R. M., Gimma, A., Edmunds, W. J., Jombart, T., O’Reilly, K., Endo, A., Hellewell, J., Nightingale, E. S., Quilty, B. J., Jarvis, C. I., Russell, T. W., Klepac, P., Bosse, N. I., Funk, S., Abbott, S., Medley, G. F., Gibbs, H., … Liu, Y. (2020). Effects of non-pharmaceutical interventions on COVID-19 cases, deaths, and demand for hospital services in the UK: A modelling study. The Lancet Public Health, 5(7), 375–385. [doi:10.1016/s2468-2667(20)30133-x]
DOCHERTY, A. B., Harrison, E. M., Green, C. A., Hardwick, H. E., Pius, R., Norman, L., Holden, K. A., Read, J. M., Dondelinger, F., Carson, G., Merson, L., Lee, J., Plotkin, D., Sigfrid, L., Halpin, S., Jackson, C., Gamble, C., Horby, P. W., Nguyen-Van-Tam, J. S., … Semple, M. G. (2020). Features of 20 133 UK patients in hospital with COVID-19 using the ISARIC WHO Clinical Characterisation Protocol: Prospective observational cohort study. BMJ, 369, m1985. [doi:10.1136/bmj.m1985]
DUNN, P., Allen, L., Cameron, G., & Alderwick, H. (2020). COVID-19 policy tracker: A timeline of national policy and health system responses to COVID-19 in England: https://www.health.org.uk/news-and-comment/charts-and-infographics/covid-19-policy-tracker. [doi:10.1136/bmj.m1937]
ECDC. (2020a). COVID-19 Surveillance Report: Week 22, 2020: https://covid19-surveillance-report.ecdc.europa.eu/.
ECDC. (2020b). Q & A on COVID-19: When is a person infectious?: https://www.ecdc.europa.eu/en/covid-19/questions-answers.
EDMONDS, B., Le Page, C., Bithell, M., Chattoe-Brown, E., Grimm, V., Meyer, R., Montañola-Sales, C., Ormerod, P., Root, H., & Squazzoni, F. (2019). Different modelling purposes. Journal of Artificial Societies and Social Simulation, 22(3), 6: https://www.jasss.org/22/3/6.html. [doi:10.18564/jasss.3993]
FERGUSON, N., Laydon, D., Nedjati Gilani, G., Imai, N., Ainslie, K., Baguelin, M., Bhatia, S., Boonyasiri, A., Cucunuba Perez, Z., Cuomo-Dannenburg, G., Dighe, A., Dorigatti, I., Fu, H., Gaythorpe, K., Green, W., Hamlet, A., Hinsley, W., Okell, L., van Elsland, S., … Ghani, A. (2020). Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand: https://www.imperial.ac.uk/mrc-global-infectious-disease-analysis/covid-19/report-9-impact-of-npis-on-covid-19/. [doi:10.12688/wellcomeopenres.15805.1]
FLAXMAN, S., Mishra, S., Gandy, A., Unwin, H. J. T., Mellan, T. A., Coupland, H., Whittaker, C., Zhu, H., Berah, T., Eaton, J. W., Monod, M., Ghani, A. C., Donnelly, C. A., Riley, S., Vollmer, M. A. C., Ferguson, N. M., Okell, L. C., & Bhatt, S. (2020). Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe. Nature, 584(7820), 257–261. [doi:10.1038/s41586-020-2405-7]
GAUDOU, B., Huynh, N. Q., Philippon, D., Brugiere, A., Chapuis, K., Taillandiar, P., Larmande, P., & Drogoul, A. (2020). COMOKIT: A modeling kit to understand, analyze and compare the impacts of mitigation policies against the COVID-19 epidemic at the scale of a city. Frontiers in Public Health, 8, 563247. [doi:10.3389/fpubh.2020.563247]
GILBERT, N., Ahrweiler, P., Barbrook-Johnson, P., Narasimhan, K. P., & Wilkinson, H. (2018). Computational Modelling of Public Policy: Reflections on Practice. Journal of Artificial Societies and Social Simulation, 21(1), 14: https://www.jasss.org/20/1/14.html. [doi:10.18564/jasss.3669]
GIT. (2020). DeepCOVID: https://deepcovid.github.io/.
GOOGLE. (2020). COVID-19 Community Mobility Reports: https://www.google.com/covid19/mobility/.
GOV.UK. (2020a). Coronavirus (COVID-19) in the UK Dashboard: https://coronavirus.data.gov.uk/.
GOV.UK. (2020b). NHS Test and Trace statistics (England): Weekly reports: https://www.gov.uk/government/collections/nhs-test-and-trace-statistics-england-weekly-reports.
HALE, T., Webster, S., Petherick, A., Phillips, T., & Kira, B. (2020). Oxford COVID-19 Government Response Tracker: https://www.bsg.ox.ac.uk/research/research-projects/coronavirus-government-response-tracker. [doi:10.1101/2020.07.04.20145334]
HILL, A. (2020). Modeling COVID-19 Spread vs Healthcare Capacity: https://alhill.shinyapps.io/COVID19seir/.
HOLMDAHL, I., & Buckee, C. (2020). Wrong but useful - What Covid-19 epidemiologic models can and cannot tell us. The New England Journal of Medicine, 383(4), 303–305. [doi:10.1056/nejmp2016822]
ICNARC. (2020). ICNARC report on COVID-19 in critical care: 29 May 2020. ICNARC 2020: https://www.icnarc.org/Our-Audit/Audits/Cmp/Reports.
IHME. (2020). COVID-19 projections: https://covid19.healthdata.org/.
JAGER, W. (2017). Enhancing the realism of simulation (EROS): On implementing and developing psychological theory in social simulation. Journal of Artificial Societies and Social Simulation, 20(3), 14: https://www.jasss.org/20/3/14.html. [doi:10.18564/jasss.3522]
JOMBART, T., Nightingale, E. S., Rees, E. M., Knight, G. M., Pearson, C. A. B., & Clifford, S. (2020). Forecasting critical care bed requirements for COVID-19 patients in England. medRxiv preprint: https://cmmid.github.io/topics/covid19/current-patterns-transmission/ICU-projections.html.
KLEPAC, P., Kucharski, A. J., Conlan, A. J. K., Kissler, S., Tang, M., Fry, H., & Gog, J. R. (2020). Contacts in context: Large-scale setting-specific social mixing matrices from the BBC Pandemic project. medRxiv preprint. [doi:10.1101/2020.02.16.20023754]
LAUER, S. A., Grantz, K. H., Bi, Q., Jones, F. K., Zheng, Q., Meredith, H. R., Azman, A. S., Reich, N. G., & Lessler, J. (2020). The incubation period of coronavirus disease 2019 (COVID-19) from publicly reported confirmed cases: Estimation and application. Annals of Internal Medicine, 172(9), 577–582. [doi:10.7326/m20-0504]
MAHMOOD, I., Arabnejad, H., Suleimenova, D., Sassoon, I., Marshan, A., Serrano-Rico, A., Louvieris, P., Anagnostou, A., Taylor, S. J. E., Bell, D., & Groen, D. (2020). FACS: A geospatial agent-based simulator for analysing COVID-19 spread and public health measures on local regions. Journal of Simulation, 1–19. [doi:10.1080/17477778.2020.1800422]
MAHMOUD, M., Liu, Y., Hartmann, H., Stewart, S., Wagener, T., Semmens, D., Stewart, R., Gupta, H., Dominguez, D., Dominguez, F., Hulse, D., Letcher, R., Rashleigh, B., Smith, C., Street, R., Ticehurst, J., Twery, M., Delden, H. van, Waldick, R., … Winter, L. (2009). A formal framework for scenario development in support of environmental decision-making. Environmental Modelling & Software, 24(7), 798–808. [doi:10.1016/j.envsoft.2008.11.010]
MISHRA, S., Scott, J., Zhu, H., Ferguson, N., Bhatt, S., Flaxman, S., & Gandy, A. (2020). COVID-19 UK: https://imperialcollegelondon.github.io/covid19local.
MOSSONG, J., Hens, N., Jit, M., Beutels, P., Auranen, K., Mikolajczyk, R., Massari, M., Salmaso, S., Tomba, G. S., Wallinga, J., Heijne, J., Sadkowska-Todys, M., Rosinska, M., & Edmunds, W. J. (2008). Social contacts and mixing patterns relevant to the spread of infectious diseases. PLoS Medicine, 5(3), e74. [doi:10.1371/journal.pmed.0050074]
NOLL, N. B., Aksamentov, I., Druelle, V., Badenhorst, A., Ronzani, B., Jefferies, G., Albert, J., & Neher, R. (2020). COVID-19 scenarios: An interactive tool to explore the spread and associated morbidity and mortality of SARS-CoV-2. medRxiv preprint. [doi:10.1101/2020.05.05.20091363]
OECD. (2020). OECD Economic Outlook: Volume 2020 Issue 1: Preliminary version: http://www.oecd.org/economic-outlook/.
ONS. (2020a). Coronavirus and the social impacts on Great Britain: https://www.ons.gov.uk/.
ONS. (2020b). Coronavirus (COVID-19) Infection Survey pilot: 18 June 2020: https://www.ons.gov.uk/.
ONS. (2020c). Deaths registered weekly in England and Wales, provisional: https://www.ons.gov.uk/.
PEREZ-GUZMAN, P. N., Daunt, A., Mukherjee, S., Crook, P., Forlano, R., Kont, M. D., Lochen, A., Vollmer, M., Middleton, P., Judge, R., Harlow, C., Soubieres, A., Cooke, G., White, P. J., Hallett, T. B., Aylin, P., Ferguson, N., Hauck, K., Thursz, M., & Nayagam, S. (2020). Clinical characteristics and predictors of outcomes of hospitalised patients with COVID-19 in a London NHS Trust: A retrospective cohort study. Clinical Infectious Diseases, ciaa1091. [doi:10.1093/cid/ciaa1091]
PIDD, M. (2009). Tools for Thinking. Hoboken, NJ: John Wiley & Sons.
ROOCKS, P. (2016). Computing Pareto Frontiers and Database Preferences with the rPref Package. The R Journal, 8(2), 393. [doi:10.32614/rj-2016-054]
SQUAZZONI, F., Polhill, J. G., Edmonds, B., Ahrweiler, P., Antosz, P., Scholz, G., Chappin, E., Borit, M., Verhagen, H., Giardini, F., & Gilbert, N. (2020). Computational models that matter during a global pandemic outbreak: A call to action. Journal of Artificial Societies and Social Simulation, 23(2), 10: https://www.jasss.org/23/3/10.html. [doi:10.18564/jasss.4298]
STRUBEN, J. (2020). The coronavirus disease (COVID-19) pandemic: Simulation-based assessment of outbreak responses and post peak strategies. System Dynamics Review, 36(3), 247-293. [doi:10.1101/2020.04.13.20063610]
SWEENEY, L. B., & Sterman, J. D. (2000). Bathtub dynamics: Initial results of a systems thinking inventory. System Dynamics Review, 16(4), 249–286. [doi:10.1002/sdr.198]