Agent-Based Modelling of Social-Ecological Systems: Achievements, Challenges, and a Way Forward

Introduction

Social-ecological systems (SES) describe the tight coupling of social and ecological systems: ecosystems are affected by humans and in turn provide societies with ecosystem services and are thus the basis of human well-being (Berkes & Folke 1998; Folke et al. 2010). These interactions are continuously changing due to feedbacks and internal or external factors, and they take place across different temporal and spatial scales, making SES highly dynamic systems (Liu et al. 2015). Understanding the dynamics of SES is therefore crucial to support sustainable management of resources to ensure ecosystem integrity (Carpenter et al. 2009; Ostrom 2009; Chapin et al. 2010; Schlüter et al. 2012; Levin et al. 2013).

SES are composed of individual decision-making agents able to follow their goals (Levin et al. 2013). These are organisms, which range from microbes in the soil to the largest plants and animals, as well as humans and their institutions. They are usually different, interact with each other locally or via networks, and are trying to achieve a certain goal by adapting their behaviour to the current states of both themselves and their living and non-living environments. In SES, human agents are at the core of interest: they affect the system dynamics in different ways, for example, by resource use decisions or by policy intervention.

Agent-based models (ABMs) (in ecology also referred to as individual-based models) are a natural and promising tool for improving understanding and increasing our ability to predict and successfully manage SES (Gilbert and Troitzsch 2005; Grimm and Railsback 2005; Squazzoni 2012). ABMs have thus been widely used to represent and analyse land use/land cover change (Parker et al. 2003; Matthews et al. 2007), ecosystem or environmental management (Bousquet and Le Page 2004; Kelly et al. 2013; Le Page et al. 2013), and collective actions in common pool resource problems (Poteete et al. 2010; Schill et al. 2016).

ABMs tend, by the very nature of the systems they represent, to be complex, which makes them difficult to parameterize and analyse (e.g., Smajgl et al. 2011; Lee et al. 2015). It is thus important to critically reflect upon the state-of-the-art in ABMs of SES. A key question is whether ABMs should be mainly viewed as a heuristic tool to explore ideas, gain system understanding, and test hypotheses or whether they can also serve as a management and decision support tool for specific case studies. Moreover, what is the state-of-the-art for both types of models? Is the potential of modelling fully exploited, and if not, what is missing?

To answer these questions, we reviewed current achievements and open challenges in social-ecological agent-based modelling in order to provide suggestions for future directions in this field. As a reference, we considered the development of ecological modelling over the last two decades to possibly learn from this field with its long history. It should be kept in mind, though, that the main difference between agent-based modelling of ecological and social-ecological systems is that representing the decision-making of humans is far more complex than representing the decision-making of animals or plants (Müller et al. 2013).

In ecology, individual-based modelling started to be widely used approximately ten years earlier than agent-based modelling in social sciences, i.e., around 1990 rather than 2000 (C. Vincenot, pers. communication). The same key question about the scope of ABMs in ecology was already asked more than 15 years ago (Grimm 1999). The state-of-the-art at that time was considered incoherent and inefficient. Therefore, general approaches for model descriptions (Grimm et al. 2006; Grimm et al. 2010) and making models more structurally realistic (Grimm et al. 2005; Grimm and Railsback 2012) were developed and increasingly used. Furthermore, achievements have been made concerning model development and analysis (Railsback and Grimm 2012; Grimm and Berger 2016a). Therefore, it is worthwhile to try to make use of established concepts and methodological achievements in ecological modelling for social-ecological modelling.

A further development to put forward good practice in ecological modelling is TRACE (TRAnsparent and Comprehensive model Evaludation), a framework for documenting the development and testing of ecological models (Schmolke et al. 2010; Grimm et al. 2014). It follows the "modelling cycle" (Grimm and Railsback 2005): problem formulation, defining the model purpose, model description, model development, analysis, and testing. For testing, the question of how well a model represents a real system is usually related to the issues of evaluation, verification, and validation. However, the terminology in this field is confusing.

Independent prediction usually is not possible at the system level for agent-based complex systems, as we cannot perform controlled experiments, cannot know the future type and dynamics of drivers, and cannot wait for years or decades to see whether our predictions are true. Still, theories can be tested at the level of the behaviour of agents, and structural realism can be established by making models reproduce multiple observed patterns ("pattern-oriented modelling"; Grimm et al. 2005; Grimm and Railsback 2012). Additionally, a particular challenge for model evaluation of agent-based modelling is the quality assessment of the representation of decision-making, particularly when modelling complex decisions made by human agents (An 2012; Müller et al. 2013).

Augusiak et al. (2014) put the different elements of model evaluation, verification, and validation into a coherent framework. Evaluation in terms of quality assurance should relate to all elements of the model development, which is an iterative process. In this "modelling cycle", Augusiak et al. (2014) distinguish the following elements: data evaluation, conceptual model evaluation, implementation verification, model output verification, model analysis, and model output corroboration (Table 1, Figure 1). Augusiak et al. (2014) suggest the new term "evaludation", a merger of "evaluation" and "validation", for these steps. The term validation can no longer be reserved for any practical purpose, as it has been given virtually any possible and contradicting meaning in the literature (Augusiak et al. 2014). Grimm et al. (2014) included the "evaludation" steps within the TRACE framework (Schmolke et al. 2010) to document the full modelling cycle (Table 1). TRACE documents are designed to document, using a standardized format and terminology, that a model was "thoughtfully designed, correctly implemented, thoroughly tested, well understood, and appropriately used for its intended purpose" (Grimm et al. 2014, p.129).

Here, we use the TRACE framework including the evaludation scheme (Table 1) as the basis for our attempt to answer the key question: where does social-ecological agent-based modelling currently stand regarding the different elements of model evaludation? Specifically, we address the following questions about the use of ABMs as a scientific method: What is the model’s purpose? Is its design ad hoc or based on generic principles? How is human decision-making represented? Are qualitative and/or quantitative data used to develop and parameterize the model? Has it and its rationale been thoroughly documented? Is it reproducible? Has its implementation been tested? Is its behaviour well understood? How much calibration does it include? Does it deliver any testable new predictions?

To answer these questions, we first checked whether and how these issues are addressed in reviews and method papers of social-ecological ABMs. Then, in a mini-review of published ABMs of SES, we evaluated whether and to what degree the elements of model evaludation were addressed in publications on specific ABMs. The purpose of these two reviews is not to distinguish ABMs in this field into "good" or "bad" but to document the current awareness and practice regarding important elements of model development and use. The purpose of our review is to identify poorly developed elements in the current culture of agent-based modelling of SES, suggest improvements, and thereby contribute to the maturation of agent-based modelling of SES.

Methods

Review of reviews

We assessed reviews and method papers on social-ecological ABMs and extracted information on achievements, open challenges and ways ahead along the following categories suggested by the TRACE framework (Grimm et al. 2014): problem formulation/purpose, model description, data evaluation, conceptual model evaluation, implementation verification, model output verification, model analysis, model output corroboration, iteration of the modelling cycle, and upscaling/transferability (Table 1). In addition to the TRACE categories (Grimm et al. 2014), we added the category upscaling/transferability because it is currently being discussed as one the major open challenges for social-ecological ABMs (Arneth et al. 2014; Verburg et al. 2016). The final category "iteration of the modelling cycle" illustrates the cyclic character of the TRACE framework.

Table 1: Elements of model "evaludation" (Augusiak et al. 2014) that comprise the structure of TRACE documents (TRAnsparent and Comprehensive model Evaludation), a standard format for documenting the development and testing of ecological models (Schmolke et al. 2010; Grimm et al. 2014 ). The last two elements were added for the purpose of this review.

TRACE element	Explanation
Problem formulation	This TRACE element provides supporting information on: The decision-making context in which the model will be used; the types of model clients or stakeholders addressed; a precise specification of the question(s) that should be answered with the model, including a specification of necessary model outputs; and a statement of the domain of applicability of the model, including the extent of acceptable extrapolations.
Model description	This TRACE element provides supporting information on: The model. Provide a detailed written model description. For individual/agent-based and other simulation models, the ODD protocol is recommended as the standard format. For complex submodels, it should include concise explanations of the underlying rationale. Model users should learn what the model is, how it works, and what guided its design.
Data evaluation	This TRACE element provides supporting information on: The quality and sources of numerical and qualitative data used to parameterize the model, both directly and inversely via calibration, and of the observed patterns that were used to design the overall model structure. This critical evaluation will allow model users to assess the scope and uncertainty of the data and knowledge on which the model is based.
Conceptual model evaluation	This TRACE element provides supporting information on: The simplifying assumptions underlying a model’s design, both with regard to empirical knowledge and general, basic principles. This critical evaluation allows model users to understand that the model design was not ad hoc but based on carefully scrutinized considerations.
Implementation verification	This TRACE element provides supporting information on: (1) whether the computer code implementing the model has been thoroughly tested for programming errors, (2) whether the implemented model performs as indicated by the model description, and (3) how the software has been designed and documented to provide necessary usability tools (interfaces, automation of experiments, etc.) and to facilitate future installation, modification, and maintenance.
Model output verification	This TRACE element provides supporting information on: (1) how well the model output matches observations and (2) how much calibration and the effects of environmental drivers were involved in obtaining good fits of the model output and data.
Model analysis	This TRACE element provides supporting information on: (1) how sensitive model output is to changes in model parameters (sensitivity analysis) and (2) how well the emergence of model output has been understood.
Model output corroboration	This TRACE element provides supporting information on: How model predictions compare to independent data and patterns that were not used, and preferably not even known, while the model was developed, parameterized, and verified. By documenting model output corroboration, model users learn about evidence that, in addition to model output verification, indicates that the model is structurally realistic so that its predictions can be trusted to some degree.
Upscaling/Transferability	This category encompasses the application of social-ecological ABMs to large geographical areas as well as the transfer of ABMs to different case studies.
Iteration of the modelling cycle	The development of models is an iterative process, and model versions continually improve based on knowledge gained in previous model versions. Therefore, the categories above should ideally be visited several times during the model development.

Mini-review of ABMs

We conducted a Web of Science Topic Search (TS) with the search term TS = (((agent AND based AND model*) OR (multi AND agent)) AND social-ecological). We restricted our research to articles published in the last 16 years (2000-2016). After cursory scanning of the resulting 60 articles, we excluded 23 articles because they did not describe ABMs and retained 37 for our mini-review. In this review, we scanned for the more technical elements of Table 1, i.e., model purpose, model description, data evaluation, implementation and model output verification, model analysis, and model output corroboration (Table 1). The other elements are covered in the "review of reviews".

Results and Discussion

Review of reviews

The list of the 46 reviews and 18 method papers we evaluated is provided in Supplementary Material S1. Although we aimed to assess current achievements, we included articles from the last two decades because only a few recent reviews exist and because change to modelling cultures within specific fields is generally slow (Grimm and Berger 2016b), usually at the time scale of decades. At some points, we included modelling examples to complement the reviews and method papers by highlighting examples of recent developments in the field (28 papers). One purpose of our additional mini-review of ABMs of SES was to also systematically cover more recent publications. For the following, it is important to have in mind the specific definition of the corresponding categories (Table 1) to avoid misunderstandings.

Problem formulation/model purpose

A key question is whether social-ecological ABMs should focus on demonstrating and exploring ideas and testing hypotheses, or whether they can also provide decision and management support (Matthews et al. 2007). The boundaries between these model purposes are porous, and both are important. Nevertheless, these different model purposes call for different model types, ranging from toy models, which focus on specific features relevant for the research question under study, to structurally more-realistic models (Schlüter et al. 2013). While, in principal, models from both ends of this spectrum can be valuable for management support, ABMs aimed at supporting the appropriate design of a management strategy in a specific case study should have greater structural realism compared to ABMs designed for improving system understanding of general principles.

In an extensive review of land use ABMs, Groeneveld et al. (2017) showed that the overwhelming majority of ABMs are used for system understanding. So far, ABMs have been used to understand processes in different contexts, such as land use change, natural resource management, and urban processes (Heckbert et al. 2010), or they have been used to study processes of systemic change in coupled SES, such as regime shifts (Filatova et al. 2016; Polhill et al. 2016). The use of ABMs has already improved our understanding of collective actions and the governance of common pool resources (Janssen and Ostrom 2006; for examples, see Agrawal et al. 2013, Zellner et al. 2014 or Schill et al. 2016) and of human-environmental interactions in the distant past (for examples, see Crabtree et al. 2012, Barton et al. 2016 or Kohler et al. 2012). Despite this huge potential of ABMs to understand SES, there are multiple processes relevant for these systems that have so far only rarely been addressed in ABMs, such as the role of institutions, hierarchical interactions, multi-level decision-making (Hare and Deadman 2004; Rounsevell et al. 2012; Groeneveld et al. 2017), interaction network structures (Janssen and Ostrom 2006), and iterative influences between attitude and behaviour (Edwards-Jones 2006).

We want to highlight work that has been done in recent years in some of these directions. For example, Agrawal et al. (2013) use an ABM to study the impact that informal norms in the form of social networks and formal organizations have on forest consumption. Wang et al. (2013) used an ABM to analyse the role of institutions, such as sedentary grazing or pasture rental markets, for climate change adaptation in Mongolian grasslands. Finally, Manson et al. (2016) developed an ABM of social network dynamics and showed the importance of social networks for the adoption of rotational grazing in dairy farming systems in the northern United States. As these are only examples, new reviews are needed on specific topics such as the state-of-the-art in the incorporation of institutions or, more specifically, the representation of social networks in ABMs.

While ABMs have been frequently used to support system understanding, there seems to be a substantial gap in their use for solving real-world problems by providing guidance for the design of management and policy strategies in specific case studies. There is still a lack of predictive power of ABMs, and it is an open challenge to demonstrate the value of ABMs to solve real-world problems and for operational decision support (Matthews et al. 2007; Schlüter et al. 2013). In the same line, Verburg et al. (2016) stress the limited application of ABMs as policy tools.

There are different approaches to making progress in this field. For example, the realism of social-ecological ABMs has been improved by recent developments in the combination of ABMs with GIS studies (Heckbert et al. 2010). Furthermore, the value of ABMs for decision support can be pushed forward through stakeholder participation (Matthews et al. 2007), transdisciplinary project design (Levin et al. 2013), or through testing the impacts of policies over different futures and evaluating the probability of a policy to perform well (Lempert 2002). Still, experience from other fields of research suggests that simulation modelling may rather be useful to derive rules of thumb instead of being used directly by the end user (Matthews et al. 2007).

One particular strength of social-ecological ABMs is that they enable the incorporation of stakeholders and decision makers through the use of participatory modelling approaches (Schlüter et al. 2013). The number of studies using such participatory approaches has been increasing considerably over the last decade, and ABMs are one of the main modelling techniques to support it (Seidl 2015). In contrast, in a review of land use ABMs, O’Sullivan et al. (2016) find that not many models have a participatory focus; they conclude that the potential of ABMs in the field of participation is currently underexplored. This potential encompasses different functions, such as collecting specific knowledge, informing decision makers, and social learning (Seidl 2015). Participatory modelling facilitates trans- and interdisciplinary communication, education and outreach (Matthews et al. 2007) and developing a shared representation in a companion modelling process (Bousquet and Le Page 2004). The latter can be used to develop and test social-ecological ABMs and has been practiced in a number of case studies (Bousquet et al. 2002; Castella et al. 2005). One method for participatory model development that has gained more attention over the last decade is the use of role-playing games to identify the decision-making processes of stakeholders and to include them in ABMs (Voinov & Bousquet 2010; Seidl 2015). While the current achievements in this area are promising, there are still multiple open challenges such as the subjectivity of stakeholders, conflicts between parties or the representative selection of stakeholders (Doole 2013). Additionally, so far, there is no understanding of how accumulated knowledge derived in one situation can be generalised (Rouchier 2006). Here, Voinov & Bousquet (2010) predict that it will still take some time for a theory to be established. Until then, there will be only a few generalisations beyond the local applications of participatory modelling. Finally, standards for reporting on participatory modelling projects are currently missing (Seidl 2015).

Voinov & Bousquet (2010) gathered several valuable general principles to foster efficient participatory modelling such as early involvement of stakeholders or the selection of a diverse group of stakeholders. Seidl (2015) present a general template to structure transdisciplinary modelling projects, including the definition of the functions of different stakeholders during the project phases.

Model description

In recent years, great effort has been put into improving and standardizing how to describe ABMs. Examples include protocols such as the ODD (short for Overview, Design and Details; Grimm et al. 2006; Grimm et al. 2010); its extension for ABMs, including human decision-making ODD+D (Müller et al. 2013); the TRACE framework for documenting the full modelling cycle (Schmolke et al. 2010; Grimm et al. 2014 ); and ontologies describing entities and their relationships (Parker et al. 2008b; Polhill and Gotts 2009). Müller et al. (2014) review different types of model descriptions and propose as a minimum standard the provision of natural language descriptions, for example, in the form of the ODD, combined with the provision of source code. ODD is widely used in ecology, and its use for social-ecological modelling seems to be increasing. Also, it is recommended by one of the main outlets for ABMs on SES, the journal JASSS, and by the OpenABM initiative (www.openabm.org). In any case, an established standard for model descriptions reduces the effort required by model developers, reviewers and peers to describe and understand a model because the same kind of information is always to be found in the same part of the model description. Moreover, standardised model descriptions also facilitate and harmonize model development because the categories of, e.g., the ODD framework can be used as a checklist for the design decisions a modeller has to make. This also implies that for ABMs of SES, ODD+D might be more advisable, as representing human decision-making is the key element of these models.

Data evaluation

ABMs profit from the availability of quantitative and qualitative data to directly or indirectly parameterize the model. In particular, collecting micro-level and interaction data can improve the representation of decision-making in ABMs (Filatova et al. 2013). Here, multiple approaches toward empirically informing social-ecological ABMs have emerged in recent years, including surveys, interviews, laboratory experiments, participatory and companion modelling, stylized facts (also referred to as "patterns"; Grimm and Railsback 2012), GIS and remotely sensed data (Rouchier 2006; Janssen and Ostrom 2006; Robinson et al. 2007; Heckbert et al. 2010; Voinov and Bousquet 2010). Also, "citizen observatories or crowd sourcing have the potential to contribute information on individual decision making" (Verburg et al. 2016, p.337).

Despite this large number of approaches and the progress made in empirically informing ABMs (Robinson et al. 2007; Windrum et al. 2007; Smajgl et al. 2011), ABMs still face multiple challenges with regard to parameterization and calibration (Robinson et al. 2007; Parker et al. 2008a; Nolan et al. 2009). These challenges are especially pronounced for empirically informing human decision-making in ABMs, as "data collection is a very expensive task in [social science] and in most cases it is even impossible to generate long time series for individual or group behaviour" (Troitzsch 2004, p.5). For example, methods to collect data on temporal processes in decision-making, such as frequency, duration, order or changes over time, are lacking (Gilbert 2004; Robinson et al. 2007). A further example of a gap in data collection is that survey data often "treats individuals as isolated ‘atoms’ and pays little attention to the impact of people’s interactions with others" (Gilbert 2004, p.4).

However, promising approaches exist that can help overcome challenges in parameterizing human decision-making in ABMs. In the following paragraphs, we will discuss some of these. One promising approach to model calibration established in ecological modelling is pattern-oriented modelling (Wiegand et al. 2003; Grimm et al. 2005), where different patterns/stylized facts are combined and used to calibrate the model. For social-ecological ABMs, such stylized facts or patterns can be used to reject agents' decisions rules that fail to reproduce patterns (Heckbert et al. 2010). As a further approach, Smajgl et al. (2011) provide a framework for the parameterization of human decision-making in ABMs: based on the modelling context, a combination of empirical methods for different steps in model building is proposed.

Finally, like in other modelling fields, for social-ecological ABMs, the "balance between fitting the data and generalizability remains another problem" (Janssen and Ahn 2006). In automated optimization procedures, this might be tackled by including penalties for model complexity due to, for example, the number of parameters (Janssen and Ahn 2006). Reducing complexity will increase the tractability of models (Verburg et al. 2016). However, we would like to note that model complexity per se is not a bad thing but rather that the right level of complexity depends on the model purpose (Sun et al. 2016).

Conceptual model evaluation

In social-ecological ABMs, the human decision-making sub-models are key elements, so their conceptual design is crucial for model evaluation. The adequate representation of human decision-making is a prerequisite for models to provide reliable policy recommendations (Milner-Gulland, 2012; World Bank Report, 2015). There are already a variety of multiple decision models and architectures that are partly already used in ABMs. For example, Balke and Gilbert (2014) review 14 architectures, including belief-desire-intention, norms, cognition, and learning. An et al. (2012) differentiate among nine categories: microeconomic models, space theory-based models, psychological and cognitive models, institution-based models, experience- or preference-based models, participatory models, empirical or heuristic rules, evolutionary programming, and assumption-based models. However, this large number of different techniques and theories also hampers advancement in the field (Parker et al. 2003).

Despite this variety of potential decision models, the representation of decision-making in ABMs is often ad hoc and only seldom based on established theory. Decision-making is most often based on simple heuristics (Hare and Deadman 2004) and merely on psychological theories (Groeneveld et al. 2017). To advance the use of complex decision theories in social-ecological models, Schlüter et al. (2017) proposed the MoHuB framework for mapping and comparing behavioural theories. The MoHuB framework (Modelling Human Behaviour) specifies the necessary elements of the decision process (e.g., perception or selection process) and provides a guideline to map decision theories onto those elements. It aims to support informed selection of the appropriate decision theory and its formalisation in models. In contrast to this rather theoretical starting point, Bayesian belief networks present a promising approach to include empirically observed patterns of human decision-making in ABMs (e.g., Sun and Müller 2013).

Interestingly, very few ABMs of SES compare alternative models of decision-making and check how well they are able to lead to system-level behaviour that is realistic. One example is Janssen and Baggio (in press), who compare different behavioural theories to experimental data on irrigation games. This "pattern-oriented theory development" (Grimm and Railsback 2012; Railsback and Grimm 2012), which is increasingly implemented in ecology, is nothing more than following the scientific method. It was explicitly formulated by Platt (1964) and dubbed "strong inference". This basic principle, which is commonly used in natural sciences, seems to be less acknowledged in social-ecological research. For example, Schlüter and Pahl-Wostl (2007) state the obvious in saying that "The choice of how to represent the behavior of human actors in an agent-based model has a strong influence on model results". They refer to Hare and Pahl-Wostl (2001) in saying, "They showed that the sensitivity of model results to structural uncertainties in the social model largely exceeded the effect of parameter uncertainties in the natural system." However, the potentially high sensitivity of model results to how a certain sub-model is formulated is inherent to any modelling discipline and does not necessarily prevent progress. Rather, corresponding to quantifying the sensitivity to changes in model parameters, modellers also need to systematically explore the sensitivity to different formulations of a certain sub-model. Basically, the modeller has to define a set of patterns or stylized facts characterizing the system of interest and its behaviour and see how well the ABM, with the different alternative formulations of the decision model, is able to reproduce these patterns simultaneously. In this way, the most appropriate representation of behaviour, in the context of the question addressed and the system characteristics considered, is filtered from the candidate representations.

A particular challenge regarding the conceptual development of social-ecological ABMs lies in the needed integration of social and ecological systems: the majority of ABMs tackles only one-way linkages and not two-way feedbacks between the social and the ecological system (Parker et al. 2008a; Cooke et al. 2009; Heckbert et al. 2010; Schlüter et al. 2012; Filatova et al. 2013; a similar challenge was reported by Drechsler et al. (2007) for ecological-economic models addressing biodiversity conservation). Filatova et al. (2016) name a few exceptions that address closed-loop couplings (e.g., Evans and Kelley 2008; Rouleau et al. 2009; Le et al. 2012; or Heckbert et al. 2013).

Further effort is needed to represent (1) social networks (An et al. 2012), (2) spatial or scale (e.g., harvest decisions lead to biodiversity loss at a higher scale) mismatch between decisions and impacts (e.g., Parker et al. 2003) and (3) adaptive behaviour and learning (Bousquet and Le Page 2004; Filatova et al. 2013). For the latter, "laboratory and field experiments are perhaps best suited to identifying the structure of the learning process where the repeated decisions of actors are recorded (e.g., Evans et al. 2006)." (Robinson et al. 2007, p.51).

Implementation verification

Software development for the implementation of ABMs has advanced in recent years, with software packages becoming available that no longer require extensive programming skills (Railsback et al. 2006; Heckbert et al. 2010) and thereby also facilitate the checking of model implementations. However, due to their complex nature, verification of ABM implementations is challenging (Nolan et al. 2009), and the detection of errors and artefacts is hampered (Galan et al. 2009). To avoid errors and artefacts, Galan et al. (2009) propose different strategies such as the "repetition of experiments in different platforms, reimplementation of the code in different programming languages, reformulation of the conceptual model using different modelling paradigms, and mathematical analyses of simplified versions or particular cases of the model". Independent replication of models has also been suggested as a general strategy not only for implementation verification but also for theory development, because modellers will less often start from scratch, preferring to give more time to new analyses and identify the processes that actually control model behaviour (Thiele and Grimm 2015). Doole (2013) suggests identifying the errors of each model component separately instead of testing the fully integrated model, a strategy also advised in an ABM textbook by Railsback and Grimm (2012), who demonstrate how re-implementing key sub-models in, e.g., spreadsheets can help identify even subtle implementation errors. A further approach is to make source code publicly available (Parker et al. 2003), which can be realized via online platforms such as OpenABM (http://www.openabm.org; for a discussion of experiences with OpenABM, see Janssen et al. 2008; Rollins et al. 2014; or Janssen 2017). Further platforms include github (http://www.github.com) and Modelling4All (http://m.modelling4all.org/).

Model output verification

Verification of social-ecological ABMs in the sense of comparing model results to real-world data or patterns is still in its infancy. For example, Balbi and Giupponi (2009) reviewed ABMs in the field of climate change adaptation and found that half of the reviewed studies did not address validation and verification due to the level of model abstraction, which hampers model testing. Following Ahrweiler and Gilbert (2005), there are two ways to verify model quality: a) the constructivist approach, where the observer compares the results of a constructed simulation model and constructed real world observations, and b) the user community approach, where model outcomes are compared to expert expectations (also in line with (Troitzsch 2004). In a similar vein, Polhill et al. (2016) argue that validation methods appropriate for ABMs could be expert validation (Smajgl and Bohensky 2013) or pattern-oriented modelling (Grimm et al. 2005). For participatory modelling endeavours, the quality of developed models is related to the degree of stakeholders’ agreement (Voinov and Bousquet 2010). Verburg et al. (2016) state that instead of predicting how an SES will develop in the future, ABMs could explain why SES "produce currently observed behaviour" (Verburg et al. 2016, p.332).

A particular challenge for verifying the output of ABMs is knowing how to choose the appropriate level of testing, as different model structures at the micro-scale can lead to the same emergent patterns at the macro-scale (Gilbert 2004). Hence, Takadama et al. (2008) propose conducting both micro- and macro-level validation of ABMs, and they provide an example using a simulation of a bargaining game. Bert et al. (2014) offer the same suggestion and distinguish between iterative conceptual validation, together with experts and stakeholders, and iterative empirical validation via calibration. They demonstrate their protocol for validation using an ecological-economic land use model that includes many practical examples of useful model analysis techniques. It should be noted, however, that Augusiak et al. (2014) suggest a more refined framework for model evaluation and validation; in particular, they distinguish between "output verification" and "output corroboration". "Output verification" includes model tweaking via calibrations and pattern-oriented theory development. "Output corroboration" refers to predictions of patterns in the structure and behaviour of the modelled system that were not used or, preferably, not even known during model development and calibration (see corresponding section below). Moreover, with output verification, environmental drivers should be reported together with model output because sometimes these drivers largely determine output, while details of model structure do not matter much. Janssen (2009) reports a case where the seemingly impressive match of model output and data was almost entirely driven by the imposed time series of the carrying capacity of the system. This does not necessarily render a model useless, but it should be communicated to avoid over-interpretations and to stimulate critical discussion and further development.

Model analysis

Model analysis comprises assessing the impact of model parameters and assumptions on model output and deriving an understanding of how model results emerge (Augusiak et al. 2014). Due to their flexibility in representing processes at the micro-scale in great detail, as well as their stochastic nature, ABMs deliver "a high volume of output data rendering the identification of salient and relevant results (such as trends) and the assessment of model sensitivities to varying experimental conditions a challenging problem" (Lee et al. 2015, par. 1.3). Moreover, "verification of model output" usually includes calibration, which implies running the model for the whole range of inputs, which is computationally infeasible for complex models, since "real-life systems have too many different kinds of inputs, resulting in a combinatorial explosion of test cases" (Cole 2000; after Ahrweiler and Gilbert 2005). Additionally, the field currently lacks guidelines for appropriate analysis and presentation of complex ABM results (Lee et al. 2015).

Therefore, the ABMs of SES often focus on scenario comparison (as discussed for ABMs on regime shifts by Filatova et al. 2016; see also the mini-review below). Usually, the sensitivity of a few highly aggregated model outputs, such as average income or years staying in business, to different management, policy, or institutional scenarios is tested. The results may be interesting and important, but without understanding how the different responses of the system actually emerge, we have to trust the model blindly and we will not gain much in terms of how, why, and when certain mechanisms are dominant. Consequently, the "insights" from the model remain limited and cannot easily be transferred to other systems, scenarios, and questions.

There are a few examples of social-ecological ABMs for which at least an extensive sensitivity analysis has been performed (Filatova et al. 2013). However, these are exceptions, and "multidimensional parameter sweeps across several scenarios of change, and accompanying large-scale data analysis techniques and visualisations" are urgently needed (Filatova et al. 2016, p.342; O’Sullivan et al. 2016). Lee et al. (2015) discuss several available techniques for ABM analysis following three themes: an appropriate number of runs, sensitivity analysis and processing spatial and temporal output data. The authors also conclude that user-friendly software products for performing available analysis techniques are in high demand. First attempts in this direction include the BehaviorSpace of Netlogo (Wilensky 1999). BehaviorSearch (http://www.behaviorsearch.org) is a tool for calibrating models implemented in NetLogo. Many approaches for model calibration and sensitivity analysis that are widely used in other modelling disciplines are implemented in R (www.r-project.org). Using the R-package RNetLogo (Thiele et al. 2012), they can be easily used for ABMs implemented in NetLogo or, via file exchange of parameters and outputs, in any programming language. Thiele et al. (2014) provide a "cookbook" for calibration and sensitivity analysis based on R packages. In ecological modelling, local or one-at-a-time sensitivity analyses are included in most model analyses, but increasingly more-comprehensive techniques are being used. An example is the so-called "Morris screening" to identify the most-sensitive parameters, followed by global sensitivity analyses using, e.g., the Sobol of FAST methods (Thiele et al. 2014; Ayllón et al. 2016).

Model output corroboration

The gold standard for model validation, or for theory in general, is to make independent or secondary predictions. Dependent predictions are those that the model was more or less forced to make by choosing model structure and parameter values. However, this tweaking, which is inherent to modelling but does not yet often occur with ABMs of SES, might lead to the right outcome by combining the wrong mechanisms. "Testing" a theory in physics means making predictions that can be tested with existing but not yet used data or new experiments, for example, the gravitation waves predicted by Einstein, which were only confirmed in 2016.

In ABMs, such predictions are difficult but not impossible. The beech forest model BEEFORE (Rademacher et al. 2004) predicted a certain tree age structure in the canopy and a certain spatial distribution of very old trees; neither model feature was considered or even known during model development, but they were later confirmed by analysing historical data.

For ABMs of SES, this means thoroughly analysing the model using more than just a few aggregated variables as model output and being alert to unusual patterns in model structure and behaviour. Similar to the discussion of "model output verification" above, expert assessment can help advance "model output corroboration". Especially in participatory modelling projects, the final step is to disseminate results beyond the stakeholder group that was originally involved in the model development process (Voinov and Bousquet 2010). This can help advance "model output corroboration" by the feedback of a larger audience.

Iteration of modelling cycle

Modelling has to start from simple versions, which are analysed to be understood and then refined to add only necessary detail and to develop understanding incrementally. In principle, iterative model development in this "modelling cycle" is never finished, but it should ideally be run through at least several times. This is of special value for theory development: the inability of a model to reproduce empirical findings can indicate flaws in the theoretical foundation of the model and lead to an update of the underlying theory (Schutte 2010). This is especially interesting for advancing the representation of human decision-making in social-ecological ABMs and the understanding of how humans make decisions in general (see comments on "pattern-oriented theory development" above).

Upscaling and transferability

A recently discussed open question is how to upscale social-ecological ABMs to larger geographical areas. So far, an upscaling theory is missing (Parker et al. 2003; Rounsevell et al. 2012; Arneth et al. 2014; Verburg et al. 2016). This would enable the coupling of ABMs with environmental and vegetation models at different spatial scales (Rounsevell et al. 2012) and would thereby help realize hybrid approaches that couple or tightly integrate different models (O’Sullivan et al. 2016). In particular, it is an open question of "how to scale up processes of interactions of a few agents to interactions between many agents" (Janssen and Ostrom 2006). Upscaling of information on societies "demands substantive methodological development beyond simple statistical aggregation" (Verburg et al. 2016, p.334). Here, Verburg et al. (2016) discuss two methodological approaches: (1) outscaling to represent all individuals in a large geographic area by the use of increased computational power and (2) upscaling based on observed response patterns at an aggregated level (i.e., instead of individuals, the behaviour of the entire community is represented).

Another recently discussed approach to tackling these challenges is using agent functional types, which group individual agents by type (Arneth et al. 2014), comparable to global vegetation models, where a small number of plant functional types are used (e.g., LPJ GUESS - http://iis4.nateko.lu.se/lpj-guess/). We could also learn from political science, where ABMs have been applied at national and state levels (Cederman 2002).

A related challenge is to develop models that facilitate the transferability and generalizability of ABMs over different case studies while still being applicable for specific case studies (Janssen and Ostrom 2006; Schlüter et al. 2012). In this context, O’Sullivan et al. (2016) warn against the tendency of the ABM field to develop increasingly more independent ABMs for specific case studies (the "YAAWN syndrome - Yet Another Agent-Based Model…Whatever…Nevermind…") and propose that researchers working on specific case studies should "more fully articulate how they contribute to theoretical and/or methodological debates and concerns" (O’Sullivan et al. 2016, p.8).

Mini-review

Of the 37 articles found in our scan of the Web of Science database (see Supplementary Material S2), we excluded eight articles because they did not include a particular model but presented modelling frameworks (e.g., Schlüter et al. 2014) or larger projects of which ABMs are just a part and are not described in more detail (e.g., Astier et al. 2012; Forrester et al. 2014), or because so far only the conceptual model exists but not its implementation as a computer program (Spies et al. 2014). The fact that approximately 20% of the articles we found discuss ABMs and their role instead of actually presenting and analysing one confirms how young the field of ABMs of SES still is. Also, in ecology, there was initially a large number of review, methodological, or framework papers, but nowadays, ABMs are accepted while lengthy discussions of their potentials and limitations are no longer needed, or wanted, in research articles.

In the following, we summarize and briefly discuss the main findings of our mini-review (Figure 2). Most models were designed to mimic a specific real system about which data or observations exist. Only in four cases, however, were the models designed to directly support decision-making in a specific context (Carpenter and Brock 2004; Parrot et al. 2011; Chion et al. 2013; Pizzitutti et al. 2014). In 17 cases, the more or less realistic model was used to address a generic or theoretical question, whereas in seven cases, the real system was represented in a stylized way. The difference between these two categories is that "specific but theoretical" models aimed at a more or less realistic representation of a specific system, for example, by including GIS data, but then they explored theoretical questions, for example, how different types of governance affect resilience. In contrast, "specific but stylized" still refers to a specific system, but without aiming for a realistic but more or less stylized representation, with the benefit of being less complex and, therefore, easier to analyse. Only one model (Perez and Janssen 2015) was a toy model, as it did not relate to any specific system.

This distribution of model types may reflect the fact that the SES addressed with ABMs are complex systems. Representing them in generic models usually leaves too many degrees of freedom in the model structure and parameters. Confinement to real systems limits the degrees of freedom but at the same time limits the general insights that can be gained. Stylized representations of classes of systems seem to be a way to combine realism with the potential for general insights, but so far, it has not been tried to define classes of systems, in contrast to ecology, where only a limited number of ecosystem types exist.

Model description turned out to be a major issue. It is impossible to say just by reading a model description whether it is complete enough to be the basis of an independent re-implementation. We might therefore err on the positive side, but it seems that 13 of the 29 model descriptions were more or less complete. Of the remainder, 12 were incomplete and five were presented in earlier publications. Five presumably full model descriptions were in online supplements that were proprietary to journals and therefore were not generally accessible. On the other hand, 14 online model descriptions were accessible, eight of them on OpenABM. The use of ODD (Grimm et al. 2010) and ODD+D (Müller et al. 2013) has increased in recent years. Nine models used these formats, but two were incomplete (Guzy et al. 2008; Huber et al. 2013), one used a modified format (Janssen 2009), and one existed on OpenABM but was not mentioned in the publication (Heckbert 2013).

Reading the model descriptions revealed the necessity for substantial improvements in this area. Even though the ODD descriptions were not all entirely consistent, at least the reader knew where to expect what kind of information; often, they were complemented by the corresponding computer program. In contrast, the non-ODD descriptions did not have any consistent structure across the publications. Some were overly short and incomplete, while some were very long but did not follow a consistent structure so that despite the efforts of the authors, it remained unclear whether they were complete. Usage of the ODD format is obviously not perfect in practice, similar to the fact that being a native speaker in, e.g., English does not mean one is a good writer. However, even a suboptimal ODD/+D is much better than most free-format model descriptions.

Consistently useful ODDs can be achieved by taking the requirement to follow a standard format seriously, using the template provided by Grimm et al. (2010) and Müller et al. (2013), reading a few existing ODDs and checking in what way they are good or incomplete, and, most importantly, by factually describing what the model actually does, describing the program code, not any meta-representation of the model.

Still, even the best ODD will contain the ambiguities of any verbal description. Therefore, one should always also provide the very program code that was used to produce the results presented, preferably on an open-access online platform such as OpenABM (cf. Müller et al. 2014). The link between ODD and program code can be improved by using hyperlinks, or by numbering model equations and algorithms and using the same numbers, as comments, in the code. Referring to full model descriptions in earlier papers is also a cumbersome practice, as readers might not have access to that journal. Again, platforms like OpenABM are a much better solution and should be used even if earlier model versions have already been published elsewhere.

As stated above, except for one single model, all models reviewed referred to specific systems. However, only 11 publications mentioned a comparison of model output to observation and data, and a strong case for model realism was not presented in any of these models. Model corroboration with independent data was tried only once (Drake et al. 2014), perhaps because this model was published in an applied ecology journal where providing evidence for sufficient realism is required.

It seems that this seemingly limited effort of model output verification is mainly a matter of communication. Modellers who have a specific system in mind actually do use criteria by which they decide whether or not their model offers a sufficiently realistic representation for their purposes (Rykiel 1996; Augusiak et al. 2014), but in social-ecological modelling, no culture has yet been established to communicate these criteria. However, even if the criteria are qualitative or categorical, they can be good indicators of a model’s realism and generality, and they should be listed in the methods section and their tests briefly presented in the results section.

The greatest issue we identified in our mini-review is model analysis. We found very few cases, e.g., Wilson et al. (2007), where the major effort of the modeller was model analysis in order to understand how model results emerged. In 13 cases, model analysis was non-existent, extremely limited, or confined to scenario analysis. Only in seven cases were some elements of the sensitivity analysis presented, and none of them in a systematic way. Only eight publications included elements of model analysis that went beyond sensitivity analysis and provided at least some mechanistic understanding, e.g., sensitivity experiments (varying individual parameters over a larger range: Wang et al. 2013; Perez and Janssen 2015), global sensitivity analysis (Rasch et al. 2016), comparison of model output with multiple observed patterns (Heckbert 2013; Parrot et al. 2011), extensive calibration (Janssen 2009), and unrealistic scenarios (Wilson et al. 2007).

It seems that modellers of ABMs for SES focus too much on the question of how to represent SES and too little on how they could actually learn from their models. Guzy et al. (2008) admit this very openly:

"We arrived at a very costly, complex model configuration that is difficult to explain, interpret, and generalize from. Because we had already exhausted the resources available to us, we could not explore the consequences of alternative agent representations, including interactive role-playing, dynamic generation of agent preferences, mechanisms for specifying agent behavior, and other advances in the core mechanism, such as those from game theory and behavioral psychology that involve agent interactions."

To improve this situation, we recommend iterative model development where early oversimplified model versions are thoroughly analysed, with all relevant model outputs and testing methods implemented. Only in this way can we already learn from early versions how to incrementally refine the model and actually understand how model behaviours emerge.

Conclusions

With this review, we recorded achievements made in the field of agent-based modelling of SES. However, we also identified poorly developed elements in the current modelling culture and suggested improvements to overcome these. To conclude, we would like to highlight promising ways forward in advancing the field of social-ecological ABMs.

Improving the representation of human decision-making: Here, interdisciplinary collaboration among behavioural economics, social psychology and agent-based modelling should be fostered (Groeneveld et al. 2017). This can improve the availability of empirical data on decision processes, our understanding of how humans make decisions and how these decisions can be formalized in models. In this context, participatory modelling approaches also represent a valuable tool.

Sharing/reusing models: In a concerted action, we should advance the reuse and sharing of social-ecological ABMs. This can be achieved by the provision of source codes of full models as well as of model modules (cf. Janssen 2017 and Bell et al. 2015; see also Thiele and Grimm 2015 on the benefits of model replication). This would enable the pooling of energies, as modellers would not begin from scratch but would learn from each other. Furthermore, the sharing of models could foster the transferability of models to other research contexts.

Model development and analysis: Substantial improvements are needed to improve the current culture of model analysis in the field of agent-based modelling of SES. Method compilations for an adequate, transparent and systematic model analysis in combination with iterative model development need to be provided. This is the only way trust in social-ecological ABMs can be increased and their full potential - for example, for decision and management support - can be realized.

In particular, tools and strategies such as the evaludation framework, standard protocols for model communication such as ODD and ODD+D, and sharing code on online platforms such as OpenABM are simple but efficient means to learn more from our models and to make modelling in this field a community effort. We believe that following these routes and the suggestions provided throughout our review will help advance the modelling culture and thereby make agent-based modelling of SES more efficient and more coherent in the future.

References

AGRAWAL, A., Brown, D.G., Rao, G., Riolo, R., Robinson, D.T. & Bommarito, M. (2013). Interactions between organizations and networks in common-pool resource governance. Environmental Science & Policy, 25, 138-146. [doi:10.1016/j.envsci.2012.08.004 ]

AHRWEILER, P. & Gilbert, N. (2005). Caffè Nero: the Evaluation of Social Simulation. Journal of Artificial Societies and Social Simulation, 8(4), 14: https://www.jasss.org/8/4/14.html.

AN, L. (2012). Modeling human decisio,ns in coupled human and natural systems: Review of agent-based models. Ecological Modelling, 229, 25-36. [doi:10.1016/j.ecolmodel.2011.07.010 ]

ARNETH, A., Brown, C. & Rounsevell, M. D. A. (2014). Global models of human decision-making for land-based mitigation and adaptation assessment. Nature Climate Change, 4(7), 550-557.

ASTIER, M., García-Barrios, L., Galván-Miyoshi, Y., González-Esquivel, C. E. & Masera, O. R. (2012). Assessing the sustainability of small farmer natural resource management systems. A critical analysis of the MESMIS Program (1995-2010). Ecology and Society, 17(3). [doi:10.5751/ES-04910-170325 ]

AUGUSIAK, J., van den Brink, P. J. & Grimm, V. (2014). Merging validation and evaluation of ecological models to ‘evaludation’: a review of terminology and a practical approach. Ecological Modelling, 280, 117-128.

AYLLÓN, D., Railsback, S. F., Vincenzi, S., Groeneveld, J., Almodóvar, A. & Grimm, V. (2016). InSTREAM-Gen: modelling eco-evolutionary dynamics of trout populations under anthropogenic environmental change. Ecological Modelling, 326, 36-53. [doi:10.1016/j.ecolmodel.2015.07.026 ]

BALBI, S. & Giupponi, C. (2009). Reviewing agent-based modelling of socio-ecosystems: a methodology for the analysis of climate change adaptation and sustainability. University Ca' Foscari of Venice, Dept. of Economics Research Paper Series No. 15_09. Available at SSRN: https://ssrn.com/abstract=1457625.

BALKE, T. & Gilbert, N. (2014). How do agents make decisions? A survey. Journal of Artificial Societies and Social Simulation, 17(4), 13: https://www.jasss.org/17/4/13.html. [doi:10.18564/jasss.2687 ]

BARTON, C. M., Ullah, I. I. T., Bergin, S. M., Sarjoughian, H. S., Mayer, G. R., Bernabeu-Auban, J. W., Heimsath, A. M., Acevedo, M. F., Riel-Salvatore, J. G. & Arrows, J. R. (2016). Experimental socioecology: Integrative science for anthropocene landscape dynamics. Anthropocene, 13, 34-45.

BELL, A. R., Robinson, D. T., Malik, A. & Dewal, S. (2015). Modular ABM development for improved dissemination and training. Environmental Modelling and Software, 73, 189-200. [doi:10.1016/j.envsoft.2015.07.016 ]

BERKES, F. & Folke, C. (1998). Linking Social and Ecological Systems. Cambridge University Press, Cambridge, UK.

BERT, F. E., Rovere, S. L., Macal, C. M., North, M. J. & Podesta, G. P. (2014). Lessons from a comprehensive validation of an agent based-model: The experience of the Pampas Model of Argentinean agricultural systems. Ecological Modelling, 273, 284-298. [doi:10.1016/j.ecolmodel.2013.11.024 ]

BOUSQUET, F., Barreteau, O., d’Aquino, P., Étienne, M., Boissau, S., Aubert, S., Le Page, C., Babin, D. & Castella, J.-C. (2002). Multi-agent systems and role games: collective learning processes for ecosystem management. In: M. A. Janssen (Ed), Complexity and ecosystem management: the theory and practice of multi-agent systems (pp. 248– 285). Edward Elgar Publishing, Cheltenham, UK.

BOUSQUET, F. & Le Page, C. (2004). Multi-agent simulations and ecosystem management: a review. Ecological Modelling, 176, 3-4, 313-332. [doi:10.1016/j.ecolmodel.2004.01.011 ]

CARPENTER, S. R. & Brock, W. A. (2004). Spatial complexity, resilience, and policy diversity: Fishing on lake-rich landscapes. Ecology and Society, 9(1).

CARPENTER, S. R., Mooney, H. A., Agard, J., Capistrano, D., DeFries, R. S., Diaz, S., Dietz, T., Duraiappah, A. K., Oteng-Yeboah, A., Pereira, H. M., Perrings, C., Reid, W. V., Sarukhan, J., Scholes, R. J. & Whyte, A. (2009). Science for managing ecosystem services: beyond the millennium ecosystem assessment. PNAS, 106, 1305-1312. [doi:10.1073/pnas.0808772106 ]

CASTELLA, J. C., Trung, T. N. & Boissau, S. (2005). Participatory simulation of land-use changes in the northern mountains of Vietnam: the combined use of an agent-based model, a role-playing game, and a geographic information system. Ecology and Society, 10(1).

CEDERMAN, L. E. (2002). Endogenizing geopolitical boundaries with agent-based modelling. PNAS, 99, 7296-7303. [doi:10.1073/pnas.082081099 ]

CHAPIN, F. S., Carpenter, S. R., Kofinas, G. P., Folke, C., Abel, N., Clark, W. C., Olsson, P., Stafford Smith, M., Walker, B., Young, O. R., Berkes, F., Biggs, R., Grove, J. M., Naylor, R. L., Pinkerton, E., Steffen, W. & Swanson, F. J. (2010). Ecosystem stewardship: sustainability strategies for a rapidly changing planet. Trends in Ecology & Evolution, 25(4), 241-249.

CHION, C., Cantin, G., Dionne, S., Dubeau, B., Lamontagne, P., Landry, J.-A., Marceau, D., Martins, C. C. A., Ménard, N., Michaud, R., Parrott, L. & Turgeon, S. (2013). Spatiotemporal modelling for policy analysis: application to sustainable management of whale-watching activities. Marine Policy, 38, 151-162. [doi:10.1016/j.marpol.2012.05.031 ]

COLE, O. (2000). White-box testing. Dr. Dobb´s Journal, 23-28.

COOKE, I. R., Queenborough, S. A., Mattison, E. H. A., Bailey, A. P., Sandars, D. L., Graves, A. R., Morris, J., Atkinson, P. W., Trawick, P., Freckleton, R. P., Watkinson, A. R. & Sutherland, W. J. (2009). Integrating socio-economics and ecology: a taxonomy of quantitative methods and a review of their use in agro-ecology. Journal of Applied Ecology, 46, 269-277. [doi:10.1111/j.1365-2664.2009.01615.x ]

CRABTREE, S. A. & Kohler, T. A. (2012). Modelling across millennia: Interdisciplinary paths to ancient socio-ecological systems. Ecological Modelling, 241(0), 2-4.

DOOLE, G. J. & Pannell, D. J. (2013). A process for the development and application of simulation models in applied economics. Australian Journal of Agricultural and Resource Economics, 57, 79-103. [doi:10.1111/j.1467-8489.2012.00607.x ]

DRAKE, D. A. R. & Mandrak, N. E. (2014). Bycatch, bait, anglers, and roads: quantifying vector activity and propagule introduction risk across lake ecosystems. Ecological Applications, 24(4), 877-894.

DRECHSLER, M., Grimm, V., Myšiak, J. & Wätzold, F. (2007). Differences and similarities between ecological and economic models for biodiversity conservation. Ecological Economics, 62, 232-241. [doi:10.1016/j.ecolecon.2006.03.026 ]

EDWARDS-JONES, G. (2006). Modelling farmer decision-making: concepts, progress and challenges. Animal Science, 82(6), 783-790.

EVANS, T. P., Sun, W. & Kelley, H. (2006). Spatially explicit experiments for the exploration of land use decision-making dynamics. International Journal of Geographic Information Science, 20(9), 1013-1037. [doi:10.1080/13658810600830764 ]

EVANS, T. P. & Kelley, H. (2008). Assessing the transition from deforestation to forest regrowth with an agent-based model of land cover change for south-central Indiana (USA). Geoforum, 39(2), 819-832.

FILATOVA, T., Verburg, P. H., Parker, D. C. & Stannard, C. A. (2013). Spatial agent-based models for socio-ecological systems: Challenges and prospects. Environmental Modelling and Software, 45, 1-7. [doi:10.1016/j.envsoft.2013.03.017 ]

FILATOVA, T., Polhill, J. G. & van Ewijk, S. (2016). Regime shifts in coupled socio-environmental systems: Review of modelling challenges and approaches. Environmental Modelling and Software, 75, 333-347.

FOLKE, C., Carpenter, S. R., Walker, B., Scheffer, M., Chapin, T. & Rockström, J. (2010). Resilience thinking: integrating resilience, adaptability and transformability. Ecology and Society, 15(4), 20. [doi:10.5751/ES-03610-150420 ]

FORRESTER, J., Greaves, R., Noble, H. & Taylor, R. (2014). Modeling social-ecological problems in coastal ecosystems: a case study. Complexity, 19(6, SI), 73-82.

GALÁN, J. M., Izquierdo, L. R., Izquierdo, S. S., Santos, J. I., del Olmo, R., López-Paredes, A. & Edmonds, B. (2009). Errors and Artefacts in Agent-Based Modelling. Journal of Artificial Societies and Social Simulation, 12(1), 1: https://www.jasss.org/12/1/1.html.

GILBERT, N. (2004). Agent-based social simulation: dealing with complexity. http://wiki.commres.org/pds/AgentBasedModeling/AbssDealingWithComplexity.pdf.

GILBERT, N. & Troitzsch, K. (2005). Simulation for the social scientist. McGraw-Hill Education (UK).

GRIMM, V. (1999). Ten years of individual-based modelling in ecology: what have we learned and what could we learn in the future? Ecological Modelling, 115, 129-148.

GRIMM, V. & Railsback, S. F. (2005). Individual-based modeling and ecology. Princeton University Press, Princeton, NJ. [doi:10.1515/9781400850624 ]

GRIMM, V., Revilla, E., Berger, U., Jeltsch, F., Mooij, W. M., Railsback, S. F., Thulke, H.-H., Weiner, J., Wiegand, T. & DeAngelis, D. L. (2005). Pattern-oriented modeling of agent-based complex systems: lessons from ecology. Science, 310(5750), 987-991.

GRIMM, V., Berger, U., Bastiansen, F., Eliassen, S., Ginot, V., Giske, J., Goss-Custard, J., Grand, T., Heinz, S. K., Huse, G., Huth, A., Jepsen, J. U., Jørgensen, C., Mooij, W. M., Müller, B., Pe'er, G., Piou, C., Railsback, S. F., Robbins, A. M., Robbins, M. M., Rossmanith, E., Rüger, N., Strand, E., Souissi, S., Stillman, R. A., Vabø, R., Visser, U. & DeAngelis, D. L. (2006). A standard protocol for describing individual-based and agent-based models. Ecological Modelling, 198(1-2), 115-126. [doi:10.1016/j.ecolmodel.2006.04.023 ]

GRIMM, V., Berger, U., DeAngelis, D. L., Polhill, J. G., Giske, J. & Railsback, S. F. (2010). The ODD protocol: A review and first update. Ecological Modelling, 221(23), 2760-2768.

GRIMM, V. & Railsback, S. F. (2012). Pattern-oriented modelling: a ‘multi-scope’ for predictive systems ecology. Philosophical Transactions of the Royal Society B Biological Science, 367(1586), 298-310.

GRIMM, V., Augusiak, J., Focks, A., Frank, B. M., Gabsi, F., Johnston, A. S. A., Liu, C., Martin, B. T., Meli, M., Radchuk, V., Thorbek, P. & Railsback, S. F. (2014). Towards better modelling and decision support: Documenting model development, testing, and analysis using TRACE. Ecological Modelling, 280, 129-139.

GRIMM, V. & Berger, U. (2016a). Robustness analysis: deconstructing computational models for ecological theory and application. Ecological Modelling, 326, 162-167. [doi:10.1016/j.ecolmodel.2015.07.018 ]

GRIMM, V. & Berger, U. (2016b). Structural realism, emergence, and predictions in next-generation ecological modelling: synthesis from a special issue. Ecological Modelling, 326, 177-187.

GROENEVELD, J., Müller, B., Buchmann, C. M., Dressler, G., Guo, C., Hase, J. N., Hoffmann, F., John, F., Klassert, C., Lauf, T., Liebelt, V., Nolzen, H., Pannicke, N., Schulze, J., Weise, H. & Schwarz, N. (2017). Theoretical foundations of human decision-making in agent-based land use models – A review. Environmental Modelling and Software, 87, 39-48. [doi:10.1016/j.envsoft.2016.10.008 ]

GUZY, M. R., Smith, C. L., Bolte, J. P., Hulse, D. W. & Gregory, S. V. (2008). Policy research using agent-based modeling to assess future impacts of urban expansion into farmlands and forests. Ecology and Society, 13(1).

HARE, M. & Pahl-Wostl, C. (2001). Model uncertainty derived from choice of agent rationality: a lesson for policy assessment modelling. In N. Giambiasi & C. Frydman (Eds.), Simulation in industry: 13th European Simulation Symposium (pp. 854-859). SCS Europe Bvba, Ghent, Belgium.

HARE, M. & Deadman, P. (2004). Further towards a taxonomy of agent-based simulation models in environmental management. Mathematics and Computers in Simulation, 64, 1, 25-40.

HECKBERT, S., Baynes, T. & Reeson, R. (2010). Agent-based modeling in ecological economics. Annals of the New York Academy of Sciences, 1185, 39-53. [doi:10.1111/j.1749-6632.2009.05286.x ]

HECKBERT, S. (2013). MayaSim: an agent-based model of the ancient Maya social-ecological system. Journal of Artificial Societies and Social Simulation, 16(4), 11: https://www.jasss.org/16/4/11.html.

HECKBERT, S., Isendahl, C., Gunn, J., Brewer, S., Scarborough, V., Chase, A. F., Chase, D. Z., Costanza, R., Dunning, N., Beach, T., Luzzadder-Beach, S., Lentz, D. & Sinclair, P. (2013). Growing the ancient Maya social-ecological system from the bottom up. In C. Isendahl & D. Stump (Eds.), The Oxford Handbook of Historical Ecology and Applied Archaeology. Oxford University Press..

HUBER, R., Briner, S., Peringer, A., Lauber, S., Seidl, R., Widmer, A., Gillet, F., Buttler, A., Le, Q. B. & Hirschi, C. (2013). Modeling Social-Ecological Feedback Effects in the Implementation of Payments for Environmental Services in Pasture-Woodlands. Ecology and Society, 18(2).

JANSSEN, M. A. & Ahn, T. K. (2006). Learning, signaling, and social preferences in public-good games. Ecology and Society, 11(2), 21. [doi:10.5751/ES-01752-110221 ]

JANSSEN, M. A. & Ostrom, E. (2006). Empirically based, agent-based models. Ecology and Society, 11(2), 37.

JANSSEN, M. A., Alessa, L. N. I., Barton, M., Bergin, S. & Lee, A. (2008). Towards a community framework for agent-based modelling. Journal of Artificial Societies and Social Simulation, 11(2), 6: https://www.jasss.org/11/2/6.html.

JANSSEN, M. A. (2009). Understanding Artificial Anasazi. Journal of Artificial Societies and Social Simulation, 12 (4), 13: https://www.jasss.org/12/4/13.html.

JANSSEN, M. A. (2017). The practice of archiving model code of agent-based models. Journal of Artificial Societies and Social Simulation, 20 (1), 2: https://www.jasss.org/20/1/2.html. [doi:10.18564/jasss.3317 ]

JANSSEN, M. A. & Baggio, J. A. (in press). Using agent-based models to compare behavioral theories on experimental data: Application for irrigation games. Journal of Environmental Psychology.

KELLY, R. A., Jakeman, A. J., Barreteau, O., Borsuk, M. E., ElSawah, S., Hamilton, S. H., Henriksen, H. J., Kuikka, S., Maier, H. R., Rizzoli, A. E., van Delden, H. & Voinov, A. A. (2013). Selecting among five common modelling approaches for integrated environmental assessment and management. Environmental Modelling and Software, 47(0), 159-181. [doi:10.1016/j.envsoft.2013.05.005 ]

KOHLER, T. A., Bocinsky, R. K., Cockburn, D., Crabtree, S. A., Varien, M. D., Kolm, K. E., Smith, S., Ortman, S. G. & Kobti, Z. (2012) Modelling prehispanic Pueblo societies in their ecosystems. Ecological Modelling, 241, 30-41.

LE, Q. B., Seidl, R. & Scholz, R. W. (2012). Feedback loops and types of adaptation in the modelling of land-use decisions in an agent-based simulation. Environmental Modelling and Software, 27-28, 83-96. [doi:10.1016/j.envsoft.2011.09.002 ]

LEE, J.-S., Filatova, T., Ligmann-Zielinska, A., Hassani-Mahmooei, B., Stonedahl, F., Lorscheid, I., Voinov, A., Polhill, G., Sun, Z. & Parker, D.C. (2015). The complexities of agent-based modeling output analysis. Journal of Artificial Societies and Social Simulation, 18(4), 4: https://www.jasss.org/18/4/4.html.

LEMPERT, R. (2002). Agent-based modeling as organizational and public policy simulators. PNAS, 99, 7195-7196. [doi:10.1073/pnas.072079399 ]

LE PAGE, C., Bazile, D., Becu, N., Bommel, P., Etienne, M., Mathevet, R., Souchère, V., Trébuil, G. & Weber, J. (2013). Agent-based modelling and simulation applied to environmental Management. In B. Edmonds & R. Meyer (Eds.), Simulating Social Complexity (pp- 499-540). Berlin, Heidelberg: Springer.

LEVIN, S., Xepapadeas, T., Crépin, A.-S., Norberg, J., de Zeeuw, A., Folke, C., Hughes, T., Arrow, K., Barrett, S., Daily, G., Ehrlich, P., Kautsky, N., Mäler, K.-G., Polasky, S., Troell, M., Vincent, J. R. & Walker, B. (2013). Social-ecological systems as complex adaptive systems: modeling and policy implications. Environment and Development Economics, 18(2), 111-132. [doi:10.1017/S1355770X12000460 ]

LIU, J., Mooney, H., Hull, V., Davis, S. J., Gaskell, J., Hertel, T., Lubchenco, J., Seto, K. C., Gleick, P., Kremen, C. & Li, S. (2015). Systems integration for global sustainability. Science, 347, 6225.

MANSON, S. M., Jordan, N. R., Nelson, K. C. & Brummel, R. F. (2016). Modeling the effect of social networks on adoption of multifunctional agriculture. Environmental Modelling and Software, 75, 388-401. [doi:10.1016/j.envsoft.2014.09.015 ]

MATTHEWS, R. B., Gilbert, N. G., Roach, A., Polhill, J. G. & Gotts, N. M. (2007). Agent-based land-use models: a review of applications. Landscape Ecology, 22, 1447.

MILNER-GULLAND, E. J. (2012). Interactions between human behaviour and ecological systems. Philosophical Transactions of the Royal Society B Biological Science, 367(1586), 270-278. [doi:10.1098/rstb.2011.0175 ]

MÜLLER, B., Bohn, F., Dreßler, G., Groeneveld, J., Klassert, C., Martin, R., Schlüter, M., Schulze, J., Weise, H. & Schwarz, N. (2013). Describing human decisions in agent-based models - ODD+D, an extension of the ODD protocol. Environmental Modelling and Software, 48, 37-48.

MÜLLER, B., Balbi, S., Buchmann, C. M., de Sousa, L., Dressler, G., Groeneveld, J., Klassert, C. J., Le, Q. B. & Millington, J. D (2014). Standardised and transparent model descriptions for agent-based models: Current status and prospects. Environmental Modelling and Software, 55(0), 156-163. [doi:10.1016/j.envsoft.2014.01.029 ]

NOLAN, J., Parker, D., Van Kooten, G. C. & Berger, T. (2009). An overview of computational modeling in agricultural and resource economics. Canadian Journal of Agricultural Economics/Revue canadienne d'agroeconomie, 57, 417-429.

OSTROM, E. (2009). A general framework for analyzing sustainability of social-ecological systems. Science, 325, 419-422. [doi:10.1126/science.1172133 ]

O’SULLIVAN, D., Evans, T., Manson, S., Metcalf, S., Ligmann-Zielinska, A. & Bone, C. (2016). Strategic directions for agent-based modeling: avoiding the YAAWN syndrome. Journal of Land Use Science, 11(2), 177-187.

PARKER, D. C., Manson, S. M., Janssen, M. A., Hoffmann, M. J. & Deadman, P. (2003). multi-agent systems for the simulation of land-use and land-cover change: a review. Annals of the Association of American Geographers, 93, 314-337. [doi:10.1111/1467-8306.9302004 ]

PARKER D. C., Hessl A. & Davis S. C. (2008a). Complexity, land-use modeling, and the human dimension: fundamental challenges for mapping unknown outcome spaces. Geoforum, 39, 789-804.

PARKER, D. C., Brown, D. G., Polhill, J. G., Deadman, P. J. & Manson, S. M. (2008b). Illustrating a new 'conceptual design pattern' for agent-based models and land use via five case studies: the MR POTATOHEAD framework, In A. L. Paredes & C. H. Iglesias (Eds.), Agent-based modelling in natural resource management (pp. 23-51). Universidad de Valladolid: Valladolid, Spain.

PARROTT, L., Chion, C., Martins, C. C. A., Lamontagne, P., Turgeon, S., Landry, J. A., Zhens, B., Marceau, D. J., Michaud, R., Cantine, G., Ménard, N. & Dionne, S. (2011). A decision support system to assist the sustainable management of navigation activities in the St. Lawrence River Estuary, Canada. Environmental Modelling and Software, 26(12), 1403-1418.

PLATT J. R. (1964). Strong inference. Science, 146, 347-352. [doi:10.1126/science.146.3642.347 ]

PEREZ, I. & Janssen, M. A. (2015). The effect of spatial heterogeneity and mobility on the performance of social-ecological systems. Ecological Modelling, 296, 1-11.

PIZZITUTTI, F., Mena, C. F. & Walsh, S. J. (2014). Modelling tourism in the galapagos islands: an agent-based model approach. Journal of Artificial Societies and Social Simulation, 17(1), 14: https://www.jasss.org/17/1/14.html. [doi:10.18564/jasss.2389 ]

POLHILL, J. G. & Gotts, N. (2009). Ontologies for transparent integrated human-natural system modelling. Landscape Ecology, 24(9), 1255-1267.

POLHILL, J. G., Filatova, T., Schlüter, M. & Voinov, A. (2016). Modelling systemic change in coupled socio-environmental systems. Environmental Modelling and Software, 75, 318-332. [doi:10.1016/j.envsoft.2015.10.017 ]

POTEETE, A. R., Janssen, M. A. & Ostrom, E. (2010). Working together: collective action, the commons, and multiple methods in practice. Princeton, NJ: Princeton University Press.

RASCH, S., Heckelei, T., Oomen, R. & Naumann, C. (2016). Cooperation and collapse in a communal livestock production SES model - A case from South Africa. Environmental Modelling and Software, 75(SI), 402-413. [doi:10.1016/j.envsoft.2014.12.008 ]

RADEMACHER, C., Neuert, C., Grundmann, V., Wissel, C. & Grimm, V. (2004) Reconstructing spatiotemporal dynamics of central European natural beech forests: the rule-based model BEFORE. Forest Ecology and Management, 194, 349-368.

RAILSBACK, S. F., Lytinen, S. L. & Jackson, S. K. (2006). Agent-Based Simulation Platforms: Review and Development Recommendations. Simulation, 82(9), 609-623. [doi:10.1177/0037549706073695 ]

RAILSBACK, S. F. & Grimm, V. (2012). Agent-based and individual-based modeling: a practical introduction. Princeton University Press, Princeton, NJ.

ROBINSON, D. T., Brown, D. G., Parker, D. C., Schreinemachers, P., Janssen, M. A., Huigen, M., Wittmer, H., Gotts, N., Promburom P., Irwin, E., Berger, T., Gatzweiler, F. & Barnaud, C. (2007). Comparison of empirical methods for building agent-based models in land use science. Journal of Land Use Science, 2, 31-55. [doi:10.1080/17474230701201349 ]

ROLLINS, N. D., Barton, C. M., Bergin, S., Janssen, M. A., & Lee, A. (2014). A computational model library for publishing model documentation and code. Environmental Modelling and Software, 61, 59-64.

ROUCHIER, J. (2006). Data gathering to build and validate small-scale social models for simulation. In J. P. Rennard (Ed), Handbook of research on nature inspired computing for economics and management (pp. 198-210).

ROULEAU, M., Coletti, M., Bassett, J. K., Hailegiorgis, A. B., Gulden, T. & Kennedy, W. G. (2009). Conflict in complex socio-natural systems: using agent-based modeling to understand the behavioral roots of social unrest within the Mandera Triangle. Proceedings of the Human Behavior-Computational Modeling and Interoperability Conference 2009, Joint Institute for Computational Science, Oak Ridge National Laboratory, Oak Ridge, Tennessee, USA.

ROUNSEVELL, M., Robinson, D., & Murray-Rust, D. (2012). From actors to agents in socio-ecological systems models. Philosophical Transactions: Biological Sciences, 367(1586), 259-269. [doi:10.1098/rstb.2011.0187 ]

RYKIEL, E. J. (1996). Testing ecological models: the meaning of validation. Ecological Modelling, 90(3), 229-244.

SCHILL, C., Wijermans, N., Schlüter, M. & Lindahl, T. (2016). Cooperation is not enough—exploring social-ecological micro-foundations for sustainable common-pool resource use. Plos One, 11(8), e0157796. [doi:10.1371/journal.pone.0157796 ]

SCHLÜTER, M. & Pahl-Wostl, C. (2007). Mechanisms of resilience in common-pool resource management systems: an agent-based model of water use in a river basin. Ecology and Society, 12(2).

SCHLÜTER, M., McAllister, R. R. J., Arlinghaus, R., Bunnefeld, N., Eisenack, K., Hölker, F., Milner-Gulland, E. J., Müller, B., Nicholson, E., Quaas, M. & Stöven, M. (2012). New horizons for managing the environment: A review of coupled social-ecological systems modeling. Natural Resource Modeling, 25(1), 219-272. [doi:10.1111/j.1939-7445.2011.00108.x ]

SCHLÜTER, M., Müller, B. & Frank, K. (2013). How to use models to improve analysis and governance of social-ecological systems - the reference frame MORE. Working paper. Available at SSRN: http://ssrn.com/abstract=2037723.

SCHLÜTER, M., Hinkel, J., Bots, P. W. G. & Arlinghaus, R. (2014). Application of the SES Framework for Model-based Analysis of the Dynamics of Social-Ecological Systems. Ecology and Society, 19(1). [doi:10.5751/ES-05782-190136 ]

SCHLÜTER, M., Baeza A., Dressler G, Frank K., Groeneveld, J., Jager W., Janssen, M. A., McAllister, R. R. J., Müller, B., Orach, K., Schwarz, N. & Wijermans. N. (2017). A framework for mapping and comparing behavioural theories in models of social-ecological systems. Ecological Economics, 131, 21-35.

SCHMOLKE, A., Thorbek, P., DeAngelis, D. L., & Grimm, V. (2010). Ecological models supporting environmental decision making: a strategy for the future. Trends in Ecology & Evolution, 25(8), 479-486. [doi:10.1016/j.tree.2010.05.001 ]

SCHUTTE, S. (2010). Optimization and falsification in empirical agent-based models. Journal of Artificial Societies and Social Simulation, 13(1), 2: https://www.jasss.org/13/1/2.html.

SEIDL, R. (2015). A functional-dynamic reflection on participatory processes in modeling projects. Ambio, 44(8), 750-765. [doi:10.1007/s13280-015-0670-8 ]

SMAJGL, A., Brown, D. G., Valbuena, D. & Huigen, M. G. A. (2011). Empirical characterisation of agent behaviours in socioecological systems. Environmental Modelling and Software, 26(7), 837–844.

SMAJGL, A. & Bohensky, E. (2013). Behaviour and space in agent-based modelling: poverty patterns in East Kalimantan, Indonesia. Environmental Modelling and Software, 45, 8-14. [doi:10.1016/j.envsoft.2011.10.014 ]

SPIES, T. A., White, E. M., Kline, J. D., Fischer, A. P., Ager, A., Bailey, J., Bolte, J., Koch, J., Platt, E., Olsen, C. S., Jacobs, D., Shindler, B., Steen-Adams, M. M. & Hammer , R. (2014). Examining fire-prone forest landscapes as coupled human and natural systems. Ecology and Society, 19(3).

SQUAZZONI, F. (2012). Agent-based computational sociology. John Wiley & Sons. [doi:10.1002/9781119954200 ]

SUN, Z. & Müller, D. (2013). A framework for modeling payments for ecosystem services with agent-based models, Bayesian belief networks and opinion dynamics models. Environmental Modelling and Software, 45(0), 15-28.

SUN, Z., Lorscheid, I., Millington, J. D., Lauf. S., Magliocca, N. R., Groeneveld, J., Balbi, S., Nolzen, H., Müller, B., Schulze, J. & Buchmann, C. M. (2016). Simple or complicated agent-based models? A complicated issue. Environmental Modelling and Software, 86, 56-67. [doi:10.1016/j.envsoft.2016.09.006 ]

TAKADAMA, K., Kawai, T. & Koyama, Y. (2008). Micro- and macro-level validation in agent-based simulation: reproduction of human-like behaviors and thinking in a sequential bargaining game. Journal of Artificial Societies and Social Simulation, 11(2), 9: https://www.jasss.org/11/2/9.html.

THIELE, J. C., Kurth, W. & Grimm, V. (2012). RNetLogo: an R package for running and exploring individual-based models implemented in NetLogo. Methods in Ecology and Evolution, 3, 480-483. [doi:10.1111/j.2041-210X.2011.00180.x ]

THIELE, J. C., Kurth, W. & Grimm, V. (2014). Facilitating Parameter Estimation and Sensitivity Analysis of Agent-Based Models: A Cookbook Using NetLogo and R. Journal of Artificial Societies and Social Simulation, 17(3), 11: https://www.jasss.org/17/3/11.html.

THIELE, J. C. & Grimm, V. (2015). Replicating and breaking models: good for you and good for ecology. Oikos, 124(6), 691-696. [doi:10.1111/oik.02170 ]

TROITZSCH, K. (2004). Validating simulation models. Proceedings of 18th European Simulation Multi-conference, Magdeburg, 98-106: http://scs-europe.net/services/esm2004/pdf/esm-43.pdf.

VERBURG, P. H., Dearing, J. A., Dyke, J. G., Leeuw, S. v. d., Seitzinger, S., Steffen, W. & Syvitski, J. (2016). Methods and approaches to modelling the Anthropocene. Global Environmental Change, 39, 328-340. [doi:10.1016/j.gloenvcha.2015.08.007 ]

VOINOV, A. & Bousquet. F. (2010). Modelling with stakeholders. Environmental Modelling and Software, 25, 1268-1281.

WANG, J., Brown, D. G., Riolo, R. L., Page, S. E. & Agrawal, A. (2013). Exploratory analyses of local institutions for climate change adaptation in the Mongolian grasslands: An agent-based modeling approach. Global Environmental Change, 23(5), 1266-1276. [doi:10.1016/j.gloenvcha.2013.07.017 ]

WILSON, J., Yan, L. & Wilson, C. (2007). The precursors of governance in the Maine lobster fishery. PNAS, 104(39), 15212-15217.

WIEGAND, T., Jeltsch, F., Hanski, I. & Grimm, V. (2003). Using pattern-oriented modeling for revealing hidden information: a key for reconciling ecological theory and application. Oikos, 100, 209-222. [doi:10.1034/j.1600-0706.2003.12027.x ]

WILENSKY, U. (1999). NetLogo. http://ccl.northwestern.edu/netlogo/. Center for Connected Learning and Computer-Based Modeling, Northwestern University. Evanston, IL.

WINDRUM, P., Fagiolo, G. & Moneta, A. (2007). Empirical validation of agent-based models: Alternatives and prospects. Journal of Artificial Societies and Social Simulation, 10(2), 8: https://www.jasss.org/10/2/8.html.

WORLD BANK, 2015. World Development Report 2015: Mind, Society, and Behavior. World Bank, Washington, DC. doi: 10.1596/978-1-4648-0342-0. License: Creative Commons Attribution CC BY 3.0 IGO.

ZELLNER, M., Watkins, C., Massey, D., Westphal, L., Brooks, J. & Ross, K.(2014). Advancing collective decision-making theory with integrated agent-based modeling and ethnographic data analysis: an example in ecological restoration. Journal of Artificial Societies and Social Simulation, 17(4), 11: https://www.jasss.org/17/4/11.html. [doi:10.18564/jasss.2605 ]