key: cord-0255005-e4knnfh6
authors: Zavalis, E. A.; Ioannidis, J.
title: A meta-epidemiological assessment of transparency indicators of infectious disease models
date: 2022-04-16
journal: nan
DOI: 10.1101/2022.04.11.22273744
sha: 64f6f11ac97d834dfa093e157d181d3c9d91c564
doc_id: 255005
cord_uid: e4knnfh6

Mathematical models have become very influential, especially during the COVID-19 pandemic. Data and code sharing are indispensable for reproducing them, protocol registration may be useful sometimes, and declarations of conflicts of interest (COIs) and of funding are quintessential for transparency. Here, we evaluated these features in publications of infectious disease-related models and assessed whether there were differences before and during the COVID-19 pandemic and for COVID-19 models versus models for other diseases. We analysed all PubMed Central open access publications of infectious disease models published in 2019 and 2021 using previously validated text mining algorithms of transparency indicators. We evaluated 1338 articles: 216 from 2019 and 1122 from 2021 (of which 818 were on COVID-19); almost a six-fold increase in publications within the field. 511 (39.2%) were compartmental models, 337 (25.2%) were time series, 279 (20.9%) were spatiotemporal, 186 (13.9%) were agent-based and 25 (1.9%) contained multiple model types. 288 (21.5%) articles shared code, 332 (24.8%) shared data, 6 (0.4%) were registered, and 1197 (89.5%) and 1109 (82.9%) contained COI and funding statements, respectively. There was no major changes in transparency indicators between 2019 and 2021. COVID-19 articles were less likely to have funding statements and more likely to share code. Manual assessment of 10% of the articles that were identified by the text mining algorithms as fulfilling transparency indicators showed that 24/29 (82.8%) actually shared code, 29/33 (87.9%) actually shared data; and all had COI and funding statements, but 95.8% disclosed no conflict and 11.7% reported no funding. On manual assessment, 5/6 articles identified as registered had indeed been registered. Transparency in infectious disease modelling is relatively low, especially for data and code sharing. This is concerning, considering the nature of this research and the heightened influence it has acquired.

A large number of infectious disease-related models are published in the scientific literature and their production and influence has rapidly increased during the COVID-19 pandemic. Such models can inform and shape policy, and have also been the subject of much debate (1) (2) (3) (4) , surrounding a range of issues, including their questionable predictive accuracy and their transparency (5) (6) (7) .

Sharing of data and of code is totally indispensable for these models to be properly evaluated, used, reused, updated, integrated, or compared with other efforts. Without being able to rerun a model, it resembles a black box where blind trust is requested on its function and credibility. Moreover, other features of transparency, such as declaration of funding and of potential conflicts of interest (COI) are also important to have since many of these models may be very influential on deciding policy with major repercussions. Another feature of transparency that may aid reproducibility and trust in these models sometimes is the registration of their protocols, ideally in advance of their conduct. Registration is concept that receives increasing attention in many scientific fields (8-10) as a safeguard of trust. Registration may not be easy or relevant to have for many mathematical models, especially those that are exploratory and iterative (5) . However, it may be feasible and desirable to register protocols about models in some circumstances (5) .

There have previously been empirical evaluations of research practices, including documentation and transparency in subfields of mathematical modeling (11) (12) (13) that have shown that data and code/algorithm sharing has improved somewhat over time but that it still remains suboptimal. Yet, to our knowledge, in the field of infectious disease modelling there has been no comprehensive, large-scale analysis of such transparency and reproducibility indicators. It would . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022. ;  https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint be of interest to explore the state of transparency in this highly popular field, especially in the context of the rapid and massive adoption of mathematical models during the COVID-19

pandemic. Therefore, we decided to evaluate infectious disease modeling studies using largescale algorithmic extraction of information on several transparency and reproducibility indicators (code sharing, data sharing, registration, funding, conflicts of interest). We compared these features in articles published before and during the pandemic (in 2019 and 2021, respectively) and in articles on COVID-19-related models and models related to other infectious diseases.

We screened 2903 records in their titles and abstracts according to the eligibility criteria. 

Of the 1338 eligible papers (Table 1) , 216 had been published in 2019 and 1122 in 2021. 818 (61.1%) were COVID-19 papers and the second largest group contained 130 (9.7%) publications and was the group of General (theoretical models). More than 70 different diseases had altogether been modelled in the eligible publications. The model types were more evenly . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted April 16, 2022 Table 2 shows the transparency indicators overall and in the three main categories based on year and COVID-19 focus. We found that based on the text mining algorithms 288 (21.5%) articles shared code, 332 (24.8%) shared data, 6 (0.4%) used registration, and 1197 (89.5%) and 1109 (82.9%) contained a COI and funding statement, respectively. 919 (68.7%) of publications shared neither data nor code, while 199 (14.9%) of all papers shared both data and code.

We found no differences between years and between COVID-19 and non-COVID-19 papers in terms of probability of sharing data, registration, or mentioning of COIs. COVID-19 papers were more likely to share their code openly than the non-COVID- 19 

As shown in Table 3 , data sharing varied significantly across journals, e.g. it was 54.8%

in PLoS One, but 12.7% in International Journal of Environmental Research and Public Health.

Code sharing varied significantly across diseases, e.g. it was most common for dengue and least common for malaria (34.3% v 5.4%); and it varied significantly among types of models, (highest in agent-based models with 33.9% of publications sharing code). Registration was uncommon in all subgroups. COI disclosures were most common in dengue and least common in general . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted April 16, 2022. ; https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint models and they also varied by type of model (least common in compartmental models). Funding information was most commonly disclosed in dengue models and least commonly disclosed in general models; it also varied by type of model (being lowest in compartmental models); and by journal. Table 3 . Key transparency indicators per disease type, model type, and journal

Multivariable regressions (not shown) showed similar results. Code sharing was more common in COVID-19 models and in agent-based models. Data sharing was more common in spatiotemporal and agent-based models and also depended substantially on the journal. We did not perform multivariable regressions for the presence of COI and funding statements, since these depended almost entirely on the journal (several journals had 100% frequency of having a placeholder for such statements). Registration was too uncommon to subject to multivariable analysis.

We also checked a random sample of 29 (10%) of papers that were found to be sharing code, 33 (10%) of those sharing data, and all 6 that were registered. Of these, 24/29 (82.8%) actually shared code, 29/33 (87.9%) actually shared data and 5/6 (83.3%) were indeed registered.

The papers that used registration were two malaria models (14, 15) , one vector model (16) (which focused on malaria vectors) one polio (Sabin 2 virus (17)) model and one rotavirus model (18) .

The majority were from 2021 (14, 16, 17) and were also malaria models (two malaria and one vector that was essentially malaria (14) (15) (16) ) the majority we also classified as spatiotemporal (14) (15) (16) . Finally, of the 120 articles (10%) that text mining found that they contained a COI statements, there was indeed a placeholder for this ststement in all articles, but the vast majority of the statements (115 (95.8%)) disclosed no conflict at all. Of the 111 (10%) articles where text . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted April 16, 2022. ; https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint mining found that they contained a funding statement, all of them had indeed such a statement, but 13 (11.7%) stated that they had no funding.

Analysing 1338 recent articles from the field of infectious disease modelling we found that less than a quarter of these publications shared code or data, and only 14% shared both. This is concerning since it does not allow other scientists to check the models in any depth and it also limits their further uses. Moreover, registration was almost nonexistent. On a positive note, the large majority of models did provide some information on funding and COIs. However, the vast majority of COI statements simply said that they was no conflict. Furthermore, we saw no major differences between 2019 and 2021. COVID-19 and non-COVID-19 papers showed largely similar patterns for these transparency indicators, although the former were modestly more likely to share code and modestly less likely to report on funding. There were some differences for some of the transparency indicators across journals, model types and diseases.

Jalali et al. (11) analysed 29 articles on COVID-19 models in 2020 and found that 48% shared code, in 60% data was shared, whilst 80% contained a funding and COI disclosure respectively. Our findings show much lower rates of code sharing and data sharing. The Jalali et al. sample was apparently highly selective as it focused on the most referenced models among a compilation of models by the US Centers for Disease Control (19) . In another empirical assessment of the reproducibility of 100 papers in simulation modelling in public health and health policy published over half a century (until 2016) and covering all applications (not just infectious diseases), code was available for only 2% of publications (20) . Finally, in an empirical evaluation in decision modelling by Emerson et al. (13) , when the team tried to get authors of papers to share their code 7.3% of simulation modelling researchers responded and in the end . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022. ; https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint only 1.6% agreed to share their code. This suggests that infectious disease models are not doing worse than other mathematical models, and may be doing even substantially better, but there is still plenty of room for improvement in sharing practices.

There have been many initiatives for improving sharing code and better documentation in the modelling community (21-24) as well, repositories for COVID-19 models (25,26) . The modelling community including COVID-19 (27) modelling has had multiple calls for transparency and the debate of reproducibility has been ongoing for decades (28) (29) (30) . Several journals have tried to take steps in enhancing reproducibility. For example, Science changed their policy for code and data sharing to make both essentially mandatory (31) . However, Stodden et al. (32) found no clear improvement after such interventions. Models are published in a vast array of journals and sharing rate as well as reporting and documentation requirements tend to be highly journal specific.

The frequency of code and data sharing in our sample was higher than what was documented for the general biomedical literature that was assessed in Serghiou et al. (33) using the same algorithm. COI and funding disclosures were almost equally common. On the other hand, we observed a ten-fold lower registration rate in our sample compared with the overall biomedical literature, which may reflect the difficulty of registering models and the lack of sufficient sensitization of the field to this possibility (5) . We found that essentially 5 of our studies were registered (after validating the initial 6 that we found). Realising that registration may be difficult and even impossible for a large portion of models (exploratory models for instance) (5), it would still be advisable to register confirmatory studies of models that are destined to be used for policy to reduce the "vibration of effects" (the range of possible results obtained with multiple analytical choices) (34, 35) . Otherwise, promising output or excellent fit . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022. ; https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint may in reality be due to bias alone. When the stakes are high and wrong decisions may have grave implications, more rigor is needed.

The rates of COI and funding disclosures are satisfactory on face value, considering they both are above 80% both in our sample and across other empirical assessments (11, 33, 36) . This may also be due to the fact that both these types of disclosures have been introduced into many journal's routinely published items and there is a standard placeholder for them. Typically journals mandate a COI and funding statement. However, the fidelity and completeness of these statements is difficult to probe. We cannot exclude that undisclosed COIs may exist. Our random sample validation found that the COI disclosures almost never mentioned any conflict. Given the policy implications of many models, especially in the COVID-19 era, this pattern may represent under-reporting of conflicts. Funding disclosures were more informative with only 12% stating no funding, but even then unstated sources of funding cannot be excluded.

There are limitations in our evaluation. Our sample focused on the PubMed Central Open Access subset and not all PubMed-indexed papers. It is unclear if non-open access papers may be less likely to adopt sharing practices. If so, the proportion of sharing in the total infectious disease modeling literature may be over-estimated.

Second, we used a text-mining approach which has been extensively validated across the entire biomedical literature, but the algorithms may have different performance specifically in the infectious disease modeling field. Nevertheless, in-depth evaluation of a random sample of papers suggests that identification of these indicators is quite accurate and false positives are uncommon. We did not check manually for false negatives, because the algorithms have very . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022. ;  high negative predictive value, but it is possible that in a few articles some transparency indicators may have been missed.

Third, the presence of a data sharing or code sharing statement doesn't promise full functionality and the ability to fully reuse the data and code. This can only be decided after spending substantial effort to bring a paper to life based on the shared information. For COI and funding statements, we also only established their existence, but did not appraise in depth the content of these statements, let alone their veracity. Evaluations in other fields suggests that many COIs are undisclosed and funding information is often incomplete (37) (38) (39) .

Finally, we only assessed a sample that is drawn from two calendar years that are not very far apart, thus major changes might not have been anticipated at least for non-COVID-19 models. Nevertheless, 2021 was a unique year with a pandemic which of course affected the field not merely through inflation of publications(40) but also through specific funder and governmental initiatives and incentives. Therefore, only time will tell if any of the COVID-19 impact on the scientific literature will be long-lasting and if it may also affect the landscape of mathematical modeling in general after the pandemic phases out.

We found that in the highly influential field of infectious disease modeling that relies as much on its assumptions and underlying code and data, transparency and reproducibility have large potential for improvement. Yet, there is a growing literature of recommendations and tutorials for researchers and other stakeholders (41) (42) (43) (44) , plus the EPIFORGE guidelines (45) which are guidelines for the reporting of epidemic forecasting and prediction research. They all explicitly urge for code sharing, and data sharing and transparency in general. The current lack of transparency may cause problems in the use, reuse, interpretation, and adoption of these . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022. ;  models for scientific or policy activities. It also hinders evidence synthesis and attempts to build on previous research to facilitate progress within the field. Improved transparency and reproducibility may help reinforce the legacy of this important field.

This study is a meta-epidemiological survey of transparency indicators present in four common types of infectious disease models (compartmental, spatiotemporal, agent-

Subset of PubMed. The study is reported using the STROBE guidelines (46) . The code needed for the analysis of our data used R(47) and Python (48) .

We developed a search strategy to identify papers published from 2019 and 2021 in 

Compartmental models assign subsets of the population to different classes according to their infection status (e.g., susceptible exposed, recovered etc.) and models the . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022. ; https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint population parameters of the disease according to assumed transmission rates between these subsets [1] .

Spatiotemporal models explore and predict the temporal and geographical spread of infectious diseases (usually using geographic time series data).

(iii) Agent-based/ individual based models are computer simulations of the interaction of agents with unique attributes regarding spatial location, physiological traits and/or social behavior (50, 51) . Finally, (iv) Time-series models other than spatiotemporal were also included that use trends in number of infected or deaths or any other parameter of interest to predict future trends and numbers of spread (52) .

We excluded clinical predictive, prognostic, and diagnostic models and included only models of infectious agents that can infect humans (i.e. both zoonotic diseases as well as diseases exclusive to humans). All screening and analysis was conducted by EAZ in two eligibility assessment rounds. In the first round, eligibility was assessed based on the title and abstract; in the second where the model type and disease type was extracted, eligibility was also assessed by perusing the article in more depth. After this round, in unclear cases EAZ consulted JPAI and these cases were settled with discussion.

For each eligible study, we extracted information on the model type and disease type manually. For model type, whenever cases came up that were not clear-cut EAZ and JPAI conferred as to what category was sensible. Some phylogenetic models were included and classified as spatiotemporal if they had spatiotemporal aspects. When there were multiple model types in a single paper it was classified as 'Multiple'. For disease, we used categories defined . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022. ; https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint based on the infectious agent of interest that was studied. The "Unspecified" category included studies not mentioning a specific infectious agent but a clinical syndrome (i.e. urinary tract infection or pneumonia etc.), the "General (theoretical models)" category included studies that didn't model a specific disease (i.e. a theoretic pandemic). Finally, where multiple diseases were mentioned, the papers were categorised in a separate category as 'Multiple different agents' (i.e.

HIV and tuberculosis). Where vectors of diseases such as mosquitos were modelled to predict spread of multiple diseases, we classified the disease as 'Vector'.

For each eligible article we used PubMed to extract information on metadata that included PMID, PMCID, publication year, journal name and the R package rtransparent (33) to extract the following transparency indicators: (i) code sharing (ii) data sharing (iii) (pre-)registration, (iv) COI and (v) funding statements. rtransparent searches through the full text of the papers for specific words or phrases that strongly suggest that the aforementioned transparency indicators are present in that particular paper. The program uses regular expressions to adjust for variations in expressions. For example, to identify code sharing, rtransparent looks for "code" and "available" as well as the repository "GitHub" and its variations, and in a paper selected (53) from our dataset it finds the following: is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022 To further validate the performance of the algorithms in detecting code sharing and data sharing reliably, a random sample of 10% of publications that the algorithm identified as sharing code and 10% of those that the algorithm identified as sharing data were manually assessed looking into whether the statements indeed represented true sharing. All papers that were identified by the algorithm to have registration were assessed manually to verify whether registration had been performed. Moreover, arandom sample of 10% of papers that were found to contain a COI statement and 10% of those found to include a funding statement were assessed manually to see not only whether such statements were indeed present, but also to assess how many of them contain actual disclosures of specific conflicts or funding sources, respectively, and not just a statement that there are no COIs/funding, e.g. 'There is no conflict of interest', No funding was received' or 'Funding disclosure is not applicable'.

The primary outcome studied was the percentage of papers that include each of the transparency indicators. We considered three primary comparisons that were conducted using Fisher's exact tests. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022. ;  https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint Subsequently we also explored whether other factors may have correlated with the transparency indicators using Fisher's exact tests to see whether there was any statistically significant association (significance level set at 0.005 (54)) when comparing model types, year, disease modelled, as well as journal separately. We had pre-specified that whenever any statistically significant results were found, we would conduct multivariable logistic regressions as well.

We deviated from the protocol in that we didn't perform chi-square tests due to too low counts in some variables rendering it unreliable, therefore we decided to conduct these analyses using Fisher's exact tests instead of chi-square tests. The 10% manual assessment of a random sample of articles with COI and funding statements was added post hoc, when we realized that many articles could have such statements but they might simply state that there was no COI and/or no funding.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022. ;  https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint

The protocol for this study can be found at doi:10.17605/OSF.IO/JGWVK and the code and data will also be made available upon publication on the same site.

The authors declare no conflict of interest.

Stiftelse' and by the Meta-Research Innovation Center at Stanford (METRICS).

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022. ; https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 16, 2022 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted April 16, 2022. ; https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted April 16, 2022 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted April 16, 2022. ; https://doi.org/10.1101/2022.04.11.22273744 doi: medRxiv preprint

Wrong but Useful -What Covid-19 Epidemiologic Models Can and Cannot Tell Us

Effect estimates of COVID-19 nonpharmaceutical interventions are non-robust and highly model-dependent

Mathematical models to guide pandemic response

A case study in model failure? COVID-19 daily deaths and ICU bed utilisation predictions in New York state

Pre-registration of mathematical models

A Review of Multi Compartment Infectious Disease Models

Modelling the Global Spread of Diseases: A Review of Current Practice and Capability

The Challenge of Transparency and Validation in Health Economic Decision Modelling: A View from Mount Hood

Transparency assessment of COVID-19 models

On code sharing and model documentation of published individual and agent-based models. Environ Model Softw

Publication of Decision Model Source Code: Attitudes of Health Economics Authors

Spatiotemporal analysis and prediction of malaria cases using remote sensing meteorological data in Diébougou health district

Spatiotemporal clustering of malaria in southern-central Ethiopia: A community-based cohort study

Data-driven and interpretable machine-learning modeling to explore the fine

Rotavirus Seasonality: An Application of Singular Spectrum Analysis and Polyharmonic Modeling

Centers for Disease Control and Prevention

Evolution and Reproducibility of Simulation Modeling in Epidemiology and Health Policy Over Half a Century

A Systematic Review of Cost-Effectiveness Studies Reporting Cost-per-DALY Averted

The CellML Model Repository. Bioinforma Oxf Engl

Twenty years of ModelDB and beyond: building essential modeling tools for the future of neuroscience

Evaluating epidemic forecasts in an interval format

Call for transparency of COVID-19 models

Towards reproducible computational drug discovery

Open innovation: Towards sharing of data, models and workflows

Evidence informing the UK's COVID-19 public health response must be transparent. The Lancet

Making Data Maximally Available

Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals

Assessment of transparency indicators across the biomedical literature: How open is open?

Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations

Reproducible Research Practices and Transparency across the Biomedical Literature

Evaluating Industry Payments Among Dermatology Clinical Practice Guidelines Authors

Evaluation of Industry Relationships Among Authors of Otolaryngology Clinical Practice Guidelines

Top Cancer Researcher Fails to Disclose Corporate Financial Ties in Major Research Journals. The New York Times

The rapid, massive growth of COVID-19 authors in the scientific literature

Ten Simple Rules for Reproducible Computational

Enhancing reproducibility for computational methods

Tools and techniques for computational reproducibility

The FAIR Guiding Principles for scientific data management and stewardship

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies

The R Project for Statistical Computing

Welcome to Python.org

Toward Standardizing a Lexicon of Infectious Disease Modeling Terms. Front Public Health

Agent-based modeling: Methods and techniques for simulating human systems

Agent-Based Modelling: An Overview with Application to Disease Dynamics. ArXiv200704192 Cs Q-Bio

Use of time-series analysis in infectious disease surveillance

OpenABM-Covid19-An agent-based model for non-pharmaceutical interventions against COVID-19 including contact tracing

Redefine statistical significance