key: cord-0925028-5nsra16a authors: Van Gordon, M. M.; McCarthy, K. A.; Proctor, J. L.; Hagedorn, B. L. title: Evaluating COVID-19 reporting data in the context of testing strategies across 31 LMICs date: 2021-02-15 journal: nan DOI: 10.1101/2021.02.11.21251590 sha: ce19a761f7de2c47e4fbf825a1f519007d1e4a01 doc_id: 925028 cord_uid: 5nsra16a 0.1. BackgroundCOVID-19 case counts are the predominant measure used to track epidemiological dynamics and inform policy decision-making. Case counts, however, are influenced by testing rates and strategies, which have varied over time and space. A method to consistently interpret COVID-19 case counts in the context of other surveillance data is needed, especially for data-limited settings in low- and middle-income countries (LMICs). 0.2. MethodsWe leverage statistical analyses to detect changes in COVID-19 surveillance data. We apply the pruned exact linear time change detection method for COVID-19 case counts, number of tests, and test positivity rate over time. With this information, we categorize change points as likely driven by epidemiological dynamics or non-epidemiological influences such as noise. 0.3. FindingsHigher rates of epidemiological change detection are more associated with open testing policies than with higher testing rates. The non-pharmaceutical intervention most correlated with epidemiological change is workplace closing. LMICs have the testing capacity to measure prevalence with precision if they use randomized testing. Rwanda stands out as a country with an efficient COVID-19 surveillance system. Sub-national data reveal heterogeneity in epidemiological dynamics and surveillance. 0.4. InterpretationRelying solely on case counts to interpret pandemic dynamics has important limitations. Normalizing counts by testing rate mitigates some of these limitations, and open testing policy is key to efficient surveillance. Our findings can be leveraged by public health officials to strengthen COVID-19 surveillance and support programmatic decision-making. 0.5. FundingThis publication is based on models and data analysis performed by the Institute for Disease Modeling at the Bill & Melinda Gates Foundation. O_TEXTBOXResearch in ContextO_ST_ABSEvidence before this studyC_ST_ABSWe searched for articles on the current practices, challenges, and proposals for COVID-19 surveillance in LMICs. We used Google Scholar with search terms including "COVID surveillance." Existing studies were found to be qualitative, anecdotal, or highly location-specific. Added value of this studyWe developed a quantitative method that makes use of limited information available from LMICs. Our approach improves interpretation of epidemiological data and enables evaluation of COVID-19 surveillance dynamics across countries. Implications of all the available evidenceOur results demonstrate the importance of open testing for strong surveillance systems, bolstering existing anecdotal evidence. We show strong alignment across LMICs between workplace restrictions and epidemiological changes. We demonstrate the importance of considering sub-national heterogeneity of epidemiological dynamics and surveillance. C_TEXTBOX dence that these insights cannot be readily generalized to LMIC settings. 12,2 23 This leaves an important knowledge gap in understanding how to evaluate 24 and interpret COVID-19 epidemiological data from LMICs. 25 To address the gap in systematic interpretation and evaluation methods, we ical surveillance data. We make use of imperfect information despite data 32 weaknesses, deriving insights from information available in LMICs that may 33 otherwise be overlooked. The approach is fast and highly portable, well 34 suited to looking across countries, and has minimal data requirements. 35 In this article, we first present the methods for our analysis, including the The methods are outlined in Figure 1 for two example countries: South Africa 44 and Bangladesh. Details about each step are presented in the following sub- . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Change detection identifies changes that may be related to data quality, 87 stochasticity, and testing dynamics, in addition to epidemiological changes. 88 We classify the likely cause of changes identified by the PELT algorithm based 89 on which changes co-occur. This categorization simplifies the interpretation 90 of epidemiological surveillance, separates signal from noise, and enables broad 91 comparison across countries and testing dynamics. 92 We combine detected change points across cases, tests, and positivity time 93 series to create change point groups. The tolerance for temporal association 94 is set at ± seven days to account for seven-day smoothing and weekly data 95 reporting practices. These change groups are then categorized as shown in 96 Figure 2 , with details of the interpretation described in Appendix B. To 97 capture all changes that may be epidemiological, we include both categories 98 D and E as epidemiological change in our analysis. We note that these cate-99 gories are heuristically defined, however they are informed both by validation 100 using the EMOD simulations and a qualitative understanding of epidemio-101 logical surveillance dynamics. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) Figure 3 . . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251590 doi: medRxiv preprint ties, change point detection is sensitive to parameterization when applied to interpreted. 131 We illustrate the relevance of testing rates and the influence of testing policy 132 using time series for Bangladesh in the context of local events ( Figure 1 ). Case rates peaked in early July, an apparent epidemiological turning point if 134 case rates were considered alone. Simultaneously, however, there was a new 135 9 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted February 12, 2021. ; policy implemented to charge for testing and thus a decline in testing rate. 22 This resulted in no change in positivity and contradicts the interpretation of is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251590 doi: medRxiv preprint can approximate random sampling more closely than symptomatic testing. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. 3.4. Change detection rates and NPI alignment frequency vary across LMICs. 170 Figure 6 shows frequencies of change category detection across countries. Rwanda is an outlier with high epidemiological detection accompanied by low 172 non-epidemiological and noise detection rates. High rates of noise detection 173 are generally associated with low rates of epidemiological change detection, 174 whereas the relationship between noise detection and non-epidemiological 175 change detection is not consistent. 176 12 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251590 doi: medRxiv preprint R w a n d a . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251590 doi: medRxiv preprint 3.5. Alignment with detected epidemiological changes varies by NPI type. The NPIs most frequently aligned with epidemiological changes are work-184 place closures, public transport closures, and stay at home requirements. The percentage of these NPIs that are aligned with epidemiological changes 186 are 15·1%, 14·3%, and 12·2%, respectively. Note that there are substantially 187 fewer total public transit NPIs than workplace closures and stay at home CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251590 doi: medRxiv preprint requirements. Cancelling public events has the lowest frequency of epidemiological alignment at 6·12%, and also the fewest number of implementation 190 incidences. : 3.6. National-level results obscure sub-national heterogeneity in epidemiolog- ical dynamics and surveillance. To investigate sub-national heterogeneity, we conduct the same analyses as 194 above, but at the province level in South Africa. Figure 9A shows substantial 195 variability in provinces both by NPI alignment rate and by epidemiological 196 change detection rate. In line with results from national-level data, epi-197 demiological change detection rate is not correlated with mean testing rate. Because of reporting limitations, the NPIs here are national policies only. 199 We select three edge cases from the scatter plot in Figure 9A (Limpopo, 200 15 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251590 doi: medRxiv preprint that co-occurrence does not establish causality. In PELT change detection, 248 the changes detected are influenced by the choice of the sparsity parameter. In a sensitivity analysis of our novel parameterization approach, however, 250 we find that Rwanda remains the leader in surveillance system performance, 251 regardless of the parameterization choice. Results from this analysis highlight that surveillance data must be used care- . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. ; The case rate is defined as the number of individuals confirmed positive Weekly cases, testing, and death data are interpolated using a cubic spline. All daily cases, testing and death data are smoothed using a centered seven- CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. ; To evaluate the influence of penalty selection on our analysis results, we con-346 duct a parameterization sensitivity analysis. We compare results of country 347 23 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. ; ranking by epidemiological change detection rate for different penalty plateau . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. ; Detected change points across cases, testing, and positivity time series are CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) Population components: Testing components: Eligible rand = P opulation = SARS -CoV -2 + Non-SARS -CoV -2 Eligible sympt = CLI = Non-SARS -CoV -2 sympt CLI + SARS -CoV -2 sympt testing rate = T ests/P opulation testing coverage = T ests/Eligible 30 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251590 doi: medRxiv preprint Surveillance metrics: Cases detected = T ests * Cases Eligible P ositivity = Cases detected T ests = Cases Eligible Symptomatic testing: Cases detected sympt = T ests sympt * Cases sympt CLI P ositivity sympt = Cases sympt CLI Random testing: Cases detected rand = T ests rand * Cases total P opulation P ositivity rand = Cases total P opulation = P revalence . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251590 doi: medRxiv preprint of tests is very small, or probabilities are very close to zero or one. Un-482 der the condition of true random testing, positivity is a direct measure of 483 prevalence. At any given prevalence, margin of error can be calculated for 484 the number of tests administered and the total population. This calcula-485 tion is carried out for all LMIC countries in our dataset. Margin of error is 486 then normalized by the given prevalence rate. Based on these relationships, 487 ME (95 %)/P revalence is higher at lower prevalence. In other words, precise 488 measurement becomes increasingly more difficult as prevalence decreases. [27] WHO Regional Office, Africa, COVID-19 in Rwanda: A country's re-569 sponse, WHO -Regional Office for Africa (2020). CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251590 doi: medRxiv preprint CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251590 doi: medRxiv preprint Lockdown measures in response to COVID-19 in nine 494 sub-Saharan African countries Lab-496 oratory medicine in Africa: A barrier to effective health care Epidemiologic surveillance for controlling COVID-19 499 pandemic: types, challenges and implications ture, characteristics and components of COVID-19 surveillance system Surveillance is underestimating the burden of the COVID-505 19 pandemic Estimating 512 the effects of non-pharmaceutical interventions on COVID-19 in Europe The 516 effect of large-scale anti-contagion policies on the COVID-19 pandemic Inferring change points in the spread of 520 COVID-19 reveals the effectiveness of interventions Scenario analysis of non-pharmaceutical interventions 522 on global COVID-19 transmissions Physical distancing inter-526 ventions and incidence of coronavirus disease 2019: natural experiment 527 in 149 countries Coronavirus pandemic 529 Oxford 532 COVID-19 government response tracker Latest confirmed cases of COVID-19 in South Africa National Institute for Communicable Diseases, Weekly testing summary P0302 -mid-year population estimates Selective review of offline change 542 point detection methods 546 Implementation and applications of EMOD, an individual-based multi-547 disease modeling platform Estimation of incu-549 bation period distribution of covid-19 using disease onset forward time: 550 A novel cross-sectional and forward follow-up study Bangladesh's COVID-19 testing criticised Ac-555 curate statistics on COVID-19 are essential for policy guidance and de-556 cisions Linking statistics