key: cord-1049556-14m53hyi
authors: Bennett, Magdalena
title: All Things Equal? Heterogeneity in Policy Effectiveness against COVID-19 Spread in Chile
date: 2020-09-24
journal: World Dev
DOI: 10.1016/j.worlddev.2020.105208
sha: 5a6407fba39b0b8ff8e47421b532737bb21f80c9
doc_id: 1049556
cord_uid: 14m53hyi

Several variables and practices affect the evolution and geographic spread of COVID-19. Some of these variables pertain to policy measures such as social distancing, quarantines for specific areas, and testing availability. In this paper, I analyze the effect that lockdown and testing policies had on new contagions in Chile, especially focusing on potential heterogeneity given by population characteristics. Leveraging a natural experiment in the determination of early quarantines, I use an Augmented Synthetic Control Method to build counterfactuals for high and lower-income areas that experienced a lockdown during the first two months of the pandemic. I find substantial differences in the impact that quarantine policies had for different populations: While lockdowns were effective in containing and reducing new cases of COVID-19 in higher-income municipalities, I find no significant effect of this measure for lower-income areas. To further explain these results, I test for difference in mobility during quarantine for high and lower-income municipalities, as well as delays in test results and testing availability. These findings are consistent with previous results, showing that differences in the effectiveness of lockdowns could be partially attributed to heterogeneity in quarantine compliance in terms of mobility, as well as differential testing availability for higher and lower-income areas.

Like in most countries, the COVID-19 pandemic has deeply disrupted all aspects of daily life in Chile, starting from the way we approach our health care system, to job security, education, and even how we mobilize, among many others. Mitigating measures such as non-pharmaceutical interventions have become essential for preventing massive outbreaks and reducing systemic stress in hospitals and clinics. However, some of these interventions may have had differential effects depending on the population characteristics.

In this paper, I provide evidence of heterogeneity in the effectiveness of quarantines or lockdowns at the municipality level in Chile's capital city during the first two months of the pandemic. I also shed light on some of the reasons that could explain these differential effects, such as the probability of compliance with lockdown measures by area, as well as differences in testing availability for different groups. The Chilean context provides a particularly rich setting to analyze the effectiveness of lockdown measures, as quarantines were implemented during the first two months of the pandemic at the municipality level, which are the smallest administrative divisions in the country. Municipalities are some of the smallest areas that have experienced quarantine in comparison to other countries and allow me to analyze differential effects within a major city. Leveraging administrative data at the municipality level and an Augmented Synthetic Control Method approach, I build a counterfactual for higher and lower-income municipalities that entered quarantine from a pool of untreated areas that resemble the evolution pattern of the treated units before the policy was implemented, finding substantial differences in the effectiveness of quarantine measures by income.

Understanding the effectiveness of containment measures is of critical importance during a pandemic, but not only its average impact, but also the effect lockdowns had on specific populations of interest. In terms of policy evaluations, average treatment effects are often of limited value as they hide potential null or even opposite effects for specific groups of interest (Imai & Ratkovic, 2013) . In the case of quarantine policies, if certain populations are less likely to comply with lockdown measures due to differences in opportunity costs of staying at home, or asymmetry in information in terms of infections, then it is important to put in place complementary measures that will improve compliance.

Evidence related to the effectiveness of lockdown or shelter-in-place measures shows positive effects on the containment and reduction of the COVID-19 spread (Bonaccorsi et al., 2020; Bonardi et al., 2020; Dave et al., 2020; Flaxman et al., 2020; Hsiang et al., 2020; Patel et al., 2020; Prem et al., 2020; Vinceti et al., 2020) . For example, Hsiang et al. (2020) conduct an analysis of 1,700 non-pharmaceutical measures in six different countries, finding that quarantines and lockdowns were associated with a substantial slow-down of the spread of COVID-19. Flaxman et al. (2020) echo the previous results by modeling the transmission of the disease using data from 11 European countries. The authors find that lockdown measures are positively correlated with the containment of In the context of the current literature, this paper presents two main contributions: (i) the estimation of causal effects of small-area lockdowns on the spread of COVID-19 by leveraging a natural experiment, and (ii) the identification of heterogeneous effects by socioeconomic characteristics.

One of the main difficulties of identifying causal effects for quarantines is that their adoptions are associated with other confounding variables that make causal identification difficult or even unfeasible. However, the Chilean setting provides a solid natural experiment given the loose definition used to impose the first quarantines. Additionally, unlike in many other countries, lockdowns in Chile were applied at the smallest administrative level (municipalities). These small areas provide variation even within a city, allowing for the construction of better counterfactuals and estimation of differential effects according to municipality's characteristics. Given that measures that are effective for certain groups might not have the same effect on others, estimating the impact of lockdowns by income level can provide relevant feedback for policymakers to better understand how to target and complement current policies. This paper is structured as follows. Section 2 provides context of the Chilean case and the measures implemented to fight the spread of COVID-19. Section 3 outlines the augmented synthetic control method used for estimating the effects of lockdown policies by income at the municipality level and its results. In section 4, I discuss potential mechanisms that could explain part of the differential effectiveness of quarantines. Finally, section 5 concludes with some final remarks and further discussion.

The spread of COVID-19 in Chile started slowly, with its first confirmed COVID-19 case on March 3rd, 2020. During the first weeks of the pandemic, most of the spread of the virus was contained in the east side of Santiago, in the Metropolitan Region, which is the most affluent area of the city. However, by mid-April, the virus had already spread throughout the city, as it can be seen in Figure 1 .

To try to mitigate the spread of the virus, the Chilean government declared the first lockdowns during the final week of March 1 in 7 municipalities of the Metropolitan Region, and a few other cities across the country. The second wave of quarantines was around mid-April, which included three municipalities. Finally, by the end of April, three additional municipalities in Santiago entered lockdown. Figure 2 shows the location of the different quarantines in the Metropolitan Region by timing, and Table 4 in the Appendix shows the exact dates municipalities went into and out of lockdown. To provide some context on the units of analysis, Chile has 346 municipalities in total, which drastically vary in size: from a couple of hundred residents to over 500,000 in 2017 (Biblioteca 1 Only two other municipalities had been in quarantine before March 25th: Caleta Tortel and Rapa Nui. Caleta Tortel is a remote and small southern municipality which declared an early quarantine due to a cruise ship arriving with a confirmed case. Rapa Nui, or Easter Island, is an island far from the coast of Chile and took early precautions after its firsts confirmed cases. municipality (or part of a municipality) was declared on quarantine, people that lived in that area could not leave their residence without legal authorization.

Additionally, non-residents were not allowed to transit in quarantined areas either. Essential services were still open during this time.

To assess the effect that quarantines had on the evolution of new cases at the municipality level, a natural approach would be to compare treated areas with those that are similar but were not affected by lockdowns, assuming that conditional on some observable features, the assignment of quarantines was random. In this context, Synthetic Control Method lends itself nicely to estimate a causal effect of these policies (Athey & Imbens, 2017) .

Synthetic Control Method (SCM) is a popular approach in causal inference settings that, under certain assumptions, provides a valid counterfactual for a unit that was treated at a specific point in time (Abadie & Gardeazabal, 2003; Abadie et al., 2010 Abadie et al., , 2015 . SCM uses a weighted combination of untreated units from a donor pool to build a "synthetic" version of the treated one that resembles its behavior prior to the intervention. Some of the main advantages of the SCM are that it does not rely on extrapolation (weights are non-negative and sum to one), and that it is a transparent method in the sense that makes the differences between treatment and its counterfactual explicit, as well as the contribution of each of the control units (Abadie, 2019).

Using the notation on Ben-Michael et al. (2020) , Equation 1 shows the typical setting for SCM where one unit i is treated (i = 1) in period T 0 < T . W i is a treatment indicator for unit i, and weights γ scm i ∈ [0, 1] are estimated to minimize the difference in pre-intervention trends between the treated unit and the synthetic control. These weights are then used to approximate the potential outcome under control of the treated unit in the post-intervention period T ,

Following a similar setup, the Augmented Synthetic Control Method (ASCM)

proposed by Ben-Michael et al. (2020) also builds a counterfactual for the treated observations based on a weighted combination of untreated units. However, the main advantage of ASCM over traditional synthetic control methods is that it provides "bias correction" when pre-treatment fit is imperfect. This bias correction means that even when the synthetic control does not closely follow the path of the treatment group in the pre-intervention period, ASCM provides a method to de-bias the original SCM estimate using, for example, a ridgeregularized linear regression as an outcome model. The trade-off, in this case, is that ASCM allows non-negative weights to improve pre-treatment fit, but the method focuses on minimizing extrapolation outside the convex hull.

Using the same setup as the one described for 1, Equation 2 shows how ASCM would be applied in a setting were there is potentially a poor fit in a particular function of pre-intervention outcomes,m(·) (Ben-Michael et al., 2020) . In this case, the estimated counterfactual for SCM is "corrected" by the observed imbalance. This setup can be extended for more complex settings, such as multiple treated units, staggered designs, and different choices of estimators (Ben-Michael et al., 2019 .

In SCM and ASCM settings, three assumptions need to hold to estimate a valid average treatment effect on the treated: i) assignment of the treatment is random conditional on the donor pool, observable covariates, and preintervention path of the outcome, ii) Stable Unit Treatment Value Assumption (SUTVA), and iii) the intervention had no effect prior to its start date.

The first assumption relates to the idea that synthetic control methods use a weighted average of units from the donor pool to build an estimate of the missing potential outcome, and those units are chosen based on pre-intervention fit. Overall, the decision to declare quarantine in a municipality at the beginning of the pandemic depended on diverse factors, but the most important one was the progression of the spread: Areas with a higher number of total cases and an increasing number of new cases were likely candidates for this policy. However, decisions had a political component as well, which provides a level of exogeneity to the decision that makes the conditional ignorability assumption likely to hold in this setting. For example, when the first lockdowns were declared in March 25th, there were three other municipalities not affected that had the same or even a higher number of total cases than most of the areas that entered quarantine, and actually, the municipality of Independencia, which entered an early lockdown, was just top 30 in number of cumulative cases. The story is similar when we look at daily new cases: 3 out of the top 10 municipalities with the highest number of daily cases on March 25th did not experience quarantine.

These three municipalities were also located high in the ranking of total cases, and are high-density population areas.

The second assumption, SUTVA, implies that the intervention only affects the treated units and does not affect non-treated municipalities. The geographic nature of lockdowns, however, might make this assumption less likely to hold: If residents from a municipality on lockdown were more likely to leave and mobilize to nearby areas that were not subjected to quarantines, these spillovers would break SUTVA. To avoid potential effects on neighboring municipalities, I build a buffer zone around treated units using the areas that would be more likely to experience spillovers due to quarantines and exclude these zones from the donor pool. I also exclude two coastal municipalities (Vina del Mar and Concon), given that they are a common second-home destination for residents of Santiago.

Regarding the final assumption, anticipation effects could potentially play a role in the estimation of the effect of lockdowns if there is an important lag between the announcement of the measure and the start of quarantines. Given that people usually need to prepare for quarantine, it is not uncommon to see surges in mobility in the period between the announcement and the beginning of lockdowns. In the case of Chile, the first and second waves of quarantines were announced one and two days before their implementation, respectively. 2 Even though the period between announcement and enforcement is short, I use the day of the announcement as the starting period of treatment, to avoid potential confounding.

To estimate the effects of lockdowns on new cases over time, I use publicly available data from the Ministry of Health Epidemiological Reports (Departamento de Epidemiologia, 2020) on the number of new cases by municipality over time. Because these reports are only delivered every 2 to 3 days, I rely on interpolation to build daily data based on daily regional contagion rates, closely approximating the number of daily cases at the municipality level. 3

For analyzing heterogeneous effects by income at the municipality level, I take three sets of municipalities for the period between March 15th and May 8th: 1) high-income areas that had quarantines before April 30th, 2020 in the capital city, 2) lower-income areas that had quarantines prior to the same date in the same region, and 3) other large municipalities that did not have lockdown measures before April 30th. 4 The latter group serves as the "donor" pool or counterfactual pool for the first two. are not considered in the previously described high-income group. Average characteristics for these different groups are shown in Table 1 . Using a Ridge ASCM approach (Ben-Michael et al., 2020) and following the previously described groups of analysis, I build counterfactuals for three groups of municipalities in Santiago that experienced quarantines: (i) highincome municipalities (H), (ii) lower-income municipalities (L), and (iii) the combination of the previous two groups (A), such that A = H ∪ L. Then, for each group g, I estimate the optimal weights for the municipalities in the donor pool that balance pre-intervention trends (e.g. daily number of cases in group g) and baseline covariates (i.e. income per capita, population density, total number of cases, and poverty rate), allowing for extrapolation but penalizing the departure from traditional SCM weights. Table 2 and Figure 3 show the overall effect of quarantines on the number of new cases over time for all municipalities that had quarantines in the Metropolitan Region. 5 I use a 12-day mark to assess the effectiveness of the quarantine because that is the number of days needed for most people to develop symptoms (Lauer et al., 2020) . Estimates for each period are obtained using the Ridge ASCM weights previously described, and standard errors are estimated using a jackknife method. 6 . Figure 3 shows that after the 12 days since the start of the lockdown period, treated municipalities experience a lower number of new cases over time, though the difference is not statistically significant at conventional levels and the magnitude of the effect is also modest.

This seemingly positive result, however, hides an important degree of heterogeneity in terms of the effectiveness of quarantines. When I analyze higherincome municipalities in Santiago compared to other municipalities, we can see that the results differ by socioeconomic level (Figure 4 ). municipalities. By the end of the series, confidence intervals are too wide and I do not have enough statistical power to reject a null effect, but the pattern shows However, Figure 4b shows another side of the story. In this case, even though the effect is not statistically significant at 10% level, the estimate is opposite to what we would expect: there is an increase in the number of new cases over time even after the 12-day mark. In this case, given that some lower-income municipalities entered lockdown later in time, the post-quarantine period of analysis is shorter compared to higher-income municipalities. However, the same results stand when using only early-adopters of quarantines for the lowerincome group.

One potential concern for the identification of causal effects, in this case, would be that people change their mobility patterns from one municipality in lockdown to a neighboring area which is not in quarantine, breaking SUTVA.

To avoid confounding the effect with potential spillovers, I run a robustness check of the ASCM by including a buffer zone around treated municipalities, which are excluded from the control pool (see Appendix for a map of the buffer zones). Results are shown in Table 3 , and they are very similar to the original estimated average treatment effects on the treated for both groups.

As it was previously mentioned, another concern in terms of comparison of the effects between high-and lower-income municipalities is the timing of the lockdowns. If lower-income areas were systematically treated later in the pandemic, it could be the case that the timing factor is confounding my results. To avoid this potential issue, I only compare high-and lower-income municipalities that were treated roughly at the same time, using the same set of high-income areas, but only Santiago and Independencia as lower-income municipalities. Conclusions remain unchanged, as overall effects of lockdown measures were not effective in Santiago or Independencia, lower-income areas that were subject to quarantine at the same time that high-income municipalities ( Figure   5 ).

Finally, for each of the groups analyzed, I exclude the municipality with the highest weight to see whether is one unit from the donor pool that is driving the results. 7 The exclusion of these municipalities do not substantially affect the conclusions, and results remain mainly unchanged. 8 7 For high-income areas I exclude Concepcion, for lower-income areas I exclude Renca, and for all the municipalities that were in quarantine I exclude Maipu 8 Results available in the public replication code. 

In this section, I study two different, but potentially complementary, hypotheses that could explain the heterogeneity in the effect of lockdowns on the number of new infections. The first one relates to the differential ability of households to stay at home during a lockdown, and the second one to the availability and timing of testing in different municipalities.

Mobility has been highly reduced during the pandemic, both because of individual changes in behavior, but most importantly due to government policies aimed at avoiding contagion. Figure 6 shows data from Google Mobility Reports for the Metropolitan Region, which compares the number of visitors to transit stations over time between mid-February and early May to a baseline before the pandemic. 9 I focus on transit data, as it is a measure that most likely approximates mobility to and from work during labor days. As Figure 6 shows, the biggest drop in mobility was given by the closure of schools, with a 56% decrease. After that, there are no clear changes in mobility rates considering the active quarantines in Santiago. Excluding the day right before and after quarantines, there is no distinctive change in mobility for labor days when only the first wave of lockdowns was in place (i.e. high-income municipalities plus Santiago and Independencia), and when the second one started during that same period (i.e. half of the municipality of Puente Alto).

Given that quarantines were targeted to specific areas in the Metropolitan Region and were not widespread, the aggregated mobility map may be hiding movement reductions for those areas. To test potential differences, I use data from the Mobility Index Report created by the Data Science Institute from UDD and Telefonica (Universidad del Desarrollo, 2020) to analyze changes in mobility before and after quarantine, using the same Augmented Synthetic Control

Method described in the previous section. The mobility index presented here 9 Baseline consists of the period between January 3rd, 2020 and February 6th, 2020. To further analyze these mobility patterns, I compare data on subway validations 11 provided by the Ministry of Transportation. Figure 8 shows year-over-year percentage changes in subway 12 validations during the morning for the same period between 2019 and 2020. Dates have been adjusted in 2019 to align with major events in 2020 (first Monday in March and Easter weekend).

As it can be seen in Figure 8 , high-income municipalities in quarantine had a higher reduction of their subway mobility compared to both lower-income areas in quarantine and other municipalities that were not affected. On average, after the announcement of the first wave of lockdowns, the difference between high and lower-income areas in quarantine was 8 percentage points. High-income municipalities appear to be more sensitive to overall policy measures and reduced their overall mobility patterns even earlier than expected.

Some of these differences could be due to a higher ability for smoothing con- sumption (e.g. savings) or work-from-home opportunities that lower-income households do not have available, but more data would be required to study these hypotheses.

One important factor that highly influences the containment of COVID- Publicly available testing data supports these differences in testing availability. By analyzing the correlation between private center testing and estimated positivity rate using publicly available data (Ministerio de Salud de Chile, 2020), 14 I find a significant negative correlation of -0.61 for the month of April. 15 This means that when private centers increased their share of testing, the number of new cases over total testing dropped (Figure 9 ), while the opposite was true for public-center testing. These patterns provide additional evidence that testing was more extensive in the private sector, where patients got tested with a lower probability of actually being infected. The difference between private and public center availability becomes particularly relevant when we consider that in high-income municipalities that had lockdown policies only 32% of residents were subscribed to the public health care system in 2017, while 13 The need for a physicians' referral, as well as the cost of the test, are some of the barriers that differentially affected lower and higher-income populations. 14 Estimated positivity rate refers to the number of new daily cases over the number of informed tests for that day. 15 Testing data is available continuously for type of testing center since 3/31/2020, and I use this data until 04/28/2020, given that the testing strategy changed to broader testing after that date.

that number was 78% for lower-income areas that had quarantines before May 5th (Observatorio Social, 2017). In addition to the availability of testing for different groups, proper timing plays a key factor in the spread of COVID-19. Even though there is no official data related to the timing from testing to diagnosis, some reports show that test results could take between two to five days to get a result confirmation, depending on the lab (La Tercera, 2020). If we add these differences to the time it takes for patients to get tested, the period between getting infected and having a confirmed diagnosis could likely be over a week.

Delays between test and diagnosis present major pitfalls in containment COVID-19 test, and also some degree of non-compliance with self-quarantine measures for patients that are waiting for their results.

Using publicly available data provided by the Ministry of Health (Departamento de Epidemiologia, 2020), I combine different reports that gather the week of first symptoms by municipality over time. With this data, I estimate the difference in new cases between reports and the difference in the number of patients that reported initial symptoms each week. Using the fact that reports are made available every two to four days, I estimate the maximum days between first symptoms and diagnosis for these new cases and calculate the estimated proportion for days between first symptoms and confirmation, shown in Figure 10 .

As it can be seen in Figure 10 , for both high and lower-income municipalities,

there is a lag of at least three days between showing first symptoms and getting a confirmed diagnosis. However, differences become significant after the 4-day mark: While nearly a quarter of new cases in high-income municipalities have a confirmed diagnosis by day 5, only 15% of newly infected cases in lower-income municipalities have a confirmed test by that date. These differences remain fairly stable 7 days since first symptoms and shed light on the fact that lower-income areas that experienced quarantine have almost consistently less timely diagnosis, which can result in a higher number of infections of close contacts during that time. One caveat of the data provided by the Ministry of Health is that is subject to changes between reports that could be due to reasons other than new cases (e.g. further investigation about initial symptoms); however, if we assume this measurement error is not systematically correlated with municipality's income, the difference between both curves is still informative. 

The effectiveness of mitigation measures plays a paramount role in contagion containment during a pandemic. For the same reason, understanding how interventions work for different populations is key for designing and implementing policies that will actually help reduce the spread of COVID-19. In this case, average treatment effects can hide potentially harmful evidence for populations that are more at risk.

The differential risks of contagion between socioeconomic groups is particularly important when considering the health, economic, and general welfare ramifications of the pandemic. Lower-income groups are usually the ones that have fewer protection mechanisms to overcome a health crisis, so mitigation policies should be especially tailored to protect vulnerable populations.

In this paper, I show evidence that small-area lockdown measures had a differential effect on high and lower-income populations. While quarantines proved effective in reducing new daily cases in more affluent areas, they did not have a significant effect on lower-income municipalities. Timely testing and difference in opportunity costs for staying at home might have played a role in explaining part of this difference. These results suggest that mobility-reduction measures are not equally effective, and current policies should potentially be accompanied by complementary measures that boost the effectiveness of lockdowns.

The causal identification of these effects relies on the assumption that a valid counterfactual for municipalities in quarantine can be built from those areas that did not experience lockdowns. Even though the loose criteria used by the Chilean government lends itself nicely as a natural experiment in this setting, the method employed here does rely to a certain extent on extrapolation to build a valid comparison group. Even though the use of Ridge ASCM penalizes the departure from traditional SCM weights, it is important to consider the context of the problem and deem whether the trade-off between extrapolation and better fit is appropriate for the setting.

Finally, given the speed with which governments need to act during a pandemic, more data, better data, and timely data are needed to assess some of these policies when they are implemented and provide swift feedback that could help improve or complement current interventions. Note: High-income municipalities in blue, and lower-income municipalities in green. Municipalities in white are excluded from the analysis because (i) their quarantine period is too short within the analysis window or (ii) they are not located in the capital region. Source: Data collected from the Chilean Health Ministry and publicly distributed by @soyDgo

Using synthetic controls: Feasibility, data requirements, and methodological aspects

Synthetic control methods for comparative case studies: Estimating the effect of california's tobacco control program

Comparative politics and the synthetic control method

The economic costs of conflict: A case study of the basque country

The state of applied econometrics: Causality and policy evaluation

Synthetic controls and weighted event studieswith staggered adoption

The augmented synthetic control method

Reportes Estadisticos Comunales

Economic and social consequences of human mobility restrictions under COVID-19

Fast and local: How lockdown policies affect the spread of COVID-19

Encuesta de Ocupacion y Desocupacion en el Gran Santiago

When do shelter-in-place orders fight COVID-19 best? Policy heterogeneity across states and adoption time

Informe Epidemiologico: Enfermedad SARS-CoV-2 (COVID-19)

Imperial College COVID-19 Response Team

Estimating the effects of non-pharmaceutical interventions on COVID-19 in europe

Informe Trimetral Mercado Inmobiliario Gran Santiago

Mobility Reports

The effect of large-scale anti-contagion policies on the COVID-19 pandemic

Estimating treatment effect heterogeneity in randomized program evaluation

Impacts of delays on effectiveness of contact tracing strategies for COVID-19: A modelling study

Cuanto tardan los resultados de los test? Gobierno admite demora de al menos 48 horas y hospitales, de hasta cinco dias

The incubation period of

 Figure 11 : Map for high and lower-income municipalities that had lockdowns before May 5th, including buffer zones used for robustness test.