key: cord-0109727-6z6g211p
authors: Hwang, Neil; Chatterjee, Shirshendu; Di, Yanming; Bhattacharyya, Sharmodeep
title: Estimating the treatment effect of the juvenile stay-at-home order on SARS-CoV-2 infection spread in Saline County, Arkansas
date: 2020-09-18
journal: nan
DOI: nan
sha: 04a44f876f6fd7016dd6b8a4bbb8e211700b4016
doc_id: 109727
cord_uid: 6z6g211p

We investigate the treatment effect of the juvenile stay-at-home order (JSAHO) adopted in Saline County, Arkansas, from April 6 to May 7, in mitigating the growth of SARS-CoV-2 infection rates. To estimate the counterfactual control outcome for Saline County, we apply Difference-in-Differences and Synthetic Control design methodologies. Both approaches show that stay-at-home order (SAHO) significantly reduced the growth rate of the infections in Saline County during the period the policy was in effect, contrary to some of the findings in the literature that cast doubt on the general causal impact of SAHO with narrower scopes.

In response to rising numbers of Covid-19 cases, state governments have implemented a wide-ranging array of policies in the form of non-pharmaceutical interventions (NPIs) aimed at slowing the rate of growth of the SARS-CoV-2 infections. At times, local governments have stepped up and issued orders when the policies they deemed necessary were not implemented at the state level. Much of this policy response has focused on enforcing social-distancing through measures ranging from temporary closures of public-facing businesses and shelter-in-place orders (SIPO) to mandatory mask-wearing orders [Abouk and Heydari (2020); Friedson et al. (2020) ; Courtemanche et al. (2020) ; Chernozhukov et al. (2020) ; Dave et al. (2020) ; Hsiang et al. (2020) ]. The prominence of state and local governments in implementing policy responses vís-a-vís the federal government is explained by the fact that the jurisdictional authority to do so rests with the former [Dave et al. (2020) ].

Of the various policies that have been implemented, SIPO has been found to be among the most effective in slowing the growth of infections in the U.S. There have been a plethora of studies on association between SIPO and SAHO and SARS-CoV-2 infection spread [Gao et al. (2020) ; ; Lurie et al. (2020) ]. Abouk and Heydari (2020) used the difference-in-differences methodology to conclude that statewide stayat-home orders (SAHO) showed the strongest causal effect on reducing social interactions at the state-level [Abouk and Heydari (2020) ]. Courtemanche et al. (2020) noted significant causal impact of interventions, including SIPO, in reducing case growth rates at the county-level [Courtemanche et al. (2020) ]. Chen et al. (2020) also investigates the causal impact of SAHO on mobility and SARS-CoV-2 infection spread. Regarding SIPO, Dave et al. (2020) noted that while its effectiveness is most notable when adopted in dense areas early in the pandemic, its impact declines significantly when implemented later [Dave et al. (2020) ]. Friedson et al. (2020) used synthetic control research design to find that SIPO in California reduced both cases and deaths. It was also noted that SIPO is the most restrictive form of social-distancing measure with its compliance assurance coming from law enforcement and punitive fines [Dave et al. (2020); Friedson et al. (2020) ; Caswell (2020a); Napoleon (2020)], as well as social pressures [Dave et al. (2020) ; Ronayne and Thompson (2020)] .

A similar but more lenient "advisory" SIPO or stay-at-home order (SAHO) discourages members of the public from leaving their homes other than for medical emergencies, commuting to work, or shopping for necessities. SAHO with narrower scopes have also been issued, pertaining to only a certain segment of the resident population. Juvenile stay-at-home orders (JSAHO), such as that issued in Saline County of Arkansas [Caswell (2020b) ], comprise one such example and allow juveniles to leave their homes if accompanied by an adult. However, strict adherence to such measures is often not enforced; instead, its effectiveness to a large extent relies on the public's willingness to modify their behaviors to comply in light of the pandemic and SAHO [Caswell (2020b) ]. Abouk and Heydari (2020) found that SAHOs with narrower scopes did not show significant causal effect in reducing infection rates at the state level [Abouk and Heydari (2020) ]. Then, the natural question arises as to how effective such measures are that rely, at least in part, on the public to voluntarily comply.

We answer this question by examining JSAHO that Saline County in Arkansas adopted as a policy response to the pandemic and the nature of its impact, if any, while the policy was in effect. As one of five states that did not impose either SIPO or SAHO as of September 5, 2020, Arkansas is unique in that one of its counties, Saline County, nonetheless issued its own SAHO with the scope restricted to those under the age of 18. This provides an invaluable setting for natural experiments to assess causal effects of policy actions adopted at the county level, such as JSAHO, while controlling the effects of other policies that have been concurrently implemented at the state-level. In particular, examining counties in one state allows us to control for the effect of the virus testing on cases since the availability of tests is constant at the state level.

A number of methodologies have been used in the Covid-19 policy evaluation literature to estimate average treatment effects using panel data. Among the more popular methodologies include Difference-in-Differences (DID) [Abouk and Heydari (2020) ], propensity scores matching (PSM) [Courtemanche et al. (2020) ], event-studies [Abouk and Heydari (2020) ; Courtemanche et al. (2020) ; Dave et al. (2020) ], and synthetic control research designs (SC) [Friedson et al. (2020) ; Abadie and Gardeazabal (2003) ; Abadie (2005) ; Abadie et al. (2010 Abadie et al. ( , 2015 ]. Of these methods, SC is a relatively novel method in the broader econometrics literature and has gained an increasingly larger following in recent years, spawning several varieties and enhancements in the process.

The rest of the paper is organized as follows. In Section 2, we state the terms and notations used throughout this paper. In Section 3, we provide an overview of the standard DID methodology, apply it empirically to the Covid-19 data in Arkansas, discuss the results, and note key shortcomings of the approach. Similarly, in Section 4, we introduce the SC methodology in its standard form and discuss empirical findings resulting from its application to the data. In Section 6, we reference the sources of the data used for empirical analyses in this paper. We conclude with some suggestions for further study on both the empirical and methodological fronts. Lastly, various visualizations and regression output are shown in the Appendix.

As is standard in the literature on policy evaluation, we use the term "treatment" to specifically refer to JSAHO and "treatment group" to refer to Saline County, AK, since it was the only county in which residents received the treatment. Similarly, "control group" refers to some set of counties in Arkansas other than Saline County, where individual counties of the control group are referred to as "control units."

We denote infection rates in county i belonging to group g ∈ {1(treatment), 0(control)} in Arkansas at time t by Y g it . As noted earlier, the only county in the treatment group in our study is Saline County, and the membership composition of the control group would vary depending on the methodology under study, as discussed below. Without loss of generality, we assign Saline County the index i = 1, and denote its infection rate by Y 1 1t , and the counties in the control group Y 0 jt , with j ∈ [2, 75]. The time variable is binary, 0 for pre-treatment period, i.e., before JSAHO was issued on April 6, and 1 for treatment period, i.e., April 6 through May 7 when the order was lifted in Saline County. The days after May 7 are referred to as the post-treatment period.

K denotes the number of pertinent covariates included in a given model to explain the variation in pretreatment infection rates of Saline County and N refers to the number of control units. Then, X 0 is a K × N matrix containing the values of the covariates for N control units. For the treatment unit (Saline County), we denote by X 1 a K × 1 vector of pre-treatment values of K covariates.

3 Difference-in-Differences (DID) Also known as a methodology for "natural" or "quasi-experiments," DID has been a popular methodology of choice among applied researchers in policy analysis for its simplicity and intuitive appeal. In particular, the JSAHO setting in Arkansas lends itself well for DID analysis. We first discuss the DID methodology in light of the pandemic setting, followed by our empirical study design and findings.

The estimate for the treatment effect is simply the difference between mean reduction in infection rate in the treatment county and that in a control county, where this double differencing is meant to remove biases due to county fixed effects and time effects.

In its canonical form, the estimand for the treatment effect of the policy, denoted τ , is expressed in turn as follows:

The foremost assumption in DID (known as the parallel trends assumption) is that the time trends for both groups are identical, and hence the subtraction of the two expectations is designed to eliminate the common time effect. With this assumption, the DID estimate of the treatment effect is unbiased. τ is typically estimated fitting a linear regression line of the following form [Ashenfelter and Card (1984) ]:

where α, β, and γ are model parameters for the intercept, the time coefficient applicable to both groups, and the group-specific coefficient, respectively. Finally, ε i represents unobservable traits of county i, and are assumed to be independent of g i and t i , and that E[ε i ] = 0.

The estimate for τ is thuŝ

As stated earlier, the most crucial assumption in the DID design is that the groups exhibit parallel trends over time so as to allow for the cancellation of the time effects in differencing the differences. This assumption is crucial since one does not observe the counterfactual infection rate of the treatment unit in absence of the treatment Y 0 11 , i.e., the infection rate that Saline County would have had it not implemented JHAHO. The parallel trends assumption allows us to infer the counterfactual outcome based on Y 1 10 , Y 0 i0 , and Y 0 i1 with g i = 0 and i ∈ [2, 75].

For this reason, a standard approach is to choose as a control a county that has the pre-treatment period time trend similar to that of Saline County, where the pre-treatment period starts from the date the first case was reported until the day before the start of the JSAHO effective dates. More concretely, one would choose a county that exhibits the slope coefficient that is close to that of Saline. However, one cautionary note regarding time trends in the pre-treatment period is that control units typically exhibit multiple time trends in the pre-treatment period.

For example, Figures 3.1a and 3.1b show the time series plots of cumulative case counts in Conway and Benton Counties. The dotted red lines represent the dates when the slopes of the fitted curves change, also known as "change-points" or "knots."

It is obvious from the figures above that the time trends for both counties change depending on the specific date ranges one considers.

For this reason, when selecting candidate counties for control units, we first identified for each county change-points where the changes in the slopes of fitted lines were significant. We used the knot-selection algorithm in the adaptive splines method discussed in Goepp et al. (2018) and implemented in the A-Splines R package.

Then, we considered only the dates after the most recent change-point when fitting a line for each county to estimate the time trend. We selected the counties that had slope coefficients significant at the 5% level with adjusted R-squared of 0.75. Figure 3 .2 shows 11 counties obtained as a result of this procedure and constitute potential control units listed in order of the absolute deviation of the slope coefficient from that of Saline County. Figure 7 .2 the treatment unit is more dense and has lower mortality rates, although whether the latter affects the infection rates is uncertain. The control unit has a lower median age. In Figure 7 .3, the density appears to be about the same for both counties. The treatment unit has lower mortality and poverty rates.

When considering each potential control unit separate, we found that the treatment effect was significant when using 7 of the 11 control units. To illustrate, we first consider 2 such control units, Pulaski and Garland. Both counties had relatively small absolute deviations in pre-treatment trends and had similar values for several key covariates in comparison to Saline County. Then, we assess results based on the other remaining control units. Figures 3.4a and 3 .4b below show the infection rate trends of Saline vs. Pulaski and Garland counties juxtaposed in the pre-treatment period. The blue dotted line indicates the date when JSAHO was issued on April 2 and the solid blue line indicates the start of the effective dates of April 6. One can confirm that the time trends starting on March 30 and onwards for the treatment and control units are very similar in both graphs. Indeed, based on the infection rates between the last change-point (March 29 for Saline and Pulaski, and March 30 for Garland) and the start of the treatment period, the slope coefficients are 2.52, 2.38, and 2.17 for Saline, Pulaski, and Garland counties, respectively.

Lastly, as noted in Bertrand et al. (2004) , conventional standard error estimates for the treatment effect using the OLS (3.1) often suffer from downward bias due to serial correlation in infection rates within each county. Prior works in the literature have addressed this issue by clustering standard errors at the geographical level where measurements are taken [Chernozhukov et al. (2020) , Abouk and Heydari (2020) ]. For this reason, we cluster standard errors at the county level.

It has been documented that the incubation period of the Coronavirus ranges from 2 to 14 days, with the median of 5 days [Guan et al. (2020) Figures 3.5a and 3.5b show the the mean infection rates of the 5-county and 11-county control groups compared to that of the treatment unit over the pre-treatment and treatment periods, plus 7 days. Visually, one can confirm that there is a relative reduction in the growth rate of infections in Saline during the treatment period in both cases. In both cases, 3.5a, the downward effect on the infection rates appears to continue after May 7, the last day of the treatment period, which is consistent with observations made in prior works regarding the lag in policy effectiveness [Abouk and Heydari (2020) ]. To estimate the treatment effect parameter, we fit Equation 3.1 to assess the value of the coefficient τ . As shown in Figures  3.6 and 3 .7, the estimand for τ is the coeffient for the parameter dc : dt, where dc is an indicator for the county fixed effects and dt is an indicator for the time effects. The coefficient for dc : dt is −1.85 × 10 −0.4 with the 5-county control group and −4.08 × 10 −0.4 with all 11 counties comprising the control group. Both estimates are shown to be significant based on clustered standard errors at the county level. Interestingly, the time trends in the pre-treatment period are not parallel, with Saline County showing a steeper slope than the mean of the control counties' trends. However, the reduction in infection rate for Saline County vís-a-vís the control is nonetheless obvious. Figure 3 .12 shows the comparative time series of infection rates for all control counties except Clark county. One can visually confirm the presence of the treatment effect in 5 of the 8 remaining counties, namely, Craighead, Pope, Cross, Jefferson, and Crittenden. Figure 3 .12: Infection rates for Saline (red) vs. the Remaining Counties as the Control (blue) In Appendix, OLS output with clustered standard errors for all 11 control units in comparison to treatment unit (Saline county) is given in Figure 7 .12. Figure 7.1-7.11 shows how all the counties compare to the treatment unit (Saline county) along the covariates identified in Section 4. Saline is more dense with lower mortality rates and residents in poverty, while Garland has a lower median age and more hospitals.

SC was proposed by Abadie et al. in a series of seminal papers [Abadie and Gardeazabal (2003) ; Abadie et al. (2010 Abadie et al. ( , 2015 ] to estimate the counterfactual outcome of the treatment unit in absence of the treatment by using a weighted average of control units. Here, the weights of the control units are considered nuisance parameters that are estimated to arrive at the SC estimator. A useful quality that makes SC particularly apt for our current problem setup in Arkansas is that SC considers situations with one treatment and multiple control units.

Recall that X 1 represents the coefficients for the treatment unit for the covariates that are significant in explaining its pre-treatment infection rates, and X 0 is a K × N matrix representing the values of the covariates for each of N control units. The main task in the standard SC is to estimate the relative weights for the control units, called the SC weights, by solving the following optimization problem [Abadie and Gardeazabal (2003) ]:

where W * is the solution to 4.1. Let Y 1 be a T 0 × 1 vector representing the infection rates for Saline County where T 0 is the length of the pre-treatment period. Let Y 0 be a T 0 × N matrix containing the infection rates for N potential control units. Then,

where V := {Diag(v 1 , ..., v K )|v i 0} and subject to the condition V = 1 to ensure identifiability of the solution. Then, the SC weights are given by W * (V * ).

Once the SC weights are computed, time series of the weighted average of the control group's infection rates and Saline County's infection rates are used to estimate the treatment effect, denoted by τ below:

where g i ∈ {1(Saline), 0(Synthetic Control)}.

The identification of meaningful covariates to explain variations in cases and deaths has been active research area since the inception of the pandemic. Wright et al. showed that low-income counties comply less with SIPO [Wright et al. (2020) ]. Griffith et al. noted that men are more likely to become infected and have higher mortality due to biological, psychological, and behavioral factors [Griffith et al. (2020) ]. Goldstein et al. found that the prevalence of the disease among 15-34-year-olds increased significantly faster than 34-49amd 10-14-year-olds, suggesting the behavioral role in spreading the disease among the former [Goldstein and Lipsitch (2020) ]. In addition, we consider other covariates in identifying X 1 . Given reports in the literature about the incubation period of the virus [Guan et al. (2020); Lauer et al. (2020) ] and the delay in policy effectiveness [Abouk and Heydari (2020) ], a linear regression would not be a preferred tool of choice for assessing covariates. In addition, given the wide range of infection rates across the counties in Arkansas, we consider a more general negative binomial GLM that allows variance of the response to vary.

For K 0 potential covariates, let X be a K 0 ×N matrix that contains the normalized values of those covariates for N control units. Then, we run the following event study Negative Binomial GLM to select K signficant covariates to base X 1 and X 0 in equation 4.1.

with the variance of Y given by var

and k is the model dispersion parameter. The potential covariates considered included normalized values of the following in each county: males; those living below the poverty income threshold; juveniles; seniors (age 65 and over); population density per square mile; those with diabetics; county's CDC Social Vulnerability Index; number of full-time equivalent practitioners needed; those eligible for Medicare; ratio of voters who voted democratic versus republican; number of hospitals; the respiratory morality rate; and the heart disease mortality rate. Percentages of residents living below poverty threshold were not available for some of the counties on the CovidSeverity.com website. Hence, for poverty rates for all counties, we used the 2015 Arkansas Department of Health Report [DoH (2015)]. [Goldstein and Lipsitch (2020) ]. Intuitively, seniors, population density, median age, and the number of hospitals in a county are significant. As noted in [Wright et al. (2020) ], poverty rate is shown to be significant. We estimated V in (4.1) using the magnitudes of the coefficient estimates in Figure 4 .1 for use in (4.1). Then, we estimated the SC weights W * (V * ) using the iterative process involving (4.1) and (4.2). 

There has been active multi-disciplinary research on Covid-19. However, to date little has been said about the causal impact of SAHO with limited scopes, such as JSAHO in Saline County, Arkansas. Using difference-in-differences and the synthetic controls design approaches, this paper presents evidence of a causal effect of county-level JSAHO implemented in a state that had not adopted a SAHO on reducing the growth rate of infection rates. While we studied the case in Saline County, the methods used here can be applied to assess the situations in other counties or local jurisdictions, and in the process strengthen the external validity of the findings by addressing the issues of limited duration and geographic specificity of the present study. There are other states that had not adopted statewide SIPO or SAHO when some of their local governments went ahead with their own orders at some point in the past, such as Utah. In addition, the analyses conducted in this paper can be applied to study the causal impact of other policy treatments.

Daily case counts for the counties in Arkansas were accessed on September 3, 2020, at The New York Times Covid-19 site available at https://github.com/nytimes/covid-19-data. The data on countylevel poverty rates were obtained from the Arkansas Department of Health report [DoH (2015) ]. With the exception of the poverty rates data, data points for all other covariates were accessed on September 3, 2020, at the Covid-19 Severity Prediction project repository available at http://covidseverity.com/. 

Semiparametric difference-in-differences estimators

The economic costs of conflict: A case study of the basque country

Synthetic control methods for comparative case studies: Estimating the effect of california's tobacco control program

Comparative politics and the synthetic control method

The immediate effect of covid-19 policies on social distancing behavior in the united states

Using the longitudinal structure of earnings to estimate the effect of training programs

How much should we trust differences-indifferences estimates? The Quarterly journal of economics

Stay-at-home order now in effect: What you need to know. Dayton Now

Saline county judge issues juvenile stay at home' executive order. Saline County

Causal estimation of stay-at-home orders on sars-cov-2 transmission

Causal impact of masks, policies, behavior on early covid-19 pandemic in the us

Strong social distancing measures in the united states reduced the covid-19 growth rate: Study evaluates the impact of social distancing measures on the growth rate of confirmed covid-19 cases across the united states

Economic Inquiry, 2020. Arkansas DoH. Arkansas department of health office of rural health and primary care primary care needs assessment. Arkansas Department of Health

Did california's shelter-in-place order work? early coronavirus-related public health effects

Association of mobile phone location data indications of travel and stay-at-home mandates with covid-19 infection rates in the us

Spline regression with automatic knot selection

Temporal rise in the proportion of younger adults and older adolescents among coronavirus disease (covid-19) cases following the introduction of physical distancing measures, germany

Men and covid-19: A biopsychosocial approach to understanding sex differences in mortality and recommendations for practice and policy interventions

Clinical characteristics of coronavirus disease 2019 in china

The effect of large-scale anti-contagion policies on the covid-19 pandemic

The incubation period of coronavirus disease 2019 (covid-19) from publicly reported confirmed cases: estimation and application

Impact of governmentimposed social distancing measures on covid-19 morbidity and mortality around the world

Covid-19 epidemic doubling time in the united states before and during stay-at-home restrictions

police hoping public adheres to holcomb's shelter-in-place order, but will be vigilant about enforcement". chicago tribune

California governor issues statewide stay-at home order

Poverty and economic dislocation reduce compliance with covid-19 shelter-in-place protocols

We would like to thank Peter Bickel and Lihua Lei for helpful suggestions.