key: cord-0602988-zu2mns82
authors: Gao, Jian; Yin, Yian; Jones, Benjamin F.; Wang, Dashun
title: Quantifying Policy Responses to a Global Emergency: Insights from the COVID-19 Pandemic
date: 2020-06-24
journal: nan
DOI: nan
sha: 681e7c518b13e6cf4d0772416eeafe0e425b2679
doc_id: 602988
cord_uid: zu2mns82

Public policy must confront emergencies that evolve in real time and in uncertain directions, yet little is known about the nature of policy response. Here we take the coronavirus pandemic as a global and extraordinarily consequential case, and study the global policy response by analyzing a novel dataset recording policy documents published by government agencies, think tanks, and intergovernmental organizations (IGOs) across 114 countries (37,725 policy documents from Jan 2nd through May 26th 2020). Our analyses reveal four primary findings. (1) Global policy attention to COVID-19 follows a remarkably similar trajectory as the total confirmed cases of COVID-19, yet with evolving policy focus from public health to broader social issues. (2) The COVID-19 policy frontier disproportionately draws on the latest, peer-reviewed, and high-impact scientific insights. Moreover, policy documents that cite science appear especially impactful within the policy domain. (3) The global policy frontier is primarily interconnected through IGOs, such as the WHO, which produce policy documents that are central to the COVID19 policy network and draw especially strongly on scientific literature. Removing IGOs' contributions fundamentally alters the global policy landscape, with the policy citation network among government agencies increasingly fragmented into many isolated clusters. (4) Countries exhibit highly heterogeneous policy attention to COVID-19. Most strikingly, a country's early policy attention to COVID-19 shows a surprising degree of predictability for the country's subsequent deaths. Overall, these results uncover fundamental patterns of policy interactions and, given the consequential nature of emergent threats and the paucity of quantitative approaches to understand them, open up novel dimensions for assessing and effectively coordinating global and local responses to COVID-19 and beyond.

Human society is often challenged by emergencies, from natural disasters [1] [2] [3] and pandemics [7, 8, 13, 14] to human conflicts [15, 16] and other emergent crises [3, 17, 18] . Despite the consequential nature of public response [6] [7] [8] [9] [10] [11] [12] , governments often struggle to manage these emergent problems, potentially due to various reasons ranging from the inherent difficulties in altering rules and allocating attention and resources [19, 20] to uncertainties about the evolving nature of the problems [19] [20] [21] .The COVID-19 pandemic provides a high-stakes case for examining the nature of policy response [5] [6] [7] [8] [9] [10] [11] [12] 20 ]. Indeed, the coronavirus pandemic is not only a public health crisis but also creates enormous challenges to social, economic, and political systems around the world [19, 20, 22] . At the same time, evolving knowledge about the nature of the disease [23, 24] , as well as evolving knowledge about containment policies and social implications [5, 7-11, 19, 20, 22] , presents a dynamic and uncertain policy environment.

To quantify the global policy response to the COVID-19 pandemic, we analyze a novel dataset of policy documents (37,725 documents published from January 2 nd to May 26 th , 2020). The dataset categorizes the source, topics, publication date of each policy document, and further includes each document's cited references. The institutions include government agencies, think tanks, and intergovernmental organizations (IGOs). The coverage is worldwide, with policy documents from 114 countries, and includes all major economies and large population centers, with a notable exception of mainland China, overall covering ~66.3% of the world population, ~79.3% of total Gross Domestic Product (GDP), and ~95.6% of confirmed deaths worldwide due to (as of May 30 th , see Fig. S1 and SI S1.1 for further description). Within this corpus, we identify COVID-19 related policy documents through keyword filtering (7, 730 documents in total, see SI S1.1), allowing us to compare COVID-19 policy documents with all other policy documents published in 2020. For each policy document, we further analyze all the references cited therein, allowing us to link policy documents to scientific papers and other policy documents they reference. These linkages can inform how different institutions interact in policy making and also illuminate the nature of the scientific knowledge upon which these policies draw.

Overall, the total number of policy documents recorded in our database grows roughly linearly with time, averaging approximately 280 new policy documents per day (Fig. 1A) . By contrast, the number of COVID-19 related policy documents features highly non-linear growth, approximated by an exponential pattern ( Fig. 1B and inset) , showing rapid shifts in global policy attention over the past few months. To understand how policy attention tracks the coronavirus outbreak, we measure the share of policy attention to COVID-19 by calculating the proportion of COVID-19 related policy documents among all policy documents published on a given day. We then compare this share with how the pandemic unfolded globally, as traced by the Johns Hopkins COVID-19 tracking map [23] . We find that the share of COVID-19 policy attention shows a remarkably similar trajectory as the total confirmed cases of COVID-19 over time (Fig. 1C , red vs blue). We observe the same pattern when using total confirmed deaths (Fig. S2) . Together, these results show that the share of policy attention to COVID-19 closely corresponds to the global unfolding of the pandemic (see SI S3.1 for more detail).

To inform how the COVID-19 policy attention may reflect policy-making institutions' priorities, we further examine the content of the COVID-19 related policy documents, breaking them down by field (Fig. 1D ) and topic (Fig. 1E ). For both analyses, we observed clear shifts in policy attention related to the pandemic, with policy priorities shifting from issues in public health to economy and society. Indeed, we first classify COVID-19 related policy documents by field, grouping them into three major field categories ("science & health", "economy & labour", and "society & others", see SI S2.1 for more detail). Plotting the share of COVID-19 policy documents across these field categories (Fig. 1D ), we find that in the early stage of the outbreak (January and February 2020), about 90% of COVID-19 policies belong to the health and science category, showing a clear, initial focus on public health issues. The policy priorities show a visible shift, however, since early March 2020, with a notable rise in policy attention to issues around the economy and society, suggesting a growing policy balance between health and socio-economic implications of the pandemic (see Fig. S3B for results at the individual field level).

We further break down these policy documents at the narrower topic level, plotting dynamics related to COVID-19 for the top 10 topics (Fig. 1E) . We find again an initial focus on healthrelated topics (in blue), followed by a clear increase over time in the share of topics related to economy and society (in red). Indeed, Fig. 1F -G respectively show the word cloud of all topics pre-and post-WHO's pandemic declaration, which illustrates a stark difference in policy priorities across the two periods, shifting sharply from public health to socioeconomic issues (see Fig. S3A for topic changes by month). Furthermore, the shifts documented in Figs. 1D-G are observed in COVID-19 policy documents only, as we repeated all our analyses for other policy documents published in the same period, finding that their shares by field or topic are relatively stable over time (Fig. S4 ).

Overall, Fig. 1 demonstrates a rapidly evolving global policy frontier that closely corresponds to the evolution of the pandemic. Yet at the same time the scientific research related to COVID-19 also evolves rapidly, as exemplified by the strong response from the global research enterprise [5, 20] . Indeed, as of May 30 th 2020, over 40,000 papers on coronavirus research have been published in 2020, with preprint servers playing an outsize role in disseminating the latest science [22, 25] .

The parallel advancement of the policy and scientific frontier in response to the pandemic raises a crucial question: is our policy understanding of COVID-19 closely linked with scientific understanding, or largely separated from the ivory tower? To answer this question, here we trace all scientific papers cited by policy documents in our corpus and match them to large-scale publication and citation databases (SI S1.2), offering a unique opportunity to quantify the role of science in the global policy response to COVID-19.

We find that COVID-19 policy documents rapidly evolve with the scientific frontier. First, Fig.   2A shows that the probability for these policy documents to cite at least one scientific paper fluctuates until WHO's pandemic declaration, and features a steady increase afterward. Fig. 2B plots the age distribution of scientific evidence referenced by COVID-19 policy documents, showing that COVID-19 policies are disproportionately centered on the latest scientific frontier.

Indeed, out of all scientific references drawn upon by COVID-19 policy documents, 19.9% of the scientific papers were published in 2020. This rate of utilizing the most recent scientific knowledge is over ten times larger than seen for other policy documents. Not surprisingly, we find that the latest scientific evidence cited by COVID-19 policies is primarily related to research on coronavirus (88.4%). However, as policy attention shifts gradually from health to economy over the course of the pandemic (Fig. 1D-E) , we also observe a similar shift in the fields of science that COVID-19 policies cite (Fig. 2C) , showing a clear shift from drawing primarily upon the biomedical literature to citing economics, society, and other fields of study (Figs. S3C).

Together, these results suggest that despite the extremely recent development in COVID-19 related scientific research, new scientific work substantially finds its way into policy documents, prompting us to next examine the quality of scientific evidence that informs policy. To proceed, we separate COVID-19 related papers into two groups based on whether or not they are referenced by policy documents. We then measure each paper's scientific impact, approximated by the number of citations the paper receives from other scientific papers. We find a dramatic impact difference between the two groups ( publication venues. We find that different venues differ widely in publication volume, with preprint servers such as medRxiv, bioRxiv, and SSRN publishing an order of magnitude more COVID-19 related papers than peer-reviewed journals. Yet, despite the large amount of papers posted on preprint servers, their impact in policy is rather limited, as these preprint servers show consistently fewer policy citations than average. By contrast, COVID-19 policies disproportionately reference peer-reviewed insights, drawing especially heavily on respected medical journals, both general (e.g., Lancet) and specialized (e.g., Clinical Infectious Diseases).

Amid growing concerns over the quality and abundancy of coronavirus research posted on preprint servers [26], these results show that peer-reviewed journals appear to remain an important institution in supplying scientific evidence for policy making.

Overall, the COVID-19 policy frontier appears grounded in extremely recent, peer-reviewed scientific insights, and science directly drawn by this policy frontier is viewed as especially impactful within the scientific community itself. Our next set of analyses present the same pattern but on the other side of the policy-science interface, showing that policy documents that are grounded in the scientific frontier tend to gather substantially more use within the global policy network. Figure 2F separates COVID-19 policy documents by whether they cite any scientific reference or not, and compares the average number of citations from other policy documents. We find that COVID-19 policy documents that cite at least one scientific paper are associated with a significantly higher number of citations from other policies, more than doubling its policy use than those without scientific references. To test if the difference observed in Fig. 2F can be explained by other covariates, we further use a regression model (see Table S1 and SI S3.3 for more detail)

to control for policy sources, the total number of scientific references, the date of policy documents, and self-citations, finding our conclusions remain the same (P < 0.001). Together, the results in Fig. 2 show that, despite the rapidly evolving nature of the pandemic, the policy and scientific frontier of COVID-19 are closely interlinked, with advances directly along the policy-science interface being notably more impactful within their own domains. (government agencies, think tanks, and IGOs), we find that overall IGOs produce the fewest number of documents ( Fig.   3A ), yet the policy documents they produce tend to be most closely connected to the scientific frontier ( Fig. 3B ) and occupy a central position within the global policy network (Fig. 3C-E) . We visualize the policy citation network among COVID-19 policy documents in Fig. 3E . Each node is a COVID-19 policy document with at least one citation linkage to other COVID-19 policy documents, and two nodes are connected if one policy cites the other. We color the nodes based on the institution that published the policy (IGOs in blue, think tanks in orange, and governments in green). A node's size is proportional to the number of citations received within this policy network. We also add a node border (in black) for those that draw on scientific papers.

The networks convey three primary insights ( Fig. 3C-E) . First, although IGOs produced the fewest number of policy documents among the three types of institutions (Fig. 3A) , blue nodes are the most visible within the network, indicating IGOs are disproportionately more likely to produce hubs within the policy network (Fig. 3F) . Second, hubs in this policy network mostly have black borders, indicating that those that attracted more policy use are also directly grounded in science, consistent with Fig. 2F . Third, while the connectivity among policy documents is relatively sparse (2,994 linkages between 2,098 nodes, <kin> = <kout> = 1.43), a high fraction of policy documents coalesce into the same giant weakly connected component (GWCC, 70.3%), illustrating the connectedness within the global policy frontier.

To quantify the relative locations of policy documents within the network, we use the k-core algorithm to calculate the coreness of each node (Fig. 3F inset, SI S3.2). A popular concept to measure the core-periphery structure of complex networks, k-core is a maximal subgraph of the original network, all of which are connected to at least k other nodes in the group [27] . Figure 3F inset compares the distribution of coreness for policy documents produced by the three institutions, showing policy documents produced by IGOs systematically occupy central positions within the network, characterized by a high k-core score. More importantly, these IGO documents appear critical for the global connectivity of this policy network. In Fig. 3G we randomly remove a fixed number of nodes from certain types and measure the remaining GWCC size [28, 29] , finding IGO documents carry disproportionate importance in connecting the network. Indeed, as Fig. 3E shows, removing IGOs' policy documents drastically alters the structure of the policy network. While IGOs only account for 31.0% of the nodes (Fig. 3C ), removing them shrinks the GWCC from 70.3% to 18.0 % (Fig. 3DG ). Contrasting IGOs with other institutions, we find that the effects of removing policy documents by think tanks or government agencies are not nearly as significant (Fig. 3D -EG). Lastly, the fraction of the GWCC decreases to merely 5.77% if we further remove all think tank policy documents, keeping only those by government agencies (Fig. 3E ), highlighting that policies by individual countries rarely connect directly across geopolitical boundaries.

Taken together, Fig. 3 shows that the overall connectedness in the COVID-19 policy frontier is mainly mediated by IGOs, the absence of which alters the global policy landscape. And among IGOs, the WHO plays a dominant role across all 55 organizations recorded in our data (Fig. 3H ).

While these results highlight the critical role of IGOs and especially the WHO in policy responses to 31] , they also suggest that the apparent connectivity in the global COVID-19 policy network hides a high degree of isolation at the individual governmental level. Indeed, if we only consider policy responses by government agencies, individual countries are fragmented into small isolated clusters, as their COVID-19 policies primarily interact within their geopolitical boundaries. This suggests that individual countries may differ in their policy response to COVID-19 [6, 32] , prompting us to examine whether country-level variations in COVID-19 policy attention may predict country-specific realities of the pandemic. Therefore, in our final analyses, we consider the COVID-19 policy response at the individual country level, and explore its correlation with the country's effectiveness in containing the pandemic (Fig. 4) . For each country in our database, we mark the calendar date when three COVID-19 deaths were first recorded within the same day [33] . We then measure the country's total deaths from COVID-19 over the following 30 days, which approximates the country's effectiveness in containing the disease. We further calculate the share of policy attention to COVID-19 in that country during the prior 30 days, i.e., in the 30 days before three daily deaths were first recorded in that country. Plotting the results in Fig. 4A , we find a strong negative correlation (n = 59, P < 0.00001), showing that countries with a greater share of policy documents addressing COVID-19 experienced substantially fewer subsequent deaths.

Several additional analyses help further assess this result. Specifically, countries with larger populations may naturally experience greater deaths. Countries with greater per-capita income may also be more susceptible (e.g., through higher influxes of international travel). Finally, countries that experienced later onset of the outbreak may have more time to prepare and learn from others. Table S2 considers various regression models to control for these additional features.

We find robust evidence that greater population predicts greater death and that later onset of the disease is associated with less ensuing death, which is consistent with global learning about how to handle the pandemic. At the same time, the explanatory power of a country's early policy attention remains robust. Figure 4B shows the ongoing predictive role of a country's prior policy attention, net of other controls (P = 0.043). Figure 4C shows the collective power of prior policy attention and the other controls in predicting subsequent deaths, demonstrating the substantial predictive capacity of this simple model.

We further test the robustness of these findings across several additional dimensions. We restrict the analysis to countries with higher volumes of policy documents recorded in our database (Tables   S3-S4) , examine different daily death thresholds to mark the onset of the disease for each country ( Fig. S7 ) and consider alternative functional forms (Tables S2-S4). We also adopt alternative data sources for the COVID-19 tracking data, as provided by the European Centre for Disease Prevention and Control (Fig. S5D) , alter the measures of a country's effectiveness in containing the pandemic by calculating total confirmed cases instead of deaths (Fig. S5A) , examine per-capita death rates for each country (Fig. S5B) , and restrict the analysis to countries with a minimum number of COVID-19 policy documents (Fig. S6 ). The findings are broadly robust to these many alternatives. Finally, we use an alternative policy attention measure -the stringency index provided by the Oxford COVID-19 Government Response Tracker (OxCGRT) [6] -which is developed independently of the Overton policy document database we studied. This alternative source shows a similar relationship (Table S5) , where greater policy stringency predicts fewer subsequent deaths, and further suggests the robustness of our result.

Overall the results suggest a country's early policy attention to COVID-19 predicts its containment of the outbreak. The patterns presented in Fig. 4 are correlations and do not identify causal mechanisms. Indeed, there are a large number of social, cultural, economic, and political factors that may influence a country's response to COVID-19. Yet, despite these many potential influences and complex factors, a country's early policy attention provides a simple yet powerful proxy that closely predicts a country's effectiveness in containing the pandemic, offering a new dimension in measuring heterogeneous policy responses and predicting differences in outcomes.

While the data used in this paper represents among the largest collection of policy documents worldwide, there are limitations of the data that should be considered when interpreting the results (SI S1.4, S3.4-S3.10). First, policy documents from mainland China, where the COVID-19 originated, are notably missing from the data. Second, many policy documents are written in the native language of a country, requiring an additional machine translation step to integrate the corpus, which was performed at the data source. It is important to keep this language effect in mind, which may generate differential data quality for non-English speaking countries. It is worth noting that potential sampling biases in our data are likely to be conservative. Indeed, we repeated our analyses in Fig. 4 for English-speaking countries only (SI S3.7), finding that the relationship further strengthens (Fig. S5C) . Lastly, the data focus on policy responses at the country level, while across regions within each country there can be important local differences in response [12, 34, 35 ], which are not captured by our data.

The data and methodologies presented in our paper are not limited to analyzing COVID-19 policy, 

The authors declare no competing interests.

Data availability Deidentified data necessary to reproduce all plots and statistical analyses will be made freely available. COVID-19 cases and deaths data are publicly available. Those who wish to access raw data should contact the data sources directly.

Code availability Code will be made freely available. 

Quantifying Policy Responses to a Global Emergency:

Jian Gao ¶ , Yian Yin ¶ , Benjamin F. Jones * , Dashun Wang * ¶ These authors contributed equally to this work.

*Correspondence to: bjones@kellogg.northwestern.edu, dashun.wang@northwestern.edu. [2] . Note that the list of keywords also relies on pre-translation to English by Overton (see S1.4 below). While the data cover a large number of countries, there are two notable exceptions in our analyses. First is the Netherlands.

One empirical limitation is that our access to the API only allows for at most 2,000 results per query, and there is one country that has exceeded this limit. Indeed, the Government of the Netherlands (Rijksoverheid) has published over 4,000 documents in 2020, making it impossible to collect the full set of its policy documents. We therefore exclude Netherlands from our analysis to avoid potential bias. Second is mainland China. Although COVID-19 originated from Wuhan, China, policy documents from mainland China are missing in the Overton database for 2020.

Personal correspondence with the Overton team has not pinpointed the reason for this missing data, but it may have to do with web crawling issues on Chinese government websites [3] .

We focus on policy documents published by (1) government agencies and think tanks in country members of the United Nations, and (2) intergovernmental organizations (defined by "IGO" and "EU" source labels in the Overton data). For the temporal coverage, we froze our data on May 26th, 2020. We noticed that there is an unusual number of documents published on January 1st 2020 (1,036 documents, about 4 times of a normal day), possibly due to default classification to this date. We further excluded these documents from our analyses to avoid including policy documents that have inaccurate publication dates. Lastly, since our main focus is on policy documents, we follow Overton's suggestion [3] and further filter on the document type by using only "publications" (95% of the total documents), removing other types such as "working papers", "transcripts", "blog posts", and "clinical guides".

In total, we analyzed 7,730 COVID-19 documents out of 37,725 documents published across 114 countries (including 402 think tanks), 55 IGOs (including 4 in EU).

We further link the scientific references cited within the policy documents to a database of We also constructed another set of COVID-19 related scientific publications by searching for papers published in 2020 with the following query suggested by Dimensions [5] :

"2019-nCoV" OR "COVID-19" OR "SARS-CoV-2" OR "HCoV-2019" OR "hcov" OR "NCOVID-

novel)), yielding in total 40,733 papers published in 2020 out of all articles indexed by Dimensions.

We use country-level daily statistics for COVID-19 confirmed cases and deaths from two independent sources. Our primary data source is the COVID-19 Data Repository provided by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University [6] , which has been extensively used in COVID-19 related studies. To ensure the robustness of our results, we also repeated our analyses using an alternative dataset provided by the European Centre for Disease Prevention and Control (ECDC) [7] , which includes not only information on COVID-19 confirmed cases and deaths, but also country-level covariates such as GDP per capita and population (see also S2.3). In our robustness checks (S3.4), we repeated our analyses using the COVID-19 confirmed deaths reported by ECDC, confirming that our results are not sensitive to the specific COVID-19 tracking data source. producing their own COVID-19 analysis [8] reveals that the keyword list contains several COVID-

Wikidata [9] ). Although this list is still not exhaustive, such efforts may help to reduce potential language biases.

Taken together, there are several potential sampling biases by countries and languages that readers should keep in mind. However, in the context of regression relationships (e.g., as in Fig. 4 of the main text), it is worth noting that noise in explanatory variable (e.g., the COVID-19 policy share) will attenuate the results toward zero, making it harder to find a significant statistical relationship.

Indeed, as measurement errors are likely to introduce noise to our data, they suggest conservative biases to the regression relationships we observed. Related, in several different robustness checks, we find that if we focus on the portion of the data that has a higher reliability, the relationships we observe tend to strengthen further. For example, S3.7 presents robustness checks by separating English and non-English speaking countries, showing that if we only focus on English speaking countries (less bias by language), the correlations we observe become stronger. Similar trends are found in S3.8, where we limit the minimum number of COVID-19 policy documents published by each country, which again increases the overall correlation.

We leverage field and topic classifications from Overton and Dimensions to determine the primary focus of each policy document and scientific paper.

Overton uses machine learning approaches to assign fields and topics to policy documents. The

Telecommunications Council (IPTC) Subject Codes taxonomy, the global standards body of the news media. In our analysis, we use 18 top-level fields in Overton and further group them into three major field categories:

Science & Health: "health", "science and technology".

Economy & Labour: "economy, business and finance", "labour", "prices".

Society & Others: "arts, culture and entertainment", "conflicts, war and peace", "crime, law and justice", "disaster, accident and emergency incident", "education", "environment", "human interest", "lifestyle and leisure", "politics", "religion and belief", "society", "sport", "weather".

The Overton policy topic also implements a more fine-grained classification system. For example, in Fig. 1e , we used this classification scheme and show the top 10 topics by volume, after excluding one generic topic ("coronavirus disease 2019").

system, which follows a three-level hierarchy (divisions, groups and fields) and covers a broad set of research fields from the sciences and engineering, social sciences, and arts and humanities. In our analysis, we use 22 top-level divisions as the research fields of scientific papers. Analogously, as we do for policy documents, we further group 22 top-level divisions into the same three major field categories:

Science & Health: "Agricultural and Veterinary Sciences", "Biological Sciences", "Chemical Sciences", "Earth Sciences", "Engineering", "Environmental Sciences", "Information and Computing Sciences", "Mathematical Sciences", "Medical and Health Sciences", "Physical Sciences", "Psychology and Cognitive Sciences", "Technology".

Economy & Labour: "Commerce, Management, Tourism and Services", "Economics".

Society & Others: "Built Environment and Design", "Education", "History and Archaeology", "Language, Communication and Culture", "Law and Legal Studies", "Philosophy and Religious

Studies", "Studies in Creative Arts and Writing", "Studies in Human Society".

We construct a citation network using all COVID-19 related policy documents, where each directed link A→B corresponds to a citation from document A to document B. Self-citations (selfloops) are removed and only nodes with non-zero degree (i.e. documents that either cite or are cited by at least one other COVID-19 policy document) are visualized in the network (Fig. 3 of the main text).

To compare the importance of documents from various institutions, we simulate a random node removal procedure and study the relative size of the giant weakly connected component when a small fraction f of the nodes is removed [10, 11] . For each institution type (IGO/think tank/government), we perform 50 independent simulations.

The relative importance of individual nodes is measured by k-core decomposition [12, 13] , where k-core is defined as the maximal subgraph with all nodes having degree no smaller than k. A node has coreness of k if it belongs to k-core but not (k+1)-core. Nodes with higher/lower coreness are located in the core/periphery of the network. We treat the network as undirected (defining degree as the sum of in-degree and out-degree) when calculating the coreness index.

We quantify the COVID-19 policy attention at the individual country level and explore its correlation with the country's effectiveness in containing the pandemic. For each country in our analysis, we mark the calendar date on which three deaths from COVID-19 in one day were first recorded [14] . We then calculate the share of COVID-19 policy documents among all policy documents in a country during the prior 30 days (i.e., in the 30 days before three daily deaths were first recorded in that country), which approximates the country's COVID-19 policy attention. We further measure the country's total deaths from COVID-19 over the following 30 days (i.e., in the 30 days after three daily deaths were first recorded in that country), which approximates the 

We employ an ordinary least squares (OLS) regression model to estimate the relations between the country-level COVID-19 policy attention and the country's effectiveness in containing the pandemic. The OLS regression takes the following form:

where ' indexes a country and & ! is the error term. Other variables are defined as follows:

Dependent variable: The dependent variable ! ! = log #" +,-.ℎ ! is the total COVID-19 confirmed deaths in the logarithmic scale in country ' in the 30 days after three daily deaths were first recorded in that country (i.e., the following 30 days), which approximates the country's effectiveness in containing the pandemic. 2) % !,# = log #" (#2345+ ! /#788 ! ) is the share of COVID-19 policy documents, now measured on the logarithmic scale. Note that the logarithmic value is undefined when 3) % !,# = log #" [(#2345+ ! + 1)/(#788 ! + 1)] is the COVID share in the logarithmic scale.

As in 1), we add one to both the number of COVID-19 policy documents and the number of all policy documents before taking the log. This provides a simple alternative to avoid dropping cases where #2345+ ! = 0 and is visualized in Fig. 4 of the main text. 

We compare the share of COVID-19 policy attention and total confirmed deaths from COVID-19 over time, finding a high degree of similarity between the two trajectories (Fig. S2A) . To test if there are systematic delays between the share of COVID-19 policy attention and the pandemic progress (both cases and deaths), we further calculate the correlation between the two time-series.

To account for the approximate exponential growth of both curves, we take the first-order difference of logarithm transformation, defined as

Correlations between the transformed series suggest that the relationship between the time series is closest when the time offset, Δt, between the share of COVID-19 policy documents and COVID-19 deaths is close to 0 (see Fig. S2B for deaths and Fig. S2C for cases) , suggesting a high degree of synchronicity between the share of COVID-19 policy attention and how the pandemic evolves.

In Fig. 1D of the main text, we group COVID-19 related policy documents into three major field categories (see also S2.1) and observe clear shifts in policy attention related to the pandemic, with policy priorities shifting from issues in public health to economy and society. At the narrower topic level, in Fig. 1e and 1f we find a clear decrease in the share of health-related topics and an increase of topics related to economy and society since early March 2020.

Here, we test the robustness of these observations by looking into data by month and for individual fields. We find similar shifts in topics of COVID-19 policy documents around March 2020 ( policy documents published in the same period. We find that the shares of policy documents by three major field categories (Fig. S4A ) and individual fields (Fig. S4B) stay relatively stable over time, suggesting that the shifts are observed in COVID-19 policy documents only. Table S1 . Regressions considering the effects of policy publication date, the country, and the type of the institution.

Notes: Column (4) and (5) shows the results on using citations from same/different sources only. Standard errors in parentheses. Significant level: *p<0.1; **p<0.05; ***p<0.01.

We explore the relationship between a country's COVID-19 policy response and its effectiveness given by three alternative functional forms, namely, the linear share of COVID-19 policy documents, #2345+/#788; the logarithmic share of COVID policy documents, log #" ( #)*+,-#233 ); and the logarithmic COVID share after adding 1 for both numbers of COVID and all policy documents: log #" ( #)*+,-0# #2330# ) (see S2.4 for more detail). Here, together with each of these functional forms for the independent variable of interest, we gradually include other control variables in the regression. Table S2 . Regressions considering the effects of COVID-19 policy attention on total deaths. Notes: The dummy variable ! "#$%& = 1 if ##$%&! = 0, and ! "#$%& = 0 if otherwise. Countries where##$%&! = 0 are dropped in column (4), and these countries are included in column (5) after resetting the value of log '( (##$%&!/#/00) to 0. Standard errors in parentheses. Significant level: *p<0.1; **p<0.05; ***p<0.01.

In columns (1-3) of Table S2 , we use the COVID share #2345+/#788 as a proxy for the share of COVID-19 policy attention. We find that this COVID-19 policy attention share by itself exhibits a significant, negative correlation with total deaths (see column (1)). Further, in column (2) we include the dummy variable + )*+,that indicates whether a country has yet published COVID-19 policy documents and find the negative correlation further improves. In column (3) we include three other control variables, namely, GDP per capita, population, and the calendar day of 3 daily deaths were first recorded for each country. We find that the COVID share still has a significant predictive power for the total deaths after controlling for all other variables.

In columns (4) (5) , we use the COVID share in the logarithmic scale log #" ( #)*+,-#233 ) as a proxy for the share of COVID policy attention. When including log #" ( #)*+,-#233 ) solely in column (4), the regression drops countries with #2345+ = 0 (i.e., where log #" ( #)*+,-#233 ) is undefined). In columns (5-6), we include all countries after resetting log #" H #)*+,-#233 I = 0 for countries where #2345+ = 0 and use the dummy variable + )*+,to account for publishing no COVID-19 policy documents. We find that the negative correlation between the COVID share and total deaths remain strong and significant.

In columns (7-9), we repeat the regression by using log #" ( #)*+,-0# #2330# ) as a proxy for the share of COVID policy attention. We find that the negative correlations remain robust, with a 10 percent increase in the adjusted COVID-19 policy share variable predicting 3.19 percent fewer deaths.

GDP per capita has no significant effect on total deaths, whereas population size has a positive effect. Moreover, the calendar day of the first 3 daily deaths has significantly negative effects, suggesting that a later onset of COVID-19 outbreak is associated with fewer deaths in the following 30 days. These variables together can explain up to 55.4% of the variation in total deaths.

We further check the robustness of the negative correlation between the COVID-19 policy attention in the prior 30 days and the total deaths in the following 30 days (i.e., the 30 days after 3 daily deaths were first recorded), by restricting the analysis to a minimum threshold number of policy documents that a country published in the prior 30 days (i.e., the 30 days before 3 daily deaths were first recorded), Table S3 summarizes the results of regressions where we only include countries that published at least 10 policy documents in the prior 30 days. We find that the negative correlations remain largely significant. In particular, we notice from column (9) that the regression coefficient of the COVID share increases to -0.375 (from -0.319 in Table S2 ), suggesting that that COVID share exhibits a slightly stronger predictive power for total deaths in countries that published more policy documents. Table S3 . Regressions considering the effects of COVID-19 policy attention on total deaths for countries that published at least 10 policy documents in the prior 30 days.

Notes: The dummy variable ! "#$%& = 1 if ##$%&! = 0, and ! "#$%& = 0 if otherwise. Countries where##$%&! = 0 are dropped in column (4) , and these countries are included in column (5) after resetting the value of log '( (##$%&!/#/00) to 0. Standard errors in parentheses. Significant level: *p<0.1; **p<0.05; ***p<0.01.

In Table S4 , we further limit our regression analysis to countries that published at least 25 policy documents in the prior 30 days. We notice that the regression coefficient of COVID share in column (9) further increases to -0.467, showing that the share of COVID-19 policy attention in the prior 30 days in countries that published more policy documents has again a steeper relationship with the total deaths in the following 30 days. With the smaller number of country observations that result from these policy document count restrictions, the standard errors tend naturally to increase in these regressions, yet most specifications remain statistically significant. Overall, these analyses tend to further support the robustness of our findings. Table S4 . Regressions considering the effects of COVID policy attention on total deaths for countries that published at least 25 policy documents in the prior 30 days.

Notes: The dummy variable ! "#$%& = 1 if ##$%&! = 0, and ! "#$%& = 0 if otherwise. Countries where##$%&! = 0 are dropped in column (4) , and these countries are included in column (5) after resetting the value of log '( (##$%&!/#/00) to 0. Standard errors in parentheses. Significant level: *p<0.1; **p<0.05; ***p<0.01.

In Fig. 4A of the main text, we presented the negative correlation between the share of COVID-19 policy attention in the prior 30 days and the total confirmed deaths in the following 30 days, where we use the COVID-19 death tracking data provided by Johns Hopkins University (JHU).

Here, we consider alternative measures and show that our results are robust when using total confirmed cases ( Fig. S5A ; columns (1-2) of Table S5 ) and confirmed deaths per million population of a country ( Fig. S5B ; columns (3) (4) of Table S5 ) in the following 30 days.

We further leverage a new dataset for daily COVID-19 case and death statistics, provided by the European Centre for Disease Prevention and Control (ECDC). Country-level death statistics collected by JHU and ECDC are highly correlated with each other. For example, the Pearson correlation coefficient between the total deaths (in the logarithmic scale) in the following 30 days recorded by the two data sources is 0.996. Here, we show that our results remain robust when using ECDC death statistics ( Fig. S5D ; columns (5-6) of Table S5 ). Table S5 . Regressions considering the effects of COVID policy attention on the effectiveness in containing the pandemic.

Notes: The dummy variable ! "#$%& = 1 if ##$%&! = 0, and ! "#$%& = 0 if otherwise. Standard errors in parentheses. Significant level: *p<0.1; **p<0.05; ***p<0.01.

To address possible concerns on the potential bias by language in Overton policy documents, we further performed robustness checks by separating English and non-English speaking countries in our analysis. Based on whether English is an official language [17] , we consider a combination of three sets of countries as English-speaking countries in our analysis: (1) Countries where English is a de facto (i.e., practices that exist in reality) official language; (2) Countries where English is a de jure (i.e., practices that are legally recognized) and de facto official language; (3) Countries where English is a de facto official, but not primary language.

Out of all 59 countries in our analysis for 

We further focus on the portion of the data that has a higher reliability in our analysis by limiting countries that have published a minimum threshold number of COVID-19 policy documents in the prior 30 days. We find that the negative correlation between the share of COVID-19 policy attention and total deaths that we observe tends to strengthen. Specifically, when increasing the minimum number of COVID-19 policy documents from 1 to 10, the Pearson correlation coefficient increases from -0.63 to -0.78 (Fig. S6) , and the correlation remains largely significant when controlling for other variables (Table S6) . Table S6 . Regressions considering the effects of COVID policy attention on total deaths for countries that published a minimum number of COVID policy documents in the prior 30 days.

Notes: Standard errors in parentheses. Significant level: *p<0.1; **p<0.05; ***p<0.01.

As a reference point approximating the onset of the outbreak, the paper takes the day where 3 daily deaths were first recorded. Here we present robustness checks on this reference point ( Fig. S7 ; Table S7 ). Specifically, when taking the day where the first death was recorded in each country as the reference point [15, 16] , a total of 82 countries are included in the analysis, and we find a significantly negative correlation (-0.550) between COVID share and total deaths ( Fig. S7A; columns (1-2) of Table S7 ). Taking the reference point as the day when 5 daily deaths were first recorded ( Fig. S7B; columns (3-4) of Table S7 ) and the day where 10 daily deaths were first recorded ( Fig. S7C ; columns (5-6) of Table S7 ), the negative correlation between COVID share and total deaths remains strong and significant, again confirming the robustness of our results. Besides the number of daily deaths, we use an alternative method by taking the day where a cumulative number of 100 deaths was first recorded as the reference point. After checking the relations between the share of COVID-19 policy documents in the 30 days before total 100 deaths was first recorded in a country and the total deaths after total 100 deaths was first recorded in that country, we find that the negative correlation further improves and remains significant ( Fig. S7D ; columns (7-8) of Table S7 ). Overall, these results support the robustness of our results.

Finally, we leverage an independent measure on COVID-19 policy, the stringency index provided by the Oxford COVID-19 Government Response Tracker (OxCGRT) [15, 16] , to further test the robustness of our finding on the relationship between COVID-19 policy attention and the effectiveness in containing the pandemic. For each country we calculate an average stringency index during the prior 30 days and employ it to predict the total deaths in the following 30 days. Table S8 summaries the results of regressions that include both the share of COVID-19 policy documents and the average stringency index together with other control variables. In order to compare the values of regression coefficients, here we take the z-score for all variables. (1) and (4), and these countries are included in other columns after resetting the value of log '( (##$%&!/#/00) to 0. Standard errors in parentheses. Significant level: *p<0.1; **p<0.05; ***p<0.01.

In columns (1-3) of Table S8 , we include alternative functional forms for the share of COVID-19 policy documents and the average stringency index, finding that the COVID share and the stringency index have independent and comparable predictive power for total deaths. In columns (4) (5) , we include both the policy document share and the stringency index and find that they augment each other's predictive power and have similar coefficient magnitudes. In columns (6-10), we include the dummy variable whose value + )*+,-= 1 if a country did not publish any COVID-19 policy documents in the prior 30 days and + )*+,-= 0 if otherwise. We find that the predictive powers of COVID share and stringency index are again similar to each other and augmentative.

In columns (11) (12) (13) (14) (15) , we further include controls for GDP per capita, population, and the calendar day where 3 daily deaths were first recorded in each country. After controlling for all variables, we find that the effects of the share of COVID-19 policy documents remain negative and significant in columns (11) (12) . Moreover, the COVID share exhibits a slightly stronger power than the stringency index in predicting total deaths in columns (14) (15) . These observations are encouraging, as the stringency index measures policies that are specifically designed toward containing the pandemic, which exhibits a comparable level of predictive power as our measure.

Overall these results further confirm the robustness of our results, showing that at the country level, greater COVID-19 policy attention predicts fewer subsequent deaths. 

Predictability of population displacement after the 2010 Haiti earthquake

Rapid assessment of disaster damage using social media activity

Collective response of human populations to large-scale emergencies

The parable of Google Flu: Traps in big data analysis

Policy opportunities to enhance sharing for pandemic research

Variation in government responses to COVID-19, in Blavatnik School of Government Working Paper

The effect of human mobility and control measures on the COVID-19 epidemic in China

The effect of travel restrictions on the spread of the 2019 novel coronavirus (COVID-19) outbreak. Science

How will country-based mitigation measures influence the course of the COVID-19 epidemic?

Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe

The effect of large-scale anti-contagion policies on the COVID-19 pandemic

Interdependence and the cost of uncoordinated responses to COVID-19

Population flow drives spatio-temporal distribution of COVID-19 in China

Modeling the worldwide spread of pandemic influenza: Baseline case and containment interventions

Pattern in escalations in insurgent and terrorist activity

Quantifying the dynamics of failure across science, startups and security

Bursts: The Hidden Patterns Behind Everything We Do, from Your E-mail to Bloody Crusades

The Black Swan: The Impact of the Highly Improbable

Advice and answers from the Overton Team

Who are COVID-19 guidelines & policy documents citing?

Dimensions: Bringing down barriers between scientometricians and data. Quantitative Science Studies

Dimensions COVID-19 publications, datasets and clinical trials

An interactive web-based dashboard to track COVID-19 in real time

European Centre for Disease Prevention and Control. COVID-19 situation update worldwide

Who are COVID-19 guidelines & policy documents citing? 2020. 9. Wikidata contributors. COVID-19

Error and attack tolerance of complex networks

Percolation in directed scale-free networks

A model of Internet topology using k-shell decomposition

Identification of influential spreaders in complex networks

Coronavirus tracked: Has your country's epidemic peaked

Variation in government responses to COVID-19, in Blavatnik School of Government Working Paper

Whose coronavirus strategy worked best? Scientists hunt most effective policies

List of territorial entities where English is an official language