key: cord-0685581-tz2ava1s
authors: Klumpp, Matthias; Loske, Dominic; Bicciato, Silvio
title: COVID-19 health policy evaluation: integrating health and economic perspectives with a data envelopment analysis approach
date: 2022-01-11
journal: Eur J Health Econ
DOI: 10.1007/s10198-021-01425-7
sha: 905cbbf6acb671515c526f647cfda1e031ae0787
doc_id: 685581
cord_uid: tz2ava1s

The COVID-19 pandemic is a global challenge to humankind. To improve the knowledge regarding relevant, efficient and effective COVID-19 measures in health policy, this paper applies a multi-criteria evaluation approach with population, health care, and economic datasets from 19 countries within the OECD. The comparative investigation was based on a Data Envelopment Analysis approach as an efficiency measurement method. Results indicate that on the one hand, factors like population size, population density, and country development stage, did not play a major role in successful pandemic management. On the other hand, pre-pandemic healthcare system policies were decisive. Healthcare systems with a primary care orientation and a high proportion of primary care doctors compared to specialists were found to be more efficient than systems with a medium level of resources that were partly financed through public funding and characterized by a high level of access regulation. Roughly two weeks after the introduction of ad hoc measures, e.g., lockdowns and quarantine policies, we did not observe a direct impact on country-level healthcare efficiency, while delayed lockdowns led to significantly lower efficiency levels during the first COVID-19 wave in 2020. From an economic perspective, strategies without general lockdowns were identified as a more efficient strategy than the full lockdown strategy. Additionally, governmental support of short-term work is promising. Improving the efficiency of COVID-19 countermeasures is crucial in saving as many lives as possible with limited resources.

The ongoing global COVID-19 pandemic is a challenge to humankind with a high death toll -more than 2.2 million persons lost their lives, with a large number of countries worldwide affected, and in excess of 102 million people had contracted the viral disease as of January 2021 (Johns Hopkins University 2021) [1, 2] . COVID-19 is an exogenous health threat that poses a particular challenge for health systems worldwide [9] because (1) vaccines do not yet apply any relevant influence, (2) forecasting the number and severity of infections is difficult, and (3) empirical evidence regarding the suitability of pharmaceutical (COVID-19 testing) or non-pharmaceutical interventions (travel restrictions, social distancing measures, partial or complete lockdowns) is missing [119] . As the pandemic is still ongoing, this poses a specific and comprehensive challenge to research and science to address this problem with all available tools, methods, and insights. The efficient allocations of health care resources, e.g., testing policies [27] , hospital admissions [73] , and intensive care capacity [132] , are major health policy challenges [43, 55] . Therefore, an evaluation of the actions taken to date to address the pandemic is highly relevant for health economics [23, 72] . Of particular interest in the context of pandemics is the tension between medical efficacy and economic efficiency [69] , which is increased by the quality of health care and the equitable use of health goods. When evaluating medical or health policy interventions during or after pandemic outbreaks, health economics analyses adopt different perspectives: (1) the perspective of health service providers (doctors, hospitals) studying, e.g., direct costs necessary to treat patients [22, 84] , stockpiling of drugs [7, 63] , or withholding effective novel antidotes [90] , (2 patient-centered investigations on, e.g., consumer learning in vaccination decisions [88] , (3) examinations through the lens of vaccine producers, e.g., analyzing the profit-maximizing capacity [53] , and (4) studies evaluating the economic effect of pandemics, e.g., on companies through the lens of employee absences from work [40] , effects on tourism and certain production sectors [108] , and overall effects on a country's economy [74, 77] . The objective of our study was to provide a countryspecific efficiency evaluation of the fight against COVID-19 for 19 OECD countries while focusing on the role of prepandemic health care policy and its interconnection to ad hoc interventions (case 1 and case 2), as well as the impact of COVID-19 as an exogenous health threat to the country's economy (case 3) [8] . We aspired to answer the following research questions: (1) "How efficiently did OECD countries handle the COVID-19 outbreak?" and (2) "What are the reasons for efficient or less efficient COVID-19 handling?" We employed a multi-factor evaluation approach based on data envelopment analysis (DEA) as a neutral efficiency measurement method to enable fact-based discussions and decision processes. DEA is an established and widely used method for efficiency measurement in healthcare management [71, 79, 81, 118, 135] . The evaluation approach integrates different levels of governmental decision-making and from a cross-country, also chosen by, e.g., Vogler and Fischer [128] ,: (1) factors considering pre-pandemic government health strategies measurable through governmental expenditures, (2) pre-pandemic characteristics of the respective health system representing health resources, (3) indicators expressing the consequences of governmental interventions during the COVID-19 outbreak on the economy, and (4) governmental interventions against COVID-19 taking into account the country-specific state of the pandemic. While our input factors are chosen to quantify the direct costs that are necessary for the treatment of a patient (health expenditures, number of doctors and hospital beds), the output factors are not quantified in monetary terms but as therapeutic outcomes and presented as clinical or physical quantities (infections, deaths, recoveries) because we wanted to consider patient-relevant measures.

The contribution of this paper is (1) the specific quantitative efficiency measurement approach with the DEA technique applied to a strategic-level evaluation of COVID-19 responses in the 19 examined OECD countries. In addition, this enables (2) a new perspective on COVID-19 countermeasures from an integrated health care and economics perspective based on empirical real-life data. Furthermore, (3) evaluations of individual measures directed toward the objective of an overall resource-efficient answer to viral pandemics that can be analyzed as a general objective measure are introduced. Therefore, we add new insights on the existing lessons learned from the management of the COVID-19 pandemic presented by Forman et al. [52] .

This paper is structured as follows: the literature review section highlights the intersection of efficiency measurement and health economic activities during epidemics and pandemics from an interdisciplinary perspective (Sect. Literature view). Then, the methodology section describes the data used in this paper and elaborates on the use of DEA models to measure the efficiency of health care policies during the COVID-19 outbreak (Sect. 3). Section 4 presents the results of three longitudinal efficiency analyses (1) investigating the health system efficiency during COVID-19, (2) decompensating the health system and governmental ad hoc intervention efficiency, and (3) examining the impact of interventions on the country's economies. The results are discussed in Sect. 5. Finally, conclusions and the outlook toward future research are presented in Sect. 6.

A brief literature review was applied to structure a conceptual basis for the specific analysis implemented in this paper regarding COVID-19 overall efficiency evaluation. This is important as though it seems from the 2020 perspective that pandemic challenges are new on a global scale, this is not the case when looking back in a larger timeframe. In order to identify the key words and topics from previous research on this issue for the last 150 years, an overview is implemented as follows: For keywords, the following entries were used in searches of international academic journals: (1) "pandemic", in combination with (2a) "public economics", (2b) "government policy", and (2c) "efficiency". In a second search round, "pandemic" was replaced by "COVID-19". Papers were selected to achieve a topical representation.

The structured results are presented in Table 1 . Pandemic situations have been regarded as rare but special public economic and health challenges since the start of scientific discussions thereof in the nineteenth century (No 1-3 in Table 1 ). Additionally, characteristically, efficacy evaluation to date has occurred from many different medical science perspectives, including health care, public economics, and public health science as well as social sciences; disciplinespecific journals are indicated in . This pandemic is nevertheless seldom seen as a public economic challenge, as only very large pandemic situations actually strain public resources in the sense of crucial political and 6 . Further interesting comparative analyses with earlier results can and will be addressed in future research with further insights. For the subsequent parts of this paper it is important to recognize that pandemic crises have been subject to research from many disciplines already -but without the methodology to integrate that into a coherent quantified evaluation scheme as it is proposed in this paper [18, 19] .

The data used for the efficiency analysis were retrieved from several databases. First, general country-specific data, e.g., health spending, number of doctors and hospital beds, and unemployment rates, were obtained through database queries from the OECD database OECD Stat 

To calculate the efficiency of each country, this paper proposed a DEA model with an output-oriented ratio form under constant returns to scale (CRS). In general, DEA is a non-parametric optimization method of mathematical programming for measuring the relative efficiency of decisionmaking units (DMUs) that have multiple inputs and outputs. A basic model was introduced by Charnes, Cooper, and Rhodes [29] based on the Koopmans activity analysis concept [82] together with the publications of Debreu and Farrell dealing with radial efficiency measurement [36, 49] . The optimization method can be based on CRS in the CCR model [29] or variable returns on scale (VRS) in the BCC model [11] , and each case has an input or output orientation. The linear program for the CCR model is [32] , pp. 23-24):

The basic idea is to calculate an efficiency frontier that is used as a best practice input-output combination for the underlying production scenario. A score of 1.0 indicates that a DMU is efficient and on the efficiency frontier, whereas the relative inefficiency of a DMU can be determined by measuring the distance between the individual DMU performance and the efficiency frontier. Measuring efficiency under the assumption of CRS is known as overall technical efficiency (OTE). This includes the determination of inefficiency/efficiency based on (1) the input/output transformation, meaning pure technical efficiency (PTE), as well as (2) the size of operations, meaning the scale efficiency (S.E.). DEA is a frequently applied methodology for public health question on several aggregation levels, e.g., regarding (1) hospitals [3, 4] ,D. C. [50, 51, 87] , (2) the evaluation of healthcare reforms [44, 105, 106] , (3) healthcare infrastructures [26] , and (4) health care systems [94] . It has also been applied to evaluate the health care production of OECD countries [125, 126] .

The traditional DEA methodology was extended in several directions: To evaluate the efficiency of panel data and enable time series efficiency measurement per DMU, Charnes et al. [28] proposed DEA window analysis. The dynamic perspective of this model treats the same DMU occurring in different periods as entirely different DMUs. The major benefit of its moving average method is that the number of DMUs increase, and dynamic changes of the efficiency per DMU can be evaluated. Furthermore, to disclose the black-box assumption of the traditional DEA model, Färe and Primont [48] and Färe [46] proposed Network DEA where a production process is decompensated in sub-technologies or nodes. The specification of these nodes enables the examination of input/output allocations as well as intermediate products, which represent the entire production process [47, 120] .

The selection of applicable inputs and outputs, as well as the design of a suitable DEA model, is a well-known source of pitfalls within the DEA literature [41] . Researchers are often facing a choice between the (1) empirical quantitative data that are published, e.g., by governments or organizations, and (2) the restrictions of the DEA model concerning input/output selection. Therefore, we followed a four-step framework for DEA application proposed by Jain et al. [70] : (1) select applicable inputs and outputs for the subject of research, (2) validate the inputs and outputs, (3) select the applicable DEA model, and (4) execute the DEA model in step 4.

First, possible input and output factors must be directly related to government policy and health policy, as well as the COVID-19 outbreak, and can be separated into four groups: (1) factors considering pre-pandemic health strategies measurable through governmental expenditures or revenues, e.g., health or pharmaceutical spending; (2) prepandemic characteristics of the respective health systems representing resources that are available during an epidemic outbreak and measurable through the existing health infrastructure, e.g., number of doctors and nurses, number of medical and nursing graduates, number of hospital beds and intensive care beds; (3) indicators expressing the quantifiable economic consequences of governmental interventions during the COVID-19 outbreak, e.g., unemployment rates or trade volumes; and (4) governmental interventions against COVID-19 taking into account the country-specific state of the epidemic outbreak, e.g., number of COVID-19 tests performed, number of COVID-19 infections, number of COVID-19 deaths, or number of patients recovered from COVID-19.

Second, to validate the inputs and outputs through empirical data, we follow the steps proposed by Dyson et al. [41] . First, mixing indices and volume measures, as well as integrating percentages, can lead to distortions of the efficiency values. Since most data are available in the form of ratios, volume measures, e.g., total population, are excluded from the analysis or converted to ratios. Consequently, the DEA model is calculated by applying indices per inhabitant or per 1,000,000 inhabitants. Second, linked input/output values have to be avoided, e.g., considering the total number of tests performed and the number of tests per inhabitant leads to distortions of the efficiency scores. Third, a cross-correlation of the available factors has to be avoided. For instance, health spending per capita and pharmaceutical spending per capita (r = 0.68), as well as the number of nurses and doctors (r = 0.76), are highly correlated. To finish step two, we define the following input (I) and output (O) factors: I 1 health expenditures in US dollars per inhabitant, I 2 number of doctors per 1,000,000 inhabitants, I 3 number of hospital beds per 1,000,000 inhabitants, I 4 number of COVID-19 tests per 1,000,000 inhabitants, O 1 number of COVID-19 deaths per 1,000,000 inhabitants, O 2 number of patients recovered from COVID-19 per 1,000,000 inhabitants, O 3 number of COVID-19 cases per 1,000,000 inhabitants, and O 4 unemployed individuals per 1,000,000 inhabitants assigned to the labor force. Especially for I 1 to I 3 , we are aware of the fact that the possible regional organization of health care systems is a key issue for the efficient handling of pandemics. E.g., hospitals traditionally organized to deliver patient-centric care are ill-equipped to deliver the type of community-focused care needed during a pandemic (For research regarding the further COVID-19-related development of public hospitals, the reader is referred to Rodríguez et al. [110] . But as most health care systems are highly decentralized, it is difficult to quantify that different regions tried various policy responses. As spatial doctor density or any index accounting for the local public health infrastructure and surveillance (e.g., the number of community doctors as compared to that of hospital specialists) is not available in cross-country databases, our approach is based on the currently available data aggregation level of countries. Appendix Table 4 summarizes the key attributes of the dataset by applying descriptive statistics, and a further correlation analysis proves that there is no linear statistical relationship between the applied input and output measures. Hereafter, we explain the variables in detail: I 1 Health expenditures in US dollars per inhabitant measures the final consumption of health care goods and services that are financed through a mix of financing arrangements, including government spending and compulsory health insurance, as well as voluntary health insurance and private funds, such as households' out-of-pocket payments, NGOs and private corporations [98] . This factor quantifies the prepandemic government health care policies determining the majority of the health care resources that are available during the COVID-19 pandemic (fixed). I 2 Number of doctors per 1,000,000 inhabitants, which defines doctors as practicing doctors providing direct care to patients. Doctors are usually generalists who assume responsibility for the provision of continuing care to individuals and families or specialists, such as pediatricians, obstetricians, gynecologists, psychiatrists, medical specialists, and surgical specialists [97] . As doctors are an essential resource in fighting pandemics, the input factor expresses a major pre-pandemic variable that reflects the resources of a health system (fixed . It is a factor that quantifies the nature and extent of ad hoc government policies against COVID-19 and enables its integration in an efficiency analysis. An even better fit to quantify this aspect could be the number of tests performed per day compared to the total number of tests that can be processed per day. However, as this capacity is mostly depending on highly decentralized lab capacity, it was not quantifiable in the course of this research [21, 25] .

O 1 Number of COVID-19 deaths per 1,000,000 inhabitants expresses the mortality of the pandemic and is, therefore, one of the most important measures to quantify the burden of COVID-19. Countries throughout the world have reported very different case-fatality ratios, e.g., the number of deaths divided by the number of confirmed cases [39] . O 1 solely includes the reported deaths associated with COVID-19. As DEA would value a large number of deaths as a large output and, therefore, as highly efficient, O 1 is integrated as an undesirable output. O 2 Number of patients recovered from COVID-19 per 1,000,000 inhabitants quantifies the number of individuals who successfully recovered from their COVID-19 infection and were discharged from hospitals and self-isolation facilities. As the exact date of recovery is unknown in most cases, the Robert Koch Institute and Johns Hopkins University use algorithms to estimate the number of recovered cases [39] .

O 3 Number of COVID-19 cases per 1,000,000 inhabitants quantifies the number of individuals who are infected within an examined time period. Reducing the number of new COVID-19 cases will help countries prevent overloading of their health care systems. Because the main transmission pathway for COVID-19 is the respiratory absorption of virus-containing fluid particles that are produced during breathing, coughing, speaking, and sneezing, governmental restrictions include partial or total lockdown, traveling limitations, and social distancing restrictions. O 3 can be used to integrate an evaluation regarding the suitability of timing, nature, and extent for ad hoc governmental measures [39] . As DEA would value a large number of cases as a large output and, therefore, as highly efficient, O 3 is integrated as an undesirable output.

O 4 Unemployed individuals per 1,000,000 inhabitants assigned to the labor force are integrated into a separate model to measure the impact of governmental restrictions on the economy, as well as the success of governmental interventions through, e.g., tax reductions, monetary subsidies for short-term work, or loan programs. Unemployed inhabitants are people of working age who are without work, are available for work and have taken specific steps to find work. This indicator is measured in numbers of unemployed people per 1,000,000 inhabitants of the labor force. The labor force is defined as the total number of unemployed people plus those in employment [101]. Thus, inhabitants outside of the working age are excluded. O 4 is integrated as an undesirable output [31, 33, 35] .

In step three of the DEA application framework, before the execution of the computations in step four, the number of defined inputs and outputs can also provide another pitfall, as discussed by Dyson et al. [41] , who proposes that the number of observation points must be at least 2 times the number of inputs times the number of outputs. With four inputs, four outputs, and 228 observation points (19 countries times 12 periods, each lasting one week), our model fulfills this criterion. Furthermore, DEA models can aspire to (1) maximize desirable outputs, (2) minimize undesirable outputs, (3) maximize desirable inputs, or (4) minimize normal inputs [64] . In the course of this paper, we use an output-oriented model. As the output-oriented model seeks to maximize outputs, O 1 , O 3 , and O 4 are treated as undesirable outputs that have to be minimized [61, 114, 115] .

To support the choice of available scale assumption, semi-parametric statistical tests, known as Banker's tests or tests of goodness of fit, were applied; H 0 was the assumption of CRS against the alternative of H 1 with VRS [10, 12] . For execution, we follow the example of Giokas et al. [58] , assuming that h C and h B are the DEA inefficiency scores estimated from the BBC and CCR models. In the first hypothesis test (formula 1), h is assumed to follow the exponential distribution in the two models and evaluated relative to the critical value of the F-distribution with (2 N, 2 N) degrees of freedom, while in the second (formula 2), h is assumed to be half-normally distributed for both models and evaluated relative to the critical value of the F-distribution with (N, N) degrees of freedom [58] , p. 1941).

The results of T EX = 2.03 and T HN = 4.11 indicate that the null hypothesis of constant returns to scale cannot be rejected at the 5% level of significance for an exponential distribution and normally distributed h score. Therefore, we applied the CCR model with CRS.

Being able to evaluate and compare the efficiency of healthcare systems during the COVID-19 pandemic requires considering two circumstances that need to be addressed before applying the evaluation framework elaborated in the previous chapter: (1) the comparability of the health systems of OECD countries, and (2) time-delayed COVID-19 outbreaks and their operationalization. Several dimensions of a healthcare system, e.g., the level and financing of resources, the role of key actors including the state and other societal or private actors, and access regulation, should be noted when comparing various countries [45] . To systematize country-specific characteristics of health systems, research approaches have developed typologies of health systems that aim to better characterize and categorize health systems for European (P. L. [50, 51] and OECD countries [20, 109, 121, 131] . Böhm et al. [20] applied the typology developed by Rothgang et al. [111] that distinguishes the three dimensions of (1) regulation, (2) financing, and (3) service provision, as well as the three types of actors, namely (1) the state, (2) societal actors, and (3) private actors. Wendt [131] proposed four types of healthcare systems that mainly focus on health expenditure with public financing and outof-pocket payments by patients, as well as on the payment of general practitioners and access regulation. Reibling et al. [109] develop five typologies integrating the dimensions: (1) the level of resources of healthcare systems operationalized through expenditure on healthcare per capita and the number of general practitioners per thousand inhabitants, (2) the public-private mix operationalized through the share of public health expenditures of the total health expenditure, the share of out-of-pocket payments of the total health expenditure, and the payment of specialists; (3) access regulation, including whether individuals are required to register at a

general practitioner and how a person can access specialist care; (4) primary care orientation operationalized through the ratio of general practitioners to specialists and the share of health expenditure on outpatient care of the total health expenditure, and (5) the performance measured by the level of achievement for goals in the prevention and the quality of care. In our analysis, we included all countries that were and categorized then into the following clusters: (1) (4), the low-supply and low-performance mixed type, was excluded because the available data were suitable, not suitable, or sufficient for the purpose of our analysis. Therefore, we only compare public health systems and exclude the supply-and performance-oriented private types [42] . Time-delayed outbreaks with a varying number of infections lead to country-specific scenarios in the context of COVID-19. While (1) (4) AUS and JPN started the second infection wave in July 2020. Being able to deal with these time lags requires the normalization of all countries to a starting time t 0 [92] . Thus, we chose the date of the first COVID-19-related death reported by Johns Hopkins University as the beginning of the country-specific COVID-19 outbreak. With this normalization approach, we were able to compare the healthcare system efficiency for the selected OECD countries with the individual progression of COVID-19-related deaths and infections.

To evaluate the efficiency of health system policies before the epidemic outbreak, we separated the dataset into periods lasting seven days each and conducted a DEA window analysis. The first period started when the first COVID-19-related death appeared. We chose w = 2 as the window width for this analysis, as the maximum incubation time of the COVID-19 virus is 14 days, which is, on the other hand, the minimum period until the effects of governmental practices are measurable. The first model used I 1 , I 2 , and I 3 , as well as O 1 and O 2 . Table 2 summarizes the results.

(1) AUT: Two announcements for partial lockdowns were implemented on March 16, 2020, and March 19, 2020, which was at the end of period 1 in our analysis (Austrian Federal Ministry of Social Affairs, Health, Care and Consumer Protection, 2020). Two weeks later, the efficiency level of Austria increased significantly. After releasing the lockdown in period 6, several local COVID-19 outbreaks led to COVID-19-related death peaks and consequently to a low level of efficiency in subsequent periods. (2) BEL: An equal logic can be inferred in the case of Belgium, where a general lockdown starting on March 18, 2020 [15] limited the leaving of homes, except for emergencies, in period two of our analysis. At this time, the efficiency level was 0.53. Two weeks later, the efficiency level of Belgium increased significantly to 0.80 [59] . only allowed movements for travels to make essential purchases (food, hygiene, health, first necessity), in strict compliance with basic precautionary measures, travels to access banking services, or travels to provide care and assistance to vulnerable people [117] . After introducing these measures in period 2, with an average efficiency of 0.52, positive development started in period 4 (eff. = 0.77). The decline in efficiency in periods 10 to 13 can be traced back to a high death rate in relation to other countries in these late periods. Therefore, the speed of flattening the curve had a significant impact on countries' efficiency in fighting the pandemic. (6) FRA: One of the countries with the lowest efficiency level among all periods implemented the lockdown on March 17, 2020, and from a calendrical perspective, FRA had efficiency levels quite similar to those in other European countries [54] . However, France was already in period 5, meaning one month and two days after the first reported COVID-19-related death. It is furthermore interesting to observe that the late lockdown did not have the same effect as the early (period 1 or 2) lockdowns of other countries, as the efficiency level of France constantly remained low in subsequent periods (period 7 and further). (7) UK: The government implemented a lockdown on March 24, 2020 (end of period 3), which was late compared to more efficient European countries, e.g., Germany or Spain. Additionally, the health system has significantly fewer doctors per million inhabitants and hospital beds per million inhabitants compared to Italy and France, which have similar populations. Finally, the resulting constantly high death ratio, with 500 to 1,100 deaths per day during April, as well as high death ratios in the late periods (compared to other countries), led to an overall weak health system performance [60, 66] . (8) IRE: The government enforced a partial lockdown on March 23, 2020 (period 2); people could only leave their homes to travel to or from work if they were providing an essential service, to shop for food, to collect medical prescriptions and medical supplies and attend medical appointments, or to carry out vital services, such as caring for a family member, within 2 kms of their houses, while keeping 2 m away from others for social distancing. Similar to other states, the results were observable after two periods in period 4 [67] . Ireland's health system and additional measures made it possible to maintain a high efficiency level for the rest of the pandemic. On March 9, this lockdown was extended to the whole country. All public spots were closed; travel was limited to the strictly necessary (health, food, and work when working remotely is impossible). All gatherings were prohibited and punishable by fines. On March 11, adoption of a new decree from the Council of Ministers imposed the closure of all businesses except for pharmacies and grocery stores. March 11 was therefore basically the start of the nation-wide lockdown. On March 22 and 23, this was de jure adopted by an ordinance by the Ministries of Health and of the Interior prohibiting citizens from leaving their city of lockdown, but this simply did not add anything at all to March 11 decree [68] . About three weeks later, the efficiency level in Italy increased significantly. However, the low average efficiency of Italy can be connected to the fact that the lockdown was implemented late when comparing with the timing in other states. The same observation was made for Great Britain [75, 76, 78] . For a deeper discussion on the measures taken by the Italian government, the reader is referred to Mauro and Giancotti [89] and Valent et al. [124] . (10) KOR: Until July 2020, South Korea was the only country with a population of over 50 million inhabitants that had slowed the spread of the virus and flattened the curve of new infections without shutting down the whole country. Furthermore, there were no extreme personal travel or movement restrictions and no closure of airports. After the first COVID-19 case on January 19, 2020, and the first COVID-19-related death on February 20, 2020, South Korea raised the alert level to red on February 23, 2020 (period 1) (The Government of the Republic of Korea, 2020). The country managed to avoid high case numbers through intelligent and digital COVID-19 management, as well as persistent tracking, tracing, and testing of infected persons who were quickly identified and treated at an early stage. The country is thus a unique example of efficient COVID-19 control without lockdown. However, in many European and North American countries, involving police in matters of public health combined together with data sharing is seen as an act of criminalizing illnesses and, therefore, highly critical [93] . (11) SVN: The country with the least inhabitants in the whole dataset also had the lowest input values for the average health spending per capita. On the other hand, the number of doctors per million inhabitants and the number of hospital beds per million inhabitants are higher in better-developed countries, e.g., Belgium, Great Britain, the Netherlands, or Norway. Due to the number of inhabitants (2 million), it is difficult for Slovenia to find measures that keep the relative number of deaths or new cases of infections as low as for countries with more than 50 times more inhabitants (e.g., Japan). Therefore, Slovenia can be seen as an example for the case that fewer infections should have been possible through a theoretic performanceoriented view, while this seems unrealistic from a practical viewpoint [86] . (12) AUS, CAN, JPN, and NLD have early lockdowns in the first period and consequent infection tracking in common, which led to a stable and, compared to other countries, highly efficient health system with a mean efficiency score above 0.90. (13) Similar policies were adopted in the Northern European countries of DNK, FIN, NOR, and SWE, to keep restaurants and primary schools open and rely on citizens adhering to social distancing recommendations themselves. This seems to be another highly successful strategy, as the mean efficiency score was constantly above 0.90. Furthermore, these states have highly developed public health systems.

To determine more about the interdependency of pre-epidemic health system strategies and ad hoc interventions when fighting COVID-19, the following network DEA approach separates the efficiencies of both contributing factors. Figure 1 summarizes the approach [95] . The results of the network DEA (appendices 2 and 3) indicate that the efficiency of the pre-epidemic health strategy and ad hoc COVID-19 intervention are positively related to each other (r = 0.66). This seems reasonable when remembering the example of Germany expanding the intensive care hospital capacity based on the existing health care system capacity. Furthermore, the possibility of testing for COVID-19 infections depends on the available health care resources, e.g., doctors or medical professionals. Therefore, an important finding of the network DEA is that the institution of only ad hoc decisions, e.g., lockdowns, testing, or expanding intensive care capacity, will not lead to an overall successful fight against COVID-19. Rather, ad hoc decisions can contribute to the fight against COVID-19 and increase efficiency. Figure 2 demonstrates the relationship between efficiency for pre-epidemic health strategy and COVID-19 testing policy, where the size of the bubbles depends on the population of the country and the color indicates the Through a closer examination of the efficiency values per typology, it can be stated that performance-and primarycare-oriented public healthcare systems have been the most efficient ones in fighting the COVID-19 pandemic. This is true for the pre-pandemic health system efficiency, as well as for the efficiency of ad hoc measures. According to Reibling et al. [109] , this type is dominated by public financing but spends less money and uses fewer human resources to provide healthcare. The focus of the FIN, JPN, KOR, NOR, and SWE healthcare systems was clearly on primary care orientation, with a comparatively high proportion of primary care doctors compared with specialists [107, 108] .

Furthermore, the regulation-oriented public type was the least efficient in regard to the efficiency of the pre-pandemic healthcare strategy. This type was characterized by a medium level of resources that come primarily through public funding, but it has the highest level of access regulation and limits the choice of providers. The system is also characterized by the absence of formalized cost-sharing and the lowest level of out-of-pocket expenditures [109] . It is important to mention that countries in the regulation-oriented public type can increase their overall performance in fighting COVID-19 with good ad hoc management, e.g., through lockdowns or social distancing policies (DNK, NLD). However, incorrect decisions such as lockdowns that are introduced late (GBR) or only in small steps and inconsistently (ITA), together with regulation-oriented or inefficient public pre-pandemic healthcare system policies, led to the lowest efficiency levels among all OECD countries. A more counter-intuitive finding is that the efficiency was not dependent on the population size of the countries (node 1: r =-0.12; node 2: r =-0.06), population density (node 1: r =-0.01; node 2: r =-0.24), median age (node 1: r =-0.11; node 2: r =-0.26), proportion of the population that is older than 65 years (node 1: r = 0.03; node 2: r =-0.17), the proportion of the population that is older than 70 years (node 1: r =-0.08; node 2: r =-0.23), the GDP per capita (node 1: r = 0.05; node 2: r = 0.33), or the Human Development Index (node 1: r =-0.14; node 2: r =-0.11) [123].

Governmental regulations to lower the spread of COVID-19 through, e.g., isolation and quarantine policies, public gatherings limitations, or lockdowns, had versatile impacts on the economy. While the northern European countries Sweden (Government Response Stringency Index, GRSI 46.3 of 100) and Finland (GRSI 57.41 of 100) were defensive in their regulation policies compared to, e.g., France (GRSI 90.74 of 100) or Germany (GRSI 60.65 of 100). Therefore, the question of which policy is the best for a country's economy arises [62] . A second important factor is a direct intervention through, e.g., limiting product imports/exports and government aid programs. Table 3 of the DEA window analysis using I 1 , I 2 , and I 3 , as well as O 1 , O 2 , and O 4 . Furthermore, the average efficiency level in the economic model was correlated with the average GRSI between periods 1 and 12 per country. Taking the averages for the whole dataset (periods 1-12) showed a negative correlation of efficiency and GRSI (r =-0.45); the average of periods 5-12 showed a strong negative relationship, with r =-0.54, and no significant relation for the last periods (10-2; r =-0.28). For all moving average calculations, the correlation coefficient was negative (Appendix Table 7 ).

Summarizing these findings from an economic point of view, the higher the strength of the governmental response to COVID-19, e.g., through lockdowns and social distancing, the less efficient the country response is through the lens of economic efficiency. As presented in the previous chapters, lockdowns lead to a direct increase in efficiency, but the price is more unemployment and a weakening of the economy. The most promising approach to respond to COVID-19-related lockdowns of the economy and society is short-term work with governmental support, which had a positive impact on the efficiency observed the top 4 countries of Sweden (eff. = 1.00), Germany (eff. = 0.97), Spain (eff. = 0.93), as well as Switzerland (eff. = 0.84) 1 and was not reported for the rest of the examined OECD countries [113, 116] .

The following items for discussion stem from the presented results and are important for the objective of improving the public health economic answer to pandemic crises in a political and managerial sense:

(1) Traditional health systems and economic success factors for wealth and development, such as country and population size or the respective Human Development Index (HDI), are found to be less relevant to the efficiency of pandemic management in some countries. In contrast, we have found that small and large countries, Eastern and Western counties, and developed and less developed are represented in both the efficient group and the less efficient group in terms of their efforts and results against the COVID-19 pandemic. This is important for health economics research and decisions, as new predictor variables for decisions and analysis regarding the resource-efficient answer to pandemic situations have to be identified and applied in future (e.g., the discussion by [30] or in a pre-COVID-19 perspective the results by [83] and [34] ). (2) It is important to note that the development situation during a pandemic crisis is very dynamic, and individual countries are transitioning between various phases of success and failure at different times -successful crisis management in the past does not guarantee resistance to the pandemic for the future. This has already been found before the COVID-19 pandemic but has to be remembered nevertheless [24, 91] . In some cases, even a reciprocal connection can be recognized as successful crisis management is, by and large, lowering the acceptance of future restrictive measures in the fight against COVID-19. This is not only relevant to the ongoing discussions about "second waves" of COVID-19 but also the general risk of a long-term perspective and trade-off regarding health economic decisions. Research has to improve the analytical and prognosis perspective to enhance the pandemic foresight aspect of health economics. This also addresses a continuous learning perspective for the health care management discipline [65] . (3) Looking at the integrated model of health care management and economic evaluation of country-specific crisis management provides insight that governmental support of short-term work is an efficient concept to balance COVID-19 intervention and economic success. This might be true for further instruments developed during the pandemic in 2020, including the latest measures taken by China, the U.S. and the European Union countries. Further methods and research discourse are required to systematize and evaluate all the comprehensive approaches regarding analysis and policies in this regard; in many cases, only long-term hindsight might reveal the full picture, as we have learned from previous pandemics (Almond & Bhashkar Mazumder, 2005 for the Spanish flu of 1918/19) [122, 127, 129] . (4) Analysis and decision endeavors in health care management science have to take into account that international pandemic situations are real "rule changers" -many assumptions applied in health economics analysis and discourse simply do not hold in such situations, as already shown by Beutels et al. [17] . Therefore, dynamic and "out of the box" approaches have to be developed for application in situations that are beyond the routine steady state of health systems economics on normal days [85] . (5) For further discussion regarding successful COVID- 19 interventions, it may be beneficial to integrate data from deeper aggregation levels. An example is the impact of population size which has been found to be influential comparing rural and urban areas in Turkey [13] , but had no impact on in our country-related analysis. The same applied for approaches that seek to evaluate regional changes of peoples' actions, such as a more strict adoption of social distancing measures among the population, that reduce the spread [14] . However, as our research was intended to provide a bigger picture going beyond regional developments, the data quality on this aggregation level does not exist for a majority of OECD countries until now. The country managed to avoid high case numbers through intelligent and digital COVID-19 management, as well as persistent tracking, tracing, and testing of infected persons who were quickly identified and treated at an early stage. The country is thus a unique example of efficient COVID-19 control without lockdown. (6) The implementation of telemedical health services can be a promising concept for people seeking for a rapid first advice regarding a possible infection while keeping to social distancing measures at the same time. It could also help relieve pressure on the often overburdened primary care systems or emergency departments [56] . Parsons and Romanis [104] discusses an interesting case of a mobile abortion care service for the United Kingdom [103] . (7) During our analysis, we found that all countries chose diverse strategies that are mostly disconnected. This applies for the OECD countries, as well as when looking at European countries in isolation (see also, [112] . The paradigms of strict legally backed lockdowns in central Europe stand against softer responses of northern European countries. Therefore, we agree with Forman et al. [52] and call for a unified responses to pandemics. (8) Additionally, our results on OECD countries may be supplemented by integrating developing OECD countries that we did not evaluate within the DEA model, e.g., (a) Chile, (b) Columbia, and (c) Mexico: We start with (a) Chile and the results presented by Oliveira et al. [102] showing that timely and coordinated social distancing was a powerful non-pharmaceutical intervention, but difficulties in keeping the population under control decreased the impact of this measure. For (a) Chile and (b) Columbia, Benítez et al. [16] show that although stringent measures of containment and mitigation were introduced, pre-pandemic conditions, e.g., high informal employment and social inequalities, have undermined the effectiveness of the countries' responses to the pandemic. For (c) Mexico, Knaul et al. [80] demonstrate the absence of a uniform national response of Mexico and argue that coordinated, timely, rigorous response to the pandemic did not occur in Mexico. Arellanos-Soto et al. [5] evaluated Mexican Governmental measures by comparing data on influenza diagnosis, finding that the implementation of public health measures leads to a significant decrease. This is supported by Díaz-Castro et al. [38] , concluding that Mexico's health policies had an effect on slowing the pandemic's propagation, but population density and mobility played a fundamental role. In summary, the results for developing OECD countries are in line with our previously stated quantitative findings, including the suitability of social distancing measured and the fact that a lack of uniform, coordinated, and timely response may be reasons for countries to fall behind in an early stage. Furthermore, the cases of (a) Chile and (b) Columbia underline the relevance of pre-pandemic conditions [130, 133, 134, 136] . Altogether, this approach, involving country-specific modeling of input and output types for a health and general economic efficiency analysis to be used as a central evaluation measurement for bundles of health care and political measures, has proven to be worthwhile and should be subject to further research and discussion. Further research approaches are warranted regarding, for example, the identification of further relevant input and output types, as well as the inclusion of further individual countries or bundles of countries besides the OECD set applied here. Additionally, as we identified (partial) lockdowns and social distance measures as a possible key to fight the pandemic, more research on the evolvement of public values and perceptions during the COVID-19 pandemic should foster our understanding on the acceptance level of these measures as discussed by Denburg et al. [37] .

This paper has outlined specific approaches and results regarding the efficiency of health economic measures and interventions for the COVID-19 pandemic worldwide, including country-specific analysis of 19 OECD countries. The results show that country-specific efficiency regarding multiple input and output factors varies very broadly and is not connected to traditional clustering factors or economic success factors, such as size, population, or development status. Instead, specific individual types of measures like selected lockdown or dedicated economic support instruments seem to make a difference, which is an important message for health economic research and management. Looking into further analyses based on these finding presented here will pay off in future as government interventions can be improved and finetuned to optimize the resource-dependent results from health policy-related answers to the COVID-19 crisis. The contribution of this paper consists of (1) the specific identification of efficient countries in the fight against the COVID-19 pandemic during the first waves in 2020, which can serve as role models and sources of further information through research into comparative efficiency the specific measures taken in these countries; (2) the establishment of an evaluation scale through multidimensional efficiency measures that provides room for further elaboration in health economic research to identify success factors; and (3) the introduction of specific political measures that are expected to be useful in many future situations similar to the COVID-19 situation. Therefore, the paper provides a valuable contribution toward an improvement regarding public measures for pandemic preparation and management.

See Appendix Tables 4, 5, 6 and 7. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.

On the uncertainty of real-time predictions of epidemic growths: A COVID-19 case study for China and Italy

The 1918 influenza pandemic and subsequent health outcomes: an analysis of SIPP data

The impact of New Public Management on efficiency: An analysis of Madrid's hospitals

Productivity growth, case mix and optimal size of hospitals. A 16-year study of the Norwegian hospital sector

Decline in influenza cases in Mexico after the implementation of public health measures for COVID-19

COVID-19 Government Measures Dataset

Investment in antiviral drugs: A real options approach

Austrian Federal Ministry of Social Affairs, Health, Care and Consumer Protection.: COVID-19 emergency measures regulation (Coronavirus -Aktuelle Maßnahmen)

Practicing Corona -towards a research agenda of health policies

Hypothesis tests using data envelopment analysis

Some models for estimating technical and scale inefficiencies in Data Envelopment Analysis

A field study of scale economies in Software maintenance

Population density index and its use for distribution of Covid-19: A case study using Turkish data

COVID-19: R0 is lower where outbreak is larger

Corona virus: strengthened measures (Coronavirus: versterkte maatregelen

Responses to COVID-19 in five Latin American countries

Partially wrong? Partial equilibrium and the economic analysis of public health emergencies of international concern

Does the AIDS epidemic threaten economic growth?

Nonlinear and dynamic programming for epidemic intervention

Five types of OECD healthcare systems: Empirical results of a deductive classification

Understanding COVID-19 diffusion requires an interdisciplinary, multi-dimensional approach

Microsimulation of financial impact of demand surge on hospitals: The H1N1 influenza pandemic of fall

Initial impacts of global risk mitigation measures taken during the combatting of the COVID-19 pandemic

Global stability of an age-structure epidemic model with imperfect vaccination and relapse

Safety and efficacy of amiodarone in a patient with COVID-19

On the role of environmental corruption in healthcare infrastructures: An empirical assessment for Italy using DEA with truncated regression approach

Repeat SARS-CoV-2 testing models for residential college populations

A developmental study of data envelopment analysis in measuring the efficiency of maintenance units in the U.S. air forces. Ann

Measuring the efficiency of decision making units

Beyond COVID-19: How the "dismal science" can prepare us for the future

The Lancet

Data envelopment analysis: A comprehensive text with models, applications, references and DEA-solver software

Forecasting Brazilian and American COVID-19 cases based on artificial intelligence coupled with climatic exogenous variables

Epidemics control and logistics operations: A review

Emerging from the other end: Key measures for a successful COVID-19 lockdown exit strategy and the potential contribution of pharmacists

The coefficient of resource utilization

Does moral reasoning influence public values for health care priority setting? A population-based randomized stated preference survey

The Impact of Health Policies and Sociodemographic Factors on Doubling Time of the COVID-19 Pandemic in Mexico

An interactive web-based dashboard to track COVID-19 in real time

The effect of the 2009 influenza pandemic on absence from work

Pitfalls and protocols in DEA

Multi-stage group testing improves efficiency of large-scale COVID-19 screening

The Impact of Sociocultural Influences on the COVID-19 Measures-Reflections From Singapore

Physician payment schemes and physician productivity: Analysis of Turkish healthcare reforms

The role of the state in financing and regulating primary care in Europe: a taxonomy

Measuring Farrell efficiency for a firm with intermediate inputs

Efficiency measures for multiplant firms

The measurement of productive efficiency

Doctors, nurses, and the optimal scale size in the Portuguese public hospitals

Eu health systems classification: A new proposal from EURO-HEALTHY

12 Lessons learned from the management of the coronavirus pandemic

On the optimal production capacity for influenza vaccine

Attestation de déplacement et de voyage)

Implementation of altered provider incentives for a more individual-risk-based assignment of dental recall intervals: Evidence from a health systems reform in Denmark

When fear backfires: Emergency department accesses during the Covid-19 pandemic

Extension of the adopted guidelines for the restriction of social contacts (Erweiterung der beschlossenen Leitlinien zur Beschränkung sozialer Kontakte)

Efficiency and productivity of the food and beverage listed firms in the pre-recession and recessionary periods in Greece

The Government of the Republic of Korea.: Tackling COVID-19: Health, Quarantine and Economic Measures of South Korea

A decision support system for demand management in healthcare supply chains considering the epidemic outbreaks: A case study of coronavirus disease 2019 (COVID-19)

Non-parametric productivity analysis with undesirable outputs: an application to the canadian pulp and paper industry

Oxford COVID-19 Government Response Tracker: Blavatnik School of Government

Stockpiling anti-viral drugs for a pandemic: The role of Manufacturer Reserve Programs

A flexible cross-efficiency fuzzy data envelopment analysis model for sustainable sourcing

Learnings for Health Economics from the Early Stages of the COVID-19 Pandemic

Public Health

Latest updates on COVID-19 (Coronavirus)

Government decides to close noncore or strategic production activities (Governo decide chiusura attività produttive non essenziali o strategiche. Aperti alimentari, farmacie, negozi di generi di prima necessità e i servizi essenziali)

Predicting the impacts of epidemic outbreaks on global supply chains: A simulation-based analysis on the coronavirus outbreak (COVID-19/SARS-CoV-2) case

Manufacturing performance measurement and target setting: A data envelopment analysis approach

Fuzzy DEA-based classifier and its applications in healthcare management

Containing 2019-nCoV (Wuhan) coronavirus

Aligning SARS-CoV-2 indicators via an epidemic model: Application to hospital admissions and RNA detection in sewage sludge

The impact of the 1918 Spanish flu epidemic on economic performance in Sweden: An investigation into the consequences of an extraordinary mortality shock

The concept of vector efficiency in experimental studies of plague

The relationship between fever rate and telework implementation as a social distancing measure against the COVID-19 pandemic in Japan

The possible macroeconomic impact on the UK of an influenza pandemic

Simulation model of security control lane operation in the state of the COVID-19 epidemic

A study on healthcare supply chain management efficiency: Using bootstrap data envelopment analysis

Not far enough: Public health policies to combat COVID-19 in Mexico's states

The use of Data Envelopment Analysis (DEA) in healthcare with a focus on hospitals

An analysis of production as an efficient combination of activities

Long-distance effects of epidemics: Assessing the link between the 2014 West Africa Ebola outbreak and U.S. Exports and employment

Excess influenza hospital admissions and costs due to the 2009 H1N1 pandemic in England

Dynamic versus static models in cost-effectiveness analyses of anti-viral drug therapy to mitigate an influenza pandemic

A minimal common outcome measure set for COVID-19 clinical research

Congestion analysis to evaluate the efficiency and appropriateness of hospitals in Sicily

Learning to Trust Flu Shots: Quasi-Experimental Evidence from the 2009 Swine Flu Pandemic

Italian responses to the COVID-19 emergency: Overthrowing 30 years of health reforms? Health Policy

Investing in antibiotics to alleviate future catastrophic outcomes: What is the value of having an effective antibiotic to mitigate pandemic influenza?

Escaping epidemics through migration? Quarantine measures under incomplete information about infection risk

Covid-19: How to make between-country comparisons

Alternatives to sharing COVID-19 data with law enforcement: Recommendations for stakeholders

An international comparison of efficiency of inpatient mental health care systems

Fighting transient epidemics-optimal vaccination schedules before and after an outbreak

Total COVID-19 Tests Performed by Country

Health resources: Total in US dollars per capita

Evaluating social distancing measures and their association with the Covid-19 pandemic in South America

-19 governmental decisions to allow home use of misoprostol for early medical abortion in the UK

2020 developments in the provision of early medical abortion by telemedicine in the UK

Healthcare reforms in Cyprus 2013-2017: Does the crisis mark the end of the healthcare sector as we know it?

Corporate governance in Czech hospitals after the transformation

On the recent epidemic of diphtheria

The economic impact of H1N1 on Mexico's tourist and pork sectors

Worlds of healthcare: A healthcare system typology of OECD countries

Assessing the future direction of sustainable development in public hospitals: Timehorizon, path and action

9 The changing role of the state in healthcare systems

United but divided: Policy responses and people's perceptions in the EU during the COVID-19 outbreak

Effectiveness of the measures to flatten the epidemic curve of COVID-19: The case of Spain

Undesirable outputs in efficiency valuations

Modeling undesirable factors in efficiency evaluation

Internet of things (IoT) applications to fight against COVID-19 pandemic

COVID-19)

Modelling efficiency, screening and preferences within healthcare systems

The COVID-19 pandemic: a global natural experiment

Network DEA: A slacks-based measure approach

Classification of healthcare systems: Can we go further?

United Nations Development Programme

Frequency of nasopharyngeal swab collection and positivity for SARS-CoV-2 infection in the population of the Italian province of Udine with and without chronic conditions

The efficiency of health care production in OECD countries: A systematic review and meta-analysis of cross-country comparisons

International comparisons of the technical efficiency of the hospital sector: Panel data analysis of OECD countries using parametric and non-parametric approaches

Early assessment of the impact of mitigation measures on the COVID-19 outbreak in Italy

How to address medicines shortages: Findings from a cross-sectional study of 24 countries

Can masks be reused after hot water decontamination during the COVID-19 pandemic? Engineering

Psychological impact of Coronavirus Disease 2019 (COVID-19) epidemic on medical staff in different posts in China: A multicenter study

Changing Healthcare System Types

Covid-19 scenario modelling for the mitigation of capacity-dependent deaths in intensive care

Covid-19 social distancing in the Kingdom of Saudi Arabia: Bold measures in the face of political, economic, social and religious challenges

Preventing COVID-19 from the perspective of industrial information integration: Evaluation and continuous improvement of information networks for sustainable epidemic prevention

A hybrid data envelopment analysis and game theory model for performance measurement in healthcare

The epidemiology, diagnosis and treatment of COVID-19