key: cord-1017392-ouwzewms authors: Woldetsadik, Mahlet A; Bratton, Shelly; Fitzpatrick, Kaitlin; Ravat, Fatima; Del Castillo, Lisetta; McIntosh, Kelsy J; Jarvis, Dennis; Carnevale, Caroline R; Cassell, Cynthia H; Chhea, Chhorvann; Prieto Alvarado, Franklyn; MaCauley, Jane; Jani, Ilesh; Ilori, Elsie; Nsanzimana, Sabin; Mukonka, Victor M; Baggett, Henry C title: Qualitative evaluation of enabling factors and barriers to the success and sustainability of national public health institutes in Cambodia, Colombia, Liberia, Mozambique, Nigeria, Rwanda and Zambia date: 2022-04-01 journal: BMJ Open DOI: 10.1136/bmjopen-2021-056767 sha: 2d8e0398e470687d46cc8b360da3a775257bdb0b doc_id: 1017392 cord_uid: ouwzewms OBJECTIVES: The success of National Public Health Institutes (NPHIs) in low-income and middle-income countries (LMICs) is critical to countries’ ability to deliver public health services to their populations and effectively respond to public health emergencies. However, empirical data are limited on factors that promote or are barriers to the sustainability of NPHIs. This evaluation explored stakeholders’ perceptions about enabling factors and barriers to the success and sustainability of NPHIs in seven countries where the U.S. Centers for Disease Control and Prevention (CDC) has supported NPHI development and strengthening. DESIGN: Qualitative study. SETTING: Cambodia, Colombia, Liberia, Mozambique, Nigeria, Rwanda and Zambia. PARTICIPANTS: NPHI staff, non-NPHI government staff, and non-governmental and international organisation staff. METHODS: We conducted semistructured, in-person interviews at a location chosen by the participants in the seven countries. We analysed data using a directed content analysis approach. RESULTS: We interviewed 43 NPHI staff, 29 non-NPHI government staff and 24 staff from non-governmental and international organisations. Participants identified five enabling factors critical to the success and sustainability of NPHIs: (1) strong leadership, (2) financial autonomy, (3) political commitment and country ownership, (4) strengthening capacity of NPHI staff and (5) forming strategic partnerships. Three themes emerged related to major barriers or threats to the sustainability of NPHIs: (1) reliance on partner funding to maintain key activities, (2) changes in NPHI leadership and (3) staff attrition and turnover. CONCLUSIONS: Our findings contribute to the scant literature on sustainability of NPHIs in LMICs by identifying essential components of sustainability and types of support needed from various stakeholders. Integrating these components into each step of NPHI development and ensuring sufficient support will be critical to strengthening public health systems and safeguarding their continuity. Our findings offer potential approaches for country leadership to direct efforts to strengthen and sustain NPHIs. National Public Health Institutes (NPHIs) provide countries with the tools to conduct strategic, evidence-based public health system strengthening. [1] [2] [3] They serve as focal points for the coordination and implementation of critical global health security functions, including workforce development, emergency preparedness and response, laboratory and disease surveillance. 2 They also have strategic functions relevant to the long-term Strengths and limitations of this study ► The study involved the participation of key stakeholders from National Public Health Institutes (NPHIs), government organisations, as well as nongovernmental organisations in seven low-income and middle-income countries (LMICs), providing rich and diverse perspectives. ► Data were collected from seven countries that were purposively selected and might not be representative of all NPHIs supported by U.S. Centers for Disease Control and Prevention (CDC) in LMICs. ► Stakeholder perspectives from NPHIs that receive CDC support might differ from those in countries not supported by CDC, as perceived indicators of sustainability can differ by country context. Open access strategic development of health systems, such as partnership development, programme management and longterm planning. By unifying key national public health functions, NPHIs can improve coordination and increase efficiency, 4 which is especially critical during public health emergencies. Without an NPHI, public health programme implementation can often be fragmented across multiple entities within a health system. An NPHI aims to consolidate activities in a coordinated science-based structure with strong leadership and accountability. 4 In low-income and middle-income countries (LMICs), NPHIs are often established with financial and technical support from partners, including the World Bank, the International Association of NPHI (IANPHI) and the U.S. Centers for Disease Control and Prevention (CDC). Since 2011, CDC has collaborated with country government ministries of health (MoH) and other partners to support the establishment or strengthening of NPHIs in over 30 countries. 5 As part of its global health strategy, 6 CDC provides technical and financial support to NPHIs to strengthen their functions in various areas, including strategic and operational planning, relevant scientific projects, human resource development, financial management and communications. CDC supports countries with NPHIs in all stages of development: those looking to establish a new NPHI, nascent NPHIs recently established (less than 5 years), and those with mid to long year leadership roles in the public health system (ranges from 10 years to 100+ years). A successful NPHI is able to effectively meet its mandate and perform the functions for which it is responsible. In addition, success in NPHI capacity strengthening should be measured not only by short-term gains, but also by the sustainability of those gains and of the NPHI itself. There is increasing interest among country governments and partners, including CDC, to understand what is necessary for NPHIs to become sustainable in the long term. 7 Sustainability, within the context of international development, refers to the continuation of programmes, 8 specifically a programme's ability to successfully deliver intended benefits for an extended period after major financial and technical assistance from external donors is terminated. 9 This can directly apply to NPHIs that endure through changes in government leadership, have direct government funding, and a dedicated workforce. The NPHI may also have bilateral and multilateral partners but is not fully dependent on these partnerships for continuity. The sustainability of NPHIs is critical to countries' timely detection of, and response to, public health threats. The novel COVID-19 pandemic has shown the vital role NPHIs can play to mitigate the impact of infectious diseases in LMICs. 10 11 However, there is a dearth of data on enabling factors and barriers to the sustainability of NPHIs and other public health programmes in LMICs. In late 2019, we assessed the impact of CDC's investment to the development and strengthening of NPHIs in seven countries. As part of this evaluation, we sought to understand stakeholder perceptions of what helps and hinders NPHIs achievement of success and sustainability. We purposively selected seven countries from 30 countries where CDC has partnered on NPHI development and strengthening: Cambodia, Colombia, Liberia, Mozambique, Nigeria, Rwanda and Zambia. We included countries where CDC's financial investment was relatively high and accounted for geographic variation, and whether CDC support was provided directly or through cooperative agreements with implementing partners, and the country's NPHI development stage. At the time of the study, Liberia and Zambia were established less than 5 years ago; Cambodia, Nigeria and Rwanda were midrange (6-25 years); and Mozambique and Colombia were longstanding organisations (40+ years). Potential participants were selected through a collaborative process among NPHI and CDC staff in the seven countries and CDC's NPHI Programme. We conducted interviews in each country with a broad range of stakeholders with diverse perspectives. Participants included NPHI leadership (eg, NPHI director, emergency operations centre leader, laboratory leader), non-NPHI government staff (eg, MoH permanent secretaries, public health directors, district health office directors) and partners collaborating with the NPHIs (eg, universities, non-governmental organisations, United Nations agencies, international organisations). We sought to interview 13-15 participants in the capital of each country after considering the human and financial resources available to implement the evaluation. 12 13 We contacted participants by email, shared the evaluation goals and asked for their participation. Data collection MAW and KF conducted interviews with all participants from August 2019 through January 2020. MAW was the lead of the evaluation and is a PhD researcher with extensive experience conducting evaluations in low-income settings. KF was a fellow with CDC's NPHI Programme and had experience working in sub-Saharan Africa. MAW provided refresher training on qualitative methods to team members who participated in data collection, data analysis and writing. After obtaining information about participants' demographics and their roles in their respective institutes, the semistructured interview protocol explored participants' perception around factors they identified to be important for their NPHIs' success and sustainability, if there were activities that were currently being implemented towards these goals, potential barriers or threats to achieving NPHIs' success and sustainability, the types of activities needed to ensure the Open access continuation of these institutes, and the key stakeholders or partners that should be involved in this process. Interviews lasted 41-96 min (median=47.5 min) and explored participants' perceptions of enabling factors and barriers to NPHIs' sustainability. All interviews were conducted in private in English, except 14 interviews in Colombia and one interview in Cambodia, which were conducted through experienced interpreters in Spanish and Khmer, respectively. All recordings were transcribed verbatim and coded using MAXQDA V.20.0.2. 14 We analysed transcripts using a directed content analysis approach, which began with the interview questions as a guide for developing initial themes but allowed flexibility for additional themes to emerge directly from the data. 15 Four team members, including MAW and KF, coded the interviews. The first iteration of the codebook was used to code six transcripts independently and was then checked, refined and expanded. The team reviewed a random selection of coded transcripts to ensure consistent application of theme categorisations and used an iterative process to resolve any discrepancies in the coding application. This process established intercoder reliability, including consistency and consensus coding application within the MAXQDA platform. On coding completion, the team reviewed the coded excerpts for key themes and identified themes through repetitions, a well-established technique to identify themes. 16 To assess the validity of our conclusions, we employed 'member checking' (ie, sharing and soliciting feedback on the results and conclusions from a small group of representatives at each of the seven NPHIs) and triangulation (ie, collecting data from multiple sources and using more than one interviewer). 17 18 The team regularly debriefed to discuss the results, emerging themes and potential conclusions, which mitigated the potential for researcher bias during data analysis. 19 Patient and public involvement It was not appropriate or possible to involve patients or the public in the design, or conduct, or reporting, or dissemination plans of our evaluation. A total of 96 stakeholders from the seven countries participated in the interviews (10-19 stakeholders per country). Twelve stakeholders who had initially agreed were unable to participate due to last-minute scheduling conflicts. Of the 96 persons interviewed, 43 (45%) were NPHI staff, 29 (30%) were non-NPHI government staff and 24 (25%) worked in non-governmental or international organisations. Sixty-six (69%) participants were male. Institutions represented included MoH, District Health Departments, Ministry of Foreign Affairs, Ministry of Agriculture, Ministry of Statistics, Universities, World Bank, WHO and the West African Health Organisation. A detailed list of participants' positions and institutions for the seven countries is presented elsewhere. 20 Enabling factors for success and sustainability of NPHIs All NPHI staff, 23 (79%) non-NPHI government staff and 21 (88%) non-government partner staff identified factors they considered critical for the success and sustainability of NPHIs in their countries. Five themes emerged: (1) strong leadership, (2) financial autonomy, (3) political commitment and government ownership, (4) building capacity of NPHI staff and (5) maintaining strong relationships with partners (table 1) . Participants across all groups and countries identified strong leaders at the top and within the NPHIs' divisions as one of the most important enabling factors for NPHIs' success and sustainability. Participants noted the importance of leaders who understand the value of NPHIs and can articulate a clear vision and a strategic direction for the institutes. Participants added that having leaders that are respected and recognised by senior governments officials can foster trust and confidence, while also contributing to NPHIs' reputations both internally and among partners. Trusted leaders can navigate evolving political landscapes, negotiate with other government officials, and secure the resources needed to run and sustain the organisation. Participants also emphasised the importance of having a leader who was committed to improving the competencies of the public health workforce both at national and subnational levels, which they considered critical for NPHIs' sustainability. Participants noted that most NPHIs, including six of the seven included in this evaluation (Colombia being the exception), depend heavily on donor funding to support core public health activities, such as laboratory and surveillance. Therefore, participants underscored financial autonomy as a key driver for long-term NPHI sustainability. Some suggestions for reaching financial autonomy included applying for research grants, charging for public health services, conducting cross-cutting research and building strategic partnerships with both the public and private sectors, such as research institutes and universities. Political commitment and country ownership Participants identified political commitment as another important factor for the success and sustainability of NPHIs. They emphasised the link between government ownership of NPHIs and increased government funding, and that both are essential for NPHIs' continuation and longevity. In addition, participants noted that political commitment and financial support facilitate public recognition and credibility of NPHIs, which are crucial to building trust with citizens and effective risk communications during public health emergencies. Moreover, Table 1 Participants' perceptions of key enabling factors for the success and sustainability of National Public Health Institutes (NPHIs) in the seven countries included in the NPHI evaluation (N=96), August 2019-January 2020 Center] to fulfill its mission of educating, implementing, but also conducting research.'(NPHI staff (10) , Rwanda) 'In our establishment as an entity, we are also given the right to raise money as an entity. And so, that is an area where we really want to have some strength so that we can generate income to support our self instead of just waiting and asking for help.'(NPHIL staff (5) (8), Zambia) 'We have to work on getting more funding from the government, and that will come from being better at what we do, communicating what we do, and becoming, as much as possible, a success story within the government. So, that will allow us to negotiate more budget from the government. We have been doing that, but I think that is an avenue that we need to pursue.'(NPHI staff (4), Mozambique) 'I think there needs to be dedication of government funds to enable the institute. Yes, more funding and early release of funds. I know it's a general problem but given the critical role that we play in epidemic and pandemic response, I think committing that funding pool for NCDC is essential.'(NPHI staff (8) (1), Rwanda) '… in terms of building skills, I think this is an area of ongoing need because we've identified some gaps. But, for example, I have noticed that perhaps there's a bit more focus on the technical skills of the staff here, and that's why PHE [Public Health England] is providing human resources and management support. This has been lacking. I'm sure that the technical people cannot exist without the needed logistics and cross-cutting and organizational support. I think this is something that needs to grow better within the organization.'(PHE staff (1), Nigeria) 'I think building that capacity for the region would be an asset especially if one country experiences threats, we can mobilize from the other countries and provide support accordingly. So, we needed to build that capacity for our region.'(NPHI staff (1), Zambia) Forming strategic relationships with partners 'So, the key to sustainability is to ensure that they [NCDC] are delivering for their partners, and their partners are also delivering. The idea it to achieve this nice balance and for everyone to be happy. With this comes the trainings, the capacity building, and the funding.'(WHO staff (3), Nigeria) 'This is a good question because sustainability is a big thing. So, we need support from the government. We need commitment from ourselves, but we also need a very strong relationship with our partners in order to make sure that have a strong sustainability plan.'(NPHI staff (9) , Rwanda) 'They [CDC] helped us bring the institute back to the 21st century and that meant having the necessary resources in place, but more importantly the experts. We could not find experts in Colombia because there is no other institute stronger than us… through the maturity framework exercise, CDC has given us the understanding of the areas where we need to close gaps in order to get to that forefront status.'(NPHI staff (5), Colombia) '…going forward, if we have the right management, NPHIL needs to continue to be supported by our partners, CDC, to ensure that we, not only have capacity in place, but that we are able to generate the resources we need going forward to maintain the gains that we've made.'(NPHI staff (5) , Liberia) The number in parenthesis for each quote label signifies the unique number that was assigned to the interviewee in that specific country. political commitment could lead to increased government funding for NPHIs, which reduces their dependence on donor funding and positions them to respond effectively to major public health threats as they occur. Building capacity of NPHI staff Participants across countries and groups described the importance of strengthening staff knowledge and skills to conduct the core public health work of NPHIs, such as laboratory, surveillance and research activities. Furthermore, participants underscored that sustainability of NPHIs will depend on having skilled staff who are committed to the mission of their institutes; are subject matter experts in core public health functions, including emergency preparedness and response and disease detection; and can implement programmes and manage resources efficiently. Participants also emphasised that having a skilled workforce can improve the quality of public health activities, strengthen NPHIs to carry out current and future public health work and increase the perception of value of NPHIs among the public, which can contribute to regional, national and global health security. Participants noted that forging strategic relationships with sub-national health entities, other government sectors, other nations and multilateral organisations was essential for NPHIs' sustainability. Participants shared that improving collaborations across various partners can lead to a more integrated public health response during emergencies. In addition, partnerships built on trust and mutual benefits can broaden the scope of NPHIs' public health activities, lead to additional economic opportunities and result in financial autonomy. Thirty-one (72%) NPHI staff, 11 (38%) non-NPHI government staff and 9 (38%) non-government staff identified potential barriers or threats to NPHI sustainability, which were categorised into three themes: (1) reliance on partner funding to maintain NPHI activities, (2) changes in NPHI leadership or countries' political landscape and (3) NPHI staff attrition and turnover (table 2) . Reliance on partner funding to maintain NPHI activities Over 50% of participants said that dependence on partner funding to maintain core public health activities was a significant threat to NPHIs' long-term sustainability. They underscored the risk of heavily relying on inconsistent funding, which can jeopardise NPHIs' ongoing public health work and can hinder timely and effective response to public health emergencies. Participants cautioned that NPHIs would not be able to fulfil their mandates without sufficient financial support from their governments. They believed that it was the responsibility of country governments to allocate adequate funding for NPHIs, which can mitigate the impact of reductions in donor support and demonstrate national commitment to public health priorities. Participants across all countries said that government funding for NPHIs is inadequate when compared with the financial support received from partners. In many cases, most operational costs were covered by donor funding, which participants found to be problematic and unsustainable, especially because partners often do not fund public health activities at the local level. They added that insufficient funds could limit long-term strategic public health and preparedness planning, consistently forcing NPHIs into a reactive posture. Participants stressed that a combination of consistent government support and NPHIs' ability to generate income, in addition to donor funds, is needed for NPHIs to address the most salient public health challenges in their countries and become sustainable. A second threat to NPHIs' success and sustainability included changes in NPHI leadership as a result of evolving political conditions in countries. Participants emphasised that NPHI leadership positions should be selected based on individuals' skills and experience to lead these institutions. This issue was identified as a risk to NPHI sustainability because leadership changes often shift public health priorities based on political pressures that can be misaligned with the NPHI strategic plan and public health needs of the country. Participants indicated that political changes could weaken collaborations between NPHIs and partners if the new political direction does not support these relationships. Suggestions to mitigate these issues included implementing a plan to maintain partner relationships even if the leadership of the country changes. NPHI staff attrition and turnover NPHI staff attrition and turnover were identified as threats to NPHI success and sustainability. Losing trained staff can be costly for institutions because they must spend limited resources on training new people on a rolling basis. Participants' recommendation to mitigate these issues included attracting and retaining experts with diverse backgrounds by providing competitive salaries, enhancing knowledge and skills through trainings and offering opportunities for advancement in the organisation. Our findings provide new evidence to inform NPHI development from local stakeholders in countries at varied income levels on three continents, with different political systems and at different stages of NPHI maturity. Although the importance of sustainability and factors contributing to sustainability have been described in guidance documents and commentaries, 2 21 empirical data in peer-reviewed journals have been limited. In a mixed-methods study of leadership roles in sustaining Open access evidence-based interventions in the USA, Aarons et al found leadership to be a predictor of sustainability. 22 In a best practices document on Legal Mandates and Governance of NPHIs, IANPHI described strong leadership as essential to an NPHI's identity, 2 an assertion supported by our findings. Our findings demonstrate that strong leadership is anchored both in the traits and success of individuals empowered to lead an NPHI. The authority intrinsically provided by the positioning of the NPHI within the broader government structure tends to Table 2 Participants' perceptions of barriers or threats to the success and sustainability of National Public Health Institutes (NPHIs) in the seven countries included in the NPHI evaluation (N=96), August 2019-January 2020 Relying on external partners for unpredictable funds to maintain NPHI activities '… the big issue for RBC is their dependence on partners to sustain key programs.'(WHO staff (7), Rwanda) 'The threats to sustainability are that we still rely on the partners significantly.'(NPHI staff (3), Cambodia) 'Ninety percent of the time, there's no operational support from the government. We exclusively get funding for salaries. So, if NPHIL did not receive support from CDC, the institute wouldn't be able to carry out the public health functions that they are supposed to work on.'(NPHI staff (5) , Liberia) 'To be honest, I don't think that we would have the capacity to run them [laboratories] if we ran out of donor funds or donor agencies leave.'(University of Zambia (UNZA) staff (11) , Zambia) 'I think a few years ago, there was a study from the World Bank that showed that we had the 3rd most dependent health system in the world when it comes to foreign help. So, the whole health system is unsustainable. The Ministry of Health, our parent institution, is not sustainable. I mean, most of our overall state budget comes from foreign partners. So, the whole country is not sustainable. Being completely independent of external sources of funding is probably not achievable within my lifespan.'(NPHI staff (4), Mozambique) Lack of continued government support for NPHIs, including funding 'The issue of lack of funding from the government affects the sustainability of the institute. They [INS] need domestic financial resources to cover the deficit that they have.'(MoH staff (13) , Mozambique) 'Sustainability for me in largely has to do with funding. Currently, most of the support comes from external funding so NPHIL need to find other means of support internally from the government and maybe through other grants that could help them become sustained. Relying almost entirely on donor funding is a risk.'(MoH staff (17) , Liberia) 'We are a public institution anyway. So, I don't see any other way for RBC to be sustainable …as external funding decreases, the government should be putting money.'(NPHI staff (6), Rwanda) 'Right now, one of the key limitations is resources and funding. So, while they're [NCDC] advocating for a bigger budget space at the national level, their impact at state levels is still quite minimal. And if they got state buy-in, then they'd be a lot more stable. They would also have less pressure to supply everything from the national level to the states.'(WHO staff (3), Nigeria) 'The institute might not be able to be sustainable without outside support. By outside support, I mean support from Ministry of Health and support from U.S. CDC, as well as other partners that keep pushing the institute to keep moving.'(IPC staff (4), Cambodia) 'We depend on a national budget that is allocated to us and this will always be a limitation for any institution… sustainability is a challenge because we still have support through the government, but it's still not enough to carry out our core functions and activities. The funds that the government gives us are not enough.'(NPHI staff (9) , Colombia) Changes in NPHI leadership or countries' political landscape 'We don't know what's going to happen [with the government reform] because they could use this reform to move the NIPH out. We never know. But we say that the best way to prevent this from happening is if we do our job well. It's not about the argument for keeping it, but it is about doing our job well and showing that we are needed by others.'(NPHI staff (5), Cambodia) 'I'm not sure if it's a threat now that INS has a new mandate. So, the changes in the government might also impact INS' structure. But the function is already established, and I don't think that much will be affected even with the government. Because although we are autonomous, we are still in the same machine and we are still being affected by changes in policy and so on. But with the new mandate, I think somehow, we've created a kind of protection against changes that might affect us.'(NPHI staff (2), Mozambique) 'While NPHIL is a semi-autonomous agency, the government controls a lot of things because the first three positions are appointed by the government. And then those positions are politicized, and they don't keep them in the context of the function. So, we are at a very critical stage as an entity. I think that's the biggest challenge for the current situation'(NPHI staff (5) (1), Nigeria) One thing that actually affects sustainability in general terms, not specific to NCDC, is personnel turnover. High staff turnover affects sustainability. I think NCDC, should minimize this risk. because it makes no sense to train someone and then the person leaves the next month. And then another person comes, you have to spend resources to train a person again. So, it will be important if they are able to maintain staff. High staff retention is very key in sustainability at whatever level.'(Department of Public Health at Federal Capital Territory staff (12) , Nigeria) 'Turnover of staff happens because people go wherever the salary is better. Even me, perhaps I don't see myself staying at RBC for 10 years. I see this happening at my labs. I see it at transfusion services. I see it within RBC and other programs. People move.'(NPHI staff (6) , Rwanda) '…the retention of the critical staff is key. I have three or four PhD level staff here, and they know they are competitive. Everyone is looking for them, so we need to find a way so that we don't lose them.'(NPHI staff (1), Zambia) 'Human resource is one major concern that might be an issue. This is serious because workforce is needed to sustain the health system, and to be able to provide the health services. So, I think that NIPH and the health system as a whole face problem with human resources.'(MoH staff (6) , Cambodia) The number in parenthesis for each quote label signifies the unique number that was assigned to the interviewee in that specific county. Open access increase visibility, credibility and the ability to implement interventions and mobilise funds during an emergency. Thus, a crucial decision point in the process of establishing an NPHI is anchoring the NPHI within or closely linked to the government structure to effectively fulfil its mandate. Political commitment and country ownership of NPHIs are essential guiding principles of donors who support NPHI development, 23 and were identified as critical enabling factors by our participants. A clear understanding of the purpose and objective of an NPHI within the public health system of a country is key to country ownership and political support behind the organisation. Legislation can play an important role in defining an NPHI's mandate and also helps mitigate against uncertainties that may arise with changes in political leadership. 7 Often, this comes from demonstrating value of an NPHI and its role in the economics of public health investments. Recent global pandemics have highlighted the need for effective coordinating entities to prevent, detect and respond to health emergencies. 10 11 However, chronic underspending in public health and the difficulty of demonstrating how public health investments decrease morbidity and mortality remains a challenge. In addition, long-term government commitment, including dedicated financial, infrastructural and human resources support for NPHIs in the national budget, is critical to their sustainability. 7 Inconsistent government funding is a challenge to the sustainability of NPHIs in LMICs. 24 Our findings indicate that reliance on donor funding was believed to be a major threat to NPHIs' ability to maintain key public health activities, including emergency preparedness and response. We found that an important distinction was made between increased funding from the government and the NPHI being able to raise funds to contribute to its sustainability. This distinction may lay in the phased approach of NPHI establishment as more developed NPHIs-after having sustainable funding from the government or foundational funding from donors-can expand to generate income for increased financial flexibility. Our findings indicate that having a semiautonomous financial system in place, which would allow NPHIs to quickly deploy resources, could result in effective emergency response. In addition to self-generated income, NPHIs could be authorised to access emergency contingency funding to scale up operations, ensure critical operations and reduce the reliance on donor funding. Staff attrition and turnover were identified as additional threats, a finding that comports with those of previous studies that identified a skilled workforce and expertise as essential components for organisations' sustainability. 25 26 Building a skilled NPHI workforce is crucial to undertaking new and existing public health challenges, tackling complex health problems and the overall sustainability of NPHIs. [27] [28] [29] Our evaluation had some limitations. We collected data from only seven countries that were purposively selected and might not be representative of all NPHIs supported by CDC in LMICs. In addition, stakeholder perspectives from NPHIs that receive CDC support might differ from those in countries not supported by CDC, as perceived indicators of sustainability can differ by country context. 30 Our evaluation was exploratory in nature and did not use validated indicators to measure NPHIs' sustainability, which currently do not exist. Therefore, we could not assess how different indicators might contribute to the sustainability of NPHIs and evaluate how they might interact. However, the main enabling factors and barriers identified in our evaluation can be used as the first step to creating quantitative measures of sustainability. These measures can then be validated and used to assess the progress toward sustainability of young NPHIs receiving donor support. Moreover, validated sustainability measures can be incorporated into existing NPHI maturity models, like the Stage Development Tool, 31 to measure ongoing status and develop plans for improvement. NPHIs success and sustainability is an important concern of country governments, CDC and organisations interested in the success of these vital public health agencies. Our findings contribute to the scant literature on the sustainability of NPHIs by identifying essential components of sustainability and the types of support needed from various stakeholders. Integrating these components into each step of NPHI development, 7 and ensuring sufficient support from different actors, especially country governments, will be critical to strengthening public health systems and safeguarding NPHIs' continuity. As next steps, countries' leadership might consider the potential implications of our findings and determine what may work best for their institution and country. Contributors MAW, SB, DJ, CRC, FR, CHC and HB conceptualised the study. CC, FPA, JM, IJ, EI, SN and VMM facilitated the implementation of the evaluation in the seven countries. MAW and KF collected the data. MAW, KF and LDC analysed the data. MAW, KF, FR, KJM and HB drafted the manuscript. All authors reviewed all drafts and approved the final version of the manuscript and MAW is the guarantor author. Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors. Competing interests None declared. Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research. Patient consent for publication Not applicable. Ethics approval This evaluation was reviewed through CDC's project determination process and the Office of Management and Budget and was exempted and did not require review from institutional review boards. Participants were notified of data confidentiality, data safeguarding procedures and their right regarding participation. All participants provided written consent before the interview. Interviews were recorded if the participant agreed. Two participants declined to be recorded. Provenance and peer review Not commissioned; externally peer reviewed. Data availability statement Data are available on reasonable request. The data presented in this article are not readily available because of confidentiality agreements with participants. Any reasonable requests should be directed to Dr. Mahlet A Woldetsadik ( mwoldetsadik@ cdc. gov). Open access This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/. Mahlet A Woldetsadik http://orcid.org/0000-0001-7780-2915 National public health Institutes: contributing to the public good International Association of National Public Health Institutes. Saving lives through national public health Institutes Building operational public health capacity through collaborative networks of national public health Institutes US Centers for Disease Control and Prevention. CDC's National Public Health Institute Program US Centers for Disease Control and Prevention and Its Partners' Contributions to Global Health Security Cdc global health strategy African centers for disease control and prevention, African Union. Framework for development of national public health Institutes in Africa Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy Sustainability of development programs: a compendium of donor experience Regional Office for Africa, World Health Organization. Weekly bulletin on outbreaks and other emergencies Africa's response to COVID-19 How many qualitative interviews is enough? expert voices and early career reflections on sampling and cases in qualitative research Characterising and justifying sample size sufficiency in Interview-Based studies: systematic analysis of qualitative health research over a 15-year period Available: maxqda.com/help-max20/welcome Three approaches to qualitative content analysis Techniques to identify themes Naturalistic inquiry Qualitative research design: an interactive approach Qualitative data analysis: an expanded sourcebook Stakeholders' assessment of US centers for disease control and prevention's contributions to the development of national public health Institutes in seven countries Stronger national public health Institutes for global health The roles of system and organizational leadership in system-wide evidence-based intervention Sustainment: a Mixed-Method study International Association of National Public Health Institutes. Framework for the creation and development of national public health Institutes Public health program capacity for sustainability: a new framework The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change Sustaining innovations in community prevention systems: a data-informed sustainability strategy: a data-informed sustainability strategy Improving the world's health--the role of National Public Health Institutes National Public Health Institute partners. Improving the world's health through national public health institutes The role of public health institutions in global health system strengthening efforts: the US CDC's perspective Advancing the application of systems thinking in health: analysing the contextual and social network factors influencing the use of sustainability indicators in a health system--a comparative study in Nepal and Somaliland Use of the staged development tool for assessing, planning, and measuring progress in the development of national public health Institutes Acknowledgements The authors thank Natalie Brown and Basia Tomczyk for protocol review, Camila Florez, Dilia Ballen and Kanha Sar for interpretation services in Colombia and Cambodia. We are grateful to Martha Knuth, Alice Williams, Stephanie K. Young, Jacob Clemente and Whitney Remy for assistance with literature reviews, Augusto Lopez for Portuguese and Spanish translations of tools and CDC country offices and NPHIs in the seven countries for logistical support and recruitment of participants. We also thank the Division of Global Health Protection's Science and Strategic Information Office in the Centre for Global Health at CDC for reviewing and providing constructive feedback on various drafts of this manuscript. Lastly, we greatly appreciate all the participants who agreed to be interviewed and shared their perspectives. This publication was supported in part by RTI International Contract Number # 200-2017 F-96261, funded by CDC.