key: cord-0027399-sa8myogu
authors: Jaramillo, Jahn; Ning, Mariangeli Freitas; Cadena, Loren; Park, Michael; Lo, Terrence; Zielinski-Gutierrez, Emily; Espinosa-Bode, Andres; Reyes, Marines; Del Rosario Polo, Maria; Henao, Olga
title: Evaluation of the collaborative integrated surveillance system (ViCo) in Guatemala: a qualitative study on lessons learned and future perspectives
date: 2022-02-18
journal: BMC Public Health
DOI: 10.1186/s12889-022-12719-7
sha: 918b2a16485d17389dfbf0a7b57ec9bb09fcef2b
doc_id: 27399
cord_uid: sa8myogu

BACKGROUND: The collaborative integrated surveillance system known as Vigilancia Integrada Comunitaria (ViCo) was implemented in 2007 to better understand and characterize the burden of diarrheal, respiratory and febrile illnesses in Guatemala. METHODS: To evaluate the usefulness of ViCo and inform a redesign of the system and new surveillance activities in the Central American region, personnel from the United States Centers for Disease Control and Prevention (CDC) conducted thirty-nine in-depth interviews from June—December 2018 with key stakeholders responsible for the design and implementation of ViCo in Guatemala. A semi-structured questionnaire adapted from the Updated CDC Guidelines for Evaluating Public Health Surveillance Systems was used for data collection. We used a grounded theory approach to explore stakeholder perceptions of ViCo and generate recommendations for improvement. Primary qualitative findings were organized based on thematic areas using ATLAS.ti version 8 software. RESULTS: Emergent themes relevant to the usefulness of ViCo were organized across strengths, weaknesses, and recommendations pertaining to the: (1) Size and Complexity of ViCo, (2) Stakeholder Expectations About the Objectives of ViCo, (3) Data Management and Structure of the Information System, (4) Local Control of Data, (5) Integration of ViCo within the Ministry of Health, and, (6) Improvement of the Operational and Design Aspects of ViCo across System, Process, and Output levels. CONCLUSIONS: Stakeholders perceived ViCo to be useful. They recommended measures to improve system performance and quality, including simplifying the surveillance system, routine data analysis and feedback, and channeling efforts towards integrating surveillance data into the national health information system. To create a well-performing surveillance system and achieve the intended objective of surveillance for public health action, ongoing evaluation and assessment of surveillance activities are necessary. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12889-022-12719-7.

Public health surveillance is the continuous and systematic collection, analysis, interpretation, and dissemination of health-related data for public health action to improve health [1] . An effective surveillance system can provide early warning for public health emergencies and help guide the planning, implementation, and evaluation of public health activities and policies [1] . Periodic evaluations of surveillance systems are therefore essential for enhancing the quality, efficiency, and usefulness of information that drives informed decision-making in rapidly changing environments [1] [2] [3] .

There is a need for evaluation data to guide the development of robust and useful disease surveillance systems that are also sustainable [4] . Understanding the extent to which surveillance systems are able to successfully collect and analyze data to estimate the size of a health problem in a population and document the distribution and spread of disease is critical to improving their effectiveness and their relevance [5] . Significant challenges in evaluating surveillance systems include identifying the benefits of surveillance and response systems and the costs involved with conducting systematic evaluations [1, 6] . To address these challenges, assessing the non-monetary costs and benefits of surveillance holds promise for generating valuable insights for decision-makers [7] . Despite their impact on system performance, non-monetary costs are rarely considered in evaluations, revealing a fundamental gap in the literature that requires attention to move the assessment of surveillance systems forward [6] .

Additional research is necessary to understand the views of stakeholders responsible for the design and implementation of disease surveillance systems, including the strategies, strengths, weaknesses, and recommendations for improving surveillance efforts [7, 8] . Stakeholders play an active role in the definition and analysis of problems encountered during the implementation of surveillance activities. This expertise can be leveraged for finding solutions to these problems [8] . This evaluation assessed the Integrated Community Surveillance (ViCo) system in Guatemala, where emerging infectious diseases continue to be a significant burden [9] , from the perspectives of stakeholders. We explored the multi-level factors and processes that shaped the implementation of the system. To evaluate the non-monetary benefits of the surveillance system, realistic and context-adapted recommendations were obtained from stakeholders to enhance the system's acceptability and guide future iterations of ViCo [5, 10] . We expect that results will improve future surveillance in Guatemala while also adding to the larger global discussion on how to best harness the benefits of surveillance activities.

To better understand and characterize the burden of diarrheal, respiratory and febrile illnesses in Guatemala, a sentinel surveillance system known as Vigilancia Integrada Comunitaria (ViCo), or the integrated community surveillance, was established in 2007. The Universidad del Valle de Guatemala (UVG), in collaboration with the Guatemala Ministry of Public Health and Social Assistance (MSPAS) and the U.S. Centers for Disease Control and Prevention (CDC), designed and implemented the facility-based syndromic surveillance system to generate high-quality data among a network of sentinel reporting sites, document the distribution and spread of disease, evaluate control strategies and interventions, identify research needs, facilitate national planning for surveillance activities, and lay the foundations for early warning of outbreaks and emerging infections.

Surveillance activities were carried out in the departments of Santa Rosa (population 320,000), located 50 km southeast of Guatemala City, and Quetzaltenango (population 705,000), located 120 km northwest of the capital, across three levels of health services (i.e., hospital, health center, and health post) from 2007-2018 [9, 11, 12] . Briefly, project-hired surveillance nurses reviewed registers and emergency department logs to identify and admit patients with acute infectious disease symptoms (e.g., diarrheal, respiratory, febrile). Clinical, demographic, and epidemiological data were obtained from face-to-face interviews with patients who met the case definition for the illnesses under surveillance and from chart reviews.

Depending on the syndrome, biological specimens were collected and tested for a variety of emerging pathogens of interest including but not limited to dengue, norovirus, rotavirus, Campylobacter, Salmonella, Cyclospora, influenza, respiratory syncytial virus, Streptococcus pneumoniae [11, [13] [14] [15] [16] [17] [18] . Linked epidemiologic and laboratory data were shared with MSPAS and CDC on a weekly basis. The data collected were used to better understand the distribution and trends of the different syndromes under surveillance and reviewed to ascertain potential changes in occurrence or in populations being affected. Reports were generated and periodically shared among stakeholders, including the health facility and hospital epidemiology staff, and MSPAS officials. In addition to sentinel surveillance, ViCo was also used as a foundation for smaller-scale research studies which tapped into existing surveillance activities to evaluate interventions and diagnostics [19] [20] [21] .

A logic model for ViCo (Additional file 1) was developed during the formative stages of ViCo to highlight the objectives and outcomes of the surveillance system inputs and activities. To assess the applicability of the logic model to the specific research objectives of ViCo, CDC's Updated Guidelines for Evaluating Public Health Surveillance Systems were used as a framework to guide the development of the interview topic guide, as well as analysis of the data collected [1] . The updated guidelines included topic areas related to the integration of surveillance and health information systems, the establishment of data standards, the exchange of health data, and changes in public health surveillance objectives to facilitate the response to emerging health threats.

Stakeholders from CDC, UVG, and MSPAS who had worked in Guatemala and the United States during ViCo were selected purposively to reflect different organizational structures and levels, scientific agendas, degrees of research activity, and historical engagement across ViCo's 10-year operation. Our target sample goal was 40 participants, which is consistent with qualitative research with heterogeneous samples to achieve data saturation [22] . A comprehensive list was compiled by CDC staff with the following inclusion criteria: current and past CDC, UVG, or MPSAS staff; considered a key informant with first-hand knowledge about ViCo and the community; direct involvement with and influence in the development of ViCo (e.g., protocol development, management/ supervision, implementation); and worked directly with ViCo at any given point in time from 2007-2018. Those without current contact information were excluded from the list. Our sample consisted of the ViCo system users from across the program's implementation period and included central, district and health-facility level MSPAS staff, university and field staff from UVG, and CDC staff from headquarters in Atlanta and in the Guatemala country office location. Stakeholders were instrumental in the design and management of the surveillance system at varying time periods and could offer a comprehensive view of the successes and challenges of implementing ViCo.

A semi-structured interview guide was developed based on prior research on the barriers and facilitators to implementing community-based surveillance systems and the CDC Updated Guidelines for Evaluating Public Health Surveillance Systems [1] . The interview guide was designed with an open framework to guide the interview, while allowing for a conversational format (Additional file 2). Interviews explored stakeholder perspectives of the surveillance system's objectives, the evolution of ViCo, the usefulness of ViCo, quality of data outputs, the impact of ViCo, and recommendations for improving ViCo. In-person or phone interviews were conducted with Spanish-and English-speaking stakeholders involved in the design and operationalization of ViCo; all were carried out by the second author (MFN), who was not involved in the design or the implementation of ViCo. To communicate their knowledge of ViCo and its major components, key informants were asked to visually represent their understanding of the ViCo surveillance system, including data pathways, from initial data collection to data dissemination. Notes were taken, and conversations recorded, with permission, using a hand-held digital recorder. Each recording and associated materials (i.e., participant drawings and interview notes) were assigned a study identification number. Written informed consent was received from each stakeholder interviewed. This activity was reviewed by CDC and was conducted consistent with applicable federal law and CDC policy. 1 This evaluation was nested as an essential activity within the ViCo surveillance system study protocol which received IRB approval from CDC (#5150), UVG (#005-04-2008), and MSAPS (#18-2014).

Notes and recordings from in-depth interviews were transcribed verbatim and translated from Spanish to English by the fifth and sixth authors (MR and MDRP). Interview transcripts were reviewed and anonymized, and any names or other individually identifying references were removed. Data were analyzed using framework analysis to ensure the validity, reliability, and replicability of qualitative research [23, 24] . The first author (JJ) and coinvestigators (MP and TL) developed the initial analytical framework containing broad preliminary codes to define concepts (e.g., trends, evaluation, dissemination), which was supplemented by an inductive approach to identify themes emerging from the data, based on a Grounded Theory approach [23, 25] . For each construct that emerged from the data, the construct's strength and consistency with the conceptual framework were explored by documenting the frequency, importance, and representativeness. The framework was subsequently refined through a reflexive and iterative process. The first and second authors (JJ and MFN) independently coded two transcripts, after which divergent coding was discussed, codes revised, and new codes added to the codebook. A subset of 10 interviews (with equal representation from CDC, UVG, and MSPAS stakeholder groups) were double coded by JJ and MFN, and inconsistencies were discussed among the coders to ensure coding reliability and consistency and add rigor to the coding process [26, 27] . JJ independently completed the final coding of all transcripts. ATLAS.ti qualitative software version 8 was used to manage and organize the interview transcripts [24] . Quotes from respondents are included in the text to highlight findings.

From a total of forty-one interview invitations sent to stakeholders, thirty-nine (95%) in-depth interviews with key informants were conducted across three different stakeholder groups and at different levels (e.g., central, district, health facility) [UVG (n = 17), MSPAS (n = 12), and CDC (n = 10)] before it was assessed that saturation was reached (Table 1) . Each interview lasted from 30 min to two hours. Participants represented epidemiologists (n = 14), directors (n = 11), laboratorians (n = 7), project nurses (n = 5), and informatics (n = 2) who were involved in the design and implementation of ViCo. Most participants were female (n = 24, 62%) and from Guatemala City (n = 24, 59%). Two stakeholders could not be reached via email or phone for an interview request.

Across interviews, key themes were organized to demonstrate the: (1) Size and Complexity of ViCo, (2) Stakeholder Expectations About the Objectives of ViCo, (3) Data Management and Structure of the Information System, (4) Local Control of Data, (5) Integration of ViCo within MSPAS, and (6) Improvement of the Operational and Design Aspects of ViCo across System, Process, and Output levels. (Fig. 1 ).

When asked to draw a flowchart of ViCo processes, all stakeholders expressed a basic knowledge of the purpose and objectives of the surveillance system. Stakeholders from UVG gave the most "systems-focused" description, focusing on the organization and functionality, its major components (e.g., population, time period, data sources, what/how data were collected, policies and procedures regarding data management and privacy) and strengths including data management, dissemination, and analysis ( Fig. 2 ). Stakeholders explained that weaknesses in the design and structure of ViCo were related to its complexity, often referencing the large amount of data elements collected for the priority diseases. The amount of epidemiologic, clinical and laboratory data collected under ViCo contributed to its size, the personnel needed, and the funding required to sustain it.

VICO could have been much simpler. In the end we created a rather complex system that worked but did need sophisticated personnel or methods to be able to work, then replicating VICO or inheriting VICO by the Ministry of Health is quite a task ... It is quite expensive because it would mean transferring a lot of technology and knowledge and the Ministry believes that it could not sustain it, so another lesson..

Respondents in all stakeholder groups felt that the design of the patient interview questionnaire could be simplified as it included many questions that, in their perspective, were never completely used. 

According to participants from UVG, a key advantage that emerged from ViCo was the generation of a robust database that helped generate research publications and reports on both a national and international scale.

[ViCo] databases have been used to generate publications. And… the data themselves, for the use of other people in public health. P27, UVG Director, Guatemala

However, as the surveillance system evolved over the years, stakeholders alluded to changing expectations and objectives of ViCo. Respondents from MSPAS expressed their thoughts and concerns about whether the data ViCo was generating were used for research and surveillance purposes or to diagnose and treat patients. CDC stakeholders observed that the addition of ancillary studies contributed to a sense that ViCo's original objectives had changed and that the working parts of the system were becoming increasingly difficult to manage. While respondents perceived the integration of shortterm research studies to generate useful data, it was unclear whether it was necessary to continue collecting this data as part of the core surveillance activities, even after smaller scale studies concluded. ViCo used an approach in which patients were assigned to one of three priority syndromes (i.e., febrile, diarrheal, and respiratory). Stakeholders from UVG reported that the syndromic case definitions changed due to the addition of other research studies. These changes made it difficult to generate compatible and comparable data across the three priority syndromes and over time.

As time goes by, although we try to keep 

ViCo was praised for its ability to support MSPAS, particularly the Epidemiology Unit, to detect and collect high-quality data on diseases of public health importance, and their causative organisms. Participants from the MSPAS mentioned that ViCo was considered most valuable in identifying the 2009 H1N1 influenza outbreak, guiding decision-making on the influenza immunization schedule and also in assessing measuring the occurrence of rotavirus post-vaccination.

More than anything else [ViCo] Despite strengths in high-quality data collection, stakeholders expressed a sense of "information overload" when describing the database and described a learning curve for the end-user given the platform that was used to house the data.

Everything is there… every single case… the methods that were used, the branching pattern, everything exists [laughs] in this program that people in general don't use...it's just it's very dense to sift through it. P20, UVG Epidemiology, Guatemala There were far too many variables being collected… and made the database cumbersome… ViCo was collecting close to 3,000 individual data points per enrollee, which is excessive and then you run into the second issue of trying to maintain the quality of that data. P17, CDC Guatemala Country Office Stakeholders from UVG mentioned the need to troubleshoot technology to collect, clean, integrate, and obtain the necessary high-quality data within a reasonable time frame. Throughout the data collection phase, nurses reported on the challenges faced when collecting data. Data managers reported challenges in correcting data errors, missing information, and inconsistencies from the database. As time passed, and as leadership shifted across all levels, the consistency in communication, and the approach were affected.

Even though the data was shared, it was not as so... frequent… there was a period of time when data was no longer shared because apparently, they were doing a restructuring of all emails because the data they sent was to people who were no longer within the Ministry... in the Ministry, we had a lot of rotation, two or three years that many people rotated… so it was difficult to maintain constant communication.

According to CDC stakeholders, ViCo was credited for signaling community trends in the epidemiology of shigellosis, assessing infection prevention effectiveness in hospitals, and evaluating control measures related to nosocomial diseases. The development of interventions for addressing antimicrobial resistance and antibiotic use in the population were also attributed to data from ViCo.

When asked about using ViCo data for public health decision making, there was agreement that additional steps could have been taken to expand its utility especially at the local level.

Local-level stakeholders from MSPAS pointed to an inadequate feedback mechanism from the project to the healthcare providers as the main challenge for leveraging data for public health action and providing immediate analysis to those charged with investigation and followup of potential outbreaks. Although the information was routinely shared among the higher levels in MSPAS, CDC, and the UVG, communication with the local level was perceived to be fragmented by stakeholders who reported that care providers external to ViCo could not access nor use existing health data in real-time to impact patient care.

We fell short... It would be very important that the information to be presented at some point, but in a general way… and give it more impact by delivering it to all the doctors and then we the doctors could say, "these are the recommendations that I would have, " right? To be able to replicate the knowledge.

As well as mentioning limitations in human resources, the time needed to analyze data, and huge workloads as potential reasons for lower data utilization rates, respondents also identified technical constraints to data use for decision making. They agreed that data analysis was a technological capability that required specialized skills.

You need not just anyone... even though the data is there, it is needed as a technical part to be able to analyze it, and that is not learned overnight… Maybe that's one of the reasons why it wasn't used as much… but rather that they used the reports as managers to see… how many [cases] were from such syndromes... P3, UVG Informatics, Guatemala [It has] a lot to do with the capacity of the ministry to analyze data, so most of the ministry had not really gone through epidemiology programs or had a lot of opportunity to analyze data. They were used to looking at reports, but not… to really play with data.

Across all stakeholder groups, ViCo was perceived to be a valuable research platform that developed surveillance and laboratory capacity in Guatemala, nationally and at the health facility level. Another notable positive observation among stakeholders was the training provided to a range of staff within MSPAS in outbreak investigation and response, surveillance data collection, and research dissemination.

Aside from the strengthening local capacity, stakeholders described ViCo as resource-intensive, expensive, and time-consuming and questioned whether it could be replicated and sustained by MSPAS. Given the human resource and financial limitations within MSPAS, stakeholders acknowledged ViCo as a project necessarily supported by external project-hired staff and funding. These limitations impacted the sense of initiative and ownership felt by MSPAS staff.

ViCo had personnel exclusively for epidemiological surveillance. They were dedicated exclusively to that... here, we have staff who are dedicated to three or four different things… we don't have the digital part that is much easier… like having an electronic survey, which can be emptied in a database. P32, UVG, Epidemiology, Guatemala Among stakeholders, ViCo was considered a parallel, independent system designed to feed data to CDC, MSPAS, and health centers but was not integrated within the national health information system. Disparate data collection systems, database interfaces, and data formats complicated the consolidation of data across systems, which according to stakeholders from MSPAS, produced inconsistencies and made it difficult for staff to analyze data.

There was coordination with the National Epidemiology Center, with the health area and the services, and, weekly, we received the feedback from the database… but, it was a platform that was run by [UVG] . It w'sn't a database that was run by the Ministry of Health… As an epidemiologist in the health area, I could not include the ViCo information in SIGSA, the Health Management Information System... In other words…we always have a parallel information system. Never integrated. P34, MSPAS, District Director, Guatemala At some point, the surveillance was completely in the hands of ViCo, and… did not enter our surveil-lance… it was important to coordinate more to have the information, both ViCo and the project, and the Ministry of Health. P11, MSPAS Central Epi, Guatemala Staff turnover and limited resources were also mentioned as major obstacles that limited the incorporation of ViCo into the national system. Although ViCo provided the opportunity to transfer capacities to MSPAS, this exchange was perceived by stakeholders as neither consistent nor sustainable.

Understanding how the Ministry works… there is a lot of personnel turnover and little availability of resources. So, almost all activities that are done that generate new capacities, or improve capacities in the Ministry, are temporary due to the very nature of the Minis'ry's operation. P28, UVG Epidemiology, Guatemala

Stakeholders provided recommendations for strengthening future surveillance projects in Guatemala (Table 2) across system, process, and output levels.

A clear recommendation was to focus the surveillance system's design on the needs of MSPAS by reviewing and ensuring case definitions, laboratory turn-around time, data elements, survey tools, and data platforms align with Ministry systems, capacities, and expectations.

One of the main recommendations would be to simplify the system. One way to do that is to better direct the collection of information, so it's done to collect only the information that's essential. P28, UVG Epi, Guatemala

Suggestions on the representativeness of the geographical area and pathogens vis-à-vis the objectives of surveillance were provided to ensure data are representative of a catchment population to facilitate generalization where this is a priority.

give us… greater representativeness. My country is a country with a lot of geographic diversity, so I consider that having other ViCo points could perhaps extend to other pathologies to investigate. P8, MSPAS Central Director, Guatemala

According to stakeholders, a strong feedback mechanism could support the continued development of surveillance system management and organizational structure while providing iterative evaluation on data sharing and customization based on surveillance needs. Increased interaction could be nurtured by having more dedicated staff members who hold a stake in the overall system and its direction. 

Additional collaboration with national authorities (preferably from the formative stages of the project) to concretely institutionalize components of the surveillance system within the MSPAS, considering financial and human resource constraints was also endorsed by respondents. 

Working with MSPAS staff to integrate data within the national health information system, either through addons or modifications to the platform, or transfer the database and server to the ministry website could be beneficial according to stakeholders.

Make sure the questions you're asking and the information you're gathering… is collected and used in a timely manner… [and] well integrated into the ministry from the start. P17, CDC Guatemala Country Office

Participants across all stakeholder groups recommended additional training opportunities for data analysis, interpretation, and visualization with the 

Dissemination of information at the local level to raise awareness of surveillance system findings was encouraged by stakeholders. The promotion of a more participatory approach with a broad range of participants, including those actively responsible for data collection, could generate trust and a stronger sense of ownership.

This study provides qualitative examples of stakeholder perspectives about the usefulness of a decade-long surveillance project in Guatemala, addresses challenges to its implementation and recommends ways forward. We summarized views, assumptions, expectations, and feelings regarding ViCo's impact. While all stakeholders acknowledged multiple positive aspects of the project, perspectives varied depending on participant role, knowledge, engagement with the surveillance system, and the length of their involvement. Emergent themes included key strengths related to the detection and collection of high-quality data on various diseases and their etiologies; key challenges related to the size and complexity of ViCo, data management, and limited use of data for public health decision-making. Recommendations focused on the need for simplification, training on data utilization and interpretation, and integration with national systems.

This evaluation highlights some of the potential tensions between routine (i.e., ministry of health-run) and enhanced (i.e., external non-host government funding) surveillance platforms [28] . Enhanced surveillance efforts, such as ViCo, bring additional staff, diagnostic resources, and more detailed design criteria than routine surveillance efforts. More extensive diagnostics are performed in the hope that understanding the detailed causes of illnesses in the area under enhanced surveillance will shed light on what may be occurring in other areas where enhanced diagnostics are unavailable. Over time, stakeholders should revisit the project's fundamental principles when considering making changes to the surveillance system or adding attractive ancillary studies.

In an assessment to strengthen Ethiopia's maternal death surveillance and response system, researchers suggested adequate supervisory support from the start to ensure the system became embedded within the health system as a routine practice rather than perceived as a stand-alone activity [29] . Our evaluation generated similar results, as evidenced by stakeholder recommendations to focus additional efforts on integration with MSPAS systems. When enhanced surveillance expertise is housed outside a vertically organized surveillance system, stakeholder roles, responsibilities, and expectations may become fragmented and goals understood differently [29, 30] .

Our results emphasize that all details can be crucial to how a system is understood and utilized-including progress towards accepted data standards and guidelines, particularly for how data are cataloged, and case definitions used over time to ensure consistency. As echoed by participants from the health facilities that expressed challenges to the timely access and use of data at the local level, a feedback structure would encourage more dialogue and opportunities for improving the system to meet the surveillance needs of local facilities and the Ministry of Health. This same mechanism could also help define a clear transition strategy for sustaining enhanced surveillance functions once program goals are met. A qualitative assessment of data management and reporting systems in Botswana found that it was essential to build feedback loops into the system, not only through epidemiological bulletins, but also by defining and determining appropriate actions for the investigation and control of outbreaks and measuring progress towards surveillance targets [31] . These strategies would focus on reducing parallel structures, long-term external dependency, and strengthening the existing system, while ensuring that the immediate successes and long-term impacts of enhanced surveillance are not jeopardized [30, 32, 33] .

Restricting enhanced surveillance systems to only methods and materials manageable with local resources will not solve the problems of inadequate identification of the causes of illness and death [34] . Instead, there needs to be an a priori plan to extend learning from enhanced surveillance to the broader system and facilitate inclusive analysis of enhanced surveillance data [31, [35] [36] [37] [38] [39] [40] [41] . Our findings advocate that transferring knowledge and equipment is not enough to strengthen health systems. In our evaluation, data sharing was particularly impacted by staff turnover and rotations across the facility and national levels. Continuous staff rotation impacted the sense of morale and ownership felt among stakeholders. A One-Health evaluation of the Southern African Centre for Infectious Disease Surveillance found similar operational challenges, citing a lack of resources attributed to information and data sharing, and institutional memory [35] .

Further steps to integrate supervision, mentoring and collaboration within the project aims to assure that the expertise generated extends to ministry staff and health workers external to the project are required. Surveillance systems do not serve their purpose when data are not analyzed and interpreted on a consistent basis [42] . As expressed by stakeholders in our study, data utility, data visualization, and data analysis were crucial components that were necessary for an effective disease surveillance system, as it guided decision-making and helped identify and contain outbreaks. Staff views about the quality of a communicable disease surveillance system in Sudan stressed the importance of simplicity in the structure and design of a surveillance system for generating data that can inform decision-making [43] . Similarly, in a study on data-informed decisions in high HIV-prevalence settings in South Africa, inefficient use of information was attributed to the organizational culture and limited capacity of program and facility managers to analyze, interpret, and use information -highlighting the training needs for surveillance [44] . To address this gap, researchers and government leaders tasked with managing surveillance systems have focused on developing local infrastructure to build more effective surveillance and response systems with local motivation and centralized support [36, 41, 45, 46] . Digital health interventions can also be leveraged to facilitate training, supervision, communication, and feature components of performance feedback to promote further engagement of ministry staff in routine data utilization and evaluation [47, 48] . Future surveillance projects will benefit from a comprehensive assessment of skill gaps and identification of opportunities to foster professional growth locally as part of the partnership with local and national staff. Further evaluation is warranted to determine the most effective continuing education approaches to strengthen the surveillance public health workforce.

This study has several limitations. One limitation is the potential for recall and response bias particularly from respondents who were involved with ViCo in the distant past. The information shared by stakeholders was based on personal experiences and subjective impressions. To minimize social desirability bias (a tendency to present reality to align with what is perceived to be socially acceptable) a consistent interviewer was used who was external to ViCo and who could build rapport with a broad base of respondents across agencies and positions to elicit more honest responses about challenges and gaps. Most participants interviewed had moved on to other positions and had little incentive to steer the project's future direction. These interviewees were not currently involved with ViCo so their opinions may not reflect the current use of ViCo. Another limitation was that we were unable to collect quantitative indicators such as trends in funding for health, diarrheal diseases, respiratory diseases, febrile illnesses, time and use of staff on ViCo and response/control activities, and trends in the number of samples processed. Inclusion of such variables would have provided an additional sense of the monetary investment in the surveillance system and added complementary information to our theme of integration within MPSAS. This assessment nonetheless adds to a growing body of literature on surveillance system evaluations to identify areas for improvement, promote policy changes, and inform future interventions.

This evaluation has identified system-wide, process, and output challenges related to the delivery, capacity, and management of a surveillance system implemented as a collaborative research effort. These lessons learned may help focus more effective system design for ViCo and similar surveillance projects. To promote surveillance system development, it is essential that data collection is efficient, and provides timely information for public health decision-making. Training in applied epidemiology and data analysis must be an important component of a surveillance system to have capable and empowered personnel to use data, and data use should be a primary evaluation component. Ongoing evaluation of surveillance will be essential to ensure activities meet countries' needs and generate the data needed to plan and implement effective public health policies and strategies not only in Guatemala but in other countries as well. Future assessments should consider economic evaluations to support the non-monetary findings in this study, critical for policymakers who need to make decisions based on limited funding streams.

The findings and conclusions in this report are those of the author(s) and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

Updated guidelines for evaluating public health surveillance systems; recommendations from the Guidelines Working Group

Strengthening Global Public Health Surveillance through Data and Benefit Sharing

Centers for Disease Control and Prevention. Global health surveillance MMWR Suppl

a concurrent embedded mixed quantitative/qualitative facility-based cross-sectional study

Applying participatory approaches in the evaluation of surveillance systems: A pilot study on African swine fever surveillance in Corsica

Applying participatory approaches in the evaluation of surveillance systems: A pilot study on African swine fever surveillance in Corsica

Surveillance systems evaluation: a systematic review of the existing approaches

The Added-Value of Using Participatory Approaches to Assess the Acceptability of Surveillance Systems: The Case of Bovine Tuberculosis in Belgium

Factors associated with fatal cases of acute respiratory infection (ARI) among hospitalized patients in Guatemala

Qualitative Methods in Health Policy and Systems Research: A Framework for Study Planning

The epidemiology and clinical characteristics of young children hospitalized with respiratory syncytial virus infections in Guatemala

Surveillance for Hospitalized Acute Respiratory Infection in Guatemala

Burden of laboratory-confirmed Campylobacter infections in Guatemala 2008-2012: results from a facility-based surveillance system

Prevalence and genetic diversity of viral gastroenteritis viruses in children younger than 5 years of age in Guatemala

Prevalence and Genetic Diversity of Norovirus Among Patients With Acute Diarrhea in Guatemala

Effectiveness of Monovalent and Pentavalent Rotavirus Vaccines in Guatemala

Full genomic characterization and phylogenetic analysis of a zoonotic human G8P[14] rotavirus strain detected in a sample from Guatemala

Inverse association between dengue, chikungunya, and Zika virus infection and indicators of household air pollution in Santa Rosa, Guatemala: A case-control study

Diagnostic performance of rectal swab versus bulk stool specimens for the detection of rotavirus and norovirus: implications for outbreak investigations

• fast, convenient online submission • thorough peer review by experienced researchers in your field • rapid publication on acceptance • support for research data, including large and complex data types • gold Open Access which fosters wider collaboration and increased citations maximum visibility for your research: over 100M website views per year submit your research ? Choose BMC

An evaluation of water, sanitation, and hygiene status and household assets and their associations with soil-transmitted helminthiasis and reported diarrhea in Nueva Santa Rosa

Characterising and justifying sample size sufficiency in interview-based studies: systematic analysis of qualitative health research over a 15-year period

Using the framework method for the analysis of qualitative data in multi-disciplinary health research

The Practice of Qualitative Research: Engaging Students in the Research Process

Discovery of Grounded Theory: Strategies for Qualitative Research

Benefits to Qualitative Data Quality with Multiple Coders: Two Case Studies in Multi-coder Data Analysis

Measuring the Typicality of Text: Using Multiple Coders for More Than Just Reliability and Validity Checks

Health and Demographic Surveillance System Sites: Reflections on Global Health Research Ethics

We identify, discuss, act and promise to prevent similar deaths': a qualitative study of Ethiopia's Maternal Death Surveillance and Response system

Lessons learned from the polio eradication initiative in the Democratic Republic of Congo and Ethiopia: analysis of implementation barriers and strategies

Improving the quality of health information: a qualitative assessment of data management and reporting systems in Botswana. Health Research Policy and Systems

Effect of changing case definitions for COVID-19 on the epidemic curve and transmission parameters in mainland China: a modelling study

Striking a Balance between Program-Specific and Portfolio-Level Evaluation: Lessons Learned from a Multi-Site Evaluation on the Texas-Mexico Border

Building scientific capacity in developing countries

A One Health Evaluation of the Southern African Centre for Infectious Disease Surveillance

Trust and the ethical challenges in the use of whole genome sequencing for tuberculosis surveillance: a qualitative study of stakeholder perspectives

Progress, challenges and ways forward supporting cancer surveillance in Latin America

Perceived usefulness of a distributed community-based syndromic surveillance system: a pilot qualitative evaluation study

Staff views about the quality of the communicable diseases surveillance system in Khartoum state

Health information generation and utilization for informed decision-making in equitable health service management: The case of Kenya Partnership for Health program

Data for action: Collecting and using local data to more effectively fight tuberculosis

Suicide surveillance and health systems in Nepal: a qualitative and social network analysis

Staff views about the quality of the Communicable Diseases Surveillance System in Khartoum State

Perceptions about data-informed decisions: an assessment of information-use in high HIVprevalence settings in South Africa

Improving access for Latino immigrants: evaluation of language training adapted to the needs of health professionals

Examining state health agency epidemiologists and their training needs

Digital Technologies for Health Workforce Development in Low-and Middle-Income Countries: A Scoping Review

Can Public Health Workforce Competency and Capacity be built through an Agent-based Online, Personalized Intelligent Tutoring System?

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations

We gratefully acknowledge the key informants who participated in our study and provided invaluable insight and recommendations. We would like to especially acknowledge all staff from CDC, MSPAS and UVG who contributed to the development of ViCo in Guatemala.

The online version contains supplementary material available at https:// doi. org/ 10. 1186/ s12889-022-12719-7.Additional file 1.

Authors' contributions