key: cord-0027074-nxcgo4lz
authors: Barchielli, Chiara; Rafferty, Anne Marie; Vainieri, Milena
title: Integrating Key Nursing Measures into a Comprehensive Healthcare Performance Management System: A Tuscan Experience
date: 2022-01-26
journal: Int J Environ Res Public Health
DOI: 10.3390/ijerph19031373
sha: 2b33ef84941adc2afeb3ffeff1628d5081a69ccb
doc_id: 27074
cord_uid: nxcgo4lz

This paper addresses the evaluation of nursing quality and safety beyond nursing tasks in specific healthcare settings and sets it in a context that conveys the sense of complexity and multifaceted nature of the contribution that nursing makes to the whole system. The paper describes research conducted in Tuscany during 2019 involving regional managers and heads of nursing departments. This research has led to the development of an integrated evaluation framework through focus groups and consensus process with the latter, which includes Performance Organizational climate data, Patient-Reported Experience Measures (PREMs), and Patient-Reported Outcome Measures (PROMs). This integrated framework aims at both making sense of extant measures as key performance indicators shared among different professionals while recognizing the important role of nursing care by adding specific measures and can be seen as a tool that boosts the sense of “teamness” in healthcare.

Since 2000, New Public Management reforms have ushered in a variety of changes and innovations into different parts of the public sector [1, 2] including healthcare, propelling the development of multidimensional performance management systems [3] . Given the complexity of healthcare, performance evaluation tools have been designed to measure and monitor different dimensions, such as (i) health outcomes, (ii) health service access, (iii) efficiency, and (iv) service quality and appropriateness [4, 5] . Motivations for measurement and assessment in the public sector are several and extensively reported in the literature. For instance, Behn [6] proposed eight reasons for measuring performance: evaluating, controlling, budgeting, motivating, promoting, celebrating, learning, and improving. When measures are used to evaluate, they become elements that help managers and policymakers to make decisions. Indeed, performance evaluation estimates "the quality of health services with the ultimate goal of improving health outcomes" [7] . This is particularly relevant when the performance evaluation system is based on solid measures from administrative and survey data [8] that can help select strategies to continuously improve care and accountability towards citizens [9, 10] . Although many scholars uphold the importance of designing and implementing a performance evaluation system at different governance levels [8] [9] [10] [11] , there are several factors to be considered (different organizational models, financing mechanisms, governance and resources in service provision [11] , stakeholders 'perspectives, uncertainty, and organizational fragmentation [12] ), all of which make performance evaluation in the healthcare field particularly complex. To cope with this complexity, performance evaluation systems in healthcare are characterized by multiple dimensions; Hence, the impetus for integrating nursing in a broader framework on healthcare quality and safety came from the need to make the system aware of the role played by nursing into the whole healthcare performance but also to make nurses aware of the fact that they are an integral part of the system, where their effectiveness and accountability can be enhanced.

This paper presents a performance model where nursing quality and security (Q&S) indicators are integrated into an existing framework on healthcare system performance evaluation to demonstrate the multifaceted nature of the nursing contribution to the whole healthcare system and to give visibility to nursing management. The Performance Evaluation System (PES) framework was developed in an Italian Regional healthcare system (RHS) in the early 2000s [16, 31] and benchmarks all the organizations that take part in it. The integration of nursing indicators is instrumental in two ways: (1) making nurses accountable in the eyes of the system and (2) make the latter aware of how their work can improve Q&S through feedback from the system-heightened awareness as its main engine of improvement, as shown in Figure 1 . has been supported through the healthcare strategies until financed by the "next-generation UE" funds. In this evolving context, it is more urgent than ever to give nursing's impact on the health system strength and visibility. Hence, the impetus for integrating nursing in a broader framework on healthcare quality and safety came from the need to make the system aware of the role played by nursing into the whole healthcare performance but also to make nurses aware of the fact that they are an integral part of the system, where their effectiveness and accountability can be enhanced. This paper presents a performance model where nursing quality and security (Q&S) indicators are integrated into an existing framework on healthcare system performance evaluation to demonstrate the multifaceted nature of the nursing contribution to the whole healthcare system and to give visibility to nursing management. The Performance Evaluation System (PES) framework was developed in an Italian Regional healthcare system (RHS) in the early 2000s [16, 31] and benchmarks all the organizations that take part in it. The integration of nursing indicators is instrumental in two ways: (1) making nurses accountable in the eyes of the system and (2) make the latter aware of how their work can improve Q&S through feedback from the system-heightened awareness as its main engine of improvement, as shown in Figure 1 . The paper describes the constructive research, an approach used to define and solve problems [32] carried out in Tuscany to include NSI into PES and to make sense of the PES for nurses.

Research in health management has the goal of building a theory, a solution, a tool to address a real need and improve an existing system in terms of design and or performance [31, 32] . Our study aimed to build a performance evaluation system that would be useful to nurses and capable of integration into an existing system based on routine administrative and survey data. The Tuscan RHS designed and implemented a multidimensional PES in 2014 [31] . To date, PES contains over 700 indicators [1, 33] and has spread to other Italian Regions and their RHSs. Through a benchmarking process, it shed evidence on crucial dimensions like population's health status, health assessment, ability of the healthcare organizations to follow set regional health strategies, evaluation of the experience of users and employees, and operational efficiency [16] . Nursing was not involved The paper describes the constructive research, an approach used to define and solve problems [32] carried out in Tuscany to include NSI into PES and to make sense of the PES for nurses.

Research in health management has the goal of building a theory, a solution, a tool to address a real need and improve an existing system in terms of design and or performance [31, 32] . Our study aimed to build a performance evaluation system that would be useful to nurses and capable of integration into an existing system based on routine administrative and survey data. The Tuscan RHS designed and implemented a multidimensional PES in 2014 [31] . To date, PES contains over 700 indicators [1, 33] and has spread to other Italian Regions and their RHSs. Through a benchmarking process, it shed evidence on crucial dimensions like population's health status, health assessment, ability of the healthcare organizations to follow set regional health strategies, evaluation of the experience of users and employees, and operational efficiency [16] . Nursing was not involved in its original design as a stand-alone profession, and our proposal of an adapted PES version that includes nursing performance measures has a dual intent: (a) make nurses aware of their impact on the system, empowering them to seek improvement and being accountable at a system level; (b) make the system aware of the nursing role in achieving better outcomes.

First, we surveyed the existing literature on nursing quality and safety systems to inform our subsequent focus group with Directors of Nursing Departments in Tuscany that were formally and directly recruited explaining the research relevance of the project. Appendix A reports the information according to the consolidated criteria for reporting qualitative research (COREQ).

Due to the estimated impact of the significance of finding more accurate ways to measure nursing quality and safety measures beyond the boundaries of the profession itself to the advantage of a more systemic estimation, all the identified components of the group accepted to be part of the focus group. It lasted about 3 h with the eight Directors of Nursing Departments-all of them experienced Registered Nurses and with seniority of no less than 30 years in the field of the Tuscan RHS, and it was dual-moderated by two of the authors: a Nurse (CB) and an Associate Professor (MV), both with specific knowledge in the fields of nursing, performance evaluation, and management. It was conducted in a formal RHS location, and the participants were introduced to the open discussion by the presentation of compliance guidelines to it. The used interview guide was developed based on an extending study of the existing literature on nursing quality and safety indicators. The overall recorded process led to the identification of the key measures they use in their organizations and found that all the measures identified were comprised in two systems: CaLNOC (Collaborative Alliance for Nursing Outcomes) and NDNQI ® (National Database of Nursing Quality Indicators) as more complete, comprehensive, and useful. A description of these two instruments, as well as the PES, are reported in Appendix B to this paper.

Nursing indicators were identified according to a series of criteria: (i) nurses and other health professionals were actively involved in the selection process and the ongoing item review, (ii) clinical trials had validated each indicator [34] [35] [36] [37] , (iii) measures linked to whole system performance measurement at a national level, and (iv) indicators that linked to organizational factors and human resources (e.g., nurse to patient ratio). For an in-depth analysis of these themes, see Appendix B. During the focus group with the Directors of Nursing Departments of the Tuscan RHS, the selected nursing indicators to be included in PES have been grouped into four categories: clinical area, regional strategies, patient voice, and human resource management, which are described below.

The second phase consisted in screening indicators included in the CaLNOC [38] and NDNQI ® [33] bundles that were already in the PES.

The final phase was the graphical representation together with the consensus received from the Directors of Nursing Departments and from regional and national experts, such as members of the Tuscany Region and the Ministry of Health.

The focus group and the consensus process generated 56 nursing indicators to highlight nursing within the RHS evaluation process more broadly. The indicators were grouped into four domains, as in Table 1: 1. Clinical domain: nurses, like other healthcare professionals, contribute to the achievement of regional goals like addressing people's needs in a safe way, providing qualified and attentive help to the customer/patient, etc. In this area, indicators such as the average stay performance index (as a proxy of hospital efficiency) were selected. The information sources are, again, ministerial and regional data, for example, the information flow on hospital discharge records, outpatient services, supplied drugs, etc.

2. Patients' voice domain: this dimension includes the indicators that derive from patients' satisfaction and experience (PREMs and PROMs). These patient perception measures have the goal of shaping the healthcare system in a patient-centered way [39, 40] by the continuous analysis of the feedback that comes from the end users of the system itself. Through these tools, it is also possible to estimate specific aspects such as the degree of humanization of care and the perceived level of collaboration between the different professionals.

3. Human resource management: this dimension includes both administrative measures like the absence rate and survey data deriving from the Organizational Climate Survey [41] administered every two years in Tuscany from which the indicators on nurses' job satisfaction, the willingness to recommend, the intention to leave, etc., are selected. In addition, the selected indicators relate to different settings, such as organizational (firm), hospital, community, and nursing home. Table 1 lists the indicators that were selected.

The PES provides the opportunity to expand understanding and action on identified organizational targets to enhance performance improvement. Introducing nursing measures in a consolidated performance measurement system, which is used by policy makers and managers, is a way to ensure that these measures are available to be used by the whole system and not only by one professional category. Moreover, disclosure of data can have positive effects: the process of public sharing activated reputational mechanisms capable of stimulating Health System improvement [42] . Such improvements could result in increasing managerial accountability [43] . On the other hand, there is possible reputational damage that managers could experience otherwise [44] [45] [46] [47] . It has a high visual impact, is easy to understand [48] , and is based on a rigorous methodology of collection, processing, and disseminating, providing robust information to track change and benchmarking over time. The indicators can be used to conduct analyses on several levels: national, regional, hospital-based, department-based, nursing home. The level of detail allows strategic, tactical, operational, nurse-specific analysis to be targeted and actions taken. The added value of this enriched framework also lies in the fact that it is co-created with (i) professionals, focusing on their performance but also considering the organizational climate in which they operate and (ii) taking account of patients, to make their point of view the litmus test of the appropriateness of the care provided. The twofold aim that we want to achieve is to highlight the role of nurses within the health system and make it objective in their eyes, thus increasing their motivation [49] ; as pointed out at the beginning of this discussion, the aim is, in fact, to motivate, promote, and improve [6] the healthcare systems is of paramount importance. The PES, not only indicators, cover the activities for which nurses are responsible, but all processes in which they participate, to consider their systemic contribution. It also becomes clear that their contribution is integrated with that of other professions, strengthening the position of the need for multi-professionality in healthcare [50] . For their part, the Directors of Nursing Departments of Tuscany that engaged in the focus group did not oppose resistance in the choice of indicators to be accountable for; this is probably because they expressed the will of the whole professional body to be more visible and recognized ad deeply rooted in the various settings of the healthcare system. If the nurses had chosen to use only and only the NSI from CaLNOC and NDNQI ® , they would have evidenced the will to be accountable in a separate way from all the other professionals. This inclusive and integrative approach was greatly appreciated by the regional and national levels who feared a breakup.

The present experience, based on the Italian context, can be an inspiration or an example for countries in which organizational models are still based on the role of the physician, as the Italian model is [51] . The pandemic highlighted the importance of Primary Health Care (PHC) [52, 53] , an approach that follows a person and his/her needs throughout their lifetime and that puts nurses in a central position [54] .

This analysis is Tuscany based and reflects the strategies and the characteristics of a single region. The aim is to replicate it in different Italian regions that use PES as an evaluation tool, eventually reaching the national level. Regarding the diffusion of the tool, consideration must be made. Although the PES has been in place for several years, it never encountered a radical, game-changing event like a pandemic, which distorted traditional programming and control processes. We expect a growing role of nursing for the effect of the necessity of a task shifting [26] and for the indispensable expansion of community cares, and we expect PES to reflect it. The ultimate aim of this tool is to offer safe and quality care to patients, and the measures that it provides are meant to inform the managers on the state of the art of the overall process of care itself. When the "awareness virtuous circle" is triggered, a sense of "teamness" moves along, as all the actors of the system can monitor the performance they all contributed to. Furthermore, nurses, physicians, managers, etc., perceive how strong their interdependence along the entire patient pathway is. Managers in particular can enhance their ability to set the way in multiple settings and immediately monitor the effects of their policies and decisions. The result is that the overall performance is enhanced [55] .

Although this investigation has benefited from multiple contributions and points of view at different levels, it still needs further considerations and confrontations with professionals from other backgrounds and places. There are still aspects to be discussed and measures to eventually be included, for example, specific metrics on interprofessional teamwork or on specific areas, such as the mental health setting. The state of advancement of this framework strictly reflects the consensus reached on it, and it is open to future changes. One of the possible future improvements is the application of quantitative methods, even very sophisticated like the vector evaluation genetical algorithm (VEGA) that are used to solve real-world optimization problems [56] in cases in which management has to pursue more than one objective and they need to consider multiple factors. 

No approval by the ethics committee was necessary to carry out this investigation, as it doesn't involve patients, patients' data, or any participants' sensitive data.

Data Availability Statement: Data concerning PES can be found at: https://performance.santanna pisa.it/pes/start/start.php (accessed on 12 December 2021).

The authors declare no conflict of interest.

Appendix A. Interviewer characteristics 8

What characteristics were reported about the interviewer/facilitator? E.g., bias, assumptions, reasons, and interests in the research topic 4

Methodological orientation and theory 9

What methodological orientation was stated to underpin the study? E.g., grounded theory, discourse analysis, ethnography, phenomenology, content analysis The National Database of Nursing Indicators (NDNQI ® ) is an American database that provides quarterly and annual reports on nursing indicators. NDNQI ® originated from the American Nursing Association (ANA) in 1998. Its main purpose is the systematic data collection of nursing quality indicators to measure nursing performance in a national context. This voluntary system compares more than 1100 USA health organizations until the unit level. It proposes traditional and operative measures (e.g., hours of nursing care per patient day, the prevalence of pressure ulcer, falls and falls with injury, nosocomial selective infection, and patient/family satisfaction with nursing care), includes staff mix measures (e.g., nurse-coworkers ratio), and it monitors policy and human resources management aspects (e.g., turnover, educational level, job satisfaction, work safety perception, nursing work index) regularly. NDNQI ® 's novelty and foresight lie on dimensions as the decision-making process, nurse-physician relational aspects quality, personnel composition adequacy, work organization based on the presence of nursing models. As Lockhart [1] points out, the measures that this system provides are important because they allow hospitals not to be self-referential, that is to say that every organization can compare itself with other organizations at a national and regional level, and the measures are able to give information down to unit level.

Collaborative Alliance for Nursing Outcomes (CaLNOC) started in 1996 in California as a no-profit initiative. CaLNOC provides performance measures on hospital quality trends allowing a start baseline for continuous benchmarking. As for the NDNQI ® , adherence to CaLNOC is voluntary. The assessed indicators measure many dimensions as skill mix, nurse to patient ratio, voluntary turnover, nurse staff characteristics (e.g., educational level, work seniority), and the level of nurse process implementation (e.g., risk assessment, PICC use). The use of this measurement has been extended to other settings beyond acute care to include ambulatory care [2] . Benchmarking is the underlying factor that encourages healthcare systems to focus on the quality and efficiency of their outcomes. CalNOC is particularly valued for its capacity to support the responsiveness and strategic value of the data to members [3] since they have the possibility to use a customizable virtual dashboard that boosts its reporting capacity through making available selectable parameters of interest (ibidem).

Appendix B.3. The Performance Evaluation System Experience (PES) and the Nursing Dimension Figure A1 represents an example of the Tuscany Region dartboard, and the blue circles show some of the indicators considered in the proposal of nursing performance measurement integration inside the PES. Indicators enriched with nursing dimension are circled in blue: percentage of absence from work, palliative care, ER, mental health, hospital-territory integration, territorial care effectiveness, obstetric care, hospitalization efficiency, territorial and residential care, and health of initiative (proactive health).

The dartboard is divided into five different bands, associated with different levels of performance: while the dark green centre indicated a high performance, the external red band indicates a very poor performance (dark green to red: excellent performance, good performance, average performance, poor performance, and very poor performance) according to the set standards [4] . The reference criteria for the positioning of performance are gathered from international standards-if existing, regional standards set out by the Regional Government or, if no measure is set, the "Regional mean" is used (standardized by all the factors that will allow a comparison hence a benchmarking among all the organizations) [5] . Figure A1 represents an example of the Tuscany Region dartboard, and the blue circles show some of the indicators considered in the proposal of nursing performance measurement integration inside the PES. Indicators enriched with nursing dimension are circled in blue: percentage of absence from work, palliative care, ER, mental health, hospital-territory integration, territorial care effectiveness, obstetric care, hospitalization efficiency, territorial and residential care, and health of initiative (proactive health).

The dartboard is divided into five different bands, associated with different levels of performance: while the dark green centre indicated a high performance, the external red band indicates a very poor performance (dark green to red: excellent performance, good performance, average performance, poor performance, and very poor performance) according to the set standards [4] . The reference criteria for the positioning of performance are gathered from international standards-if existing, regional standards set out by the Regional Government or, if no measure is set, the "Regional mean" is used (standardized by all the factors that will allow a comparison hence a benchmarking among all the organizations) [5] .

A graphic representation of a pentagram joins the dartboard, as shown in Figure A2 . A graphic representation of a pentagram joins the dartboard, as shown in Figure A2 . Color bands are horizontal, framing the different stages of care pathways, and patients' experience measures (PREMs and PROMs are continuously collected by the Observatory that is located inside the MeS Lab) are included and coexist with indicators gathered from administrative data, [6] and the notes represent performance indicators. While the dartboard refers to the provider of the service, the pentagram refers to the steps the patients take along the pathway, beyond organizational and physical boundaries crossing different care settings, and presents interactions with different subjects (different professionals) conjointly operating towards the same objective: the achieving of the best result possible for the patient [7, 8] . To date, the clinical pathways represented through the pentagram are the oncological path, the mental health path, the maternal-infant path, the chronicity path, the emergency-urgency path, the home and residential care path for the elderly non-self-sufficient patients and the orthopaedic-traumatology path. Color bands are horizontal, framing the different stages of care pathways, and patients' experience measures (PREMs and PROMs are continuously collected by the Observatory that is located inside the MeS Lab) are included and coexist with indicators gathered from administrative data, [6] and the notes represent performance indicators. While 

A Public Management for All Seasons? Public Adm

Accounting and the New Public Management: Instruments of Substantive Efficiency or a Rationalising Modernity? Financ. Account. Manag

A Performance Management System in Healthcare for All Seasons?

A performance assessment framework for hospitals: The WHO regional office for Europe PATH project

A conceptual framework for the OECD Health Care Quality Indicators Project

Why measure performance? Different purposes require different measures

Assisting allied health in performance evaluation: A systematic review

Health System Performance Assessment in the WHO European Region: Which Domains and Indicators Have Been Used by Member States for Its Measurement? In Health Evidence Network Synthesis Report; WHO Regional Office for

Analysing and assessing accountability: A conceptual framework

Health System Performance Comparison: An Agenda for Policy, Information and Research: An Agenda for Policy

The challenge of complexity in health care

Wicked problems: Implications for public policy and management

Health Systems Performance Assessment: Goals, Framework and Overview. Health System Performance Assessment: Debates, Methods and Empiricism; World Health Organization

Assessing health system performance in developing countries: A review of the literature

A Road Map for Improving the Performance of Performance Measures. Health Aff

Let's play the patients music: A new generation of performance measurement systems in healthcare

State of the Art Metrics for Nursing: A Rapid Appraisal

Nursing-sensitive indicators: A concept analysis

Using nurse-sensitive outcomes to improve clinical practice

Nusing-sensitive indicadors: An opportunity for measuring the nurse contribution

Is the availability of hospital IT applications associated with a hospital's risk adjusted incidence rate for patient safety indicators: Results from 66 Georgia hospitals

Models for Organizing the Delivery of Personal Health Services and Criteria for Evaluating Them

Performance measurement of nursing care: State of the science and the current consensus

The Evolution of Professional Nursing Culture in Italy: Metaphors and Paradoxes

Health at a Glance: Europe 2020: State of Health in the EU Cycle

Task Shifting and Health System Design: Report of the Expert Panel on Effective Ways of Investing in Health

The support needs of Australian primary health care nurses during the COVID-19 pandemic

COVID-19 Impact in Community Care-A Perspective on Older Persons with Dementia in Singapore

Available online: alute.gov.it/portale/news/p3_2_1_1_1.jsp?lingua=italia no&menu=notizie&p=null&id=4001

DPCM n. 73 del 17 Maggio 2020

Assessing the effectiveness of a performance evaluation system in the public health care sector: Some novel evidence from the Tuscany region experience

The constructive research approach in project management research

The National Database of Nursing Quality IndicatorsTM (NDNQI ®)

Outcomes of adoption: Measuring evidence uptake by individuals and organizations

Beyond nursing quality measurement: The nation's first regional nursing virtual dashboard

Reliability and validity of the NDNQI®injury falls measure

Reliability testing of the National Database of Nursing Quality Indicators pressure ulcer indicator

Evolving concepts of patient-centered care and the assessment of patient care experiences: Optimism and opposition

Rethinking healthcare performance evaluation systems towards the peoplecentredness approach: Their pathways, their experience, their evaluation

L'esercizio delle competenze manageriali e il clima interno. Il caso del Servizio Sanitario della Toscana

Reputations count: Why benchmarking performance is improving health care across the world. Health Econ

Benefits and Hazards of Reporting Medical Outcomes Publicly

Reporting health care performance: Learning from the past, prospects for the future

Does publicizing hospital performance stimulate quality improvement efforts? Health Aff

What might the English NHS learn about quality from Tuscany? Moving from financial and bureaucratic incentives towards 'social' drivers. Soc

House of Commons Health Committee Inquiry into Commissioning

Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups

Influences over Italian nurses' job satisfaction and willingness to recommend their workplace

Competencies relevant for gerontological nursing: Focus-group interviews with professionals in the nursing of older people

The global role of the doctor in healthcare

Promoting Health: The Primary Health Care Approach

What is the role of Primary Health Care in the COVID-19 pandemic? SciELO Brasil 2020

A Vision for Primary Health Care in the 21st Century: Towards Universal Health Coverage and the Sustainable Development Goals

The effects of managerial ability on firm performance and the mediating role of capital structure: Evidence from Taiwan

Optimal computing budget allocation for the vector evaluated genetic algorithm in multi-objective simulation optimization