key: cord-0028721-zvu0ygj7 authors: Almeda, Nerea; García-Alonso, Carlos Ramón; Killaspy, Helen; Gutiérrez-Colosía, Mencía R.; Salvador-Carulla, Luis title: The critical factor: The role of quality in the performance of supported accommodation services for complex mental illness in England date: 2022-03-17 journal: PLoS One DOI: 10.1371/journal.pone.0265319 sha: cf0dc499e7db2500a1763a6c38dd16367ab72cfd doc_id: 28721 cord_uid: zvu0ygj7 Rehabilitation services have a key role in ensuring integrated and comprehensive mental health (MH) care in the community for people suffering from long-term and severe mental disorders. MH-supported accommodation services aim to promote service users’ autonomy and independence. Given the complexity associated with MH-supported accommodation services in England, a comparative evaluation of critical performance indicators, including service provision and quality of care, seems to be necessary in designing evidence-informed policies. This study aims to explore the influence of service quality indicators on the performance of MH-supported accommodation services in England. The analysed sample includes supported accommodation services from 14 nationally representative local authorities in England from the QuEST study grouped by three main types of care: residential care homes (divided into two subgroups: move-on and non-move-on oriented), supported housing and floating outreach. EDeS-MH (efficient decision support-mental health) was used to assess the performance indicators for the selected services by combining a Monte Carlo simulation engine, data envelopment analysis and a fuzzy inference engine for integrating expert knowledge. Depending on the type of care, six/seven quality domains were sequentially included after a baseline scenario (only technical) was analysed. Relative technical efficiency scores for the baseline scenarios revealed high performance in all the selected supported accommodation services, but the statistical variability was high. Quality domains significantly improved performance in every type of care. The inclusion of quality indicators has a positive impact on the global performance of each type of care. Remaining at the corresponding services more than expected for two years has a negative impact on performance. These findings can be considered from a planning perspective to facilitate the design of pathways of care with more realistic expectations about gaining autonomy in two years. The Hospital Plan of 1962 in England established its position as a pioneer in the transition of psychiatric services from largely hospital-based settings to community-based mental health (MH) care [1] . Supported accommodation services are a key component of the 'theoretical whole system care pathway' that provides support in the community for people with more complex MH needs [2] . Ideally, these should be organized into a 'care pathway' with the expectation that people graduate from higher to lower levels of support over time. The QuEST (Quality and Effectiveness of Supported Tenancies for People with Mental Health Problems) study analysed supported accommodation services across England in 2014. This study identified three main types of services, residential care homes, supported housing and floating outreach, and investigated their quality and effectiveness. Residential care homes cater to those with the highest needs and include communal facilities staffed 24 hours that provide meals, medication supervision, cleaning, etc., where placements are not time-limited. Supported housing services provide shared or individual, self-contained, time-limited tenancies with staff based on-site up to 24 hours a day to help residents gain skills to move on to less supported accommodations. Finally, floating outreach services provide visiting support consisting of a few hours per week to people living in permanent, self-contained, individual tenancies with the aim of reducing support over time to zero. Users of residential care and supported housing are more likely to have a diagnosis of schizophrenia or other psychosis, whereas users of floating outreach are more likely to have a diagnosis of a common mental disorder, but the level of needs of people using supported housing and floating outreach is similar [3, 4] . This study found that the quality of care was higher in MH-supported housing than in residential care or floating outreach services using multivariate statistical techniques. To increase its applicability and transferability, this analysis should be completed with standard information on service availability and modelling based on system dynamics and complexity [5] . The standard performance assessment of any kind of comparable service includes studying their input consumption and output production (directly related to the inputs consumed). These inputs/outputs are mainly technical, and in the end, the best service performance is always related to the most appropriate balance between the available inputs and the produced outputs. Researchers and decision-makers can seek to reduce (minimize) the amount of inputs for a given amount of outputs (input orientation) or vice versa to increase (maximize) the output production for a specific amount of inputs (output orientation). The incorporation of quality variables in performance assessments (represented by six/ seven quality domains in this study) is always complicated because these variables are subjective (perceived quality from users, managers, etc.). To our knowledge, this study is the first to include quality domains estimated by the managers of the selected MH services in a performance analysis [6] . Independent of the technical performance of the services (the baseline scenario analysed without quality domains), the main research objective is to assess the impact on service performance when quality domains are incorporated into the analysis. Considering that MH service managers have a specific amount of inputs and always try to obtain the corresponding best results, a neutral/positive relationship between managerial processes (input and output management) and their quality perceptions of performance is expected. Relative technical efficiency (RTE) is a decision support measure that can be used to guide health informed evidence-based policy-making, mainly to improve resource allocation [7, 8] . RTE assesses the relationship among the amount of inputs consumed and outputs produced by a set of comparable decision-making units [9] . RTE can be regarded as a synthetic metaindicator that facilitates monitoring the evolution of a system and the dynamic relationships or connections across different performance indicators [10, 11] . It has been used to identify tailor-made improvement strategies for health care ecosystems such as the provision and resourcing of addiction treatment clinics [12, 13] , residential MH facilities [14] , homes for people with mental disability [15] , clinics for children and youth [16] and community-based youth services [17] . The RTEs of primary care and MH ecosystems have been systematically assessed in Basque County (Spain) [5, [18] [19] [20] . Data envelopment analysis (DEA) has been widely used to assess the RTE of health services [6, 18, 21] . This group of nonparametric techniques is very robust and flexible because they do not need any preliminary assumption on the variable statistical structure, implying that variables (inputs/outputs) from different origins and types (e.g., the number of beds, technical input, the quality of care, manager perception) can be analysed at the same time. Quality variables have been included in DEA to assess the RTE of systems operating in different socioeconomic contexts (e.g., hospital care, schools or the banking industry) [22] [23] [24] [25] . DEA can be included in a Monte Carlo simulation engine to include uncertainty and randomness in data values (all of which are considered statistical distributions) and design more realistic models [5] . As DEA is an operational model, it is completely blind. Variable values must be interpreted according to existing expert knowledge (usually a theoretical paradigm). A fuzzy inference engine allows us to operationalize the formalization of the balanced care model following the Expert-based Collaborative Analysis-EbCA methodology [5] . This engine interprets variable values before RTE was calculated. This study aims to assess the impact of quality indicators (managerial perspective) on the performance (RTE) of selected MH-supported accommodation services in the English pathway of care. This objective includes the formalization of specific quality domains into variables (rates), their integration in RTE assessment, and a comparative impact analysis of quality variables on ecosystem performance to support decision-making and investment by providing relevant information for service managers to inform practice and service planning. Accordingly, this paper first presents a description of the ecosystem under study (a representative sample of supported accommodation services in England). Then, the selected variables are described and grouped into scenarios to highlight different perspectives of the ecosystem situation. Finally, the methodology used to assess ecosystem performance (including quality domains) is briefly described. Data for supported accommodation services from 14 nationally representative local authorities in England were collected for the QuEST study [4] . Face-to-face interviews were conducted with service managers, key staff, and service users to assess the quality and characteristics of the services and those using them. The Quality Indicator for Rehabilitative Care-Supported Accommodation (QuIRC-SA) was completed with service managers. This standardized tool assesses service quality in seven domains: living environment, therapeutic environment, treatments and interventions, self-management and autonomy, social interface, human rights and recovery-based practice [26] . Data on the service's annual budget, weekly cost per user and service resources were also provided by the service managers to complement standard service costs for estimation of the cost-effectiveness of services [4] . The dataset is available at the Dryad digital repository (https://doi.org/10.5061/dryad.j0zpc86dz). Measurement units, usually expressed as rates, for each variable were discussed by an expert group comprising senior clinicians, policy-makers, providers, and researchers with expertise in MH-supported accommodation. Specifically, the experts who were involved included a rehabilitation psychiatrist who works with people living in MH-supported accommodations and is an international leader in this field, a psychiatrist with national-level expertise in policy pertaining to people with severe MH problems, two national leaders in MH-supported accommodation service provision and policy and three researchers who collected data from 87 supported accommodation services across England during the QuEST study. According to the background on evaluating the performance of MH-supported accommodation services [26] [27] [28] , the relevant inputs were service budget (£ per place), places (number), and full-time equivalent staff (professionals per service user), and the relevant outputs were the average length of stay (years), occupied beds (%), the number of service users who moved to more independent accommodations (users per place), and the seven QuIRC-SA quality of care variables considering the service size (value of the domain×available places/100). By using the last mathematical transformation, original quality indicators considered the size of the service (an indicator value of 90 does not have the same meaning for a service with 10 places/beds and a service with 100 places/beds). All the considered transformations of original data render the selected services comparable by eliminating the potential "size" effect on performance assessments. Eight different scenarios, or input/output variable combinations, were designed to assess the RTE of residential care and supported housing services (Table 1) . For floating outreach services, only seven scenarios were identified because this type of care does not include the "living environment" QuIRC-SA service quality domain. Scenario 1 can be considered the "reference" scenario because it does not include any quality domains. An adaptation of the hybrid Decision Support System (DSS) EDeS-MH (Efficient Decision Support-Mental Health) was used [19] to assess the performance of the services. This computer-based tool included a Monte Carlo simulation engine, DEA and a fuzzy inference engine. The simulation engine was developed to address the uncertainty (data imprecision and vagueness) and randomness (unexpected facts) of real environments and to artificially multiply the number of observations [29] . The inner uncertainty of any ecosystem can be overcome by transforming original data values into statistical distributions (from standard datasets to statistical distribution bases). In each simulation, the Monte Carlo simulation engine analyses a new dataset selected at random. The statistical analysis of the final results (the process is stopped when the statistical error is lower than 2.5% for the mean) includes a sensitivity analysis of the ecosystem under study. The results (RTE scores) for each DMU and scenario are statistical distributions that can be studied in a more (basic statistics) or less (stability and entropy) standard manner [19, 21] . The characteristics of these statistical distributions represent the potential reaction of the DMU to data changes. DEA [9] was selected to evaluate the RTE of MH-supported accommodation services. This nonparametric technique has been widely used to assess health service performance [30] and, to a lesser extent, to evaluate MH services [6] . The standard DEA model is a linear programming one which structure is: where d is the number of DMU, i the number of inputs and j the number of outputs. The DMU m consumes x hm of input h and produces y rm of output r. θ is the efficiency score and S À h and S þ r are the slacks [21] . In this research, the variable returns to scale DEA [31] was selected because when studying MH services, real output variations cannot be considered proportional to the corresponding input modifications [32] , and constant returns to scale would involve a constant variation that cannot be considered realistic. The input-oriented DEA model was applied to assess whether service input consumption can be reduced while assuming a constant output level [9] , which is especially relevant for decision-makers who must allocate finite resources to meet population needs. Output maximization (output-oriented DEA) is especially difficult and sometimes not recommendable (for example, when the system artificially tries to maximize the number of users who are moved to a service with greater independence, this can be mathematically correct but from a health care perspective it has no sense at all) when supported accommodation services are assessed. Finally, no weight control of the variables was considered. When the data envelopment analysis (DEA) did not include weight control of the inputs/outputs, a two-step process was followed to verify the impact of quality variable needs. First, RTE must be assessed without them, which yields 'operational results' (baseline). Second, quality variables must then be included as outputs to investigate the influence of quality on performance. Finally, when the number of observations is low (here, the number of move-on residential services is especially low), DEA cannot be sufficiently discriminative (in the end, the methodology tends to show that all DMUs are efficient). However, as explained before, the uncertainty analysis (Monte Carlo simulation engine) multiplies the number of observations by the number of selected simulations, which overcomes this DEA drawback. Prior to RTE assessment, variable (inputs and outputs) data values must be interpreted according to a preselected paradigm; otherwise, performance results can be biased. The fuzzy inference engine carries out this process automatically by including an expert-based rule base (IF . . . THEN . . .) according to the Balanced Care Model developed by Thornicroft and Tansella [33] and the pathway of care of MH-supported accommodation services [34] . This paradigm was used to define the range of adequacy for each variable. For example, the variable values for the service budget (£ per place)-input-in floating outreach services were considered adequate between the range [5000, 6000]. In this range, a greater value corresponds to better competence (it is mathematically transformed to be interpreted by the analytical procedure). Outside this range, the variable value is penalized by multiplying it by a parameter (in this specific case, by 2) because it is considered less adequate than when inside the range. The specific references for data value interpretation have been defined by a panel of experts according to the paradigm selected and their expertise. This process followed the EbCA model [5] , where an iterative sequence of expert-based reviews culminates in a consensus. Once the references for interpreting variable values are defined, the EDeS-MH automatically runs a mathematical transformation based on an equation (linear monotone transformation) or a fuzzy operator (product-sum gravity method) to obtain the "transformed" value [21] . These transformed values will be analysed by DEA to determine the corresponding RTE scores (statistical distributions). To evaluate the statistical significance of the differences between the baseline (without quality variables, scenario 1) and the rest of the designed scenarios (corresponding to each quality domain, scenarios 2 to 7/8), the nonparametric statistical Wilcoxon signed-rank test was used. First, the types of services were coded according to the DESDE-LTC (Description and Evaluation of Services and DirectoriEs for Long Term Care) [35, 36] , an international service classification system that facilitates comparisons across different jurisdictions and studies and is used to describe service availability in our previous RTE studies using EDeS-MH. Second, expertbased cooperative analysis (EbCA) [5] was used to formalize expert knowledge on MH-supported accommodation services into a knowledge base structured by rules. This knowledge base facilitates interpretation of variable values according to the selected paradigm. EbCA has previously been used to evaluate the RTE of MH areas in Basque County [19] . The EbCA panel of experts on MH-supported accommodation services in England included persons in charge of the QuEST study, MH planners, MH managers and academic researchers (see the Scenario section for details). The interaction among these agents in the meetings was considered crucial for acquiring explicit their implicit knowledge: EbCA is an iterative procedure. The fuzzy inference engine interpreted variable values according to the resulting ranges of adequacy [19] according to the selected paradigm. Services considered outliers in terms of the three main supported accommodation service groups were also identified by the EbCA panel. For example, some of the residential care services were oriented to work with people to helping them move on to more independent accommodations, whereas others worked with people who were likely to need high levels of support long term. Missing values (very few) were statistically imputed using a Monte Carlo simulation procedure based on the real statistical distribution of the corresponding variable. Original data were randomized using symmetric triangular statistical distributions (5% variation on each side of the corresponding original value). This range includes feasible data variations (imprecision and vagueness). No critical stress on the ecosystem was included in the analysis (randomness). This procedure includes both data variations corresponding to ecosystem evolution (population, user mobility, etc.) and the effect of the time. The Monte Carlo simulation engine selects a specific data value from the dataset in each computer run. Then, the fuzzy inference engine interprets these values according to the paradigm. The obtained RTE performance scores for each MH service are statistical distributions that can be analysed accordingly. RTE scores are always in a [0, 1] range. If RTE = 0, then the service is completely inefficient; if RTE = 1, then the service is efficient, and if the RTE score is between (0, 1), then the service is inefficient (a greater RTE score corresponds to higher service efficiency). For example, if a service has an RTE = 0.45 and another has an RTE = 0.98, both are inefficient, but the second service is very close to the efficiency. Nine move-on residential services, 13 non-move-on residential services, 34 supported housing services and 30 floating outreach services were finally analysed because they have complete quality datasets. Five hundred simulations were run by the DSS. Basic statistics for the original data are shown in Table 2 . The results show that the variability among services in each type of care is very high. The representativeness of the variables (inputs/outputs) for the RTE assessment was confirmed by the EbCA panel. Therefore, they can be considered indicators for real informed evidencebased decision-making. Quality domains (assessed by the QuIRC-SA [26] ) were considered outcomes for performance assessments. Only one service was considered an outlier and removed from the analysis. On the other hand, the analytical process of setting the measurement units for the variables highlighted the existence of two different groups in the residential care services dataset: move-on and nonmove-on oriented services. Regarding the move-on-oriented MH residential care services, the average RTE oscillated from 0.7, the worst, (scenario 1, baseline) to 0.86, the best, (scenario 2, therapeutic environment) ( Table 3 ). In the baseline scenario, the highest performance (0.94) was achieved by service 22, while service 18 had the lowest score (0.5). Therefore, the performance of this type of care is very heterogeneous. When quality variables are included as outputs in scenarios 2-8, the RTE mean (which was always greater than 0.84) and the performance of each service increased significantly (p < 0.001) compared to the baseline. Services 4 and 26 had the worst performance. In the non-move-on MH-oriented residential care services, the average RTE was 0.79, with service 90 (0.96) having the best and services 61 and 81 the worst RTE scores (Table 4 ). Again, including quality variables had a statistically significant (p < 0.001) positive impact (the performance increased), but the heterogeneity remained high. Scenarios 7 (treatments and interventions), 8 (recovery-based practices) and 4 (self-management and autonomy) were associated with the highest increase in performance. Services 61 and 81 showed low performance. For MH-supported housing services, the average RTE oscillated from [0.54 to 0.64 (Table 5 ). In the baseline model, the average RTE was 0.54 (with service 86 (0.89) performing the best and service 89 (0.33) performing the worst; the aggregation of quality variables increased the average RTE, and these differences were significant (p < 0.001). Here, scenario 7 (treatment and interventions) showed the lowest increase. Services 104, 4, 114, 70, 84, 19, 101, The role of quality in the performance of mental health supported accommodation services in England 113, 60, 52, 89 and 95 had an average RTE lower than 0.5. The performance variability was very high in all scenarios. In MH floating outreach services, the average RTE was in the 0.5 to 0.65 range (Table 6 ). For the other two types of supported accommodations, the baseline model showed the worst performance, and the inclusion of quality variables significantly increased both global and individual average service RTE (p < 0.001). Services 1, 63, 37 and 35 had very high performance. The heterogeneity of the sample was also very high. Generally, the inclusion of quality domains highlights a neutral-positive or positive impact on the performance of supported accommodation services in England, which is especially relevant in some services that showed a lower average RTE: 13, 18 and 21 (33.3% of the surveyed services, Table 3 Table 6 ). In these services, the manager´s perception of quality provided by their services surpasses technical results (baseline scenario), probably because of non-evaluated variables or circumstances in care provision. On the other hand, quality variables decrease service performance only in supported accommodation services 15, 31, 32, 86 and 98 (14.7% of the surveyed services, Table 5 ). This behaviour can be considered strange because here, the manager´s perception of quality provided by their services underestimates their own technical results. Again, additional non assessed variables or circumstances can affect how the manager processes information with respect to detecting frameworks of improving quality of care. Services where the inclusion of quality variables results in a neutral behaviour of their performance constituted the majority, respectively: 66.7%, 69.2%, 61.8% and 73.3% of the surveyed services. In these cases, the manager´s perception of quality provided by their corresponding services matches their technical results; they obtain a quality according to their resources and outcomes. This study aimed to assess the impact of quality domains (formalized as variables) on the performance (RTE) of selected MH-supported accommodation services in the English pathway of care. To our knowledge, this study is the first to evaluate the performance of MH-supported accommodation services with the inclusion of service quality indicators in the analyses. The integration of the quality construct (structured by a set of specific domains with varying relevance) in DEA is complex. Authors have addressed this issue using different techniques: quality-adjusted DEA (Q-DEA) [37] , the two-model approach [38] , quality and operating efficiency models with weight restrictions [39] , and the multiple objective approach to DEA (MODEA) [40] . The EDeS-MH decision support system has been successfully adapted to assess MH services instead of catchment areas [5, 18, 19, 21] . This DSS transfers the robust methods for resource allocation and health technology assessment used at national (macro) and regional (macro) levels to the local (meso) and service (micro) levels, reducing the lack of transparency and accountability in relation to resource management, which often occurs at this level [41] . Finally, the EbCA highlights the existence of two different residential services: move-on and non-move-on oriented services, which must be studied separately. To date, MH ecosystem performance has been assessed by combining operational and technical variables without including quality indicators [6] . Quality of care scores were determined by a standardized instrument applied through interviews with service managers; therefore, they are managerial perceptions. However, the ratings produced have been shown to correlate well with service user experiences of care [42] . Expert knowledge, formalized in a knowledge base, was crucial for interpreting the level of adequacy of variable values. For this process, selecting the appropriate paradigm was mandatory to overcome the limitations of classical DEA, where less input (resources) consumption, given an output level, was related to higher efficiency. In this research, the selected paradigm was structured by the Balanced Care Model [33] and the English pathway of care in MH-supported accommodation services [34] . The knowledge base is the core of the fuzzy inference engine because it provides expert-based information for transforming the original data values (from the Monte Carlo simulation engine) according to the balanced care model. This transformation is based on experts' opinions on the adequacy of each data value and represents a specific framework that must change throughout the time span and depend on the socioeconomic environment. Therefore, the same dataset (raw data) can result in different results according to experts' perceptions of reality. The fuzzy inference engine can "understand" the nuances that decision-makers assume in system management. A recent national survey of MH-supported accommodation services across England found that the quality of care was higher in MH-supported housing than in residential care or floating outreach services [4] . Considering that quality variables have not been included in any RTE assessment of MH services and ecosystems [6, 13, 15, 18, 43] , the question to answer here is whether the perception (given by the corresponding managers) about the quality provided by an MH service is consistent with its technical performance. If the impact on RTE scores is negative, then the perception of the quality provided is not aligned with the technical performance (baseline RTE). In this situation, technical results do not achieve an equivalent quality according to the manager´s opinion. If the impact can be considered neutral, then technical results and quality perception are balanced. Finally, if it is positive, managers know that the quality provided by their services is better than that of the corresponding neutral impact and probably manage other variables that could not be gathered in the QuEST study. The incorporation of quality domains as variables (outputs) in DEA had a neutral-positive or positive global impact on the performance of MH-supported accommodation services. The MH residential care services dataset was divided into those aiming to promote autonomy among service users (move-on oriented) and those focused on providing residential care (non-move-on oriented). This fact was highlighted by the EbCA panel of experts and allowed DEA to avoid the bias mainly induced by the interpretation of a critical output: "the number of service users who have moved to more independent accommodation". Instead of being the most expensive and probably complex services, RTE scores revealed relatively high performance in both groups (especially when quality variables were included), but the variability was high, revealing probable differences in management strategies. Quality variables increased (sometimes not significantly) the average RTE in a relevant number of the selected services, but most showed a neutral-positive profile. Quality indicators included nontechnical but real service characteristics estimated by the respective managers that ultimately improved service performance scores. The average RTE of MH-supported housing services was lower than that of residential care homes, and the input/output balance was slightly worse because of the higher statistical variability. This variability can be associated with differences in service users' needs and personal characteristics. The incorporation of quality variables significantly increased service performance and confirms the conclusions of a previous study by Killaspy et al. [4] : supported housing is a cost-effective type of care, as it provides support while promoting autonomy. Nevertheless, service performance scores were lower than expected due to the high variability among services and the low number of service users who moved to more independent accommodations (a critical technical output for the paradigm). A considerable number of services in MH-supported housing had an average RTE below 0.5 (relatively low performance), indicating that these types of services adjust their structures and obviously obtain different outcomes considering users' needs (tailor-made structures). For these specific services, quality variables did not significantly increase performance. Again, considering that quality was assessed by the service managers (managerial perspective), they implicitly stated that in these services, major changes can be developed. The English pathway of care is not a one-way road, and service users can be moved from one supported housing service to another (instead of to a floating outreach service) or even remain in the same place longer than expected. This variable (N˚of service users who moved to a more independent accommodation per bed/place) is likely to be outside the service's control to some degree, i.e., the service may be working hard with service users to help them gain skills to move on, but the supply of more independent accommodations where they can move on may be insufficient. In this type of care, a relatively pertinent number of services decreases performance scores when quality variables are included, which may indicate that managers consider other variables or service characteristics that lead to poorer quality results than they were expecting considering the corresponding resource level and outcome production. The performance of MH floating outreach services was the lowest on average, but the incorporation of quality variables still led to a statistically significant increase in the average global RTE. The results reported by Killaspy et al. [4] also showed that these services provided lowerquality care than supported housing or residential care, but they also achieved the highest rate of movement to more independent accommodations. The high statistical variability of the services in this type of care indicates that they may be struggling to meet a wide range of service user needs. After adjusting for differences in patient characteristics, the move-on characteristic was a critical variable considering the Balanced Care Model (a paradigm for interpreting variable values). As occurs in residential houses, a relevant number of floating outreach services significantly increased their performance scores. Again, this increase is not aligned with the service management (resources and outcomes) shown in the baseline scenario. MH service names were not revealed in this study to maintain data privacy. The ideal integration of supported accommodation services in an MH care pathway is unfortunately difficult to determine outside of highly integrated care systems. The existence of functioning care pathways is a major example of integration. With increasing integration, the inclusion of service quality variables in the RTE assessment of MH-supported accommodation services in England has become possible. Quality increased global service performance in the three types of care: residential care, supported housing and floating outreach. This neutral-positive or positive impact showed that RTE assessment using only technical variables is not sufficient to achieve a holistic view of service performance. Expert knowledge formalization was critical for distinguishing the types of service, identifying outliers and interpreting variable values according to a specific care paradigm. EbCA demonstrated its practical utility when appropriate experts were available to join the panel. The adaptation of the EDeS-MH to include quality variables required a two-step process. The results allowed us to have a better understanding of the performance of individual services and the supported accommodation care pathway. This approach may have utility in designing tailor-made improvement strategies for specific services as well as in service planning. The input/output balance of MH residential care services was appropriate because they were more structured from a managerial point of view (user needs are usually very well defined), while supported housing and floating outreach could be improved. These types of care are not structured due to the diversity of user needs associated with their higher level of autonomy compared to residential care service users. Considering that the English pathway of MH-supported accommodation is not a one-way road, the impact of remaining in supported housing and floating outreach services longer than the usual two years appears to have a major negative influence on service performance. Services showing a significant increase or decrease in their performance scores when quality variables are included in the analysis should be studied. The observed differences from the baseline scenario (only technical) must be a consequence of something related to specific structural or managerial characteristics. These characteristics are a crucial source of information for the design of new interventions, policies or strategies to improve MH care. Future research including other components of the whole system (such as inpatient, day and outpatient care) is recommended to understand global MH ecosystem performance in England. By selecting service benchmarks, key variables requiring improvement can be identified to design specific policies and interventions. The integration of the user's satisfaction construct into the analysis is also a major trend. Mental health supported accommodation services in England and in Italy: a comparison Rehabilitation for adults with complex psychosis: summary of NICE guidance Predictors of moving on from mental health supported accommodation in England: national cohort study Quality of life, autonomy, satisfaction, and costs associated with mental health supported accommodation services in England: a national survey Integrating clinicians, knowledge and data: expertbased cooperative analysis in healthcare decision support Relative Technical Efficiency Assessment of Mental Health Services: A Systematic Review An efficiency data envelopment analysis model reinforced by classification and regression tree for hospital performance evaluation Efficiency of private and public primary health facilities accredited by the National Health Insurance Authority in Ghana Measuring the efficiency of decision making units Reporting the effects of interventions to the tutor with meta-indicators Development and spatial representation of synthetic indexes of outpatient mental health care in Andalusia (Spain) Analyzing performance in addiction treatment: An application of data envelopment analysis to the state of Maryland system Employing super-efficiency analysis as an alternative to DEA: An application in outpatient substance abuse treatment Investigating sources of inefficiency in residential mental health facilities A modified three-stage data envelopment analysis: The Netherlands Efficiency and Organization in Norwegian Psychiatric Outpatient Clinics for Children Efficiency evaluation of community-based youth services in Virginia Assessment of Relative Technical Efficiency of Small Mental Health Areas in Bizkaia A decision support system for assessing management interventions in a mental health ecosystem: The case of Bizkaia Evaluació n de la eficiencia té cnica de la atenció n primaria pública en el País Vasco Evaluation of system efficiency using the Monte Carlo DEA: The case of small health areas A two-stage double-bootstrap data envelopment analysis of efficiency differences of New Zealand secondary schools Quality, and Efficiency for a Partially Super-Additive Cost Function: Connecticut Public Schools Revisited Data envelopment analysis applied to quality in primary health care The Use of Data Envelopment Analysis in Banking Institutions: Evidence from the Commercial Bank of Greece Adaptation of the Quality Indicator for Rehabilitative Care (QuIRC) for use in mental health supported accommodation services (QuIRC-SA) Predictors of quality of care in mental health supported accommodation services in England: a multiple regression modelling study The development of the Quality Indicator for Rehabilitative Care (QuIRC): a measure of best practice for facilities for people with longer term mental health problems Uncertainty and information: foundations of generalized information theory A survey and analysis of the first 40 years of scholarly literature in DEA: 1978-2016 Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis Use of an operational model of community care to assess technical efficiency and benchmarking of small mental health areas in Spain The balanced care model for global mental health Housing services for people with mental disorders in England: Patient characteristics, care provision and costs Standardised description of health and social care: A systematic review of use of the ESMS/ DESDE (European Service Mapping Schedule/Description and Evaluation of Services and DirectoriEs) Developing a tool for mapping adult mental health care provision in Europe: the REMAST research protocol and its contribution to better integrated care Benchmarking with quality-adjusted DEA (Q-DEA) to seek lower-cost high-quality service: Evidence from a U.S.bank application A Two-Model approach to measuring operating and quality efficiency with DEA Weights restrictions and value judgements in Data Envelopment Analysis: Evolution, development and future directions A multiple objective approach to data envelopment analysis LSE-Lancet Commission on the future of the NHS: re-laying the foundations for an equitable and efficient health and care service after COVID-19 Quality of longer term mental health facilities in Europe: validation of the quality indicator for rehabilitative care against service users' views Managed care and technical efficiency in outpatient substance abuse treatment units We would like to thank the experts of the EbCA nominal group: Jed Boardman, Andrew van Doorn, Kathy Roberts, Peter McPherson, Christian Dalton-Locke and Sarah Dowling. Conceptualization: Nerea Almeda, Carlos Ramón García-Alonso.