Deb Performance Assessment in Community Mental Health Care and At-Risk Populations Ann M. Holmes, Ph.D. and Partha Deb, Ph.D. We examine whether community mental health care centers (CMHCs) dif fer in their ability to serve at-risk populations, includ­ ing clients with dual diagnoses for sub­ stance abuse, comorbid disabilities, and par ticularly severe functional impairment. Our analysis uses data from Indiana’s pub­ lic mental health system. Although at-risk clients experience, on average, worse out­ comes than other clients, we find that some CMHCs achieve statistically significantly better outcomes than others. Although this information is useful to consumers and providers who wish to identify the most ef fective providers and treatment models for at-risk clients, it is not generated in standard per formance assessments. INTRODUCTION While the health care system overall has retreated from managed care, such compe­ tition-based strategies continue to be increasingly prevalent in public mental health care (McIntyre, Rogers, and Heier, 2001). The organizational and financial arrangements associated with managed care are designed to increase efficiency and reduce waste in health care deliver y. Concerns have arisen that these same incentives may lead providers to under- ser ve clients, particularly individuals with severe or complicated conditions (Ellis, 1998; Ware et al., 1996). To guard against Ann M. Holmes is with Indiana University-Purdue University Indianapolis. Partha Deb is with Hunter College, City University of New York. The statements in this article are those of the authors and do not necessarily reflect the views or policies of Indiana University-Purdue University; Hunter College, City University of New York; the Indiana Family and Social Ser vices; or the Centers for Medicare & Medicaid Ser vices. such potential negative outcomes, man­ aged care is typically supplemented with monitoring of provider performance. Yet, “[d]espite recent research on methods of risk adjustment…, the application of this research to Medicaid populations has lagged” (Ireys, Thornton, and McKay, 2002). For instance, standard methods of performance assessment focus on average outcomes, and may not detect suboptimal quality of care provided to select groups of at-risk clients. Our analysis is based on data from the Indiana Division of Mental Health and Addictions (IDMHA). IDMHA is the public agency that ser ves as payer of last resort for persons with persistent and severe mental illness in Indiana. Care is delivered through 1 of 30 not-for-profit CMHCs, which act as gatekeepers to the 6 State hos­ pitals. In 1996, the IDMHA adopted the Hoosier Assurance Plan that reformed the deliver y system along managed care prin­ ciples (Family and Social Ser vices Admin­ istration, 1997). Subsequently, IDMHA produced provider repor t cards that describe various aspects of the centers that reflect the quality of care provided, includ­ ing differences in assessed mental health outcomes experienced by clients at these centers (Family and Social Ser vices Administration, 2000). While the IDMHA analysis controls for baseline functioning, it ignores variance that may be due to non- clinical client factors. In addition, the IDMHA analysis produces only limited subgroup analysis, in part because it uses a stratified approach that severely limits the HEALTH CARE FINANCING REVIEW/Fall 2004/Volume 26, Number 1 75 extent to which different subgroups can be compared. As a result, the report cards cannot identify the vulnerability of some at- risk client groups, at-risk clients cannot use the information to identify optimal choices for people most like themselves, and treatments that work best on average may be applied to some clients for whom other treatment approaches may be more appropriate. In previous analysis of these data (Deb, Holmes, and Deliberty, 2004), we showed the importance of adjusting performance measures for non-clinical client character­ istics (e.g., sociodemographic variables and income), and different rates of client attrition across CMHCs. In this article, we extend this analysis to examine whether per formance dif ferentials obser ved in aggregate apply to specific, vulnerable sub­ populations of clients, including clients with dual diagnoses for substance abuse, comorbid disabilities, and mental illnesses that cause particularly severe functional impairment. Methods Typically, estimates of provider perfor­ mance have been generated in a fixed effects framework. We use instead a mixed random effects model to evaluate provider performance. The model includes both fixed coefficients (which permit control of client risk factors on outcomes) and ran­ dom coefficients associated with provider- specific variation. We estimate the mixed random effects model in SAS® with the PROC MIXED procedure (SAS Institute Inc., 1999). In addition, we adjust provider performance for different rates of client attrition using a non-linear selection equa­ tion. The formulation of the non-linear selection equation with fixed and random coefficients is described in Deb, Holmes, and Deliberty (2004) and estimated using the NLMIXED procedure in SAS® (SAS Institute Inc., 1999). As a multilevel modeling technique, the mixed random effects model offers a num­ ber of advantages over standard fixed effects specifications and is particularly attractive for the objectives of this analysis. First, because outcome analysis is typically based on data with a natural hierarchical structure (clients are grouped according to CMHC), multilevel models appropriately correct standard errors for clustering ef fects. Second, unlike analysis that is stratified by patient subgroups, mixed ran­ dom effects models can accommodate the cell sizes that can arise when centers ser ve relatively small numbers of particular at- risk client types. Finally, multilevel models can easily accommodate interaction terms to evaluate if relative provider performance is conditional on type of client ser ved (Goldstein and Spiegelhalter, 1996). Data The data for this analysis are taken from the Indiana Managed Care Provider Client Based Data Reports for fiscal years (FYs) 1998 and 1999. These data are collected on an ongoing basis for the universe of all clients who qualify for enrollment in the Hoosier Assurance Plan. An individual is eligible for care if (1) he or she has a severe mental, behavioral, or emotional disorder (as defined by the fourth revision of the Diagnostic and Statistical Manual of Mental Disorders) that is expected to last for more than 12 months, and that impairs functioning, (2) is at least 18 years of age, and (3) is eligible for Medicaid or food stamps, or has income that is below 200 percent of the Federal poverty level. The Indiana data include information on 35,098 individuals who were enrolled in the HEALTH CARE FINANCING REVIEW/Fall 2004/Volume 26, Number 1 76 Hoosier Assurance Plan in FYs 1998 and 1999. Performance is assessed using the sample of 16,516 individuals who were enrolled continuously through this period and, thus, for whom we can obser ve changes in health status over 1998. Dependent Variable “High quality care for chronic conditions entails a focus on optimizing functional sta­ tus” state Clauser and Bierman (2003). In this spirit, IDMHA requires CMHCs to rou­ tinely and comprehensively conduct func­ tional assessment for every client for whom the division covers care, and bases its report cards on this information. Functioning is assessed using the Hoosier Assurance Plan Instrument - Adults (HAPI-A) (Family and Social Services Admini-stration, 1997). The HAPI-A captures severity of illness on four behavioral health dimensions (symptoms of distress and mood, community functioning, social support, and risk behavior and sub­ stance abuse) and one dimension of physical health. The HAPI-A has been shown to yield reliable and valid measures of health out­ comes for this population (Newman et al., 1997; Deliberty, Newman, and Ward, 2001). Centers are contractually obliged to report functioning scores biannually, and these data must be supplied before reimburse­ ments are paid. Each center must have at least one designated person who receives training from IDMHA staff on an annual basis, where training is focused on achieving reliable scoring of clients. Reliability is fur­ ther enhanced by an annual audit of a sam­ ple of HAPI-A scores undertaken by an accounting firm that uses trained medical personnel to evaluate the consistency with which functioning is assessed across clients and centers. We base our analysis on one subscale from the HAPI-A, mental health symptoms and mood, which is constructed from rat­ ings of the client’s depression, anxiety, and symptom distress. The scale takes on val­ ues between 3 (most ill) and 21 (least ill). We chose to base our analysis on this one subscale because it is the one most highly correlated with the Global Assessment of Functioning Scale (a commonly employed measure of functioning used in the mental health care field), and because it has been shown to be more sensitive to changes in mental health, with effect sizes measured at 3-month inter vals twice that of the Global Assessment of Functioning Scale (Newman et al., 1997). Our measure of outcome is the change in the mental health symptoms and mood score between the beginning of FY 1998 and the beginning of FY 1999. Because the focus of our analysis is on persons with per­ sistent severe mental illness, measured improvements on any outcome scale tend to be modest. Indeed, the average absolute change in our functioning measure is only 3.2 in this sample, and one-third of such changes were less than one in absolute value. Given that the instrument used to measure outcome in this analysis has been shown to have superior sensitivity to other instruments used in the field, we believe these minimal changes simply reflect the difficulty of achieving recover y in persons with such severe illnesses. Case-Mix Variables Given our choice of dependent variable, it is necessar y to include baseline mental health functioning (as measured at the beginning of FY 1998) to control for possi­ ble effects of regression to the mean. We also consider a number of client socioeco­ nomic characteristics to control for possi­ ble differences in illness perception, treat­ ment efficacy, and compliance across dif­ ferent client groups. These include age and age squared (to account for possible HEALTH CARE FINANCING REVIEW/Fall 2004/Volume 26, Number 1 77 non-linearities found in previous studies [Cuffel et al., 1996]), education level, sex, family income, marital status, and race and ethnicity. We also include two clinical vari­ ables, measured at baseline, including whether or not the client had a disability other than a mental disorder (including being blind, deaf, mute, non-ambulator y, neurologically impaired, developmentally disabled, or illiterate), and whether the center considered the client to be at risk for substance use. To correct for possible sample selection bias, we also include a second order approximation of the true selection index as recommended by Vella (1998). The selection index is derived from an analysis in which client retention across the 30 CMHCs is expressed as a function of obser vable client characteristics. Refer to Deb, Holmes, and Deliberty (2004) for more detailed information. Center-Level Variables Center-level effects are inferred from indicator variables in the mixed random effects model. To determine if CMHCs dif­ fer in their ability to ser ve particular at-risk populations, we interact these center indi­ cator variables with risk indicators for client groups of concern. These groups include clients with dual diagnoses for sub­ stance abuse (ICD-9 codes 303, 304, and/or 305), clients with comorbid dis­ abling conditions (including being blind, deaf, mute, non-ambulator y, illiterate, and/or having a developmental disability or neurological impairment), and clients with mental illness that causes severe func­ tional impairment (as determined by an IDMHA algorithm that considers mental and physical health, social and community functioning, and risk behaviors). These interaction terms allow the slope coeffi­ cients on the client risk factors to var y by CMHC, and can be used to determine if a Table 1 Prevalence of At-Risk Populations Sub-Group Prevalence Percent Mental Illness with Severe Functional Impairment 3.95 Co-occurring Substance Abuse 12.05 Comorbid Disability 28.67 NOTE: Results are based on all 35,098 clients served in FYs 1998 and 1999. SOURCE: Holmes, A.M., Indiana University-Purdue University Indianapolis and Deb, P., Hunter College, City University of New York: Data based on estimates from the Indiana Managed Care Provider Client Based Data Reports, FYs 1998-1999. particular CMHC performs significantly better or worse when ser ving individuals from specific vulnerable client groups. Summar y statistics for these variables are presented in Table 1. Analysis Plan We estimate two models, one with center indicators interacted with at-risk variables, and one without for comparison purposes. Overall model fit is evaluated using both Akaike and Bayesian Information Criteria (AIC and BIC). Both are related to the adjusted R2 statistic, but with slightly dif­ ferent adjustments for the number of inde­ pendent variables, and with smaller values of the test statistic associated with better model fit. Because the BIC also adjusts for sample size, it is less likely to favor over-fit­ ted models. To evaluate the robustness of center per­ formance differentials across the at-risk client subgroups, we consider the follow­ ing: First, the overall importance of the variation in per formance across all CMHCs for different client subgroups is assessed by examining the covariance parameter estimates associated with each group of interaction terms. Standard t-sta­ tistics associated with the individual inter­ action coefficients can be used to deter­ mine the extent to which any one CMHC may produce statistically significantly bet­ ter or worse outcomes for a particular at-risk HEALTH CARE FINANCING REVIEW/Fall 2004/Volume 26, Number 1 78 Table 2 Model Fit and Covariance Parameters Category Overall Model Fit Covariance Parameter Model AIC BIC Base Model 88750.6 88753.4 — Intercept — — 0.2382 (0.0004) Interaction Model *88686.7 *88693.7 Intercept — — 0.2363 (0.0007) Mental Illness with Severe Functional Impairment — — 1.1781 (0.0118) Co-occurring Substance Abuse — — 0.3473 (0.0221) Comorbid Disability — — 0.1502 (0.0290) *Indicates best model based on fit criteria. NOTES: N=16,516. AIC is Akaike Information Criteria. BIC is Bayesian Information Criteria. Numbers in parentheses are p-values. SOURCE: Holmes, A.M., Indiana University-Purdue University Indianapolis and Deb, P., Hunter College, City University of New York: Data based on estimates from the Indiana Managed Care Provider Client Based Data Reports, FYs 1998-1999. client subgroup. Second, we calculate the largest change, both positive and negative, in ranks inferred from the center indica­ tors in the overall model and the coeffi­ cients on the interaction terms associated with each at-risk client group. Third, we calculate the proportion of changes in rela­ tive ranks between each at-risk group and the overall center ranks. The number of changes in relative ranks is given by 1-W/2, where W is Kendall’s measure of concor­ dance between the ranks implied by the two groups being compared. Fourth, we calculate the correlation in relative perfor­ mance and implied ranks between each at- risk group and the overall client popula­ tion. Results Model fit statistics are presented in Table 2. Based on both selection criteria, the model that includes interaction terms dominates the model that assumes relative performance differentials are the same for all client subgroups. The covariance para­ meter estimates, also presented in Table 2, indicate that there are significant differ­ ences in relative center performance for each at-risk group, with the most signifi­ cant differences being obser ved for the client group with severe functional impair­ ment due to mental illness (p=0.012), fol­ lowed by the group with co-occurring sub­ stance abuse (p=0.022). Estimates of the fixed coefficients that capture the effect of client case-mix vari­ ables (available from the authors on request) are robust to the inclusion of interaction terms. The random ef fects solutions in Table 3 provide information on the relative performance of the 30 commu­ nity mental health centers. These random effects coefficients represent the estimat­ ed deviations for each center from the mean performance score, with positive (negative) estimates indicating the center per formed above (below) the average level. The coefficients on the interaction terms, by comparison, represent the esti­ mated deviations in performance score between the at-risk group considered and the not at-risk group for that particular CMHC. In the overall model, four CMHCs are found to perform significantly (p<0.05) bet­ ter than average, and six perform signifi­ cantly worse than average. Although the magnitude of these provider-level coeffi­ cients may appear to be small, it is impor­ tant to note that they measure the devia­ tions from the average change in function­ ing score. Given the mean absolute change in functioning score is only 3.2, even a HEALTH CARE FINANCING REVIEW/Fall 2004/Volume 26, Number 1 79 Table 3 Random Effects Solutions for Provider Performance Differentials Interaction Model Mental Illness with Base Model Severe Co-occurring Implied Intercept Implied Functional Implied Substance Implied Comorbid Implied CMHC Intercept Rank (Not At-Risk) Rank Impairment Rank Abuse Rank Disability Rank A *1.1144 1 *1.2174 1 -0.0647 3 **-0.4342 3 -0.2577 1 B *0.7459 2 *0.7375 2 0.9782 1 0.2031 1 -0.1537 4 C *0.6995 3 *0.7048 3 0.0214 7 0.063 4 NA — D *0.4767 4 *0.5459 4 0.6069 2 *-0.6715 13 0.3127 2 E **0.4011 5 0.2679 5 -0.977 23 0.0488 8 **0.5370 3 F **0.2751 6 0.1917 7 0.6472 5 0.0834 9 0.3918 5 G 0.253 7 0.178 10 0.286 9 0.3794 5 0.0106 11 H 0.1783 8 0.1807 9 0.9 4 -0.4708 17 0.0081 10 I 0.1763 9 0.0159 14 0.558 8 NA — 0.2405 7 J 0.148 10 0.2134 6 -0.0222 11 0.2845 6 -0.2852 15 K 0.1233 11 0.1831 8 -0.244 16 -0.3539 14 NA — L 0.1147 12 0.147 11 0.65 6 -0.2181 12 -0.1091 13 M 0.1127 13 0.1003 13 *-1.944 29 *-0.7003 20 **0.3756 6 N 0.0632 14 0.1467 12 -0.0001 13 0.2302 7 *-0.6207 21 O 0.0528 15 -0.0129 16 0.0013 15 -0.1696 15 0.2083 9 P 0.0223 16 -0.083 18 -0.7651 24 *0.9786 2 -0.3302 20 Q -0.0092 17 -0.1926 22 -1.1396 28 -0.0239 16 0.4182 8 R -0.0191 18 0.0127 15 -0.3434 21 -0.0551 11 -0.1921 16 S -0.0374 19 -0.0942 19 0.0832 14 0.1055 10 0.077 14 T -0.0382 20 -0.1956 23 -0.0369 19 NA — 0.3198 12 U -0.1355 21 -0.06 17 **-0.8093 25 NA — -0.2875 19 V -0.2475 22 -0.1518 21 0.0164 18 **-0.8166 23 NA — W -0.2524 23 -0.1301 20 -0.3294 22 *-0.8571 24 -0.1792 18 X **-0.3646 24 *-0.5014 26 0.9018 10 0.032 19 0.2013 17 Y *-0.3793 25 **-0.3481 24 -0.6612 27 -0.0916 18 -0.2189 24 Z *-0.4412 26 **-0.4256 25 0.3262 17 NA — -0.1033 22 AA *-0.5352 27 *-0.5064 27 -0.4176 26 -0.1367 21 -0.0477 23 BB *-0.5923 28 *-0.5581 28 0.7308 12 -0.31 22 -0.0865 25 CC *-0.7936 29 **-0.5870 29 0.2699 20 NA — -0.2682 26 DD *-1.112 30 *-0.9962 30 *-2.295 30 NA — 0.0393 27 *p<0.05. **p<0.10. NOTES: CMHC is community mental health care center. NA indicates the CMHC served no clients in the at-risk category. At-risk coefficients under the interaction model represent the marginal differences in performance, while at-risk ranks are based on levels of performance. SOURCE: Holmes, A.M., Indiana University-Purdue University Indianapolis and Deb, P., Hunter College, City University of New York: Data based on estimates from the Indiana Managed Care Provider Client Based Data Reports, FYs 1998-1999. 1-point difference would be considered substantial. Also, the coefficients measure the average deviation for all clients treated at the center. Thus, a coefficient of +1 would correspond with improving the func­ tioning of ever y client at the center by one additional point (on average) compared with the mean center. Improvements above +1 are in the top one-third of all improve­ ments for this population, so a center with a coefficient greater than one would have essentially moved their clients from out­ comes in the middle one-third of the distri­ bution to outcomes in the top one-third of the distribution. Across all centers, outcomes are much worse for clients whose mental illnesses caused severe functional impairment (aver­ age interaction coefficient of -0.10), and somewhat worse for clients with co-occur­ ring substance abuse (-0.027) and comor­ bid disabling conditions (-0.027). Although there are only a small number of signifi­ cant differences for the at-risk groups, at HEALTH CARE FINANCING REVIEW/Fall 2004/Volume 26, Number 1 80 Table 4 Correlations in Performance Differentials Across Patient Subgroups Mental Illness with Severe Functional Co-occurring Comorbid Category Not At-Risk Impairment Substance Abuse Disability Overall *0.977/0.968 *0.286/0.653 *0.089/0.719 *0.130/0.889 Not At-Risk — *0.282/0.626 *-0.021/0.718 *-0.012/0.811 Mental Illness with Functional Impairment — — *0.025/0.443 *-0.106/0.500 Co-occurring Substance Abuse — — — **-0.271/0.395 *p<0.05. **p<0.10. NOTES: Pearson’s correlation of performance differentials/Spearman’s correlation of ranks. SOURCE: Holmes, A.M., Indiana University-Purdue University Indianapolis and Deb, P., Hunter College, City University of New York: Data based on estimates from the Indiana Managed Care Provider Client Based Data Reports, FYs 1998-1999. least one CMHC performs statistically sig­ nificantly worse for ever y at-risk group considered. In addition, the variability in performance is much greater for clients whose mental illnesses cause severe func­ tional impairment than for other clients. A comparison of the implied ranks across the subgroups considered (also pro­ vided in Table 3) reveals the largest change is between the ranks for the overall client population and the ranks for the group with mental illness causing severe functional impairment. The maximum changes were a 53-percentile increase and a 57-percentile decrease in rankings), fol­ lowed by the group with co-occurring sub­ stance abuse (with a 54-percentile increase and 38-percentile decrease, respectively), followed by the group with a disabling comorbidity (with a 26-percentile increase and a 33-percentile decrease, respectively), and lastly followed by the not at-risk group (with a 13-percentile increase and a 17-per­ centile decrease, respectively). This order­ ing is preser ved when comparing the pro­ por tion of relative ranks that change between the overall and at-risk rankings: rank reversals are nearly nine times more likely between the overall ranks and the ranks for the group with mental illness causing severe functional impairment than between the overall ranks and the ranks for the not at-risk group. Rank reversals for the groups with co-occurring substance abuse and other disabling conditions are, respectively, seven and three times more likely than for the not at-risk group. Correlation coefficients are presented in Table 4. The correlations across estimated performance differentials are statistically insignificant and only weakly positive in size for all at-risk groups considered and the overall client population. In contrast, the correlation between the overall differ­ entials and the not at-risk differentials is 0.98 and highly statistically significant (p=0.000). Thus, it appears that relative center performance overall is determined largely by its ability to ser ve less vulnera­ ble clients. Although correlations between implied ranks are, by contrast, statistically significant, the strength of association is only moderately strong, par ticularly between the overall ranks and the ranks for the group with mental illness causing severe functional impairment. DISCUSSION The President’s New Freedom Commis­ sion on Mental Health (2003) identified out­ come assessment and accountability as unique challenges to the successful func­ tioning of the mental health care system. Problems of asymmetr y of information, in which providers know more about patients’ conditions than either insurers or patients themselves, are particularly acute in mental HEALTH CARE FINANCING REVIEW/Fall 2004/Volume 26, Number 1 81 health care and, combined with incentives for risk selection, can place the neediest patients in peril (Frank and McGuire, 2000). Outcome assessment is needed to ensure these quality problems are not exac­ erbated by managed care deliver y systems that increasingly characterize publicly fund­ ed community mental health care. Access to community-based care for per­ sons with even the most debilitating men­ tal illnesses was advocated by the New Freedom Commission on Mental Health (2003) which recognized that mental health care should be consumer and fami­ ly driven. With this authority comes the responsibility for selecting optimal care from community providers and the need for policymakers to provide the informa­ tion consumers need to make these choic­ es, including those consumers with partic­ ularly severe or complicated conditions. The Commission also reported that dis­ parities exist in access to appropriate men­ tal health care and the burden of mental ill­ ness borne by certain segments of the population. In particular, the care for per­ sons with co-occurring disorders was found to be inadequate. Administrators of public mental health care systems need to consider the extent to which they meet the needs of such at-risk subgroups. Similarly, researchers who undertake effectiveness research to identify best treatment prac­ tices need to consider not only what works best for the typical client, but also whether these same practices are optimal for more vulnerable clients. Standard provider-profiling exercises fail to identify whether some providers are par­ ticularly effective in the treatment of the most vulnerable at-risk clients, and these clients cannot use the resulting informa­ tion to identify optimal choices for people most like themselves (Elliott et al., 2001). While stratified analysis has been suggest­ ed as a possible solution to these problems, strata-specific risk rates typically have unsatisfactor y statistical properties, partic­ ularly for under-represented client groups (Gatsonis et al., 1995). This feature is par­ ticularly undesirable if the most vulnerable at-risk clients are infrequently encountered in CMHCs. In this article, we used a mixed random effects model to evaluate provider perfor­ mance. Compared with standard provider- profiling exercises, such models yield more precise estimates of relative performance, especially when sample sizes are small. In addition, the model easily accommodates interaction terms to evaluate whether per­ formance differentials are robust across various client subgroups. Our results sug­ gest that, for some CMHCs, relative perfor­ mance is significantly dependent on the type of client ser ved; while, on average, centers attained poorer outcomes for at-risk clients than less vulnerable clients, the dis­ crepancy was larger for some centers than others. Furthermore, the estimated perfor­ mance differences for at-risk populations were only moderately related to overall per­ formance differences, with the result that standard provider profiles sometimes failed to identify the most effective providers of care for at-risk clients. We also found that per formance dif ferentials varied much more for clients with mental illnesses that resulted in severe functional impairment than for clients with less severe illnesses. Policymakers need to be aware that in such situations these at-risk clients may have to travel relatively greater distances to obtain quality care, further aggravating disparities in health status and access to health care (Dranove et al., 2003). While the number of centers with statis­ tically significantly better (or worse) out­ comes for various at-risk client groups may be small, the results still have practical relevance. By identifying a small number of exemplar y centers, we have identified HEALTH CARE FINANCING REVIEW/Fall 2004/Volume 26, Number 1 82 centers whose practices, etc., may be wor th emulating by other providers. Similarly, by identifying a small number of centers with subpar performance, we have identified centers where quality improve­ ment initiatives by State agencies could be most effectively applied. Our results can also be used to assess the distribution of quality care across different regions of the State, both overall and with respect to vul­ nerable subgroups. Although we believe our empirical model offers a number of advantages over standard specifications, a number of caveats deser ve mention. First, the analy­ sis is based on only one clinical measure— change in mental health symptoms and mood over a 1-year period. Our relative rankings may discriminate against centers that place more priority on other dimen­ sions of mental health (e.g., community functioning, reduction of substance abuse risk), or that focus on longer or shorter time horizons. Second, our results are based on data for a single State over a sin­ gle year. The external validity of our find­ ings may be limited to the extent that sys­ tem, practice, or client differences may exist across geographical regions or time, although the methods we have presented for detecting differences in provider per­ formance for vulnerable at-risk populations remain valid regardless of setting. Third, our analysis can only consider differences in performance across CMHCs, and not differences within a given CMHC. As a consequence, our results cannot inform consumers and insurers about the relative ef fectiveness of individual providers or treatments. However, given that IDMHA clients must select annually a CMHC to ser ve as a mental health care gatekeeper (rather than a specific provider or treat­ ment protocol), center-level comparisons remain useful. Finally, our results only indi­ cate that differences exist, not why they exist. One of the advantages of the meth­ ods used in this article is that it is possible, in theor y, to incorporate center-level vari­ ables in the mixed random effects specifi­ cation to identify center characteristics associated with better per formance. Empirically, however, our ability to assess the impact of multiple center-level charac­ teristics is limited given the small number of centers on which our analysis is based. The results of this article do provide a crit­ ical first step in quality improvement—hav­ ing identified exceptional centers, policy- makers can use this information in future studies to help determine the staffing, practice patterns or organizational struc­ tures that are associated with superior or inferior outcomes. ACKNOWLEDGMENTS We would like to thank Richard N. Deliberty for his assistance. REFERENCES Clauser, S.B. and Bierman, A.S.: Significance of Functional Status Data for Payment and Quality. Health Care Financing Review 24(3):1-12, Spring 2003. Cuf fel, B.J., Jeste, D.V., Halpain, M., et al.: Treatment Costs and Use of Community Mental Health Ser vices for Schizophrenia by Age Cohorts. American Journal of Psychiatry 153(7):870-876, 1996. Deb, P., Holmes, A.M., and Deliber ty, R.N.: Adjusting for Patient Characteristics and Selection Ef fects in Assessment of Community Mental Health Centers. Medical Care 42(3):251-258, 2004. Deliberty, R. N., Newman, F.L., and Ward, E.O.: Risk Adjustment in the Hoosier Assurance Plan: Impact on Providers. Journal of Behavioral Health Services & Research 28(3):301-318, 2001. Dranove, D., Kessler, D., McClellan, M., et al.: Is More Information Better? The Effects of Report Cards on Health Care Providers. Journal of Political Economy 111(3):555-588, 2003. Elliott, M. N., Swartz, R., Adams, J., et al.: Case-Mix Adjustment of the National CAHPS Benchmarking Data 1.0: A Violation of Model Assumptions? Health Services Research 36(3):555-573, 2001. HEALTH CARE FINANCING REVIEW/Fall 2004/Volume 26, Number 1 83 Ellis, R. P.: Creaming, Skimping and Dumping: Provider Competition on the Intensive and Extensive Margins. Journal of Health Economics 17(5):537-556, 1998. Family and Social Ser vices Administration Biennial Report. Division of Mental Health, Office of Public Policy: Indianapolis, IN. 1997. Family and Social Ser vices Administration The Hoosier Assurance Plan Provider Profile Report Card. Treatment Outcomes for Adults with a Serious Mental Illness. Division of Mental Health, Office of Public Policy: Indianapolis, IN. 2000. Frank, R. G. and McGuire, T.G.: Economics and Mental Health. In Culyer, A.J., and Newhouse, J.P. (eds.): Handbook of Health Economics, Volume 2. Elsevier Press. New York. 2000. Gatsonis, C. A., Epstein, A.M., Newhouse, J. P., et al.: Variations in the Utilization of Coronar y Angiography for Elderly Patients with an Acute Myocardial Infarction. An Analysis Using Hierarchical Logistic Regression. Medical Care 33(6):625-42, 1995. Goldstein, H. and Spiegelhalter, D.J.: League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance. Journal of the Royal Statistical Society (A) 159(3):385-409, 1996. Ireys, H.T., Thornton, C., and McKay, H.: Medicaid Managed Care and Working-Age Beneficiaries with Disabilities and Chronic Illnesses. Health Care Financing Review 24(1): 27-41, Fall 2002. McIntyre, D., Rogers, L., and Heier, E.J.: Over view, Histor y, and Objectives of Per formance Measurement. Health Care Financing Review 22(3):7-21, Spring 2001. New Freedom Commission on Mental Health: Achieving the Promise: Transforming Mental Health Care in America. Final Report. DHHS Pub. No. SMA-03-3832. Rockville, MD. 2003. Newman, F., Deliberty, R., Hodges, K., et al.: The Hoosier Assurance Plan: Repor t of Research/ Implementation Strategy and Assessment Instruments to Support Level of Care Determination. Indiana Division of Mental Health Working Paper. Indianapolis, IN. 1997. SAS Institute Inc.: SAS®/STAT User’s Guide: Version 8. SAS Publishing. Car y, NC. 1999. Vella, F.: Estimating Models with Sample Selection Bias: A Sur vey. The Journal of Human Resources 33(1):127-169, 1998. Ware, J.E., Bayliss, M.S., Rogers, W.H., et al.: Differences in 4-Year Health Outcomes for Elderly and Poor, Chronically Ill Patients Treated in HMO and Fee-for-Ser vice Systems. Results from the Medical Outcomes Study. Journal of the American Medical Association 276(13):1039-1047, 1996. Reprint Requests: Partha Deb, Ph.D., Economics Department, Hunter College, 695 Park Avenue, 1524W, New York, NY 10021. E-mail: partha.deb@hunter.cuny.edu HEALTH CARE FINANCING REVIEW/Fall 2004/Volume 26, Number 1 84