A Pro-performance appraisal system for the university Expert Systems with Applications 37 (2010) 2108–2116 Contents lists available at ScienceDirect Expert Systems with Applications j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / e s w a A Pro-performance appraisal system for the university Jui-Kuei Chen a,1, I-Shuo Chen b,* a Graduate Institute of Futures Studies, Tamkang University, 4F, No. 20, Lane 22, WenZhou Street, Taipei City 10648, Taiwan b Institute of Business & Management, National Chiao Tung University, 4F, No. 20, Lane 22, WenZhou Street, Taipei City 10648, Taiwan a r t i c l e i n f o a b s t r a c t Keywords: Fuzzy analytical network process (FANP) Decision-making trial and evaluation laboratory (DEMATEL) Original performance appraisal system (OPAS) Pro-performance appraisal system (PPAS) 0957-4174/$ - see front matter � 2009 Elsevier Ltd. A doi:10.1016/j.eswa.2009.07.063 * Corresponding author. Tel.: +886 911393602. E-mail addresses: chen3362@ms15.hinet.net (J.- com.tw (I-S. Chen). 1 Tel.: +886 912272961. Due to economic pressures and declining birth rates, universities in Taiwan are seeking ways to evaluate and improve operational performance to acquire a competitive advantage to attract more students. How- ever, current performance evaluation models have been criticized for two reasons. First, the measure- ment criteria currently used are not completely in accordance with the characteristics of different university types, research-intensive university, teaching-intensive university, and professional-intensive university. Second, the models assume independence of measured criteria. Nonetheless, in the real world, such measured criteria are seldom independent. To address these issues, we first reviewed the literature and interviewed Taiwanese higher education experts to integrate critical measurement criteria and develop an original performance appraisal system (OPAS). Next, we adapted a decision-making trial and evaluation laboratory (DEMATEL) method to present complex interdependent relationships and to construct a relation structure among measurement criteria for performance appraisal. A fuzzy analytic network process (FANP) was generated to address the dependence and feedback among each of the mea- surement criteria. Finally, we proposed a Pro-performance appraisal system (PPAS). This study offers a Pro-performance appraisal system (PPAS) to aid in future performance appraisals and improvements for all three university types. � 2009 Elsevier Ltd. All rights reserved. 1. Introduction Higher education is the foundation for fostering high-tech tal- ent, the key factor in increasing national quality, and the main way to upgrade a nation’s competitive status (Fairweather, 2000; Meek, 2000). In recent years, the number of Taiwanese universities has increased to 157 in according to the Taiwanese Ministry of Education (Ministry of Education, 2006). However, the quality and operational performance among them has not increased pro- portionally (Taiwan Assessment and Evaluation Association, 2006). This has been a serious issue for the Taiwanese government and universities (Department of Higher Education, 2004). Visiting and standard procedure evaluations are the two meth- ods the Ministry of Education uses to evaluate universities. There are three types of Taiwanese universities: research-intensive uni- versities, teaching-intensive universities, and profession-intensive universities (Li, 2007). The current measurement criteria utilized are not completely in accordance with the characteristics of these three different university types. This is the main characteristic of the current system that some universities have argued has been ll rights reserved. K. Chen), ch655244@yahoo. unfair. In this study, we argue that since the measurement criteria utilized are assumed to be independent, each criterion may signif- icantly influence the operation performance appraisal results. However, the independence assumption is not consistent with the conditions in the real world. For the reasons above, we propose a more professional perfor- mance appraisal mechanism with more suitable measurement cri- teria for the three university types. In this study, critical measurement criteria were determined by summarizing the litera- ture and interviewing Taiwanese higher education experts. Then, a decision-making trial and evaluation laboratory (DEMATEL) meth- od was adapted to present complex interdependent relationships and to construct a relation structure among measurement criteria for performance appraisal. A fuzzy analytic network process (FANP) was constructed to solve the problem of dependence and feedback among each measurement criterion (Liou, Tzeng, & Chang, 2007). Here, we combined a DEMATEL, and a fuzzy ANP method to form a Pro-performance appraisal system (PPAS). 2. Pro-performance appraisal system (PPAS) Many studies offer insights on performance appraisal in higher education and some studies even develop evaluation models. However, there is inconsistency among different types of univer- sities and the relationships between measurement criteria are not http://dx.doi.org/10.1016/j.eswa.2009.07.063 mailto:chen3362@ms15.hinet.net mailto:ch655244@yahoo.com.tw mailto:ch655244@yahoo.com.tw http://www.sciencedirect.com/science/journal/09574174 http://www.elsevier.com/locate/eswa J.-K. Chen, I-S. Chen / Expert Systems with Applications 37 (2010) 2108–2116 2109 considered. Recent research suggests that the influential factors for operational performance for higher education vary across university types. After summarizing related literature and studies and performing in-depth interviews with higher education experts, we first developed an original performance appraisal system (OPAS). Then, we took a hybrid approach, combining DEMATEL, and Fuzzy ANP, which accounts for complex relation- ships using them to construct a Pro-performance appraisal system (PPAS). 3. Research methods To precisely quantify values with a complex measurement sys- tem is difficult. Nevertheless, such systems can be categorized into subsystems to make it easier to distinguish and evaluate each sub- group (Liou et al., 2007). Here, based on the original performance appraisal system (OPAS), DEMATEL is adapted to assess the inter- relations between each of the measurement criteria. Next, each cri- terion is weighted by conducting fuzzy ANP. Finally, we construct a Pro-performance appraisal system (PPAS) in accordance with above results. 3.1. The decision-making trial and evaluation laboratory (DEMATEL) It is difficult for a decision-maker to evaluate the single effect of a single factor while avoiding interference from the rest of the sys- tem because factors in a complex system may relate to each other directly or indirectly (Liou et al., 2007). In addition, an interdepen- dent system may result in passive positioning. For example, a sys- tem with a clear hierarchical structure may give rise to linear activity with no dependence or feedback, which may cause prob- lems that are distinct from those found in non-hierarchical systems (Tzeng, Chiang, & Li, 2007). The Battelle Geneva Institute created DEMATEL in order to solve difficult issues using interactive man-model techniques to measure qualitative and factor-linked aspects of societal problems (Gabus & Fontela, 1972). DEMATEL has been utilized in many additional contexts, such as industrial planning, deci- sion-making, regional environmental assessing, and even ana- lyzing world problems (Huang, Tzeng, & Ong, 2007). In each case, DEMATEL was used to confirm criteria interdependence and restrict the relationships that affect characteristics within an essential system and its developmental trends (Liou et al., 2007). The DEMATEL method is founded on graph theory. It allows decision-makers to analyze as well as solve visible problems. In doing so, decision-makers can separate multiple measurement cri- teria into cause and effect groups to identify causal relationships. In addition, directed graphs, called digraphs, are more useful than directionless graphs since they depict the directed relationships among subsystems. In other words, a digraph represents a commu- nication network or a domination relationship between entities and their groupings (Huang et al., 2007). The calculation steps of the DEMATEL are as follows (Liou et al., 2007; Yu & Tseng, 2006): Step 1: Calculate the initial average matrix by scores. Sampled experts are asked to point the direct effect based on their perception that each element i exerts on each other element j, as presented by aij, measured on a scale ranging from 0 to 4. No influence is repre- sented by 0, while a very high influence is represented by 4. Based on groups of direct matrices from samples of experts, we can generate an average matrix A in which each element is the mean of the corresponding elements in the experts’ direct matrices. Step 2: Calculate the initial influence matrix. After normalizing the average matrix A, the initial influ- ence matrix D, [dij]n�n, is calculated so that all principal diagonal elements equal zero. In accordance with D, the initial effect that an element exerts and/or acquires from each other element is given. The map depicts a contextual relationship among the elements within a complex system. Each matrix entry can be seen as its strength of influence. As a result, we can easily translate the relationship between the causes and effects of vari- ous measurement criteria into a comprehensive struc- tural model based on the influence degrees using DEMATEL. Step 3: Develop the full direct/indirect influence matrix. The indirect effects of problems decrease as the powers of D increase, e.g. D2, D3, ... , D1, which guarantees con- vergent solutions to the matrix inversion. Therefore, we can generate an infinite series of both direct and indirect effects. Let the (i, j) element of matrix A be pre- sented by aij, then the direct/indirect matrix can be ac- quired by following Eqs. (1)–(4). � D ¼ s A; s > 0 ð1Þ or ½dij�n�n ¼ s½aij�n�n; s > 0; i; j 2f1; 2; . . . ; ng; ð2Þ where 2 3 S ¼ Min 1 max1�i�n Pn j¼1 jaijj ; 1 max1�i�n Pn i¼1 jaijj 6664 7775 ð3Þ and lim m!1 Dm ¼ ½0�n�n where D ¼ ½dij�n�n; 0 � dij < 1: ð4Þ The total influence matrix T can be acquired by utilizing Eq. (5). Here, I is the identity matrix. T ¼ D þ D2 þ���þ Dm ¼ DðI � DÞ�1 when m !1: ð5Þ If the sum of rows and the sum of columns is repre- sented as vector r and c, respectively, in the total influ- ence matrix T, then T ¼ ½tij�; i; j ¼ 1; 2; . . . ; n; ð6Þ R ¼ ½ri�n�1 ¼ Xn j¼1 tij ! n�1 ; ð7Þ c ¼ ½cj� 0 1�n ¼ Xn i¼1 tij ! 1�n ; ð8Þ where the superscript apostrophe denotes transposition. If ri represents the sum of the ith row components of matrix T, then ri represents the sum of both direct and indirect effects of factor i on all other criteria. In addi- tion, if cj represents the sum of the jth column compo- nents of matrix T, then cj presents the sum of both direct and indirect effects that all other factors have on j. Moreover, note that j = i (ri + cj) demonstrates the degree to which factor i affects or is affected by j. Note that if (ri � cj) is positive, then factor i affects other fac- tors, and if it is negative, then factor i is affected by oth- ers (Liou et al., 2007; Tzeng et al., 2007). 2110 J.-K. Chen, I-S. Chen / Expert Systems with Applications 37 (2010) 2108–2116 Step 4: Set the threshold value and generate the impact relations map. Fig. 2. A fuzzy membership function for linguistic variable attributes. Table 1 Definition and membership function of fuzzy number. Fuzzy Linguistic variable Triangular fuzzy Finally, we must develop a threshold value. This value is generated by taking into account the sampled experts’ opinions in order to filter minor effects presented in ma- trix T elements. This step is needed to isolate the rela- tionship structure of the most relevant factors. In accordance with the matrix T, each factor tij provides information about how factor i affects j. In order to de- crease the complexity of the impact relations map, the decision-maker determines a threshold value for the influence degree of each factor. If the influence level of an element in matrix T is higher than the threshold va- lue, which we denote as p, then this element is included in the final impact relations map (IRM) (Liou et al., 2007). number number ~9 Extremely important/preferred (7, 9, 9) ~7 Very strongly important/ preferred (5, 7, 9) ~5 Strongly important/preferred (3, 5, 7) ~3 Moderately important/preferred (1, 3, 5) ~1 Equally important/preferred (1, 1, 3) 3.2. The fuzzy analytical network process (FANP) 3.2.1. Fuzzy set theory Fuzzy set theory was first developed in 1965 by Zadeh when he was attempting to solve fuzzy phenomenon problems, including problems with uncertain, incomplete, unspecific, or fuzzy situa- tions. Fuzzy set theory is more advantageous than traditional set theory when describing set concepts in human language. It allows us to address unspecific and fuzzy characteristics by using a mem- bership function that partitions a fuzzy set into subsets of mem- bers that ‘‘incompletely belong to” or ‘‘incompletely do not belong to” a given subset. 3.2.2. Fuzzy number We order the universe of discourse such that U is a collection of targets, where each target in the universe of discourse is called an element. A fuzzy number ~A is mapped onto U such that a random x ? U is appointed a real number, l~AðxÞ! ½0; 1�. If another element in U is greater than x, we call that element under A. The universe of real numbers R is a triangular fuzzy number (TFN), ~A, which means that for x 2 R; l~AðxÞ 2 ½0; 1�, and l~AðxÞ¼ ðx � LÞ=ðM � LÞ; L � x � M; ðU � xÞ=ðU � MÞ; M � x � U; 0 otherwise; 8>< >: Note that ~A ¼ðL; M; UÞ, where L and U represent fuzzy probabil- ity between the lower and upper boundaries, respectively, as shown in Fig. 1. Assume two fuzzy numbers ~A1 ¼ðL1; M1; U1Þ and ~A2 ¼ðL2; M2; U2Þ; then, (1) ~A1�~A2 ¼ðL1;M1;U1Þ�ðL2;M2;U2Þ¼ðL1 þL2;M1 þM2;U1 þU2Þ (2) ~A1 ~A2 ¼ðL1;M1;U1Þ ðL2;M2;U2Þ¼ðL1 L2;M1 M2;U1 U2Þ; Li > 0; Mi > 0; Ui > 0 (3) ~A1 �~A2 ¼ðL1;M1;U1Þ�ðL2;M2;U2Þ¼ðL1 �L2;M1 �M2;U1 �U2Þ (4) ~A1 ~A2 ¼ðL1; M1; U1Þ ðL2; M2; U2Þ¼ ðL1=U2; M1=M2; U1=L2Þ; Li > 0; Mi > 0; Ui > 0 ~A�11 ¼ðL1;M1;U1Þ �1 ¼ð1=U1;1=M1;1=L1Þ;Li > 0; Mi > 0; Ui > 0 Fig. 1. Triangular fuzzy number. 3.2.3. Fuzzy linguistic variable The fuzzy linguistic variable reflects different aspects of human language. Its value represents the range from natural to artificial language. When the values or meanings of a linguistic factor are being reflected, the resulting variable must also reflect appropriate modes of change for that linguistic factor. Moreover, variables describing a human word or sentence can be divided into numer- ous linguistic criteria, such as equally important, moderately important, strongly important, very strongly important, and extre- mely important, as shown in Fig. 2. Definitions and descriptions are shown in Table 1. For the purposes of this study, the 5-point scale (equally important, moderately important, strongly important, very strongly important and extremely important) is used. 3.2.4. Analytic network process (ANP) The purpose of the ANP approach is to solve problems involving interdependence and feedback between criteria or alternative solutions. ANP is the general form of the analytic hierarchy process (AHP), which has been used in multi-criteria decision-making (MCDM) in order to consider non-hierarchical structures. MCDM has been applied to numerous disciplines (Huang, Tzeng, & Ong, 2005). The beginning stage of an ANP uses pair-wise comparisons of the measured criteria to form a super matrix. The relative impor- tance-values of pair-wise comparisons can be categorized from 1 Fig. 3. The general form of the super matrix (Liou et al., 2007; Yu and Tseng, 2006). Table 2 Two simple cases. Number Case 1 Case 2 Structure type Matrix forming Table 4 The average initial direct-relation 10�10 matrix A. D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D1 0 3.17 1.02 0.24 0.11 0.43 0.11 0.30 2.14 0.10 D2 0.23 0 1.17 0.01 0.13 0.32 0.01 0.00 2.45 0.15 D3 3.33 3.21 0 1.12 1.86 3.37 0.23 0.14 1.07 0.40 D4 3.16 2.42 1.12 0 0.31 3.23 1.06 1.12 3.27 0.41 D5 0.13 0.08 0.03 1.03 0 2.18 1.22 1.19 3.36 0.27 D6 0.03 2.09 0.01 1.19 1.01 0 1.11 1.16 3.17 1.05 D7 2.21 0.12 2.33 3.05 3.42 3.18 0 1.10 3.16 3.31 D8 1.00 0.06 1.96 1.13 3.24 3.19 2.87 0 3.55 1.11 D9 1.23 0.23 0.05 1.06 2.04 1.31 0.38 0.21 0 0.20 D10 2.12 0.10 0.03 1.10 2.27 1.45 2.01 3.42 1.28 0 J.-K. Chen, I-S. Chen / Expert Systems with Applications 37 (2010) 2108–2116 2111 to 9 in order to represent pairs of equal importance (1) to extreme inequality in importance (9) (Saaty, 1980). Fig. 3 shows the general form of the super matrix where cm represents the mth cluster, emn represents the nth element in the mth cluster, and wij is the prin- cipal eigenvector measuring the influence of the jth cluster ele- ments on the ith cluster elements. In addition, if the jth cluster has no influence on the ith cluster, then wij = 0 (Yu & Tseng, 2006). The form of the super matrix depends on the variety of its struc- ture. In order to demonstrate how the structure is affected by the super matrix, Huang et al. (2005), Yu and Tseng (2006), and Liou et al. (2007) offer two simple cases that both involve three clusters to illustrate how to form a super matrix in accordance with differ- ent structures (see Table 2). Case 1 is much simpler than case 2, and based on each structure, the super matrices are given under each. Next, the weighted super matrix is generated by transforming all column sums to unity (Huang et al., 2005; Yu & Tseng, 2006). Then, we use the weighted super matrix to generate a limiting Table 3 An original 10-dimensional performance appraisal system (OPAS). System Measurement dimensions The original performance appraisal system (OPAS) Learning performance (D1) Life development (D2) Learning behavior (D3) Quality of teaching (D4) Research performance (D5) Professional skill performanc Organizational development External interactions (D8) School prestige (D9) Budget handling performance super matrix by using Eq. (9) to calculate global weights (Huang et al., 2005). lim k�1 wk ð9Þ Measurement criteria The enrollment rate of new students (C1) The graduation rate of current students (C2) The job acquiring rate of students (C3) The relations degree of students’ major and jobs (C4) Rate of continuing education (C5) Student innovative/creative ability (C6) The rate of borrowing books (C7) The rate of club participation (C8) The rate of course performance appraisal (C9) The rate of practical training course opening (C10) The rate of optional course opening (C11) Number of plans given by NSC (C12) The job promotion rate of faculty (C13) Number of articles published in international journals (C14) e (D6) Number of thesis winning of students (C15) Number of patents (C16) (D7) The performance of occupational refresher courses (C17) The supplemental budget of faculty research (C18) The budget of scholarships (C19) The budget of industry-university relations (C20) The budget of international relations (C21) Number of international conferences (C22) The satisfaction degree of students and faculty (C23) Employee turnover (C24) Score given by Ministry of Education (C25) (D10) The unit cost of each student (C26) The rate of tuition and schooling in school income (C27) The rate of government supplement in school overall income (C28) Table 5 Total influence T. D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D1 0.029 0.169 0.064 0.032 0.038 0.058 0.020 0.027 0.201 0.017 D2 0.033 0.020 0.059 0.017 0.031 0.043 0.011 0.010 0.203 0.015 D3 0.195 0.214 0.035 0.095 0.139 0.223 0.049 0.045 0.180 0.047 D4 0.202 0.184 0.092 0.059 0.097 0.236 0.094 0.092 0.286 0.058 D5 0.053 0.043 0.029 0.094 0.068 0.171 0.094 0.088 0.246 0.045 D6 0.052 0.128 0.033 0.100 0.114 0.080 0.090 0.089 0.243 0.078 D7 0.210 0.109 0.200 0.228 0.278 0.306 0.088 0.137 0.355 0.203 D8 0.093 0.077 0.133 0.141 0.260 0.283 0.200 0.069 0.336 0.110 D9 0.082 0.033 0.017 0.075 0.213 0.106 0.042 0.034 0.072 0.027 D10 0.158 0.063 0.053 0.121 0.201 0.186 0.156 0.205 0.220 0.052 Table 6 The sum of influences on measurement dimensions. Measurement dimensions ri + ci ri � ci D1 1.709 �0.503 D2 1.417 �0.661 D3 1.894 0.549 D4 2.362 0.441 D5 2.281 �0.419 D6 2.700 �0.688 D7 2.912 1.232 D8 2.494 0.901 D9 2.838 �1.615 D10 2.067 0.763 Fig. 4. The impact relations map of this study. 2112 J.-K. Chen, I-S. Chen / Expert Systems with Applications 37 (2010) 2108–2116 In this step, if the super matrix shows signs of cyclicity, then more than one limiting super matrix must exist. In this case, the Cesaro sum must be calculated to obtain the priority order of the multiple super matrices (Yu & Tseng, 2006). The Cesaro sum is calculated using Eq. (10) (Huang et al., 2005; Yu & Tseng, 2006) lim k�1 1 N � �XN k¼1 wk ð10Þ Table 7 The example of the local weight of criteria 23–25 under the effect of criteria 3. Measurement criteria C23 C24 C23 1.000 1.000 3.000 0.126 0 C24 4.583 6.708 7.937 1.000 1 C25 2.297 4.711 6.433 0.116 0 Eq. (10) calculates the average effect of a limiting super matrix. Otherwise, the super matrix would be raised to a large power to generate the priority weights (Liou et al., 2007; Yu & Tseng, 2006). 4. Empirical study of Pro-performance appraisal system (PPAS) Due to economic pressures and declining birth rates, Taiwanese universities are seeking ways to improve operational performance to acquire a competitive advantage and attract more students. In the previous section, we explained that performance appraisal cri- teria vary greatly from one university to another. In addition, crite- ria can affect each other. There are three university types in Taiwan, the research-intensive universities, the teaching-intensive universities, and the professional-intensive university, and the per- formance improvement and evaluation focuses are different for each university type. To overcome the challenges outlined and to meet the requirements above, we take the interrelationship be- tween criteria and the types of universities into account while developing a Pro-performance appraisal system (PPAS). 4.1. Forming an original performance appraisal system (OPAS) Due to the different measurement criteria involved, construct- ing a Pro-performance appraisal system (PPAS) for universities is complicated. The Pro-performance appraisal system (PPAS) must allow for interdependence and be in accord with real practice. In this study, we first categorized related measurement criteria from existing studies (Cameron, 1978; Lysons, Hatherly, & Mitchell, 1998). For a detailed summary of the measurement criteria, refer to Mei and Lee (2006). Twenty-five higher education experts were also consulted: ten from research-intensive universities, seven from professional-intensive universities, and eight from teaching- intensive universities. The works of the National Science Council (NSC) were consulted to form an original ten-dimensional perfor- mance appraisal system (OPAS) (Learning performance (D1), Life development (D2), Learning behavior (D3), Quality of teaching (D4), Research performance (D5), Professional skill performance (D6), Organizational development (D7), External interactions (D8), School prestige (D9), and Budget handling performance (D10)). Each category includes 2–3 measurement criteria (Table 3). A questionnaire was adapted and given to 66 experts from the three university types. Of the 66 questionnaires, 41 were used for this study: sixteen from the research-intensive universities (16/ C25 Local weight .149 0.218 0.155 0.212 0.435 0.075 .000 3.000 5.433 7.504 8.631 0.742 .133 0.184 1.000 1.000 3.000 0.182 Table 8 The un-weighted matrix. C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18 C19 C20 C21 C22 C23 C24 C25 C26 C27 C28 C1 0 0 0 0 0 0 0 0 0.09 0.11 0.08 0 0 0 0 0 0.03 0.07 0.05 0 0 0 0 0 0 0 0 0 C2 0 0 0 0 0 0 0 0 0.03 0.02 0.02 0 0 0 0 0 0.02 0.006 0.03 0 0 0 0 0 0 0 0 0 C3 0 0 0 0 0 0.06 0.08 0.11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C4 0 0 0 0 0 0.04 0.03 0.06 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C5 0 0 0 0 0 0.03 0.07 0.04 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.05 0.04 0.06 0 0 0 0 0 0 0 0 0 C7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.02 0.04 0.03 0 0 0 0 0 0 0 0 0 C8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.01 0.01 0.02 0 0 0 0 0 0 0 0 0 C9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.07 0.09 0.07 0 0 0 0 0 0 0 0 0 C10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.06 0.07 0.08 0 0 0 0 0 0 0 0 0 C11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.03 0.03 0.04 0 0 0 0 0 0 0 0 0 C12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.11 0.11 0.1 0.17 0.2 0.16 0.24 0.31 0.27 0.21 0.16 0.15 C13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.09 0.14 0.08 0.15 0.11 0.13 0.18 0.22 0.24 0.11 0.14 0.13 C14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.13 0.12 0.19 0.18 0.23 0.24 0.58 0.47 0.49 0.24 0.19 0.16 C15 0 0 0 0 0 0.20 0.28 0.31 0.26 0.22 0.27 0 0 0 0 0 0.07 0.05 0.06 0.08 0.07 0.07 0 0 0 0 0 0 C16 0 0 0 0 0 0.67 0.54 0.48 0.41 0.36 0.36 0 0 0 0 0 0.10 0.11 0.09 0.13 0.08 0.16 0 0 0 0 0 0 C17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.06 0.07 0.06 0 0 0 0 0 0 C18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.09 0.09 0.08 0 0 0 0 0 0 C19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.04 0.06 0.03 0 0 0 0 0 0 C20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.09 0.12 0.13 C21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.1 0.13 0.11 C22 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.15 0.11 0.13 C23 0.05 0.16 0.07 0.08 0.21 0 0 0 0.02 0.04 0.02 0.17 0.16 0.16 0.19 0.17 0.01 0.004 0.005 0.01 0.02 0.01 0 0 0 0.01 0.01 0.03 C24 0.68 0.61 0.57 0.74 0.55 0 0 0 0.13 0.17 0.14 0.62 0.48 0.51 0.43 0.59 0.06 0.03 0.04 0.03 0.02 0.02 0 0 0 0.03 0.06 0.07 C25 0.27 0.23 0.36 0.18 0.24 0 0 0 0.06 0.08 0.11 0.21 0.36 0.33 0.38 0.24 0.04 0.06 0.05 0.06 0.05 0.04 0 0 0 0.06 0.08 0.09 C26 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.01 0.007 0.001 0 0 0 0 0 0 0 0 0 C27 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.03 0.01 0.001 0 0 0 0 0 0 0 0 0 C28 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.02 0.003 0.003 0 0 0 0 0 0 0 0 0 J.-K . C h en , I-S. C h en /E xp ert System s w ith A p p lica tio n s 3 7 (2 0 1 0 ) 2 1 0 8 – 2 1 1 6 2 1 1 3 Table 9 The weighted matrix. C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18 C19 C20 C21 C22 C23 C24 C25 C26 C27 C28 C1 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 0.0024 C2 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 0.0007 C3 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 0.0021 C4 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 0.0011 C5 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 0.0012 C6 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 0.0004 C7 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 C8 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 C9 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 0.0077 C10 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 0.0071 C11 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 0.0034 C12 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 0.1659 C13 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 0.1303 C14 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 0.2439 C15 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 0.0762 C16 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 0.1371 C17 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 0.0069 C18 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 0.0095 C19 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 0.0047 C20 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 C21 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 0.0210 C22 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 0.0241 C23 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 0.0181 C24 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 0.0738 C25 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 0.0406 C26 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 C27 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 0.0002 C28 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 2 1 1 4 J.-K . C h en , I-S. C h en /E xp ert System s w ith A p p lica tio n s 3 7 (2 0 1 0 ) 2 1 0 8 – 2 1 1 6 Fig. 5. The impact of direction map of measurement criteria. J.-K. Chen, I-S. Chen / Expert Systems with Applications 37 (2010) 2108–2116 2115 36), seven from teaching-intensive universities (7/11), and 18 from professional-intensive universities (18/19). The experts ranked the level of interrelationships among each measurement dimension on a scale from 0 to 4 (No influence, 0, to Very high influence, 4), and the importance between each measurement performance appraisal criterion on a 5-point scale described in Table 1. Table 10 Study result summary. System Measurement dimensions Meas The original performance appraisal system (OPAS) Learning performance (D1) The e The g Life development (D2) The j The r Rate Learning behavior (D3) Stude The r The r Quality of teaching (D4) The r The r The r Research performance (D5) Num The j Num (C14) Professional skill performance (D6) Num Num Organizational development (D7) The p (C17) The s The b External interactions (D8) The b The b Num School prestige (D9) The s Empl The s Budget handling performance (D10) The u The r The r incom 4.2. Evaluating the interrelationships of dimensions Here, the questionnaires from 41 higher educational experts were used to determine the level of relationship among each dimension. With the questionnaire data, we formed the average initial direct-relation 10�10 matrix A using pair-wise comparisons as shown in Table 4. Next, using Eqs. (1)–(3), we acquired the normalized direct- relation D from matrix A. After that, Eq. (5) was adapted to obtain total influence T (Table 5). Finally, we used Eqs. (7) and (8) to determine the total influence given to and received by each mea- surement dimension. The results are provided in Table 6. To avoid relationships too complex for our system, a threshold value under 0.20 was adopted after consulting with higher educa- tion experts. Fig. 4 shows the impact relations map (IRM). 4.3. Weighting the criteria and constructing a Pro-performance appraisal system (PPAS) After calculating the level of interrelationships among each measurement dimension, we utilized fuzzy ANP to acquire the weights of each measurement criterion. The importance of rela- tionships between measurement criteria is paralleled based on the impact relations map above. Such pair-wise comparisons are in accordance with the data in Table 5. In Table 7, we provide an example of the local weight calculated by the principle eigenvector of comparison, specifically of criteria 23–25 when under the effect of criterion 3. The complete result is an un-weighted super matrix, shown in Table 8. Next, we calculate the limiting power of the un-weighted matrix until it remains stable using Eq. (9). The result is shown urement criteria Global weights Ranking nrollment rate of new students (C1) 0.0024 C14 raduation rate of current students (C2) 0.0007 C12 ob acquiring rate of students (C3) 0.0021 C16 elations degree of students’ major and jobs (C4) 0.0011 C13 of continuing education (C5) 0.0012 C15 nt innovative/creative ability (C6) 0.0004 C24 ate of borrowing books (C7) 0.0003 C25 ate of club participation (C8) 0.0001 C22 ate of course performance appraisal (C9) 0.0077 C21 ate of practical training course opening (C10) 0.0071 C20 ate of optional course opening (C11) 0.0034 C23 ber of plans given by NSC (C12) 0.1659 C18 ob promotion rate of faculty (C13) 0.1303 C9 ber of articles published in international journals 0.2439 C10 ber of thesis winning of students (C15) 0.0762 C17 ber of patents (C16) 0.1371 C19 erformance of occupational refresher courses 0.0069 C11 upplemental budget of faculty research (C18) 0.0095 C1 udget of scholarships (C19) 0.0047 C3 udget of industry-university relations (C20) 0.0210 C5 udget of international relations (C21) 0.0210 C4 ber of international conferences (C22) 0.0241 C2 atisfaction degree of students and faculty (C23) 0.0181 C6 oyee turnover (C24) 0.0738 C7 atisfaction degree of industries (C25) 0.0406 C27 nit cost of each student (C26) 0.0001 C28 ate of tuition and schooling in school income (C27) 0.0002 C8 ate of government supplement in school overall e (C28) 0.0001 C26 Subsystem Appraisal Dimension Appraisal Criteria Core appraisal system Organizational development The employee turnover The percentage of promotion Academy performance Number of articles published in international journals Number of patents Number of winning student thesis External behavior Number of plans given by NSC The satisfaction degree of industries Financial support and budget planning Support appraisal system Fig. 6. A Pro-performance appraisal system (PPAS). 2116 J.-K. Chen, I-S. Chen / Expert Systems with Applications 37 (2010) 2108–2116 in Table 9. Last, using the above result, the impact-direction map is derived and is shown in Fig. 5. In Table 10, we summarize all of the above results. We propose a novel Pro-performance appraisal system (PPAS) in which the top seven most heavily weighted measurement criteria are extracted from the whole set of criteria. We argue that the operational per- formance appraisal and improvement considering only the highly weighted criteria will be better than that using all measurement criteria. Also, the content of organizational development (D7) is chosen due to its highest influential degree on the performance improvement conducting (as Fig. 6). In the PPAS we propose, a core appraisal system, containing three dimensions with seven mea- surement criteria, represents the key suitable items for all three university types to conduct operational performance appraisals and improvements with accuracy. This appraisal system supports financial and budget planning for all three university types to eval- uate appropriate financial decisions to support key initiatives and to control the budget. 5. Conclusions With increasing economic pressures and declining birth rates, Taiwanese universities are seeking ways to improve operational performance to acquire competitive advantages and to attract more students. Current performance appraisal models are inade- quate due to inconsistencies among different types of universities, inconsistencies in measurement criteria, and the treatment of interdependent criteria. To address these problems, we first re- viewed previous research and interviewed several higher educa- tion experts. Then, we combined a DEMATEL, and a fuzzy ANP to construct a Pro-performance appraisal system (PPAS), considering the interdependence and the relative weights of measurement cri- teria and the characteristics of three university types. The PPAS will be helpful in performing future performance appraisals and suggesting improvements for all three university types. References Cameron, K. S. (1978). Measuring organizational effectiveness in institutions of higher education. Administrative Science Quarterly, 23(4), 604–632. Department of Higher Education (2004). Operation book. Taipei: Taiwan Assessment and Evaluation Association. Fairweather, J. S. (2000). Diversification or homogenization: How markets and governments combine to shape American higher education. Higher Education Policy, 13, 79–98. Gabus, A., & Fontela, E. (1972). World problems, an invitation to further thought within the framework of DEMATEL. Geneva, Switzerland: Battelle Geneva Research Center. Huang, J. J., Tzeng, G. H., & Ong, C. S. (2005). Multidimensional data in multidimensional scaling using the analytic network process. Pattern Recognition Letters, 26(6), 755–767. Huang, J. J., Tzeng, G. H., & Ong, C. S. (2007). Marketing segmentation using support vector clustering. Expert Systems with Applications, 32(2), 313–317. Li, U. C. (2007). Enter university by 2.8 point each subject. Retrieved August 18, 2007, from http://news.msn.com.tw/print.aspx?id=210245. Liou, J. H., Tzeng, G. H., & Chang, H. C. (2007). Airline safety measurement using a hybrid model. Journal of Air Transport Management, 13(2007), 243–249. Lysons, A. F., Hatherly, D. J., & Mitchell, D. A. (1998). Comparison of measures of organizational effectiveness in U.K. higher education. The International Journal of Higher Education and Educational Planning, 36(1), 1–19. Meek, V. L. (2000). Diversity and marketisation of higher education: Incompatible concepts. Higher Education Policy, 13, 23–39. Mei, Y. F., & Lee, L. S. (2006). A study to develop performance indicators for colleges of technology in Taiwan. Journal of Education Research, 1(1), 25–67. Ministry of Education (2006). Department of Statistics. Retrieved November 23, 2008, from http://www.edu.tw/statistics/. National Science Council (2008). Fast connection. Retrieved November 23, 2008, from http://web1.nsc.gov.tw/mp.aspx. Saaty, J. T. (1980). The analytic hierarchy process. Columbus: McGraw-Hill. Taiwan Assessment and Evaluation Association (2006). Evaluation Bimonthly, 1, 48– 49. Tzeng, G. H., Chiang, C. H., & Li, C. W. (2007). Evaluating intertwined effects in e-learning programs: A novel hybrid MCDM model based on factor analysis and DEMATEL. Expert Systems with Applications, 32, 1028–1044. Yu, R., & Tseng, G. H. (2006). A soft computing method for multi-criteria decision making with dependence and feedback. Applied Mathematics and Computation, 180(2006), 63–75. http://news.msn.com.tw/print.aspx?id=210245 http://www.edu.tw/statistics/ http://web1.nsc.gov.tw/mp.aspx A Pro-performance appraisal system for the university Introduction Pro-performance appraisal system (PPAS) Research methods The decision-making trial and evaluation laboratory (DEMATEL) The fuzzy analytical network process (FANP) Fuzzy set theory Fuzzy number Fuzzy linguistic variable Analytic network process (ANP) Empirical study of Pro-performance appraisal system (PPAS) Forming an original performance appraisal system (OPAS) Evaluating the interrelationships of dimensions Weighting the criteria and constructing a Pro-performance appraisal system (PPAS) Conclusions References