key: cord-0045755-kh3musfl authors: Sałabun, Wojciech; Urbaniak, Karol title: A New Coefficient of Rankings Similarity in Decision-Making Problems date: 2020-06-15 journal: Computational Science - ICCS 2020 DOI: 10.1007/978-3-030-50417-5_47 sha: ce7ff7a79e44b02a63df2cb0beed9046f7acde10 doc_id: 45755 cord_uid: kh3musfl Multi-criteria decision-making methods are tools that facilitate and help to make better and more responsible decisions. Their main objective is usually to establish a ranking of alternatives, where the best solution is in the first place and the worst in the last place. However, using different techniques to solve the same decisional problem may result in rankings that are not the same. How can we test their similarity? For this purpose, scientists most often use different correlation measures, which unfortunately do not fully meet their objective. In this paper, we identify the shortcomings of currently used coefficients to measure the similarity of two rankings in decision-making problems. Afterward, we present a new coefficient that is much better suited to compare the reference ranking and the tested rankings. In our proposal, positions at the top of the ranking have a more significant impact on the similarity than those further away, which is right in the decision-making domain. Finally, we show a set of numerical examples, where this new coefficient is presented as an efficient tool to compare rankings in the decision-making field. Decision making is an integral part of human life. Every day, every person is faced with different kinds of decision-making problems, which can affect both professional and private life. An example of a decision-making problem can be a change of legal regulations in the state, choice of university, purchase of a new car, determination of the amount of personal income tax, selection of a suitable location for the construction of a nuclear power plant, adoption of a plan of research, or the sale or purchase of stock exchange shares. In the majority of cases, decision-making problems are based on many, often contradictory, decision-making criteria. Therefore multi-criteria decision-analysis the similar coefficients presented in [6, 10, 13] . For this purpose, numerical experiments are discussed. The rest of the paper is organized as follows: In Sect. 2, some basic preliminary concepts are discussed. Section 3 introduces a new coefficient of rankings similarity in the decision-making problems. In Sect. 4, the practical feasibility study of the WS coefficient is discussed. In Sect. 5, we present the summary and conclusions. An important issue is how to compare the correctness of the order of the two rankings. The simplest method is to check whether the rankings are consistent or inconsistent. Such an approach is not sufficient and can be used almost exclusively for 2 or 3 elementary rankings [27] . The much more common approach is to use one of the coefficients of monotonous dependence of two variables, where the obtained rankings for a set of considered alternatives are our variables. The most commonly used symmetrical coefficient of such dependence is the Spearman's coefficient [9, 17, 18, 23] , which is expressed by the following formula (1): where d i is defined as the difference between the ranks d i = R xi − R yi and n is the number of elements in the ranking. The Spearman's coefficient is interpreted as a percentage of the rank variance of one variable, which is explained by the other variable [31] . The most frequently used asymmetrical monotonous coefficients of two variables are Kendall [12, 20] and Goodman-Kruskal coefficients [12, 14] . They are expressed in formulas (2) and (3) respectively: where N s is the number of compatible pairs, N d is the number of non-compliant pairs, and n is the number of all pairs. The Kendall and Goodman-Kruskal coefficients, unlike Spearman, are interpreted in terms of probability. They represent the difference between the probability that the compared variables will be in the same order for both variables and the probability that they will be in the opposite order. The presented coefficients are the most frequently used measures of the analysis of the rankings similarity in decision-making problems [9, 12, 17, 18, 23] . However, we want to indicate a significant shortcoming, which is related to the place of difference occurrence. The idea of measuring the rankings similarity is not new and has been the subject of many works [11, 30] . Particularly interesting in the context of the presented approach are works related to Blest's measure of rank correlation v and the weighted rank measure of correlation r w [6, 10, 13] . They are expressed in formulas (4) and (5) respectively: The presented coefficients (1-3) are regardless of whether the error occurs at the top or bottom; the values of the factors will be identical. In Table 1 , the simple example shows five rankings, including one reference (R x ) and four test rankings (R . The test rankings were created by a change in the correct ranking of the two adjacent alternatives. We want to remind that the rankings are determined to choose the best possible solution, and the value of the preferences decreases with each position in the ranking. The difference at the top should be more significant than an error at the bottom of the ranking. The exchange of alternative locations from the first and second position is a more considerable error than the swap of the second and third position. However, the values of the coefficients indicate that similarity of the test rankings to the reference ranking is the same for all test sets. The new ranking similarity factor should be resistant to the situation described in the previous section, and at the same time, should be sensitive to significant changes in the ranking. Besides, this factor should be easy to interpret, and its values should be limited to a specific interval. We assumed that the new indicator should be strongly related to the difference between two rankings on particular positions. An additional assumption is that the top has a more significant influence on similarity than the bottom of the ranking. Based on these assumptions, a new indicator was developed, which can be presented as (6): where WS is a value of similarity coefficient, N is a length of ranking, R xi and R yi mean the place in the ranking for i − th element in respectively ranking x and ranking y. The proof of convergence for the WS factor is quite simple. The formula (6) can be divided into two main components. The first one (7) is responsible for making the WS value dependent on the position in the reference ranking (R x ). y ) with the calculated correlation factors and proposed WS coefficient for the set of five alternatives (A1 − A5), each having a different position in the ranking. Rx R Coefficients rs 0.9000 0.9000 0.9000 0. We are dealing with a geometric series which is convergent. As proof, we can calculate a trivial limit. The second component (9) determines to what extent the difference in rankings affects the similarity of rankings. This value can be obtained from zero (the positions are identical) to one. If we multiply the (7) by (9) then this series cannot be higher than one. Therefore, it is clear that the WS coefficient can only take values from zero to one. We can compare all coefficients for a simple example in Table 1 . The W S, r w , and v coefficients take into account the position of the error occurrence, and the rest of them remain the same regardless of where the error occurs. In the next section, other tests comparing the performance of the indicators will be presented and discussed. The first experiment here presents tied ranks, i.e., the same values in the ranking. It happens when two alternatives get the same place. For example, if two decision variants receive the first place together, the ranking will contain a value of 1.5 for y ) with the calculated correlation factors and proposed WS coefficient for the set of five alternatives (A1 − A5), where one pair has the same position in the ranking. Rx R Table 2 shows the results of calculations for the five-element ranking, where the different location of tied pairs is considered. One more again, W S, r w , and v coefficients show the change of value together with the change of position on which there are the tied pairs. It is a property that was identified as a significant drawback of the currently used methods, i.e., ρ Spearman, τ Kendall, and γ Goodman-Kruskal. Ranking R (4) y is more similar than R (1) y because full correctness occurs on the first three positions and not on the last three. Another simple experiment consists in creating test rankings, where successive rankings differ from the base ranking by the alternative indicated as the best. The results for the five-element ranking are shown in Table 3 . Replacing the best option with the worst in all coefficients results in a negative value result (except W S). It means a negative correlation, which is not trivial to interpret in decision-making problems. Besides, interesting is the case of the R (3) y ranking because it obtained a total lack of correlation (for τ and G coefficients). It means that the order of the base and second rankings is utterly unrelated to each other. It is a confirmation that these classical rank coefficients do not examine the similarity of the two rankings thoroughly. In general, all coefficients assess test rankings against the base ranking in a somewhat similar way. The rationale for this is that the three positions in the ranking have been indicated flawlessly. The last example in this subsection examines the r w coefficients in two cases, i.e., for test rankings R (1) y −−R (2) y and R (3) y −−R (4) y ). Once again, the R x ranking is used as a reference point. The detailed results are presented in Table 4 . Rankings R (1) y and R (2) y have equal values for most of the coefficients, where only W S and v are exceptions. Ranking R (1) y is significantly better than ranking y . Even though the A 5 alternative has been identified as the best in R (1) y . The rest of this ranking has been correctly identified according to the right order. However, in the R (2) y , the best alternative was wrongly rated as the worst. Therefore, the best alternative (A 1 ) has a chance to be chosen in the first case and not in the second. These rankings cannot be evaluated as being the same. This shows the superiority of W S and v coefficients in decision-making ranks analysis. Rankings R (3) y and R (4) y show greater variability of coefficients, i.e., the ranking R (3) y has a coefficient value less, equal to or greater than the ranking R (4) y . It all depends on which coefficient is taken into account, but W S and v again point to the superiority of the ranking R In this subsection, we want to indicate the impact of the ranking size on the achieved value of the indicator. Figure 1 shows comparisons of W S, r s , r w , and v coefficients. Only alternatives on the first and second positions have been replaced. It is a consequence of the conclusion drawn from Table 1 . We take into account the rankings with the number from 5 to 50 elements. We can observe that with the increased ranking size, the similarity with the assumed assumptions increases. The Ws coefficient is characterized by the greatest variability, depending on the size of the ranking. Figure 2 shows the changes in the WS value when replacing the best elements with the second, third, fourth, and fifth ones (the position of one adjacent pair is swapped). As we can see, the W S values decrease accordingly, as the quality of the rankings decreases as the best solution moves away from the top of the ranking. In the next step, we attempted to visualize the distributions for three indicators. Figure 3 presents the distribution of the τ Kendall factor for all possible permutations of the sets of five, six, seven, eight, nine, and ten elements. Figures 4 and 5 show the distribution of the ρ Spearman coefficient and the W S coefficient, respectively. The shape of the ρ Spearman values is smoother than the τ Kendall. Both indicators have a symmetrical distribution, unlike the W S coefficient. The problem may be the interpretation of the WS value, because it is a new approach. However, the question arises when the similarity of the W S coefficient is low, medium, and high. A statistical analysis of the distribution of the W S should be carried out to define three appropriate linguistic terms and answer on this research question. All possible permutations and values of the WS coefficient are determined for ranks of the size from 3 to 10 elements. Based on the obtained values, we calculated basic statistics, which are presented in Table 5 . For larger rankings, statistics are based on random samples of 100,000 rankings. Both population and random sampling data are used. Note the convergence of the arithmetic mean, standard deviation, and typical value ranges. The biggest differences concern the arithmetic mean, and it is equal to 0.207 (for a ranking of 10 and 1000 elements). Fig. 6 . The definitions of three linguistic terms, i.e., low, medium, and high similarity of rankings by using trapezoidal fuzzy numbers. Based on the analysis of typical values, i.e. interval of [x − S x ;x + S x ], we identified the linguistic terms low, medium and high similarity of rankings. It can indeed be said that if the WS is less than 0.234, then the similarity is low. If the value is higher than 0.808, then the similarity is high. The medium of likeness, which corresponds to a typical value, belongs to the range from 0.352 to 0.689. The remaining values are values where we can talk about a partial belonging to linguistic concepts according to the theory of fuzzy sets, or just low/medium and medium/high concept can be used. Detailed definitions are presented in Fig. 6 . Linguistic values are important because they can be used to evaluate the adjustment of the reference and test rankings. The main contribution of the paper is a proposal of the new coefficient of the rankings similarity. For this purpose, the short analysis of classical factors are presented, and some of their shortcomings are emphasized. The most critical is the equality of the values of the classical coefficients in case the ranking error concerns the replacement of a pair of adjacent alternatives ( Table 1 ). The paper presents a theoretical foundation of proposed WS coefficient, which ensures that a new factor is free of identified shortcomings. The results of numerical experiments compare all analyzed coefficients and their correctness, i.e., ρ Spearman, τ Kendall, G Goodman-Kruskal, and WS coefficients. Then, the distributions of τ Kendall, ρ Spearman, and W S coefficients were compared. WS values can be used to measure similarity of rankings. Finally, three linguistic concepts were formulated for the low, medium, and high similarity of the two rankings. The properties of the WS coefficient indicate that it is a useful tool for comparing the similarity of rankings and is better suited for this purpose than the currently used correlation coefficients. During the research, some improvement areas have been identified. The future work directions should concentrate on: -further comparing between existing coefficients and the proposed W S; -testing the use of the WS coefficient in real-life examples; -detection and correction of WS coefficient shortcomings; -adaptation of the proposed coefficient to uncertain (fuzzy) rankings. Multicriteria modelling for a repair contract problem based on utility and the ELECTRE I method Evaluation of power plants technologies using multicriteria methodology MACBETH TOPSIS-based service arbitration for autonomic internet of things Ranking of suppliers with MCDA technique and probabilistic criteria Application of fuzzy probabilistic TOPSIS on a multi-criteria decision making problem Theory & methods: Rank correlation -an alternative measure A comparison of ranking methods for classification algorithm selection A preventive maintenance decision model based on multicriteria method PROMETHEE II integrated with Bayesian approach A comparative analysis of multi-criteria decision-making methods A weighted rank measure of correlation Comparing top k lists Decision making with uncertainty using hesitant fuzzy sets On blest's measure of rank correlation Measures of association for cross classifications Selecting a best compromise direction for a powered wheelchair using PROMETHEE Use of multi-criteria decision analysis to make collection management decisions Are multi-criteria decision-making tools useful? An experimental comparative study of three methods Multiple-criteria comparative analysis of magnetic resonance imaging systems A new approach in the evaluation of team chess championships rankings A new measure of rank correlation A study on stock ranking and selection strategy based on UTA method under the condition of inconsistence Assessing the quality of different MCDA methods Comparative analysis of mcdm methods for the assessment of sustainable housing affordability Evaluation of rankings with regard to the possible number of agreements and conflicts The characteristic objects method: a new distance-based approach to multicriteria decision-making problems Handling data uncertainty in decision making with comet Comparative analysis of MCDM methods for the assessment of mortality in patients with acute coronary syndrome Deciding endemic area of dengue fever using simple multi attribute rating technique exploiting ranks A refined DRSA model for the financial performance prediction of commercial banks A weighted Kendall's tau statistic The proof and measurement of association between two things AHP, gray correlation, and TOPSIS combined approach to green performance evaluation of design alternatives Generalised framework for multi-criteria method selection Comparison of AHP and Monte Carlo AHP under different levels of uncertainty A stochastic ANP-GCE approach for vulnerability assessment in the water supply system with uncertainties Virtual network embedding using node multiple metrics based on simplified ELECTRE method The work was supported by the National Science Centre, Decision number UMO-2018/29/B/HS4/02725.