key: cord-0786326-jsyuh368
authors: Chu, Hui-Chun; Hwang, Gwo-Jen
title: A Delphi-based approach to developing expert systems with the cooperation of multiple experts
date: 2007-05-18
journal: Expert Syst Appl
DOI: 10.1016/j.eswa.2007.05.034
sha: c8eea85b7ab160588489e237470aff0ca1f1c5cd
doc_id: 786326
cord_uid: jsyuh368

Knowledge acquisition has been a critical bottleneck in building knowledge-based systems. In past decades, several methods and systems have been proposed to cope with this problem. Most of these methods and systems were proposed to deal with the acquisition of domain knowledge from single expert. However, as multiple experts may have different experiences and knowledge on the same application domain, it is necessary to elicit and integrate knowledge from multiple experts in building an effective expert system. Moreover, the recent literature has depicted that “time” is an important parameter that might significantly affect the accuracy of inference results of an expert system; therefore, while discussing the elicitation of domain expertise from multiple experts, it becomes an challenging and important issue to take the “time” factor into consideration. To cope with these problems, in this study, we propose a Delphi-based approach to eliciting knowledge from multiple experts. An application on the diagnosis of Severe Acute Respiratory Syndrome has depicted the superiority of the novel approach.

In the past decades, expert systems have been applied to various applications. Subject domains that are supported by experts systems include bioengineering, defense, education, engineering, finance, and medical diagnosis. For example, MYCIN project is a well-known medical expert system for diagnosing infectious diseases (Buchanan & Shortliffe, 1985) ; ISODEPOR was developed to evaluate the muscle strength of Spanish top-competition athletes (Barreiro et al., 1997) ; FRBS-GP is a fuzzy rule-based system for diagnosing aphasia's subtypes and the classification of pap-smear examinations (Jantzen, Axer, & Keyserlingk, 2002) .

The successful cases of the expert system approach not only demonstrated the benefits of applying expert system approach to coping with medical diagnosis problems, but also depicted the difficulty of applying it. In building an expert system, the critical bottleneck is to obtain the knowledge of the special domain from the domain experts, which is called knowledge acquisition. In past decades, several methods and systems have been proposed to cope with this problem. However, most of these methods and systems were proposed to deal with the acquisition of domain knowledge from single expert. However, as multiple experts may have different experiences and knowledge on the same application domain, it is necessary to elicit and integrate knowledge from multiple experts in building an effective expert system. Recent literature also indicated that ''time'' is an important parameter that might significantly affect the accuracy of inference results of an expert system; therefore, while discussing the elicitation of domain expertise from multiple experts, it becomes a much more challenging and important issue to take the ''time'' factor into consideration (Hwang, Chen, Hwang, & Chu, 2006) .

To cope with these problems, we shall propose a Delphibased approach to eliciting knowledge from multiple experts. An application of developing a medical expert system has depicted the superiority of the novel approach. 

To cope with the knowledge acquisition problem, many knowledge acquisition tools or methods have been proposed to build rapid prototypes and to improve the quality of the elicited knowledge, e.g., ETS (Boose, 1984 (Boose, , 1985 , TEIRESIAS (Davis, 1979) , MORE (Kahn, Nowlan, & McDermott, 1985) , SALT (Marcus, 1987; Marcus, McDermott, & Wang, 1985) , NeoETS (Boose & Bradshaw, 1986; Kitto & Boose, 1986) , KNACK (Klinker, Bentolila, Genetet, Grimes, & McDermott, 1987) , AQUINAS (Boose & Bradshaw, 1987; Shema & Boose, 1988) , KRITON (Diederich, Ruhmann, & May, 1987) , Student (Gale, 1987) , Rule-Cons (O'Bannon, 1987) , MOLE (Eshelman, Ehret, McDermott, & Tan, 1987) , KITTEN (Shaw & Gaines, 1987) , KSSO (Gaines, 1987) , ASK (Gruber, 1988) , Word-Net (Millar, 1990; Navigli, Velardi, & Gangemi, 2003) , KADS (Schreiber, Wielinga, & Breuker, 1993; Wielinga, Schreiber, & Breuker, 1992) , MCRDR (Kang, 1996) , Med-Frame/CADIAG-IV (Boegl, 1997; Kolousek, 1997; Leitich et al., 2001) . Most of these systems were developed based on the repertory grids method originated from Kelly's Personal construct theory (Kelly, 1955) , which assists in identifying different objects in a domain and distinguishing among these objects.

A single repertory grid is represented as a matrix whose columns have elements labels and whose rows have construct labels. A 5-scale rating mechanism is usually used in filling the grid; i.e., each rating is an integer ranging from 1 to 5, where ''1'' represents that the element is very likely to have the trait; ''2'' represents the element may have the trait; ''3'' represents ''unknown'' or ''no relevance''; ''4'' represents that the element may have the opposite characteristic of the trait; ''5'' represents that the element is very likely to have the opposite characteristic of the trait.

As repertory grid approach has been widely used by researchers, some extensions have been made to enhance its representative ability. For example, Jose, Nicholas, Jennings, Luo, and Shadbolt, 2003 developed a technique using a fuzzy repertory grid for acquiring the finite set of attributes or variables that the expert uses in a classification problem, characterizing and discriminating a set of elements. In addition, several models have been proposed to generate more meaningful rules from the repertory gridoriented approaches, such as the EMCUD method, which can generate embedded meanings from repertory grids by defining the impacts of the constructs to each element (Hwang & Tseng, 1990) . Recently, Hwang et al. (2006) indicated that, in building medical expert systems, most of the previous knowledge acquisition methods only pay attentions to the relationships between diseases and symptoms, while, the variant of the symptoms in different time scales of the diseases are not taken into account. Consider the repertory grid given in Table 1 which depicts an example of eliciting knowledge for diagnosing various kinds of gastrointestinal diseases. Note that the rating of the (Acute bronchitis, Throat pain) entry is 4, which implies highly tendency for Acute bronchitis to have Throat pain. However, in practical situation, Influenza has significant appearance of Throat pain in the early time scale. What has been addressed in the repertory grid is not happened in the last time scale of acute bronchitis. For later time scale, the throat pain symptom will become not so significant. Such variant of disease symptoms with respect to different time scales cannot be precisely presented by those conventional knowledge acquisition approaches.

In developing a knowledge-based system, it is very difficult to elicit and integrate knowledge from multiple experts (Hwang et al., 2006) , especially the application domains in which various time scales of elements need to be taken into account. To cope with this problem, a novel approach, Knowledge Acquisition for Multiple Experts with Time scales (KAMET), is proposed in this section, which takes time scales into consideration while eliciting expertise from multiple experts. In addition to time scales, KAMET takes importance degree for each construct to each element in different time scales into account, such that more embedded knowledge can be explicitly presented.

Let e t i denote tth stage period of element (or disease) e i and c j denote a construct (or symptom), where i = 1 to n, and j = 1 to m. Each KAMET entry is a triplet that consists of three values: a rating to indicate the relevance of e t i and c j , a certainty degree for giving the rating and an impact factor to represent the importance of c j to e t i , which are represented by the following three functions:

(1) Rating (e t i , c j ): the degree of relevance for element e i in tth time scale to construct c j , ranging from 1 to 5: ''1'' represents that the element is very likely to have the opposite characteristic of the trait; ''2'' represents the element may have the opposite characteristic of the trait; ''3'' represents ''unknown'' or ''no relevance''; ''4'' represents that the element may have the trait; ''5'' represents that the element is very likely to have the trait.

(2) Certainty (e t i , c j ): the degree of certainty for giving Rating (e t i , c j ), which is either ''S'' or ''N'' representing ''sure'' or ''not sure''.

(3) Impact_factor (e t i , c j ): the degree of importance for construct c j to element e i in tth time scale. Impact_ factor (e t i , c j ) can be one of the following values: ''X'' represents no relationship between the element and the construct; ''D'' means that the construct dominates the element, i.e., if the value of the construct is not matched, it is impossible for the element to be implied; an integer, ranging from 1 to 5, indicates that the construct is of some degree of importance to the element, but does not dominate the implication of the element.

In the following subsections, we shall introduce each step of KAMET in detail. Without loss of generality, we shall use the knowledge elicitation process for developing a medical diagnosis expert system with seven medical experts as an example to demonstrate our novel approach. In this illustrative example, each element is called a ''disease'', and each construct is called a ''symptom''.

To elicit and integrate time scale information of elements from multiple experts, KAMET employs the Delphi method in its initial negotiation stages. Delphi has been defined by Delbecq, Van de Ven, and Gustafson (1975) as ''a method for systematic solicitation and collection for judgments on a particular topic through a set of carefully designed sequential questionnaires interspersed with summarized information and feedback of opinions derived from earlier responses''. It has been considered a reliable qualitative research method with potential for use in problem solving, decision making, and group consensus reaching in a wide variety of areas (Cochran, 1983; Uhl, 1983) . Delphi is generally characterized by three important features (Murry & Hammons, 1995) :

(1) Anonymous group interaction and responses.

(2) Multiple iteration or rounds of questionnaires or other means of data collection with researchercontrolled statistical group responses and feedback. (3) Presentation of statistical group responses.

Once a working problem is defined, the Delphi procedure begins with identifying and selecting the domain experts who will participate in the Delphi panel. When a pre-determined number of experts agree to participate, the researcher uses multiple iterations or rounds of questionnaires to collect data.

The first-round questionnaire uses an open-ended format to elicit individual judgments or opinions from each member of the panel about the particular issue or problem under study. In essence, round one is an anonymous brainstorming session. After all the round one questionnaires are returned, the researcher reviews, edits, and compiles the panel's responses, then prepares the round two questionnaires. In the second-round questionnaire, the researcher requests the panel of experts to consider, to rank and/or rate, to edit, and to comment upon the responses developed during round one. Typically the ranking or ratings (or both) is represented by a Likert scale. When all the round two questionnaires are returned, the researcher tabulates results, and then calculates frequency distributions, means, and standard deviations for each questionnaire item. During the third and any subsequent questionnaire rounds, the panel is given feedback about the previous round. The given information includes panel comments, the composite results, and individual rankings or ratings (or both) for each questionnaire item. Panel members are again asked to rank and/or rate, edit, and comment upon each item. The goal of the third round and any other subsequent round of questionnaires is to achieve consensus or stability of panel member responses. Once consensus or stability (or both, depending on individual case) is gained, the Delphi procedure is completed (Murry & Hammons, 1995) .

The typical first-round questionnaire uses an openended format to elicit individual judgments or opinions from each member of the panel about the particular issue or problem under study. In this phase, each expert is asked to provide the names of the elements to be discussed (e.g., the Respiratory Tract Infect diseases). Assume that Expert A provided {Upper respiratory tract infection (URI), Acute tonsillitis, Acute bronchitis, Bronchopneumonia, Pneumonia, Pulmonary TB, Influenza, SARS (Severe Acute Respiratory Syndrome)}, Expert B provided {URI (Upper respiratory tract infection), Acute tonsillitis, Acute bronchitis, Bronchopneumonia, Pneumonia, Pulmonary TB, Influenza, Severe Acute Respiratory Syndrome (SARS), Allergic rhinitis, COPD/Emphysema}, etc. After collecting the elements (diseases) given by the experts and removing the redundant ones. KAMET then employs the Delphi technology to inquire for each expert's opinions and comments on the union set of the elements. The Delphi questionnaire is shown in Table 2 , where E 1 , E 2 , . . ., E 15 Table 1  Illustrative example of a repertory grid for gastrointestinal diseases   Acute bronchitis Bronchopneumonia   Diarrhea  2  2  No diarrhea  Cough  3  5  No cough  Throat pain 4  4  No throat pain  Fever  5  4 No fever Table 3 shows the analysis results of the ratings given by seven experts for the Respiratory Tract Infect diseases in the first round.

To analyze the consistency and the stability of the ratings given by the experts, a second-round questionnaire is conducted after the summary of the first-round rating is depicted to each expert. In the second-round questionnaire, KAMET requests the panel of experts to consider, to rank and/or rate, to edit, and to comment upon the responses developed during the first round. When all the second round's questionnaires are returned, KAMET tabulated the results, and then calculates frequency distributions, means, and standard deviations for each questionnaire item. Table 4 shows the summary of the second-round ratings given by the experts.

To calculate the mean and standard deviation, the fourpoint rating scales are converted to numeric values. If an expert failed to respond to certain criteria statement, the data will not be included in the computation of means or standard deviations. Usually, the threshold (minimum requirement) of having agreement on any particular item at the second round or later is set to 75%. The stability or convergence is defined as few or no further shifting of panel responses from round to round (Murry & Hammons, 1995) . Based on this principle, the rules for employing the Delphi technology in KAMET are given in Table 5 , where Rating_Mean(q i ) represents the mean of the ratings for questionnaire item q i , Rating_Variant(q i ) represents the ratio of experts who change their ratings for q i and Q is the quartile range.

The Delphi questionnaire is ended if one of the following situations occurs:

(1) All of the questionnaire items are either accepted or rejected.

(2) There still exist some undetermined questionnaire items; nevertheless, over 75% questionnaire items have their Rating Variant values being less then 15%. Table 6 shows the analysis results of the ratings given by the seven experts for the two rounds of questionnaires. As the rating means of Allergic rhinitis, chronic obstructive pulmonary disease (COPD), COPD/Emphysema and Asthma are all less then 3.5, and their Rating Variant values are all less than or equal to 15%, questionnaire items Allergic rhinitis, chronic obstructive pulmonary disease (COPD), COPD/Emphysema and Asthma disease are removed; that is, those elements (diseases) are considered to be irrelevant to the domain. As the rating mean of COPD/Bronchiectasis disease is less then 3.5, and their Rating Variant value is greater than or equal to 15%, a third-round questionnaire is also needed.

For Pneumonia, the rating mean is grater than 3.5, and the Rating_Variant value is greater than 15%, and hence a third-round questionnaire is needed. Moreover, for Empyema disease, the rating mean in the first round is less than 3.5 and that in the second round is greater than 3.5, and hence a third-round questionnaire is also needed. The Delphi questionnaire is repeatedly conducted until no further round of questionnaire is needed. Table 5 Rules for analyzing the ratings from multiple experts with Delphi approach

Round t for Delphi questionnaire Round t + 1 for Delphi questionnaire Round t + 2 for Delphi questionnaire Rating_Mean(q i ) P 3.5 IF Rating_Mean(q i ) P 3.5 and Q 6 0.5 and Rating_Variant(q i ) < 15% Then q i is accepted, and no further discussion concerning q i is needed Rating_Mean(q i ) < 3.5

Rating_Mean(q i ) P 3.5 or Rating_Variant(q i ) > 15% If Rating_Mean(q i ) P 3.5 and Q 6 0.5 and Rating_Variant(q i ) 6 15% Then q i is accepted, and no further discussion concerning q i is needed Rating_Mean(q i ) < 3.5 IF Rating_Mean(q i ) < 3.5 and Q 6 0.5 and Rating_Variant(q i ) 6 15%. Then q i is rejected, and no further discussion concerning q i is needed Table 6 Analysis of the ratings in the two rounds of ratings given by the seven experts

Degree of relevance to the respiratory tract infect diseases domain 16. Asthma Round 1 2 2 3 3 3 2 1 2.29 0.5 14.29 Round 2 2 2 2 3 3 2 1 2.14 0.5

After determining the union element set of the application domain, KAMET employs the Delphi method to elicit corresponding time scale information of each element from multiple experts.

In this phase, each expert is asked to provide the number of time scales of the elements to be discussed. Based on the feedbacks of the experts, the elements with inconsistent numbers of time scales need to be discussed by employing the Delphi technology. For example, if Expert A proposed three time scales for SARS and two time scales for URI, while Expert B proposed three time scales for SARS and one time scale for URI, the Delphi method will be employed to discuss the number of time scales of URI. Table 7 depicts an illustrative of the number of time scales of each element given by seven experts.

In Table 7 , SARS is said to have three time scales by all of the experts, that is, a consistent conclusion has been arrived; therefore, no further discussion concerning the number of time scales for SARS is needed. For the elements containing inconsistent feedbacks from the experts, a Delphi questionnaire is generated for further discussions, as shown in Table 8 . The process of analyzing the questionnaire and determining the number of time scales of each element is similar to that of previous phase.

After the elements and their corresponding numbers of time scales have been determined, each expert is asked to give a set of constructs (symptoms) to distinguish the elements (diseases) in different time scales by invoking the interviewing technology of the repertory grid approach (Kelly, 1955) . If the experts have provided inconsistent c'onstructs (symptoms), the Delphi method will be employed to determine the accepted construct set based on the agreement of all of the experts.

Each entry of the KAMET is a triplet, as defined by Hwang et al. (2006) , consisting of a Rating (e t i , c j ) to indicate the relevance of the disease and the symptom, a Certainty (e t i , c j ) for giving the rating, and an Impact_factor (e t i , c j ) to represent the importance of the symptom to the disease. A 5-scale rating mechanism is used for Rating (e t i , c j ), where each rating is an integer ranging from 1 to 5, representing the tendency of e t i to each pole of construct c j : ''1'' means ''strongly support the positive trait of the construct'', ''2'' means ''support the positive trait of the construct'', ''3'' means ''not clearly support the positive trait or the opposite of the construct'', ''4'' means ''support the opposite of the construct'', ''5'' means ''strongly sup-port the opposite of the construct''. The value of Certainty (e t i , c j ) could be ''S'' or ''N'', where ''S'' represents ''Sure'' for giving Rating (e t i , c j ) and ''N'' means ''Not sure''. The value of Impact_factor (e t i , c j ) represents the degree of importance for construct c j to element e i in tth time scale. Impact_ factor (e t i , c j ) can be ''X'' to represent no relationship between the element and the construct, ''D'' to represent that a necessary construct for the element, and an integer, ranging from 1 to 5, to indicate that the construct is of some degree of importance to the element.

An example of a KAMET grid is given in Table 9 , where element e i in the kth time scale is recorded as T k of e i or e k i . For example, the second time scale of Acute bronchitis is recorded as T 2 of Acute bronchitis. There are two time scales for URI 3.

There is one time scale for Acute tonsillitis 4.

There are two time scales for Acute tonsillitis 5.

There are two time scales for Pulmonary TB 6.

There are three time scales for Pulmonary TB 7.

There are two time scales for SARS 8.

There are three time scales for SARS 9.

There are two time scales for Pneumonia 10.

There are three time scales for Pneumonia 11.

There is one time scale for Allergic Influenza 12.

There are two time scales for Allergic Influenza 13.

There is one time scale for Flu/influenza 14.

There are two time scales for Flu/influenza 3.5. Integrating corresponding ratings from multiple experts

In developing an expert system, one of the most difficult tasks is to collect domain knowledge from multiple experts; therefore, implying a knowledge integration technology is needed. To cope with the knowledge integration problem, the ratings given by Exp i for element e t p and construct c q are represented as Rating (Exp i , e t p , c q ), Certainty (Exp i , e t p , c q ) and Impact_Factor (Exp i , e t p , c q ), and a set of meta rules is defined as follows:

If Rating (Exp i , e t p , c q ) = 3 and Rating (Exp j , e t p , c q ) = 3 and Certainty (Exp i , e t p , c q ) = ''S'' and Certainty (Exp j , e t p , c q )=''N'' Then Rating (e t p , c q ) = Rating (Exp i , e t p , c q ) and Certainty (e t p , c q ) = ''S''

This rule is used to handle the case that two experts both assign opposite side ratings to represent an element/construct relationship with different degrees of confidence. In this case, the rating of the expert with higher confidence is adopted as the integrated rating, and the confidence degree is set to ''S''.

If Rating (Exp i , e t p , c q ) = 3 and Rating (Exp j , e t p , c q ) = 3 and Certainty (Exp i , e t p , c q ) = ''S'' and Certainty (Exp j , e t p , c q ) = ''S'' Then Rating (e t p , c q ) = MAX(Rating (Exp i , e t p , c q ), Rating (Exp j , e t p , c q )) and Certainty (e t p , c q )=''S''

This rule is used to handle the case that two experts both assign opposite side ratings to represent an element/construct relationship with degree of confidence ''S''. In this case, the larger rating (the stronger opposite tendency) is adopted as the integrated rating, and the confidence degree is set to ''S''.

If Rating (Exp i , e t p , c q ) = Rating (Exp j , e t p , c q ) = 3 and Certainty (Exp i , e t p , c q ) = ''N'' and Certainty (Exp j , e t p , c q ) = ''N'' Then Rating (e t p , c q ) = Rating (Exp i , e t p , c q ) and Certainty (e t p , c q ) = ''N'' This rule is used to handle the case that two experts both assign opposite side ratings to represent an element/construct relationship with degree of confidence ''N''. In this case, the larger rating (the stronger opposite tendency) is adopted as the integrated rating, and the confidence degree is set to ''N''.

If Rating (Exp i , e t p , c q ) 5 3 and Rating (Exp j , e t p , c q ) 5 3 and Certainty (Exp i , e t p , c q ) = ''S'' and Certainty (Exp j , e t p , c q ) = ''N'' Then Rating (e t p , c q ) = Rating (Exp i , e t p , c q ) and Certainty (e t p , c q ) = ''S''

This rule is used to handle the case that two experts both assign positive side ratings to represent an element/construct relationship with different degrees of confidence. In this case, the rating of the expert with higher confidence is adopted as the integrated rating, and the confidence degree is set to ''S''.

If Rating (Exp i , e t p , c q ) 5 3 and Rating (Exp j , e t p , c q ) 5 3 and Certainty (Exp i , e t p , c q ) = ''S'' and Certainty (Exp j , e t p , c q ) = ''S'' Then Rating (e t p , c q ) = MIN(Rating (Exp i , e t p , c q ), Rating (Exp j , e t p , c q )) and Certainty (e t p , c q ) = ''S'' This rule indicates that, if an expert assign ''X'' to represent the importance degree with ''Sure'' while others assign an integer with ''not sure'', the integrated result is ''X''.

If Impact(Exp i , e t p , c q ) = ''X'' and Impact(Exp j , e t p , c q ) = integer and Certainty (Exp i , e t p , c q ) = ''S'' and Certainty (Exp j , e t p , c q ) = ''S''

Then Ask the experts to check their ratings This rule indicates that, if an expert assign ''X'' to represent the importance degree with ''Sure'' while others assign an integer with ''Sure'', further negotiation is needed.

After eliciting and integrating knowledge from multiple experts, the rule-generating algorithm proposed by Hwang et al. (2006) can be applied to generate a set of rules with certainty factors. 

Based on our novel approach, a knowledge acquisition system, KAMET, has been implemented on Windows 2000 Server. KAMET consists of five main components, namely, knowledge elicitation module, knowledge transformation module, Delphi-oriented Group Decision Module and Expertise Integration Module. The knowledge acquisi-tion module is a web-based knowledge elicitation tool with KAMET approach, which provides facilities of retrieving, maintaining, creating, and storing knowledge. The knowledge transformation module can transform domain expertise into the rule-based knowledge format of some inference engine. The Delphi-oriented Group Decision Module is implemented based on the Delphi technology (Murry & Hammons, 1995) . It employs anonymous controlled- feedback procedures to assist multiple experts in determining elements, constructs and number of times scales for each element during the knowledge acquisition process. The Expertise Integration Module employs a set of knowledge integration rules to integrate ratings, certainty degrees and importance degrees of corresponding element/construct relationships.

After all of the experts provide the elements for some specific application domains; KAMET generates a union element set by integrating the element sets elicited from individual experts and removing the redundant elements. KAMET then employs the Delphi technology to inquire for each expert's opinions and comments on the union set of the elements. An illustrative example of a Delphi questionnaire is shown in Fig. 1 .

KAMET then summarizes the answers given by the experts in the first-round questionnaire, and shows the analysis results to the experts before starting the secondround questionnaire. Fig. 2 demonstrates the KAMET interface for presenting the analysis results. Fig. 3 shows the summary of the ratings given by seven experts in the first and the second rounds of questionnaires, which is then used to analyze the consistency and stability of the ratings for each questionnaire item.

As shown in Fig. 3 , the rating means of COPD (chronic obstructive pulmonary disease) and Pneumonia are both grater than 3.5, and their Rating Variant values are both greater than 15%, and hence the third round questionnaire is needed. Moreover, for Empyema disease, the rating mean in round 1 is less than 3.5 and that in round 2 is greater than 3.5, and hence the third round questionnaire is also needed (see Fig. 4 ). This phase will be repeatedly executed until all of the elements are determined to be relevant or irrelevant to the application domain.

Consequently, the experts are asked to determine the number of time scales for each element (disease), as shown in Fig. 5 . If there are elements with inconsistent number of time scales given by individual experts, a Delphi questionnaire is then used to discuss the number of time scales for each element, as shown in Fig. 6 . After the diseases and their relevant constructs (pairs of symptoms and antonyms) are elicited, each expert is asked to enter the triplet values for each <disease, construct> pair. Following the procedure given in previous section, the final integrated knowledge is generated as a set of IF-THEN rules.

To evaluate the performance of our novel approach, seven domain experts were invited to participate in the experiment; the backgrounds of the experts are given as follows:

(1) Four medical doctors from teaching hospital, one from the Department of Thoracic Internal Medicine, one from the Department of Pediatrics, and the other from the Department of Emergency Medicine.

(2) Two family doctors.

(3) One dietician.

The application domain is ''Severe Acute Respiratory Syndrome'', including twelve diseases: URI (upper respiratory tract infection), Acute tonsillitis, Acute bronchitis, Bronchopneumonia, Pneumonia, COPD/Chronic bronchi-tis, COPD/Bronchiectasis, Empyema, Pulmonary TB/ tuberculosis, Flu/influenza, Severe Acute Respiratory Syndrome (SARS), Avian Influenza. Twenty two symptoms are used to identify those diseases, including three categories ''Respiratory tract symptoms'', ''Physiology index and Physical examination'' and ''Others symptoms'', ''Respiratory tract symptoms'' including seven symptoms including ''Sneeze, Running nose, Snuffle'', ''Snot regorge'', ''Throat pain'', ''Cough'', ''Dense phlegm'', ''Pant/ Breath hard'', ''Pleurisy chest pain''. The second category has eight symptoms including ''Fever'', ''Headache'', ''Muscle ache'', ''Serious tired'', ''Longterm cough'', ''Night sweat'', ''Weight adate''. The third category is ''Physiology index and Physical examination'' has seven symptoms including ''The body temperature >38°C'', ''Tachypnoea'', ''Pulse frequency'', ''Hypertrophy or fester of tonsils'', ''The pharynx and the larynx hyperemia'' ''Wheezing'' and ''Respiratory rhonchus''.

Thirty-three cases given by the medical experts were used as the test data. Table 10 shows comparison for the correct-diagnosis rates of the knowledge bases created by individual experts and the one by employing KAMET. It can be seen that the performance of the KAMET approach is better than the individual knowledge acquisition approach. 

In this study, we propose a knowledge acquisition method to elicit expertise from multiple experts, in which each element is considered to have one or more time scales; moreover, a systematical procedure is proposed to elicit embedded meanings based on the degree of relevance for each symptom to each time scale of elements. To evaluate the effectiveness of the novel approach, a knowledge acqui-sition system has been developed, and seven medical experts were asked to participate in an experiment. From the experimental results, it can be seen that our novel approach able to achieve significantly better performance, and hence, we conclude that the new approach is helpful in enhancing repertory grid efficacy. Now, we are planning to employ the novel approach to several issues concerning e-learning and medical education. 

ISODEPOR: System for the Interpretation of Isoquinetics. Res umenes de Comunicaciones del IV congreso internacional de entrenamiento deportivo

Design and implementation of a web-based knowledge acquisition toolkit for medical expert consultation systems, Doctoral thesis

Personal construct theory and the transfer of human expertise

A knowledge acquisition program for expert systems based on personal construct psychology

NeoETS: Capturing expert system knowledge in hierarchical rating grids

Expertise transfer and complex problems: using AQUINAS as a knowledge-acquisition workbench for knowledge-based systems

Rule-based expert systems: The MYCIN experiments of the Stanford heuristic programming paper

The Delphi method: Formulation and refining group judgements

Interactive transfer of expertise

Group techniques for program planning: A guide to nominal group and Delphi processes

KRITON: a knowledge acquisition tool for expert systems

MOLE: a tenacious knowledge acquisition tool

An overview of knowledge-acquisition and transfer

Knowledge-based knowledge acquisition for a statistical consulting system

The acquisition of strategic knowledge

EMCUD: A knowledge acquisition method which captures embedded meanings under uncertainty

A time scale-oriented approach for building medical expert systems

Diagnosis of aphasia using neural and fuzzy techniques

Acquiring domain knowledge for negotiating agents: A case of study

MORE: An intelligent knowledge acquisition tool

Multiple classification ripple down rules

The psychology of personal constructs

Heuristics for expertise transfer: The automatic management of complex knowledge acquisition dialogs

KNACK -Report-driven knowledge acquisition

The system architecture of an integrated medical consultation system and its implementation based on fuzzy technology

Taking backtracking with a grain of SALT

Knowledge acquisition for constructive systems

WordNet: An on-line lexical resource

Delphi, a versatile methodology for conducting qualitative research. The Review of Higher Education

Ontology learning and its application to automated terminology translation

An intelligent aid to assist knowledge engineers with interviewing experts

KADS: A principle approach to knowledge-based system development

KITTEN: Knowledge initiation and transfer tools for experts and novices

Refining problem-solving knowledge in repertory grids using a consultation mechanism

KADS: A modeling approach to knowledge engineering

Using the Delphi technology in institutional planning

This study is supported in part by the National Science Council of the Republic of China under Contract numbers NSC 95-2524-S-024-002 and NSC 95-2520-S-024-003.

This rule is used to handle the case that two experts both assign positive side ratings to represent an element/construct relationship with degree of confidence ''S''. In this case, the smaller rating (the stronger opposite tendency) is adopted as the integrated rating, and the confidence degree is set to ''S''.

If Rating (Exp i , e t p , c q ) 5 3 and Rating (Exp j , e t p , c q ) 5 3 and Certainty (Exp i , e t p , c q ) = ''N'' and Certainty (Exp j , e t p , c q ) = ''N'' Then Rating (e t p , c q ) = MIN(Rating (Exp i , e t p , c q ), Rating (Exp j , e t p , c q )) and Certainty (e t p , c q ) = ''S'' This rule is used to handle the case that two experts both assign positive side ratings to represent an element/construct relationship with degree of confidence ''N''. In this case, the smaller rating (the stronger opposite tendency) is adopted as the integrated rating, and the confidence degree is set to ''N''.

If (Rating (Exp i , e t p , c q ) À 3) · (Rating (Exp j , e t p , c q ) À 3) < 0 and Certainty (Exp i , e t p , c q ) = ''S'' and Certainty (Exp j , e t p , c q ) = ''N'' Then Rating (e t p , c q ) = Rating (Exp i , e t p , c q ) and CertaintyThis rule is used to handle the case that two experts assign ratings in different sides (one is greater than 3, and one is less than 3) to represent an element/construct relationship. In this case, the rating with higher confidence is adopted as the integrated rating, and the confidence degree is set to ''N''.

If (Rating (Exp i , e t p , c q ) À 2) · (Rating (Exp j , e t p , c q ) À 2) < 0 and Certainty (Exp i , e t p , c q ) = ''N'' and Certainty (Exp j , e t p , c q ) = ''N'' Then Ask the experts to check their ratings This rule is used to check if the experts assign conflict values to represent an element/construct relationship. If this case does happen and all of the experts are not confident about the value they gave, the system returns to the previous phase to get more confident information.Rating_Integration_Rule 9:

This rule is used to check if the experts assign conflict values to represent an element/construct relationship. If this case does happen and all of the experts are confident about the value they gave, the system returns to the previous phase to get more confident information.

If Impact(Exp i , e t p , c q ) = integer and Impact(Exp j , e t p , c q ) = integer Then Impact(e t p , c q ) = Impact(Exp i , e t p , c q ) * Ratio(Certainty (Exp i , e t p , c q )) + (Impact(Exp j , e t p , c q ) * Ratio(Certainty (Exp j , e t p , c q )))/(Ratio(Certainty (Exp i , e t p , c q )) + Ratio(Certainty (Exp j , e t p , c q ))) This rule is used to combine the importance degrees with integer types based on the confidence values, where Ratio(''S'') = 1 and Ratio(''N'') = 0.8. For example, consider two experts Exp i and Exp j who assign 2 and 4 as the importance degrees for construct c q to element e t p ,, and Certainty (Exp i , e t p , c q ) = ''S'' and Certainty (Exp j , e t p , c q ) = ''N''. We have Impact(e t p , c q ) = (2 * 1 + 4 * 0.8)/(1 + 0.8) = 2.9.

If Impact(Exp i , e t p , c q ) = ''X'' and Impact(Exp j , e t p , c q ) = integer and Certainty (Exp i , e t p , c q ) = ''N'' and Certainty (Exp j , e t p , c q ) = ''S''Then Impact(e t p , c q ) = Impact(Exp j , e t p , c q ) * Ratio(Certainty (Exp j , e t p , c q ))/(Ratio(Certainty (Exp i , e t p , c q )) + Ratio(Certainty (Exp j , e t p , c q )))This rule indicates that, if an expert assign ''X'' to represent the importance degree with ''not sure'' while others assign an integer with ''Sure'', the ''X'' is treated as 0 during the integration process.

If Impact(Exp i , e t p , c q ) = ''D'' and Impact(Exp j , e t p , c q ) = integer and Certainty (Exp i , e t p , c q ) = ''N'' and Certainty (Exp j , e t p , c q ) = ''S'' Then Impact(e t p , c q ) = 6 * Ratio(Certainty (Exp i , e t p , c q )) + (Impact(Exp j , e t p , c q ) * Ratio(Certainty (Exp j , e t p , c q )))/(Ratio(Certainty (Exp i , e t p , c q )) + Ratio(Certainty (Exp j , e t p , c q )))This rule indicates that, if an expert assign ''D'' to represent the importance degree with ''not sure'' while others assign an integer with ''Sure'', the ''D'' is treated as 6 during the integration process.

If Impact(Exp i , e t p , c q ) = ''D'' and Impact(Exp j , e t p , c q ) = ''X'' Then Ask the experts to check their ratings This rule indicates that, if one expert assigns ''D'' to represent the importance degree while the other assigns ''X'', further negotiation is needed.

If Impact(Exp i , e t p , c q ) = ''D'' and Impact(Exp j , e t p , c q ) = ''D'' Then Impact(e t p , c q ) = ''D''This rule indicates that two D's are treated as one ''D'' during the integration process.

If Impact(Exp i , e t p , c q ) = ''X'' and Impact(Exp j , e t p , c q ) = ''X'' Then Impact(e t p , c q ) = ''X''This rule indicates that two X's are treated as one ''X'' during the integration process.

If Impact(Exp i , e t p , c q ) = ''D'' and Impact(Exp j , e t p , c q ) = integer and Certainty (Exp i , e t p , c q ) = ''S'' and Certainty (Exp j , e t p , c q ) = ''N'' Then Impact(e t p , c q ) = ''D''This rule indicates that, if an expert assign ''D'' to represent the importance degree with ''Sure'' while others assign an integer with ''not sure'', the integrated result is ''D''.

If Impact(Exp i , e t p , c q ) = ''D'' and Impact(Exp j , e t p , c q ) = integer and Certainty (Exp i , e t p , c q ) = ''S'' and Certainty (Exp j , e t p , c q ) = ''S'' Then Ask the experts to check their ratings This rule indicates that, if an expert assign ''D'' to represent the importance degree with ''Sure'' while others assign an integer with ''Sure'', further negotiation is needed.

If Impact(Exp i , e t p , c q ) = ''X'' and Impact(Exp j , e t p , c q ) = integer and Certainty (Exp i , e t p , c q ) = ''S'' and Certainty (Exp j , e t p , c q ) = ''N'' Then Impact(e t p , c q ) = ''X''