paper.dvi Applying KADS to KADS� knowledge based guidance for knowledge engineering John K�C� Kingston AIAI�TR���� January ���� This paper was published in the journal �Expert Systems The International Journal of Knowledge Engineering � ��� �� February ���� Arti cial Intelligence Applications Institute University of Edinburgh �� South Bridge Edinburgh EH� �HN United Kingdom c� The University of Edinburgh� ����� Abstract The KADS methodology �Schreiber et al� ����� �Tansley � Hayball� ����� and its successor� CommonKADS �Wielinga et al� ��� � have proved to be very useful approaches for modelling the various transformations involved be� tween eliciting knowledge from an expert and encoding this knowledge in a computer program� These transformations are represented in a series of mod� els� While it is widely agreed that these methods are excellent approaches from a theoretical viewpoint� the documentation provided concentrates on de ning what models should be produced� with only general guidance on how the models should be produced� This has the advantage of making KADS and CommonKADS widely applicable� but it also means that considerable training and experience is required to become pro cient in them� This paper reviews three projects� which investigated the feasibility of producing speci c guidance for certain decisions which are required when using KADS or CommonKADS to develop a knowledge based system� Guid� ance was produced for the identi cation of the generic task addressed by a knowledge based system� for the selection of appropriate AI techniques for implementing the analysed knowledge� and for selecting a suitable tool for implementing the system� Each set of guidance was encoded in its own knowledge based system� which was itself developed with the assistance of KADS or CommonKADS� These projects therefore both studied and applied KADS and CommonKADS in order to produce knowledge based guidance for knowledge engineers� The projects showed that it was feasible to produce heuristic guidance which could be understood� applied� and occasionally overridden by knowl� edge engineers� The guidance provides reasonably experienced knowledge engineers with a framework for making the key decisions required by Com� monKADS� in the same way that CommonKADS provides knowledge engi� neers with a framework for representing knowledge� The projects also pro� duced some new insights about CommonKADS domain modelling and about the process of task identi cation� � Introduction The KADS methodology �Schreiber et al� ����� �Tansley � Hayball� ����� and its successor� CommonKADS �Wielinga et al� ������ are collections of structured meth� ods for building knowledge based systems� analagous to methods such as SSADM for software engineering� The development of these methods was funded by the European Community�s ESPRIT programme between ���� and ����� KADS and CommonKADS view the construction of KBS as a modelling activity� and so these methods require a number of models to be constructed which represent di�erent views on problem solving behaviour� in its organisational and application context� CommonKADS recommends the construction of six models � � a model of the organisational function and structure� � a model of the tasks required to perform a particular operation� � a model of the capabilities required of the agents who perform that operation� � a model of the communication required between agents during the operation� � a model of the expertise required to perform the operation� which is divided into three sub�levels � models of declarative knowledge about the domain� � models of the inference processes required during problem solving� � an ordering of the inference processes� � a model of the design of a KBS to perform all or part of this operation� For more details on these models� see �deHoog et al� ������ Experience has shown that the models recommended by KADS and Com� monKADS provide an excellent basis for representing the various transformations required between eliciting knowledge from an expert and encoding it in a com� puter program� In addition� KADS and CommonKADS provide various libraries of generic models which have proved very useful to knowledge engineers �see �Kingston� ������ for example�� However� experience has also shown that the task of developing these models is non�trivial� There are a number of decisions to be taken at each stage in the modelling process� some of which have a major impact on the models� or on the implemented system� These decisions include �� Deciding the approach which should be taken to modelling should models be produced bottom�up from acquired knowledge� top�down by instantiating generic problem�solving methods which CommonKADS provides� or by an intermediate approach which selects a generic model and then modi es it in the light of domain knowledge� �� Selecting models from a library� The library of generic inference structures is indexed according to the type of task which is being tackled so the knowledge engineer must determine the most appropriate task type for the current task� �� Deciding whether the design should preserve the structure of the expertise model� �� Deciding which knowledge representations and inference techniques should be used within the design� � �� Deciding on the most appropriate tool for implementing this knowledge based system� Much of the CommonKADS documentation concentrates on de ning what should be done in order to produce models� while only specifying how it should be done in general terms� This has the advantage that it speci es the content of mod� els without enforcing an approach on practitioners� thus making CommonKADS widely applicable and compatible with many di�erent approaches to knowledge engineering� however� by the same token� it requires knowledge engineers to be fairly experienced at making good knowledge engineering decisions before they can make full use of KADS or CommonKADS� The CommonKADS project has pro� vided guidance on making some of the decisions outlined above �e�g� guidance on top�down�bottom�up approaches to model construction �Wielinga� ����� and model instantiation �L�ockenho� � Valente� ������� however� many of the organisa� tions which have recognised the value of KADS and CommonKADS have had to obtain training and accept a long learning curve for each of their sta� who wants to become pro cient in the KADS approach� An alternative approach to providing extensive training and experience for all sta� would be to provide speci c guidance on developing KADS models� thus pro� viding in�house KADS guidance based on the company�s own knowledge engineering experiences� The purpose of this paper is to review three projects which investi� gated the feasibility of producing guidance for some of the important decisions listed above� The tasks for which guidance was produced were � identifying the task type� � selecting appropriate AI techniques at the design stage�� � choosing a suitable implementation tool� These projects were all performed by students in the Department of Arti cial Intelligence� University of Edinburgh as part of an M�Sc course in Intelligent Knowl� edge Based Systems� The students were supervised by sta� from the Department of Arti cial Intelligence� and by members of the Knowledge Engineering Methods group at AIAI� the supervisors also acted as experts from whom knowledge was elicited� It is a requirement that all students on Edinburgh�s M�Sc course produce func� tioning software as part of their project� which meant that the students were re� quired to acquire knowledge� analyse the knowledge� and to implement this knowl� edge in a knowledge�based system� It therefore seemed sensible for the students to use KADS to help them in this process� since practical experience of using �Guidelines on whether a design should preserve the structure of an expertise model are cur� rently being considered� � KADS ought to help them in producing useful guidance� The main body of this paper therefore shows how KADS was used to model the tasks involved in making KADS�related decisions in these three projects� � Identifying task types The rst project looked at the task of identifying the most appropriate task type for a particular problem �Krueger� ������ ��� The task The student who took on this project was set the task of developing a technique for distinguishing and classifying di�erent expert tasks� with a focus on the tax� onomy of expert tasks developed by the KADS methodology �see Figure ��� This taxonomy de nes the contents of the library of generic inference structures� there is a generic inference structure associated with �almost� every leaf node in the tax� onomy� The selection of a suitable generic inference structure therefore boils down to the identi cation of the most appropriate task type from this taxonomy� SYSTEM MODIFICATION SYSTEM SYNTHESIS SYSTEM ANALYSIS Prediction Identifying Repairing Controlling Remedying Modelling Transformation Design Planning Configuration Refinement design Transformational design Monitoring Classification Prediction of values Prediction of behaviour Maintaining Assessment Diagnosis Simple classification Multiple fault diagnosis Single fault diagnosis Localisation Causal Tracing Systematic Diagnosis Heuristic Classification Multiple stream refinemt design Single stream refinemt design Exploration- based design Hierarchical design Figure �� Taxonomy of task types � ��� The project Given that the student has been presented with a problem to solve using knowledge� based techniques� the rst stage of KADS analysis is to identify the type of task which is carried out in order to select a generic inference structure� Fortunately� the obvious �chicken and egg situation which arose here was circumvented by reading an early KADS report �Breuker�� ������ which suggests that the appropriate task type here is assessment � that is� assessing how well each of the task types matches the task under consideration� and then selecting the one which matches most closely� The student therefore designed and implemented a KBS which used an assessment approach to the identi cation of task types� KADS� generic inference structure for assessment tasks �Figure �� recommends that assessment is carried out by abstracting �i�e� generalising� key features of a particular problem case description� and specifying the preferred value of these features from a system model which represents the �ideal world � The features of the case are then matched against the preferred features to determine the degree of acceptability of the case� For this task� the �problem case is an expert task� and the �ideal world is �one of� a set of generic inference structures� The generic inference structure for assessment tasks was therefore adapted and instantiated in the ways described below the resulting inference structure is shown in Figure �� Case description System model Norms Decision class Abstract case description match abstract specify Figure �� Generic inference structure for assessment tasks In order to instantiate this generic structure to problem in hand� the student was required to make some alterations to the generic inference structure � The key features of an expert task were obtained by asking the user� rather than by abstracting from a detailed description� The abstraction inference was therefore replaced by an obtain step�� �In CommonKADS� obtain is considerd to be a transfer task rather than an inference step� Transfer tasks are represented by rounded rectangles� � � The set of inference structures goes through a two�stage speci cation the rst stage determines the features which must be asked of the user� and the second stage speci es parameters of these features which can be compared against the user�s replies� � A �feedback from the di�erences discovered to the set of inference structures under consideration is explicitly recorded in the problem�speci c inference structure� user’s knowledge about expert task differences parameters expected (with weightings) features of structures under consideration set of inference structures under consideration user-supplied parameters refine compare specify-2 specify-1 obtain Figure �� Instantiated inference structure for assessing inference structures The resulting system� known as SEXTANT�� asks users to identify inputs of � Selection of EXpert TAsks by Nature of the Task � a task� outputs of a task� and knowledge roles which exist in the problem solving process� it then matches this information against the corresponding attributes of each generic inference structure in order to attach a likelihood weighting to each inference structure� As weightings increase or decrease� some inference structures are removed from consideration until only one �or a few� are left� The remaining inference structure�s� are then recommended to the user� As the project progressed� it became obvious that an alternative technique for identifying task types could be developed� based on the taxonomy of task types shown in Figure �� By starting at the topmost node in the taxonomy� it is possible to traverse the taxonomy by asking a single question at each node to decide which is the most appropriate subcategory for the current problem� For example� at the topmost level� the following question could be asked � Does the task involve �� Establishing unknown properties or behaviour of an object within the domain� �� Composing a new structural description of a possible object within the domain� �� A combination of the above� If the rst answer is chosen� then the most appropriate subcategory is System Analysis tasks� if the second� then System Synthesis� if the last� then System Mod� i�cation is the most appropriate task type� By de ning suitable questions for each non�leaf node in the taxonomy� it becomes possible to identify task types simply by answering a series of questions� E�ectively� the taxonomy is transformed into a decision tree� and the identi cation of task types becomes a comparatively simple classi cation task� The nal version of the SEXTANT system incorporates both the �decision tree approach and the �assessment approach� Novice users of KADS can progress through the decision tree� however� if they are unable to answer questions in the decision tree� the assessment approach takes over� using the remaining candidates from the decision tree as the set of inference structures on which assessment is performed� Experienced users of KADS should be able to specify a relatively small set of task types at the outset� and so will only need to use the assessment part of the SEXTANT system in order to support them in their nal decision� The use of SEXTANT also forces knowledge engineers to consider the nature and the dependencies of each inference step carefully� which should provide assistance in the task of instantiating the chosen generic inference structure to the actual task being performed� SEXTANT was implemented in CLIPS ���� and is therefore capable of running on a range of hardware� � ��� Results This project demonstrated that it was feasible to provide guidance on task selec� tion� using either an assessment�based approach or a relatively simple decision tree approach� The creation of appropriate questions for the decision tree required con� siderable thought about the nature of the choices being made at each choice point� The questions which were generated were reasonably consistent in their format� which suggests that the tasks in the library form a coherent set� It is not clear that the tasks form a complete set of all possible expert tasks� however� and even those tasks which are in the library do not always have an associated generic inference structure� It is likely that more work is needed on these tasks types � particularly system synthesis and system modi cation tasks � in order to produce a complete set of knowledge based tasks �cf� �Tansley � Hayball� ����� �Kingston� ������� Since this project was performed� the CommonKADS project has rede ned generic inference structures� by simplifying the basic structures in the library and providing extensive guidance on con guring the generic structures to the require� ments and features of a particular task �L�ockenho� � Valente� ������ This alter� ation has greatly increased the scope for de ning many expert tasks� as variations of a smaller number of �basic tasks � It has also reduced the utility of the �as� sessment approach in SEXTANT� because the di�erences between the simpli ed inference structures in the library now require less detailed analysis� and because the con guration process requires users to think deeply about the nature of the inferences performed in the task� However� SEXTANT�s �decision tree approach is still useful� indeed� it could usefully be extended to incorporate guidance on con� guring inference structures� since this guidance consists of a set of questions which are used to recommend particular alterations to a basic inference structure� These �con guring questions could therefore be considered to be extensions to the lowest levels of the decision tree� The �decision tree component of SEXTANT has been used on other KADS� related projects� including the projects described later in this paper� � Selecting appropriate AI techniques The second project was intended to help knowledge engineers select appropriate knowledge representation and inference techniques when constructing a KBS design �MacNee� ������ ��� The task Once a task type has been identi ed� the acquired knowledge can be analysed by building the KADS expertise model� This consists of � � an inference structure� con gured and instantiated to represent the reasoning processes which take place in the task under consideration� � a set of domain models� identifying key concepts in the domain and the rela� tionships between them� � a task structure� enforcing an ordering on the inference steps�� The resulting expertise model must then be transformed into a program speci� cation� this is the task of the KADS design phase� The approach recommended for the CommonKADS design phase prescribes a ��stage design process� and includes suggested approaches for modelling the rst two phases �van de Velde et al� ����� � Application design typically consists of a conceptual decomposition of the expertise model into a number of functional units and�or data objects� � Architecture design requires decisions on whether to use rules� objects� or other representational techniques� for di�erent parts of the application design� � Platform design matches these chosen techniques with the facilities and environment o�ered by the chosen programming tool� with consequent alter� ations to the architecture design and�or the programming tool� This approach has been used successfully on some projects� and appears to represent a su�ciently detailed breakdown of the design process�� however� unless a strongly prescriptive top�down approach to modelling is being used� there is little speci c guidance on the design process� The aim of the project described in this section was to elicit and acquire some guidelines for the production of an architectural design� The approach used was based on the probing questions approach� developed at Rome Air Force Base� USA �Kline � Dolins� ����� and further developed at AIAI �Inder et al� ������ in which a KBS designer is asked a number of questions about the analysed knowledge� with the answers being used to produce some recommendations of suitable representa� tional techniques� The general format of the questions is if a certain feature exists in the analysed knowledge then consider using a particular knowledge representation� or imple� mentation technique� The designer is asked whether the if condition is true if it is� the recommen� dation supplied by the latter part of the �probing question is added to the set of recommendations� An example of a probing question might be �See �Wielinga et al� ���� for a fuller description of the expertise model� �These � phases approximately correspond to the stages of functional decomposition� be� havioural design and physical design speci ed in the original KADS project� � if the problem�solving task is such that a pre�enumerated set of so� lutions can be established �as distinct from the type of task in which solutions are constructed as a result of the satisfaction of constraints� then use goal�driven reasoning weighting� � else use data�driven reasoning weighting� � It can be seen that probing questions are e�ectively heuristics for knowledge engineers� encoded in a rule�based format� The student�s task was to acquire prob� ing questions �either by eliciting knowledge from experienced knowledge engineers� or by reading the available literature�� to collate the resulting collection of heuris� tic rules into a structure of some kind� and then to implement a knowledge based system to run these rules� ��� The project� acquisition and analysis The student chose to concentrate his e�orts on eliciting knowledge from experts� Three experts were used� each of whom was interviewed �or asked to make extensive comments on documents sent by electronic mail� on several occasions� Knowledge elicitation techniques used included introductory interviews� card sorting� and tri� adic comparison of actual KBS systems� One of the key results of knowledge elicitation was the identi cation of a number of functional requirements and design features for knowledge base systems� Many functional requirements relate to the KBS� need to model the domain objects and their relationships� and the need to model the inferences which are made about these domain objects� These requirements may lead to various needs for example� there may be requirements for certain knowledge representations� for the ability to represent uncertainty in knowledge� and for the ability to generate and examine large numbers of solutions� Design features are knowledge representation and in� ference techniques used to satisfy the functional requirements� Examples include data�driven reasoning� rules� semantic networks� and blackboard architectures� There is also a sizeable group of functional requirements� which are concerned with the capabilities of the KBS environment and with producing a smooth inter� action between the user and the KBS� These requirements may specify a need for dialogue with and explanations to the user� a need for a high level of computational e�ciency in the program� or a need to consider how �or whether� to present the user with large numbers of possible solutions� It was therefore decided that these envirnomental features would also be considered as design features� The categorisation of functional requirements and design features can be seen in Figure �� Note that �thoughts of God is intended to be a catch�all category for ill�de ned subjective in uences� such as design for elegance� or parsimonious design� �� expert knowledge Knowledge of KBS environment and user/KBS interaction (0) ‘THOUGHTS OF GOD’ (1a) KNOWLEDGE STRUCTURE -- (a): domain objects and relationships (1b) KNOWLEDGE STRUCTURE -- (b): inferences and generic task (2) UNCERTAINTY IN KNOWLEDGE (3) SOLUTIONS (4) DATA (5) DIALOGUE AND EXPLANATION (6) COMPUTATIONAL EFFICIENCY (7) DEVELOPMENT: Analysed construction, maintenance, and CATEGORIES OF DESIGN FEATURE expansion (8) SAFETY CRITICALITY (A) DEPTH-OF-REASONING AND ARCHITECTURE -- shallow, model-based, blackboard (B) KNOWLEDGE REPRESENTATION STRUCTURES -- rules, frames, OOP, networks, ... (C) INFERENCE TYPES -- goal-driven, data-driven (D) CONTROL OF FLOW OF INFERENCE: search strategy, constraints on search, meta-control (E) HANDLING OF UNCERTAINTY IN KNOWLEDGE AND OF INCOMPLETENESS AND INACCURACY OF DATA (F) USER INTERFACE: KBS input/output (G) KNOWLEDGE ACQUISITION CATEGORIES OF FUNCTIONAL REQUIREMENT Figure �� Categories of functional requirements and design features The next stage was to create a KADS expertise model to represent all the ac� quired knowledge from the di�erent viewpoints speci ed by KADS� In this project� the domain level of the model of expertise was developed rst� it was decided that the domain level consisted of the various functional requirements and design fea� tures� Once the domain level had been completed� the inference structure was developed� It was decided that the task type being performed was heuristic classi� �cation� The generic inference structure for heuristic classi cation and the instan� tiated inference structure which forms part of the expertise model can be seen in Figure �� It can be seen that the level of abstraction of the probing questions varies considerably� the conditions might be based on general knowledge of the operation of KBS systems� or on speci c knowledge of the task under consideration� and the recommendations might be to use a particular design feature� or to use one of a category of design features� Finally� a task structure was developed� which speci ed that the KBS should continue to ask questions until all possible relevant questions had been asked� in order to ensure that all functional requirements are considered� �� DESIGN FEATURE MATCH ANALYSED EXPERT KNOWLEDGE KNOWLEDGE OF KBS ENVIRONMENT CLASS FEATURE DESIGN REFINE FUNCTIONAL REQUIREMENTS Inference structure -- adapted and instantiated AND INTERACTION MATCH SOLUTION ABSTRACTION SOLUTION ABSTRACTION PROBLEM ABSTRACT PROBLEM REFINE Inference structure -- generic: heuristic classification Figure �� Generic and instantiated inference structures for a heuristic classi cation task ��� The project� design As the design is a relatively late stage in the development of the KBS� the student was able to apply the elicited probing questions to his own design problem� The re� sults of this exercise can be seen in an appendix to the project thesis �MacNee� ������ in which the nal implemented system was �retrospectively� applied to itself� The main resulting recommendations are summarised in the following table �� Design feature Recommendation Shallow reasoning Strong Rules Strong Goal driven reasoning Moderate Depth rst search Moderate Truth maintenance Moderate Certainty factors Moderate !Canned� text for explanations Moderate Data driven reasoning Weak Model�based reasoning Strong negative It is hardly surprising that a knowledge base which is based around heuristic probing questions should elicit strong recommendations for shallow reasoning and for the use of rules� The justi cation for some of the other recommendations is less obvious� this was noted during the project� and it was therefore decided that the implemented system would need to be able to justify its reasoning to the user in a clear and coherent manner�� An example of an explanation� based on the table above� is that the recommendation for depth� rst search arose because of the need to ask questions of the user� and because the student stated that the natural ow of dialogue was to ask detailed questions about one subject before asking general questions about another subject� Depth� rst search supports this mode of reasoning� and therefore makes it easy to ask questions in a natural order� ��� The project� platform design � implementation Having performed architectural design� the nal stage of design must be performed the matching up of the recommended design features with the chosen programming tool� The decisions required at this stage are described in more detail in section �� but for now� it is su�cient to note that the the probing questions should be consid� ered as a starting point for tool selection� not as a prescription� For this project� a choice of two programming tools was available CLIPS and Prolog� The table above indicates a stronger recommendation for goal�driven reasoning than for data� driven reasoning� which would favour Prolog� however� bearing in mind that the probing questions are heuristics� the factors contributing to this recommendation were examined in more detail� It turned out that the recommendation for goal� driven reasoning was based on a combination of three weak contributing factors� and one of these factors is actually not applicable in this case� because the system is designed to make the KBS ask all possible questions� instead of stopping when a single solution has been found� The recommendation for goal�driven reasoning was therefore downgraded to !weak�� �This decision accounts for the recommendation for �canned� text in the above table� �� CLIPS and Prolog are both equally strong in most of the other recommended de� sign features �although CLIPS does provide its own facilities for truth maintenance� which Prolog does not�� the conclusion is that� from the viewpoint of providing ad� equate design features� CLIPS and Prolog are both equally recommended for this project� The choice of tool was therefore heavily in uenced by other factors� such as the student�s previous experience� The tool chosen was CLIPS� a largely rule�based tool whose primary reasoning mechanism is depth� rst forward chaining� CLIPS also provides facilities for truth maintenance and certainty factors� Using CLIPS� the PDQ system �which� in this context� stands for �Probing Design Questions � was implemented in approximately � weeks� ��� Results PDQ is a workable system which produces recommendations which are helpful� although heuristic� as the above example concerning goal�driven reasoning shows� The questions require the knowledge engineer to have a good understanding of the problem� and some preliminary ideas about possible designs� for example� one question asks if the problem �can be subdivided into �� or more distinct modules � This suggests that PDQ is most approriate for knowledge engineers who have some experience of building knowledge based systems� Some further work has been done on the set of probing questions since the PDQ system was completed� in an attempt to separate those questions which produce abstract recommendations from those which recommend more speci c design fea� tures� This process has lead to the transferral of a small set of probing questions to the rst stage of the design process �application design�� because some design features �such as blackboard architectures or constraint�based programming� have such a profound e�ect on the architecture of a program that they must be con� sidered as alternative ways of performing the initial decomposition of the analysed knowledge� The probing questions are therefore able to provide some guidance on whether to perform a structure�preserving design� or whether to make use of an es� tablished AI paradigm� This decision is another of the key decisions for knowledge engineers listed in the introduction to this paper� It is possible that the �prob� ing questions technique could be extended to provide extensive support for this important decision� � Choosing a suitable implementation tool The nal project described in this paper aimed to produce guidance on the se� lection of a suitable shell or toolkit for implementing a knowledge based system �Robertson� ������ �� ��� The task The two projects described above have shown that knowledge�based guidance can be provided for certain areas of KADS modelling� However� the knowledge engineer�s decision making does not end when the �probing questions have been answered and the KADS design has been completed� as section ��� suggests� the choice of a programming tool requires careful consideration� taking into account both the recommendations of the probing questions and other factors external to the knowledge representation requirements� The requirements for this project were that the student should identify factors important to the selection of a KBS building tool� and develop a program which would recommend appropriate tools for a project� Given that the PDQ system had already been developed� it seemed sensible to use PDQ�s recommendations as a starting point for the identi cation of important factors� The student also referred to a number of books on knowledge engineering �e�g� �Price� ������ which gave their own advice on tool selection� ��� The project� acquisition Knowledge acquisition for this project was initially performed by reading various textbooks� and examining the output of the PDQ system� The results of this work were compiled� and represented graphically using a simple node and arc represen� tation� These diagrams� and associated text� were used as input to a knowledge elicitation interview with an experienced knowledge engineer� This interview pro� duced further useful information� which was used to alter and extend the diagrams� Finally� the student was directed to investigate a set of expert system building tools which was representative of all the categories which he had identi ed� The major result of the knowledge acquisition was two classi cations of KBS building tools � The rst classi cation divided tools into shells� toolkits and AI languages� These categories were further subdivided� shells were divided into those which evaluate rules by pattern matching �e�g� Xi Plus� OPS�� early versions of CLIPS� and those which are e�ectively �rule networks �e�g� Crystal��� Toolkits were subdivided into top�range toolkits �such as ART and KEE� and mid�range toolkits �such as Nexpert Object� ProKappa and Kappa PC�� Languages were not subdivided� since there are comparatively few popular languages for implementing knowledge based systems� � The second classi cation was based on the historical background of the tools� tools were classi ed as belonging to the �ART Camp �e�g� ART�IM� CLIPS� �This distinction was rst de ned in �Inder et al� ��� � �� ECLIPSE� or the �KEE Camp �including ProKappa and Kappa PC�� Tools in the same !camp� have similar features to each other� since many of them are derived from similar programming philosophies or tools� for example� all tools in the ART camp o�er e�cient forward chaining rules based on an implementation of the RETE algorithm� but are comparatively weak on backward chaining� because the algorithm used o�ers almost no support for backward chaining� whereas all tools in the KEE camp have good support for object�oriented programming� but comparatively ine�cient rules�� The other main conclusion from the knowledge acquisition phase �largely drawn from �Rothenberg� ������ was that the importance of particular features of a pro� gramming tool is dependent on the phase of the project� For example� at the very early stages of the project� rapid prototyping might be used to support knowledge acquisition� to investigate the feature of the tool� or to help the knowledge engineer learn to use the tool� at this stage� the training and debugging features of the tool are very important� However� when the nal system is delivered� the training and debugging tools are of little or no importance� whereas the speed and e�ciency of the execution of the program become very important� The student identi ed ve phases of program development �exploration� prototyping� development� elding and operation� and seven features of a tool �ease of use� e�ciency� extendability� exibility� portability� reliability and support� which are more or less desirable at various phases� For further details of the proposed relationship between phases of development and tool features� see �Rothenberg� ����� or �Robertson� ������ ��� The project� analysis � � � Domain level analysis The domain level of the expertise model comprised the classi cations identi ed in the knowledge acquisition phase� plus a selection of factors which in uence the choice of tool �such as the cost of the tool� the required platform for development� and the experience of programmers with the tool�� It also de ned a prototype !frame�� with a number of attributes �or� in the terminology of CommonKADS� a concept� with a number of properties� for de ning tools� This frame proved remarkably di�cult to de ne� because of di�culties in distinguishing properties of tools from tool�related concepts� For example� there was considerable discussion on whether the frame should have �forward chaining as a property �with values such as rete� procedural or none� or whether it should have �reasoning types as a multiple�valued property� with forward chaining being one of the values of �See �Mettrey� ���� for some benchmarking tests� Note� however� that the vendors of KAPPA� PC claim considerable performance improvements in newer versions of KAPPA�PC� which have appeared since Mettrey�s study was done� �� this property� This discussion led to an important obaservation concerning KADS� which is described in section ���� � � � Inference level analysis When the inference level is considered� it can be seen that this project is similar in structure to the PDQ project PDQ identi ed functional requirements which had to be mapped to design features� whereas this project uses various design features and other requirements to select a suitable tool� or class of tool� However� while the task of PDQ was to identify all relevant design features� the task required when selecting a tool is to select the best tool for the task� This requires comparison of a set of possible tools against all relevant factors� and assessment of how well each tool matches the overall set of criteria� The most appropriate task type in the KADS taxonomy is therefore assessment� rather than heuristic classi cation� By the time this project was carried out� some guidance had been published by members of the CommonKADS consortium �L�ockenho� � Valente� ����� on the con guration of inference structures to match particular assessment tasks� Starting with a minimal generic inference structure �Figure ��� this guidance consists of a set of questions which are asked of the user in order to decide which of the inference steps relevant to assessment tasks are actually performed in this task� The results of these questions are used to devise a problem�speci c version of the inference structure �e�g� Figure ��� which is then instantiated to the problem in hand by changing the generic labels on the knowledge roles into problem�speci c labels� The nal result �Figure �� forms the inference level of the model of expertise� A worked example of the con guration of an inference structure can be found in �Kingston� ������ System ModelCase Description Decision Class Match Cases Figure � Minimal generic inference structure for assessment tasks �� Abstract Case Description Measurement System Decision Classes Measure Case Set of Norms Specify Set of Norms Specify Conflict Resolution Conflict Resolution Criteria Resolve Conflicts Decision ClassDecision Class Resolve Conflicts Conflict Resolution Criteria Specify Conflict Resolution Specify Set of Norms Set of NormsMeasure Case Decision Classes Measurement System Abstract Case Description Abstract Case Description Measurement System Decision Classes Measure Case Set of Norms Specify Set of Norms Specify Conflict Resolution Conflict Resolution Criteria Resolve Conflicts Decision ClassDecision Class Resolve Conflicts Conflict Resolution Criteria Specify Conflict Resolution Specify Set of Norms Set of NormsMeasure Case Decision Classes Measurement System Abstract Case Description Figure �� Con gured inference structure for selecting a KBS tool �� Desired Product Features Available Products Measure Case Specify Set of Norms Available Product Features Chosen Products Conflict Resolution Criteria Specify Conflict Resolution Resolve Conflicts Ideal Product Figure �� Instantiated inference structure for selecting a KBS tool It can be seen that the task of selecting a KBS tool requires comparison of the features of available tools against the features desired for a particular project� this produces a shortlist of suggested tools� which is then further re ned to choose the ideal tool for the job� ��� The project� design and implementation The con gured inference structure contributed greatly to the quick and accurate development of an expertise model� Once the model was complete� design was per� formed� with the assistance of the PDQ system� PDQ produced very strong recom� mendations for using both rules �to represent the applicability of di�erent factors� �� and frames �to represent individual tools�� it also produced a strong recommen� dation for goal�driven reasoning� and a moderate recommendation for data�driven reasoning� At this point� the student was o�ered the choice of two tools CLIPS ��� or Sic� stus Prolog� The choice was restricted to two tools because these tools were easily available� and because the student had some familiarity with both these tools� the latter factor was important because of the tight timescales of the project� The lim� ited choice simpli ed the decision process for reasons described below� Prolog was chosen as the most appropriate tool� The system was therefore implemented using Prolog� using a goal�driven approach which investigated several di�erent categories of tool features one by one� The details of fteen di�erent KBS tools were also implemented� using a predicate called �frame which was de ned to allow the rep� resentation of object�like structures� The features of each tool were then matched against the features required by the user� the relative importance of each feature in the overall assessment was scaled according to the phase of development for which the tool would be used� and according to an importance value entered by the user� The nal list of tools� ordered by their overall score� was then displayed to the user� In order to test the system� it was retrospectively applied to the choice between CLIPS and Prolog for the student�s own project� The results of the consultation showed that Sicstus Prolog and CLIPS were among the more highly favoured tools� although they were not at the top of the list� however� most of the tools which were preferred were considerably more expensive� The main factors which led to Prolog being preferred were the recommendation for goal�driven reasoning from the probing questions� the problems in integrating rules and objects in version � of CLIPS� � and the student�s greater degree of familiarity with Prolog� which led the system to believe that features which were unavailable in Prolog �such as frames� could easily be programmed� It therefore seems that the choice of Prolog was the best decision from the available options� ��� Results This system can be considered to be a sophisticated prototype� Its reasoning ca� pabilities are wide�ranging �over many di�erent tool features and aspects of KBS development� and its algorithm for assessing tools provides results which closely resemble the opinions of expert knowledge engineers� in short� the heart of the sys� tem is su�ciently good to be commercially usable� It might nd favour as a tool for organisations to assess the adequacy of their existing tools for a proposed project� rather than to recommend a particular too� for a project which is already under Version � of CLIPS included a full object�oriented programming system� but these objects could not be pattern matched in the conditions of rules� Version � of CLIPS has introduced this facility� �� way� However� its knowledge base needs to be extended to include more tools� and it would bene t from improvements to its user interface and e�ciency� The use of KADS modelling and the probing questions has led to a knowledge base architecture which is well supported by documentation and justi cation more importantly it can be extended easily� because new tools can be added to the knowledge base simply by de ning a new !frame�� Ease of maintenance is a vital feature for a system which provides expertise about a domain which changes as rapidly as this one� � Discussion The three projects described above have produced knowledge based systems which provide guidance on particular decisions which users of CommonKADS have to make� The SEXTANT system assists knowledge engineers in the early stages of knowledge analysis� when they are selecting a suitable generic inference structure from CommonKADS� library of inference structures� PDQ provides suggestions for the architectural design of a knowledge based system� and the tool selection system provides guidance on matching the recommendeddesign with available shells or toolkits� The guidance which has been produced seems to be useful PDQ in particular has been used by later generations of students� and on commercial projects� It is important to realise that these systems are not intended to take over the decision�making role of a knowledge engineer� The guidance which these systems provide is heuristic� in some cases� careful analysis might suggest that the guidance should not be followed �see section ��� for an example�� The guidance also requires an understanding of knowledge engineering terminology� as well as an in�depth understanding of the problem to be solved� in order to make the best use of the advice� The key bene t of the guidance is in assisting knowledge engineers by providing a framework for performing the required analysis� just as CommonKADS provides a framework for representing knowledge� It does this by identifying the questions which need to be asked in order to make a particular decision� as well as providing suggested answers to those questions and �in some cases� justi cation for those answers� The projects also produced new insights about the modelling techniques rec� ommended by KADS and CommonKADS� The students found that using KADS helped them to think clearly about their knowledge bases and to identify speci c areas where problems occurred� However� KADS and CommonKADS are intended to assist rather than to replace the judgement of a knowledge engineer� the students all discovered that the KADS approach did not provide a �magic wand solution to all the problems which they encountered� For example� the SEXTANT project originally considered the choice of an appropriate task type to be an assessment �� task� but it was later discovered that a simpler approach� based on a decision tree� could be used to accomplish most of the job� Another example can be seen in the domain modelling for the tool selection project� where the student had di�culty in deciding whether �forward chaining should be a property or a value of a property� The crux of the problem was that neither approach could be ruled incorrect at a theoretical level� the choice between them depended on purely pragmatic con� straints� such as the number of possible values of the property� and the number of possible attributes of the concept� In other words� the de nition of concepts and properties in a domain appears to be context�dependent� This observation has far�reaching implications for CommonKADS domain modelling� in that it suggests that there is more than one !correct� model of any chosen domain of knowledge� There is therefore scope for further research on CommonKADS� identifying di�er� ent styles of domain modelling and the circumstances in which di�erent approaches are most appropriate� In terms of a single project� the choices of classi cations at the domain level are unlikely to a�ect the functionality of the nal system very much� although they may a�ect the ease of achieving that functionality� � Summary In summary� CommonKADS provides a declarative framework for representing knowledge� which can be used to promote clearer thinking and structuring of a knowledge base� CommonKADS is most appropriate for knowledge engineers who have enough experience to be able to make their own decisions about knowledge analysis or design in those exceptional circumstances when CommonKADS� rec� ommendations prove to be unhelpful� CommonKADS itself is su�ciently complex that knowledge engineers require experience with and�or guidance on the KADS approach to use it to its full extent� The projects described in this paper provide guidance for some of the key decisions which must be made when using Com� monKADS� This guidance is aimed at those people who would be capable of using CommonKADS it o�ers heuristic advice� which is normally good advice but may need to be overridden in exceptional circumstances� The results of these projects suggest that it is feasible to produce knowledge� based guidance for users of KADS or CommonKADS� as well as producing some new insights about domain modelling and task selection� It is hoped that the results of the projects above can be amalgamated with other projects which are producing guidance of CommonKADS �for tasks such as con guring inference structures �L�ockenho� � Valente� ����� and converting CommonKADS� Concep� tual Modelling Language to its Formal Modelling Language �Aben� ������� in order to make CommonKADS a more powerful tool for structuring knowledge engineer� ing� �� Acknowledgements I would like to thank Dr Dave Robertson and Dr Mandy Haggith of the Department of Arti cial Intelligence� University of Edinburgh� Ian Filby of AIAI� and Dr Robert Inder of the Human Communications Research Centre� University of Edinburgh for providing the students with supervision and expertise� I would also like to thank Dr Robertson� Dr Haggith and Robert Rae of AIAI for their comments on drafts of this paper� References �Aben� ����� Aben� M� ������� Formal methods in Knowl� edge Engineering� Unpublished Ph�D� thesis� SWI� University of Amsterdam� The relevant chapter is also available as CommonKADS report KADS� II�T����WP�UvA��������� �Breuker�� ����� Breuker�� J� A� ������� Model�driven Knowledge Ac� quisition� University of Amsterdam and STL� ES� PRIT project ����� Deliverable A�� �deHoog et al� ����� de Hoog� R�� Martil� R�� Wielinga� B�� Taylor� R�� Bright� C� and van de Velde� W� ������� The Common KADS model set� ESPRIT Project P���� KADS�II KADS�II�M��DM���b�UvA����� ���� University of Amsterdam and others� �Inder et al� ����� Inder� R�� � Aylett� R�� Bental� D�� Lydiard� T� and Rae� R� �October ������ Study on the Evaluation of Expert Systems Tools for Ground Segment Infras� tructure Final Report� Technical Report AIAI�TR� ��� AIAI� �Kingston� ����� Kingston� J�K�C� ������� Re�engineering IMPRESS and X�MATE using CommonKADS� In Expert Sys� tems ��� British Computer Society� Cambridge Uni� versity Press� Also available as AIAI�TR����� �Kingston� ����� Kingston� J�K�C� ������� Design by Exploration A Proposed CommonKADS Inference Structure� Sub� mitted to �Knowledge Acquisition�� �� �Kline � Dolins� ����� Kline� P�J� and Dolins� S�B� ������� Designing ex� pert systems � a guide to selecting implementation techniques� Wiley� �Krueger� ����� Krueger� A� �Sept ������ Classi cation of Expert Tasks the SEXTANT system� Unpublished M�Sc� thesis� Dept of Arti cial Intelligence� University of Edinburgh� �L�ockenho� � Valente� ����� L�ockenho�� C� and Valente� A� �March ������ A Li� brary of Assessment Modelling Components� In Pro� ceedings of �rd European KADS User Group Meeting� pages �������� Siemens� Munich� �MacNee� ����� MacNee� C� �Sept ������ PDQ A knowledge�based system to help knowledge�based system designers to select knowledge representation and inference tech� niques� Unpublished M�Sc� thesis� Dept of Arti cial Intelligence� University of Edinburgh� �Mettrey� ����� Mettrey� W� �February ������ Expert systems and tools Myths and realities� IEEE Expert� �Price� ����� Price� C� ������� Knowledge Engineering Toolkits� Ellis Horwood� This book concentrates on available toolkits� and on how to go about selecting an appro� priate toolkit� While some of the tools it mentions are now somewhat dated� the principles of tool se� lection are still valid� It also gives clear examples of implementation� �Robertson� ����� Robertson� S� �Sept ������ A KBS to advise on se� lection of KBS tools� Unpublished M�Sc� thesis� Dept of Arti cial Intelligence� University of Edinburgh� �Rothenberg� ����� Rothenberg� J� ������� Expert System Tool Evalu� ation� In Guida� G� and Tasso� C�� �eds��� Topics in Expert System Design� North�Holland� �Schreiber et al� ����� Schreiber� A� Th�� Wielinga� B� J� and Breuker� J� A�� �eds��� ������� KADS� A Principled Approach to Knowledge�Based System Development� Academic Press� London� �� �Tansley � Hayball� ����� Tansley� D�S�W� and Hayball� C�C� ������� Knowledge�Based Systems Analysis and Design� A KADS Developers Handbook� Prentice Hall� �van de Velde et al� ����� van de Velde� W�� Duursma� C� and Schreiber� G� ������� Design model and process� Unpublished KADS�II�M��MM�VUB� Vrije Universiteit Brussel� �Wielinga� ����� Wielinga� B� �October ������ Expertise Model Model De nition Document� CommonKADS Project Report� University of Amsterdam� KADS� II�M��UvA��������� �Wielinga et al� ����� Wielinga� B�� Van de Velde� W�� Schreiber� G� and Akkermans� H� ������� The KADS Knowledge Mod� elling Approach� In Proceedings of the Japanese Knowledge Acquisition Workshop JKAW�� �� �Wielinga et al� ����� Wielinga� B�� van de Velde� W�� Schreiber� G� and Akkermans� H� �Jun ������ Expertise Model Def� inition Document� CommonKADS Project Report� University of Amsterdam� ��