The development of an expert system for managerial evaluation of internal controls Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) MANAGERIAL EVALUATION OF INTERNAL CONTROLS 103INTELLIGENT SYSTEMS IN ACCOUNTING, FINANCE AND MANAGEMENT Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002 /isaf.246 Copyright © 2004 John Wiley & Sons, Ltd. * Correspondence to: Clyde W. Holsapple, School of Management, Carol M. Gatton College of Business Administration, University of Kentucky, Lexington, KY 40506-0034, USA. E-mail: cwhols@pop.uky.edu THE DEVELOPMENT OF AN EXPERT SYSTEM FOR MANAGERIAL EVALUATION OF INTERNAL CONTROLS CHULEEPORN CHANGCHITa AND CLYDE W. HOLSAPPLEb* a College of Business, Texas A&M University-Corpus Christi, USA b School of Management, Carol M. Gatton College of Business Administration, University of Kentucky, USA SUMMARY Both practitioners and researchers have devoted significant effort to the study of decision aids, especially expert systems, to assist auditors in internal control evaluations. In addition to being used as a decision aid, researchers have long contended that expert systems could be used to train non-expert users. Even though the professional accounting literature makes it clear that responsibility for maintaining an effective internal control system rests with management rather than auditors, the focus to date has been on expert systems aimed at assisting/training auditors, not an organization’s management. In contrast, this study focuses on management as users of an expert system for internal control evaluation. We describe the development process, explain how the resulting system was evaluated, and discuss results of that evaluation. These results suggest that such a system gives a new way to help managers increase effectiveness and efficiency of a critical organizational process: the evaluation of internal controls. Copyright © 2004 John Wiley & Sons, Ltd. 1. INTRODUCTION A number of reviews have been published dealing with the nature of expertise in auditing (Marchant, 1989; Bonner and Pennington, 1991; Fedorowicz et al., 1992; Libby, 1995; Lieb and Gillease, 1996; Weber, 1999). Among these reviews, several have examined experienced auditors’ internal control knowledge (Eining and Dorr, 1991; Libby, 1995). The study and evaluation of internal controls is a problem involving the expertise of well-trained auditors. The internal control evaluation process is characterized by the use of heuristic rules to determine how well the client’s controls support specific assertions for specific accounts (Gadh et al., 1993). Both practitioners and researchers have devoted significant effort to the study of audit decision aids to assist auditors in internal control evaluations (Bailey et al., 1985; Boritz, 1985; Gal, 1985; Cummings and Apostolou, 1987; Cummings et al., 1988; O’Leary and Watkins, 1989; Brown and Phillips, 1990; Murphy, 1990; Graham et al., 1991; Vinze et al., 1991; Messier, 1995). Several researchers suggest that audit judgment can be improved through the use of decision aids (Messier, 1995; Brown and Jones, 1998). Among these decision aids, expert systems have evolved to become practical tools used in aiding decision making (Bonczek et al., 1981; Holsapple and Whinston, 1987; O’Leary, 1988; Biggs and Morrison, 1990; Wong and Monaco, 1995). 104 C. CHANGCHIT AND C. W. HOLSAPPLE Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) In addition to being used as a decision aid, researchers have long contended that expert systems could also be used to train non-expert users (Biggs et al., 1987: Borthick and West, 1987; Ege and Sullivan, 1990; Gal and Steinbart, 1992; Pei and Reneau, 1992; Bonner and Walker, 1994; Steinbart and Accola, 1994; Odem and Dorr, 1995). Prior studies found that subjects who practiced making decisions with the aid of an expert system were better and quicker at reaching decisions than subjects who practiced without the support of the expert system (Oz, 1989; Eining and Dorr, 1991; Fedorowicz et al., 1992). Expert systems for internal control evaluation are generally developed using either professional literature for a knowledge source, or interview or protocol analysis to acquire knowledge from human experts (Meservy et al., 1986; Holsapple and Raj, 1994). The resultant system is able to emulate the results of an auditor’s internal control evaluation process. The professional accounting literature makes it clear that the responsibility for maintaining an effective internal control system rests with management rather than with the auditors (COSO, 1992; Arens and Loebbecke, 1994). However, the focus to date has been on expert systems aimed at assisting or training auditors, not an organization’s management. Most of these systems are designed to help auditors determine the extent of other tests they will perform in conducting an audit. If an auditor determines that the client’s internal control systems are designed properly and are functioning as designed, he or she can reduce direct testing of account balances accordingly (Gadh et al., 1993). Although the literature does not report on expert systems that can facilitate the transfer of internal control knowledge to managers, there are several reasons why such systems deserve to be invest- igated. First, establishment and supervision of internal control systems are the responsibility of management, not external auditors (COSO, 1992). Second, managers have more immediate and detailed insight into operations (including their internal control facets) than external auditors. Third, decisions made by managers often have internal control implications, but they may not be recog- nized or adequately considered due to managers’ insufficient internal control knowledge. Fourth, evaluation of internal control systems ideally should be treated as a continuous process, allowing weaknesses to be prevented or detected as soon as possible. Moreover, in the wake of recent financial scandals, the Sarbanes–Oxley Act of 2002 was enacted to emphasize the importance of vigilant and effective corporate governance participants—including the board of directors, the audit committee, management, internal auditors, external auditors, and governing bodies (e.g. Securities and Exchange Commission, American Institute of Certified Public Accountants)—can play in preventing and detecting financial statement fraud (Rezaee, 2004). The act requires public companies to validate the accuracy and integrity of their financial management. The processes and documentation required for compliance are rigorous: companies must have estab- lished procedures for meeting their reporting obligations, and CEOs and CFOs must personally certify that their own company’s statements are complete and accurate. Having said this, we do not imply that internal control evaluations should be performed by management instead of auditors. Rather, we contend that ongoing attention to internal control issues by management would be a useful complement to ordinary auditing activity. However, this requires that managers possess or have easy access to internal control evaluation knowledge. The expert system described in this paper is aimed at transferring such knowledge to managers. The remainder of this paper begins with an overview of system objectives. This is followed by a description of the development process, including knowledge acquisition, rule set specification, and expert testing. We discuss how the resulting system was evaluated in experimental sessions with practicing managers. The results give strong evidence that an expert system can be valuable in both directly supporting managers’ internal control decisions and training them to make such decisions. Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) MANAGERIAL EVALUATION OF INTERNAL CONTROLS 105 The major contribution of this work is its exploration of a new way to increase the effectiveness and efficiency of a critical organizational process: the evaluation of internal controls. 2. OBJECTIVES This study is unique in its focus on managers as users of an expert system for internal control evaluations. Even though there is evidence in some problem domains that the use of expert systems can facilitate the transfer of expertise to novice subjects, we cannot automatically assume that this is feasible or effective for the transfer of internal control evaluation knowledge to experienced managers. Managers come from diverse backgrounds (marketing, operations, technical, etc.) and do not necessarily have an accounting educational background (Viator and Curtis, 1998). Differences in educational background have been directly linked to differences in knowledge structure (Curtis and Viator, 2000). Studies have shown that a mismatch between knowledge structure and task structure can have a detrimental effect on performance (Pei et al., 1994; Nelson et al., 1995), and differences in knowledge structure are systematically associated with the quality of performance in reviewing internal control systems (Curtis and Viator, 2000). Given the variations in managers’ educational backgrounds, work experiences, and knowledge structures, constructing an expert system for effec- tive internal control evaluation by managers (as compared with external auditors or accounting student novices) may not be feasible. Moreover, real managers may resist learning about internal controls, or they may feel uncomfortable using an expert system to do so. A manager may provide incorrect inputs, not understand system outputs, find it difficult to work with an expert system, or resist considering the advice it gives. The central objective of this study is to explore whether the foregoing reservations can be over- come. This exploration has two aspects: building an expert system aimed at helping managers evaluating internal control weaknesses and demonstrating that practicing managers can use this system in a way that results in the transfer of internal control knowledge to them. Because the targeted users are managers, who tend not to be well versed in internal control evaluation concepts, the required features of the system are different from those of expert systems developed for auditors. Specifically, the system was designed to meet the following objectives: • The system should aid users in understanding why internal control is important. It is divided into five sections based on internal control objectives. Each section begins with an introduction to the basic concepts of such control. • The system should aid users in understanding how a particular problem is solved. For instance, several diagrams are incorporated into the interface in order to help users capture the concept of how a weakness is detected so that the learning is enhanced. • The system should be easy to understand and use by persons experienced with business operations but who are unfamiliar with internal control concepts. For example, the system prompts users to enter data relevant to detecting internal control weaknesses (e.g. the name of the person who re- ceives cash, the name of the person who records cash receipts), and reports each weakness found. • The system should report each weakness found, not just the adequacy of the overall organization’s internal control system. • The system should provide the reason why each weakness found is considered to be an internal control weakness. 106 C. CHANGCHIT AND C. W. HOLSAPPLE Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) 3. DEVELOPMENT PROCESS Expertise has been defined as the knowledge about a particular domain, understanding of domain problems, and skill in solving such problems. The nature of expertise includes the ability to (1) solve the problem, (2) explain the result, (3) learn, (4) restructure knowledge, (5) break rules, (6) deter- mine relevance, and (7) degrade gracefully (Davis and Lenat, 1982). An expert’s knowledge has both public and private aspects. Public knowledge includes the facts, theories, and definitions as found in texts and journals referenced by those studying in the domain, whereas much private knowledge is in the form of rules of thumb referred to as heuristics (Hayes-Roth et al., 1983). Heuristics allow experts to make educated guesses when necessary, to recognize promising ap- proaches to problems, and to deal effectively with errorful or incomplete data (Meservy, 1985). The review and evaluation of internal accounting controls is a critical step in every financial audit and is an area in which the auditor exhibits substantial expertise (Meservy, 1985). To meet the objectives outlined in Section 2, it is necessary to determine the reasoning that auditors use in evaluating a client’s internal control system, formalize and represent that reasoning knowledge as rule sets stored in a knowledge system, and then test the usefulness of the resultant expert system. The knowledge incorporated into this expert system formalizes the expertise of an auditor experi- enced in evaluating an internal control system of the sales and collection cycle in medium-size merchandising organizations. Typically, the strengths and weaknesses of an internal accounting control system are evaluated by determining control objectives, identifying controls and faults from a description of the system, and then combining the controls and faults into an overall evaluation of the sufficiency with which each control objective has been met. The expert from whom the knowledge for this expert system was acquired is a partner in a major international accounting firm. He has more than 10 years of experience in the area of internal control evaluation and demonstrated significant interest in the research project. The tool selected for developing the expert system was the GURU integrated artificial intelligence environment, version 3.01 (Holsapple and Whinston, 1987; Micro Data Base Systems, 1991). Beyond the rule management facilities typically found in expert system shells, GURU provides integral facilities for relational database management, forms management, program management, and spreadsheet management. These are integrated into a single problem-processing system in such a way that any of these knowledge handling techniques can be used independently or several can be used interdependently. In this particular expert system application, the techniques used included rule management, program management, forms management, and database management. For in- stance, conclusions of some rules invoked program modules which, in turn, invoked operations on form specifications. Once the source of expertise was identified and the development tool was selected, knowledge acquisition began. 3.1. Knowledge Acquisition Method Knowledge acquisition plays an important role in expert system development. It is evident that the quality of the resulting system is dependent on the quality of the knowledge originally elicited (Bolger and Wright, 1994; Moody et al., 1998). The knowledge acquisition process began with the question: What are the processes that auditors are using to arrive at internal accounting control evaluations? The task of reviewing and evaluating internal accounting was then analyzed, resulting in the identification and characterization of important aspects of the task. Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) MANAGERIAL EVALUATION OF INTERNAL CONTROLS 107 Knowledge for the expert system was acquired via a 6 month series of interviews with the expert. The expert was asked to identify all potential weaknesses that might occur in the sales and collection cycle of a medium-size merchandising organization. The expert was asked further to describe, in detail, the techniques and processes that were used to discover each of these weaknesses in a client’s internal control system. The rationale for each heuristic was also acquired in an attempt to develop an expert system that would be able to emulate both the expert’s knowledge and their reasoning behavior. Figure 1 shows how the interview process was conducted. The interview questions were divided into two streams. The first line of questions was asked to examine what descriptive knowledge was needed by the expert auditor in order to evaluate clients’ internal control systems. Such questions were then incorporated into the presentation system of the expert system for eliciting the users’ inputs (which comprised the expert system’s language system). The second stream of questions was asked to understand what reasoning knowledge the expert auditor employed in drawing inferences from the descriptive knowledge. This reasoning knowledge was then incorporated into the knowledge system of the expert system as rule sets. Interestingly, consistent with the finding of Frederick (1991), during the interview process, the expert first identified each transaction required in sales and collection for a merchandizing organization schematically (by flows of transactions). All the potential weaknesses that could occur in each of these transactions were then identified. The expert then regrouped these weaknesses taxonomically by objective. Figure 2 presents a resultant model for assessing internal controls in the sales and collection-cycle functions of a medium-size merchandising organization. Figure 1. A diagram of the interview process 108 C. CHANGCHIT AND C. W. HOLSAPPLE Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) Figure 2. Model of internal controls for a sales and collection cycle in a merchandising organization 3.2. Rule Set Specification In order to construct the expert system’s rule sets, all variables needed were identified in each of the basic internal controls grouped by objective. For example, in order to identify all variables needed for checking the control for adequate segregation of duties, the expert started by describing all departments involved in sales and collection cycles of a merchandising organization. Each func- tion required in each department was then identified and represented as a variable. Logical variables (Yes/No) were also used to check on the existence of certain controls. For instance, in order to check whether the functions of recording cash/cheques and the function of receiving cash/cheques were performed by the same person or persons who have a family relationship, the variables needed were identified and represented as follows: • The function of recording cash/cheques was represented as RCQ. • The function of receiving cash/cheques was represented as ECQ. • A logical variable was used to check whether the persons who perform these functions are related. Once all variables needed were identified, knowledge acquired from the expert was represented in sets of rules. These rule sets were also grouped according to the basic internal controls by objective. Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) MANAGERIAL EVALUATION OF INTERNAL CONTROLS 109 Figure 3. A diagram for checking the existence of proper authorization of sales orders, sales invoices, credit memo, and changes in payment conditions memos 3.3. System Outputs Several diagrams were devised to facilitate learning by the system’s users. For example, each restriction between functions was analyzed and represented first as a diagram describing all require- ments for adequate segregation of duties. The diagrams were then incorporated into the expert system as form specifications to provide users with an idea of what duties would be detected by the subsequent rule sets. Figure 3 presents an example of a diagram for detecting whether there is a proper authorization for sales orders, sales invoices, credit memos, and changes in payment conditions memos. The screen displaying this diagram is followed by a menu that prompts the user to select the person required to approve a specific document. For instance, Figure 4 presents a menu for selecting the person required to approve a sales invoice. In order to ensure that the user does not select a menu option accidentally, the next screen prompts the user for verification of his/her selection. Figure 5 presents an example of a verification screen. By processing sets of rules, the system infers a recommendation of potential internal control weaknesses. This recommendation identifies significant internal control weaknesses discovered in the situation being evaluated and indicates resulting exposures that could occur in a user’s organi- zation. For example, if the user does not identify the right person(s) required to approve sales invoices, then a weakness in internal control is reported, along with reasons why the expert considers this circumstance to be a weakness. Figure 6 presents an example of the weakness found for improper authorization of sales invoices. 110 C. CHANGCHIT AND C. W. HOLSAPPLE Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) Figure 4. A menu for selecting the person who is required to approve sales invoices Figure 5. A verification of user’s selection Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) MANAGERIAL EVALUATION OF INTERNAL CONTROLS 111 Figure 6. A report of a weakness found for improper authorization of sales invoices Figures 7 and 8 present additional examples of recommendations for weaknesses found in the internal control system. Such weaknesses indicate an inadequate segregation of duties by: • Having the same person doing both the functions of preparing sales invoices and recording sales invoices. • Having a relationship between the person who records cash /cheques and the person who receives cash /cheques. 3.4. Evaluating the System’s Validity Validation is often considered the cornerstone of expert system evaluation (Back, 1994). It is the process of analyzing the knowledge and decision-making capabilities of the expert system (O’Leary, 1988). Comparison of an expert’s recommendations with those of the expert system is a major criterion for evaluating that system’s validity. Test cases are developed and used to examine whether the expert system offers sufficiently good and timely advice compared with the human expert. For the expert system described here, test cases were generated from the manipulation of several cues for detecting potential weaknesses in an internal control system over the sales and collection cycle. These cues were obtained from a review of auditing texts, accounting texts, and input from accounting professors and experienced auditors. The expert was asked to evaluate each test case and detect its potential internal control weaknesses. Reasons for each potential weakness were also requested. Then, the prototype expert system was used to detect the potential weaknesses and offer reasons for such weaknesses as well. The results were then compared. 112 C. CHANGCHIT AND C. W. HOLSAPPLE Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) It turned out that the expert and expert system had some discrepancies in detecting the internal control weaknesses in the first case and the last (i.e. the fourth) case. The expert could identify only eight weaknesses in the first case and only nine weaknesses in the last case (instead of 10 as the expert system did). Where there were discrepancies, the expert reconsidered his responses and agreed that the expert system’s responses were indeed correct. Interestingly, this illustrates that an expert system can sometimes be useful even to an expert (e.g. to double check the expert’s reason- ing). After all refinements were made, the resulting system consisted of 256 rule sets and 952 additional files (i.e. form specifications, database tables, program fragments). 3.5. Evaluating the System’s Utility Beyond validity, it is important to assess an expert system’s utility with respect to its objective. Usefulness can be evaluated via empirical testing. To examine the impact of using an expert system to facilitate the transfer of the auditor’s internal control evaluation knowledge to management, an experiment was conducted to assess the effectiveness of managers using it. The experiment was conducted in a conference room and a laboratory room of a college, providing an isolated and controlled environment for the study. The research model used is illustrated in Figure 9. Two decision-aid treatments were used as values of the independent variable: expert system (ES) and no decision aid (NDA). Three response variables were measured as follows: 1. Effectiveness—accuracy of decision making was examined as a measure of the system’s effectiveness. Figure 7. A report of a weakness found for inadequate segregation of duties: having the same person perform both functions of preparing and recording sales invoices Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) MANAGERIAL EVALUATION OF INTERNAL CONTROLS 113 Figure 8. A report of a weakness found for inadequate segregation of duties: the functions of receiving cash / cheque and recording cash /cheque are performed by persons who are related 2. Participant perception of the task—post-experiment questionnaires were used to measure the participants’ perceptions about the task. 3. Participant satisfaction with performance—post-experiment questionnaires were used to measure the participants’ satisfaction with performance. The following hypotheses were tested to examine the utility of the expert system in facilitating transfer of an auditor’s internal control evaluation knowledge to management. Figure 9. Research model 114 C. CHANGCHIT AND C. W. HOLSAPPLE Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) H1. The improvement in accuracy scores of participants trained with the expert system (partici- pants in the ES group) is higher than the improvement in accuracy scores of participants in the NDA group. H2a. Participants in the ES group are more satisfied with their accuracy than participants in the NDA group. H2b. Participants in the ES group are more satisfied with their speed than participants in the NDA group. H3a. Participants in the ES group perceive that performing tasks requires less effort than partici- pants in the NDA group perceive. H3b. Participants in the ES group perceive that performing tasks is more interesting than partici- pants in the NDA group perceive. Sixty-five practitioners voluntarily participated in this experiment. All played managerial roles in their respective organizations. No participant had either specific background in internal control concepts or prior hands-on experience with an expert system. Table I shows the industries repre- sented by the participants. The experiment involved four sessions, as presented in Table II. Each session made use of one of four cases. Each case presented an eight-page, single-spaced narrative description of a scenario Table II. Experimental tasks in each session ES group NDA group Session II Participants detect internal control weaknesses in case D with the ES Participants detect internal control weaknesses in case D without the ES Session III Participants detect internal control weaknesses in case C with the ES Participants detect internal control weaknesses in case C without the ES Session I Participants detect internal control weaknesses in case B without the ES Participants detect internal control weaknesses in case B without the ES Session IV Participants detect internal control weaknesses in case A without the ES Participants detect internal control weaknesses in case A without the ES Table I. Participant demographics Industry No. of subjects Consultant 12 Legal 8 Manufacturing 8 Government 11 Insurance 3 Banking 2 Construction 2 Distribution 2 Mining 2 Transportation 2 Education 3 Finance 3 Others 7 Total 65 Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) MANAGERIAL EVALUATION OF INTERNAL CONTROLS 115 involving an organization with some internal control weaknesses. To prevent any bias in designing the case studies, the expert who participated in developing the prototype expert system was not allowed to participate in generating case studies. Three experienced auditors and three managers were asked to pilot test the case studies to validate their contents, as well as ensure that each case had the same level of difficulty. Ten MBA students performed pre-tests to validate the clarity of the cases. Prior to the experiment, the four case studies were randomly assigned to the sessions. Participants played the role of a manager trying to detect the potential weaknesses of the internal control system described in a case study. Ten internal control weaknesses were intentionally incor- porated into each case. The maximum time allowed for each session was 2 h. For each session, an accuracy score was calculated and the decision time was recorded. Questionnaires were distributed at the end of each session to measure participants’ satisfaction with their performances, as well as their perceptions about the task (see Appendix A). Accuracy of decision making is regarded as a measure of the system’s effectiveness. Two expe- rienced auditors and two accounting professors evaluated the list of internal control weaknesses previously described. Each evaluator assigned a score to each weakness using a scale of 0 to 10, based on the degree of importance of each weakness (0 being the least important, 10 being very important). The average of the scores for each weakness (i.e. total divided by four) was assigned as its importance. In arriving at an accuracy score, all points related to correctly identified weaknesses are added. In order to prevent the subjects from trying to detect weaknesses by guessing, they were informed at the beginning of the experiment that one-third of all points for inaccurately identified weaknesses would be subtracted as the penalty for guessing. A non-response for a potential weakness results in neither addition nor subtraction. In testing the hypotheses, we examined the deviations of participants’ performances between session I and session IV. In both sessions, participants had to perform tasks without the expert system support. The deviations examined are attributed to the learning effects that occurred while performing tasks in sessions I to III. There were two types of potential learning effect that we expected here. First, there is the learning effect that could occur from participants having used the expert system; this could happen only for the ES group. Second, there is the learning effect that could occur from participants having performed four cases consecutively; the degree of this type of learning effect could be expected to be comparable for the ES group and NDA group. To verify homogeneity of internal control evaluation knowledge between the ES and NDA groups, an analysis of variance (ANOVA) was performed on the first session results for the experimental group (ES group) versus the control group (NDA group). The ANOVA was performed to test for differences in the Accuracy Score between the groups. At an α level of 0.05, no significant differ- ence was found between the groups. 4. EXPERIMENTAL FINDINGS The experimental data were analyzed as a completely randomized design with a one-way treatment structure using the ANOVA technique. Treatments had a grouping structure and fixed effects. Each participant was an experimental unit. The F-test was used to test hypotheses about group means for each response variable: (i) accuracy score (H1), (ii) participants’ satisfaction with performance (H2a and H2b), and (iii) participants’ perceptions of the task (H3a and H3b). Hypothesis H1 was tested by comparing the deviations of performances of each participant in session IV versus session I 116 C. CHANGCHIT AND C. W. HOLSAPPLE Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) between the expert system group and the NDA group. Hypotheses H2a, H2b, H3a, and H3b were tested by comparing participants’ responses from the questionnaires given at the end of session IV. Regarding hypothesis H1, session I versus session IV results show that participants could detect internal control weaknesses significantly more accurately after being trained with the expert system ( µ I = 4.575 versus µ IV = 16.073) than before. Even though there is also an increase in the accuracy score for the NDA group ( µ I = 8.584 versus µ IV = 12.250), this increase was not nearly at the same significance level as in the experimental group. It is clear that the increase in accuracy in the ES group (having practiced with the expert system) is much greater than the increase in accuracy in the NDA group. The accuracy score in the experimental group is more than three times better after being trained with the expert system, whereas the improvement in the NDA group is less than 50%. A t-test conducted on the improvement score between two groups (11.498 for the ES group and 3.666 for the NDA group) also confirms that the improvements in performance of participants in the ES group were significantly higher than the improvements in performance of participants in the NDA group from an accuracy standpoint ( P = 0.0035). The results are presented in Table III. The major findings for testing hypotheses H2a, H2b, H3a, and H3b are summarized in Table IV. The participants’ answers to the post-experiment questionnaires reveal the attitudes of participants in both groups toward the task as follows: Table IV. Results of testing hypotheses H2a, H2b, H3a, and H3b Response Variable H0 Session IV P-value a ES group NDA group Mean of participants’ attitude on the difficulties of the task H2a 3.65 2.93 0.085 (1: very difficult; 7: very easy) Mean of participants’ attitude about their satisfaction with H2b 3.94 3.00 0.05 the accuracy in answering the case study (1: very unsatisfactory; 7: very satisfactory) Mean of participants’ attitude about their satisfaction with the H3a 4.04 2.87 0.025 speed in answering the case study (1: very unsatisfactory; 7: very satisfactory) Mean of participants’ attitude on how interesting was the task H3b 4.18 2.87 0.025 (1: very boring; 7: very interesting) a Represents significance of at least α = 0.05. Table III. Results of testing hypothesis H1 Mean of accuracy score Deviation between P-value between (range: −20 to 38) Sessions I and IV ES and NDA groupsc Session Ia Session IVb ES group (49 participants)d 4.575 16.073 11.498 0.0035 NDA group (15 participants) 8.584 12.250 3.666 a Participants perform case B without the expert system. b Participants perform case A without the expert system. c Represents significance of at least α = 0.01. d One participant had an emergency call and left while performing case II. His data were taken out. Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) MANAGERIAL EVALUATION OF INTERNAL CONTROLS 117 • Participants in the ES group were more satisfied with their perceived accuracy than participants in the NDA group ( µ ES = 3.94 versus µ NDA = 3.00, P = 0.05). • Participants in the ES group were more satisfied with their perceived speed than participants in the NDA group ( µ ES = 4.04 versus µ NDA = 2.87, P = 0.025). • There was no significant difference in participants’ attitude between the ES group and the NDA group on the difficulty of the task. That is, expert system exposure and training does not appear to affect perceptions of task difficulty. These findings might be attributed to the following: 1. Because participants in the ES group were allowed to use the expert system in sessions II and III, they may lose some confidence after the expert system was removed in session IV. 2. The increase in participants’ perceptions in the NDA group may be the result of learning that may have accrued from having the participants perform three consecutive cases. • Participants in the ES group perceived that the task was more interesting than participants in the NDA group perceived ( µ ES = 4.18 versus µ NDA = 2.87, P = 0.025). • Answers to open-ended questions also reveal satisfaction with the use of the expert system. Most of the participants were interested in the availability of the system and the idea that they could learn how to evaluate internal controls. A number of these participants even asked whether the system was commercially available. 5. CONCLUSIONS, LIMITATIONS, AND DIRECTION FOR FUTURE RESEARCH This research is an initial investigation of the use of an expert system to train practicing managers on making internal control evaluations. The major finding of this research is that it is feasible to transfer an auditor’s internal control evaluation knowledge to management via the use of an expert system. However, as with any computerized system, the expert system must be used with care. It is important to note that the objective of the system is not to replace the auditor’s work. An auditor still plays the traditional role. The main purpose of the system is to provide managers with a practical applied understanding of internal controls. Having this knowledge can help managers maintain an effective internal control system, thus providing more reliable accounting data and better safeguard- ing of assets. It could let organizations save time and money by allowing weaknesses in internal control systems to be detected and solved more quickly. The communication between management and auditors relating to the importance of internal controls might also be improved. Generalizability of the results in this study may be constrained in several respects. First, this research concentrates only on the evaluation of controls commonly found in the sales and collection cycle. Second, it investigates internal control systems commonly found in the merchandising indus- try. It is not expected to handle novel (uncommonly different) accounting systems. In addition, because this is just an initial study, there may still be some limitations concerning the design of system’s user interface (e.g. the inability to go back directly to change or correct an answer in the previous screen). These limitations point to directions in which the research presented here can be extended by future investigations. The future development of this system might use an Internet-compatible tool so that access to the system can be significantly increased. Researchers might try to incorporate additional transac- tion cycles, industries, or other auditing functions into the expert system. They could develop and study another expert system by acquiring the expertise from an internal auditor instead of the external auditor. Another direction is to acquire expertise from multiple auditors and then conduct experiments similar to the one reported here. Finally, researchers may investigate the feasibility of 118 C. CHANGCHIT AND C. W. HOLSAPPLE Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) integrating this kind of expert system with a company’s databases so that users are not required to input as much information. APPENDIX A: POST-EXPERIMENT QUESTIONNAIRES Please answer the following questions. Please keep in mind that the following questions refer ONLY to the last case study you have done. 1. On a scale of 1 (very unsatisfactory) to 7 (very satisfactory), how satisfied were you with your accuracy in answering the case study? 1 2 3 4 5 6 7 very unsatisfactory slightly neutral slightly satisfactory very unsatisfactory unsatisfactory satisfactory satisfactory 2. On a scale of 1 (very difficult) to 7 (very easy), how difficult was it to do the case study? 1 2 3 4 5 6 7 very difficult slightly neutral slightly easy very difficult difficult easy easy 3. On a scale of 1 (very unsatisfactory) to 7 (very satisfactory), how satisfied were you with your speed in answering the case study? 1 2 3 4 5 6 7 very unsatisfactory slightly neutral slightly satisfactory very unsatisfactory unsatisfactory satisfactory satisfactory 4. On a scale of 1 (very boring) to 7 (very interesting), how interesting was the task you performed? 1 2 3 4 5 6 7 very boring slightly neutral slightly interesting very boring boring interesting interesting REFERENCES Arens AA, Loebbecke JK. 1994. Auditing: An Integrated Approach, 6th edn. Prentice Hall: New Jersey. Back B. 1994. Validating an expert system for financial statement planning. Journal of Management Information Systems 10(3): 157–177. Bailey Jr AD, Duke GL, Gerlach J, Ko C, Meservy R, Whinston AB. 1985. TICOM and the analysis of internal controls. Accounting Review 60(2): 186 –211. Biggs SF, Morrison TA. 1990. What accountants need to know about expert systems. The CPA Journal 60: 98 –101. Biggs SF, Messier Jr WF, Hansen JV. 1987. A descriptive analysis of computer audit specialists’ decision- making behavior in advanced computer environments. Auditing: A Journal of Practice & Theory 6(1): 1–21. Bolger F, Wright G. 1994. Assessing the quality of expert judgement: issues and analysis. Decision Support Systems 11: 1–24. Bonczek RH, Holsapple CW, Whinston AB. 1981. Foundations of Decision Support Systems. Academic Press: New York. Bonner SE, Pennington N. 1991. Cognitive processes and knowledge as determinants of auditor expertise. Journal of Accounting Literature 10: 1–50. Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) MANAGERIAL EVALUATION OF INTERNAL CONTROLS 119 Bonner SE, Walker PL. 1994. The effects of instruction and experience on the acquisition of auditing know- ledge. The Accounting Review 69(1): 157–178. Boritz JE. 1985. The effect of information presentation structures on audit planning and review judgments. Contemporary Accounting Research 8(1): 193–218. Borthick AF, West OD. 1987. Expert systems—a new tool for the professional. Accounting Horizons 1(1): 9–16. Brown CE, Phillips ME. 1990. Expert system for management accountants. Management Accounting 7: 18 –20. Brown DL, Jones DR. 1998. Factors that influence reliance on decision aids: a model and an experiment. Journal of Information Systems 12(2): 75 – 94. COSO. 1992. Internal Control-Integrated Framework, Committee of Sponsoring Organizations of Treadway Commission. AICPA: New York. Cummings BK, Apostolou NG. 1987. Expert systems in auditing: an emerging technology. Internal Auditing 3(2): 3–10. Cummings BK, Lauer J, Baker R. 1988. Expert systems in internal auditing. Internal Auditing 4(1): 49 – 64. Curtis MB, Viator RE. 2000. An investigation of multidimensional knowledge structure and computer auditor performance. Auditing: A Journal of Practice and Theory 19(2): 83–103. Davis R, Lenat DB. 1982. Knowledge-based Systems in Artificial Intelligence. McGraw-Hill: California. Ege G, Sullivan WG. 1990. Expert system for management accountants. Management Accounting 7: 18–20. Eining MM, Dorr PB. 1991. The impact of expert system usage on experiential learning in an auditing setting. Journal of Information Systems 5(1): 1–16. Fedorowicz J, Oz E, Berger PD. 1992. A learning curve analysis of expert system use. Decision Sciences 23(3): 797–818. Frederick DM. 1991. Auditor representation and retrieval of internal control knowledge. The Accounting Review 66(2): 240–258. Gadh VM, Krishnan R, Peters JM. 1993. Modeling internal control and their evaluation. Auditing: A Journal of Practice & Theory 12(Supplement): 113–129. Gal GF. 1985. Using auditor knowledge to formulate data model constraints: an expert system for internal control evaluation. PhD dissertation, Michigan State University. Gal GF, Steinbart PJ. 1992. Interface style and training task difficulty as determinants of effective computer- assisted knowledge transfer. Decision Sciences 23: 128–143. Graham LE, Damens J, Ness GV. 1991. Developing Risk AdvisorTM: an expert system for risk identification. Auditing: A Journal of Practice & Theory 10(1): 69 – 96. Hayes-Roth F, Waterman D, Lenat D. 1983. Building Expert Systems. Addison-Wesley: Reading, MA. Holsapple CW, Raj VS. 1994. An exploratory study of two KA methods. Expert Systems 11(2): 77–88. Holsapple CW, Whinston AB. 1987. Business Expert Systems. Irwin: Homewood, IL. Libby R. 1995. The role of knowledge and memory in audit judgment. In Judgment and Decision-Making Research in Accounting and Auditing, Ashton RH, Ashton AH (eds). Cambridge University Press: New York. Lieb EB, Gillease JK. 1996. Du Pont uses a decision support system to select its audit portfolio. Interfaces 26(3): 10 –21. Marchant GA. 1989. Analogical reasoning and hypothesis generation in auditing. Accounting Review 64(3): 500 –513. Meservy RD. 1985. Auditing internal control: a computational model of the review process. PhD dissertation, University of Minnesota. Meservy RD, Bailey Jr AD, Johnson PE. 1986. Internal control evaluation: a computational model of the review process. Auditing: A Journal of Practice & Theory 6(3): 44 –74. Messier Jr WF. 1995. Research in and development of audit decision aids. In Judgement and Decision-Making Research in Accounting and Auditing, Ashton RH, Ashton AH (eds). Cambridge University Press: New York. Micro Data Base Systems. 1991. Getting Started with GURU. Micro Data Base Systems, Inc.: Lafayette, IN. Moody JW, Blanton JE, Cheney PH. 1998. A theoretically grounded approach to assist memory recall during information requirements determination. Journal of Management Information Systems 15(1): 79 – 98. Murphy DS. 1990. Expert system use and the development of expertise in auditing: a preliminary investigation. Journal of Information Systems 4(2): 18 –35. Nelson MW, Libby R, Bonner SE. 1995. Knowledge structure and the estimation of conditional probabilities in audit planning. Accounting Review 70(1): 27–47. 120 C. CHANGCHIT AND C. W. HOLSAPPLE Copyright © 2004 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt. 12, 103 –120 (2004) Odem MD, Dorr PB. 1995. The impact of elaboration-based expert system interfaces on de-skilling: an epis- temological issue. Journal of Information Systems 9(1): 1–17. O’Leary DE. 1988. Methods of validating expert systems. Interfaces 18(6): 72–79. O’Leary DE, Watkins PR. 1989. Review of expert systems in auditing. Expert Systems Review for Business & Accounting 9(2): 3–22. Oz E. 1989. A study of improvement in decision-making skills through the use of expert systems. Unpublished PhD dissertation, Boston University. Pei BKW, Reneau JH. 1992. The effect of memory structure on using rule-based expert systems for training: a framework and an empirical test. Decision Sciences 21: 263–286. Pei BKW, Steinbart PJ, Reneau JH. 1994. The effect of judgment strategy and prompting on using rule-based expert systems for knowledge transfer. Journal of Information Systems 8(1): 21– 42. Rezaee Z. 2004. Causes, consequences, and deterrence of financial statement fraud. Critical Perspectives on Accounting in press. Steinbart PJ, Accola WL. 1994. The effect of explanation type and user involvement on learning from and satisfaction with expert systems. Journal of Information Systems 8(1): 1–17. Viator RE, Curtis MB. 1998. Computer auditor reliance on automated and non-automated controls as a function of training and experience. Journal of Information Systems 12(1): 19–30. Vinze AS, Karan V, Murthy US. 1991. A generalizable knowledge-based framework for audit planning expert systems. Journal of Information Systems 5(2): 78–91. Weber R. 1999. Information Systems Control and Audit. Prentice-Hall: New Jersey. Wong BK, Monaco JA. 1995. Expert system applications in business: a review and analysis of the literature (1977–1993). Information & Management 29: 141–152.