No Job Name Research Submission An Expert System for Headache Diagnosis: The Computerized Headache Assessment Tool (CHAT) Morris Maizels, MD; William J. Wolfe, PhD Background.—Migraine is a highly prevalent chronic disorder associated with significant morbidity. Chronic daily headache syndromes, while less common, are less likely to be recognized, and impair quality of life to an even greater extent than episodic migraine. A variety of screening and diagnostic tools for migraine have been proposed and studied. Few investigators have developed and evaluated computerized programs to diagnose headache. Objectives.—To develop and determine the accuracy and utility of a computerized headache assessment tool (CHAT). CHAT was designed to identify all of the major primary headache disorders, distinguish daily from episodic types, and recognize medication overuse. Methods.—CHAT was developed using an expert systems approach to headache diagnosis, with initial branch points de- termined by headache frequency and duration. Appropriate clinical criteria are presented relevant to brief and longer-lasting headaches. CHAT was posted on a web site using Microsoft active server pages and a SQL-server database server. A convenience sample of patients who presented to the adult urgent care department with headache, and patients in a family practice waiting room, were solicited to participate. Those who completed the on-line questionnaire were contacted for a diagnostic interview. Results.—One hundred thirty-five patients completed CHAT and 117 completed a diagnostic interview. CHAT correctly identified 35/35 (100%) patients with episodic migraine and 42/49 (85.7%) of patients with transformed migraine. CHAT also correctly identified 11/11 patients with chronic tension-type headache, 2/2 with episodic tension-type headache, and 1/1 with episodic cluster headache. Medication overuse was correctly recognized in 43/52 (82.7%). The most common misdiagnoses by CHAT were seen in patients with transformed migraine or new daily persistent headache. Fifty patients were referred to their primary care physician and 62 to the headache clinic. Of 29 patients referred to the PCP with a confirmed diagnosis of migraine, 25 made a follow-up appointment, the PCP diagnosed migraine in 19, and initiated migraine-specific therapy or prophylaxis in 17. Conclusion.—The described expert system displays high diagnostic accuracy for migraine and other primary headache disorders, including daily headache syndromes and medication overuse. As part of a disease management program, CHAT led to patients receiving appropriate diagnoses and therapy. Limitations of the system include patient willingness to utilize the program, introducing such a process into the culture of medical care, and the difficult distinction of transformed migraine. Key words: headache diagnosis, expert systems, disease management (Headache 2008;48:72-78) Annually, over 10% of the population suffers at least one migraine headache,1 with a significant im- pact on the individual’s quality of life,2 as well as a major economic and societal burden.3 Up to 5% of the adult population may suffer from daily headache syndromes, with medication overuse identified in one- third of these patients.4 The impact of chronic daily headache on the individual and society is even greater than that of episodic migraine. Despite significant advances in acute and pre- ventive therapy, migraine remains underdiagnosed, and “drug rebound headache” an “unrecognized epidemic.”5 A population-based survey found that From the Kaiser Permanente – Family Medicine, Woodland Hills, CA (Dr. Maizels); and California State University Chan- nel Islands – Computer Science, Camarillo, CA (Dr. Wolfe). Address all correspondence to Dr. Morris Maizels, Kaiser Per- manente – Family Medicine, 5601 De Soto Avenue, Woodland Hills, CA 91365-4084. Accepted for publication June 27, 2007. Conflict of Interest: None ISSN 0017-8748 doi: 10.1111/j.1526-4610.2007.00918.x Published by Blackwell Publishing Headache © 2007 the Authors Journal compilation © 2008 American Headache Society 72 only 48% of participants who fulfilled International Headache Society (IHS) criteria for migraine had been diagnosed as having migraine.6 Only 50% of patients who fulfill criteria for migraine and seek medical care are correctly diagnosed.7 The burden of undiagnosed migraine is significant: 24% of patients with undiag- nosed migraine missed at least 1 day of work or school in the previous 3 months, and 45% reported at least a 50% reduction in productivity.6 Several authors have also proposed simplified cri- teria for diagnosis of migraine. These typically include some subset of IHS criteria.8-10 A variety of question- naires to diagnose migraine have been developed, with reported sensitivities of 76–84%, and specificities of 92-99%.11,12 None of these instruments has come to be used commonly in clinical practice. There have been few reports of utilizing computer technology to diagnose migraine and other primary headache disorders. Drummond and Lance utilized a computer algorithm to determine to what extent clusters of symptoms differentiated diagnoses along the “migraine-tension headache spectrum.”13 Andrew et al.14 and Gobel et al.15 incorporated the 1988 IHS classification system16 into a computer program. Go- bel’s group later utilized the program to standardize inclusion of patients into a sumatriptan trial.17 Diagnostic headache diaries18,19 and structured medical records20 have been incorporated into com- puter programs to aid in headache diagnosis. Several authors have described programs but not reported val- idating the instruments with diagnostic interviews.21-23 Most of the above programs are essentially checklists of symptoms which the computer program tries to fit into a specific diagnosis. The author (Morris Maizels) sought to develop a headache diagnostic program which would employ simple branching decisions to mimic the logic of a clin- ician. The Computerized Headache Assessment Tool (CHAT) was developed with the following features in mind (summarized in Table 1): completed by patients on-line; questions are systematically presented de- pending on prior answers (patients are only presented with relevant questions); screens for all common pri- mary headache disorders; distinguishes chronic from episodic subtypes; recognizes medication overuse; and screens for potentially worrisome headaches. It was hoped that the output of the program would be in a format suitable for a physician to review with the pa- tient, and to include in the medical record. This article describes the development of the program, the diag- nostic accuracy for primary headache syndromes, and the outcomes of screening. METHODS Description of the Computerized Headache Assess- ment Tool (CHAT).—CHAT consists of 4 sections: an explanatory and disclaimer page, the interview section, feedback of answers, and diagnostic output. The disclaimer page requires patients to confirm their understanding that the tool is intended for use in conjunction with a health care professional. The interview section begins with 2 screens which all subjects see: the first asks the frequency of moderate-to-severe headaches, and the second the duration of these headaches. Based on these answers, the questioning branches. Patients with headaches last- ing 3 hours or less next answer questions relevant to brief headache syndromes (number of attacks/day, location, quality, autonomic features, triggering). Pa- tients with severe headaches 4 hours or longer see Table 1.—Desired Characteristics of a Computerized Headache Diagnostic System Diagnostic Features • Recognize the major primary headache disorders, including migraine, tension-type headache, cluster headache, and brief headache syndromes • Distinguish chronic from episodic subtypes of migraine, tension-type headache, and cluster headache • Recognize medication overuse • Screen for “worrisome” features User-friendly • Patient self-administered • Rapid completion Human Interviewer Characteristics • Branching logic • Subsequent questions selected based on prior answers (patient only sees relevant questions) • Feedback answers • Recognize and correct inconsistencies Clinically Useful Output • Criteria for generated diagnoses are explained in a format understandable for patients • Hardcopy of output is suitable for physician to review with patient, and to include into a medical record Headache 73 a screen, which asks questions based on IHS crite- ria for migraine and tension-type headache (severity, laterality, throbbing vs dull, exacerbation with phys- ical activity, associated nausea or vomiting, light or noise sensitivity). This screen also determines the fre- quency of milder headaches and medication usage. Pa- tients with headaches lasting longer than 4 hours are also asked questions about their previous headaches: whether they were daily from onset or became daily suddenly or gradually; whether there were any sig- nificant events associated with headache onset or “transformation”; whether there has been a signifi- cant change in headache pattern; and how long the current headache pattern has persisted. For patients whose current headaches did not fulfill migraine cri- teria on the previous screen, they see an additional screen to determine whether previous headaches met migraine criteria. Feedback. A screen summarizes the patients an- swers, and the patient has an opportunity to correct any answers. Diagnostic Output.—The screen shows an “assess- ment” (the word “diagnosis” is not used, to reduce the subject’s reliance on the CHAT assessment with- out clinician review). Assessments were based on the 1988 IHS classification system,16 and the Silberstein– Lipton revised criteria for transformed migraine.24 Po- tential assessments for headaches lasting 4 hours or longer include migraine (episodic or transformed), mi- grainous, episodic or chronic tension-type headache, and new daily persistent headache. The assessment in- cludes “. . . with medication overuse” for subjects who indicated symptomatic medication use 3–4 days/week or more. For brief headache syndromes, diagnoses in- clude cluster (episodic or chronic) and atypical cluster headache, idiopathic stabbing headache, and trigemi- nal neuralgia. Importantly, headaches that do not ful- fill criteria of the above conditions generate an as- sessment of “cannot assess.” All diagnoses are fol- lowed by a brief explanation of how the diagnosis was made. Patients with daily or near-daily headaches see a caution in bold print that frequent or daily headaches require evaluation to exclude worrisome causes. Errors and Inconsistencies.—During initial testing, it was noted that patients might indicate that headache duration was 3 hours or less when in fact it was over 4 hours. To confirm that this answer is correct, pa- tients with episodic headache who indicate duration as 3 hours or less see a screen that says “Do these headaches ever last more than 4 hours?” Further, for patients who are initially routed to the brief headaches screen, if they do not fulfill criteria for any of the brief headache syndromes, they are routed through the questions for patients with headaches lasting 4 hours or longer. Study Setting.—The patient population included members of a suburban health maintenance organi- zation. Patients completed the internet questionnaire at home or work. Patient Selection.—A poster in the triage section of the adult urgent care (AUC) department described the study, and nurses were asked to identify headache patients and hand them a flyer. However, because of poor compliance with this method, we later mailed out a study flyer to patients who had presented to AUC with headache. A flyer describing the study was also posted in the family medicine department at the receptionist’s desk. There were no inclusion or exclusion criteria for participation, other than age over 18. Diagnostic Confirmation.—Study participants who granted permission were contacted by telephone. A headache clinic nurse experienced in headache diagnosis performed diagnostic interviews. The in- terview began with the San Diego migraine ques- tionnaire, a well-validated instrument for migraine diagnosis.12 Patients with frequent headache were further classified by the modified Silberstein–Lipton criteria.24 Patients with brief headache syndromes were classified by IHS criteria. All interviews with diagnostic uncertainty were reviewed by the author (Morris Maizels). Patient Follow-Up.—Patients were offered follow- up appointments to their primary care physician if headaches were diagnosed as episodic migraine or tension-type headache, and to the headache clinic if headaches were diagnosed as chronic migraine or an- other daily headache syndrome. Chart Review.—Medical records of patients who completed their diagnostic interview were reviewed within 6 months to determine if they had kept their 74 January 2008 follow-up appointment, and whether any new treat- ments were initiated as a result. Data Analysis.—Simple descriptive analyses were performed on the data. This study was approved by the Institutional Re- view Board. Patient consent was obtained through the introductory screen online. RESULTS One hundred thirty-five subjects completed the online survey, 103 from AUC and 32 from primary care. Diagnostic interviews were completed on 117 subjects. Physician diagnoses were available for 98 pa- tients from the AUC sample but not for the primary care waiting room sample. The confirmed diagnoses, CHAT assessments, and AUC physician diagnoses are shown in Table 2. CHAT correctly identified 35/35 (100%) patients with episodic migraine and 42/49 (6%) of patients with transformed migraine. Urgent care physicians correctly diagnosed 26/30 (86.7%) of patients with episodic migraine, and recognized 27/40 (66.7%) of patients with transformed migraine as migraine, al- though none as a daily headache disorder. CHAT also correctly identified 12/12 patients with chronic tension-type headache, 2/2 with episodic tension-type headache, and 1/1 with episodic cluster headache. Al- though NDPH was correctly identified in only 3/7 (42.9%) patients, CHAT recognized 6/7 as a daily headache syndrome. CHAT recognized medication overuse in 43/52 (82.7%) patients who met criteria for the diagnosis on diagnostic interview. Three additional assessments of medication overuse by CHAT were not confirmed. Overall, CHAT recognized 85/90 (94.4%) cases of “any migraine,” (ie, migraine, migrainous, or TM) and 63/68 (92.6%) as a daily headache syn- drome (TM, CTTH, or NDPH). The diagnostic ac- curacy for all headache diagnoses, including unclas- sifiable (but not including medication overuse) was 104/117 (88.9%). Patient Follow-Up.—Fifty patients were referred to their primary care physician, and 42 made at least one visit for headache follow-up. Of 29 patients referred with a confirmed diagnosis of migraine, 25 made a follow-up appointment, the PCP diagnosed mi- graine in 19, and initiated migraine-specific therapy or prophylaxis in 17. Sixty-two patients were referred for evaluation to the headache clinic, and 51 made at least one visit. Table 2.—Confirmed Diagnoses, Assessments Generated by CHAT, and Physician Diagnoses From Adult Urgent Care Visits. Percentages (in Parentheses) of Correct Assessment/Diagnosis Confirmed Diagnosis n = 117 CHAT Assessment n = 117 Physician Diagnosis n = 98 Episodic migraine 35 Episodic migraine 35 (100%) “Migraine” 26/30 (86.7%) Migrainous* 6 Migrainous 5 (83.3%) Probable TM*** 1 TH 1 “vascular headache” 1 no specific diagnosis 1 TM** 49 TM 42/49 (85.7%) “Migraine” 27/40 (66.7%) Episodic migraine 2 CTTH 3 atypical cluster 1 “can’t assess” 1 NDPH 7 NDPH 3 (42.9%) TH 2 no specific diagnosis 3 TM 1 CTTH 2 “can’t assess” 1 CTTH 12 CTTH 12/12 (100%) migraine 2 “TTH” 1 no specific diagnosis 9 ETTH 2 ETTH 2/2 (100%) no specific diagnosis 2 Cluster headache Cluster headache • episodic 1 • episodic 1/1 (100%) • no specific diagnosis 1 • atypical 1 • atypical 1/1 (100%) • “rhinitis vs. cluster” 1 Unclassifiable 4 “can’t assess” 3 (75%) no specific diagnosis 4 NDPH 1 Medication overuse 52 Medication overuse 43 (82.7%) (none identified) * “Migrainous” now labeled “probable migraine” by ICHD-II classification. ** “Transformed migraine,” as designated by Silberstein- Lipton criteria, not recognized in ICHD-II classification. *** Probable transformed migraine, ie, patient could not recall whether headaches changed gradually. TM = transformed migraine; NDPH = new daily persistent headache; CTTH = chronic tension-type headache; ETTH = episodic tension-type headache; TH = tension headache. Headache 75 DISCUSSION An expert systems approach to headache diag- nosis, as incorporated in CHAT, achieves diagnostic precision for primary headache diagnoses. Like all of clinical medicine, accurate diagnosis depends on the accuracy of patient responses, and a skilled human in- terviewer is likely to elicit more accurate responses than a fixed computer program. In our patient sample, CHAT recognized 100% of patients correctly as episodic migraine. There were 13 incorrect diagnoses for the entire cohort, of which 11 were in patients classified by interview as TM or NDPH. The diagnosis of TM and NDPH both rely on the patient’s accurate recall of whether headaches began abruptly and/or became daily gradually or abruptly.The accuracy of this recall is difficult to deter- mine. The designation of TM has been replaced in the current ICHD-II by chronic migraine and episodic mi- graine + CTTH. However, the difficulties of this diag- nostic category led to an appendix revision of chronic migraine (appendix 1.5.1 chronic migraine), which al- lows for CM to be diagnosed with �15 headache days/month, of which 8 days are migraine-like.25 From a clinical point-of-view, especially in a primary care set- ting, the distinctions of migraine from non-migraine, and frequent/daily from episodic are important. The nuances of CM and NDPH would not change clinical practice. The ability of the program to recognize certain headaches as unclassifiable, ie, “can’t assess,” is an important feature of this program. This assessment is most often generated when patients entered incorrect data for headache duration (ie, patients with migraine may indicate headache duration less than 3 hours be- cause of relief with medication). The recognition of medication overuse is an im- portant feature of headache evaluation, and one com- monly overlooked in primary care settings. CHAT cor- rectly recognized medication overuse in 43/52 (82.7%) patients, with 3 false positive diagnoses. The false pos- itives occurred in part because some patients confused preventive with acute/abortive medication. Physicians had a high rate of correct diagnosis of migraine, although the study did not determine what percent of the AUC sample identified themselves as having migraine.The label of “transformed” or chronic migraine is not one familiar to primary care physicians. A surrogate marker to indicate whether physicians rec- ognize headache frequency and its significance might be the use of prophylaxis. This measure was beyond the scope of the study. CHAT is a unique system in that it combines simple human-like branching logic to determine the most appropriate diagnostic questions to ask. Re- cently, Sarchielli et al.19 reported on a software pro- gram which could generate ICHD-II diagnoses of all migraine subtypes, tension-type headache, cluster headache, and other trigeminal autonomic cephalgias. However, the program relies on data entered from pa- tients’ headache diaries, and is suggested as useful in tertiary headache centers rather than for primary care diagnosis. Study Limitations and Limitations of CHAT.— Headaches were most difficult to classify, both by human interviewer and by CHAT, in patients with headache of recent onset (often associated with viral- like syndromes). The tested edition of CHAT does not recognize many of the brief headache syndromes, specifically paroxysmal hemicranias, SUNCT, trigeminal neural- gia, and hypnic headache. Further, CHAT cannot generate more than a sin- gle headache diagnosis. This feature would require revision to be consistent with ICHD-II which distin- guishes chronic migraine from episodic migraine + CTTH. The appropriate clinical application of CHAT would have the physician confirm the accuracy of patient replies. This may not occur, leading to the possibility of treatment based on unvalidated patient replies. However, this problem exists for any screening instrument used in medicine. Any automated screening program must take pains to recognize potentially serious causes for headache, or caution both patient and physician users about that possibility. A later version of CHAT added the question, “Has this headache pattern been sta- ble for the past six months?” This question has not been validated as an adequate screen for ominous headaches. Future Development.—The criterion-based portion of this program can be readily modified to the cur- 76 January 2008 rent ICHD-II and future iterations of headache diag- nosis. The branching logic at present appears appro- priate although further study may lead to future modi- fication. The main modification would involve deletion of “transformed migraine,” replacing it with chronic mi- graine, episodic migraine + CTTH, or CTTH alone. “Migrainous” would be replaced with “probable mi- graine,” a change not requiring any change in the logic of the program. The less common primary care headache syndromes (paroxysmal hemicranias, SUNCT, hypnic headache, etc) can readily be included, although their rarity will make validation in a general population difficult. CONCLUSION An internet-based headache assessment tool has demonstrated a high degree of accuracy in recogniz- ing primaryheadache disorders, distinguishing chronic daily from episodic headache, and recognizing medi- cation overuse. No other computer-assisted headache diagnostic program described in the medical literature has a similar scope of diagnosis, or demonstrated ac- curacy. The major challenge of computer-assisted headache diagnosis at present is not in developing better programs, but in facilitating patients to use such programs, and encouraging their use in primary care settings. Further development might integrate online headache assessment with education about headache treatment, identification of headache triggers, and innovative online behavioral modification programs. REFERENCES 1. Stewart WF, Lipton RB, Celentano DD, Reed ML. Prevalence of migraine headache in the United States: Relation to age, income, race, and other so- ciodemographic factors. JAMA. 1992;267:64-69. 2. Dahlof CG, Solomon GD. The burden of migraine to the individual sufferer: A review. Eur J Neurol. 1998;5:525-533. 3. Hu HX, Markson LE, Lipton RB, Stewart WF, Berger ML. Burden of migraine in the United States: Disability and economic costs. Arch Int Med. 1999;159:813-818. 4. Castillo J, Munoz P, Guitera V, Pascual J. Epidemi- ology of chronic daily headache in the general popu- lation. Headache. 1999;39:190-196. 5. Edmeads J. Analgesic-induced headaches: An unrec- ognized epidemic. Headache. 1990;30:614-615. 6. Lipton RB, Diamond S, Reed M, Diamond ML, Stew- art WF. Migraine diagnosis and treatment: Results from the American Migraine Study II. Headache. 2001;41:638-645. 7. Stang PE, Osterhaus JT, Celentano DD. Migraine: Patterns of healthcare use. Neurology. 1994;44(Suppl 4):S47-S55. 8. Gervil M, Ulrich V, Olesen J, Russell MB. Screen- ing for migraine in the general population: Validation of a simple questionnaire. Cephalalgia. 1998;18:342- 348. 9. Solomon S. Criteria for the diagnosis of migraine in clinical practice. Headache. 1991;31:384-387. 10. Lipton RB, Dodick D, Sadovsky R, et al. A self- administered screener for migraine in primary care. The IDMigraine TM validation study. Neurology. 2003;61:375-382. 11. Rasmussen BK, Jensen R, Olesen J. Questionnaire versus clinical interview in the diagnosis of headache. Headache. 1991;31:290-295. 12. Tom T, Brody M, Valabhji A, Turner L, Molgaard C, Rothrock J. Validation of a new instrument for de- termining migraine prevalence: The UCSD migraine questionnaire. Neurology. 1994;44:925-928. 13. Drummond PD, Lance JW. Clinical diagnosis and computer analysis of headache symptoms. J Neurol Neurosurg Psychiatry. 1984;47:128-133. 14. Andrew ME, Penzien DB, Rains JC, Knowlton GE, McNaulty RD. Development of a computer applica- tion for headache diagnosis: The headache diagnostic system. Int J Biomed Comput. 1992;31:17-24. 15. Gobel H, Soyka D. Acceptance, reliability and va- lidity of computerized headache analysis on the ba- sis of the IHS headache classification. Cephalalgia. 1993(Suppl 13):121. 16. Headache Classification Committee of the Interna- tional Headache Society. Classification and diagnos- tic criteria for headache disorders, cranial neural- gias and facial pain. Cephalalgia. 1988;8(Suppl 7): 1-96. 17. Gobel H, Heinze A, Kuhn K, Heuss D, Lindner V. Effect of operationalized computer diagnosis on the therapeutic results of sumatriptan in general practice. Cephalalgia. 1998;18:481-486. Headache 77 18. Nielsen KD, Rasmussen C, Russell MB. The diagnos- tic headache diary – a headache expert system. Stud Health Technol Inform. 2000;78:149-160. 19. Sarchielli P, Pedini M, Coppola F, et al. Applica- tion of the ICHD-II criteria to the diagnosis of pri- mary chronic headaches via a computerized struc- tured record. Headache. 2007;47:38-44. 20. Gallai V, Sarchielli P, Alberti A, et al. Collabora- tive group for the application of the IHS criteria of the Italian society for the study of headache. Appli- cation of the 1988 International Headache Society diagnostic criteria in nine Italian headache centers using a computerized structured record. Headache. 2002;42:1016-1024. 21. Moses AJ, Lieberman M, Kittay I, Learreta JA. Computer-aided diagnoses of chronic head pain: Ex- planation, study data, implications, and challenges. Cranio. 2006;24:60-66. 22. Mainardi F, Maggioni F, Dainese F, Zanchin G. De- velopment of an ICHD-II based computer system for the general practitioner. J Headache Pain. 2005;6:211- 212. 23. De Simone R, Marano E, Bonavita V. Towards the computerisation of ANIRCEF Headache Cen- tres. Presentation of AIDA CEFALEE, a com- puter assisted diagnosis database for the manage- ment of headache patients. Neurol Sci. 2004;25:S218- S222. 24. Silberstein SD, Lipton RB, Solomon S, Mathew N. Classification of daily and near-daily headaches: Proposed revisions to the IHS criteria. Headache. 1994;34:1-7. 25. Olesen J, Bousser M-G, Diener H-C, et al. New appendix criteria open for a broader concept of chronic migraine. Headache Classification Commit- tee. Cephalalgia. 2006;26:742-746. 78 January 2008