key: cord-0291199-o0pbaalb
authors: Hersh, W. R.; Cohen, A. M.; Nguyen, M. M.; Bensching, K. L.; Deloughery, T. G.
title: Clinical Study Applying Machine Learning to Detect a Rare Disease: Results and Lessons Learned
date: 2021-12-07
journal: nan
DOI: 10.1101/2021.12.07.21267403
sha: 68b2a143038f053adaae4957b314dc4bdf465a60
doc_id: 291199
cord_uid: o0pbaalb

Machine learning has the potential to improve identification of patients for appropriate diagnostic testing and treatment, including those who have rare diseases for which effective treatments are available, such as acute hepatic porphyria (AHP). We trained a machine learning model on 205,571 complete electronic health records from a single medical center based on 30 known cases to identify 22 patients with classic symptoms of AHP that had neither been diagnosed nor tested for AHP. We offered urine porphobilinogen testing to these patients via their clinicians. Of the 7 who agreed to testing, none were positive for AHP. We explore the reasons for this and provide lessons learned for further work evaluating machine learning to detect AHP and other rare diseases.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted December 7, 2021. ; https://doi.org/10.1101/2021.12.07.21267403 doi: medRxiv preprint Machine learning has great potential to improve healthcare in many ways, from automated identification of patients eligible for testing and interventions to aiding in diagnosis and risk stratification. (1, 2) While predictive models to improve diagnosis and treatment have been developed and validated in many areas of medicine, relatively few models have been studied with real patients to determine their efficacy in real-world settings. (3) One application of machine learning is its use to identify patients who may have rare diseases. There are an estimated 1200 rare diseases that occur in humans (rarediseases.org). These diseases often go undiagnosed for many years because they may resemble common diseases and their infrequent occurrence may result in physicians not considering them in diagnostic workups.(4) They may also have genetic components that may require specialized testing. Processing of data in the electronic health record (EHR) may help determine the presence of such diseases.(5) One rare disease for which effective treatment is available that may be amenable to EHR-based detection is acute hepatic porphyria (AHP). AHP is a subset of porphyria that refers to a family of rare, metabolic diseases characterized by potentially life-threatening acute attacks and, for some patients, chronic debilitating symptoms that negatively impact daily functioning and quality of life.(6) During attacks, patients typically present with multiple signs and symptoms due to dysfunction across the autonomic, central, and peripheral nervous systems and manifesting in abdominal pain, central nervous system abnormalities, and peripheral neuropathy. The prevalence of diagnosed symptomatic AHP patients is about 1 per 100,000.(7) Due to the nonspecific symptoms and the rare nature of the disease, AHP is often initially overlooked or misdiagnosed. One study demonstrated that diagnosis of AHP is delayed on average by up to 15 years. (8) The predominant cause of AHP is a genetic mutation leading to a partial deficiency in the activity of one of the eight enzymes responsible for heme synthesis. AHP has low penetrance (~1%) and families carrying the gene may have few or only one affected member. As such, family history can be a poor diagnostic tool for this disease. The typical diagnostic approach for AHP is relatively inexpensive (typically around US$70) biochemical testing of random/spot urine for porphobilinogen.(9) Treatment of AHP has historically focused on avoidance of attack triggers, management of pain and other chronic symptoms, and treatment of acute attacks through the use of intravenous hemin. Recently, a new small interfering RNA (siRNA) molecule, givosiran, has been found to reduce the occurrence of acute attacks and impact other manifestations of the disease. (10) We previously applied machine learning to an extract of 205,571 EHR records (200,000 selected from those receiving primary care at OHSU enriched with 5,571 patients with presence of the word "porph" anywhere in their record) from Oregon Health & Science University (OHSU) to determine whether this approach could be effective in identifying patients not previously tested for AHP, and who might be candidates to receive a diagnostic workup for AHP. (11) The model was trained with 30 known patients in the OHSU system and identified patients whose records indicated AHP might be present yet had never been considered as a diagnosis. Manual review of the top-ranking 100 patients' records identified four patients where AHP diagnostic testing was . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted December 7, 2021. ;  likely indicated and 18 patients where AHP diagnostic testing was possibly indicated. Based solely on the reported prevalence of AHP, the analysis of manually reviewed patients would have expected to find only 0.001 cases, demonstrating the ability of the methodology to identify possible cases of this rare disease.

This study aimed to validate the output of the machine learning algorithm by conducting a clinical follow-up study with the patients it identified.

We obtained IRB approval to contact the 22 patients to offer free urine porphobilinogen testing (OHSU IRB00020617). Respecting the patient-clinician relationship, we structured the research protocol such that we would first obtain consent and permission to contact the patient from their primary care provider (PCP; physician, nurse practitioner, or physician assistant) or, when that the PCP could not be identified, the provider having the most clinical interaction with the patient in the OHSU record system. If approval was obtained by the provider, we would then contact the patient to obtain their consent and plan for participation. As this study was done during the COVID-19 pandemic, most interaction was done by telephone or email.

Of the 22 patients, six patients were either deceased or had moved from the Portland, Oregon area and had no follow-up address in the OHSU EHR, and we were unable to identify or contact their PCPs or other providers. Four other patients had moved their care to other local institutions and we were able to identify their PCPs. Eight patients had PCPs at OHSU, while four had non-PCP providers at OHSU significantly involved in their care.

Of the 16 patients for whom a clinician could be contacted, 13 agreed to allow us to contact their patient. (The rest did not believe testing would be clinically appropriate and declined to allow us to contract their patients.) We attempted to contact all 13 patients by phone and/or email. Four of these patients never replied to our attempts to contact them, with nine agreeing to sign the consent and come for testing. However, one patient never returned the consent form and another patient changed their mind about participation after signing the consent.

This resulted in seven patients being tested. All of these patients had virtually undetectable urine porphobilinogen, i.e., a normal test ruling out AHP. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted December 7, 2021. ; https://doi.org/10.1101/2021.12.07.21267403 doi: medRxiv preprint

Clinical validation of machine learning is a critical step to its application in patient care. Being able to detect patients with rare diseases, whose diagnosis is often missed or delayed, could be an important use case, especially when effective treatments exist. We developed what seemed to be a promising approach to detecting AHP. All of patients identified by our algorithm had the classic neurovisceral symptoms of AHP, had no other diagnosis to explain those symptoms, and had never been tested for the disease.

The results of this study did not identify any patients with AHP confirmed by diagnostic testing. However, our algorithm did identify at least seven patients who had unexplained symptoms and that were appropriate candidates for diagnostic testing to rule out AHP, as confirmed by their PCP.

This study both reiterates the need to validate machine learning approaches in medicine and highlights the challenges of doing so. In addition, there were lessons learned for the specific machine learning use case of rare disease detection. One of the lessons learned in this study concerned the challenges of machine learning researchers who are not part of the care teams of patients. This necessitated the two-step approach of first contacting the PCP and obtaining approval to contact the patient. Another challenge was our looking for a rare disease with which most PCPs had little experience.

Even for those patients who were tested, there were a number of reasons why they had negative results. To begin with, AHP is a rare disease that was already over-represented relative to its population incidence at our tertiary medical center. As such, there may be few patients for whom to make a diagnosis de novo. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted December 7, 2021. ; https://doi.org/10.1101/2021.12.07.21267403 doi: medRxiv preprint Another possibility is that our algorithm selected the wrong patients for testing, i.e., there may be patients who receive care at our medical center but did not rank high enough in the output of our model to be assessed for the need for testing. However, for the patients tested their PCP did confirm that testing was appropriate, so possibly our algorithm cut off point should be extended in the future to include more patients.

An additional challenge for this study is the nature of the symptoms of patients who have AHP. Most of these are non-specific yet can be debilitating to patients, often leading to years of frequent clinical visits and substantial use of diagnostic testing, including expensive imaging procedures.

Our study was also likely hindered by the complex recruitment process that was necessitated by respecting the patient-clinician relationship. Several steps were required to reach busy clinicians to obtain their consent and participation, which then required follow-on with patients who could be difficult to contact. Finally, the fact that we attempted to carry out this study during the COVID-19 pandemic made everything more difficult due to patients' hesitancy to visit healthcare institutions. Despite our lack of success in identifying any new AHP patients, the study did show value in identifying symptomatic patients where a diagnostic test was appropriate and had not yet been done, perhaps improving the completeness of their care. We hope that our experience in trying to conduct this study will inform the community and encourage other researchers to attempt to clinically validate their machine learning systems to help move these approaches from the basic science towards improving clinical care.

Artificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril

Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again

Clinical impact and quality of randomized controlled trials involving interventions evaluating artificial intelligence prediction tools: a systematic review

Clinically important features of porphyrin and heme metabolism and the porphyrias. Metabolites

The incidence of inherited porphyrias in Europe

Acute porphyrias in the USA: features of 108 subjects from porphyrias consortium

An update of clinical management of acute intermittent porphyria

Phase 3 Trial of RNAi Therapeutic Givosiran for Acute Intermittent Porphyria

Detecting rare diseases in electronic health records using machine learning and knowledge engineering: Case study of acute hepatic porphyria