key: cord-0993827-61s0qjeo
authors: Kim, Ue-Hwan; Kim, Moon Young; Park, Eun-Ah; Lee, Whal; Lim, Woo-Hyun; Kim, Hack-Lyoung; Oh, Sohee; Jin, Kwang Nam
title: Deep Learning-Based Algorithm for the Detection and Characterization of MRI Safety of Cardiac Implantable Electronic Devices on Chest Radiographs
date: 2021-08-19
journal: Korean J Radiol
DOI: 10.3348/kjr.2021.0201
sha: fa48057c47d0e97cfabe77bc0b80f84386bdc048
doc_id: 993827
cord_uid: 61s0qjeo

OBJECTIVE: With the recent development of various MRI-conditional cardiac implantable electronic devices (CIEDs), the accurate identification and characterization of CIEDs have become critical when performing MRI in patients with CIEDs. We aimed to develop and evaluate a deep learning-based algorithm (DLA) that performs the detection and characterization of parameters, including MRI safety, of CIEDs on chest radiograph (CR) in a single step and compare its performance with other related algorithms that were recently developed. MATERIALS AND METHODS: We developed a DLA (X-ray CIED identification [XCID]) using 9912 CRs of 958 patients with 968 CIEDs comprising 26 model groups from 4 manufacturers obtained between 2014 and 2019 from one hospital. The performance of XCID was tested with an external dataset consisting of 2122 CRs obtained from a different hospital and compared with the performance of two other related algorithms recently reported, including PacemakerID (PID) and Pacemaker identification with neural networks (PPMnn). RESULTS: The overall accuracies of XCID for the manufacturer classification, model group identification, and MRI safety characterization using the internal test dataset were 99.7% (992/995), 97.2% (967/995), and 98.9% (984/995), respectively. These were 95.8% (2033/2122), 85.4% (1813/2122), and 92.2% (1956/2122), respectively, with the external test dataset. In the comparative study, the accuracy for the manufacturer classification was 95.0% (152/160) for XCID and 91.3% for PPMnn (146/160), which was significantly higher than that for PID (80.0%,128/160; p < 0.001 for both). XCID demonstrated a higher accuracy (88.1%; 141/160) than PPMnn (80.0%; 128/160) in identifying model groups (p < 0.001). CONCLUSION: The remarkable and consistent performance of XCID suggests its applicability for detection, manufacturer and model identification, as well as MRI safety characterization of CIED on CRs. Further studies are warranted to guarantee the safe use of XCID in clinical practice.

Cardiovascular diseases (CVDs) are leading causes of global mortality and major contributors to disability. The burden of CVD has increased in most countries over the past decades [1] . The indications for cardiac implantable electronic devices (CIEDs), including pacemakers (PMs) and implantable cardiac defibrillators (ICDs), have been extended based on recent clinical trials, especially for the primary prevention of sudden cardiac death and improvement of left ventricular dysfunction [2] [3] [4] . Therefore, CIED implantations have increased dramatically over the past several years [5] [6] [7] [8] [9] . Meanwhile, rapid growth in engineering and manufacturing has led to various MRI-conditional CIEDs that have been widely approved [10] [11] [12] [13] . MRI is the most commonly used standard imaging modality because of its excellent spatial resolution, tissue characterization, and lack of radiation [14] . However, MRI for a patient with a CIED still causes trepidation among health practitioners, given the concerns about the interaction between the electromagnetic fields of MRI and CIED. Denying these patients an MRI has grave consequences since an estimated 50%-75% of patients with CIEDs require MRI during their lifetime [15] . Furthermore, when patients visit a hospital for an emergency, in the absence of a dedicated cardiologist or specific information about the patient's CIED, important medical decisions, including whether to obtain an MRI scan, may be delayed [16] . Therefore, easy assessment of MRI compatibility with CIEDs in such circumstances can facilitate medical decisionmaking, as well as allow safe and prompt patient care.

Chest radiography (CR) offers an untapped easily available tool that facilitates reliable identification of the type and manufacturer of CIED. Each CIED has certain distinctive morphological features that make radiological identification possible [17] . To date, there have been a few approaches for CIED identification that involve the manual comparison of the radiographic appearances of a device with a flow chart or deep learning-based device identification from cropped and magnified CRs [18] [19] [20] .

However, for physicians and radiologists who are unfamiliar with CIEDs, manipulating CRs to detect components of the CIED and determining the MRI safety after CIED-type identification may be rate-limiting tasks in an emergency setting. Therefore, we developed a deep learning algorithm (DLA) designed to process CIED detection in a CR and determine MRI safety.

The purpose of this study was to develop and evaluate a DLA that detects and characterizes parameters, including MRI safety, of CIEDs on CRs in a single step and compare its performance with those of other related algorithms that were recently developed.

This retrospective study was approved by the Institutional Review Boards of both hospitals (IRB No. 10-2019-29). The requirement for informed patient consent was waived because the data were analyzed retrospectively and anonymously.

This study analyzed data that comprised dataset 1 (for model training, tuning, and test sets) from Hospital 1: Seoul National University Hospital and dataset 2 (for an independent external test) from Hospital 2: SMG-SNU Boramae Medical Center. Figure 1 illustrates the flowchart of the study inclusion process. For algorithm development, we retrospectively collected 12444 CRs with CIEDs obtained between January 2014 and December 2019 at Hospital 1. Among them, 2375 CRs with invalid labels and 13 CRs for which the conversion of digital imaging and communications in medicine (DICOM) files to portable network graphics (PNG) files had failed were excluded. Training a neural network requires an adequate number of examples of each class to be identified; therefore, only device models for which at least 20 CRs were available were included. The final dataset 1 consisted of 9912 CRs of 958 patients with 968 CIEDs (male:female, 436:522; mean age ± standard deviation [SD] for male and female, 61.9 ± 12.5 and 62.6 ± 11.7 years, respectively; PM:ICD = 766:202). Among the patients with PMs, 10 patients underwent CIED changes during the target period of this study. Both portable and departmental CRs with anterior-posterior/posterior-anterior projections were included. Lateral CRs were not included. A maximum of 15 images were extracted from a patient to minimize class imbalance [21] . All CRs were anonymized according to the Health Insurance Portability and Accountability Act Safe Harbor standards. Thereafter, the final CR data were randomly assigned to one of the following three datasets: a training dataset consisting of 7925 CRs to optimize network weights, a tuning dataset consisting of 992 CRs to optimize hyperparameters, and a test dataset consisting of 995 CRs to evaluate the detection performance of the trained network. Independent dataset 2 included 2122 CRs of 198 https://doi.org/10.3348/kjr.2021.0201 kjronline.org patients (male:female, 102:96; mean age ± SD for male and female, 62.9 ± 12.1, and 69.3 ± 8.9, respectively; PM:ICD = 159:39) and was obtained from a different hospital for external testing of the DLA. The patients in the datasets did not overlap.

We categorized the CIEDs into 26 model groups and counted the number of instances of devices in each model group. We filtered the CRs containing the model groups that did not contain at least 20 CIEDs for both datasets 1 and 2. Next, we randomly split dataset 1 into training, tuning, and internal test sets with a ratio of 8:1:1 for model development. Dataset 2 was used for the independent external test. The distribution of datasets 1 and 2 is presented in Supplementary Table 1 . Information about the MRI safety (i.e., compatibility) of the devices was confirmed by each manufacturer in April 2020. Figure 2 illustrates the data flow of the proposed DLA, X-ray CIED identification (XCID). XCID consisted of two main networks: 1) a class-agnostic CIED detector (CCD) and 2) a multi-task learner (MTL) for simultaneous CIED identification and MRI safety characterization. First, the CCD receives the input CR and outputs the candidate positions of the CIED with confidence scores. If a confidence score was lower than a preset threshold, the position corresponding to the score was discarded (we empirically set the threshold as 0.5) [22] [23] [24] . Next, the CIED region was cropped, resized to 224 x 224 pixels, and normalized. Subsequently, the MTL was used in the preprocessed CIED region, and 1) the manufacturer, 2) the model group, and 3) the MRI safety of the device were identified. Further detailed engineering methodologies for the development of CCD and MTL are described in the Supplement (Materials and Methods). study inclusion process. We refined the collected data in three steps. In the first step, the validity of each label was investigated and CRs with invalid labels were removed. Next, CRs with failure to convert DICOM files to PNG files were excluded. Last, CRs whose group did not include 20 instances were filtered out. Data set 1 is further split into three sets for model development: training, tuning, and test sets. BMC = SMG-SNU Boramae Medical Center, CR = chest radiograph, DICOM = digital imaging and communications in medicine, PNG = portable network graphics, SNUH = Seoul National University Hospital kjronline.org

First, the performance of XCID was evaluated using an internal test dataset; that is, part of development dataset 1 that was not used for training. Thereafter, an external test was performed using independent dataset 2 to confirm the consistency of XCID performance. After analysis of the radiographs using the algorithm, performance measurement was assessed through the use of binary and multiclass areas under the curve (AUC) through the analysis of the receiver operating characteristics for manufacturer classification, model group identification, and MRI safety characterization. Accuracy, sensitivity, precision, and F1 scores were also evaluated for the three tasks.

PM identification with neural networks (PPMnn) was developed by Howard et al. [19] using chest X-rays in the United Kingdom with the availability of the algorithm at http://ppm.jph.am [19] . The PPMnn was trained with 1676 device images obtained from the Imperial College Healthcare NHS Trust. Weinreich et al. [20] developed PacemakerID (PID; available at http://app.pacemakerid. com/) using chest X-rays in the United States [20] . The algorithm was developed using cropped images from 1509 chest X-ray images of CIED obtained from a single institution, Montefiore Medical Center.

We randomly selected a total of 160 CRs from dataset 2, considering class balance (40 CRs per manufacturer) in comparing the performance when classifying the manufacturer between XCID and the two other algorithms. The manufacturer classification was analyzed for all three algorithms. The predictions of each algorithm showed the prediction certainty percentages. The prediction was considered correct if > 75% predicted certainty was assigned to the correct manufacturer [25] . The model group identification was evaluated for XCID and PPMnn because the PID did not return the model identification. The MRI safety characterizations could not be compared, as PID and PPMnn did not have a predictive function.

All the statistical analyses were performed in June 2020 using Python 3.6.7 with Numpy library version 1.17.3 and R version 4.0.2 (The R Foundation for Statistical Computing). kjronline.org

The 95% confidence interval (CI) was calculated using the bootstrap method with 10000 replicates for the AUC.

McNemar's test was used to compare the findings obtained with different algorithms [25] . The p values were two-sided, and p < 0.05 denoted statistical significance.

The model architectures for XCID selected in this study were the MobileNet backbone trained with transfer learning for CCD and the DenseNet backbone trained with transfer learning for MTL. The detailed comparative results are described in Supplement (Results) and Supplementary Tables 2, 3, including accuracy for device identification, model group identification, and MRI safety characterization using various deep learning architectures. for the external test dataset, respectively. The performance gap in the accuracy metric was increased in the order of manufacturer classification (3.89%), MRI safety characterization (6.71%), and model group classification (11.8%). Figure 3 shows the confusion matrix for the model group classification task in the external test dataset.

Figures 4 and 5 display two sets of exemplary CRs overlaid with heatmaps generated using the Grad-CAM method [26] . For one input CR, XCID reacted in three different ways to conduct the three tasks simultaneously. The MTL focused on the electronic circuit regions of CIED pulse generators in two slightly distinctive ways to classify the manufacturer and the model group, while the model considered the overall portrait in distinguishing MRI safety. The tendency of XCID to examine the overall portrait for MRI safety characterization was consistent with our hypothesis that the MTL would require more general features to distinguish MRI safety than the other two tasks. In addition, the last two rows in Figure 4 suggest that XCID can be performed successfully even when CR quality is moderately degraded. The failure cases (Fig. 5) implied that the MTL could not function when CR showed poor image quality with the obscured configuration of a circuit in PG, an over-exposed radiograph, over-angled projection of PG, and multiple

Diagonal components not displayed in the confusion matrix indicate that the external test data set does not contain cardiac implantable electronic devices with those model groups. kjronline.org approach to classify CIED manufacturers [18] . However, new models that are not covered by CaRDIA-X have been introduced over the past decade. Recently, PPMnn and PID algorithms have been developed in different regions and architectures. Howard et al. [19] developed a CIED identification algorithm with neural networks (PPMnn) using 1676 cropped chest radiographs, including five manufacturers (Biotronik, Boston Scientific, Medtronic, St. Jude, and Sorin) from England. Weinreich et al. [20] reported a DLA (PID) that identifies CIED using 3008 cropped chest radiographs including four manufacturers (without Sorin), which was the same as the XCID at the Montefiore Medical Center in the United States. A study that used a dataset (n = 500) from another single center in New York evaluated the accuracy of the manufacturer classification using these algorithms. The overall accuracy of PID was 89%. PPMnn showed a 71% accuracy, which was lower than that of the manual assessment using CaRDIA-X (85%) [25] . Our study included the development and external testing of the DLA with 12042 consecutive chest radiographs since 2014, when the MRI-conditional CIED was approved and had started being widely used to develop and test XCID. In contrast with the research conducted by Howard et al. [19] , with data consisting of different CIEDs in equal proportions, our data reflected the actual prevalence of different CIEDs in the institution-centered market. The CIED market share at a single institution varies with the device type and the region [27] , likely resulting in a lack of availability of Boston Scientific devices in the XCID training set and a resulting low accuracy of this subset [25] . The gaps in AUC and the accuracy of the model group identification in the internal and external test sets were relatively more prominent than those of the manufacturer classification and MRI compatibility characterization ( Table 1 ). The main reason for this could be that the model group identification had performed an external test for an independent imbalanced dataset (Supplementary Table 1) with multiclass classification for "26 subgroups" compared with the other two tasks [28] [29] [30] . Further multicenter research with data augmentation and temporal and geographic validation is needed to overcome these gaps [31] . Our one-step DLA started from a whole DICOM image, not a cropped image converted to a PNG file (used in PPMnn and PID), to determine whether a CIED was present and the PG location. Nevertheless, the overall results of XCID were not inferior for the manufacturer classification and superior for the model group identification to PPMnn. XCID was developed and tested for device identification into a single step from the entire CR and stored as a DICOM file used in picture archiving and communication systems (PACSs) without any additional manipulation or designation of a region of interest. This means that XCID may be directly applied to PACS without errors during CR manipulation in the practice of personnel unfamiliar with CIED. To be approved as a medical device software and used in clinical practice, the user interface for XCID can be embedded into PACS to analyze the CRs.

This tool would be helpful for physicians and radiologists who are not familiar with CIEDs, as well as for emergency patients with CIEDs and dedicated cardiologists using CRs, which are the most basic initial assessments in hospitals. Only an industry-employed allied professional (IEAP) would be able to communicate with the CIED using bulky portable programmers. Knowing which programmer to use would save critical time [32] . This may not only facilitate decision-making for MRI acquisition and rapid interrogation of a device in an emergency, but it would also help in the provision of emergency treatment, such as anti-tachycardia pacing [16, 33, 34] . This can avoid delays in calibrating the devices and reduce hospital costs, other indirect patient expenditures, and service costs for device manufacturers because IEAP visits can be avoided. Contact-free rapid identification of CIEDs can be useful in healthcare crises kjronline.org such as the COVID-19 pandemic [34] . Regarding the fatal complications of misjudgment, DLAs including XCID should be improved and applied to the standard workflow. The final clearance should be provided prudently by physicians having the needed relevant information. Based on the successful and failed classification results (Figs. 4, 5) , clearer demonstrations of PGs in CRs without overlapping or oversaturation would attain higher accuracy and more reliable results from XCID.

This study had some limitations. Our dataset did not include multiple heterogeneous sources of CRs with CIEDs. We obtained datasets from two medical centers using different reconstruction methods, partly overcoming the limitations of a single-center study. This could be a reason why the external test study revealed slightly lower accuracy than that achieved during DLA development. XCID cannot identify devices not listed in Supplementary Table 1 , such as CIEDs made by MicroPort (formerly LivaNova and ELA/ Sorin). The training dataset had a class imbalance because its origin was consecutive Real-Real-real-world data and reflected regional market shares. However, software engineering will allow continuous augmentation because the proposed MTL of XCID requires less than 7 minutes to train with more than 8000 images. Extending the capability of XCID requires only 20 additional examples per new device. Further research is warranted to determine the applicability of XCID to prospectively collected large multi-institutional balanced datasets. For the MRI safety characterization, we did not consider the models of leads in the XCID. It is important to note that the CIED lead tips are unaffected by static magnetic fields, as they contain no ferromagnetic materials. This negates the possibility of the lead becoming dislodged and failing to capture [35, 36] . However, the radiofrequency field may generate heat energy, particularly at the lead tips, which can result in myocardial tissue damage [37] . Nevertheless, there have been negligible effects on post-MRI troponin concentrations; there have been few participants with increased troponin concentrations in animal and human studies [38] [39] [40] [41] . Furthermore, when the MRI-conditional PG is inserted or replaced, the cardiologist and IEAP tend to use MRIconditional leads. Therefore, we decided that it would be acceptable to consider models of the PG only for MRI safety characterization during DLA development.

In conclusion, XCID may be used for detection, manufacturer and model identification, as well as MRI safety characterization of CIED on CRs with remarkable and consistent performance. Further studies are warranted to guarantee the safe use of XCID in clinical practice.

The Supplement is available with this article at https://doi.org/10.3348/kjr.2021.0201.

The authors have no potential conflicts of interest to disclose.

Kwang Nam Jin. Writing-review & editing: Ue-Hwan

Global burden of cardiovascular diseases and risk factors, 1990-2019: update from the GBD 2019 study

ACCF/AHA/HRS focused update incorporated into the ACCF/AHA/HRS 2008 guidelines for device-based therapy of cardiac rhythm abnormalities: a report of the American College of Cardiology Foundation/ American Heart Association Task Force on Practice Guidelines and the Heart Rhythm Society

ESC Guidelines on cardiac pacing and cardiac resynchronization therapy: the Task Force on cardiac pacing and resynchronization therapy of the European Society of Cardiology (ESC)

ESC guidelines for the management of patients with ventricular arrhythmias and the prevention of sudden cardiac death: the task force for the management of patients with ventricular arrhythmias and the prevention of sudden cardiac death of the European Society of Cardiology (ESC). Endorsed by: Association for European Paediatric and Congenital Cardiology (AEPC)

The 11th world survey of cardiac pacing and implantable cardioverter-defibrillators: calendar year 2009--a World Society of Arrhythmia's project

Transvenous lead extraction: Heart Rhythm Society expert consensus on facilities, training, indications, and patient management: this document was endorsed by the

Implantation trends and patient profiles for pacemakers and implantable cardioverter defibrillators in the United States

Trends in permanent pacemaker implantation in the United States from 1993 to 2009: increasing complexity of patients and procedures

Temporal trends of cardiac implantable electronic device implantations: a nationwide population-based study

HRS expert consensus statement on magnetic resonance imaging and radiation exposure in patients with cardiovascular implantable electronic devices

Magnetic resonance imaging in patients with a subcutaneous implantable cardioverter-defibrillator

Short-and long-term electrical performance of the 5086MRI pacing lead

Clinical safety of the ProMRI pacemaker system in patients subjected to head and lower lumbar 1.5-T magnetic resonance imaging scanning conditions

Cardiac implantable electronic device safety during magnetic resonance imaging

Current clinical issues for MRI scanning of pacemaker and defibrillator patients

Clinical assessment and management of patients with implanted cardioverter-defibrillators presenting to nonelectrophysiologists

Principles of cardiovascular radiology

Cardiac rhythm device identification algorithm using X-rays: CaRDIA-X

Cardiac rhythm device identification using neural networks

Development of an artificially intelligent mobile phone application to identify cardiac devices on chest radiography

Training neural network classifiers for medical decision making: the effects of imbalanced datasets on classification performance

The pascal visual object classes (voc) challenge

Microsoft coco: common objects in context

The open images dataset v4

A head-to head comparison of machine learning algorithms for identification of implanted cardiac devices

Grad-CAM: visual explanations from deep networks via gradient-based localization

Development and validation of a novel algorithm based on the ECG magnet response for rapid identification of any unknown pacemaker

Clinical implementation of deep learning in thoracic radiology: potential applications and challenges

Deep learning for chest radiograph diagnosis in the emergency department

Multiclass semantic segmentation and quantification of traumatic brain injury lesions on head CT using deep learning: an algorithm development and multicentre validation study

Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction

Radiography of pacemakers and implantable cardioverter defibrillators

The relationship between MRI radiofrequency energy and function of nonconditional implanted cardiac devices: a prospective evaluation

Managing cardiac implantable electronic device patients during a health care crisis: practical guidance

Risks to pacemaker patients undergoing magnetic resonance imaging examinations

MRI in patients with cardiac implantable electronic devices

Pacing in magnetic resonance imaging environment: clinical and technical considerations on compatibility

In vivo heating of pacemaker leads during magnetic resonance imaging

Cardiac biomarkers in patients with permanent pacemakers and implantable cardioverter-defibrillators undergoing an MRI scan

Modern pacemaker and implantable cardioverter/defibrillator systems can be magnetic resonance imaging safe: in vitro and in vivo assessment of safety and function at 1.5 T

Cardiac pacemakers and implantable cardioverter defibrillators: in vitro magnetic resonance imaging evaluation at 1.5-tesla