key: cord-1033359-6m0sr34k
authors: Hwang, Eui Jin; Kim, Hyungjin; Yoon, Soon Ho; Goo, Jin Mo; Park, Chang Min
title: Implementation of a Deep Learning-Based Computer-Aided Detection System for the Interpretation of Chest Radiographs in Patients Suspected for COVID-19
date: 2020-07-17
journal: Korean J Radiol
DOI: 10.3348/kjr.2020.0536
sha: bc0d903c0b9b564c7ca2aa3406559753e318306c
doc_id: 1033359
cord_uid: 6m0sr34k

OBJECTIVE: To describe the experience of implementing a deep learning-based computer-aided detection (CAD) system for the interpretation of chest X-ray radiographs (CXR) of suspected coronavirus disease (COVID-19) patients and investigate the diagnostic performance of CXR interpretation with CAD assistance. MATERIALS AND METHODS: In this single-center retrospective study, initial CXR of patients with suspected or confirmed COVID-19 were investigated. A commercialized deep learning-based CAD system that can identify various abnormalities on CXR was implemented for the interpretation of CXR in daily practice. The diagnostic performance of radiologists with CAD assistance were evaluated based on two different reference standards: 1) real-time reverse transcriptase-polymerase chain reaction (rRT-PCR) results for COVID-19 and 2) pulmonary abnormality suggesting pneumonia on chest CT. The turnaround times (TATs) of radiology reports for CXR and rRT-PCR results were also evaluated. RESULTS: Among 332 patients (male:female, 173:159; mean age, 57 years) with available rRT-PCR results, 16 patients (4.8%) were diagnosed with COVID-19. Using CXR, radiologists with CAD assistance identified rRT-PCR positive COVID-19 patients with sensitivity and specificity of 68.8% and 66.7%, respectively. Among 119 patients (male:female, 75:44; mean age, 69 years) with available chest CTs, radiologists assisted by CAD reported pneumonia on CXR with a sensitivity of 81.5% and a specificity of 72.3%. The TATs of CXR reports were significantly shorter than those of rRT-PCR results (median 51 vs. 507 minutes; p < 0.001). CONCLUSION: Radiologists with CAD assistance could identify patients with rRT-PCR-positive COVID-19 or pneumonia on CXR with a reasonably acceptable performance. In patients suspected with COVID-19, CXR had much faster TATs than rRT-PCRs.

The initial outbreak of coronavirus disease , caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection (1) , originated in Wuhan, China in December 2019. Following this occurrence, the disease rapidly progressed into a pandemic, with more than 5.4 million confirmed patients and more than 340000 deaths Implementation of a Deep Learning-Based Computer-Aided Detection System for the Interpretation of Chest Radiographs in Patients Suspected for COVID-19 https://doi.org/10.3348/kjr.2020.0536 kjronline.org findings of pneumonia in patients with initially negative rRT-PCR results (4) (5) (6) and can be considered as a screening tool for COVID-19 in epidemic areas. However, screening or early diagnosis using chest CT in patients suspected for COVID-19 may not be practical, owing to the risk of viral transmission during the examination, transportation of the patient, and difficulty with disinfecting the environment. Owing to this, chest CT is currently not recommended for screening or initial diagnosis of COVID-19 (7, 8) .

Chest X-ray radiograph (CXR) is the primary imaging technique in the diagnosis of pneumonia because of its easy accessibility, low cost, low radiation exposure, and reasonable diagnostic capability (9, 10) . Therefore, CXR using a portable radiography unit may be considered as a primary radiologic examination for COVID-19 because patient transportation can be minimized and disinfection is relatively easy (7) . However, because of the intrinsic limitation of the two-dimensional projection image, where various anatomic or pathologic structures are overlapped, CXR have lower sensitivity compared to chest CTs (11, 12) and are prone to reading errors and inter-or intra-reader variability (13) . Thus, a computer-aided detection (CAD) system that can accurately identify pulmonary opacities suggestive of pneumonia may help promptly and accurately diagnose pneumonia, such as that observed in COVID-19 patients (14) . In this regard, we implemented a deep learning-based CAD system for the interpretation of CXR of patients who were suspected for COVID- 19 .

We aimed to describe our experience of implementing a deep learning-based CAD system for the interpretation of CXR of suspected COVID-19 patients, as well as to investigate the diagnostic performance of CXR for COVID-19 using the CAD system.

This single-center, retrospective study was approved by the Institutional Review Board of Seoul National University Hospital, and the requirement for informed consent was waived.

We retrospectively included consecutive patients using the following criteria: 1) patients who visited a tertiary academic institution for the diagnosis of suspicious COVID-19 or management of confirmed COVID-19 between January 31, 2020 and March 10, 2020; and 2) patients who underwent CXR with a dedicated protocol for suspicious COVID-19 patients including CAD analyses. The initial CXR of each patient obtained after the visit were included in the present study. All CXR were obtained with a dedicated mobile X-ray system (DRX-revolution, Carestream Health). Erect posteroanterior X-rays or supine anteroposterior X-rays were obtained, depending on the patients' condition.

A dedicated CXR examination protocol for patients suspicious of COVID-19 ("CXR AI CAD for COVID") was established on January 28, 2020, and included CAD analysis. All subsequent CXR of patients who were suspected or already diagnosed with COVID-19 were obtained using this protocol. The CAD system integrated in the protocol was a commercialized deep-learning algorithm (Lunit INSIGHT CXR 2, Lunit Inc.) that was approved by the Ministry of Food and Drug Safety of Korea (15) that detects several thoracic abnormalities, including pulmonary nodules, consolidation, and pneumothorax. The CAD system was originally trained with 54221 normal CXR and 35613 abnormal CXR with four major thoracic diseases including pulmonary malignancy, pneumonia, pulmonary tuberculosis, and pneumothorax (15). The CAD system provided a probability score between 0% and 100% for the presence of any of the target abnormalities on each CXR and provided localization of abnormalities when the probability score was 15% or greater, with contour lines overlaid on CXR images (Figs. 1-3). The system was unable to provide detailed information on whether each detected abnormality was a nodule, mass, consolidation, or pneumothorax.

After the acquisition of the CXR, analysis by the CAD system was automatically processed, and both radiologists and referring physicians could view the CAD result side-byside with the original CXR image on the institution's picture archiving and communication system (PACS; Infinitt Gx PACS, INFINITT Healthcare). After evaluation of both CXR images and CAD results, radiologists documented a formal report using the PACS (formal radiology report, hereinafter). All CXR were interpreted by attending radiologists or by radiology residents supervised by attending radiologists. A total of 14 attending radiologists and 12 residents (1-29 years of experience in CXR interpretation) participated in CXR interpretation.

The formal radiology reports of all CXR were https://doi.org/10.3348/kjr.2020.0536

kjronline.org retrospectively reviewed by one thoracic radiologist (9 years of experience in CXR and chest CT interpretation), and classified into those indicating the presence versus those indicating the absence of any abnormality suggesting pneumonia. For the CAD results, a probability score of 15% (threshold for visualization of localization information) was defined as the threshold for binary classification between positive and negative results. The diagnostic performance of formal radiology reports and CAD results were evaluated using two different reference standards: 1) rRT-PCR results for SARS-CoV-2 infection, and 2) pulmonary abnormality suggesting pneumonia on chest CT. For evaluation based on rRT-PCR results, only patients with an available rRT-PCR result within 24 hours of the CXR were included. Positive rRT-PCR results from nasopharyngeal or oropharyngeal swabs indicated COVID-19.

For evaluation based on chest CT, only patients who underwent chest CT within 24 hours of the CXR were included. All chest CTs were obtained based on the decision of the referring physician, without pre-defined criteria for CT acquisition. Two thoracic radiologists (9 and 10 years of experience in CXR and chest CT interpretation) independently reviewed all chest CTs to determine the presence of abnormalities suggesting pneumonia. Discordant interpretations were arbitrated by the final decision of a senior thoracic radiologist (21 years of experience in CXR A. CXR of patient with COVID-19 shows no definite pulmonary opacity. B, C. Computer-aided detection system classified CXR as normal, with probability score below 15% (threshold for visualization). Formal radiology report indicated no abnormal finding on CXR. Chest computed tomography images obtained on same day show multifocal patchy consolidations and ground-glass opacities in bilateral lungs.

https://doi.org/10.3348/kjr.2020.0536

kjronline.org and chest CT interpretation). In patients with pulmonary abnormalities suggesting pneumonia on chest CTs, two thoracic radiologists determined whether the abnormality was visible on CXR until a consensus was reached. Subsequently, the formal radiology reports of the CXR and CAD results were compared to the presence of abnormalities on chest CTs. Positive radiology reports or CAD results with incorrect localization of abnormalities were regarded as false negatives. Among patients with chest CTs, in cases of false positive identification by formal radiology reports or CAD, a single thoracic radiologist reviewed CXR and chest CTs to determine the cause of the false positive detection.

We compared the rRT-PCR-and chest CT-based diagnostic performance of formal radiology reports and CAD results between patients with and without symptoms suggesting acute respiratory disease, as well as between patients with a symptom duration of ≤ 3 days or > 3 days. 

The turnaround time (TAT) of each CXR report by radiologists and rRT-PCR result (time interval between CXR or specimen acquisition and formal reporting) were obtained.

The sensitivities, specificities, positive predictive values (PPVs), and negative predictive values (NPVs) of the formal radiology reports and the standalone CAD results were obtained. The sensitivities and specificities were compared using McNemar's test, while the PPVs and NPVs were compared using the method suggested by Leisenring et al. (16) . Pearson's chi-square or Fisher's exact tests were used for the comparison of performance between different subgroups depending on the situation. Median tests were performed for the comparison of TATs.

All statistical analyses were performed using R (version 3.5.5, R Project for Statistical Computing), and p values < 0.05 were considered statistically significant.

A total of 395 patients (205 male and 190 female; mean age ± standard deviation, 53 ± 24 years) were included in this study. Among them, 283 (71.6%) patients had symptoms suggesting acute respiratory disease, and the median time interval between the symptom onset and CXR was 1 day (interquartile range [IQR], 4 days) ( Table 1 ).

In formal radiology reports, abnormalities suggesting pneumonia were reported in 31.9% (126/395) of CXR, while the CAD system identified abnormalities in 36.7% (145/395) of CXR. The CAD system identified the same abnormalities reported on the formal reports in 72.2% (91/126) of CXR with positive reports. Among the CXR in which the CAD system identified abnormalities, 33.1% (48/145) were discarded as false positives by the interpreting radiologists.

Among 332 patients (84.1%; 173 male and 159 female; mean age ± standard deviation, 57 ± 23 years) with available rRT-PCR results within 24 hours of CXR, SARS-CoV2 infections were identified in 16 patients (4.8%; 9 male and 7 female; mean age ± standard deviation, 60 ± 20 years). Among them, 12 patients (75%) were symptomatic, and the median time interval between the symptom onset and CXR was 5 days (IQR, 9 days).

The formal radiology reports exhibited sensitivity, specificity, PPV, and NPV of 68.8%, 66.7%, 9.5%, and 97.7%, respectively. The specificity (p = 0.005) and PPV (p = 0.033) of the formal radiology reports were significantly higher than those of the CAD output (sensitivity, 43.8%; specificity, 59.8%; PPV, 5.2%; NPV, 95.5%), but there was no significant difference in the sensitivity (p = 0.102) and NPV (p = 0.061) ( Table 2 ). (Table 3) . Formal radiology reports exhibited sensitivity, specificity, PPV, and NPV of 81.5%, 72.3%, 71.0%, and 82.5%, respectively. The specificity (p = 0.002) and PPV (p = 0.002) of formal radiology reports were significantly higher than those of the CAD output (sensitivity, 81.5%; specificity, 52.3%; PPV, 58.7%; NPV, 77.3%) ( Table 2) .

Among the 119 patients with available chest CTs, three (2.5%) were rRT-PCR-positive COVID-19 patients, two of whom exhibited pulmonary abnormalities on chest CT (sensitivity, 66.7%). Formal radiology reports for CXR identified abnormalities in those two patients (sensitivity, 66.7%), while the standalone CAD results identified abnormality in only one patient (sensitivity, 33.3%).

Among the 54 patients with pulmonary abnormality suggesting pneumonia on CT, 92.6% (50/54) of the CXR had visible abnormality. Both the formal radiology reports and CAD results exhibited sensitivity of 86% (43/50; 95% confidence interval, 73.3-94.2%) for CXR with visible abnormalities, which were significantly higher than those for CXR with invisible abnormalities (25.0% [1/4; 95% confidence interval, 6.3-80.6%]; p = 0.017). Two patients with chest CTs but without rRT-PCR results were excluded. *Numbers in parentheses indicate proportion among patients with any abnormality suggesting pneumonia. kjronline.org

False positives on CXR occurred in 18 formal radiology reports and 32 CAD results (Table 4) , with pleural effusion as the most common cause in both. The second most common cause of false positives was interstitial lung diseases for the formal radiology reports and pulmonary nodules for the CAD results.

For identification of rRT-PCR-positive COVID-19 patients, neither the performance of formal radiology reports nor the CAD results showed a significant between patients with and without symptoms suggesting acute respiratory disease ( For identification of pneumonia demonstrated on chest CTs, neither the performance of formal radiology reports nor CAD results showed a significant difference between patients with and without symptoms suggesting acute respiratory disease, as well as between patients with symptom duration ≤ 3 days and > 3 days ( Table 6) .

The median TAT for CXR reports was significantly shorter than that for rRT-PCR results (51 [IQR, 138] 

Herein, we have described our experience of a deep learning-based CAD system for the interpretation of CXR of suspected COVID-19 patients. The formal radiology reports with the assistance of CAD exhibited sensitivity of 68.8% and specificity of 66.7% for the identification of rRT-PCRpositive COVID-19 patients, and a sensitivity of 81.5% and specificity of 72.3% for the identification of pulmonary abnormalities suggesting pneumonia on chest CT.

The positive rate of rRT-PCR in our study (4.8%) was higher than the cumulative positive rate in Korea (2.3% as of April 4, 2020) (17) , but similar to the data published in the earlier stage of the outbreak (4.5% as of March 2, 2020) (18) .

Previous studies have reported that CXR of COVID-19 patients may appear normal (19) (20) (21) . Furthermore, reported imaging findings of COVID-19 were bilateral ground-glass opacities with or without consolidations, which are nonspecific in diagnosing COVID-19 (19) (20) (21) (22) (23) . Not surprisingly, both formal radiology reports (sensitivity, 68.8%; specificity, 66.7%) and CAD results (sensitivity, 43.8%; specificity, 59.8%) showed limited diagnostic performance for patients with SARS-CoV-2 infection in the present study. The sensitivity of formal radiology reports in our study was similar to that of baseline CXR reported in a previous study by Wong et al. (21) (69%) . Although there was no statistically significant difference, the standalone CAD results exhibited a substantially lower sensitivity compared to the formal radiology reports (68.8% vs. 43.8%), suggesting limited potential for diagnosis of COVID-19 on CXR with the CAD system only.

Despite the limited diagnostic performance, the radiological evaluation of lung lesions in COVID-19 (4) (5) (6) . In addition, Zhao et al. (24) reported that the extent of abnormalities on chest CT was greater in severe diseases. Furthermore, previous investigations on severe acute respiratory syndrome outbreak between 2002 and 2004 have reported that the extent of pulmonary opacity on CXR was a prognostic factor for the adverse patient outcome (25) (26) (27) . However, obtaining chest CT for all patients suspected for COVID-19 can be very challenging in practice due to the limited availability of CT scanners dedicated to COVID-19 patients, as well as the contagion risk of the virus. Therefore, CXR can be used as the main radiologic examination for patients suspected for COVID-19 and may assist with patient management and prognosis prediction. With regards to the identification of pulmonary abnormalities suggesting pneumonia based on chest CT, both the formal radiology reports and CAD results exhibited sensitivities of 81.5%. According to a previous multi-center cohort study by Self et al. (12) comprising 3423 patients in the emergency department, the sensitivity of CXR for pulmonary opacity, with chest CT as a reference standard, was 43.5%. Hence, we considered that the sensitivity of the CAD system would be acceptable for implementation in clinical practice to detect pneumonia. Although we were unable to determine the baseline performance of radiologists without use of the CAD system, it may enhance sensitivity for the detection of pulmonary abnormalities. In a previous study using the same CAD system, the sensitivity of radiology residents for identification of abnormal CXR were significantly enhanced (65.6-73.4%) after review of the CAD results (28) . Although the CAD system utilized in our study was only designed to detect a limited number of abnormalities, including pulmonary nodules, masses, consolidations, and pneumothorax, the CAD could actually identify other types of lung parenchymal abnormalities, such as ground-glass opacities, the representative imaging finding of COVID-19 ( Fig. 1) , since it was initially trained to identify various thoracic diseases including pneumonia (15) . Because parenchymal abnormalities related to pneumonia, such as consolidation, ground-glass opacities, reticular opacities often look similar and are indistinguishable on CXR, the CAD system was trained to identify abnormalities related to pneumonia, and was not confined to consolidations. However, the non-specific identification of several abnormalities also resulted in false positives, which led to diminished specificity (59.8% for rRT-PCR-positive COVID-19; 58.7% for CT abnormality suggesting pneumonia) of the CAD (Table 3) . However, these false positives could be significantly reduced in the formal reports, since CAD was only used as an assistant tool for radiologists, who were able to discard these false positives upon their interpretation of CXR.

In addition to morphologic types of abnormality on CXR, the visibility of abnormalities on CXR may be important for the detectability of the lesions. In our study, both the CAD system and formal radiology reports exhibited significantly higher sensitivities for visible abnormalities than invisible abnormalities. Although extensive ground-glass opacities that are clearly visible on CXR can be identified by the CAD (Fig. 1) , subtle ground-glass opacities that are barely visible on CXR may be missed by the CAD (Fig. 2) .

In our study, the formal radiology reports for CXR exhibited satisfactory TATs (median, 51 minutes), that were significantly shorter than those of the rRT-PCR results (median, 507 minutes). Therefore, we believe that CXR may help screening and triage of patients with high suspicion of COVID-19 pneumonia, especially during an outbreak where there are limited resources for hospitalization and intensive care (21, 29) . Considering that our study was performed in a tertiary-referral hospital where radiologists were readily available to interpret CXR, report TATs might be much longer in primary healthcare or community-level practice where immediate availability of radiologists may be limited. Although further studies are required, we believe the CAD system may assist with timely clinical decision making in those situations. In this situation, patients with positive radiographs could be subject to enhanced isolation, which would minimize the transmission of COVID-19 while waiting for rRT-PCR results.

In our practice, no significant difference was observed in the TATs of CXR reports and rRT-PCR results between patients with and without COVID-19, or between patients with and without pneumonia. If the CAD system could be integrated with a notification system that could inform radiologists or physicians of abnormal CAD results immediately after the acquisition of CXR, it may facilitate prioritization of patients with higher suspicion of pneumonia (30) .

Our study has several limitations. First, the formal radiology reports analyzed in the present study were results of interpretation using the CAD system. Therefore, a direct https://doi.org/10.3348/kjr.2020.0536 kjronline.org comparison of performance between the radiologists and the CAD system was not possible, and we could not evaluate whether the CAD system improved the performance of radiologists. To evaluate the effect of the CAD system on the performance of radiologists in a suspected COVID-19 population, further investigation, including the interpretation before and after use of the CAD system, is warranted. Second, our study was performed in a single tertiary institution with a limited number of COVID-19 patients. Therefore, it is difficult to generalize our results, considering that the situation for evaluating patients suspected for COVID-19 may differ significantly across institutions or countries. Third, the CAD system utilized in our study was not trained for all radiographic abnormalities, nor was it trained specifically for COVID-19.

In summary, we implemented a deep learning-based CAD system for the interpretation of CXR of patients suspected for COVID-19. The formal radiology reports with the assistance of CAD exhibited reasonably acceptable performances for identification of rRT-PCR-positive COVID-19 patients (sensitivity, 68.8%) and CT abnormalities suggesting pneumonia (sensitivity, 81.5%). Moreover, the CAD system resulted in faster TATs than rRT-PCR results. In situations where there are limited medical resources, such as during an outbreak, CXR interpretation with the assistance of CAD may assist clinical decision making and management of patients suspected for COVID-19.

A novel coronavirus from patients with pneumonia in China

Situation report-127. World Health Organization

Interim guidelines for collecting, handling, and testing clinical specimens from persons for coronavirus disease 2019 (COVID-19)

Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases

Sensitivity of chest CT for COVID-19: comparison to RT-PCR

Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing

Advocacy-and-Economics/ACR-Position-Statements/ Recommendations-for-Chest-Radiography-and-CT-for-Suspected-COVID19-Infection

STR/ASER COVID-19 position statement. Society of Thoracic Radiology Web site

ACR appropriateness Criteria® acute respiratory illness in immunocompetent patients

Infectious Diseases Society of America/ American Thoracic Society consensus guidelines on the management of community-acquired pneumonia in adults

Diagnostic value of chest radiographs in bedridden patients suspected of having pneumonia

High discordance of chest

kjronline.org tomography for detection of pulmonary opacities in ED patients: implications for diagnosing pneumonia

Common patterns in 558 diagnostic radiology errors

Can artificial intelligence improve the management of pneumonia

Development and validation of a deep learning-based automated detection algorithm for major thoracic diseases on chest radiographs

Comparisons of predictive values of binary medical diagnostic tests for paired designs

Latest updates, cases in Korea. Korean Ministry of Health and Welfare Web site

Korean Society for Antimicrobial Therapy, Korean Society for Healthcare-associated Infection Control and Prevention

Chest radiographic and CT findings of the 2019 novel coronavirus disease (COVID-19): analysis of nine patients treated in Korea

Imaging profile of the COVID-19 infection: radiologic findings and literature review

Frequency and distribution of chest radiographic findings in COVID-19 positive patients

CT imaging features of 2019 novel coronavirus (2019-nCoV)

Radiological findings from 81 patients with COVID-19 pneumonia in Wuhan, China: a descriptive study

Relation between chest CT findings and clinical conditions of coronavirus disease (COVID-19) pneumonia: a multicenter study

Radiographic-clinical correlation in severe acute respiratory syndrome: study of 1373 patients in Hong Kong

Severe acute respiratory syndrome: correlation between clinical outcome and radiologic features

Severe acute respiratory syndrome: prognostic implications of chest radiographic findings in 52 patients

Deep learning for chest radiograph diagnosis in the emergency department

Radiology department preparedness for COVID-19: facing an unexpected outbreak of the disease

Clinical implementation of deep learning in thoracic radiology: potential applications and challenges