key: cord-1039097-uhy086fn
authors: Bai, Harrison X.; Hsieh, Ben; Xiong, Zeng; Halsey, Kasey; Choi, Ji Whae; Tran, Thi My Linh; Pan, Ian; Shi, Lin-Bo; Wang, Dong-Cui; Mei, Ji; Jiang, Xiao-Long; Zeng, Qiu-Hua; Egglin, Thomas K.; Hu, Ping-Feng; Agarwal, Saurabh; Xie, Fangfang; Li, Sha; Healey, Terrance; Atalay, Michael K.; Liao, Wei-Hua
title: Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT
date: 2020-03-10
journal: Radiology
DOI: 10.1148/radiol.2020200823
sha: 38bc5123323030d9e72f3433590fc879249744e1
doc_id: 1039097
cord_uid: uhy086fn

BACKGROUND: Despite its high sensitivity in diagnosing COVID-19 in a screening population, chest CT appearances of COVID 19 pneumonia are thought to be non-specific. PURPOSE: To assess the performance of United States (U.S.) and Chinese radiologists in differentiating COVID-19 from viral pneumonia on chest CT. METHODS: A total of 219 patients with both positive COVID-19 by RT-PCR and abnormal chest CT findings were retrospectively identified from 7 Chinese hospitals in Hunan Providence, China from January 6 to February 20, 2020. A total of 205 patients with positive Respiratory Pathogen Panel for viral pneumonia and CT findings consistent with or highly suspicious for pneumonia by original radiology interpretation within 7 days of each other were identified from Rhode Island Hospital in Providence, RI. Three Chinese radiologists blindly reviewed all chest CTs (n=424) to differentiate COVID-19 from viral pneumonia. A sample of 58 age-matched cases was randomly selected and evaluated by 4 U.S. radiologists in a similar fashion. Different CT features were recorded and compared between the two groups. RESULTS: For all chest CTs, three Chinese radiologists correctly differentiated COVID-19 from non-COVID-19 pneumonia 83% (350/424), 80% (338/424), and 60% (255/424) of the time, respectively. The seven radiologists had sensitivities of 80%, 67%, 97%, 93%, 83%, 73% and 70% and specificities of 100%, 93%, 7%, 100%, 93%, 93%, 100%. Compared to non-COVID-19 pneumonia, COVID-19 pneumonia was more likely to have a peripheral distribution (80% vs. 57%, p<0.001), ground-glass opacity (91% vs. 68%, p<0.001), fine reticular opacity (56% vs. 22%, p<0.001), and vascular thickening (59% vs. 22%, p<0.001), but less likely to have a central+peripheral distribution (14.% vs. 35%, p<0.001), pleural effusion (4.1 vs. 39%, p<0.001) and lymphadenopathy (2.7% vs. 10.2%, p<0.001). CONCLUSION: Radiologists in China and the United States distinguished COVID-19 from viral pneumonia on chest CT with high specificity but moderate sensitivity. A translation of this abstract in Farsi is available in the supplement. - ترجمه چکیده این مقاله به فارسی، در ضمیمه موجود است.

Since the initial outbreak of Coronavirus disease-19 (COVID-19) from Wuhan, China in late December 2019 (1), there have been 87,137 confirmed cases and 2,873 reported deaths distributed across 60 countries as of March 1 st 2020 (2, 3) . China has had the majority of COVID 19 cases (92%) (3) . Patients infected with COVID-19 typically present with fever, cough, dyspnea, and muscle aches while imaging frequently reveals bilateral pneumonia (5) .

The standard diagnostic method being used is real-time polymerase chain reaction (RT-PCR) to detect viral nucleotides from specimens obtained by oropharyngeal swab, nasopharyngeal swab, bronchoalveolar lavage, or tracheal aspirate (6) . However, recent reports have revealed that RT-PCR has a sensitivity as low as 60-71% for detecting COVID-19 (5, 7, 8) , which can possibly be attributed to low viral load present in test specimens or laboratory error (7, 9) . These false negatives hinder quarantine efforts, necessitate repeat testing and has the potential to overload the current supply of testing kits and related infrastructure (8) . By contrast, chest CT has demonstrated about 56-98% sensitivity in detecting COVID-19 at initial presentation and can be helpful in rectifying false negatives obtained from RT-PCR during early stages of disease development (7, 8) . Chest CT of COVID-19 patients reveals areas of consolidation and ground-glass opacity (GGO) with bilateral peripheral involvement in multiple lobes progressing to "crazy-paving" patterns and consolidation. CT signs gradually improve beginning approximately 14 days post-symptom onset (10, 11, 12) .

Despite its high sensitivity in diagnosing COVID-19 in a screening population, chest CT had low specificity (25%) in a recent report of 1014 patients with COVID-19 (5) . Prior studies have not directly compared chest CT patterns of COVID-19 from viral pneumonia on chest CT. The purpose of this study was to assess the performance of United States (U.S.) and Chinese radiologists in differentiating COVID-19 from viral pneumonia on chest CT.

The institutional review board of all seven hospitals in Hunan Providence, China and Rhode Island Hospital from the U.S. approved this retrospective study and written informed consent was waived. A total of 256 patients with both positive COVID-19 by RT-PCR and chest CT imaging within two weeks were retrospectively identified from 7 Chinese hospitals in Hunan Providence, China from January 6 to February 20, 2020. The RT-PCR results were extracted from the patients' electronic medical records in the hospital information system (HIS). The RT-PCR assays were performed by using TaqMan RPP test within 7 days of each other. Then, the impression sections of these CT reports were reviewed by a research assistant (BH) and a radiologist (HXB) board-certified in general diagnostic radiology and interventional radiology with one year of practice experience to identify 205 cases with final CT impression being "consistent with" or "highly suspicious for" pneumonia. Our final cohort consisted of 424 patients. A diagram illustrating patient inclusion and exclusion is shown in Figure 1 . A diagram illustrating the final breakdown of RPP results is shown in Figure 2 .

Three Chinese radiologists, who were blinded to RT-PCR results, reviewed all chest CT images and scored each case as COVID-19, pneumonia of other etiology or neither. Fifty-eight cases age-matched between the COVID-19 and non-COVID-19 pneumonia groups were randomly selected from the entire cohort and evaluated in the same way by four U.S. radiologists. All identifying information was removed from the CT studies, which were shuffled and uploaded to 3D slicer for interpretation. Information on the radiologists regarding location of practice, years in practice, cardiothoracic imaging fellowship and COVID-10 specific training experience is shown in Table 1 .

Different CT features on the entire cohort were recorded by two Chinese radiologists in consensus. If consensus could not be reached, it was resolved by a senior radiologist (X.Z.) with more than 10 years of chest CT experience.

Continuous variables were expressed as medians and ranges, while categorical variables as counts and percentages. Metrics such as sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were calculated to evaluate the diagnostic performance of the radiologists. For the calculations, COVID-19 was considered positive result, while pneumonia of other etiology and neither were considered negative result. Exact binomial 95% confidence intervals were calculated for sensitivity, specificity, PPV, NPV, and accuracy using the epiR package in the R statistical computing language (R version 3.4.2, The R Foundation for Statistical Computing, Vienna, Austria; https://www.r-project.org). P values of <0.05 were considered statistically significant. Table 2 .

For the entire cohort, the accuracy of the three Chinese radiologists to identify COVID-19 from non-COVID-19 pneumonia was 83% (95 CI: 79-86%), 80% (95% CI: 76-83%), and 60% (95% CI: 55-65%), respectively (Table 3) . Sensitivity ranged from 72 to 94%, and specificity showed great variation (24 to 94%) ( Table 3 ).

In a randomly selected, age-matched cohort of 58 patients, the accuracy of four U.S. radiologists to differentiate COVID-19 from non-COVID-19 pneumonia was 97% (95% CI: 88-100%), 88% (95 CI: 77-I n P r e s s 95%), 83% (95% CI: 71-91%), and 84% (95% CI: 73-93%), respectively (Table 4 ). Sensitivity ranged from 70 to 93%, and specificity ranged from 93 to 100% (Table 4 ).

Compared to non-COVID-19 pneumonia, COVID-19 pneumonia was more likely to have a peripheral distribution (80% vs. 57%, p<0.001), ground-glass opacity (91% vs. 68%, p<0.001), fine reticular opacity (56% vs. 22%, p<0.001), vascular thickening (59% vs. 22%, p<0.001) and reverse halo sign (11% vs. 1%, p=0.005), but less likely to have a central+peripheral distribution (14% vs. 35%, p<0.001), air bronchogram (14% vs. 23%, p=0.014), pleural thickening (15 vs. 33%, p<0.001), pleural effusion (4 vs. 39%, p<0.001) and lymphadenopathy (2.7% vs. 10.2%, p<0.001) ( Table 5 ). 

COVID-19 is spreading rapidly worldwide while present diagnostic methods for identifying the virus have limitations. RT-PCR testing has low sensitivity early on in the disease course. Although chest CT has high sensitivity, it has low specificity (7, 11) . This low specificity may stem from the fact that it is difficult to distinguish COVID-19 findings from findings of other disease on chest CT (5). To optimize patient management, medical care, and disease control it is important to determine the efficacy of chest CT in distinguishing COVID-19 from pneumonia of other etiologies by radiologists. The study conducted herein revealed that radiologists are capable of distinguishing COVID-19 from viral pneumonia on chest CT with high specificity but moderate sensitivity.

Healthcare providers being able to reliably differentiate COVID-19 from other causes of pneumonia on chest CT would benefit diagnostic workup of the disease by compensating for the poor sensitivity of RT-I n P r e s s PCR, particularly during early disease stages. Although chest CT has demonstrated high sensitivity relative to RT-PCR testing for COVID-19 diagnosis, it may not reveal distinct patterns for COVID-19 in all cases. This can make it hard to distinguish COVID-19 from other causes of viral pneumonia. For example, influenza and COVID-19 both demonstrate GGO and consolidation on chest CT (14) . Introducing the possibility of pathologies with similar chest CT findings to those of COVID-19 ultimately complicates the differential diagnosis. Our study is significant because it demonstrates that radiologists are capable of distinguishing COVID-19 from other etiology of pneumonia on chest CT with high specificity. This suggests that there is potential that if the differential diagnosis is between COVID-19 and non-COVID-19 pneumonia, a negative diagnosis of COVID-19 by radiologists on chest CT may be good enough to exclude patients from having the disease with fairly good certainty. Our analysis of specific cases where radiologists were wrong reveals that the mistakes were made when the COVID-19 chest CT findings are either subtle (likely reflecting early time in the disease process) (Figures 3C and 3D ) or when COVID-19 has atypical chest CT findings ( Figures 3E, 3F , 3G, and 3H). It is worth noting that non-COVID-19 pneumonia can also have typical appearance of COVID-19 ( Figures 3A and 3B ). This poses a dilemma because mandated quarantine for all suspected cases can put significant strain on medical infrastructure, healthcare providers, and the lives of patients, but may need to be followed as a necessary precaution due to variation in presentation with timing of disease and atypical findings on chest CT. Future direction includes development of an artificial intelligence classifier that can further augment radiologist performance in combination with clinical information.

Our study has several limitations. First, the cohort size was small, especially when it comes to cases reviewed by U.S. radiologists. There is selection bias associated with our screening strategy as well. It remains unclear if diagnostic outcomes would improve in a more well-balanced and larger-scale prospective study of similar design. It is also noteworthy that the U.S. radiologists in this study had minimal training specific to diagnosing COVID-19 and that Chinese radiologists practiced in an area with I n P r e s s relatively low prevalence of the disease. It is possible that Chinese radiologists working near the epicenter of the disease with a higher degree of experience specific to COVID-19 would have performed significantly better than either group in the present study. In addition, although RPP test and chest CT findings within 7 days of symptom presentation were used to enrich our "pneumonia of other etiology" cohort with viral pneumonia cases, the cause and effect relationship is not 100%. Thus, it is possible that some of the selected patients had mixed viral and bacterial pneumonia or other diseases entirely.

Finally, the radiologists were not given clinical information during the evaluation, which could have further improved their performance.

As more research is done, information may be gathered by providers to make this differential more facile to navigate. Until that point, it is recommended that individuals with signs of pneumonia on chest CT be quarantined while RT-PCR testing is performed in conjunction with a thorough medical evaluation including travel history and disease contacts in order to make an accurate COVID-19 diagnosis and prevent disease spread. In conclusion, radiologists had high specificity but moderate sensitivity in distinguishing COVID-19 from viral pneumonia on chest CT. I n P r e s s (16) has been proven to be a highly sensitive and specific multiplex assay for respiratory pathogen detection. In a multicentric study, the positive percent agreement values (equivalent to sensitivity when a perfect reference method is unavailable) ranged from 85.1% to 95.1% and the negative percent agreement values (equivalent to specificity) ranged from 99.5% to 99.8% when compared to another well-established RP panel from BioFire (Salt Lack City, UT) (17).

A novel coronavirus from patients with pneumonia in China

COVID-19) Locations with Confirmed COVID-19 Cases

secondmeeting-of-the-international-health-regulations-(2005)-emergency-committee-regarding-theoutbreak-of-novel-coronavirus-(2019-ncov)

Correlation of Chest CT and RT-PCR Testing in Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases

Center for Disease Control and Prevention. Interim Guidelines for Collecting, Handling, and Testing Clinical Specimens from Persons Under Investigation (PUIs) for Coronavirus Disease

Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR

Essentials for Radiologists on COVID-19: An Update-Radiology Scientific Expert Panel

Chest CT for Typical 2019-nCoV Pneumonia: Relationship to Negative RT-PCR Testing

Chest CT Findings in Coronavirus Disease-19 (COVID-10): Relationship to Duration of Infection

Time Course of Lung Changes of Chest CT During Recovery From

CT Imaging Features of 2019 Novel Coronavirus

A Perspective from China

In: COVID-19 Pneumonia Diagnosis and Treatment Plan, trial version 6. Beijing, CN: National Health Commission of the PRC

Multicenter Evaluation of the ePlex Respiratory Pathogen Panel for the Detection of Viral and Bacterial Respiratory Tract Pathogens in Nasopharyngeal Swabs