key: cord-1015619-p2cw15qq
authors: Byrne, Danielle; O’Neill, Siobhan B.; Müller, Nestor L.; Silva Müller, C. Isabela; Walsh, John P.; Jalal, Sabeena; Parker, William; Bilawichm, Ana-Maria; Nicolaou, Savvas
title: RSNA Expert Consensus Statement on Reporting Chest CT Findings Related to COVID-19: Interobserver Agreement Between Chest Radiologists
date: 2020-07-02
journal: Can Assoc Radiol J
DOI: 10.1177/0846537120938328
sha: ef90ca11c9f3a79636cbfda164c938d6d9378c8e
doc_id: 1015619
cord_uid: p2cw15qq

PURPOSE: To assess the interobserver variability between chest radiologists in the interpretation of the Radiological Society of North America (RSNA) expert consensus statement reporting guidelines in patients with suspected coronavirus disease 2019 (COVID-19) pneumonia in a setting with limited reverse transcription polymerase chain reaction testing availability. METHODS: Chest computed tomography (CT) studies in 303 consecutive patients with suspected COVID-19 were reviewed by 3 fellowship-trained chest radiologists. Cases were assigned an impression of typical, indeterminate, atypical, or negative for COVID-19 pneumonia according to the RSNA expert consensus statement reporting guidelines, and interobserver analysis was performed. Objective CT features associated with COVID-19 pneumonia and distribution of findings were recorded. RESULTS: The Fleiss kappa for all observers was almost perfect for typical (0.815), atypical (0.806), and negative (0.962) COVID-19 appearances (P < .0001) and substantial (0.636) for indeterminate COVID-19 appearance (P < .0001). Using Cramer V analysis, there were very strong correlations between all radiologists’ interpretations, statistically significant for all (typical, indeterminate, atypical, and negative) COVID-19 appearances (P < .001). Objective CT imaging findings were recorded in similar percentages of typical cases by all observers. CONCLUSION: The RSNA expert consensus statement on reporting chest CT findings related to COVID-19 demonstrates substantial to almost perfect interobserver agreement among chest radiologists in a relatively large cohort of patients with clinically suspected COVID-19. It therefore serves as a reliable reference framework for radiologists to accurately communicate their level of suspicion based on the presence of evidence-based objective findings.

Coronavirus disease 2019 (COVID-19) is a novel infectious disease caused by severe acute respiratory syndrome coronavirus 2. It was first identified in the human population in December 2019 1 and spread rapidly around the world reaching pandemic status in March 2020. 2 Reverse transcription polymerase chain reaction (RT-PCR) of respiratory specimens is the most widely used method for diagnosing COVID-19. 3, 4 However, in many clinical settings, RT-PCR testing may not be readily available, be limited to hospitalized patients, or may be initially falsely negative. 5 Furthermore, patients with mild respiratory symptoms may be reluctant to travel to a PCR testing site or hospital. 5 Although most radiological societies and professional organizations have recommended against performing chest computed tomography (CT) in the workup of patients with suspected COVID-19, in resource-poor countries where RT-PCR is not widely available, a tentative diagnosis or exclusion is often based on clinical and CT findings alone. Additionally, several retrospective studies have shown that CT has greater sensitivity (86%-98%) and lower false-negative rate than RT-PCR. [6] [7] [8] [9] The sensitivity of nasopharyngeal swabs ranges from 42% to 71%, 6, 10 with false negatives observed more frequently early in the course of the disease. 11, 12 This has led to the widespread use of chest CT in the workup of patients with suspected COVID-19 particularly in the setting of negative RT-PCR and high clinical suspicion. 6, 9 COVID-19 results in a spectrum of chest CT manifestations, which evolve over time ranging from peripheral predominant ground-glass opacities to an organizing pneumonia reaction pattern, 13 with additional findings including crazy-paving and more diffuse ground glass, thus overlapping with imaging features of a variety of other disease processes including other infections, drug reaction, and inhalational exposure. [14] [15] [16] As there are a number of etiologies with imaging findings that overlap with those of COVID-19, the inclusion of COVID-19 in the differential diagnosis of the radiology report may lead to unwarranted anxiety among referring physicians and patients. Optimal interpretation and reporting of the chest CT and clear, unambiguous communication with the referring physician are essential for optimal patient care as well as community and health care worker safety during the COVID-19 pandemic.

An expert consensus statement on reporting chest CT findings related to COVID-19, endorsed by the Society of Thoracic Radiology, the American College of Radiology, and the Radiological Society of North America (RSNA), was published to assist radiologists in recognizing and describing lung imaging findings in a standardized manner in patients under investigation for COVID-19 pneumonia and provide clarity in communication with other health care providers. 10 The purpose of this study was to assess the interobserver variability between chest radiologists in the evaluation of CT scans in a cohort of patients with respiratory symptoms and suspected COVID-19 pneumonia using the RSNA expert consensus reporting guidelines in a setting with limited RT-PCR testing availability.

Approval was obtained from the institutional review board, and the need for informed consent was waived for this retrospective study. A total of 303 consecutive patients (158 males and 145 females) with a median age of 49 years (range, 18-96) who presented with respiratory symptoms were suspected to have COVID-19 infection and underwent CT chest imaging between March 2, 2020, and March 16, 2020, were included. Due to the resource-limited nature of the hospital, reliable RT-PCR data were not available.

Chest CT studies were acquired on a 32-slice single-source CT scanner (Siemens Somatom Scope) using a standardized CT technique: 110 kVp, 345 mA max, pitch of 1.5, 0.6 seconds rotation time, and 1 mm scan thickness. Inspiratory phase CT chest examinations were acquired with patients in the supine position without administration of intravenous contrast.

Computed tomography images were extracted from the picture archiving and communication system, anonymized, and imported onto a secure browser-based viewing system for CT scans (MD.ai, available at https://www.md.ai/). The software displayed cross-sectional CT images only with soft tissue and lung kernel and allowed for dynamic scrolling, window width-window level adjustment, panning, and zoom. Three fellowshiptrained chest radiologists, each with >10 years experience reading chest CT examinations (S.B.O.N., A.-M.B., and C.I.S.M.) performed diagnostic interpretation on all included CT studies. Observers, blinded to all patient information including RT-PCR result, recorded the presence or absence and distribution of typical COVID-19 imaging features (see Table 1 ) and scored each study according to the RSNA expert consensus reporting guidelines as to whether a study was typical, indeterminate, atypical, or negative for the presence of COVID-19 pneumonia. 10 

Statistical analysis was performed using SPSS statistics version 25 (IBM). To quantify interobserver agreement, Cramer V and Fleiss kappa were determined across observers. Kappa values were obtained by reporting scores of each observer (according to the RSNA expert consensus statement on reporting chest CT findings related to to the median of the other 2 observers. Interobserver agreement was considered slight for a kappa value of 0.01 to 0.20, fair for 0.21 to 0.40, moderate for 0.41 to 0.60, substantial for 0.61 to 0.80, and almost perfect for 0.81 to 1.00. 17 The result of a Cramer's V-test lies between 0 and 1 and is interpreted as following: 0, no association; 0.05 to 0.1, weak; 0.1 to 0.15, moderate; 0.15 to 0.25, strong; >0.25, very strong.

The Fleiss kappa for all observers was almost perfect (0.815) for typical COVID-19 appearance (P < .0001), substantial (0.636) for indeterminate COVID-19 appearance (P < .0001), and almost perfect for atypical (0.806) and negative (0.962) COVID-19 appearances (see Table 2 ). Using Cramer V analysis, there were very strong correlations between all radiologists' interpretations, statistically significant for all (typical, indeterminate, atypical, and negative) COVID-19 appearances (P < .001; see Table 3 ). Of the cases with overall typical imaging appearance for COVID-19, specific imaging findings including ground glass, consolidation, and crazy-paving were recorded in similar percentages of cases by the observers (see Table 1 ). A posterior predominance of distribution was observed in all typical cases (100%). Examples of typical COVID-19 CT appearances are presented in Figures 1 to 6 .

COVID-19 may result in a wide spectrum of CT imaging findings, which in isolation or combination can be of variable significance. Findings may vary depending on the severity of disease, time point in the infection, or the presence of background or coexistent lung disease. As a novel infection, by definition, experience of interpreting radiologists is limited, with continuously evolving described imaging features requiring time to establish diagnostic confidence. Given the potential implications of a COVID-19 diagnosis to the patient, exposed health care workers, and the wider community, concise and unambiguous radiology reporting is of paramount importance. The RSNA expert consensus statement on reporting chest CT findings related to COVID-19 was published to facilitate reporting of CT imaging studies acquired during the COVID-19 pandemic. 10 The statement's main purpose was to provide radiologists with standardized reporting language to apply to patients under investigation for COVID-19 and to allow them to assign an evidence-based impression of their level of suspicion for COVID-19 infection, thus facilitating clear communication with referring physicians. 10 Currently, there is a paucity of data on the initial experience utilizing the RSNA expert consensus statement guidelines in clinical practice.

Our results show that there is overall good agreement for typical, indeterminate, atypical, and negative imaging appearances according to the RSNA expert consensus statement reporting guidelines among expert chest radiologists in suspected COVID-19 cases where RT-PCR was largely unavailable. The RSNA expert consensus statement guidelines provide both standardized language and impression categories, which will likely reduce reporting variability and improve clarity in report interpretation among referring physicians. The CO-RADS (COVID-19 Reporting and Data System) is an additional recently described categorical reporting system to assess the degree of lung involvement by COVID-19 on chest CT in patients with moderate to severe symptoms with categories 1 to 5 according to increasing suspicion on CT. 18 The CO-RADS system was not assessed in this study but reports very good performance in predicting COVID-19 patients with substantial interobserver agreement on initial clinical application. 18 Commonly reported imaging features with greater specificity for COVID-19 pneumonia include peripheral and bilateral ground glass and/or multifocal ground glass with rounded morphology with or without consolidation or visible intralobular lines (''crazy paving'') and features of organizing pneumonia including the reverse halo sign. 10, 13, 19 Ground-glass opacities and consolidation were recorded in a high percentage of typical COVID-19 cases among observers in our study with good agreement (see Table 1 ), emphasizing the importance of identifying these features on CT in patients with suspected COVID-19 where RT-PCR results may be delayed or unavailable. Posterior predominance of distribution 20 has been reported in COVID-19 patients and was almost uniformly observed in the typical cases in this study. Bronchial dilatation in affected areas 21 is less commonly seen in this study in line with recent literature. 22 Computed tomography may be normal early on and subsequently depict an acute lung injury response to an infectious insult, with peripheral predominant multifocal ground glass with or without consolidation followed by organizing pneumonia reaction pattern. 23 Due to the highly infectious nature of COVID-19 with estimated reproduction number (R0) of 2 to 2.53, rapid and accurate diagnostic methods are needed to identify, isolate, and treat the patients in a timely fashion, which could reduce mortality rates and the risk of public contamination. In cases where RT-PCR is initially negative, typical findings on CT may encourage maintenance of infection control mechanisms and prompt repeat testing. 8, 24, 25 In cases of high clinical suspicion but negative RT-PCR, a combination of repeated swab tests and CT imaging may be useful to reach the diagnosis and assist management. 9, 26 With many countries beginning to ease lockdown restrictions and health care systems returning to near prepandemic levels of operation, it is important that radiologists remain vigilant to the typical COVID-19 CT imaging findings with the almost inevitable anticipated second wave of the pandemic. Such typical CT findings should be kept in mind for the foreseeable future, particularly in the outpatient CT reporting setting, as it is known that asymptomatic patients with COVID-19 can have abnormalities on CT. 27 Our study has several limitations. Reverse transcription polymerase chain reaction was unavailable in the majority of cases and therefore we were only able to assess interobserver agreement according to the RSNA expert consensus statement guidelines in suspected COVID-19 cases. Furthermore, findings could not be correlated with timing of onset of infection, severity of symptoms, or patient outcome due to lack of available clinical data.

The role of chest CT imaging in the era of COVID-19 continues to evolve with a particularly important role in cases with high clinical suspicion but negative or unavailable RT-PCR. We have shown that the RSNA consensus statement on reporting chest CT findings related to COVID-19 has good interobserver agreement among expert chest radiologists in a relatively large cohort of patients with suspected COVID-19 and unavailable RT-PCR testing. The guidelines provide a framework to which radiologists can easily refer and language which can easily be applied in daily clinical practice in order to accurately communicate their level of suspicion for COVID-19 based on the presence of evidence-based objective imaging findings.

Authors' Note D. Byrne and S. B. O' Neill contributed equally to this manuscript.

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Danielle Byrne, FFR (RCSI) https://orcid.org/0000-0002-8168 -2159 C. Isabela Silva Müller, MD, PhD https://orcid.org/0000-0003 -2422-2735 Figure 6 . Examples of peripheral predominant ground-glass opacities with superimposed intralobular and interlobular septal thickening (short arrows) consistent with crazy paving (A-C). D, Magnified image of the left lung in (C) more clearly demonstrates intralobular and interlobular septal thickening (long arrow) superimposed on peripheral ground-glass opacification (arrowhead).

Clinical features of patients infected with 2019 novel coronavirus in Wuhan

World Health Organization. WHO Director-General's opening remarks at the media briefing on COVID-19-11

ed: World Health Organization

World Health Organization. Laboratory testing for 2019 novel coronavirus (2019-nCoV) in suspected human cases

Diagnosing COVID-19: the disease and tools for detection

Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases

Clinical characteristics of coronavirus disease 2019 in China

Sensitivity of chest CT for COVID-19: comparison to RT-PCR

Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing

Radiological Society of North America Expert Consensus Statement on Reporting Chest CT Findings Related to COVID-19. Endorsed by the Society of Thoracic Radiology, the American College of Radiology, and RSNA

Computed tomographic imaging of 3 patients with coronavirus disease 2019 pneumonia with negative virus real-time reverse-transcription polymerase chain reaction test

COVID-19 testing: the threat of false-negative results

Coronavirus disease 2019 (COVID-19): a systematic review of imaging findings in 919 patients

Imaging of pulmonary viral pneumonia

Radiologic, pathologic, clinical, and physiologic findings of electronic cigarette or vaping product use-associated lung injury (EVALI): evolving knowledge and remaining questions

Drug-induced lung disease: high-resolution CT and histological findings

The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability

CO-RADS-a categorical CT assessment scheme for patients with suspected COVID-19: definition and evaluation

Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT

Emerging 2019 novel coronavirus (2019-nCoV) pneumonia

Chest CT findings in cases from the cruise ship ''Diamond Princess'' with coronavirus disease 2019 (COVID-19)

Coronavirus disease 2019 (COVID-19): role of chest CT in diagnosis and management

Chest CT findings in coronavirus disease-19 (COVID-19): relationship to duration of infection

SARS-CoV-2 infection in a 76-year-old man with negative results for nasopharyngeal swabs and possible nosocomial transmission

Diagnosis of the coronavirus disease (COVID-19): rRT-PCR or CT?

Use of chest CT in combination with negative RT-PCR assay for the 2019 novel coronavirus but high clinical suspicion

CT imaging and clinical course of asymptomatic cases with COVID-19 pneumonia at admission in Wuhan