key: cord-0966974-2wcaqlo6 authors: Yates, Andrew; Dempsey, Philip J.; Vencken, Sebastian; MacMahon, Peter J.; Hutchinson, Barry D. title: Structured Reporting in Portable Chest Radiographs: An Essential Tool in the Diagnosis of COVID-19 date: 2020-11-11 journal: Eur J Radiol DOI: 10.1016/j.ejrad.2020.109414 sha: a9486f014287dd47f7884961543c735a145c1afb doc_id: 966974 cord_uid: 2wcaqlo6 PURPOSE: To date, the majority of chest imaging studies in COVID-19 pneumonia have focused on CT. Evidence for the utility of chest radiographs (CXR) in this population is less robust. Our objectives were to develop a systematic approach for reporting likelihood of COVID-19 pneumonia on chest CXR, to measure the interobserver variability of this approach and to evaluate the diagnostic performance of CXR compared to real-time reverse transcription polymerase chain reaction (RT-PCR). METHOD: Retrospective review of patients suspected of having COVID-19 pneumonia who attended our emergency department and underwent both CXR and a RT-PCR were included. Two radiologists reviewed the CXRs, blind to the RT-PCR, and classified them according to a structured reporting template with five categories (Characteristic, High Suspicion, Indeterminate, Unlikely and Normal) which we devised. For analysis of diagnostic accuracy, Characteristic and High Suspicion CXRs were considered positive and the remaining categories negative. Concordance between the two assessors was also assessed. RESULTS: Of 582 patients (51 +/- 20 years), 143/582 (24.6%) had a positive RT-PCR. The absolute concordance between the two assessors was 71.1% (414/582) with a Fleiss-Cohen-weighted Cohen’s κ of 0.81 (95% confidence interval, 0.78-0.85). A patient with a positive CXR had an 88% (95% CI 80-96%) probability of having a positive RT-PCR during a period of high incidence, early in the COVID-19 pandemic. CONCLUSION: Using a structured approach, a positive CXR had a high likelihood of predicting a positive RT-PCR, with good interrater reliability. CXR can be useful in identifying new cases of COVID-19. The Coronavirus Disease 2019 pandemic is an ongoing global health emergency with persistently high mortality [1] . The Fleischner Society have provided guidance on the role of chest imaging in patients with suspected COVID-19, but they have not issued explicit guidance on the best imaging modality [2] . The majority of chest imaging studies in COVID-19 to date have focused on CT and have identified consistent patterns in COVID-19 pneumonia including bilateral, peripheral lower J o u r n a l P r e -p r o o f lobe predominant ground glass opacities and consolidation as well as a low incidence of pleural effusions [3] [4] [5] [6] [7] [8] [9] . Evidence for the utility of chest radiographs in the initial diagnosis of COVID pneumonia is much less robust and their diagnostic value has been questioned [4] . A prior meta-analysis of the diagnostic performance of CT and RT-PCR in COVID-19 found that CT had a pooled sensitivity of 94% and a pooled specificity of 37% [10] , however the disease prevalence was not always considered in these studies [11, 12] . A pattern of peripheral opacities has been shown on both CXR and CT [13, 14] . We have anecdotally found this pattern to have a high correlation with a positive real time reverse transcription-polymerase chain reaction (RT-PCR) diagnosis of severe acute respiratory syndrome Coronavirus 2 (SARS-CoV-2) infection. A number of standardized assessments of pulmonary involvement of COVID-19 have been proposed [9, 15, 16] . These are designed to rank the suspicion of pulmonary involvement of COVID-19 in patients who have signs and symptoms suggestive of COVID-19 pneumonia and are intended to be interpreted with clinical and laboratory findings. The Dutch Radiological Society have proposed the COVID-19 Reporting and Data System (CO-RADS), which is modelled on established standardized reporting systems such as BI-RADS and Lung-RADS, and have demonstrated moderate to substantial interobserver agreement in application of these categories between multiple readers [16] . Expert consensus opinion has advised the use of structured reporting in CT when COVID-19 pneumonia is suspected in order to standardize the description of lung findings as well as to decrease reporting variability [15] . Available standardized assessments for pulmonary involvement in CXR are less descriptive than those that have been established for CT [17] . These uncertainties surrounding the role and clinical utility of chest radiography in the early evaluation of patients with suspected COVID-19 pneumonia, and how structured reporting can be applied to CXR, have prompted us to examine this issue in more detail. The purposes of this study were multifold: 1) to construct a detailed description of a structured approach to reporting CXR in patients with J o u r n a l P r e -p r o o f suspected COVID-19 pneumonia, 2) to measure interobserver agreement in the application of this approach and 3) to evaluate the diagnostic performance of chest radiography compared to RT-PCR. The study was performed in a 600-bed tertiary referral hospital with an average annual emergency department attendance of 50,000. The hospital is the national referral centre for Extracorporeal Membrane Oxygenation (ECMO) and contains the National Isolation Unit for infectious diseases. The study was approved by the National Research Ethics Committee (NREC) for COVID-19 and consent was waived. All patients who attended the emergency department from March 10th -April 4th 2020, who were suspected of COVID-19 and underwent both a portable chest radiograph (CXR) and a RT-PCR test for presence of SARS-CoV-2 on initial assessment, were included for analysis. Patients were suspected of COVID-19 if they presented with symptoms of an acute respiratory infection, (including fever, shortness of breath and cough) or a fever of unknown cause. Fever was defined as a temperature >38.0°C. RT-PCR of oropharyngeal and nasopharyngeal swab was completed as part of the initial assessment of all patients suspected of having COVID-19. Serial RT-PCR was completed if the initial result was negative, at intervals of 1 day or more, if deemed required by clinicians. For patients with multiple RT-PCR assays, a positive result on an assay within 7 days of the initial swab was adopted as confirmation of diagnosis. Patients who had a positive RT-PCR >7 days after their initial assay were excluded from analysis. Patients underwent a CXR if they were suspected of COVID-19 and had moderate to severe symptoms requiring admission for treatment. J o u r n a l P r e -p r o o f All CXRs were acquired as computer or digital radiographs following local protocols. CXRs were acquired in anteroposterior (AP) projection using portable x-ray units in isolation units. Patients were positioned erect, semi-erect or supine for radiograph acquisition depending on the patients' clinical state. Radiographs were uploaded to the hospital picture archiving and communication system. Two radiologists (BH with 8 years and PMM with 13 years of experience in interpreting chest radiographs) retrospectively reviewed the baseline CXR of 592 patients, blinded to the RT-PCR result, initial CXR report and subsequent radiological tests. The clinical symptoms, prior radiological tests and prior biochemical test results were available to the radiologists to simulate real-life practice. The two radiologists categorised each CXR as Normal, Unlikely, Indeterminate, High-Suspicion or Characteristic using the reporting template described below (Figure 1 ). These initial categorisations were used to calculate interobserver agreement. Where there was a discordance in initial categorisation of the CXR the result was decided by consensus. Final categorisations were compared to the RT-PCR to assess diagnostic accuracy. Prior to the commencement of this study, a structured approach to the interpretation of CXR in suspected cases of COVID-19 was adopted by many in our radiology department based on available consensus [9, [15] [16] [17] . We decided to build on these resources to make a more prescriptive template due to anecdotal lack of consensus on how to categorise radiographs. Our template demonstrates a number of similarities to the recently published CO-RADS structured approach for CT [16] . We have A Characteristic CXR must have bilateral subpleural opacities that are relatively symmetrical. They should be of large volume (subjectively >20% of lung) and there should be no doubt that the opacities are not artefactual and related to the chest wall. A High Suspicion CXR must have either unilateral subpleural opacity or bilateral large volume patchy or ill-defined opacities with a non-upper lobe predominance. The bilateral opacities must be nonapical predominant and not of a perihilar distribution. An Indeterminate CXR may include any of the following elements: unilateral large volume patchy or multilobar opacities; unilateral or bilateral small volume opacity (<20% of lung); suspected superimposed opacity overlying significant underlying parenchymal disease which may limit evaluation for acute disease; difficulty distinguishing a peripheral opacity from an artefact related to the chest wall. An Unlikely CXR has findings that are not suggestive of COVID-19 including pleural effusion, heart failure pattern, upper lobe predominant opacities with involvement of the lower zones, pneumothorax and lobar pneumonia pattern (dense, well-defined consolidation or small volume illdefined opacity clearly confined to single). A Normal CXR must have either normal lungs on chest radiograph or definitively no interval change if there is a long-standing chronic finding such as linear scarring, goitre, healed rib fractures, cardiomegaly with chronic congestion, chronic airways inflammation or hyperinflation). Patient demographics (age, gender) and RT-PCR results were extracted from patients' electronic records. For each level of the radiological assessment, concordance between the two assessors was J o u r n a l P r e -p r o o f evaluated against the consensus diagnosis and calculated as percentage positive agreement (both assessors concord with consensus diagnosis) and percentage negative agreement (both assessors discord with consensus). Overall inter-rater reliability was expressed as Fleiss-Cohen-weighted Cohen's κ. To assess the performance of radiological assessments to diagnose RT-PCR-confirmed COVID-19, the probability of a positive RT-PCR for each level of the radiological assessment was calculated and the overall diagnostic performance was expressed as the C-index. Strength of association between methods was summarised as the Somers' D statistic. As a supplementary analysis, using the RT-PCR result as a reference, the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of baseline portable chest radiography were calculated. Characteristic and High Suspicion chest radiographs were considered positive. Chest radiographs classified as Indeterminate, Unlikely and Normal were considered negative. 95% confidence intervals were provided. Statistical analysis was performed using R v3.6.5 (R Core Team). Analysis code can be found in Appendix 1. 590 patients attended the emergency department that were initially suspected of COVID-19 underwent RT-PCR and a baseline CXR during the study period. Eight patients had a positive RT-PCR >7 days after their initial assay and were excluded from analysis. Table 1 as Highly Suspicion and 17/143 (11.9%) as Characteristic. In both RT-PCR positive and negative groups RT-PCR testing and CXR occurred on the same day for the large majority of cases (for both groups: median = 0 days. IQR: 0 -0, with a range of the RT-PCR test occurring 2 days prior to the CXR to occurring 6 days afterward the CXR). The performance of structured CXR assessments to diagnose COVID-19 was evaluated. To produce reliable estimates, the Characteristic level, which had low case frequencies and no cases with an RT-PCR negative status, was merged with the High Suspicion level to create a new level called High Suspicion-Characteristic. The probabilities of a patient having a positive RT-PCR status were calculated for each CXR assessment level and are shown in Table 2 Additional analysis was performed to assess the diagnostic performance of CXR as a binary diagnostic for COVID-19. CXRs categorised as Characteristic and High Suspicion were considered positive CXRs and CXRs classified as Indeterminate, Unlikely and Normal were considered negative CXRs for COVID- 19 . This binary CXR assessment had a sensitivity of 43% (95% CI 34-52%), specificity of 98% (95% CI J o u r n a l P r e -p r o o f 96-99%), positive predictive value of 88% (95% CI 77-95%), and negative predictive value of 85% (95% CI 81-88%). These show that the accuracy of baseline CXR in the diagnosis of COVID-19 is moderate to good, with a high specificity and a low sensitivity (Table 4 ). The majority of chest imaging studies in COVID-19 to date have focused on CT and have identified consistent imaging findings. This study found that when using a structured template reporting system, CXRs demonstrated a high positive predictive value and specificity of 88% (95% CI 80-96%) and 98% (95% CI 96-99%) respectively making it a useful tool in the diagnosis of COVID-19. This approach had a high concordance between two assessors with an absolute concordance of 71.1% (414/582) and a Fleiss-Cohen-weighted Cohen's κ of 0.81 (95% confidence interval, 0.78-0.85). Overall, our agreement was higher than that seen in the CO-RADS study [16] but this study involved multiple readers across multiple sites which is likely to account for the difference. Numerous studies have described imaging findings in an entirely positive cohort [3, 5, 13, 18, 19 ] but we chose a mixed RT-PCR positive and negative cohort so that our findings might be more generalisable to daily practice. Previous studies have evaluated sensitivity and specificity of CT for the diagnosis of COVID-19 pneumonia by assessing for the presence of pulmonary findings that have been J o u r n a l P r e -p r o o f reported to be either commonly present or absent in the disease [6, 8, 18, 20] . However in the clinical setting, how likely the patient is to have COVID-19 given a positive or negative CXR is more useful than the sensitivity or specificity [19] . Sensitivity of CT has been repeatedly shown to be high, ranging from 93% to 98% with a pooled sensitivity of 94% on meta-analysis [10] . While false negatives in CT evaluation for COVID-19 are low, to date, CT has not shown a good ability to accurately rule in COVID-19 with pooled specificity of 37% and positive predictive value (PPV) ranging from 1.5% to 30.7% [5, 10, 14, 21] . In this study, we have found CXRs to have an almost inverse performance relative to CT with a high specificity of 98% but a low sensitivity of 43%. This result can be partially explained by the inherent insensitivity of chest radiographs compared with CT. Several studies have demonstrated that early airspace opacities seen on initial CT are significantly less conspicuous on contemporaneous chest radiographs [5, 13, 18] . Furthermore, sensitivity and specificity are statistics based around positive and negative results rather than levels of confidence. By designating High Suspicion pattern and Characteristic pattern as positive and the remaining groups negative, this approach assumes a high false-negative rate. This strategy was felt to be appropriate in the patient presenting to the Emergency Department with suspected COVID-19 pneumonia as chest imaging studies in general have been shown to be frequently normal in early stages of COVID-19 infection [3, 5, 19] . This is particularly true of patients with RT-PCR confirmed COVID-19 who have no abnormalities on their chest radiograph, which accounts for 41-58.3% of patients [6, 19] Similarly 37.7% of our RT-PCR-positive patients had normal initial radiography. One study has shown a sensitivity of 69% [13] for CXRs in COVID-19 but this was in a smaller cohort of patients all of whom had RT-PCR confirmed COVID-19 prior to imaging analysis, which may account for some of the difference. The Fleischner Society has advocated for imaging in all symptomatic patients suspected to have COVID-19 with moderate/severe clinical severity, with mild respiratory disease but with risk factors for progression, and for any patient with clinically worsening respiratory disease [15] . In general, portable chest radiography is predicted to be the most commonly utilised modality for the J o u r n a l P r e -p r o o f identification and follow up of lung abnormalities in suspected cases of COVID-19 pneumonia [14] . This is mainly because portable radiography avoids the need for patient movement through the hospital and simplifies cleaning procedures [22] . Turnaround times for RT-PCR for COVID-19 vary but can take over 24 hours. Given that there are often constraints on resources (such as personal protective equipment, ventilators and healthcare staff) it is important to optimise decision-making on whether to manage a patient in a COVID-specific pathway. We have found that, using a COVID-specific structured approach to interpretation, a CXR designated as High Suspicion or Characteristic in a patient presenting to the Emergency Department with clinically suspected COVID-19 pneumonia correlates with a very high likelihood of a positive RT-PCR. Confirmation with RT-PCR is always recommended [22] but we believe the positive predictive value and specificity of a positive chest radiograph merit a presumptive diagnosis of COVID-19 until the RT-PCR result becomes available. Furthermore, we would argue that if initial RT-PCR is negative, in those with a positive CXR, particularly those with a Characteristic appearance, that precautions should still be maintained and RT-PCR repeated. We have demonstrated very good agreement in the application of this structured approach in a single institution and feel that this practice is generalisable to other institutions. CXRs that are designated as Normal, Unlikely or Indeterminate are less useful in the initial patient evaluation but awareness of this finding is essential for clinicians. Given that CT of the chest has shown high sensitivity but low specificity, radiographs and CT may be complementary tests in diagnostically difficult cases. Lastly, CXRs have been shown to be useful in prognostication and in tracking disease progression [13, 23] . The primary limitation of this study is the validation of our structured approach to radiograph interpretation. We have demonstrated very good concordance between two senior radiologists within the same institution but we have not demonstrated agreement between radiologists working in different environments, with different levels of experience and between radiologists and nonradiologist clinicians. We consider this study a proof-of-concept and hope to validate this approach on a wider scale in the future. Additionally, we have evaluated chest radiographs against COVID-19 RT-J o u r n a l P r e -p r o o f PCR positivity within 7 days but are aware that this may not reflect the true presence of the infection in our cohort and cannot accurately account for why some patients had more than one swab, however we do feel that this reflects everyday practice and this would not reduce our measured specificity or positive predictive value. We do not have a record of the timing of onset of symptoms compared with timing of radiograph and analysis of this may improve sensitivity. Lastly, we are aware that this study was carried out during a pandemic at a time of very high incidence of COVID-19 and many of the imaging findings will have a lower specificity at a time when this disease has a low incidence, however the Characteristic appearance has a narrow differential and could prove useful in identifying patients with COVID-19 in the future. A structured approach to chest radiographs provides a template to describe the changes likely attributed to COVID-19. This study found that by using a structured approach, a positive chest radiograph had a high likelihood of predicting a positive RT-PCR with good interrater reliability. Coronavirus Disease 2019 (COVID-19) Situation Report -107 The Role of Chest Imaging in Patient Management during the COVID-19 Pandemic: A Multinational Consensus Statement from the Fleischner Society Relationship to Duration of Infection Coronavirus Disease 2019 (COVID-19): A Systematic Review of Imaging Findings in 919 Patients Imaging Profile of the COVID-19 Infection: Radiologic Findings and Literature Review Chest CT Findings in Cases from the Cruise Ship "Diamond Princess Coronavirus Disease 2019: Initial Detection on Chest CT in a Retrospective Multicenter Study of 103 Chinese Subjects on behalf of the European Society of, I. the European Society of Thoracic, COVID-19 patients and the radiology department -advice from the European Society of Radiology (ESR) and the European Society of Thoracic Imaging (ESTI) Diagnostic Performance of CT and Reverse Transcriptase-Polymerase Chain Reaction for Coronavirus Disease 2019: A Meta-Analysis Chest CT accuracy in diagnosing COVID-19 during the peak of the Italian epidemic: A retrospective correlation with RT-PCR testing and analysis of discordant cases Imaging Publications in the COVID-19 Pandemic: Applying New Research Results to Clinical Practice Frequency and Distribution of Chest Radiographic Findings in COVID-19 Positive Patients Portable chest X-ray in coronavirus disease-19 (COVID-19): A pictorial review Radiological Society of North America Expert Consensus Statement on Reporting Chest CT Findings Related to COVID-19. Endorsed by the Society of Thoracic Radiology, the American College of Radiology, and RSNA CO-RADS -A categorical CT assessment scheme for patients with suspected COVID-19: definition and evaluation Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR Chest x-ray findings in 636 ambulatory patients with COVID-19 presenting to an urgent care center: a normal chest x-ray is no guarantee Correlation of Chest CT and RT-PCR Testing in Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases Extension of Coronavirus Disease 2019 (COVID-19) on Chest CT and Implications for Chest Radiograph Interpretation ACR Recommendations for the use of Chest Radiography and Computed Tomography (CT) for Suspected COVID-19 Infection Deep-Learning Approach The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.