key: cord-1012432-phvp9fl8
authors: Koc, Ali Murat; Altin, Levent; Acar, Turker; Ari, Alpay; Adibelli, Zehra Hilal
title: How did radiologists’ diagnostic performance has changed in COVID‐19 pneumonia: A single‐centre retrospective study
date: 2021-08-09
journal: Int J Clin Pract
DOI: 10.1111/ijcp.14693
sha: 8546bf1c615946de480ceb5f9fdc543366116fef
doc_id: 1012432
cord_uid: phvp9fl8

AIMS: Delay and false positivity in PCR test results have necessitated accurate chest CT reporting for the management of patients with COVID‐19–suspected symptoms. Pandemic related workload and level of experience on covid‐dedicated chest CT scans might have affected the diagnostic performance of on‐call radiologists. The aim of this study was to reveal the interpretation errors (IEs) in chest CT reports of COVID‐19–suspected patients admitted to the Emergency Room (ER). METHODS: Chest CT scans between March and June 2020 were re‐evaluated and compared with the former reports and PCR test results. CT scan results were classified into four groups. Parenchymal involvement ratios, radiology departments’ workload, COVID‐19–related educational activities have been examined. RESULTS: Out of 5721 Chest CT scans, 783 CTs belonging to 664 patients (340 female, 324 male) were included in this study. PCR test was positive in 398; negative in 385 cases. PCR positivity was found to be highest in “normal” and “typical for covid” groups whereas lowest in “atypical for covid” and “not covid” groups. 5%‐25% parenchymal involvement ratio was found in 84.2% of the cases. Regarding the number of chest CT scans performed, radiologists’ workload has found to be increased six‐folds. With the re‐evaluation, a total of 145 IEs (18.5%) have been found. IEs were mostly precipitated in the first two months (88.3%) and mostly in the “not covid” class (60%) regardless of PCR positivity. COVID‐19 and radiology entitled educational activities along with the ER admission rates within the first two months of the pandemic have seemed to be related to the decline of IEs within time. CONCLUSION: COVID‐19 pandemic made a great impact on radiology departments with an inevitable burden of daily chest CT reporting. This workload and concomitant factors have effects on diagnostic challenges in COVID‐19 pneumonia.

quick and accurate diagnosis. This, in turn, made necessary contributions to the knowledge of novel imaging findings of these diseases. [6] [7] [8] Since March 11, 2020 ; when the first case was seen in our country, CT examination was routinely performed on patients with suspicion of covid pneumonia together with the PCR testing. Regarding examples of increasing experience in the interpretation of the imaging findings in past crises, we think that our experience of CT findings in COVID-19 pneumonia might have increased with the number of cases in this timeframe. Besides, there may have been differences in the evaluation of CT examinations due to the increased workload in the peak admission days. In this study, it was aimed to find out the possible changes in the diagnostic performance level of the radiology department by retrospectively re-evaluating the reports of the chest computed tomography images of the patients who were admitted to the emergency room (ER) with the suspicion of COVID-19.

Chest computed tomography of patients who were admitted to our hospital's ER with suspicion of COVID-19 between March and June 2020 were enrolled. Among these, patients who have RT-PCR tests performed via nasopharyngeal and oropharyngeal swab specimens were selected. PCR tests had been finalised within 24 to 96 hours in the first days of the epidemic in our country. Patients whose CT scans were performed with a time gap longer than 72 hours with PCR testing were excluded from the study.

Ethics committee approval was received for this study both from the institutional Ethical Committee and The Ministry of Health.

Radiological assessment of patients included unenhanced Chest CT imaging with covid-dedicated scanning protocols in two scanners (128-MDCT Siemens Somatom Definition; 16-MDCT Toshiba Alexion): supine, end-inspiration acquisition; slice thickness, 1.0-1.5 mm; tube voltage, 120 kV; tube current, 200-300 mAs; pitch factor, 1.5; multiplanar reformations with mediastinal and lung parenchymal window settings.

CT images were retrospectively re-evaluated by three radiologists.

Typical and atypical chest CT findings related to COVID-19 pneumonia were evaluated separately by each radiologist blinded to the previously written reports. Multifocal ground-glass opacities (GGO), consolidation, GGO with superimposed consolidation, consolidation predominant pattern, crazy paving pattern, and melted sugar sign were considered as typical; pleural and/or pericardial effusion, cavity, pulmonary nodule, nodular pattern, lymphadenopathy, peribronchovascular distribution, halo and/or reverse halo sign, three-in-bud sign, bronchiectasis, airway secretions, pulmonary emphysema, pulmonary fibrosis, isolated pleural thickening and pneumothorax were considered as atypical findings for covid pneumonia. 9 Patients were categorised as "normal," "typical for covid," "atypical for covid" and "not covid" similarly with the previous structured reports. The "Not covid" condition was considered in cases where there are pathologies that are not listed under the aforementioned typical and atypical covid pneumonia findings; that is, mass, lobar consolidation.

Afterward, parenchymal involvement ratios were visually defined for "typical for covid" and "atypical for covid" groups; <25%, 26%-50%, 51%-75% and 76%-100%. In case of a conflict between evaluations, the decision was made with the consensus of three radiologists. In addition to this re-evaluation, a fourth radiologist searched for preliminary reports, final reports and radiology consultation notes of these patients. The point of this thoughtful research is to find out the very first on-call radiologist's comment on CT images.

Results of the re-evaluation were compared with previously written reports and also with PCR test results. In this way, the diagnostic performance of CT reports was aimed to be determined throughout the whole process from the first stage of the epidemic to the present.

The number of Thoracic CTs acquired for each 24-hour working period was recorded to point out the possible diagnostic differences on the high-intensity workdays. Also, the total number of reported COVID-19 pneumonia-suspected CTs were cumulatively calculated to find out the breakpoint of the departmental learning curve regarding accurate diagnosis. Schedules of in-department teaching activities, face-to-face meetings, and online meetings, conferences, webinars organised by the National Radiology Association were also noted. The possible positive effects levering the quality of CT reporting for covid-pneumonia were investigated.

• The radiologists role is crucial in interpretation of chest CT scans in cases of atypical pneumonia. 

Statistical analyses were performed using the SPSS software version 22. Distribution of clinical characteristics across groups of CT and RT-PCR results were presented with frequency tables. The chi-square test was used to compare these proportions in different groups. The chi-square test or Fisher's exact test was performed to test the significance of pairwise differences using Bonferroni correction to adjust for multiple comparisons. An overall %5 type-I error level was used to infer statistical significance.

In this cohort; a total of 5721 chest CT scans were found to have acquired at our institution during March, April and May 2020. Chest CT's were mostly requested in patients with symptoms of persistent cough, fever, history of a pulmonary nodule, mass, chest operation, trauma etc1478 patients were removed from the study due to the long latency of PCR results. A total of 3460 patients who did not have a PCR test at our institution were also removed. The remaining 783 Chest CT scans of 664 patients who have PCR test results were retrospectively included ( Figure 1 ). In total, 82 patients had multiple CT scans during their hospitalisation period. In total, 340 patients were female (51.2%) and 324 patients were male (48.8%). RT-PCR test was positive in 279; negative in 385 patients.

A total of 783 chest scans were re-evaluated with three separate radiologists blinded to the former CT scan reports and PCR test results. Parenchymal involvement analysis was executed via visual semi-quantitative analysis of the three radiologists. A significant amount of CT scans resulted in only mild (<25%) parenchymal involvement (n = 405, 84.2%). A total of 57 patients showed 25%-50% parenchymal involvement (11.85%). Only 19 patients showed a higher percentage (>50%) of parenchymal involvement (3.95%) ( Table 1) .

A total of 132 CT scans were classified as "normal" whereas 170 CT scans as "not covid." PCR test results were positive in nearly all of the "normal" group (n = 121, 91.7%); whereas negative in most of the "not-covid" group (n = 143, 84.1%). The remaining 481 CT scans were suspicious for COVID-19 pneumonia. PCR test results were positive in 59.2% of "typical for covid" classified scans, while negative in 70.3% of "atypical for covid" ones. The distribution of PCR test results among Chest CT subgroups has shown a statistically significant difference (P < .001). ( Table 2 ).

Comparison of re-evaluation with former documents related to the CT scans has shown that radiological evaluations of a total of 145 CT scans (18.5%) have changed over time. After this point, these diagnostic differences will be referred to as "interpretation errors" (IE). The number of CT scans, interpretation errors and their monthly distribution had calculated ( Figure 2) . Afterward, results of CT scans according to the aforementioned four classifications (normal, covid, typical for covid, atypical for covid and not covid) were correlated with the number of interpretation errors and PCR test results. The distribution of IEs among results of CT scans has shown a statistically significant difference (P < .001). (Table 2 ) Furthermore, it has been found that IEs were mostly precipitated in the "not covid" group regardless of PCR positivity (n = 87, 60%), ( Table 3 ). The "atypical for covid" group was found to be the second most common one that IEs have determined (n = 44, 30.34%). The ratio of IEs in "normal" and "typical for covid" groups constitute less than ten per cent of all (9.66%).

Radiologists' workload volume especially in on-call hours varies mostly depending on the ER admission rates. At our institution, primarily one radiologist at night shifts and one radiologist at working hours were responsible for emergency cases. The total number of chest CT scans reported has markedly increased up to 140 cases/ day.

Educational activities dedicated to the novel-coronavirus infection that took place within the radiology department and also within institutional panels, symposiums organised by the national F I G U R E 1 Study flow-chart radiological society, ministry of health and online meetings have been investigated with their dates (Appendix 1). It has been found that local/national educational activities mostly aggregated at the end of March and the first half of April.

Results of re-evaluation of chest CT scans were correlated with former documents and gold-standard PCR test results. IEs were correlated with the workload and educational activities. In this way, the learning curve in radiological diagnosis of COVID-19 pneumonia was tried to be established. It has been found that the number of chest CT scans increase to the top level at the beginning of April then gradually decreases. On the other hand, the distribution of the ratio of IEs to the number of chest CT scans among these three months has pointed out that the ratio of IEs gradually decreases more than half after April, from 20.2% to 11.18% (Figure 2 ).

In our study, we aimed to expose the radiologists' increasing experience on the evaluation of chest CT findings of COVID-19 pneumonia.

We have focused on some unobserved challenges on radiologists' in the COVID-19 pandemic to better understand the discrepancy in diagnostic performances.

Correlation of re-evaluated chest CT results with PCR tests has demonstrated that the PCR test was negative in 84.1% of the scans in the "not covid" group. 91.7% PCR-positivity was calculated in the cases labelled for "normal." This finding can be explained with the vigorous use of CT in the ER setting, even for the patients who have COVID-19 infection but lack pneumonia. Calculations with "typical for covid," "atypical for covid" and a combination of these two groups have resulted in 62.8% sensitivity and 40% specificity of chest CT according to PCR test results.

We have identified diagnostic differences in 18.5% of the chest CT scans in our cohort. The distribution of this IEs have shown a dramatic accumulation within the first 60 days of the pandemic in our country. After this point, the IEs decrease down to 11.18% ( Figure 2 ). Some reasons can explain the possible causes of these 1.5 million in our country. 5 We have re-evaluated the study cohort with a remarkable experience of ~17.000 chest CT scans reported in the past 9 months at our institution.

During the pandemic, the workload of radiologists has expeditiously increased parallel to the admissions with the suspicion of COVID-19. According to the COVID-19 guidelines published by the ministry of health (Appendix 2), chest CT scanning has become the secondary diagnostic tool after the gold standard PCR testing. To overcome the delay in diagnosis due to the PCR testing, clinicians were encouraged to use the CT scan as the quickest way to isolate/ hospitalise the COVID-19 infected patients. 10 Moreover, chest CT gained an important role in the clarification of PCR-false negative but clinically highly suspected cases. 11 Also as Saket D has stated, CT orders from ER have prominently risen due to the medical training focusing heavily on imaging. 12 These all have seemed to be contributed to the acquisition of 5721 chest CTs within these three months. group. IEs in the re-evaluated "not covid" group has mostly consisted of incorrect diagnosis for "atypical for covid." Similarly, analysis of re-evaluated "atypical covid" group have shown a marked incorrect diagnosis for "typical for covid." These findings showed that the radiological hallmarks of covid pneumonia were successfully identified but there was a lack of experience and knowledge, specifically about atypical radiological findings. This situation had probably created a tendency not to exclude the covid diagnosis, yet point out a suspicion with the statement of "atypical for covid" in CT reports.

Another issue to mention is that there are subspecialised academic and staff radiologists in our department. The COVID-19 pandemic has necessitated rapid responses from radiology departments to overcome the increased demand for diagnosis and management of covid pneumonia. Similar to most radiology departments' management procedures, nearly all radiologists were integrated into a team dedicated to COVID-19 diagnosis along with the partial cancellation of elective diagnostic/interventional procedures and outpatient diagnostic services in our department. 16 There is no doubt that the COVID-19 pandemic made a huge impact on radiology practices worldwide. An increasing level of experience over time promotes the performance of the radiology departments in the diagnosis of chest CT findings in COVID-19

patients. COVID-19-related admission rates and relevant changes in the daily workload, rotations of subspecialised personnel for urgent health care management are other factors affecting the diagnostic performance.

The authors declare no conflict of interest. 

Chest CT in COVID-19: what the radiologist needs to know

Reverse-transcriptase polymerase chain reaction versus chest computed tomography for detecting early symptoms of COVID-19. A diagnostic accuracy systematic review and metaanalysis

Diagnostic performance between CT and initial real-time RT-PCR for clinically suspected 2019 coronavirus disease (COVID-19) patients outside Wuhan

Chest CT accuracy in diagnosing COVID-19 during the peak of the Italian epidemic: a retrospective correlation with RT-PCR testing and analysis of discordant cases

WHO. Coronavirus Disease (COVID-19) Dashboard [Internet]. 2020

Radiological findings of abnormalities associated with congenital Zika virus infection: conclusions from World Radiology Day

Severe acute respiratory syndrome: management and reconfiguration of a radiology department in an infectious disease situation

The role of radiology in influenza: novel H1N1 and lessons learned from the 1918 pandemic

Coronavirus disease 2019 (COVID-19) imaging reporting and data system (COVID-RADS) and common lexicon: a proposal based on the imaging data of 37 studies

Sensitivity of chest CT for COVID-19: comparison to RT-PCR

Managing high clinical suspicion COVID-19 inpatients with negative RT-PCR: A pragmatic and limited role for thoracic CT

The provision of emergency radiology services and potential radiologist workforce crisis: is there a role for the emergencydedicated radiologist? Semin Ultrasound CT MRI

Policies and guidelines for COVID-19 preparedness: experiences from the University of Washington

Temporal variations in the diagnostic performance of chest CT for COVID-19 depending on disease prevalence: experience from North-Eastern France

Radiology perspective of coronavirus disease 2019 (COVID-19): lessons from severe acute respiratory syndrome and middle east respiratory syndrome

Déjà Vu or Jamais Vu? how the severe acute respiratory syndrome experience influenced a singapore radiology department's response to the coronavirus disease (COVID-19) epidemic

Severe acute respiratory syndrome: 11 years later -a radiology perspective

Preparedness and best practice in radiology department for COVID-19 and other future pandemics of severe acute respiratory infection

Learning from COVID-19: Sub-specialty radiologists should maintain general radiology skillset

The economic impact of the COVID-19 pandemic on radiology practices

How did radiologists' diagnostic performance has changed in COVID-19 pneumonia: A single-centre retrospective study