key: cord-266672-t85wd0xq
authors: Bagnera, Silvia; Bisanti, Francesca; Tibaldi, Claudia; Pasquino, Massimo; Berrino, Giulia; Ferraro, Roberta; Patania, Sebastiano
title: Performance of Radiologists in the Evaluation of the Chest Radiography with the Use of a “new software score” in Coronavirus Disease 2019 Pneumonia Suspected Patients
date: 2020-07-20
journal: J Clin Imaging Sci
DOI: 10.25259/jcis_76_2020
sha: 
doc_id: 266672
cord_uid: t85wd0xq

OBJECTIVES: The purpose of this study is to assess the performance of radiologists using a new software called “COVID-19 score” when performing chest radiography on patients potentially infected by coronavirus disease 2019 (COVID-19) pneumonia. Chest radiography (or chest X-ray, CXR) and CT are important for the imaging diagnosis of the coronavirus pneumonia (COVID-19). CXR mobile devices are efficient during epidemies, because allow to reduce the risk of contagion and are easy to sanitize. MATERIAL AND METHODS: From February–April 2020, 14 radiologists retrospectively evaluated a pool of 312 chest X-ray exams to test a new software function for lung imaging analysis based on radiological features and graded on a three-point scale. This tool automatically generates a cumulative score (0–18). The intra- rater agreement (evaluated with Fleiss’s method) and the average time for the compilation of the banner were calculated. RESULTS: Fourteen radiologists evaluated 312 chest radiographs of COVID-19 pneumonia suspected patients (80 males and 38 females) with an average age of 64, 47 years. The inter-rater agreement showed a Fleiss’ kappa value of 0.53 and the intra-group agreement varied from Fleiss’ Kappa value between 0.49 and 0.59, indicating a moderate agreement (considering as “moderate” ranges 0.4–0.6). The years of work experience were irrelevant. The average time for obtaining the result with the automatic software was between 7 s (e.g., zero COVID-19 score) and 21 s (e.g., with COVID-19 score from 6 to 12). CONCLUSION: The use of automatic software for the generation of a CXR “COVID-19 score” has proven to be simple, fast, and replicable. Implementing this tool with scores weighed on the number of lung pathological areas, a useful parameter for clinical monitoring could be available.

e severe acute respiratory syndrome coronavirus 2 is a newly emerging zoonotic agent appeared in December 2019 and causing the coronavirus disease 2019 (COVID-19). [1] e virus rapidly infected all provinces of China and spread to the rest of the world. [2] COVID-19-infected patients develop pneumonia with associated symptoms of fever (98%), cough (76%), and myalgia or fatigue (44%). [3] e infection results in a syndrome leading, in some cases, to a critical care respiratory condition, requiring specialized management at intensive care units. [4, 5] e current diagnostic criterion for COVID-19 is the positive result of a nucleic acid test by real-time reverse transcription-polymerase chain reaction (RT-PCR). [2] oracic imaging with chest radiography (or chest X-ray, CXR) and computed tomography (CT) plays an important role for diagnosis, management, and follow-up of COVID-19 pneumonia patients. [6] e typical radiological imaging demonstrated clear destruction of the pulmonary parenchyma, including interstitial inflammation and extensive consolidation. [7] Many groups describe lung opacities in more than one lobe and bilateral lower lobe consolidations (as opposed to community acquired bacterial pneumonia which tends to be unilateral and involving a single lobe). Identifying multifocal air-space disease on CXR can be a significant clue to COVID-19 pneumonia; [8] some investigators have noticed that the air-space disease tends to have a lower bilateral lung distribution (in the 33-85% of cases). [8, 9] High frequency of peripheral lung involvement (in the 77-88% of cases) is another feature readily identifiable on CXR. Ground glass opacities emerges in other cases evaluated (GGO, in the 55-79% of cases), and it is extremely difficult to detect on CXR.

CXR is a less sensitive modality in the detection of COVID-19 lung disease compared to CT (with a reported baseline CXR sensitivity of 69%). [9] CT is more sensitive than CXR for early disease and it is also important to exclude alternative diagnoses (pulmonary thromboembolism, and heart failure), [10, 11] but the use of CXR could offer some advantages. However, due to infection control issues related to patient transport to CT suites, the inefficiencies introduced in CT room decontamination, and lack of CT availability in parts of the world, portable chest radiography (CXR) will likely be the most commonly utilized modality for identification and follow-up of lung abnormalities. In fact, the American College of Radiology notes that CT decontamination required after scanning COVID-19 patients may disrupt radiological service availability and suggests that portable chest radiography may be considered to minimize the risk of cross-infection. [12] Furthermore, in cases of high clinical suspicion for COVID-19, a positive CXR may obviate the need for CT.

Many groups have tried to schematize radiologist's work providing various "diagnostic scores" with the aim to make the report more standardized; actually, all of these researches are "CT score. " e use of a specific score for CXR report has not been well investigated; the purpose of this study is to clarify the use of a specific CXR digital tool and to investigate its validity in terms of inter-reader concordance.

Fourteen radiologists participated in this study, divided into four groups based on work experience (Group 1: < to 5 years; Group 2: From 5 to 15 years; Group 3: 15-20 years; and Group 4: > to 20 years), as indicated in Table 1 .

From February 24, 2020, to April 3, 2020, more than 1560 patients with suspected diagnosis of COVID-19 viral pneumonia underwent chest radiography (CXR) in our institution. To evaluate a new tool called "COVID-19 score" made available to radiologists for lung imaging analysis, we retrospectively included in the study patients who underwent at least two consecutive chest X-rays for a total of 312 exams. is instrument is a software function available within the radiological examination reporting platform of our institute; Table 1 : ASL TO4 radiologists participated on this study and divided into four groups (based on the number of years of work experience). For each zone radiologists assigned the score based on predominant features, graded on a 3-point scale as follow: (0) For negative exam; (1) for predominant interstitial changes (as reticular and/or interlobular septal thickening);

(2) in the presence of both interstitial changes and alveolar consolidation (as interstitial changes predominance); and (3) in the presence of both interstitial changes and alveolar consolidation (as alveolar consolidation predominance).

Points from the values for each zone (from A to F) were added to obtain a final total cumulative score (from 0 to 18). e software can automatically generate the final "COVID-19 score;" the data can be recorded and filed for display in subsequent reports and it can be used as a comparison parameter in sequential radiological examinations (for a temporal evaluation of the trend).

e radiologists retrospectively and blindly assessed the pool of selected chest X-ray exams and reported the "COVID-19 score" values in an Excel database. When filing, the radiologists had no clinical information and did not know the outcome of the RT-PCR COVID test (actually considered the gold standard for the diagnosis). Chest X-ray images were viewed on Barco workstation.

e time required to complete the banner of each exam for the achievement of the overall score was reported and the insertion of the "COVID-19 score" within the structured reports of CXR was assessed, as a prognostic evaluation parameter.

e practical implications of using this new tool in the radiological CXR evaluation were considered.

All the exams were performed with patients in the supine position with a single AP projection. e portable chest X-ray device used for investigations was: GMM Mecall Easy Slide 30 (for the Ivrea Hospital), Philips Practice 33 PLUS (for the Ciriè Hospital), and Technix TMS 300 X-ray (for the Chivasso Hospital). All these mobile equipment have been used to perform chest examinations at the patient's bed.

After each procedure, the equipment was sanitized (with chlorine-containing disinfectant solutions). Additional precautions were taken by radiology technicians (TSRM), to preserve themselves and prevent the transmission of the virus in the health-care environment. ese precautions include the correct use of personal protective equipment (PPE; such as visor and mask, disposable gloves, and over-gown with headgear) and adequate awareness and training on the modalities relating to their use, dressing/undressing, and disposal.

To investigate the potential role of the "COVID-19 score" in the report of CXR, the concordance analysis among the 14 radiologists and the four groups were performed using the Fleiss' Kappa statistic. e final scores automatically generated by the software was grouped in three score classes (0-4, 5-13, and 14-18) relating to a low, moderate, and high degree of COVID-19 prognostic value.

Fourteen radiologists (three of the Group 1; five of the Group 2; three of the Group 3; and three of the Group 4) evaluated 312 chest X-ray images of COVID-19 pneumonia suspected patients (80 males and 38 females) with an average age of 64, 47 years.

e radiologists belong to the three hospitals of our company (one from Ivrea, nine from Ciriè, and four from Chivasso). No radiologist has reported difficulties in accessing the software function or in filling out the banner. No issues with the automatic registration of the "COVID-19 score" were found.

e results of the inter-rater and intra-group concordance, obtained with the Fleiss' Kappa statistic, showed a moderate agreement (0.53) among the radiologists (considering ranges: K <0.4 for a concordance definable as "poor;" 0.4 < K < 0.6 definable as "moderate;" 0.6 < K < 0.8 definable as "good;" and K > 0.8 definable as "excellent"). e intra-group analysis leads to the same conclusion, with Fleiss' Kappa values between 0.49 for Group 1 and 0.59 for Group 3, indicating that the years of work experience was irrelevant in the use of this new software application.

e average time for filling in the banner of a chest X-ray for obtaining the "COVID-19 score" with automatic software was: Seven seconds for negative exams (with score equal to zero); 13 s for tests with score between one and six; 21 s for exams with intermediate score (from 7 to 12, where the presence of more pathological findings made it more difficult to identify and interpret the image), and 15 s for images strongly indicative of pulmonary impairment (with score values from 13 to 18).

Rapid and accurate diagnosis of COVID-19 is critical during the epidemic.

oracic imaging with chest radiography (CXR) and CT is key tools for pulmonary disease diagnosis and management, but their role in the management of COVID-19 has not been considered within the multivariable context of the severity of respiratory disease, pre-test probability, risk factors for disease progression, and critical resource constraints. [13, 14] To solve these problems, we hypothesize a diagnostic model based on radiological imaging and clinical manifestations alone, independent of the nucleic acid test.

Analysis of specific cases, where radiologists were wrong reveals that the mistakes were made when the COVID-19 chest imaging findings are either subtle (likely reflecting early time in the disease process) or when COVID-19 has atypical signs. It is worth noting that non-COVID-19 pneumonia can also have typical appearance of COVID-19. is poses a dilemma because mandated quarantine for all suspected cases can put significant strain on medical infrastructure, health-care providers, and the lives of patients, but may need to be followed as a necessary precaution due to variation in presentation with timing of disease. [15] Multiple studies demonstrated that the common COVID-19 chest imaging findings are represented by bilateral involvement, peripheral distribution, and lower zone dominance; these can also be appreciated on CXR. [8] In the scenario, where there is high clinical suspicion of COVID-19 it is conceivable that a positive CXR may obviate the need for a CT, thus reducing burden on CT units in this pandemic.

Although chest CT has high sensitivity, it has low specificity. [10, 16] is low specificity may stem from the fact that it is difficult to distinguish COVID-19 findings from findings of other disease on chest CT. Another critical issues should be considered: e availability of the CT scanner and the time required to clean and disinfect the equipment after performing imaging on a suspected patient.

is problem can be solved using CXR portable equipment. e use of these mobile devices allows the exam to be performed in the same room the patient is isolated into, therefore reducing the risk of COVID-19 transmission along the transport route to a CT scanner. In hospitalized patients, CXR can be used as a first step to assess the progression of the disease or to make alternative diagnoses, such as lobar pneumonia (suggestive of bacterial superinfection), pleural effusion, or pneumothorax.

e role of imaging in COVID-19 patients' management should be considered in a multivariable context; the physicians should try to choose the best tools in a specific context considering different practice settings, different phases of the epidemic outbreak, and specific resources availability.

In this study, we tested a new software application called "COVID-19 score" that can be used in the reporting of chest X-ray imaging in patients suspected COVID-19, based on radiological semantic features. Using this tool, we found some critical issues: e range (proposed by the design company) with values from 0 to 18 is too wide (it does not allow a precise statistical analysis and generates multiple interpretative variables in radiologists). In addition, the "COVID-19 score" obtained with the cumulative method (by automatic addition and without a specific attribution of the area/s concerned) is difficult to correlate with the true clinical status of the patient.

By stratifying the scores into three classes (0-4, 5-13, and 14-18), we saw that the extreme values can only be representative of subject's symptomatology. In fact, the class with "COVID-19 score" from 0 to 4 correlates with negative imaging and pauci-symptomatic patient. e class with "COVID-19 score" from 15 to 18 correlates with pathological imaging and with severely symptomatic patient is shown in Figure 2 . On the other hand, the intermediate class with "COVID-19 score" from 5 to 13, poses major interpretative problems, due to the copresence of different situations even with identical score values, as shown in Figures 3 and 4 . For example, a score of six could derive both from the sum of a score of one in all anatomical areas (from A to F) and from the sum of a maximum score attributed in two ipsilateral anatomical areas (equivalent to a 2/3 lung pathological involvement) or contralateral (equivalent to a bilateral pathological involvement process).

To overcome this correlation problem between "score imaging" and "clinic status, " we propose that the "COVID-19 score" in CXR be weighed on the number of anatomical lung sectors (A-B) involved; this could offer clinicians a quantifiable data that can be used for patient monitoring over time.

e use of automatic software for the generation of a "COVID-19 score" within the reports of chest X-rays in COVID-19 pneumonia suspected patients has proven to be simple, fast, and replicable in different working contexts with a moderate agreement among the radiologists and without significant differences related to the work experience. Implementing this tool with scores weighed on the number of lung pathological areas can be a future parameter of clinical interest for the prognostic temporal evaluation of the improvement or worsening trend of a patient's health.

Patient's consent not required as patients identity is not disclosed or compromised.

Nil.

ere are no conflicts of interest. Figure 2 : (a) 69-year-old man with pneumonia disease who presented with fever, cough, and dyspnea. Portable CXR image shows lungs with bilateral pulmonary involvement with widespread interstitial disease and areas of consolidation in different zones of each lung with CXR "COVID-19 score" value of 18; (b) 64-yearold man with respiratory distress (intubated and subjected to mechanical ventilation). Portable CXR image shows lungs with multiple small areas of lung consolidation affecting the whole lung parenchyma bilaterally with CXR "COVID-19 score" value of 18. b a Figure 3 : 65-year-old man arrived in the emergency room with sudden high fever (39.8°C) and intense cough. (a) e first portable CXR examination shows lungs with minimal disventilative bibasal areas associated with reticular and hazy medium and the lower left lobe opacities (with interstitial changes predominance) with CXR "COVID-19 score" value of 3. (b) After a week, the second portable CXR exam shows lungs with progressive worsening with bilateral medium and lower lobe consolidations (with alveolar consolidation predominance) with CXR "COVID-19 score" value of 8. b a Figure 4 : (a) 67-year-old man with respiratory distress (intubated and subjected to mechanical ventilation). Portable CXR image shows lungs with multiple small areas of lung consolidation (with alveolar consolidation predominance) in the whole lung parenchyma bilaterally with CXR "COVID-19 score" value of 16. (b) 3-years-old man with respiratory distress (intubated and subjected to mechanical ventilation). Portable CXR image shows lungs with reticular and hazy medium and lower bilateral lobe opacities (with interstitial changes predominance) with CXR "COVID-19 score" value of 12. 

A novel coronavirus from patients with pneumonia in china

A diagnostic model for coronavirus disease 2019 (COVID-19) based on radiological semantic and clinical features: A multi-center study

Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China

e species severe acute respiratory syndrome-related coronavirus: Classifying 2019-nCoV and naming it SARS-CoV-2

SARS-CoV-2 viral load in upper respiratory specimens of infected patients

Imaging changes in patients with 2019-nCov

Imaging profile of the COVID-19 infection: Radiologic findings and literature review

Frequency and distribution of chest radiographic findings in COVID-19 positive patients

Portable chest X-ray in coronavirus disease-19 (COVID-19): A pictorial review

Chest CT findings in coronavirus disease-19 (COVID-19): Relationship to duration of infection

CT features of coronavirus disease 2019 (COVID-19) pneumonia in 62 patients in Wuhan, China

ACR Recommendations for the Use of Chest Radiography and Computed Tomography (CT) for Suspected COVID-19 Infection. United States: American College of Radiology

e role of chest imaging in patient management during the COVID-19 pandemic: A multinational consensus statement from the fleischner society

e many faces of COVID-19: Spectrum of imaging manifestations

Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT

Sensitivity of chest CT for COVID-19: Comparison to RT-PCR