key: cord-0737304-0h0c2tsh authors: Guimaraes, Joao Guilherme Boaretto; de Oliveira Salvador, Gabriel Lucca; Papp, Carolina Lobo; Boica, Adrian Sobreiro Leal; Bittencourt, Andressa Borges; Grandi, Isabela Fernanda Rohde; Suckow, Kelvin; Fonseca, Vinicius Ribas title: Diagnostic accuracy of CO-RADS in patients with suspected Coronavirus Disease-2019: A single center experience date: 2022-02-17 journal: Clin Imaging DOI: 10.1016/j.clinimag.2022.02.005 sha: 5189adb45193c4ba5ca998da5b961d552311dfc8 doc_id: 737304 cord_uid: 0h0c2tsh INTRODUCTION: COVID-19 Reporting and Data System (CO-RADS) is a tool for standardizing the reports of patients with suspected or confirmed Sars-CoV-2 infection. We performed a study of the performance of the CO-RADS in a triage scenario of patients in Brazil. METHODS: Data from 426 Computed Tomography (CT) scans from March 2020 through December 2020 were assessed in an ambidirectional, both retrospective and prospective, for the assessment in one of the six categories of the CO-RADS. We assessed sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR), negative likelihood ratio (NLR) Youden's index, Positive and Negative Clinical Utility Index (UC + and UC− respectively) and diagnostic odds ratio (DOR). We also plotted Receiver Operating Characteristics (ROC) curve with Area Under the Curve (AUC) for CO-RADS of >4 (4 + 5). RESULTS: For CO-RADS classification > 4 (4 + 5) considered positive, the AUC obtained was of 0.89 (95% CI of 0.02), sensitivity of 78% (95% CI of 0.3), specificity of 91% (95% CI of 0.3), PPV of 0.92 (95% CI of 0.02), NPV of 0.41 (95% CI of 0.03), PLR of 0.85 (95% CI of 0.2), and NLR of 0.23 (95% CI of 0.02). CONCLUSION: CO-RADS demonstrated overall good diagnostic performance in stratifying patients with suspected Sars-CoV-2 infection, even those without confirmed laboratorial diagnosis, therefore being useful in a triage scenario with lack of resources. At the end of 2019, a viral infection with a new type of Coronavirus, later called SARS-CoV-2, was reported in Wuhan, capital of Hubei province, China¹. As of December 23, 2020, there have been 78,320,614 cases of the disease and 1,723,502 deaths worldwide. In Brazil, on the same date, there were 7,318,821 cases and 188,259 deaths². Among the countries most affected by the disease are the United States, Brazil, India and Russia³. In developing countries, including Brazil, there has been a diagnostic difficulty due to the lack of a rapid laboratory test that allows the patient to be adequately screened for isolation, due to the high risk of contagion and transmission of the disease. The RT-PCR exam considered as"gold standard" for the diagnosis can take up to a few hours to a few days for a result 4 Computed tomography (CT) of the chest has been important in the diagnostic aid of COVID-19 because it allows an immediate result, anticipating actions such as isolation and clinical management, in addition to often estimating the assessment of the severity and extent of pulmonary involvement, including in early stages of the disease 5,6 . In March 2020, the Dutch Radiology Society proposed a system to standardize the result of chest tomography for the diagnosis of COVID-19, the CO-RADS (COVID- 19 Reporting and Data System) 7 , to allow the tomographic reports to be uniform and replicable. This J o u r n a l P r e -p r o o f Journal Pre-proof classification system proved to be useful in several scenarios of the initial screening of patients for admission to isolation or general units while waiting for the confirmatory test [8] [9] [10] . The objective of this study is to validate the diagnostic and screening performance of CO-RADS at a center located in a population with a high prevalence of COVID-19 and to show its usefulness in a scenario with a low availability of beds and confirmatory exams. This ambidirectional, single-center study was approved by our institutional review board and the written informed consent was waived. The reporting was done in accordance with the recommendations of the Standards for Reporting Diagnostic Accuracy Initiative (STARD) 11 . The retrospective branch of the study involved CT from March 24 to September 1, 2020, and the prospective branch from September 2 to December 31, 2020. The examinations of the included patients involved those with respiratory symptoms and / or risk factors of exposure to Sars-COV-2 and who were submitted to chest CT and to gold standard collection (RT-PCR) up to 14 days after the entry CT according to institutional guidelines and available of tests. The clinical inclusion criteria for a patient to undergo a CT were: fever higher than 37.5 °C, cough, dyspnea, close relationship with a confirmed positive individual, travel or residential history in areas with high prevalence of disease contact with individuals with fever or respiratory symptoms from those areas within 14 days prior to the CT scan based on suspicion and risk stratification guidelines 12 . As the center where the study was conducted is open to the population and receives patients referred from other medical services, the study cohort was characterized as random. Missing imaging, important clinical information in the records such as the result of the RT-PCR, symptom descriptions and outcome status (alive or dead) at the time of assessment implied the exclusion of the patient from the final study cohort. All CT scans were performed using a single 64-slice CT scanner (Philips Brilliance®). All patients were scanned in supine position during single deep-inspiration breath-hold. The contrast medium was administered according to the radiologists' opinion on the overall findings (e.g. suspicion of pulmonary thromboembolism) and according to appropriateness criteria 13 . The CO-RADS Classification was based on the original article 7 and involves seven categories:  CO-RADS 0: not interpretable scan technically insufficient for assigning a score. Patients were considered highly suspicious for COVID-19 were those who had CT with categories 4 and 5, justifying isolation and the patients with CO-RADS 1 and 2 were considered as low probability and transferred to general beds, CO-RADS 3 patients were dependent on clinical and laboratory criteria for isolation. Figure 1 summarizes the main imaging aspects for each CO-RADS classification. Three senior radiologists of the hospital, with more than ten years of experience and six other residents, were responsible for analyzing the images blinded to all clinical data and without knowledge of the RT-PCR of the patients. All exams had a double reading (senior radiologist 1+senior radiologist 2 or senior radiologist+resident), one of them being done by one of the three senior radiologists and the discrepancy between readers was resolved with the opinion of a third senior radiologist. RT-PCR testing performed on respiratory specimens obtained by nasopharyngeal and throat swabs served as a reference standard for the COVID-19 diagnosis. Clinical information and index test results were not available for the assessors of the reference standard. The index tests were one of the six CO-RADS categories. We assessed Sensitivity, Specificity, Positive Predictive Value (PPV), Negative Predictive Value (NPV), Positive Likelihood Ratio (PLR), Negative Likelihood Ratio (NLR) Youden's index, Positive and Negative Clinical Utility Index (UC + and UC− respectively). UC+ was obtained by multiplying the sensitivity with PPV and UC-by multiplying Specificity with NPV 13 . The Diagnostic Odds Ratio (DOR) was assessed by the ratio of PLR and NLR. We chose to evaluate UC+, UC-, Youden's and DOR for two reasons: these parameters give a better comprehension of the diagnostic performance in tests that the gold standard is not perfect or is under evaluation [14] [15] [16] [17] , such as the RT-PCR and also the correlation with pre-test probability by these parameters gives a better idea of the usefulness of the test when it's most needed as shown in previous studies that correlates these parameters [14] [15] [16] . We also plotted Receiver Operating Characteristics (ROC) curve with the Area Under the Curve (AUC) for patients with CO-RADS categories >4 (CO-RADS 4+5) since these were the classifications that indicated isolation of a suspected patient in our center (thus considered "positive" until proven otherwise). We excluded CO-RADS 6 since those patients already had laboratorial confirmation and were automatically conducted to the specific sectors. We also plotted the nomograms for a positive and a negative test (PPTP and NPTP respectively) for each CO-RADS classification. Categorical variables were made by the chi-square test and the statistical significance was considered when p <0.05. All the continuous values had their confidence intervals calculated with 95% reliability. Since this study involved a randomized cohort of patients, we did not perform any previous sample size estimation in our population. The software used for the tabulation was Microsoft Excel® 2016 (Microsoft Corporation), the ROC curves were plotted on SPSS version 22 (IBM Corp., Armonk, NY, USA) and for the calculations of the values and plotting of nomograms we used the software r (r foundation). From 1024 consecutive patients initially screened from our database, we included a sample of 532 patients that met all the inclusion criteria and, after exclusion of CO-RADS 6 patients, we kept 426 patients, of those 273 (64%) obtained in the retrospective branch and 153 (36%) in the prospective branch, for the final analysis. The main reasons for exclusion as well as the patient flowchart are summarized in Figure 2 . The median age of the participants was 64.9 years (95% CI of 16.7). Male patients were 330 (54.5%), and Female patients were 276 (45.5%). Patient characteristics are summarized in Table 1 . We observed the highest percentage of deaths in CO-RADS 2 patients mainly because this category contemplated several diseases with tomographic features atypical for COVID-19 but still with a high mortality rate such as extensive lobar pneumonia, pulmonary edema, metastasis etc. The other diagnostic performance indexes for each CO-RADS category are summarized in Table 2 . The ROC curve for a CO-RADS classification of >4 (4+5) is positive for COVID-19 is shown in Figure 4 . The AUC obtained was of 0.89 (95% CI of 0.02). The PPTP obtained was 65% (95% CI of 3), 67% (95% CI of 3), 84% (95% CI of 5) and 94% (95% CI of 5) for CO-RADS 2, 3, 4 and 5 respectively. The NPTP obtained was 18% (95% CI of 10), 18% (95% CI of 10), 22% (95% CI of 10) and 28% (95% CI of 5) for CO-RADS 2, 3, 4 and 5 respectively. The nomograms for CO-RADS 2, 3, 4 and 5 considered positive are shown in figure 5 . This study analyzed the diagnostic performance of CO-RADS of a population in an ambidirectional manner, in other words, with retrospective and prospective branches associated in a blind and randomized manner. To the best of our knowledge, this was one of the validation studies for CO-RADS with the largest number of patients 8, 9, 18 . In general, patients with CO-RADS> 4 showed good agreement with the RT PCR indicated by the AUC of 0.89, similar to other studies as well as the sensitivity and specificity values, especially for CO-RADS 5 and 2 (16) . Our values of specificity proved to be even superior to other multicenter studies, which may be indicative of a better adaptation of the use of CO-RADS throughout the pandemic 10 . However, the values of AUC, sensitivity and specificity are not very accurate because they do not take into account the prevalence of the disease in the population and are unable to relate to the pre-test probability of the findings and patients 15 , which is why we conducted the other analyses. To try to overcome these limitations, we used indicators recently described as of positive and negative clinical utility (> 0.64 defined as of excellent 13 ). We found UC+ values that met this for CO-RADS 4 and 5. This means that, the tomographic findings related to CO-RADS 4 and 5 have value in considering a patient positive for COVID-19, even being rare or used in isolation, however, the absence of these values does not imply in the discard of the disease as indicated in the negative clinical utility value <0.5. This inference would prove to be hidden if we considered only average values such as the Younden index for these CO-RADS, as indicated 13, 14 . The effect of the prevalence of the disease in our population as well as the false negatives also impacted the negative and positive likelihood ratios of our study. However, when analyzing the DOR for each CO-RADS we observed that all of them presented a value above 1, which could infer that the classification of CO-RADS can also discern between patients with or without COVID-19 with statistical significance. Another point to be considered refers to the values of PPTP indicated by the nomograms. Patients classified as CO-RADS>4 presented a PPTP of 84 and 94% indicating excellent probabilities of confirmation for COVID in these categories. In contrast, patients with CO-RADS 1 and 2 presented an NPTP of up to 20%, which indicates that trying to stratify patients according to CO-RADS is well founded. Analyzing all the results retrospectively, it is clear that most of the negative results occurred in the initial period of the infection (7 days), generating a decrease in the specificity and negative clinical utility of CO-RADS. Even so, our specificity values are higher than other studies 16 . Patients who were classified as CO-RADS 3 and still had a low differentiation between the presence or absence of COVID-19 (lower DOR among the categories) due to overlap with other pathologies of similar tomographic aspect, such as pulmonary edema or influenza pneumonia, mainly in early stages [17] [18] [19] [20] . Throughout the course of the pandemic, several scores were proposed by the Radiological Society of North American and American Society of Thoracic Radiology with diagnostic performance indexes 21 with an AUC ranging from 0.8-0.85 22 . Our AUCs were significantly better especially through the refinement of specificity and considering only CO-RADS above 4 positive in the final analysis, which was proved significant in our analysis. Another explanation for CO-RADS to have a better diagnostic performance involves the fact that the RSNA Statement and several others have low sensitivity values since most viral pneumonias not related to Sars-Cov-2 have similar presentation especially in the early phases 21 . The main limitations of our study involved the retrospective portion mainly related to the lack of clinical data and the quality of the tests obtained at the beginning of the pandemic. We also had to deal with the CO-RADS learning curve, although fast, and even with the double Reading of exams, we could not rule out any incorrect classifications. In Conclusion, CO-RADS proved to be useful for stratifying patients in emergency departments and screening suspect patients for COVID-19, allowing them to be separated from the non-COVID-19 patients, however low CO-RADS were not useful for ruling out infection in the early stages, and a high level of suspicion should be maintained in these patients if they have associated risk factors. with micronodules in the lower left lobe with "tree in a bud" pattern suggestive of other infections. CO-RADS 3 (B) with pleural effusion and scattered ground glass areas that could be implied in congestive states, CO-RADS 4 (C), more confluent ground glass areas predominantly in the lower lobes, but with pleural effusion that did not allow the classification in the CO-RADS 5 category and CO-RADS 5 (D) with a typical ground glass centrilobular and peripheral distribution. SARS-CoV-2/COVID-19: Viral Genomics, Epidemiology, Vaccines, and Therapeutic Interventions COVID-19): A Review COVID-19 in Brazil Diagnostic accuracy of serological tests for covid-19: systematic review and meta-analysis Role of computed tomography in COVID-19 Computed Tomography Features of Coronavirus Disease 2019 (COVID-19): A Review for Radiologists COVID-19 Standardized Reporting Working Group of the Dutch Radiological Society. CO-RADS: A Categorical CT Assessment Scheme for Patients Suspected of Having COVID-19-Definition and Evaluation Evaluation of the Usefulness of CO-RADS for Chest CT in Patients Suspected of Having COVID-19 Chest CT in COVID-19 at the ED: Validation of the COVID-19 Reporting and Data System (CO-RADS) and CT Severity Score: A Prospective, Multicenter, Observational Study A head-to-head comparison of the intra-and interobserver agreement of COVID-RADS and CO-RADS grading systems in a population with high estimated prevalence of COVID-19 STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration ACR Appropriateness Criteria® Acute Chest Pain-Suspected Pulmonary Embolism How Do We Know When a Screening Test is Clinically Useful? In Screening for Depression in Clinical Practice: An Evidence-Based Guide A new index of clinical utility for diagnostic tests Analyze a screening or diagnostic Study. Practical Guides of Psycho-Oncology Meta-analysis of screening and diagnostic tests Diagnostic Performance of Chest CT for SARS-CoV-2 Infection in Individuals with or without COVID-19 Symptoms Diagnostic Performance of CO-RADS and the RSNA Classification System in Evaluating COVID-19 at Chest CT: A Meta-Analysis Review of the Chest CT Differential Diagnosis of Ground-Glass Opacities in the COVID Era Chest CT in COVID-19: What the Radiologist Needs to Know Comparison of Chest CT Grading Systems in Coronavirus Disease 2019 (COVID-19) Pneumonia The authors declare that they have no competing interests. The authors did not received any funding from third parties or institutions for the conduction of this study.