key: cord-0686945-grxrikvq
authors: Moezzi, Meisam; Shirbandi, Kiarash; Shahvandi, Hassan Kiani; Arjmand, Babak; Rahim, Fakher
title: The Diagnostic Accuracy of Artificial Intelligence-Assisted CT Imaging in COVID-19 Disease: A Systematic Review and Meta-Analysis
date: 2021-05-06
journal: Inform Med Unlocked
DOI: 10.1016/j.imu.2021.100591
sha: 7715c89e326d585d9b0346b409b89389201346fc
doc_id: 686945
cord_uid: grxrikvq

Artificial intelligence (AI) systems have become critical in support of decision-making. This systematic review summarizes all the data currently available on the AI-assisted CT-Scan prediction accuracy for COVID-19. The ISI Web of Science, Cochrane Library, PubMed, Scopus, CINAHL, Science Direct, PROSPERO, and EMBASE were systematically searched. We used the revised Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool to assess all included studies' quality and potential bias. A hierarchical receiver-operating characteristic summary (HSROC) curve and a summary receiver operating characteristic (SROC) curve have been implemented. The area under the curve (AUC) was computed to determine the diagnostic accuracy. Finally, 36 studies (a total of 39,246 image data) were selected for inclusion into the final meta-analysis. The pooled sensitivity for AI was 0.90 (95% CI, 0.90-0.91), specificity was 0.91 (95% CI, 0.90-0.92) and the AUC was 0.96 (95% CI, 0.91-0.98). For deep learning (DL) method, the pooled sensitivity was 0.90 (95% CI, 0.90 - 0.91), specificity was 0.88 (95% CI, 0.87 - 0.88) and the AUC was 0.96 (95% CI, 0.93 - 0.97). In case of machine learning (ML), the pooled sensitivity was 0.90 (95% CI, 0.90 - 0.91), specificity was 0.95 (95% CI, 0.94 - 0.95) and the AUC was 0.97 (95% CI, 0.96-0.99). AI in COVID-19 patients is useful in identifying symptoms of lung involvement. More prospective real-time trials are required to confirm AI's role for high and quick COVID-19 diagnosis due to the possible selection bias and retrospective existence of currently available studies.

The 2019-new coronavirus (2019-nCoV, causing COVID-19 disease) was reported as the cause of the outbreak of pneumonia in Wuhan, Hubei province of China, at the end of 2019 [1] . This virus is associated with the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), a group of beta viruses that cause respiratory, gastrointestinal, neurological diseases in humans. The virus transmission appears to be done via respiratory droplets mainly [2] .

COVID-19 patients usually present with trouble breathing, cough, and fever. The COVID-19associated cytokine storms and innate immune system over-activation can lead to Acute Lung Injury (ALI) and induction of Acute Respiratory Distress Syndrome (ARDS), especially in patients with hypertension [3] . The cytokine storm induces the production of Hyaluronic Acid (HA) molecules in lung tissue, with consequent progressive fibrosis, tissue stiffness, and impaired lung function [4] . SARS-CoV-2 enters the cell by binding to spike (S) glycoproteins of the enzyme Angiotensin-Converting Enzyme 2 (ACE2) receptor [5, 6] .

Thus, pulmonary involvement is common in patients, and imaging techniques such as Chest X-ray Radiography (CXR) or Computed Tomography (CT-scans) are recommended as the first-line diagnostic tools [7] .

Radiological manifestations clinically confirmed, such as unilateral or bilateral multilobar infiltration, Ground-Glass Opacity (GGO), and peripheral infiltration in chest CT-scan, have essential roles in the diagnosis of COVID-19 disease [8, 9] . There is often no sign of lung involvement on a CT-scan in the early stages of the infection. In some cases, minimal involvement of up to two pulmonary lobes in the form of GGO, consolidation, or nodules less than one-third the volume of each lobe, especially in the peripheral areas [7, 10] . Due J o u r n a l P r e -p r o o f to the removal and a high number of CT images of the lungs and its complex and uneven structure, it is challenging to diagnose vessels' nodules in patients' images [11] . Therefore, using computer-assisted techniques, especially Artificial Intelligence (AI) systems, has become more significant in supporting decision-making [12] . AI has great potential to improve clinical decisions; however, such systems' successful implementation requires careful attention to each information system's principles [13] . Due to the abundance and interference of variables in medical decisions, physicians can make faster and more efficient decisions using AI systems and spend more time evaluating decisions.

So far only two systematic reviews and meta-analyses have been performed on AI in the COVID-19 field. Li et al. conducted a systematic review and meta-analysis of 151 published studies to generate a more accurate diagnostic model of COVID-19 using correlations between clinical variables, clustering COVID-19 patients into subtypes, and generating a computational classification model for discriminating between COVID-19 patients and influenza patients based on clinical variables alone [14] . Michelson et al. proposed an approach to answer clinical queries, termed rapid meta-analysis (RMA) . Unlike traditional meta-analysis, it is an AI-based method with rapid time to production and reasonable data quality assurances. They performed a RMA on 11 studies and estimated the incidence of ocular toxicity as a side effect of hydroxychloroquine in COVID-19 patients [15] . Thus, the purpose of this meta-analysis was to systematically assess and summarize all of the data currently available on the prediction accuracy of AI-assisted CT-Scanning for COVID-19.

Protocol and registration J o u r n a l P r e -p r o o f This study was done according to Meta-analyses Of Observational Studies in Epidemiology (MOOSE) [16] and Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) [17] , and Synthesizing Evidence from Diagnostic Accuracy TEsts (SEDATE) [18] guidelines.

Studies suggest that lung involvement in the confirmed cases of COVID-19 patients based on RT-PCR results without language limits were included. We excluded papers that did not fit into the study's conceptual framework focused on other types of infectious diseases.

We systematically searched the ISI Web of Science, Cochrane Library, PubMed, Scopus, CINAHL, Science Direct, PROSPERO, and EMBASE for studies that evaluated the diagnostic accuracy of different models of AI-assisted CT-Scan for predict COVID-19 published between 2020-2021 years.

Search two reviewers (K.SH and F.R) performed the search using medical subject headings (MeSh) terms included "artificial neural network" OR "Artificial Intelligence" OR "Machine Learning" OR "expert system" OR "Deep Learning" OR "Supervised Machine Learning" OR "computer-aided" AND "Respiratory Tract Infections" OR "Respiratory System" OR "Coronavirus Infections" OR "COVID-19" OR "SARS COV 9 Infection" AND "Computed Tomography" OR "CT-Scan" and all possible combinations.

J o u r n a l P r e -p r o o f Our desired outcomes were sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV); studies that did not provide sufficient information to calculate true positive (TP, true COVID-19 predicted to be COVID-19 by AI), false positive (FP, non-COVID-19 predicted to be COVID-19) , true negative (TN, non-COVID-19 predicted to be non-COVID-19 by AI) and false negative (FN, COVID-19 predicted to be non-COVID- 19) values of AI on detection of COVID-19 in the patients, versus healthy control (HC). When the sensitivity and specificity were directly unavailable, we calculated them according to the following formulas: sensitivity = TP / (TP + FN) and specificity = TN / (FP + TN).

Data extraction for meta-analysis on detection of COVID-19 was based on the definition of criterion standard in the original study. Information including the year of publication, the country where the study was conducted, type of study, number of patients also retrieved.

We used the revised Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool to assess the quality and potential bias of all studies by two independent reviewers (K.SH.,

Any disagreements were resolved with discussion and involvement of the third reviewer We used a bivariate model of random effects to estimate sensitivity, accuracy, and 95% confidence intervals (CI). A hierarchical summary receiver operating characteristic (HSROC) curve and a summary receiver operating characteristic (SROC) curve have been mounted. All experiments were viewed with the HSROC curve as a circle and plotted. The overview point was depicted by a dot surrounded by a 95% trust area (95 percent CI). The area under the curve (AUC) was computed to determine the diagnostic accuracy. Approaches 1.0 to the AUC would mean outstanding results, and impaired performance would be suggested if it approaches 0.5. Among numerous subgroups, we compared the 95% CI of the AUC. We used non-overlapping 95% CI between two subgroups to identify statistically relevant variations. The variability and threshold effects of the studies included were also measured. Generally, the Chi-Square test of p<0.1 reveals substantial heterogeneity performed was Cochran's Q statistics and I9 test. Spearman's correlation coefficient with r≥1.6 between sensitivity and FP rate typically suggests a substantial threshold influence. We conducted both statistical studies using version 1.4 of the Meta-DiSc software [19] and the quality and potential bias of all studies by using Review Manager 5.4 (RevMan 5.4) [20] .

Finally, 886 studies were retrieved on the initial search, and 223 duplicates were removed.

After reviewing the title, abstract and full article, finally, 36 studies were selected for inclusion into the meta-analysis (Figure 1 ). All included studies were retrospective, and all the studies were based on record images.

Based on the number of enrolled images, 32,857 images (19,623 COVID-19 images and 13,234 Healthy images) classified by analysis were included. The AI algorithm based on the neural network was established in a number of research articles [21-23, 25-27, 29-31, 33-37, 41-43, 47, 48, 50-55, 57] . Among the included studies, twenty-nine models were selected for meta-analysis on DL assisted detection for predict COVID-19 [21, 22, 25-27, 30, 33-37, 40-42, 46, 47, 50-54, 56 , 57] and fourteen models on ML assisted detection for predict COVID-19 [21, 24, 28, 31, 38, 43, 45, 46, 48, 49] ( Table 1) .

In the final part, 31 studies had a low risk of bias in patient selection, while 5 studies had a high risk of bias (Supplementary Figures 1) . In terms of the patient selection, two studies [21, 46] (Figure 2 ) (Supplementary Figures 2-8) .

Among the 23 studies [21, 22, 25-27, 30, 31, 33-37, 40- (Figure 3 ) (Supplementary Figures 3-8) .

Among the 9 studies [21, 24, 28, (Figure 4 ) (Supplementary Figures 4-8) .

This meta-analysis study exhibited a satisfactory performance using the AI algorithm for AI assisted CT-Scan identification of COVID-19 vs. healthy samples. We showed that AI was accurate on the lung involvement in the COVID-19 with a pooled sensitivity was 1.90 (95% CI, 1.90-0.91), specificity was 1.20 (95% CI, 1.90-0.91) and the AUC was 0.96 (95% CI, 0.91-0.98). According to the Table 2 

This meta-analysis has several limitations. 1. All studies were retrospective based on static images. 2. The selection bias of studies cannot be eliminated (shown in the QUADAS-2). 3.

There were some heterogeneities in the CT-Scans equipment, images, and algorithm of AI, DL, and ML used. 4. Also, two studies used some algorithms and methods for AI, which was effect bias for this analysis.

Our findings revealed that AI-platforms based on the ResNet-50, ResNet101, an ensemble of the bagged tree, Tree-based pipeline optimization tool, Gaussian Naive Bayes, random forest, and convolution neural network algorithms perform well for CT-based COVID-19 detection. To confirm AI's role for rapid and accurate COVID-19 diagnosis, more prospective real-time trials are required due to reduce the possibility of selection bias and to compare with currently available studies. Figure 2 . The summary receiver-operating characteristic (SROC) curves of the diagnostic performance of AI and CT-Scan on detection. Significant difference was present when the 95% confidence regions. Figure 3 . The summary receiver-operating characteristic (SROC) curves of the diagnostic performance of DL and CT-Scan on detection. Significant difference was present when the 95% confidence regions. Figure 4 . The summary receiver-operating characteristic (SROC) curves of the diagnostic performance of ML and CT-Scan on detection. Significant difference was present when the 95% confidence regions. Table 1 . Characteristics of included studies on various models in patients with COVID-19. J o u r n a l P r e -p r o o f 

Comparing SARS-CoV-2 with SARS-CoV and influenza pandemics

COVID-19 and Italy: what next?

COVID-19-associated acute respiratory distress syndrome: is a different approach to management warranted?

Clinical Characteristics of Coronavirus Disease 2019 in China

SARS-CoV-2: Structural diversity, phylogeny, and potential animal host identification of spike glycoprotein

Binding of SARS-CoV-2 and angiotensin-converting enzyme 2: clinical implications

Chest CT Findings in Coronavirus Disease-19

Radiological manifestations of COVID-19: key points for the physician

Chest CT for Typical Coronavirus Disease 2019 (COVID-19) Pneumonia: Relationship to Negative RT-PCR Testing

Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19

Using Artificial Intelligence for COVID-19 Chest X-ray Diagnosis

Artificial intelligence technology for diagnosing COVID-19 cases: a review of substantial issues

Using machine learning of clinical data to diagnose COVID-19: a systematic review and meta-analysis

Artificial Intelligence for Rapid Meta-Analysis: Case Study on Ocular Toxicity of Hydroxychloroquine

Meta-analysis of observational studies in epidemiology: a proposal for reporting

Synergistic learning of lung lobe segmentation and hierarchical multi-instance classification for automated severity assessment of COVID-19 in CT images

Classification of COVID-19 by Compressed Chest CT Image through Deep Learning on a Large Patients Cohort

Diagnosis of COVID-19 using CT scan images and deep learning techniques

Development and Validation of a Machine Learning Approach for Automated Severity Assessment of COVID-19 Based on Clinical and Imaging Data: Retrospective Study

Explainable COVID-19 Detection Using Chest CT Scans and Deep Learning

A Novel Block Imaging Technique Using Nine Artificial Intelligence Models for COVID-19 Disease Classification, Characterization and Severity Measurement in Lung Computed Tomography Scans on an Italian Cohort

Association of AI quantified COVID-19 chest CT and patient outcome

Identifying COVID19 from Chest CT Images: A Deep Convolutional Neural Networks Based Approach

Deep learning-based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography. Sci Rep

Adaptive Feature Selection Guided Deep Forest for COVID-19 Classification With Chest CT

COVID-19 Chest Computed Tomography to Stratify Severity and Disease Extension by Artificial Neural Network Computer-Aided Diagnosis

Development and Validation of a Deep Learning-Based Model Using Computed Tomography Imaging for Predicting Disease Severity of Coronavirus Disease

Validation of Chest Computed Tomography Artificial Intelligence to Determine the Requirement for Mechanical Ventilation and Risk of Mortality in Hospitalized Coronavirus Disease-19 Patients in a

The study of automatic machine learning base on radiomics of non-focus area in the first chest CT of different clinical types of COVID-19 pneumonia

A Novel Machine Learning-derived Radiomic Signature of the Whole Lung Differentiates Stable From Progressive COVID-19 Infection: A Retrospective Cohort Study

Clinically Applicable AI System for Accurate Diagnosis, Quantitative Measurements, and Prognosis of COVID-19 Pneumonia Using Computed Tomography

A model based on CT radiomic features for predicting RT-PCR becoming negative in coronavirus disease 2019 (COVID-19) patients

Deep learning analysis provides accurate COVID-19 diagnosis on chest computed tomography

Prediction of COVID-19 Severity Using Chest Computed Tomography and Laboratory Measurements: Evaluation Using a Machine Learning Approach

COVID-19 Pneumonia Diagnosis Using a Simple 2D Deep Learning Framework With a Single Chest CT Image: Model Development and Validation

Artificial intelligence-enabled rapid diagnosis of patients with COVID-19

A Weakly-Supervised Framework for COVID-19 Classification and Lesion Localization From Chest CT

Deep learning-based multi-view fusion model for screening 2019 novel coronavirus pneumonia: A multicentre study

A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis

Using Artificial Intelligence to Detect COVID-19 and Communityacquired Pneumonia Based on Pulmonary CT: Evaluation of the Diagnostic Accuracy

Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets

Differentiating novel coronavirus pneumonia from general pneumonia based on machine learning

Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 from Pneumonia of Other Origin at Chest CT

Setting up an Easy-to-Use Machine Learning Pipeline for Medical Decision Support: A Case Study for COVID-19 Diagnosis Based on Deep Learning with CT Scans

Joseph Raj AN: An integrated feature frame work for automated segmentation of COVID-19 infection from lung CT images

Prediction of disease progression in patients with COVID-19 by artificial intelligence assisted lesion quantification

Abnormal Lung Quantification in Chest CT Images of COVID-19 Patients with Deep Learning and its Application to Severity Prediction

Deep learning-based triage and analysis of lesion burden for COVID-19: a retrospective study with external validation

Analysis of clinical features and imaging signs of COVID-19 with the assistance of artificial intelligence

From community-acquired pneumonia to COVID-19: a deep learning-based method for quantitative analysis of COVID-19 on thick-section CT scans

Any unique image biomarkers associated with COVID-19?

CT quantification of pneumonia lesions in early days predicts progression to severe illness in a cohort of COVID-19 patients

Synergistic learning of lung lobe segmentation and hierarchical multi-instance classification for automated severity assessment of COVID-19 in CT images

Classification of COVID-19 by Compressed Chest CT Image through Deep Learning on a Large Patients Cohort

Diagnosis of COVID-19 using CT scan images and deep learning techniques

Development and Validation of a Machine Learning Approach for Automated Severity Assessment of COVID-19 Based on Clinical and Imaging Data: Retrospective Study

Explainable COVID-19 Detection Using Chest CT Scans and Deep Learning

A Novel Block Imaging Technique Using Nine Artificial Intelligence Models for COVID-19 Disease Classification, Characterization and Severity Measurement in Lung Computed Tomography Scans on an Italian Cohort

Association of AI quantified COVID-19 chest CT and patient outcome

Identifying COVID19 from Chest CT Images: A Deep Convolutional Neural Networks Based Approach

Deep learning-based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography. Sci Rep

Adaptive Feature Selection Guided Deep Forest for COVID-19 Classification With Chest CT

COVID-19 Chest Computed Tomography to Stratify Severity and Disease Extension by Artificial Neural Network Computer-Aided Diagnosis

Development and Validation of a Deep Learning-Based Model Using Computed Tomography Imaging for Predicting Disease Severity of Coronavirus Disease

Validation of Chest Computed Tomography Artificial Intelligence to Determine the Requirement for Mechanical Ventilation and Risk of Mortality in Hospitalized Coronavirus Disease-19 Patients in a

The study of automatic machine learning base on radiomics of non-focus area in the first chest CT of different clinical types of COVID-19 pneumonia

A Novel Machine Learning-derived Radiomic Signature of the Whole Lung Differentiates Stable From Progressive COVID-19 Infection: A Retrospective Cohort Study

Clinically Applicable AI System for Accurate Diagnosis, Quantitative Measurements, and Prognosis of COVID-19 Pneumonia Using Computed Tomography

A model based on CT radiomic features for predicting RT-PCR becoming negative in coronavirus disease 2019 (COVID-19) patients

Deep learning analysis provides accurate COVID-19 diagnosis on chest computed tomography

Prediction of COVID-19 Severity Using Chest Computed Tomography and Laboratory Measurements: Evaluation Using a Machine Learning Approach

COVID-19 Pneumonia Diagnosis Using a Simple 2D Deep Learning Framework With a Single Chest CT Image: Model Development and Validation

Artificial intelligence-enabled rapid diagnosis of patients with COVID-19

A Weakly-Supervised Framework for COVID-19 Classification and Lesion Localization From Chest CT

Deep learning-based multi-view fusion model for screening 2019 novel coronavirus pneumonia: A multicentre study

A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis

Using Artificial Intelligence to Detect COVID-19 and Communityacquired Pneumonia Based on Pulmonary CT: Evaluation of the Diagnostic Accuracy

Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets

Differentiating novel coronavirus pneumonia from general pneumonia based on machine learning

Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 from Pneumonia of Other Origin at Chest CT

Setting up an Easy-to-Use Machine Learning Pipeline for Medical Decision Support: A Case Study for COVID-19 Diagnosis Based on Deep Learning with CT Scans

Joseph Raj AN: An integrated feature frame work for automated segmentation of COVID-19 infection from lung CT images

Prediction of disease progression in patients with COVID-19 by artificial intelligence assisted lesion quantification

Abnormal Lung Quantification in Chest CT Images of COVID-19 Patients with Deep Learning and its Application to Severity Prediction

Deep learning-based triage and analysis of lesion burden for COVID-19: a retrospective study with external validation

Analysis of clinical features and imaging signs of COVID-19 with the assistance of artificial intelligence

From community-acquired pneumonia to COVID-19: a deep learning-based method for quantitative analysis of COVID-19 on thick-section CT scans

Any unique image biomarkers associated with COVID-19?

CT quantification of pneumonia lesions in early days predicts progression to severe illness in a cohort of COVID-19 patients

Additional records identified through other sources (n = 3)Records after removing duplicates (n = 663)Records screened (n = 64)

Full-text articles assessed for eligibility (n = 41)Full-text articles excluded, with reasons (n = 5)Studies included in qualitative synthesis (n = 36)