key: cord-1049790-d3ln897l
authors: Wang, Tianming; Chen, Zhu; Shang, Quanliang; Ma, Cong; Chen, Xiangyu; Xiao, Enhua
title: A Promising and Challenging Approach: Radiologists’ Perspective on Deep Learning and Artificial Intelligence for Fighting COVID-19
date: 2021-10-18
journal: Diagnostics (Basel)
DOI: 10.3390/diagnostics11101924
sha: f89df8635b89a98b0520900c65662af109dbf968
doc_id: 1049790
cord_uid: d3ln897l

Chest X-rays (CXR) and computed tomography (CT) are the main medical imaging modalities used against the increased worldwide spread of the 2019 coronavirus disease (COVID-19) epidemic. Machine learning (ML) and artificial intelligence (AI) technology, based on medical imaging fully extracting and utilizing the hidden information in massive medical imaging data, have been used in COVID-19 research of disease diagnosis and classification, treatment decision-making, efficacy evaluation, and prognosis prediction. This review article describes the extensive research of medical image-based ML and AI methods in preventing and controlling COVID-19, and summarizes their characteristics, differences, and significance in terms of application direction, image collection, and algorithm improvement, from the perspective of radiologists. The limitations and challenges faced by these systems and technologies, such as generalization and robustness, are discussed to indicate future research directions.

The rapid spread of the COVID-19 worldwide developed into a global pandemic, leading to a large-scale medical system burden and economic crisis. The detection and diagnosis of COVID-19 rely on the positive results of the reverse transcription-polymerase chain reaction (RT-PCR); however, false-negative results may occur due to the influence of various factors such as low viral load [1] . Chest imaging examinations, including CT and CXR images, are currently among the most important tools for screening and diagnosing COVID-19 patients [2] . However, the generation of a large amount of image data increases the workload of radiologists. Moreover, the massive information from radiologic images is also difficult to observe and summarize by sight. In recent years, due to the availability of high-performance computers and the large amount of data generated in the information age, the application of machine learning and artificial intelligence in many fields has made exciting new progress, which has also stimulated innovation in the development of artificial intelligence technology based on medical imaging [3] ( Figure 1 ). The widespread application of ML and AI has led to massive information in medical imaging, playing a pivotal role in the fight against the COVID-19 epidemic [4] ( Figure 2 ). However, generalization and robustness are still the main challenges of DL and AI models [5] . Sample size of datasets, nonstandard protocols of imaging, image segmentation, availability of public datasets, multimodal models combining medical imaging and clinical information, and extensive clinical validation and applications remain the main limitations.

Herein, the application, existing problems, and possible solution of machine learning and artificial intelligence systems based on medical images are introduced, such as diagnosis and classification, treatment decision-making, curative effect evaluation, prognosis prediction, and other relevant aspects for patients affected by COVID-19. By conducting a rapid review of published studies and the existing experience of DL and AI in COVID-19 radiological examinations, the aim of this article is to evaluate the feasibility and the effectiveness of available COVID-19 diagnostic tools, which may be helpful to develop better intelligent models by medical experts and technical researchers. Herein, the application, existing problems, and possible solution of machine learning and artificial intelligence systems based on medical images are introduced, such as diagnosis and classification, treatment decision-making, curative effect evaluation, prognosis prediction, and other relevant aspects for patients affected by COVID-19. By conducting a rapid review of published studies and the existing experience of DL and AI in COVID-19 radiological examinations, the aim of this article is to evaluate the feasibility and the effectiveness of available COVID-19 diagnostic tools, which may be helpful to develop better intelligent models by medical experts and technical researchers. Herein, the application, existing problems, and possible solution of machine learning and artificial intelligence systems based on medical images are introduced, such as diagnosis and classification, treatment decision-making, curative effect evaluation, prognosis prediction, and other relevant aspects for patients affected by COVID-19. By conducting a rapid review of published studies and the existing experience of DL and AI in COVID-19 radiological examinations, the aim of this article is to evaluate the feasibility and the effectiveness of available COVID-19 diagnostic tools, which may be helpful to develop better intelligent models by medical experts and technical researchers. 

A combination of three group of keywords including disease("COVID 19", "SARS-CoV-2", "NPC"), imaging examinations("CT", and "X-ray"), together with algorithm technology("machine learning", "deep learning", and "artificial intelligence") was used to search for articles in four databases: PubMed, Web of Scicence, Elsevier/SD, and IEEE; all types of research were evaluated, including case reports, original research, clinical guidelines, and systematic reviews. Through reading abstracts and reviewing the references of included studies, the published studies were finally screened and classified. We set the criteria for inclusion and classification. Only research written in English was included; preprints that were not peer-reviewed were excluded. The application and effectiveness of these systems were discussed from the perspective of radiologists. Although the types and development of algorithms were not the focus of this review, they were not excluded Figure 3 .

A combination of three group of keywords including disease("COVID 19", "SARS-CoV-2", "NPC"), imaging examinations("CT", and "X-ray"), together with algorithm technology("machine learning", "deep learning", and "artificial intelligence") was used to search for articles in four databases: PubMed, Web of Scicence, Elsevier/SD, and IEEE; all types of research were evaluated, including case reports, original research, clinical guidelines, and systematic reviews. Through reading abstracts and reviewing the references of included studies, the published studies were finally screened and classified. We set the criteria for inclusion and classification. Only research written in English was included; preprints that were not peer-reviewed were excluded. The application and effectiveness of these systems were discussed from the perspective of radiologists. Although the types and development of algorithms were not the focus of this review, they were not excluded Figure 3 . 

Chest X-ray and CT are widely used to diagnose and screen COVID-19 patients from suspected patients and make differential diagnosis from patients with other lung diseases [6] [7] [8] . Quantitative analysis of suspicious COVID-19 lesions on CT images is a basic application of DL and AI models. Zhang et al. [9] used the uAI intelligent auxiliary analysis system to accurately evaluate CT images. The system generates a Hounsfield unit (HU) histogram in the infected area to achieve precise quantification and visualization of the lesion. Their generalized linear mixed model showed that the favored site for COVID-19 pneumonia was the dorsal segment of the right lower lobe. Li et al. [10] developed a fully automated framework DL model named COVID-19 Detection Neural Network (COVNet) and analyzed chest CT images to distinguish COVID-19, community-acquired pneumonia (CAP), and other chest diseases. When detecting COVID-19 and CAP, the area under the curve (AUC) of the receiver operating characteristic (ROC) was 0.96 and 0.95, respectively. The study of Barbosa et al. achieved similar results [11] . This research shows that the DL and AI models trained by collecting large numbers of patient image datasets can efficiently screen and diagnose suspected COVID-19 patients. Jin et al. [12] reported that their multiclass AI diagnosis model based on deep convolutional neural networks (CNN) achieved an AUC of 0.9781 in the test cohort of 3199 CT scans to diagnosing 

Chest X-ray and CT are widely used to diagnose and screen COVID-19 patients from suspected patients and make differential diagnosis from patients with other lung diseases [6] [7] [8] . Quantitative analysis of suspicious COVID-19 lesions on CT images is a basic application of DL and AI models. Zhang et al. [9] used the uAI intelligent auxiliary analysis system to accurately evaluate CT images. The system generates a Hounsfield unit (HU) histogram in the infected area to achieve precise quantification and visualization of the lesion. Their generalized linear mixed model showed that the favored site for COVID-19 pneumonia was the dorsal segment of the right lower lobe. Li et al. [10] developed a fully automated framework DL model named COVID-19 Detection Neural Network (COVNet) and analyzed chest CT images to distinguish COVID-19, community-acquired pneumonia (CAP), and other chest diseases. When detecting COVID-19 and CAP, the area under the curve (AUC) of the receiver operating characteristic (ROC) was 0.96 and 0.95, respectively. The study of Barbosa et al. achieved similar results [11] . This research shows that the DL and AI models trained by collecting large numbers of patient image datasets can efficiently screen and diagnose suspected COVID-19 patients. Jin et al. [12] reported that their multiclass AI diagnosis model based on deep convolutional neural networks (CNN) achieved an AUC of 0.9781 in the test cohort of 3199 CT scans to diagnosing COVID-19, CAP, and influenza (Type A/B) and nonpneumonia lesions, and for the test in the two publicly available open-source datasets CC-CCII and MosMedData, it was 0.9299 and 0.9325, respectively. The results also indicate that the diagnostic ability of the trained AI system may be affected in external tests, and the application of AI requires learning with a larger range of data and larger samples to improve repeatability. Harmon et al. [13] adapted a DL model on CT data from a group of multinational patient cohorts to identify COVID-19, which showed an accuracy and sensitivity of 90.8% and 84%, respectively, thus suggesting that the trained AI system derived from heterogeneous multicountry training data provides acceptable performance indicators for chest CT classification of COVID-19

Diagnostics 2021, 11, 1924 4 of 16 infection. Zhang et al. [14] developed a CT-based AI diagnosis system by training and verifying the CT data of 3777 patients. Their AI approach could differentiate COVID-19 from two other classes with 92.49% accuracy, 94.93% sensitivity, 91.13% specificity, and an AUC of 0.9797, when validated on their internal dataset. The system has also shown similar diagnosis performance on three prospective test sets, open-source CT datasets, and datasets outside of China, proving the general applicability of the AI diagnostic approach.

Although CT was recommended in the diagnosis of COVID-19 at different stages of the disease in mainland China, the British Society of Thoracic Imaging and the American College of Radiology currently recommend CXR as the first line of COVID-19 imaging [15, 16] . As a result, a large amount of CXR image data is produced because X-ray images are easier to obtain and more numerous than CT images; furthermore, AI models using X-ray datasets have been extensively developed. Wang et al. [17] developed a deep learning system trained on a heterogeneous multicenter dataset containing 145,202 images to distinguish viral pneumonia from other types of pneumonia and absence of pneumonia, obtaining sensitivity, specificity, and an AUC of 87.04%, 92.94%, and 0.968, respectively. Wehbe et al. and Keidar et al. implemented comparable machine learning approaches for detection, achieving an AUC of 0.90 and 0.96, respectively [18, 19] .

DL and AI systems can extract massive amounts of valuable clinical decision information by analyzing medical images and clinical data from COVID-19 patients, thus revealing the great potential for the use of chest X-rays and CT for quantifying the lesions and assessing progression and related medical indicators. Zhang et al. [14] used quantitative measurements generated by AI to evaluate the effect of drug treatment on changes in lesion size and volume in CT scans, reporting the potential of the AI system to objectively and quantitatively evaluate the effects of drug treatment, which can assist medical staff in making management decisions for COVID-19 patients. Cai et al. [20] built random forest (RF) models for classification and regression to evaluate the disease severity (moderate, severe, and critical) and to predict by CT quantification the need and length of ICU hospital stay, the duration of oxygen inhalation, hospitalization, sputum NAT positive, and patient prognosis. The results showed that the root-mean-square error of the RF regressor to predict the length of hospital stay was 0.88 weeks, oxygen inhalation time was 0.92 weeks, and the AUC predicting ICU treatment and prognosis (partial recovery vs. prolonged recovery) were 0.945 and 0.960, respectively. The indicators and parameters of hospitalized critically ill patients include ICU treatment decisions, ICU duration, oxygen inhalation, and hospitalization, which are critical to clinical management. However, affected by the small patient sample size and the binary classification problems, the model of ICU demand has slightly worse results, and the performance of the model for predicting prognosis dropped precipitously, suggesting that the multitask model is affected by ML algorithms and data types; thus, it is difficult to maintain excellent performance in every task. Wang et al. [21] trained and verified a deep learning system that successfully stratified patients into highrisk and low-risk groups with significant differences in hospital stay time. Deep learning provides conveniently available tools for quickly identifying potentially high-risk patients, which may help optimize medical resources and provide early prevention before patients develop severe symptoms. Furthermore, Yue and colleagues [22] generated a CT radiomics models using logistic regression (LR) and random forest (RF), and effectively discriminated short-term and long-term hospital stay among patients with COVID-19; the AUC of the CT radiomics models, based on six second-order features, were 0.97 (LR) and 0.92 (RF), respectively. These studies have concluded that DL and AI are beneficial decision-makers that can provide more medical information, optimize medical resources, and help identify necessary steps for prevention before patients manifest severe symptoms.

A large number of clinical parameters and imaging data, which are generated during the diagnosis and treatment of COVID-19 patients, can be compared and correlated through DL and AI systems. Zhang et al. [14] selected three lesion indicators, total lesions (named lesions), ratio of lymphocytes (CL), and ground-glass opacity (GGO), for the entire lung field for correlation analysis, finding that C-reactive protein (CRP), age, serum lactate dehydrogenase (LDH), maximum body temperature (Tmax), and neutrophil/lymphocyte ratio had a strong positive correlation with lesion characteristics. This suggested that the AI system identified important clinical markers correlated with the COVID-19 lesion properties. Wang and his team [23] used a DL system to perform quantitative segmentation and calculation of the lesion area in the high-resolution CT image of COVID-19 patients. Comparison of the differences of the percentage of infection (POI) of whole lung and relationship between POI and clinical laboratory examination in all patients showed that the POI of the whole lung was negatively correlated with percentage of peripheral blood lymphocytes (L%) and lymphocytes (LY) count, and positively correlated with the percentage of neutrophils (N%). This indicated that both parameters could be used as prognostic indicators, having warning significance and contributing to clinical intervention for patients. Children are not completely immune to COVID-19. In order to determine whether CT is necessary for children as a vulnerable group, Ma et al. [24] tried to determine whether children required CT as a vulnerable group. They created a machine learning model based on advanced decision trees to predict the CT results of children with COVID-19 by combining clinical symptoms and laboratory data, which showed that age, lymphocytes, neutrophils, ferritin, and C-reactive protein were the most relevant clinical indicators for predicting CT results among pediatric patients with positive RT-PCR detection. The decision support system could effectively predict CT results. Therefore, their findings indicated that the use and necessity of CT for pediatric patients should be reconsidered, as the model can effectively predict CT outcomes. Fung et al. [25] proposed a self-supervised two-stage deep-learning model named SSInfNet to segment COVID-19 lesions (including ground-glass opacity and consolidation) from chest CT images. The authors identified several CT image phenotypes that mediate the potential causal relationship between underlying diseases and age with COVID-19 severity.

The effectiveness of DL and AI systems in diagnosing COVID-19 is a basic condition for clinical application and is often compared with the diagnostic performance of radiologists. Murphy et al. [26] used data from 24,678 chest X-rays for training and the other 1540 images for post-training verification to develop an AI system (CAD4COVID-XRay). The ROC analysis of the independent diagnosis performance of the six readers and the AI system showed that the AI system could correctly classify CXR images as COVID-19 pneumonia with 0.81 AUC. The system also significantly outperformed each reader at their highest possible sensitivity. It is suggested that the diagnostic performance of trained DL and AI systems may be a useful tool for radiologists, or may be used as an auxiliary tool for medical teams without radiology expertise. Wehbe et al. [18] developed an AI diagnostic system that used DeepCOVID-XR and compared the AI algorithm with the diagnostic performance of five experienced thoracic radiologists. ROC-AUC of DeepCOVID-XR was 0.88 compared with the consensus AUC of 0.85. With consensus interpretation as the reference standard, the AUC of DeepCOVID-XR was 0.95. The authors believed that DeepCOVID-XR is more reliable than the study by Murphy et al. for predictions produced by DeepCOVID-XR are in line with the radiological diagnosis by a consensus of experts. Hwang et al. [27] observed that DL-based CAD may improve the performance of readers, while the performance of nonradiologists significantly improved in the CADassisted interpretation. Furthermore, inter-reader agreement among physicians showed significant improvement when assisted with the CAD. To distinguish COVID-19 from other pneumonia, Bai et al. [28] used the EfficientNet B4 deep neural network architecture trained on data from CT scans of 1186 patients. Their results showed that the radiologists achieved a higher average accuracy (90% vs. 85%), sensitivity (88% vs. 79%), and specificity (91% vs. 88%) with the assistance of AI models. It is suggested that radiologists with the assistance of AI systems could improve the accuracy of diagnosing COVID-19. Moreover, Wang et al. [17] used a fully automated DL pipeline for the quantification of the severity scores in CXR images to evaluate the correlation between the DL model and radiologists. The results obtained showed the Pearson correlation coefficient was 0.81, and the AI system could predict the severity of COVID-19 pneumonia with an AUC of 0.868, indicating that the severity index of radiologists and the AI system had a strong linear relationship. Furthermore, the DL system showed that its diagnostic performance was comparable to that of senior radiologists for the CXR test set, while it also improved the performance of junior radiologists.

It can be seen from these studies that the detection and diagnosis performance of AI models based on COVID-19 CXR or CT images is similar to that of experienced radiologists and may help improve the reliability of physicians' diagnoses, especially for junior radiologists and nonradiologists.

Objectively quantifying the degree of disease by the percentage of COVID-19 lesions in the lung parenchyma is currently the most important application of AI in the evaluation of the COVID-19 patient, which can contribute to evaluating and monitoring the progression of the disease. In order to further analyze the clinical and radiological characteristics that lead to critical illness, it is necessary to develop artificial-intelligence-assisted models to estimate clinical prognosis. Shan et al. [29] used the VB-Net neural network DL to segment the COVID-19 infection area on the CT scan and compared three indicators, the Dice similarity coefficient, volume difference, and percentage of infection (POI), between automatic and manual segmentation. This DL-based segmentation system produced a Dice similarity coefficient of 91.6% ± 10.0% between automatic and manual segmentation when using multiple lung lobes and bronchopulmonary segment infections. Additionally, when quality (MOI) was used as a feature for severity prediction, the best accuracy of severity prediction was 73.4% ± 1.3%, indicating that the quantitative techniques of the DL system have potential clinical application value in predicting the severity of COVID-19. Li et al. [30] developed an algorithm based on a convolutional Siamese neural network to measure the severity of COVID-19 on CXR automatically. Their results showed that the pulmonary X-ray severity (PXS) score was correlated with the radiological chest disease severity score of COVID-19 on CXR in the internal and external test sets, and the ROC-AUC of PXS score predicting intubation or death within three days after admission was 0.80 (95% CI 0.75-0.85). A similar method is well established and used in other studies such as that reported. Lessmann et al. [31] adapted a deep learning algorithm system called CORADS-AI and followed a standardized reporting scheme to assign CO-RADS scores for suspected COVID-19 and identify COVID-19 by chest CT examinations, where the AUC of the internal cohort and the external cohort were 0.95 and 0.88, respectively, and the CT severity score was assigned to the extent of parenchymal involvement of each lobe, which was consistent with the readings of eight independent human observers. This proposed method verified the accuracy of the AI system and compared it with human observers, which avoids nonstandardized and uncalibrated output and enhances the interpretability of the output. The DL system developed by Fang et al. effectively combined the dynamic sequence of chest CT scans and static clinical information from 1040 COVID-19 patients with mild symptoms at different times after hospitalization. Their results showed that the system could effectively identify valuable indicators to predict the malignant progression of COVID-19, with an average AUC of 0.920 in the single-center study and average AUC of 0.874 in the multicenter study [32] . Zhang's team conducted research based on the quantitative chest lesion features and clinical parameters extracted by the AI system. Their results showed that the high-risk group (C-score ≥ 0.5) had a much lower survival probability and a lower risk compared to the low-risk group (C-score < 0.5), indicating that the combination of chest imaging data and clinical data could significantly promote prognostic evaluation [14] .

Although numerous studies report the efficacy evaluation and prognosis prediction of COVID-19 based on imaging, there are relatively few studies on imaging risk factors for poor prognosis of COVID-19 [33, 34] . Research by Bartolucci et al. analyzed the volume and texture of the lesion area in the CT of 115 COVID-19 patients; results suggested that the mixed radiology model, including the percentage of pulmonary consolidation, performed significantly better (AUC = 0.82) compared to the blood-laboratory-arterial-gas analysis feature alone (AUC = 0.71) in the validation set predicting ICU admission [33] . This suggested a mild but significant incremental value in predicting ICU occupancy by evaluating the volume of the consolidated lung in CT of COVID-19 patients. Yu et al. [34] used a dedicated multitask DL algorithm to build an AI system to quantitatively analyze the volume, density, location, GGO, and substance of the patient's lesions, after which they used multivariable logistic regression to evaluate the imaging features and risk factors related to prognostic endpoints (admission to ICU, acute respiratory failure occurrence, or shock during hospitalization). Their results showed that older age at admission and a larger area of consolidation in the upper lung was more closely related to the poor prognosis of COVID-19 patients. In addition, Liang et al. [35] developed a deep learningbased survival model for predicting the risk of critical illness in COVID-19 patients based on clinical characteristics (X-ray abnormalities included) at the time of admission. Their results showed that internal verification performance (0.894) and external verification of three independent cohorts (0.890, 0.852, and 0.967) achieved good consistency indexes. This suggested that the AI model could be used to classify patients at admission and identify patients at risk of serious disease.

DL and AI systems can be used to monitor and quantify disease progression and understand the time evolution characteristics of COVID-19 lesions, reducing the subjectivity of radiologists in comparing chest CT images before and after treatment. They are also conducive to accurate treatment.

Although there are currently many studies focusing on the evaluation of the radiology and clinical physiological outcomes of recovered COVID-19 patients [31, 36] , there are few studies using DL and AI models of radiological images for post-discharge follow-up and recovery evaluation. Meng et al. [37] applied a novel DL technique by learning the chest CT scan features of 270 hospitalized COVID-19 patients with two consecutive negative RT-PCR tests (sampling interval > 1 day). The discharge standard set by DL in the study is that the total lesions are less than 50% of the total lung volume. The results showed there are no positive test results or progression of pneumonia among the 230 discharged cases, thus achieving the aim to standardize the discharge criteria at a "square cabin" hospital; furthermore, the study found that vascular dilatation has the potential to predict the shortterm prognosis of COVID-19 patients. Hu et al. [38] calculated the degree of pulmonary fibrosis with the volume of fibrosis on the chest CT image using an AI assistant program machine. The basic inflammatory molecular levels at admission and after discharge were compared. The results showed that plasma interferon-γ(IFN-γ) levels were twofold lower than those without fibrosis, and there was a negative correlation between pulmonary fibrosis and basal circulating IFN-γ levels in the recovery phase. It shows that the decrease of IFN-γin circulation is a risk factor for pulmonary fibrosis of COVID-19.

Many biological data have been accumulated with the spread of the COVID-19 epidemic and numerous infected patients generating epidemic contact history, clinical data, laboratory examinations, and imaging data. Researchers have constructed various multi-modal DL and AI systems from different angles to fully mine the information contained therein, and then transform the data into reliable and referable information in the diagnosis and treatment of COVID-19 [39] . Li et al. [40] reported two false-negative results of rRT-PCR for SARS-CoV-2 infection and mentioned that a comprehensive method combining DL, CT features, and rRT-PCR results might be used to ensure early screening and accurate diagnosis of COVID-19 in clinical practice. Mei et al. [41] used AI algorithms by integrating chest CTs with clinical symptoms, exposure history, and laboratory tests to quickly diagnose COVID-19 patients, revealing that the joint model outperformed the model trained on CT images only and the model trained on clinical information only. Furthermore, the AI model also correctly identified several COVID-19 patients with a normal CT scan who were misdiagnosed by radiologists (17/25) and confirmed by rRT-PCR. Purkayastha et al. [42] collected information from 981 patients with COVID-19 from a multi-institutional international cohort to extract radiomics features of chest CTs and clinical information; they trained a machine learning system to predict severity and time-to-event model to predict progression to critical illness in combination with CT imaging omics and clinical variables. The model successfully predicted the risk of progression on the third, fifth, and seventh day, which suggested that a multimodal combination of CT radiomics characteristics and clinical variables could predict the severity and progression of COVID-19 to critical illness with high accuracy. Moreover, Schaffino et al. [43] used machine learning models to assess lung parenchyma and vascular damage. Their results showed that the best models of SVM and MLP considered the same ten input features, yielding an AUC of 0.747 and 0.844, respectively. Furthermore, in this model that integrated clinical and radiological data, pulmonary artery diameter was the third-largest predictor after age and physical involvement, which helped to reliably predict mortality during hospitalization, underscoring the value of vascular indicators for improving severity stratification patients. Jiao et al. [44] examined an EfficientNet deep neural network based on CXR images to estimate the disease severity and progression. When deep learning features were added to DL system of clinical data which used to predict progress, the concordance index (C-index) of both internal and external tests increased. The prognostic performance of the image and clinical data combination model was significantly better than the severity score and clinical data combination on the internal and external tests. It can be concluded that the performance of the multimodal model fused with imaging and clinical data is significantly better than that of models based solely on the image or clinical data.

Due to the lack of standard requirements for datasets, there are significant differences in the dataset sources used by different studies and the image scanning conditions of each medical center or database. Researchers studied the performance of various algorithms under different data sources. Shiri et al. [45] used ultralow-dose and full-dose CT images as input/output for training, testing, and external validation sets to implement full-dose prediction technology. It was found that the radiation dose expressed by CT Dose Index (CTDIvol) was reduced by 89%, and the DL algorithm could predict standard full-dose CT images with acceptable quality for clinical diagnosis of COVID-19 patients, while radiation dose could be effectively reduced, the deep learning algorithm failed to restore the correct lesion structure density for some patients who were regarded as outliers. Therefore, further research and development are necessary to solve these limitations. Sengupta et al. [46] developed a new quantum neural network (QNN) algorithm, where the system running time observed on quantum-optimized hardware was 52 min, while on K80 GPU hardware, the observed model running time was 1 h and 30 min for similar sample sizes. Simulation experiments showed that the performance of QNN in accuracy measurement was better than DNN, CNN, and 2D CNN by more than 2.92%, while the average recall rate was about 97.7%. Rapid clinical prognostic analysis could be achieved by using CT images. Wu et al. [47] proposed a weakly supervised deep active learning framework called COVID-AL to diagnose COVID-19. A hybrid sampling strategy was used to select samples with high uncertainty and diversity in each round of active learning, which greatly reduced the cost of training models for manually labeled chest CT scans. Wang et al. [48] developed a deep learning system and radiomics models for predicting and classifying COVID-19 and non-COVID-19 viral pneumonia. Their results indicated that both models achieved > 73% sensitivity and >75% specificity in the external validation cohort, while the performance of the radiomics lasso classifier was slightly superior. The diagnostic performance of human experts improved when the joint deep learning-radiomics model was used. Transfer learning and data enhancement were common strategies to combat data scarcity in deep learning algorithms among the existing literature, and it retrains the deep model on a large-scale dataset and fine-tunes it on the target image set of COVID-19 [49, 50] . In order to further evaluate the discriminative ability of different models, ensemble learning is deployed, and multiple deep networks are used to evaluate the final results. Fouladi et al. [51] proposed ResNet-50, VGG-16, Convolutional Neural Network (CNN), Convolutional Autoencoder Neural Network (CAENN), and Machine Learning (ML) methods to classify and compare chest CT images of COVID-19 patients; nearest neighbor (NN) had the highest performance with an accuracy of 94%. The limitation of this study was that it did not use image preprocessing and data enhancement techniques, and the use of pre-trained networks or data enhancement may improve accuracy. Since direct transmission between datasets from different fields may lead to poor performance, researchers have developed various strategies to mitigate the impact of domain differences on transmission performance [52] .

The most common use of deep learning and artificial intelligence based on medical images is for diagnosing and evaluating COVID-19 patients. In addition to using medical image data (Table 1) , DL and AI are also widely used in tasks related to patient outcomes, such as the diagnosis, assessment of severity, prediction of death risk, and length of hospital stay of COVID-19 patients. Imran et al. [53] developed an AI4COVID-19 system to diagnose COVID-19 by recording three 3 s cough sounds and sending them to an AI engine running in the cloud. Their results showed that the system could distinguish between COVID-19 coughs and several non-COVID-19 coughs to screen for COVID-19. DL and AI technologies have also had a vital role in epidemic monitoring and development prediction [54, 55] . Ribeiro and colleagues [56] developed an effective short-term prediction model for forecasting the number of short-term cases in Brazil. Compared with the comparative model, support vector regression (SVR) and stacked integrated learning achieved better adoption of the adopted standards. Based on a hybrid method using an autoregressive integrated moving-average model together with a wavelet-based prediction model, a real-time prediction model was proposed to predict future epidemics in different countries, which may help with the effective allocation of medical resources and serve as an early warning reference for government decision-makers [57] . In addition, it can also be used for epidemiological investigation and tracking of close contacts, discovery or reuse of drugs and vaccines, evaluation of their safety, and optimization of treatment programs [58] . The potential capabilities of DL and AI in exploring COVID-19 vaccine antibodies and therapeutic drugs are unparalleled and can accelerate the reuse and development of COVID-19 drugs [59] [60] [61] . Ong summarized the research on deep learning for COVID-19 and applied Vaxign and the newly developed machine-learning-based Vaxign-ML reverse vaccinology tool to predict COVID-19 vaccine candidates [62] . Researchers from the United States and South Korea jointly proposed a new molecular converter-drug target interaction model to solve the needs of antiviral drugs for the treatment of COVID-19 [63] . 

After the COVID-19 outbreak, data-driven DL and AI research made a breakthrough in the field of medical imaging. These studies all claimed that these algorithms could diagnose or predict COVID-19 based on CXR and CT images. Although some medical centers or medical imaging alliances have disclosed some COVID-19 dataset resources [64] ( Table 2) , there is low image quality, inconsistent CT scan parameter settings, small size samples, irregular data labeling with strong subjectivity, and dataset sources that are not uniform. Due to nonstandard datasets, algorithm bias, nonrepeatability, lack of sufficient external and forward-looking verification, and difficulty in simultaneously considering CT and CXR issues, it is difficult to clinically verify DL and AI models in complex cases and actual open scenarios, thus limiting their clinical application. Generalization and robustness have become the key and urgent problems that limit the practical application of these data-driven DL and AI model systems. Due to these limitations and challenges, most of the models in the research have no practical and clear application value and have not become the final solution for the current clinical diagnosis and treatment of COVID-19, but there are some relatively practical and more explanatory resources applied by radiologists [65] . Currently, most COVID-19 imaging is made without artificial intelligence and that human sight is remains necessary [66] . With increasing research on the application of AI systems in COVID-19, there are increasingly more studies using similar methods.Partial code access of COVID-19 DL and AI models based on radiology images were listed in Appendix A. Overall, the number of high-quality studies is relatively small, and the lack of external verification or prospective verification is a common flaw. Most studies tend to collect data from multiple institutions for testing, but lack external verification of data from multiple institutions and datasets in order to improve generalization of the AI system. The performance of models using CT data is usually better than models using CXR data; the main reason was considered to be its inherent defect of CXR images. Ordinary CXR images are the sum of the effects of X-rays on all tissues between the X-ray source and the capture membrane; compared with CT images, the tissue structure in X-rays is unclear and lacks three-dimensional information. All current models or algorithms are evaluated based on their performance. The performance of some models has been improved by improving algorithms, but these improvements are not always effective. There is no algorithm that always performs better than other algorithms and is suitable for all application types and models. The diagnostic performance of the AI system of Zhang et al. in external verification is inferior to its performance in internal testing, and the model used to assess disease prognosis is also inferior to internal testing [14] , thus, there is no best fir, only a better fit. When the sample size is sufficiently large, strategies such as transfer learning will affect the calculation efficiency and generalization ability of the system. More advanced models and algorithms provided by engineers and technicians are looked forward to in the future.

The construction of a highly standardized database is essential to improve the accuracy and generalization of the model. In addition, the development of DL and AI algorithms greatly depend on the close communication between engineer, radiologist, and clinicians, and strengthened multidisciplinary communication and cross-complementation. The establishment of multilabel and/or hierarchical classification technology evaluation and benchmark solutions will greatly promote the prevention and control of infectious diseases.

Since the outbreak of COVID-19, numerous researchers worldwide have extensively studied how to quickly and correctly diagnose COVID-19 and evaluate and control the condition. Machine learning and artificial intelligence with high reliability and huge potential in various applications were adopted by numerous healthcare providers. From multiple angles, this article introduced studies that explored DL and AI technologies based on medical imaging data so as to solve the troubles and challenges encountered in reality and directly point out the limitations of current studies. Overcoming these challenges may lead to significant progress in the use of artificial intelligence technology to combat COVID-19 and future epidemics.

Author Contributions: Conceptualization, T.W., Z.C., Q.S. and E.X.; writing-original draft preparation, T.W.; writing-review and editing, Z.C., Q.S., C.M., X.C. and E.X.; supervision and project administration, E.X. All authors have read and agreed to the published version of the manuscript.

Funding: This research received no external funding.

Informed Consent Statement: Not applicable.

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

COVID-19 diagnosis-A review of current methods

Relationship to Duration of Infection

A primer for understanding radiology articles about machine learning and deep learning

Deep learning in biomedicine

Current status and future trends of clinical diagnoses via image-based deep learning

Diagnostics 2021

Pediatric chest X-ray in COVID-19 infection

The indispensable role of chest CT in the detection of coronavirus disease 2019 (COVID-19)

Thoracic imaging tests for the diagnosis of COVID-19

Automated detection and quantification of COVID-19 pneumonia: CT imaging analysis by a deep learning-based software

Using Artificial Intelligence to Detect COVID-19 and Community-acquired Pneumonia Based on Pulmonary CT: Evaluation of the Diagnostic Accuracy

Machine learning automatically detects COVID-19 using chest CTs in a large multicenter cohort

Development and evaluation of an artificial intelligence system for COVID-19 diagnosis

Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets

Clinically Applicable AI System for Accurate Diagnosis, Quantitative Measurements, and Prognosis of COVID-19 Pneumonia Using Computed Tomography

ACR Recommendations for the use of Chest Radiography and Computed Tomography (CT) for Suspected COVID-19 Infection American College of Radiology

A deep-learning pipeline for the diagnosis and discrimination of viral, non-viral and COVID-19 pneumonia from chest X-ray images

COVID-19 classification of X-ray images using deep neural networks

CT Quantification and Machine-learning Models for Assessment of Disease Severity and Prognosis of COVID-19 Patients

A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis

Machine learning-based CT radiomics method for predicting hospital stay in patients with pneumonia associated with SARS-CoV-2 infection: A multicenter study

Correlation between lung infection severity and clinical laboratory indicators in patients with COVID-19: A cross-sectional study based on machine learning

Can Clinical Symptoms and Laboratory Results Predict CT Abnormality? Initial Findings Using Novel Machine Learning Techniques in Children With COVID-19 Infections

Self-supervised deep learning model for COVID-19 lung CT image segmentation highlighting putative causal relationship among age, underlying disease and COVID-19

COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System

COVID-19 pneumonia on chest X-rays: Performance of a deep learning-based computer-aided detection system

Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 from Pneumonia of Other Origin at Chest CT

Abnormal lung quantification in chest CT images of COVID-19 patients with deep learning and its application to severity prediction

Automated assessment of COVID-19 pulmonary disease severity on chest radiographs using convolutional Siamese neural networks

Automated Assessment of COVID-19 Reporting and Data System and Chest CT Severity Scores in Patients Suspected of Having COVID-19 Using Artificial Intelligence

Deep learning for predicting COVID-19 malignant progression

The incremental value of computed tomography of COVID-19 pneumonia in predicting ICU admission

Multicenter cohort study demonstrates more consolidation in upper lungs on initial CT increases the risk of adverse clinical outcome in COVID-19 patients

Early triage of critically ill COVID-19 patients using deep learning

Analysis of thin-section CT in patients with coronavirus disease (COVID-19) after hospital discharge

Novel Deep Learning Technique Used in Management and Discharge of Hospitalized Patients with COVID-19 in China

Lower Circulating Interferon-Gamma is a Risk Factor for Lung Fibrosis in COVID-19 Patients. Front

Machine learning is the key to diagnose COVID-19: A proof-of-concept study

False-Negative Results of Real-Time Reverse-Transcriptase Polymerase Chain Reaction for Severe Acute Respiratory Syndrome Coronavirus 2: Role of Deep-Learning-Based CT Diagnosis and Insights from Two Cases

Artificial intelligence-enabled rapid diagnosis of patients with COVID-19

Machine Learning-Based Prediction of COVID-19 Severity and Progression to Critical Illness Using CT Imaging and Clinical Data

Machine Learning to Predict In-Hospital Mortality in COVID-19 Patients Using Computed Tomography-Derived Pulmonary and Vascular Features

Prognostication of patients with COVID-19 using artificial intelligence based on chest X-rays and clinical data: A retrospective study

Ultra-low-dose chest CT imaging of COVID-19 patients using a deep residual neural network

Quantum algorithm for quicker clinical prognostic analysis: An application and experimental study using CT scan images of COVID-19 patients

The diagnosis of COVID-19 with deep active learning

Decoding COVID-19 pneumonia: Comparison of deep learning and radiomics CT image signatures

COVID-19 diagnosis from chest X-ray images using transfer learning: Enhanced performance by debiasing dataloader

Data Augmentation Using Auxiliary Classifier GAN for Improved Covid-19 Detection

Efficient deep neural networks for classification of COVID-19 based on CT images: Virtualization via software defined radio

AI-Empowered Computational Examination of Chest Imaging for COVID-19 Treatment: A Review. Front

AI4COVID-19: AI enabled preliminary diagnosis for COVID-19 from cough samples via an app

COVID-19 and the epistemology of epidemiological models at the dawn of AI

Time series forecasting of COVID-19 transmission in Canada using LSTM networks

Short-term forecasting COVID-19 cumulative confirmed cases: Perspectives for Brazil

Real-time forecasts and risk assessment of novel coronavirus (COVID-19) cases: A data-driven analysis

Review of Big Data Analytics, Artificial Intelligence and Nature-Inspired Computing Models towards Accurate Detection of COVID-19 Pandemic Cases and Contact Tracing

Deep Learning Approach for Discovery of In Silico Drugs for Combating COVID-19

Review of COVID-19 Antibody Therapies

Artificial intelligence in COVID-19 drug repurposing

COVID-19 Coronavirus Vaccine Design Using Reverse Vaccinology and Machine Learning

Artificial intelligence approach fighting COVID-19 with repurposing drugs

Available online

Chest CT in COVID-19 at the ED: Validation of the COVID-19 Reporting and Data System (CO-RADS) and CT Severity Score

Radiology indispensable for tracking COVID-19

We would like to convey our greatest respects to the medical personnel who are fighting COVID-19.

The authors declare no conflict of interest.Diagnostics 2021, 11, 1924