key: cord-0559558-r5wtvkg1 authors: Ali, Sarwan; Zhou, Yijing; Patterson, Murray title: Efficient Analysis of COVID-19 Clinical Data using Machine Learning Models date: 2021-10-18 journal: nan DOI: nan sha: 765c70098cba9ce6b3f71501c0718b6fd05b6675 doc_id: 559558 cord_uid: r5wtvkg1 Because of the rapid spread of COVID-19 to almost every part of the globe, huge volumes of data and case studies have been made available, providing researchers with a unique opportunity to find trends and make discoveries like never before, by leveraging such big data. This data is of many different varieties, and can be of different levels of veracity e.g., precise, imprecise, uncertain, and missing, making it challenging to extract important information from such data. Yet, efficient analyses of this continuously growing and evolving COVID-19 data is crucial to inform -- often in real-time -- the relevant measures needed for controlling, mitigating, and ultimately avoiding viral spread. Applying machine learning based algorithms to this big data is a natural approach to take to this aim, since they can quickly scale to such data, and extract the relevant information in the presence of variety and different levels of veracity. This is important for COVID-19, and for potential future pandemics in general. In this paper, we design a straightforward encoding of clinical data (on categorical attributes) into a fixed-length feature vector representation, and then propose a model that first performs efficient feature selection from such representation. We apply this approach on two clinical datasets of the COVID-19 patients and then apply different machine learning algorithms downstream for classification purposes. We show that with the efficient feature selection algorithm, we can achieve a prediction accuracy of more than 90% in most cases. We also computed the importance of different attributes in the dataset using information gain. This can help the policy makers to focus on only certain attributes for the purposes of studying this disease rather than focusing on multiple random factors that may not be very informative to patient outcomes. cases. We also computed the importance of different attributes in the dataset using information gain. This can help the policy makers to focus on only certain attributes for the purposes of studying this disease rather than focusing on multiple random factors that may not be very informative to patient outcomes. Keywords COVID-19 · Coronavirus · Clinical Data · Classification · Feature Selection Because of the rapid global spread of COVID-19, and the cooperation of medical institutions worldwide, a tremendous amount of public data -more data than ever before for a single virus -has been made available for researchers [14, 9, 5] . This "big data" opens up new opportunities to analyse the behavior of this virus [21, 20] . Despite these opportunities, the huge size of the data poses a challenge for its processing on smaller systems [9] . On one hand, this creates scalability issues, and on the other hand, it creates the problem of high dimensionality (the curse of dimensionality) [8, 7] . Since such data was not previously available to the research community at this magnitude and ease of access, new and more sophisticated methods are required to extract useful information from this big data. At the same time, the shortage of medical resources may occur when such a severe pandemic happens, and the surging number of patients exceeds the capacity of the clinical system. This situation happens in many countries and regions during continuous outbreaks of the COVID-19 pandemic, and clinicians have to make the tough decision of which individual patients have a higher possibility to recover and should receive a limited amount of medical care. What is more difficult is the decision of which patients have little to no chance of survival, regardless of treatment level, and should hence be abandoned for the sake of optimizing the use of limited resources for others who still have a chance. In addition to this, patients with different levels of severity and urgency of symptoms require the medical system to create a complete plan to provide various treatments in proper order [1] . Hence, a clinical decision support system is of utmost importance to optimize the use of the limited medical resources and thus save more lives overall [1] . In order to develop a such a clinical decision support system with high quality, it is necessary to build a model that can predict the possible complications of patients, assessing the likelihood that they will survive under certain levels of care. Machine learning (ML) based algorithms are proven to perform well in terms of classification and clustering. Therefore, we work on building machine learning models that can scale to larger datasets and reduce the run time by selecting the proper attributes. Since ML models take a feature vector representation as input [11, 15] , designing such vectors while preserving a maximum amount of information is a challenging task [33] . Moreover, when the size of data becomes this large, even scalability of these models becomes an issue. Much work has been done on genomic data of COVID-19 patients [19, 6, 26] . One major challenge in this case is the conversion of genomic sequences into fixed-length numerical feature vectors so that they can be given as input to the underlying ML classifiers for prediction purposes. In this paper, we propose an algorithm that efficiently predicts with high accuracy patient mortality as a function many different factors. This problem can help doctors to prescribe medications and design strategies in advance that can help to save the highest number of lives. In this paper, our contributions can be summarised as follows: 1. We propose a pipeline to efficiently predict patient mortality as a function of a few dozen factors. We show that with basic information about a patient (gender, race, exposure, etc.), we can predict in advance the likelyhood of a mortality in the future. We also predict if a patient is COVID-19 positive or negative using attributes like red blood cells and hemoglobin, etc. 2. We show that our model is scalable on larger dataset (achieves accuracies >90%). 3. From our results, it is evident that the proposed model (using efficient feature selection) outperforms the baselines (without using any feature selection) in terms of prediction accuracy and runtime. 4. We show the importance of each attribute by measuring the information gain of the attributes with the class labels. This study can help doctors and other relevant authorities to focus more on specific attributes rather than dealing with all information at once, which can be difficult for humans to fathom. The rest of the paper is organised as follows: Section 2 contains literature review for the problem. Our proposed model is give in Section 3. Dataset statistics and experimental details are given in Section 4. We show results and their comparisons in Section 5. We discuss the importance of different attributes in Section 6. Finally, in Section 7, we conclude our paper. Machine learning based models that take fixed length feature vectors as input has been successfully applied (for data analytics tasks) in many domains such as graphs analytics [11, 2] , smart grid [7, 8] , electromyography (EMG) [32] , and text classification [29, 31, 30] . It is important to perform objective evaluation of the underlying model rather than just doing subjective evaluation [16] . Many methodologies of data science have been applied to objectively analyze the data of COVID-19 and provide support to the medical system. The synergy between data scientists and the biomedical communities is helpful to improve the detection of diseases and illnesses, as well as the prediction of possible complications. Authors in [22] developed a framework with to predict cancer trends accurately. This type of framework could be used for the analysis of other clinical data. S. Ali et al. in [10] uses spike sequences to classify the variants of the COVID-19 infected humans. An effective approach to cluster the spike sequences based on the virus's variants is conducted in [6] . Several studies discussed different data mining techniques to study the behavior of the COVID-19 [23, 4, 21] . Authors in [13] uses neural networks, which takes advantage of few-shot learning and autoencoder to perform predictive analysis on COVID-19 data. A study for predicting the clinical outcomes of patients and indicating whether patients are more likely to recover from coronavirus or in danger of death is performed in [20] . They presented a tool called online analytical processing (OLAP), which can help the researchers learn more about the confirmed cases and mortality of COVID-19 by conducting machine learning methods on the big dataset of COVID-19. Most of the machine learning (ML) models take fixed-length feature vectors as an input to perform different tasks such as classification and clustering. We design a fixed-length feature vector representation, which includes the values of different attributes of the clinical data. One important point to mention here is that not all the features in the vectors are important in terms of predicting the class labels. Therefore, it is required to apply feature selection to not only improve the predictive performance of the underlying classifiers (by removing unnecessary features), but also improve the training runtime. The feature selection methods that we used in this paper are discussed below. We use different supervised and unsupervised feature selection methods to improve the runtime of the underlying classifiers and also to improve the predictive performance. For supervised models, we use Boruta (shadow features) [18] , and Ridge Regression (RR) [17] . For unsupervised methods, we use Approximate kernel approach called Random Fourier Features (RFF) [28] . The main idea of Boruta is that features do not compete among themselves but rather they compete with a randomized version of them. Boruta captures the non-linear relationships and interactions using the random forest algorithm. It then extract the importance of each feature (corresponding to the class label) and only keep the features that are above a specific threshold of importance. To compute the importance of the features, it performs the following task: From the original features set in the data, it creates dummy features (shadow features) by randomly shuffling each feature. Now the shadow features are combined with the original features set to obtain a new dataset, which has twice the number of features of the original data. Using random forest, it compute the importance of the original features and the shadow features separately. Now the importance of the original features are compared with the threshold. The threshold is defined as the highest feature importance recorded among the shadow features. The feature from the original feature set is selected is its importance (computed using random forest) is greater than the threshold (highest importance value among shadow features). In Boruta, a feature is useful only if it is capable of doing better than the best randomized feature. Note that we are using two datasets in this paper, namely Clinical Data1, and Clinical Data2 (see Section Ridge Regression (RR) is a supervised algorithm for parameter estimation that is used to address the collinearity problem that arises in multiple linear regression frequently [25] . Its main idea is to increase the bias (it first introduces a Bias term for the data) to improve the variance, which shows the generalization capability of RR as compared to simple linear regression. RR ignores the datapoints that are far away from others and it try to make the regression line more horizontal. RR is useful for Feature selection because it gives insights on which independent variables are not very important (can reduce the slope close to zero). The un-important independent variables are then removed to reduce the dimensions of the overall dataset. The objective function of ridge regression is the following where α × slope 2 is called penalty terms. A popular approach for the classification is using kernel based algorithms, which computes a similarity matrix that can be used as input for traditional classification algorithms such as support vector machines. However, pair-wise computation for the kernel matrix is an expensive task. To make this task efficient, an efficient approach, called kernel trick is used. It works by taking dot product between the pairs of input points. Kernel trick avoid the need to map the input data (explicitly) to a high-dimensional feature space. The main idea of the Kernel Trick is the following: Any positive definite function f(x,y), where x, y ∈ R d , defines an inner product and a lifting φ for the purpose of computing the inner product quickly between the lifted data points [28] . More formally: The main problem of kernel method is that when we have large sized data, they suffers from large initial computational and storage costs. To solve these problems, we use an approximate kernel method called Random Fourier Features (RFF) [28] . The RFF maps the given data to a low dimensional randomized feature space (euclidean inner product space). More formally: RFF basically approximate the inner product between a pair of transformed points. More formally: In Equation (4), z is low dimensional (unlike the lifting φ). In this way, we can transform the original input data with z. Now, z is the approximate low dimensional embedding for the original data. We can then use z as the input for different classification algorithms. For classification, we use Support Vector Machine (SVM), Naive Bayes (NB), Multiple Linear Regression (MLP), K-Nearest Neighbors (KNN), Random Forest (RF), Logistic Regression (LR), and Decision Tree (DT). All algorithms are used with default parameters. The value for K in case of KNN is taken as 5 (using standard validation set approach [12] ). We are also using a deep learning model, called Keras Classifier for the classification purposes. For this model, we use a sequential constructor. We create a fully connected network with one hidden layer that contains p neurons, where p is equal to the length of the feature vector. We use "rectifier" as an activation function and "softmax" activation function in the output layer. We also use ab efficient Adam gradient descent optimization algorithm with "sparse categorical crossentropy" loss function (used for multi-class classification problem), which computes the crossentropy loss between the labels and predictions. The batch size and number of epochs are taken as 100 and 10, respectively for training the model. Since deep learning model does not require any feature selection, we use the original data without any feature selection as input to keras classifiers. We use "sparse categorical crossentropy" instead of simple "categorical crossentropy" because we are using integer labels rather than one-hot representation of labels. In this section, we describe our dataset in detail. All experiments are performed on a Core i5 system running the Windows operating system, 32GB memory, and a 2.4 GHz processor. Implementation of the algorithms is done in Python. Our code and prepossessed dataset is available online 1 . In this paper, we are using clinical data from two different sources. The description of both datasets is given below. We use COVID-19 Case Surveillance dataset (we call it Clinical Data1 for reference), which is publicly available on center of disease control CDC, USA's website 2 . After preprocessing (removing missing values), we got 95984 patients data record. The attributes in the dataset are following: States of America) in which the patient is living (see Figure 2 for states distribution). Table 1 for the distribution of values for race attribute). [24] . We maps input data to 2d real vectors representation using t-SNE and Deceased attribute (for Clinical Data1) as class label (see Figure 4 ). We can observe in the figure that there is no visible clustering corresponding to different values of the deceased attribute. All values (No, Yes, and Unknown) are scattered around in the whole plot. This behavior shows that performing any ML task on such data will not directly give us efficient results (since the data is not properly grouped together). We got the Clinical Data2 data from [3] . This study used a laboratory dataset of patients with COVID-19 in the Israelita Albert Einstein Hospital in Sao Paulo, Brazil. The patient samples were collected to identify who were infected by COVID-19 in the beginning of year 2020. The laboratory dataset contains information on 608 patients with 18 laboratory findings. In this dataset, 520 To measure the performance of underlying machine learning classifiers, we use different evaluation metrics such as Average Accuracy, Precision, Recall, weighted and Macro F1, and Receiver Operator Curve (ROC) Area Under the Curve (AUC). We also computed the training runtime of all ML models to see which model is the best in terms of runtime. The results for Clinical Data1 are given in Table 3 . For classifying the Deceased attribute, we can see that all methods are able to classify the label (Deceased attribute) with very high accuracy (accuracy > 90 in most of the cases). Note that feature selection based models are not only better in terms of prediction accuracy, but also outperforms the setting in which we are not using any feature selection approach (No Feat. Selec.). Also, Boruta feature selection model is outperforming all other feature selection approaches. In terms of training runtime, RFF with Logistic Regression classifier is performing better than the other classifiers. The results for Clinical Data2 are given in Table 4 . For classifying whether a patient is COVID-19 positive or negative, we can see that random forest classifier with Boruta feature selection approach outperforms all other feature selection methods and also the deep learning model. In terms of runtime, logistic regression classifier with RFF is outperforming other approaches. We note that the deep learning model is slightly worse that the traditional classifiers in case of Clinical Data1 while performing worst on Clinical Data2. This is because the deep learning models are usually "Data Hungry" and require a lot more data to efficiently learn the patters in the data. Since we have small number of data points in both datasets, the deep learning model is unable to beat the traditional classification algorithms. To evaluate importance positions in spike sequences, we find the importance of each attribute with respect to class label (for Clinical Data1). For this purpose, we computed the Information Gain (IG) between each attribute and the true class label. The IG is defined as follows: where H is the entropy, and p i is the probability of the class i. The IG values for each attribute is given in Figure 5 . What is particularly interesting is that State and County code of the are two major predictors of patient outcome. This is likely due to the current vaccination situation in the US, which varies quite widely from county to county [27] . 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 We propose an efficient model for the classification of COVID-19 patients using efficient feature selection methods and machine learning classification algorithms. We show that with Boruta for feature selection, the simple classification algorithms like random forest can also beat the deep learning model when dataset size is not too big. We also show the importance of each attribute in the Clinical Data1 by computing the information gain values for each attribute corresponding to class label. In the future, we will extract more data and apply other sophisticated deep learning models such as LSTM and GRU to improve the predictive performance. We will also use other factors such as weather along with the clinical data to further improve the classification results. These results have many practical meanings. The most direct real-world application of the machine learning model is to provide support to medical doctors during the COVID-19 pandemic. By predicting the risk level of individual patients, our model enables clinicians to wisely assign, in real-time, limited medical resources, especially during periods of medical shortage, and provide immediate treatment to the most vulnerable groups. With the help of the risk prediction system, clinicians learn which individual patients may be in danger of death and can thus conduct personalized prevention treatment in due time. Moreover, our research can be used to build a general clinical decision support system that serves not only COVID-19 but also other potential future pandemics. The patterns found in this data may also help biologies to design effective vaccines or vaccination strategies. Finally, these methodologies can be applied for future studies on big data and machine learning in the broader sense. Realizing an effective covid-19 diagnosis system based on machine learning and iot in smart hospital environment Combinatorial trace method for network immunization Comparison of deep learning approaches to predict covid-19 infection Role of biological data mining and machine learning techniques in detecting and diagnosing the novel coronavirus (covid-19): a systematic review Classifying covid-19 spike sequences from geographic location using deep learning Effective and scalable clustering of SARS-CoV-2 sequences Short term load forecasting using smart meter data Shortterm load forecasting using AMI data Spike2vec: An efficient and scalable embedding approach for covid-19 spike sequences A k-mer based approach for SARS-Cov-2 variant identification Predicting attributes of nodes using network structure Pattern recognition: A statistical approach Predictive analytics of covid-19 with neural networks node2vec: Scalable feature learning for networks Locally weighted mean phase angle (lwmpa) based tone mapping quality index (tmqi-3) Ridge regression: some simulations Feature selection with the boruta package Machine learning methods accurately predict host specificity of coronaviruses based on spike sequences alone Machine learning and olap on big covid-19 data Big data science on covid-19 data Data science for healthcare predictive analytics Using machine learning of clinical data to diagnose covid-19: a systematic review and meta-analysis Visualizing data using t-SNE Ridge regression From alpha to zeta: Identifying variants and subtypes of sars-cov-2 via clustering Random features for large-scale kernel machines Language independent sentiment analysis A multi-cascaded deep model for bilingual sms classification