key: cord-0129064-bc3744c2 authors: Vijayaraghavan, Sairamvinay; Haddad, David; Huang, Shikun; Choi, Seongwoo title: A Deep Learning Technique using a Sequence of Follow Up X-Rays for Disease classification date: 2022-03-28 journal: nan DOI: nan sha: 7f76c79e34b422d4279c307ef4919f4dddcbde25 doc_id: 129064 cord_uid: bc3744c2 The ability to predict lung and heart based diseases using deep learning techniques is central to many researchers, particularly in the medical field around the world. In this paper, we present a unique outlook of a very familiar problem of disease classification using X-rays. We present a hypothesis that X-rays of patients included with the follow up history of their most recent three chest X-ray images would perform better in disease classification in comparison to one chest X-ray image input using an internal CNN to perform feature extraction. We have discovered that our generic deep learning architecture which we propose for solving this problem performs well with 3 input X ray images provided per sample for each patient. In this paper, we have also established that without additional layers before the output classification, the CNN models will improve the performance of predicting the disease labels for each patient. We have provided our results in ROC curves and AUROC scores. We define a fresh approach of collecting three X-ray images for training deep learning models, which we have concluded has clearly improved the performance of the models. We have shown that ResNet, in general, has a better result than any other CNN model used in the feature extraction phase. With our original approach to data pre-processing, image training, and pre-trained models, we believe that the current research will assist many medical institutions around the world, and this will improve the prediction of patients' symptoms and diagnose them with more accurate cure. Deep Learning in Computer Vision has found its importance in medicinal AI. Computer Vision techniques and deep learning based predictive models go hand-in-hand in case of various subsections in healthcare which involve different image based data: such as X-rays, PT scans. There has been a lot of work in predicting diseases for patients using X-rays using deep learning models such as Convolutional Neural Networks [15] [10] [1] [9] , Support Vector Machines and K-Nearest Neighbors [5] [16] . However, most of these models use strictly one X-ray image per sample for training the model for predicting disease labels as a multi classification problem. Our team decided to present a slightly unique setting for a familiar problem. In this paper, we are going to present a deep learning model, which takes in a sequence of the consecutive previous chest X-rays of a patient (termed shortly as follow-ups) and analyzes the variation and difference across this sequence. For the feature extraction phase of the images, the model would involve convolutional neural networks (CNN). We have another phase in our model which would involve an optional Long Short Term Memory layer for capturing the time based temporal relationship across the pixels three images. The goal is to distinguish between the fourteen different respiratory diseases and also the No Finding label (indicating normal) using the previous three followup X-rays of every patient by predicting the disease label associated with the third X-ray image. We frame it as a multiclassification problem by predicting any of There have been several approaches to analyze chest X-rays using deep neural networks. One of the prior works does explore the classification of the Chest X-rays using ResNet and DenseNet for feature extraction for X-ray images [1] . This paper uses all of the X-ray images for disease classification without even considering followups or view position associated with the X-ray. Our approach distinguishes from those researchers that we have been analyzing the datasets and we wanted to see a closer look into datasets and wanted to know if three follow-ups could boost up the performance of the neural network we were building. We could discover that our approach was derived mainly from the 'ChestNet' paper [14] . This paper is probably the closest research paper that is similar to the project we are working on. The authors explore the probability of each possible disease of the patient using sigmoid output activation. They had used a seq2seq model with attention for fine tuning on the convolved output of hte X-ray after feature extraction X-rays and they had also used GradCAM technique [13] for identifying key regions in the X-ray image. However, they did not explain the impact of training models with CNNs followed by LSTMs and how they could change the overall performance of the classification model. We also explored how if we do not use LSTM would change the performance of the neural network classification. There are also works which have primarily used PCA based feature reduction and simple statistical-based feature selection techniques such as forward-backward selection techniques can help in X-ray images based feature extraction. They also trained support vector machine (SVM) and K-nearest neighbors (KNN) were applied to classify the esophageal cancer images with respect to their specific types. They had performed this task for a specific task of esophageal cancer [16] . In addition to the above work, texture and edge features on a local and global level within an X-ray image are extracted while using KNNs and SVMs again for disease classification [5] . Similar to this feature extraction technique, there is another work where the authors describe how feature extraction on TB images is important for disease analysis by using PCA and kPCA and training a linear regression classification model to predict the disease [11] . CheXNeXt model was a deep learning model from which we had been heavily motivated to formulate our hypothesis. CheXNeXt is a convolutional neural network that concurrently detects the presence of 14 pathologies [9] . Some of the researchers also have used DenseNet based CNN architectures for X-ray based classification [1] . CheXNet is a 121layer Dense Convolutional Network based on DenseNet is proposed to perform better than practicing radiologists at pneumonia detection on frontal view X-rays [10] . The model is trained on the same dataset that we are also using for this paper and the results show an increase in AUROC scores over the state of the art algorithms. Research also shows that CNN's and transfer learning can improve detection of pneumonia and COVID patients [7] . This paper utilizes Grad-CAM to illustrate the radiology disease detection for all their experiments. In healthcare, Chest X-rays are expensive and are very frequently used diagnostic examinations. Chest X-rays can play a significant role in determining the lung/heart diseases associated [2] . Practically, doctors intend to use the history of a patient's health vitals and medical results for analyzing his health conditions. For example, doctors would find it very useful to analyze the current as well as the previous visit X-rays for analyzing diseases in the long term for a patient. Temporal analysis of X-ray history is very critical and it would facilitate the work of a medical doctor in analyzing such work using robust deep learning models. We then created two datasets based on the view position associated with each X-ray. We call them AP and PA sample sets. These sample sets are created from separating the previously processed sample sets based on view position. After some research on both AP and PA datasets, we have discovered that the chest Xray dataset contains both views of AP and PA. We have decided to separate these two views of X-rays to test which view of the chest Xray could train the neural network we built better, faster, and more accurately. We have reviewed many research papers on deep learning approaches to chest X-rays to predict thoracic diseases, but most of them did not split datasets into two and train their model separately. By splitting the dataset into two would distinguish which dataset may result in a better performance. To observe both sides of the chest X-rays, we have separated the chest X-ray datasets into AP dataset and PA dataset to train our deep learning model and we attempted to fully understand the performance of the deep learning when we were training the model in two different datasets. This way, we can observe a clear observation of the heart and the respiratory systems of patients to easily distinguish the respiratory diseases through the multi-classification approach of the deep learning technique. Here's some statistics of how many samples finally we had and how many unique patients we are having. We wanted to follow a two-part architecture for solving this problem. Since we had constructed a slightly different input setting but we had a multi classification problem. So, in order to consider this problem, we devised a simple way to combine two different sub models to solve this problem. The first part of our architecture involves a deep convolution neural network used for feature extraction on images. We had decided to use three different deep convolution networks for image based feature extraction namely DenseNet [4] , ResNet [3] , and MobileNetV2 [12] . We had chosen specifically: DenseNet-169, ResNet50V2, and the MobileNetV2 architectures respectively. This is basically represented as the 'CNN' network in our model diagrams. So, to this CNN model we use, we will connect 3 input Xray images, each of size (128,128), representing the consecutive followup grayscale X-ray images of a patient. We use the same CNN model for each of our 3 input X-ray images we provide while training for one particular model. There is no observable time relationship between 3 images' pixels in a set after observing auc scores between CNN models because the models perform better when LSTM layer is not included in the architecture. So, in terms of the three internal Convolutional neural network models used, we constantly observed that in most of the 4 cases: (PA/AP with/without LSTMs) ResNet50V2 does generally better than the DenseNet169 models while the MobileNetv2 performs the worst in most of the cases.The main reason we can possibly attribute for such behavior is probably due to the architecture of each of these deep learning frameworks. While ResNet has 50 hidden layers in its deep learning architecture, the Mo-bileNetV2 possesses a fully convolution layer with 32 filters, and then 19 residual bottleneck layers. Among the CNN internal layers, ResNet without LSTM performs the best to predict the disease label based on the differences occurring across given follow-up X-rays according to ROC-AUC scores. Among the CNN layers we experimented, we have discovered and proven that ResNet50V2 without LSTM performs the best to predict the disease labels. We have also proven our hypothesis that having three sequences of chest X-ray of one patient can increase the accuracy of predicting the next disease better than having only one sequence of chest x-ray of a patient. We have collected and organized ROC-AUC analysis for demonstrating the differences across the algorithms we chose, and we have shown overall good performance in the research. For future analysis, we plan to use GradCAM technique on our best models for identification of the regions on custom X-rays which may be crucial to classify the disease label. Also, using three followup images was more of a convenient choice in comparison to using a single image and we could have experimented with more follow up X-ray images for a stronger relationship across results. -Wrote and edited the report and chose the topic on AI in Medicine using chest X-ray. -Coded the single baseline, dataset preprocessing, and data visualization. -Choi also uploaded the entire dataset from the National Institutes of Health on the shared drive and his teammates could utilize the dataset with ease. -Reviewed research papers that related to the project. -Trained DenseNet on AP with and without LSTM and recoreded the results. -Maintained and debugged codes with teammates and managed GitHub repository. -Wrote code to analyze NIH dataset details and data distribution. -Coded Python code for dataset preprocessing with filters, creating sample sets from preprocessed dataset, and sample set X-ray image extraction. -Trained MobileNetV2 and ResNet50V2 on AP view position models; coded and research on ROC-AUC curve and F1-score implementation for multi-classification on python -Managed and cleaned github repository; Wrote README for github repository -Reviewed research papers for related work -Edited the report -Wrote code to Analyze NIH dataset details. Displayed bar charts and pi charts -Wrote Python code for dataset preprocessing with filters, creating sample sets from preprocessed dataset, and sample set X-ray image extraction. -Trained MobileNetV2 and ResNet50V2 PA view position models -Reviewed research papers for related work -Worked on model diagram and overall pipeline for report -Worked on paper dataset, methodology and result tables and plots for report 8.4 Sairamvinay Vijayaraghavan (saivijay@ucdavis.edu) -Devised the basic idea for this project: the 3 image input of X-rays and deep learning CNN models as a sequence -Development of ideas along with the team: worked on devising the solutions and the hypothesis for this paper -Wrote Python Code for the basic script to adapt for creating other models for the single image prediction for the disease classification and the triple image disease prediction. -Trained the single image PA baseline model ; Also the DenseNet PA models with/without LSTMs -Edited and fine tuned the report sections for the term paper. -Reviewed research papers for related work. Comparison of deep learning approaches for multi-label chest x-ray classification Multi-task learning for chest x-ray abnormality classification on noisy labels Identity mappings in deep residual networks Densely connected convolutional networks Multilevel feature extraction and x-ray image classification Ai in medicine -deeplearning.ai A deep learning and grad-cam based color visualization approach for fast detection of covid-19 cases using chest x-ray and ct-scan images Chest x-ray quality projection Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists Radiologist-level pneumonia detection on chest x-rays with deep learning Feature extraction of chest x-ray images and analysis using pca and kpca Mobilenetv2: Inverted residuals and linear bottlenecks Grad-cam: Visual explanations from deep networks via gradient-based localization Chestnet: A deep neural network for classification of thoracic diseases on chest radiography Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases Feature extraction and classification on esophageal xray images of xinjiang kazak nationality