key: cord-0790774-dv84n0v0 authors: Mathur, Jyoti; Chouhan, Vikas; Pangti, Rashi; Kumar, Sharad; Gupta, Somesh title: A convolutional neural network architecture for the recognition of cutaneous manifestations of COVID‐19 date: 2021-02-28 journal: Dermatol Ther DOI: 10.1111/dth.14902 sha: b123a7b7b7d3c174cf2108c918f12f8d1cc25c9d doc_id: 790774 cord_uid: dv84n0v0 During the COVID‐19 pandemic, dermatologists reported an array of different cutaneous manifestations of the disease. It is challenging to discriminate COVID‐19‐related cutaneous manifestations from other closely resembling skin lesions. The aim of this study was to generate and evaluate a novel CNN (Convolutional Neural Network) ensemble architecture for detection of COVID‐19‐associated skin lesions from clinical images. An ensemble model of three different CNN‐based algorithms was trained with clinical images of skin lesions from confirmed COVID‐19 positive patients, healthy controls as well as 18 other common skin conditions, which included close mimics of COVID‐19 skin lesions such as urticaria, varicella, pityriasis rosea, herpes zoster, bullous pemphigoid and psoriasis. The multi‐class model demonstrated an overall top‐1 accuracy of 86.7% for all 20 diseases. The sensitivity and specificity of COVID‐19‐rash detection were found to be 84.2 ± 5.1% and 99.5 ± 0.2%, respectively. The positive predictive value, NPV and area under curve values for COVID‐19‐rash were 88.0 ± 5.6%, 99.4 ± 0.2% and 0.97 ± 0.25, respectively. The binary classifier had a mean sensitivity, specificity and accuracy of 76.81 ± 6.25%, 99.77 ± 0.14% and 98.91 ± 0.17%, respectively for COVID‐19 rash. The model was robust in detection of all skin lesions on both white and skin of color, although only a few images of COVID‐19‐associated skin lesions from skin of color were available. To our best knowledge, this is the first machine learning‐based study for automated detection of COVID‐19 based on skin images and may provide a useful decision support tool for physicians to optimize contact‐free COVID‐19 triage, differential diagnosis of skin lesions and patient care. The clinical spectrum of COVID-19 is very heterogeneous ranging from mild symptoms to severe symptoms like respiratory failure or multi-organ dysfunction. Dermatologists in Europe, particularly in Italy, Spain and France, who were on the frontlines of managing the deluge of hospitalized COVID-19-positive patients, reported various skin rashes that appeared to be correlated with the disease. [1] [2] [3] Literature reports show, however, a great deal of variation in the skin manifestations, their latency periods and the percentage of patients who Jyoti Mathur and Vikas Chouhan are equal contributors develop these. For instance, in the first study to draw attention to skin lesions, nearly 20.4% (18 of 88) of patients, with a confirmed diagnosis of COVID-19, developed cutaneous manifestations. 4 This number contrasts with 0.2%, 7.25%, and 12.7% of cases that developed skin abnormalities in a study from China and two studies from India, respectively. [5] [6] [7] Subsequently, various reports of skin manifestations in both adults (with severe forms of and younger paucisymptomatic patients have been published. 8, 9 Marzanoet al. divided the reported skin lesions into six main clinical patterns: (a) urticarial rash, (b) erythematous/maculopapular/morbilliform rash, (c) papulovesicular exanthem, (d) chilblain-like acral pattern, (e) livedo reticularis/racemosa-like pattern, (f) purpuric "vasculitic" pattern. 8 The scientific understanding of COVID-19 and associated dermatological symptoms is currently evolving and the diagnostic and/or prognostic value of these lesions is a subject of exploration, but researchers agree that observation of cutaneous symptoms, without another explanation, should prompt confirmatory testing. 1, 9, 10 To collate cases of dermatoses in COVID-19-positive and COVID-19-suspected cases from a global network and inform doctors on the frontlines, the American Academy of Dermatology (AAD), in collaboration with the International League of Dermatologic Societies (ILDS), has launched an online COVID-19 dermatology registry. 9 Diagnosing skin manifestations in patients with COVID-19 remains a challenge even for dermatologists because it is unclear whether the skin lesions are related to the virus. 10 A decision support tool that assists physicians in differentiating skin lesions known to be associated with COVID-19 from incidental skin findings may be helpful in disease management. In this study, we present a novel machine-learning model for the identification of COVID-19-associated skin manifestations based on a single clinical image of skin lesions on both Caucasian (white) and Indian skin (skin of color). The model can discriminate COVID-19-associated skin lesions from normal skin and 18 common skin conditions including COVID-19 mimics. Although the utility of this algorithm in COVID-19 screening is limited by the observation that only a fraction of COVID-19 patients exhibit cutaneous manifestations and lesions often appear late in the infection course, this proof-of-concept study shows the potential for the model to be used as a high-level diagnostic aid in the differential diagnosis of skin lesions in COVID-19-suspected patients. We implemented a neural architecture that was an ensemble of three models-Densenet-161, 11 SeresNext-101 12 and EfficientNet-B4, 13 trained on the same training set with predictions combined at the corresponding final output layers. These models were selected based on their best individual performances across their architecture families, limited by the available GPU machine (2 gtx-1080ti cards in multi node training mode). The batch sizes were 32 for Densenet-161, 24 for EfficientNet-B4 and 32 for SeresNext-101. The number of layers were 809 for Densenet-161, 470 for EfficientNet-B4 and 2724 for SeresNext-101.The number of epoch were 70. We used pre trained weights and fine-tuned them on our dataset. All training and testing images were segmented, so a bounding box surrounded the lesion of interest. These images were split into five equal parts (folds) and five iterations of training and validation were performed so that a different fold of the data was held-out for validation, while the remaining fourfolds were used for learning. A five-fold cross-validation test ensured that each image had a chance of being validated. We initially experimented with just Densenet-161 for fine-tuning the model parameters such as loss functions and augmentation, and inference schemes. After we achieved the best possible configuration, we scaled these to SeresNext-101 and EfficientNet-B4. A schematic of the inference and training pipelines are shown in Figure 1A ,B. In the training stage, each of the three models was modified at the final output layer and trained separately with a uniquely optimized augmentation and preprocessing pipeline (the training schematic refers to Densenet-161, but is representative of all three models). Image data augmentation is a technique commonly employed to artificially expand the size of a training dataset by creating modified versions of images in the dataset. Another benefit of augmentation is to generate a model that is generalizable to different kinds of images. In the inference stage, we used six replicas of the source image to be inferred, with six unique, pre-selected augmentation schemes applied on each of them and passed them through six copies of each of the three models to get six unique prediction scores from each of those three modes. These pre-selected augmentation schemes were derived through rigorous experiments and were attuned to clinical skin images. We combined the predictions of each of these unique replicas to get a single prediction per model, which were again combined using max-pooling across all three models to arrive at the final output score. In this study, a convolutional neural network (CNN)-based algorithm was trained with clinical images of skin lesions from confirmed COVID-19-positive patients, healthy controls as well as 18 other common skin conditions, that included COVID-19-rash disease differentials described in the literature, such as urticaria, 10,14 chickenpox, 4,15 herpes zoster, 1 pityriasis rosea, 1 bullous pemphigoid, psoriasis, 1 and fixed drug eruption. 1 The remaining diseases were acne, lichen planus, normal skin, pemphigus, pityriasis versicolor, rosacea, secondary syphilis, tinea corporis, cruris or faciei, tinea manuum, tinea pedis, and paronychia. The COVID-19 associated skin lesions, used for training and testing, were further divided into 12 categories based on their morphology (Supplementary Table 1 ). Our complete clinical image dataset was comprised of 7053 total, preaugmentation images, of which 2904 were from Indian patients and 4141 were from Caucasian patients ( In this study, we included images from COVID-19-confirmed and suspected patients that fall into the six main clinical categories of urticarial rash, maculopapular rash, vesicular eruptions, chilblain-like rash, livedo and purpura-like rash and found high sensitivity, specificity, NPV and AUC values for COVID-19-rash in an in-silico validation study across 20 skin conditions (Supplementary Table 1 and Table 3 ). The utility of this algorithm is highlighted by the T A B L E 1 Number of images (pre-augmentation) per skin disease class from public databases (Caucasian skin) and private database (Indian skin) T A B L E 3 The disease-specific top-1 sensitivity, top-3 sensitivity, specificity, PPV, NPV and AUC from 5-fold validation are depicted below. The overall top-1 accuracy and top-3 accuracy for COVID-19-suspected lesions is also shown. NA: not available, ND: not determined 24 We envision that algorithms like ours could be integrated into smartphone apps to provide a decision support tool for F I G U R E 2 A, shows a representative top-1 confusion matrix for all 20 skin conditions. The numbers indicate image numbers. The y-axis represents the actual disease class and x-axis represents the predicted disease class for fold 0. The color gradient depicts the degree of correct predictions. B, shows the disease categories that represent the false-positive and false-negative cases in all 5-folds for COVID-19-rash dermatologists to recognize COVID-19-related skin lesions, guide patient care and research efforts. 18 Gupta wrote the manuscript. The data that supports the findings of this study are provided in the supplementary files which are submitted with the manuscript. Rest of the data is available from the corresponding author upon reasonable request. https://orcid.org/0000-0002-2373-4598 Somesh Gupta https://orcid.org/0000-0003-3015-8602 Classification of the cutaneous manifestations of COVID-19: a rapid prospective nationwide consensus study in Spain with 375 cases Chilblains is a common cutaneous finding during the COVID-19 pandemic: a retrospective nationwide study from France Pernio-like skin lesions associated with COVID-19: a case series of 318 patients from 8 countries Cutaneous manifestations in COVID-19: a first perspective Clinical characteristics of coronavirus disease 2019 in China Recognizable vascular skin manifestations of SARS-CoV-2 (COVID-19) infection are uncommon in patients with darker skin phototypes Dermatological findings in SARS-CoV-2 positive patients: an observational study from North India Cutaneous manifestations in patients with COVID-19: a preliminary review of an emerging issue The American Academy of Dermatology COVID-19 registry: crowdsourcing dermatology in the age of COVID-19 Acute urticaria with pyrexia as the first manifestations of a COVID-19 infection Densely connected convolutional networks Squeeze-and-excitation networks Efficientnet: rethinking model scaling for convolutional neural networks Urticarial eruption in COVID-19 infection Varicella-like exanthem associated with COVID-19 in an 8-year-old girl: a diagnostic clue? Scikit-learn: machine learning in python Cutaneous manifestations of COVID-19: report of three cases and a review of literature A machine learning-based, decision support, mobile phone application for diagnosis of common dermatological diseases Cutaneous manifestations in COVID-19: lessons learned from current evidence Characterization of acute acral skin lesions in nonhospitalized patients: a case series of 132 patients during the COVID-19 outbreak Negative SARS-CoV-2 PCR in patients with chilblain-like lesions Comment on 'Two cases of COVID-19 presenting with a clinical picture resembling chilblains: first report from the Middle East': pernio unrelated to COVID-19 Evaluation of chilblains as a manifestation of the COVID-19 pandemic Skin manifestations of COVID-19 SARS-CoV-2 infection presenting as a febrile rash A convolutional neural network architecture for the recognition of cutaneous manifestations of COVID-19