key: cord-0044764-yaffw20x
authors: Toureau, Vincent; Bibiloni, Pedro; Talavera-Martínez, Lidia; González-Hidalgo, Manuel
title: Automatic Detection of Symmetry in Dermoscopic Images Based on Shape and Texture
date: 2020-05-18
journal: Information Processing and Management of Uncertainty in Knowledge-Based Systems
DOI: 10.1007/978-3-030-50146-4_46
sha: b38c44d383068e5164141190fc538fa4a2ce9fd0
doc_id: 44764
cord_uid: yaffw20x

In this paper we present computational methods to detect the symmetry in dermoscopic images of skin lesions. Skin lesions are assessed by dermatologists based on a number of factors. In the literature, the asymmetry of lesions appears recurrently since it may indicate irregular growth. We aim at developing an automatic algorithm that can detect symmetry in skin lesions, as well as indicating the axes of symmetry. We tackle this task based on skin lesions’ shape, based on their color and texture, and based on their combination. To do so, we consider symmetry axes through the center of mass, random forests classifiers to aggregate across different orientations, and a purposely-built dataset to compare textures that are specific of dermoscopic imagery. We obtain 84–88% accuracy in comparison with samples manually labeled as having either 1-axis symmetry, 2-axes symmetry or as being asymmetric. Besides its diagnostic value, the symmetry of a lesion also explains the reasons that might support such diagnosis. Our algorithm does so by indicating how many axes of symmetry were found, and by explicitly computing them.

Skin cancer is a disease caused by the abnormal and uncontrolled proliferation of melanocytes-cells that pigment the skin-that have undergone a genetic mutation. This disease is one of the most widespread around the world as it represents 40% of all cancers [6] . There are several types of malignant skin cancers (basal cell carcinoma, squamous cell carcinoma, etc.) but the most aggressive and deadliest one is known as melanoma. In Europe, cutaneous melanoma represents 1-2% of all malignant tumors [3] but its estimated mortality in 2018 was 3.8 per 100.000 men and women per year [2] .

This type of disorder is characterized by the development of a skin lesion which usually presents an irregular shape, asymmetry and a variety of colors, along with a history of changes in size, shape, color and/or texture. Based on this, experts designed protocols, the so-called diagnostic methods, to quantify the malignancy of the lesions. Some examples are pattern analysis, the ABCD rule, the 7-point checklist, and the Menzies method. In these, the asymmetry of the lesion plays an essential role towards the assessment of the lesion. However, each of them defines symmetry in a slightly different way. While according to the Menzies method, benign lesions are associated to symmetric patterns in all axes through the center of the lesion, disregarding shape symmetry [4] . Another example: regarding the ABCD rule, there might be symmetry in 0, 1 or 2 perpendicular axes when evaluating not only the contour, but also its colors and structures. Moreover, the assessment of symmetry might be altered by the individual judgment of the observers, which depends on their experience and subjectivity [4] .

The increasing incidence of melanoma over the past decades along with the desire to overcome the variability in interpretation have promoted the development of computer-aided diagnosis systems. They provide reproducible diagnosis of the skin lesions as an aid to dermatologists and general practitioners in the early detection of melanoma.

There are several general-purpose techniques to calculate symmetry in the computer vision field, as presented in [13] . A few techniques have been applied to the detection of asymmetry in skin lesions in dermoscopic images. Seidenari et al. [14] quantify the asymmetry as the appearance of an irregular color distribution in patches within the lesion. Also, Clawson et al. [5] , following the same line, further integrates Fourier descriptors into a shape asymmetry quantifier. Other authors, such as Kjoelen et al. [9] and Hoffman et al. [7] , estimate the asymmetry by computing the nonoverlapping areas of the lesion after folding the image along the best axis of symmetry, taking into account grayscale texture and color.

However, as far as we know, the study of the presence of asymmetry in skin lesions has been used to classify lesions as malignant or benign in diagnostic aid systems. In most articles, the approaches that calculate the symmetry of the lesions do so in an integrated way in an automated system that extracts other features, simultaneously. This fact hinders both the symmetry evaluation and the interpretability of the results, such as reporting what is the impact of finding asymmetry towards the classification of a specific lesion as malignant.

Hence, our objective is two fold. First, to study the symmetry of lesions from three different points of view. The first focuses on the shape of symmetry of the lesion, while the second is based on the symmetry of the textures (including colors). Finally, these two approaches are combined into the third one. Second, to compare and analyze their impact, as well as to quantitatively assess their performance.

In order to complete the clinical analysis and the diagnosis of skin lesions at its earliest stage, physicians commonly employ a technique called dermoscopy. It is an in-vivo, non-invasive imaging technique based on a specific optical system with light that amplifies the lesion, which has previously been covered with mineral oil, alcohol or water to avoid the reflection of light on it and increase the transparency of the outermost layer of the epidermis. Dermoscopy has been shown to improve the diagnostic accuracy up to 10-30% [10] compared to simple clinical observation. In some cases, dermoscopy can capture digital images of skin lesions, providing more detailed information of morphological structures and patterns compared to normal images of skin lesions.

From the available databases of dermoscopic images we decided to use the PH 2 database [11] , which contains 200 dermoscopic images annotated by expert dermatologists. For each image, relevant information about the manual segmentation and the clinical diagnosis of the skin lesion, as well as some dermoscopic features, such as asymmetry, colors, and dermoscopic structures, are available. It is worth mentioning that the symmetry of the lesion is evaluated by clinicians according to the ABCD rule, and therefore, concerning its distribution of contour, color and structures simultaneously. There are three possible labels for this parameter: 0 for fully symmetric lesions, 1 for asymmetric lesions with respect to one axis, 2 for asymmetric lesions with respect to two axes.

In Sects. 2, 3 and 4 we detailed the three different approaches used to study the symmetry of the lesions, based respectively on shape descriptors, texture descriptors and a combination of both. In Sect. 5 we discuss on the results obtained, and conclude with the main strengths and limitations of our approach.

In this section we present the first of three computational approaches towards assessing the symmetry of skin lesions. In particular, we focus exclusively on the shape of the lesion. We parameterize the candidates to be axes of symmetry as the lines through the center of mass. Such lines are the only ones that split any continuous twodimensional figure in two parts of the same area. The assumption that symmetry axes contain the center of mass is convenient: they become characterized by their angle with respect to the horizontal axis, α. Also, the center of mass is easy to compute.

To assess whether an axis divides symmetrically the lesion we employ the Jaccard index [8] . We consider that a line is a perfect axis of symmetry if the second half is equal to the reflection of the first one with respect to the axis. Since we are dealing with the shape of the lesion, being equal refers to whether a pair of pixels are both tagged as lesion or both tagged as skin. Let be a line of the plane, M + , M − the two halves in which splits the lesion region. Let R denote the reflection with respect to and let |A| denote the area of a region. Then, we define the shape-based symmetry index of a line , S 1 ( ) to be:

The final assessment of the symmetry within the skin lesion based on shape is based on a random forest classifier. We consider a pencil of N lines through the center of mass, 180 • ·k/N , for k = 0, . . . , N − 1. Then, we obtain their shapebased symmetry index, S( 180 • ·k/N ). A random forest classifier aggregates all the indices into a final answer, being either "no symmetry", "1-axis symmetry" or "2-axes symmetry". We remark that substituting the learning classifier with experimentally-set fixed thresholds achieves worse quantitative results, but provides the insight of which lines represent the main and perpendicular axes of symmetry, if there are any.

Qualitative results of this method are found in Fig. 1 . In it, we show an accurately classified sample (left) and a wrongly classified one (right). The latter presents a symmetric shape, but it also shows some inner structures that are not symmetric with respect to one of the perpendicular axes.

In our implementation, we used N = 20, and 10 trees in the classifier. This is a fast algorithm, whose execution time is typically in the range 1-2 s.

To assess how two halves fold symmetrically with respect to their texture, we need to assess how similar two textures appear to be. Corresponding pixels are not required to be equal, but to have been drawn from a similar statistical distribution. Moreover, the distributions we found are specific: textures in dermoscopic images are not necessarily similar to textures found in other computer vision tasks. We consider a patch-based approach: we assess similarity of textures in two locations based on a local neighbourhood of them. This approach led to the creation of a dataset containing pairs of similar and different textures, introduced in the following.

The texture of skin lesions play an important role in its symmetry. As previously mentioned, its symmetry is jointly based on the shape of the lesion, and the appearance of similar structures and patterns. To discriminate such patterns, we must be able to compare the local texture in different locations of the image. However, textures found in dermoscopic images are specific, not having the same statistical distributions that textures found in textile, piles of similar objects or other settings. To discriminate such textures, we propose the extraction of a dataset from dermoscopic images, providing pairs of similar and different patches. Each patch, a n × n-pixel region cropped from the original image, contains information about the local texture in a specific point. We extract pairs of patches from a dermoscopic image in a fully automatic way. We require not only the dermoscopic image, but also a segmentation of the lesion. In order to obtain pairs of patches with similar textures, (p A , p B ), we randomly select two locations that are very close, under the restriction that both patches are completely inside the lesion or completely outside. We remark that both patches are largely overlapping. To obtain pairs of patches with different textures, (p A , p B ), we randomly select one patch completely inside the lesion, and the other one outside the lesion. In this case, there will be no overlap between them, and we will assume that they represent regions with a different underlying texture. All the samples of our dataset will be those patches, x = (p A , p B ), and the reference data will be whether they are of the first or the second, y ∈ {'Similar', 'Different'} type. In Fig. 2 we show similar and different patches extracted using this strategy. Several limitations must be acknowledged. First, the samples of differentlytextured patches are biased towards our task: they do not follow the same statistical distribution that differently-textured patches inside the lesion do. This negative effect does not seem to have a huge impact in its usage (see Sect. 3), possibly due to the appearance of different skin tones in the original dermoscopic images. Second, close patches are assumed to be similar, and patches inside the lesion are assumed to be different to patches outside of it. While this is not necessarily the case, it has proven to hold the vast majority of times. Third, a learning classifier that compares them could cheat on solving the task, using the more basic approach of detecting as positive only those patches that present a large overlap.

To overcome the second and third limitation, we limit ourselves to manually select texture-relevant features. We use the Gray Level Co-occurrence Matrix (GLCM) with two-pixel distance and a horizontal orientation to extract five texture features. They are dissimilarity, correlation, energy, contrast, and homogeneity. Correlation is understood as a measure of the linear dependence of gray levels between pixels at the specified distance. The energy feature measures the brightness of the images as well as the repetition of subunits. Contrast refers to the local gray level variations, while the homogeneity is a measure of the smoothness of the gray level distribution. Finally, we extract the 25th, 50th and 75th percentiles of the marginal distribution of the RGB channels of the pixels.

The patches were randomly extracted from the PH 2 database of dermoscopic images, selecting 10 pairs with similar texture and 10 pairs with different texture for each of the 200 images in the database. It includes a manually segmented lesion, which we used to check whether patches were completely inside or outside the lesion. Given the original resolution of images, 764 × 576, we set the size of patches to be 32 × 32, as a good trade-off to obtain a region representative enough but whose texture is approximately uniform. Besides considering the features mentioned above, we decided to use shallow-learning algorithms only, specifically selecting random forest due to the amount and diversity of features.

Using the newly created patch dataset, we trained a random forest classifier T that, given two patches p A , p B , extract the features introduced in Sect. 3 and then estimates whether they represented the same texture or not. Such classifier is represented as:

Let us continue assuming that an axis of symmetry contains the center of mass. We define a line to be an axis of symmetry with respect to its texture if symmetric patches present the same texture. We thus define the texture-based symmetry index of a line , S 2 ( ), as:

where N is the amount of patches that can be extracted from the intersection of the upper region and its reflected lower half, Similarly to the shape-based symmetry detector, we consider N equidistributed lines and aggregate their texture-based symmetry index with a different random forest classifier. Also, substituting such classifier with experimentally-set fixed thresholds provides the insight of which lines are the main and perpendicular axes of symmetry, if there are any.

Results of the above-mentioned procedure are found in Fig. 3 . In it, we show a sample with symmetric textures (right) and a sample with differently textured matched regions (left). The result of the patch-based classifier is encoded as a semi-transparent circle, ranging from green (similar) to red (non-similar) patches. Also, we emphasize that we select partially overlapped patches, and we restrict the patch selection procedure to those regions within the lesion.

At an implementation level, several details must be mentioned. The patchbased classifier, T(p A , p B ), is a random forest classifier with 200 trees and two outputs-either similar or different. Its inputs are a list of 2 · n features: n features extracted from the first patch p A , and n features from the second one p B . We used n = 5 + 3 · 3, extracting 5 features from the grey-level co-occurrence matrix, and 3 quantiles of each of the channels R, G and B (see Sect. 3). Also, we used N = 10 lines, and 32 × 32-pixel patches. These parameters were experimentally selected. The second classifier, used to aggregate information across different orientations, is also a random forest with 100 trees and, as in the case of the shape-based classifier, three outputs. This algorithm requires, with nonoptimized code, around 40-50 s. 

Finally, to answer the hypothesis of whether both shape and texture are actively contributing towards the symmetry of the lesion-as identified by a human expert-, we combine both symmetry indicators. Following the same reasoning, given a pencil of lines ( α ) α we compute for each line α its shape-based symmetry index, S 1 ( α ) and its texture-based symmetry index, S 2 ( α ). We aggregate these two lists with a 10-tree random forest classifier to output a final decision as either "no symmetry", "1-axis symmetry" or "2-axes-symmetry".

We presented a constructive approach towards symmetry detection, somehow similar to an ablation study. First, the symmetry detection has been addressed using only the shape of the lesion. However, as shown in Fig. 1 (right) , taking into account only the shape can not provide satisfactory results. Experts take into account the shapes, but also the textures and colors of the inner parts of the lesion to define symmetry. Considering this, the information loss is too high, which implies that more information must be used to be more accurate. This is why the symmetry of textures and colors has to be included and studied.

In the second approach, only texture and color symmetry has been considered to determine the presence of symmetry in skin lesions. We remark that texture and color are hardly separable, since the former is defined in terms of changes of the latter. In this case two random forest classifiers have been used: to assess the symmetry of two 32 × 32 patches, and to aggregate information across different orientations. Qualitatively, the similarity map tends to be reliable and the axes of symmetry are never aberrant regarding textures.

In the following, we present the results obtained with the aforementioned methods and conclude with some final remarks.

In this section, we present the experimental settings and results obtained. This is done quantitatively to add up to the qualitative results contained in Figs. 1 and 3. We do so by comparing with the manually labelled data of the PH 2 dataset. It contains, for each dermoscopic sample, a tag indicating either "no symmetry", "1 axis of symmetry" or "2 axes of symmetry". Therefore, we have a 3-class classification problem, where the three class are: no symmetry, 1-axis symmetry and 2-axis symmetry. In the PH2 database we have the following distribution labeled by experts: 26% for the first class, 15.5% for the second class, and 58.5% for the last class.

To assess the success of our method we consider its accuracy. That is, the ratio of correctly classified samples. We emphasize that we are considering a 3class classification problem, so binary metrics (such as recall or F-measure) can not be computed. The results obtained by the different algorithms are presented in Table 1 . We train and validate the algorithms with disjoint datasets. We split the 200 images in the training set (first 50) and the validation set (last 150). No randomization was applied to the samples since their symmetry is not ordered. We remark that the validation set is larger since we aim at estimating the generalization capacity of the model with low variance. Metrics in Table 1 have been computed over the validation set. Any other learning stage, including the classifier employed to compare patches, have been trained only with the training set. Table 1 summarizes the results obtained with each of the methods described in Sects. 2, 3 and 4. As can be observed, these classifiers provide satisfactory quantitative results. We emphasize the subjective nature of this task: in contrast to, for instance, assessing the malignancy of the lesion, the symmetry of a lesion is measured against the perceptive criteria of a human expert.

The superiority of the shape-based approach over the texture-based one is not contradictory: the latter purposely neglects information regarding the shape. That is, it exclusively uses pairs of patches such that both of them are located within the lesion, disregarding the fact that there may or may not exist additional pairs of patches such that first one represents healthy skin and the second one the lesion. Such pairs have not been provided to the texture-based method in order to avoid implicitly using information derived from the shape of the lesion.

The images leading to classification errors are different in the shape-based and texture-based methods. This implies that the two sources of information do not provide equivalent results.

As one would expect, classification based on both shape and on texture has superior results. The final accuracy reaches up to 88% which defines a reliable model that may be used in real applications. Other models may be used, but considering the features size and the quality of the results given by the random forest classifiers with a minimalist tuning, they seem to be appropriate to solve the problem raised in this work.

We have addressed the computational problem of symmetry evaluation in terms of (i) shape, and (ii) texture (including its color) as is considered by dermatologists [1, 12] . This provides the clinician with a comprehensible and interpretable tool, that indicates the presence of symmetry axis and its location. Asymmetry of skin lesions is an important indicator of the presence of irregular growth in skin lesion. Thus, it contributes substantially to its diagnosis. In the ABCD rule of dermoscopy, for example, asymmetry is the parameter that contributes with a larger coefficient to the ABCD-based diagnosis [12] . We emphasize that the aim of this work is to detect the presence of symmetry (or not) in a dermoscopic image of a skin lesion, rather than the classification of the lesion as either malignant or benign.

The algorithms designed deal with the symmetry of skin lesions, which is an important indicator of uncontrolled growth of cells. They treat the symmetry as it is evaluated by the experts considering at the same time its shape, texture and colors. Therefore, the output provided by the algorithm is interpretable by experts. The algorithms in this paper can be freely accessed online at the website http://opendemo.uib.es/dermoscopy/, as well as using it as a the standalone python package dermoscopic symmetry.

The shape-based algorithm is faster than the texture-based one: 1-2 s and 40-50 s to process a medium-sized dermoscopic image. Although the code could be optimized, the complexity of the latter is much higher. The shape-based method can be used for real-time applications, whereas both could be used off-line or for knowledge distillation into a faster classifier.

Both shape and texture information seem to be necessary towards assessing the skin lesion symmetry. The rationale lies on the fact that irregular growththe malignancy cue looked after-may cause both types of effects. However, given the quantitative metrics in Table 1 , both texture and shape provide a large amount of information.

A limitation of this work lies on the biases in the patch dataset introduced in Sect. 3. First, we assume that close regions present similar textures, which does not always hold. Second, we have a very limited amount of interesting different textures. Due to the automatic selection of patches, we can only assume that two patches represent a different texture if one of them is within the lesion and the other outside of it. This means that, in each pair of different patches, one of them was extracted from the skin, whereas we are later comparing two patches that are within the bounds of the lesion.

Finally, this study is biased towards light-skin patients: it has been quantitatively contrasted against the labels of the PH 2 dataset.

Melanoma Molecular Map Project

Dermoscopy of pigmented skin lesions: results of a consensus meeting via the Internet

Determination of optimal axes for skin lesion asymmetry quantification

A review of the quantification and classification of pigmented skin lesions: from dedicated to hand-held devices

Diagnostic and neural analysis of skin cancer (DANAOS). A multicentre study for collection and computer-aided analysis of data from pigmented skin lesions using digital dermoscopy

The distribution of the flora in the alpine zone

Performance of AI methods in detecting melanoma

Systematic review of the diagnostic accuracy of dermatoscopy in detecting malignant melanoma

Ph 2-a dermoscopic image database for research and benchmarking

What is the role of color symmetry in the detection of Melanomas?

Symmetry axis computation for almost-symmetrical and asymmetrical objects: application to pigmented skin lesions

Asymmetry in dermoscopic melanocytic lesion images: a computer description based on colour distribution