key: cord-0335908-runola9e
authors: Nau, Merlin A.; Schiffers, Florian; Li, Yunhao; Xu, Bingjie; Maier, Andreas; Tumblin, Jack; Walton, Marc; Katsaggelos, Aggelos K.; Willomitzer, Florian; Cossairt, Oliver
title: SkinScan: Low-Cost 3D-Scanning for Dermatologic Diagnosis and Documentation
date: 2021-01-31
journal: nan
DOI: 10.1109/icip42928.2021.9506335.
sha: d8ec9fe236318cca375ad7692c45e4d9da5867e4
doc_id: 335908
cord_uid: runola9e

The utilization of computational photography becomes increasingly essential in the medical field. Today, imaging techniques for dermatology range from two-dimensional (2D) color imagery with a mobile device to professional clinical imaging systems measuring additional detailed three-dimensional (3D) data. The latter are commonly expensive and not accessible to a broad audience. In this work, we propose a novel system and software framework that relies only on low-cost (and even mobile) commodity devices present in every household to measure detailed 3D information of the human skin with a 3D-gradient-illumination-based method. We believe that our system has great potential for early-stage diagnosis and monitoring of skin diseases, especially in vastly populated or underdeveloped areas.

3D scanning provides access to a plethora of useful features in the diagnosis and documentation of skin diseases [1] . For example, skin cancer is the most common cancer in the United States developed by every fifth person throughout their lifetime [2] . While skin cancer is treatable, early detection is essential for a successful prognosis. Therefore, routine screenings are vital to ensure an early-stage diagnosis and therapy [3] .

Generally, the most important evaluation for a correct diagnosis of skin cancer is based on the ABCDE [4] guided rule Thanks to Northwestern Alumnae for funding. ©2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. c) The recovered normal map. d) 3D reconstruction assessing asymmetries of the lesion shape, border irregularities, color variegation, the lesion's diameter and the evolution of shape and color. Currently, it is most common to evaluate the above criteria solely based on 2D image information. However, since lesions can also grow in height, this evaluation can be significantly improved with additional access to 3D topographic information of the skin [5] (in fact, melanomas at a young age are misdiagnosed by up to 40%, in comparison with pigmented lesions [6] ). More to the point, surface orientation on skin cancer is a feature that can support the distinction between malign and benign tissue [7] . Although increasingly used by professional physicians in their diagnosis and documentation, clinical 3D imaging techniques to measure human skin like optical coherence tomography [8] or laser-based scanners [9] , are tied to expensive hardware, are not portable, and hence are not accessible to a broad audience.

In this work, we ask the question: Can routine examinations of skin disease be augmented with low-cost but precise 3D imaging techniques to improve assessment, monitoring, and diagnosis effectiveness? Mobile applications for phones or tablets that allow the exchange of images between the medical doctor and patient are broadly available [10] . However, these apps only allow for patient-driven image capture of 2D RGB images. Currently patients cannot capture and transfer 3D information of their skin abnormalities to a medical doctor for further evaluation. This feature becomes even more critical during the time of the COVID pandemic. The main advantages of 3D over 2D imaging are that the data is invariant to the translation and rotation of the object, surface texture, and external illumination. These advantages can significantly improve the repeatability of patient-driven measurements, enabling a better dermatological evaluation across different skin tones, pigmentation variations, as well as, skin conditions or tattoos.

While some success has been made in developing stateof-the-art artificial intelligence classification systems for skin diseases from 2D images alone, the initial attempts also leave room for significant improvement [11] . In particular, additional depth information could prove of benefit for Deep-Learning (DL) techniques by enriching the feature space.

Previously, several approaches using structure-frommotion or triangulation ("structured ligh") have been explored for mobile wound documentation and measurement [12] .

However, most open source proposed methods turned out to be too imprecise to provide detailed structural images of skin topography [12] . Active triangulation techniques perform significantly better but require a suitable and expensive sensor [5] .

In this paper, we propose to use mobile gradient illumination [13] to build a low-cost measurement system for dermatological evaluation of skin. In particular, our main contributions are 1. A method for inexpensive 3D capture of skin using commodity hardware (like screens and webcams, or tablets) 2. Preliminary results of 3D skin capture using the proposed method 3. Development of an open-source framework for gradient illumination-based 3D capture of skin. While this paper's focus is on the analysis and documentation in medical 3D imaging, the method we have developed could also have much broader applications. For instance, industrial object inspection or conservation in cultural heritage poses exciting scenarios for a mobile and low-cost 3D method [14] .

3D imaging poses a set of challenges with no universal solution. Achieving reliable measurements for different materials at arbitrary spatial and temporal resolutions is not feasible. Passive methods, such as Passive stereo [15] , Structure from Motion [16] , or even single-frame DL-based depth estimation [17] are not able to resolve fine details in 3D and are not feasible for our proposed task.

Active techniques, such as time-of-flight [18] or active triangulation ("structured-light") [19] , employ an additional light-source or projector to encode depth-information into the measurement. While active triangulation techniques would provide the required measurement precision on human skin [19] , the hardware necessary to perform the respective measurement is still rarely found in consumer devices. Another active method able to measure small surface details is Photometric Stereo (PS) [20] .

Here, multiple light sources are placed around the diffuse object, capturing images with each light source switched on and the remaining lights off. Compared to 3D estimations from shading in a single frame, the illumination from various directions supports the reliability of results. To be concise, the amount of reflected light is dependent on the light source positions relative to the object. Thus, one can resolve the surface orientation. Debevec et al. [21] extended PS by the idea of placing multiple light sources and cameras in a Light stage surrounding the object of interest. However, the complicated and bulky setup makes it difficult to apply it in a mobile setting.

Ma et al. [22] showed that a screen displaying intensity gradients could replace multiple light sources. The resulting Gradient Illumination (GI) approach can recover accurate 3D information by using a single camera with a polarization lens and a half-spherical-shaped light source. In 2016, Riviere et al. [13] proposed a mobile extension by using gradients displayed on the tablet screen to recover the surface normals for specular and diffuse reflection by placing a polarization lens on the front camera. All these so-called "reflectance-based" methods are highly dependent on the reflective properties of the objects of interest. For specular objects, Willomitzer et al. [14] proposed the idea of utilizing a tablet to perform deflectometry measurements using the screen and front camera. However, this system is not tailored to measure rather diffuse objects like human skin. This paper's method offers substantially improved results on human skin. 

For our desired application of measuring human skin, we identify gradient illumination as a suitable measurement technique. A depth map can be obtained from the measured surface normal data by integration algorithms, such as, the Frankot-Chellapa algorithm [23] . Gradient illumination employs an electronic display where each pixel can be considered as an individual light-source.

Our experimental setup consists of only a programmable screen and a camera (see Fig. 1 ) and, in principle, can be implemented with just a single mobile device (e.g., smartphone or tablet) according to the solution presented in [14] . When imaging objects with Lambertian reflectance, one can define the observed radiance r from a viewing position v (e.g., a camera ray in Fig. 2h) as an integration over a product of the incident illumination direction vector ω created by the gradient patterns P ( ω) x,y programmed on the screen, and the bidirectional reflectance distribution function (BRDF). For diffuse objects the BRDF R is approximated as the product of the diffuse albedo ρ d and the foreshortening factor, dependent on ω and the surface normal direction n = (n x , n y , n z ) T :

Because a screen can only project gradients along two directions, e.g., x and y, we assume n to be aligned with z = [0, 0, 1], which coincides with the centroid of a diffuse reflectance lobe of a flat surface pointing towards the screen. By integrating Eq. 2 the radiance is resolved according to [22] as

Compared to Ma et al. [22] , we do not integrate over the entire hemisphere. Instead, the integration is only performed in the x and y directions. Displaying the gradient patterns in both x and y directions for both [0, 1] and [1, 0], we calculate r x , r y andr xry , respectively. We can resolve the x and y components of the normal by:

Finally, we estimate the z-component assuming a normal by:

Open source software framework

The methods presented in this paper are part of an opensource software framework that can be downloaded from 1 . The framework is implemented in Python and has been tested on Ubuntu, macOS, and Windows. To encourage individual adjustments and extensions, we subdivide our project in an object-oriented manner with base classes for components necessary in any structured light setup. These include a camera, projector, image reconstruction, and calibration class. Consequently, the classes for the specific cameras, projectors, and reconstruction and radiometric, intrinsic, and extrinsic calibration are inherited from these base classes, allowing specific adjustments. Namely, we have adapted camera classes for Basler cameras, the Pi-Cam and Webcams. The projection unit provides functionality for any display or projector. Radiometric calibration is accomplished by placing a color chart next to the sample, as displayed in Fig. 3a . Consequently, the camera's intensity values are linearized by applying a gamma correction, see Fig. 3b . The last pattern displayed is always a constant illumination at maximum intensity, which allows for normalization across the frames. Intrinsic calibration is implemented using the OpenCV tool kit [24] . Extrinsic calibration is accomplished by placing a mirror with ChArUco markers [25] on the edges under the screen. The pose of the mirror is retrieved from the ChArUco markers. Further, a checkerboard pattern is displayed on the screen. Consequently, the camera captures the reflection of the pattern in the mirror. Thus rotation and translation between camera, object, and screen are retrieved. The components are then used in dedicated calibration and capture sessions, yielding the calibration data, normal maps, integrated depth maps, point clouds, and color images. The results are bundled and saved in a folder.

In this section, we present reconstructed surface normal maps from different objects. Our experiments' setup consists of a 15 inch LCD screen and a Basler ace 2 grayscale surface camera, both steadily mounted on a monitor arm and tripod, facing the object of interest. A radiometric calibration has to be employed to ensure consistency of the computed normal orientations over a surface with color variance. Therefore a matte Color Gauge Micro chart, containing linearly reflective grayscale tiles, is used to correct the captured intensity values. The intensity correction function to linearise the observed grayscale values is shown in Fig. 3b . We present the acquired normal maps in a typical red, green and blue (RGB) fashion resembling normalized x, y, and z components. The normal maps alongside the grayscale albedo images are displayed in Fig. 4 . Moreover, the integration of the acquired surface normals yields a depth map. Point clouds are created from the depth maps, which are saved in a mesh for visualization, see Fig. 4 and Fig. 1 . The texture on the point cloud is obtained by averaging the images captured under each gradient illumination. As can be seen from the 3D rendering, in Fig. 1 , an accurate 3D representation of the folds and wrinkles of the skin has been captured, and the result is independent of environmental factors (e.g., ambient illumination, viewing direction, skin texture) so that the results are highly repeatable. As discussed, we postulate that such a method can significantly increase the reliability of assessment and monitoring of skin disease in remote health applications.

In addition to the results of a 3D-scan of a human hand shown in Fig. 2 , we provide three more reconstructions showcasing that our proposed setup is a viable option for dermatologic diagnosis. Firstly, we captured two moles arising from the surrounding tissue. The change of surface orientation, and therefore depth, is visible throughout the moles. This advantage can be directly tied to the ABCDE rule. Parameters of the rule may rely on color images only but can be significantly supported by shape estimation. For example, color is not directly coupled to the depth of a mole. Furthermore, shape and color boundaries do not always coincide. Finally, the color of an image is dependent on ambient lighting. Secondly, we imaged a banana peel where the skin has darkened over several areas of the peel. We have further inserted small razor cuts as well as dents pushed in with a finger. Specifically, it is essential to point out that the region around the cuts and dents has darkened. Color variations can make the assessment of 3D structures from (2D) images hard. As expected, the color variation is not visible in the normal map as the surface orientation remains unchanged. This shows that gradient illumination is independent of the surface texture albedo and indeed provides structural information. In particular, the cuts and dents are visible. While the reflectance properties of bananas are quite different from human skin, the example still shows that our method can factor the reflectance information from 3D shape features similar to what dermatological applications require.

To showcase another example of varying colors, we imitated different human skin tones with silicone used in the special effects industry. Consequently, silicone tones were molded in a plastic cup to recreate six human skin tones. The skin replicas and their corresponding 3D reconstructions of the embedded number '16' are shown in Fig. 4 . Although there is a slight variance between the samples, the number is resolved for all frames. It is essential to mention that the arising number has a height of fewer than 0.5 millimeters.

We have proposed a method for mobile 3D imaging with lowcost components using gradient illumination for dermatological imaging. The framework requires only a screen (e.g., a tablet or external monitor) and a camera (e.g. the Pi-CAM) to provide high-quality 3D measurements of human skin or similar surfaces. We have also developed an open-source framework to provide the software programming necessary for each component required by our method, including cameras, displays, calibration techniques, and image reconstruction algorithms. Although we have focused on using our method for dermatological applications, its use may be much broader, e.g., in digital cultural heritage or biological imaging on-site.

While our method assumes diffuse reflectance, the reflection properties of skin across the entire human body and for various skin tones and ethnicities is much more complicated. As the next steps for our framework, we will incorporate approximate BRDF models and wavelength-dependent illumination to improve 3D reconstruction accuracy for arbitrary skin types.

Challenges that remain are the separation of diffuse and specular components on surfaces with varying reflective features without using a polarization lens. Specifically, this poses challenges for very shiny wounds with high fat or water content.

In future studies, we will evaluate the clinical use-case of our method with our clinical partners. One aspect will be to develop machine-learning algorithms that improve the accuracy of the classical 2D-RGB skin disease classification algorithms by incorporating the structural 3D information provided by our method.

Early detection of skin cancer via terahertz spectral profiling and 3d imaging

Prevalence of a history of skin cancer in 2007: results of an incidence-based model

Advanced basal cell carcinoma: epidemiology and therapeutic innovations

Abcde-an evolving concept in the early detection of melanoma

Handheld 3d scanning system for in-vivo imaging of skin cancer

Does melanoma behave differently in younger children than in adults? a retrospective study of 33 cases of childhood melanoma from a single institution

Lesion classification using 3d skin surface tilt orientation

Oct imaging of skin cancer and other dermatological diseases

Volume estimation of skin ulcers: Can cameras be as accurate as laser scanners?

Telemedicine in dermatology: findings and experiences worldwide-a systematic literature review

Fairness of classifiers across skin tones in dermatology

A comparison of open source libraries ready for 3d reconstruction of wounds

Mobile surface reflectometry

Hand-guided qualitative deflectometry with a mobile device

Review of stereo vision algorithms: from software to hardware

Structure-from-motion revisited

Deep convolutional neural fields for depth estimation from a single image

Time-of-flight cameras in computer graphics

Single-shot 3d motion picture camera with a dense point cloud

In vivo measurement of skin microrelief using photometric stereo in the presence of interreflections

The light stages and their applications to photoreal digital actors

Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination

A method for enforcing integrability in shape from shading algorithms

A camera calibration technique based on opencv

Charuco board-based omnidirectional camera calibration method