key: cord-0065068-3l7vjkgx
authors: Kumar, Ashok; Jain, Arpit
title: Image smog restoration using oblique gradient profile prior and energy minimization
date: 2021-06-28
journal: Front Comput Sci
DOI: 10.1007/s11704-020-9305-8
sha: 523acec102fe52ece5e07dec1a635f49358aa0a0
doc_id: 65068
cord_uid: 3l7vjkgx

Removing the smog from digital images is a challenging pre-processing tool in various imaging systems. Therefore, many smog removal (i.e., desmogging) models are proposed so far to remove the effect of smog from images. The desmogging models are based upon a physical model, it means it requires efficient estimation of transmission map and atmospheric veil from a single smoggy image. Therefore, many prior based restoration models are proposed in the literature to estimate the transmission map and an atmospheric veil. However, these models utilized computationally extensive minimization of an energy function. Also, the existing restoration models suffer from various issues such as distortion of texture, edges, and colors. Therefore, in this paper, a convolutional neural network (CNN) is used to estimate the physical attributes of smoggy images. Oblique gradient channel prior (OGCP) is utilized to restore the smoggy images. Initially, a dataset of smoggy and sunny images are obtained. Thereafter, we have trained CNN to estimate the smog gradient from smoggy images. Finally, based upon the computed smog gradient, OGCP is utilized to restore the still smoggy images. Performance analyses reveal that the proposed CNN-OGCP based desmogging model outperforms the existing desmogging models in terms of various performance metrics. ELECTRONIC SUPPLEMENTARY MATERIAL: Supplementary material is available in the online version of this article at 10.1007/s11704-020-9305-8.

Images taken in poor weather conditions such as fog, haze, smog, etc., greatly suffer from the poor visibility issue. Smog is a kind of air pollution, originally named for the mixture of haze and fog in the air. Figure 1 shows the impact of fog, haze, and smog on the natural image.

Smoggy images can reduce the efficiency of many imaging applications, therefore, the development of an efficient smog removal model i.e., desmogging is desirable. But, in the literature majority of the researchers have focused on hazy or foggy images. Therefore, the existing methods can not be applied directly to smoggy images to restore them. Because it may cause adversarial artifacts. Therefore, in this paper, we have focused image, (c) shows hazy image, and (d) shows a smoggy image. This is a synthetic image. It is found that the smoggy image has very poor visibility compared to the natural, foggy, and hazy images on the development of a novel smog restoration model which can restore the visual degradation due to smog by utilizing the inversion of an optical smog imaging model. Therefore, efficient estimation of the physical attributes of smoggy images is required to be estimated carefully.

Therefore, many visibility restoration models are proposed in the literature to restore smoggy images. Enhancement based methods such as minimum information loss and histogram distribution prior [1] , partial differential equation-based enhancement [2] , optimized contrast enhancement [3] , etc., are not so effective to restore the smoggy images. These methods only try to enhance the visual characteristics of images and not try to restore them [4] . Therefore, the enhancement based restoration models fail to restore smoggy images [5] .

To overcome these issues many channel priors are implemented in literature which can be used to restore smoggy images. Some most commonly used channel priors are as optimal transmission map under scene priors (OTSP) [6] , color ellipsoid prior (CEP) [7] , block-to-pixel interpolation and adaptive dark channel prior (BDCP) [8] , dark channel prior loss (DCPL) [9] , energy minimization and dark channel prior (EMDCP) [10] , scene-aware sky-segmented dark channel prior (SSDCP) [11] , gradient channel prior (GCP) [12] , variational minimization based oblique gradient profile prior (VMOGPP) [13] , etc.

The prime contributions of the proposed smog removal model are as follows:

1) A convolutional neural network (CNN) is utilized to approximate the physical attributes of smoggy images. 2) Oblique gradient channel prior (OGCP) is used to remove the smog from still images. 3) Initially, a dataset of smoggy and sunny images are obtained. 4) Thereafter, we have trained the constitutional neural network to estimate the smog gradient on smoggy images. 5) Finally, based upon the computed smog gradient, OGCP is utilized to restore the still smoggy images.

The remaining paper is decomposed into the following sections: related work about the visibility restoration models is presented in Section 2. The proposed visibility restoration model is discussed in Section 3. Performance analyses are illustrated in Section 4. Finally, the conclusion is presented in Section 5.

Zhao et al. [14] used multi scale fusion model to improve the estimation of transmission region for image desmogging. In this, transmission map is fused patch-wise and pixel-wise to prevent misestimated transmission region. Jiang et al. [15] combined the bright and dark channel priors on superpixels to restore the smoggy images. This technique can also correct the misestimation of atmospheric light and transmission map for black and white pixels. Wang et al. [16] designed a convex model to desmog the images based on sparse dark channel prior. In this technique, l1 regularization term, a data-fitting term, and two total variation regularization terms are utilized to recover the smog free image. Gui et al. [17] implemented a model to recover the smog-free images using support vector machine. The idea behind this is that every desmogging approach shows good results when it works for some smog concentration. Yin et al. [18] utilized the parallel spatial/channel-wise attention block to recover the smoggy images. Channel-wise and spatial attention modules are used to explore the inter-dependencies among the channel-wise features and spatial positions of features, respectively. Liang et al. [19] used attenuation map and detailed preservation method to recover the smoggy images. A piecewise linear transform is used for each color channel of attenuation map to process the information. Multiscale decomposition is used to preserve the lost details. Zhang et al. [20] proposed a method to recover the smoggy images based on improved dark channel prior. It also determines the enhancement parameters optimally using logarithmic enhancement algorithm. Emberton et al. [21] recovered the smoggy images by detecting and segmenting the pure smoggy regions. A semantic white balancing method is also applied to address the spectral distortion present in the smoggy images. Xiao et al. [22] utilized deep learning to recover the smoggy images. Guided filter is also used to eliminate the halos and block artefacts. Guo et al. [23] proposed a desmogging technique based on deep convolutional network and fusion strategy. In this, five maps such as exposure map, saliency map, balance map, gamma correction map, and haze veil map are obtained from original smoggy image. a U-shaped deep convolutional network is used to blend these maps to generate a smog-free image. Gao et al. [24] studied a dual fusion approach to recover the smoggy images. It creates sky and non-sky regions using a segmentation method. Thereafter, a multi-region fusion strategy is applied to optimize the transmission map. Khan et al. [25] utilized wavelet domain to estimate the atmospheric light and transmission map of smoggy image. Smog is eliminated from the low frequency sub-band of smoggy image. Borkar and Mukherjee [26] used adaptive nearest neighbor regularization to recover the smog-free images. To preserve the texture detail, small patches with intensity distribution of smoggy images are used. Soft matting technique is used to evaluate the transmission map for smog removal. Xu et al. XU201950 used iterative desmogging technique to eliminate the smog from the images. In this technique, pixel-wise and patch-wise operations are also combined to remove the artefacts. Galdran [27] used gamma-correction and multi-scale Laplacian blending to remove the smog from the images. The benefit of this method is that it does not require estimation of costly depth in scene. Singh and Kumar [28] used a gradient profile prior to eliminate the smog from the images. Yuan et al. [29] proposed a transmission fusion method using automatic image matting to obtain the smog-free images. Wang et al. [30] implemented a desmogging technique using variational model. A gradient algorithm is used to solve the problem of constrained optimization. Alajarmeh and Salam [31] proposed a method to recover smog-free images using estimation of constant-time airlight and linear transmission.

From the extensive review, it has been found that the development of an efficient smog removal model is an open area of research. Also, the use of machine and deep learning techniques to predict smog density is ignored in the literature.

This section discusses the proposed smog removal model.

A physical smog imaging model can be represented as [12, 32, 33] :

Here, the S i (q) ∈ R N×3 shows a smoggy image. I r (q) defines actual scene radiance. γ shows atmospheric light and in color domain γ = A r , A g , A b . η(q) ∈ R N defines transmission map.q ∈ R N shows pixel coordinates. The transmission map depends upon d(q) of an actual radiance. The main objective of desmogging is to restore the smoggy images by efficiently approximating the η and γ from S i (q) [34] [35] [36] [37] [38] .

3.2 Oblique gradient channel prior An OGCP is based upon the statistical observations that in 3 × 3 mask of smog-free images, the majority of the gradient values are dark and model toward 0 (for more details please see [39] [40] [41] ). It is mathematically evaluated as:

Here, Ω(q) shows 3 × 3 mask centered at q. Δ defines oblique gradient operator. Let γ is already known and the transmission in masks is depicted usingη(q) is constant. Therefore, a gradient operator over color channels and pixels in a smog imaging model (Eq. (1)) (significantly normalized by using I r c (ι)) and obtain an estimated transmission map as:

Here ω = 0.97 is used to prevent cartoon effect issue which may occur if we completely restores smoggy images. For sky segments, an OGCP does not always exist, thus, we assume that the S i /γ → 1. Therefore,η(q) → 0. The obtainedη(q) needs to be improved by using filtering.

The smog imaging model (see Eq. (1)) is almost identical to the image matting model [42] . The obtained image is a convex linear collection of background and foreground images. These are constrained by a α matte. If we replace the α-matte with obtainedη(q), then the an energy function [42] can be defined to improve η(q) as:

Here, is control attribute and set as = 10 −39 . λ defines a Laplacian-like matrix and can be defined as [42] :

Here, pixels i, j ∈ p n around pixel n. |p n | is a mask size. µ n ∈ R 3 and Σ n ∈ R 3×3 define an average and co-variance of mask. ψ 3 defines an identity matrix. ε shows a filtering attribute and set to ε = 10 −55 .

The energy function in Eq. (4) is like a Laplacian matrix by considering their respective coefficients as in Eq. (5). By rewriting Eq. (4) in terms of coefficients as:

Here, m = 9. The summation of every overlapping masks over N pixels is used to obtain η in a 3 × 3 mask. Now vectorization is achieved by using data fidelity term as:

Here, represent element-wise projection. k ∈ [1 · · · m 2 ] show various set of pixels in a 3 × 3 mask. φ ∈ R N×m 2 define vectorization of coefficients. τ I , τ J ∈ R N×m 2 defines the computed transmission map. Initially, transmission masks are placed in I → (1, . . . , 1, 2, . . . , 2, . . . , m, . . . , m) ∈ R m 2 , and then the remaining are placed in J → (1, 2, . . . , m, 1, 2, . . . , m, . . . , 1, 2, . . . , m) ∈ R m 2 .

Te estimated transmission map is controlled using η θ . The hyper-parameters are optimized by using minimizing Eq. (7) over a training set of smoggy images, {S ir } R r=1 :

Here, R shows total number of smoggy images.

After model building, η θ (q) of a smoggy image can be estimated by using a forward-pass process. It is utilized to restore the smoggy image by considering Eq. (1) as:

Here, η 0 = 0.1 is used to prevent over restoration and cartoonist effect issues. To obtain airlight, i.e., γ, top 0.1% maximum pixels are selected from the obtained OGCP of a smoggy image. The respective indexes of the brightest pixel in S i are selected to obtain γ.

Initially, we have trained CNN [33, 37, 43, 44] on a set of smoggy and smog-free images to by setting smog density as a target label (see Fig. 2 ). We have used 0 to 9 levels of smog density. 0 means very less smog and 9 means heavy smog density. Thereafter, OGCP is used to estimate the physical The complete layout of the CNN based training and smog gradient prediction model is shown in Fig. 3 . Four maxpooling and ReLU function are used to compute the features from the smoggy and smog-free images i.e., four convolution operators are applied. Thereafter, softmax is used to evaluate the probability density function for the images. Finally, probability density function is used to predict the smog density from the images [43] .

We have collected 500 smog-free and 500 smoggy images for training purposes. It is assumed that during training purpose the smog density of every image is known. Seven haze removal models are used for evaluating the performance of the proposed smog removal model. These models are OTSP [6] , CEP [7] , [6] , (c) CEP [7] , (d) BDCP [8] , (e) DCPL [9] , (f) EMDCP [10] , (g) SSDCP [11] , (h) VMOGPP [13] and (i) proposed model BDCP [8] , DCPL [9] , EMDCP [10] , SSDCP [11] , and CLT. The proposed smog removal model is implemented on a core i5 processor with 8-GB RAM on MATLAB 2018a software. All the considered smog removal models are also implemented in the same environment. Their hyper-parameters are defined as they are reported in their respective papers. We have obtained a natural image and add a synthetic smog in them by using a MATLAB code for experimental purposes. Figure 4 shows the computed transmission map analyses among the proposed and the competitive smog restoration models. It has been found that the transmission map obtained from the proposed model is not affected by the noise and over-saturation issues as compared to the existing smog restoration models.

Visual analyses among the existing and the proposed smog restoration models are shown in Figs. 5, 6, and 7. It has been observed that OTSP [6] and BDCP [8] suffer from texture distortion, halo and gradient reversal artifacts, and color distortion [6] , (c) CEP [7] , (d) BDCP [8] , (e) DCPL [9] , (f) EMDCP [10] , (g) SSDCP [11] , (h) VMOGPP [13] and (i) proposed model Fig. 6 Visual analyses among the existing and the proposed smog restoration models (a) smoggy image, (b) OTSP [6] , (c) CEP [7] , (d) BDCP [8] , (e) DCPL [9] , (f) EMDCP [10] , (g) SSDCP [11] , (h) VMOGPP [13] and (i) proposed model Fig. 7 Visual analyses among the existing and the proposed smog restoration models (a) Smoggy image, (b) OTSP [6] , (c) CEP [7] , (d) BDCP [8] , (e) DCPL [9] , (f) EMDCP [10] , (g) SSDCP [11] , (h) VMOGPP [13] , and (i) proposed model issues. Also, the red marked areas show that these models per-form poorly especially for regions that are similar to sky-region or contain more smog gradient.

The computed smog-free images from CEP [7] and DCPL [9] perform significantly better as compared to OTSP [6] and BDCP [8] . However, these models suffer from edge and texture distortion issues.

The smog free images obtained by using EMDCP [10] , SS-DCP [11] , and VMOGPP [13] provide efficient results as compared to OTSP [6] , BDCP [8] , CEP [7] and DCPL [9] . But these techniques are unable to preserve texture information of images.

The proposed smog removal model is capable of effectively remove smog while preserving an edge, texture information, and vivid color of restored images. It is due to the use of fusionbased coarse transmission map estimation and novel regularized based variational model.

The percentage of saturated pixels values between the proposed and the existing models are depicted in Table 1 . It is desirable to be a minimum value. It has been found that the proposed model achieves significantly minimum saturated values as compared to the competitive models. Therefore, the proposed model obtains significantly remarkable images as it has a lesser number of saturated pixels.

The ratio of new visible edges (e) and ratio of mean gradient (r) analyses among the existing and the proposed visibility restoration models are shown in Tables 2 and 3 . From these tables, it has been found that the proposed model obtains significantly better values as compared to the existing smog restoration models in terms of e andr.

From the extensive review, it has been found that the restoration of smoggy images is defined as an ill-posed problem. Therefore, many smog removal (i.e., desmogging) models were implemented so far to restore smoggy images. It has been found that the smog removal models were based on a physical model. Therefore, an efficient approximation of physical attributes from a single smoggy image was achieved by utilizing CNN and OGCP to restore the smoggy images. Initially, a dataset of smoggy and sunny images was obtained. Thereafter, we have trained CNN to approximate the smog gradient of smoggy images. Finally, based upon the computed smog gradient, OGCP was utilized to restore the still smoggy images. Performance analyses have shown that the proposed CNN-OGCP based desmogging model outperforms the existing desmogging models in terms of various performance metrics. In this paper, we have not considered the optimal selection of hyper-parameters, therefore, in near future various algorithms such as genetic algorithm [45] , parallel strength pareto evolutionary algorithm-II [46] , multi-objective genetic algorithm [47, 48] , memetic differential evolution, [49] , etc., can be used to tune the hyperparameters of the proposed model.

Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior

Partial differential equation-based hazy image contrast enhancement

Optimized contrast enhancement for real-time image and video dehazing

Effective image enhancement techniques for fog-affected indoor and outdoor images

Let you see in sand dust weather: a method based on halo-reduced dark channel prior dehazing for sanddust image enhancement

Single-image dehazing via optimal transmission map under scene priors

Single image dehazing using color ellipsoid prior

Real-time single image dehazing using blockto-pixel interpolation and adaptive dark channel prior

Unsupervised single image dehazing using dark channel prior loss

Single image dehazing based on dark channel prior and energy minimization

Scene-aware image dehazing based on sky-segmented dark channel prior

Single image dehazing using gradient channel prior

Single image desmogging using oblique gradient profile prior and variational minimization. Multidimensional Systems and Signal Processing

Multi-scale optimal fusion model for single image dehazing

Image dehazing using adaptive bichannel priors on superpixels. Computer Vision and Image Understanding

A convex single image dehazing model via sparse dark channel prior

Adaptive single image dehazing method based on support vector machine

A novel image-dehazing network with a parallel attention block

Single underwater image enhancement by attenuation map guided color correction and detail preserved dehazing

Image dehazing based on dark channel prior and brightness enhancement for agricultural remote sensing images from consumer-grade cameras

Underwater image and video dehazing with pure haze region segmentation

Single image dehazing based on learning of haze layers

Single image dehazing based on fusion strategy

Single image dehazing via a dual-fusion method

Localization of radiance transformation for image dehazing in wavelet domain

Single image dehazing by approximating and eliminating the additional airlight component

Image dehazing by artificial multiple-exposure image fusion

A novel dehazing model for remote sensing images

Image dehazing based on a transmission fusion strategy by automatic image matting

A constrained total variation model for single image dehazing

Real-time framework for image dehazing based on linear transmission and constant-time airlight estimation

Dehazing of outdoor images using notch based integral guided filter

Deep learning approach for microarray cancer data classification

Single image defogging by gain gradient image filter

Analytical analysis of single-stage pressure relief valves

Study on the dynamic and static characteristics of gas static thrust bearing with micro-hole restrictors

Convolutional neural network based detection and judgement of environmental obstacle in vehicle operation

Image dehazing using window-based integrated means filter

Color image dehazing using gradient channel prior and guided l0 filter

Image dehazing using moore neighborhood-based gradient profile prior. Signal Processing: Image Communication

Engine speed reduction for hydraulic machinery using predictive algorithms

A closed form solution to natural image matting

Classification of COVID-19 patients from chest CT images using multi-objective differential evolution-based convolutional neural networks

TDD-net: a tiny defect detection network for printed circuit boards

Beta chaotic map based image encryption using genetic algorithm

Parallel strength pareto evolutionary algorithm-ii based image encryption

An efficient image encryption using nondominated sorting genetic algorithm-III based 4-D chaotic maps

Color image encryption using nondominated sorting genetic algorithm with local chaotic search based 5D chaotic map

Color image encryption approach based on memetic differential evolution

Acknowledgements The authors would like to thank their organizations especially Teerthanker Mahaveer University, Moradabad, India to provide suitable time and resources to successfully finish this research work.