key: cord-0123682-ipquknjw
authors: Tian, Chunwei; Zhang, Xuanyu; Lin, Jerry Chun-Wen; Zuo, Wangmeng; Zhang, Yanning
title: Generative Adversarial Networks for Image Super-Resolution: A Survey
date: 2022-04-28
journal: nan
DOI: nan
sha: 5c0e6632564ba8a63d7f39511a7dd85a3e2dd1e0
doc_id: 123682
cord_uid: ipquknjw

Single image super-resolution (SISR) has played an important role in the field of image processing. Recent generative adversarial networks (GANs) can achieve excellent results on low-resolution images with small samples. However, there are little literatures summarizing different GANs in SISR. In this paper, we conduct a comparative study of GANs from different perspectives. We first take a look at developments of GANs. Second, we present popular architectures for GANs in big and small samples for image applications. Then, we analyze motivations, implementations and differences of GANs based optimization methods and discriminative learning for image super-resolution in terms of supervised, semi-supervised and unsupervised manners. Next, we compare performance of these popular GANs on public datasets via quantitative and qualitative analysis in SISR. Finally, we highlight challenges of GANs and potential research points for SISR.

S INGLE image super-resolution (SISR) is an important branch in the field of image processing [1] . It also aims to recover a high-resolution (HR) image over a low-resolution (LR) image [2] , leading to in its wide applications in medical diagnosis [3] , video surveillance [4] and disaster relief [5] etc. For instance, in the medical field, obtaining higher-quality images can help doctors accurately detect diseases [6] . Thus, studying SISR is very meaningful to academia and industry.

To address SISR problem, researchers have developed a variety of methods based on degradation models of low-level vision tasks [7] . There are three categories for SISR in general, i.e., image itself information, prior knowledge and machine learning. In the image itself information, directly amplifying resolutions of all pixels in a LR image through an interpolation way to obtain a HR image was a simple and efficient method in SISR [8] , i.e., nearest neighbor interpolation [9] , bilinear interpolation [10] and bicubic interpolation [11] , etc. It is noted that in these interpolation methods, high-frequency information is lost in the up-sampling process [8] . Alternatively, reconstruction-based methods were developed for SISR, according to optimization methods [12] . That is, mapping a projection into a convex set to estimate the registration parameters can restore more details of SISR [13] . Although the mentioned methods can overcome the drawbacks of image itself information methods, they still suffered the following challenges: non-unique solution, slow convergence speed and higher computational costs. To prevent this phenomenon, the priori knowledge and image itself information were integrated into a frame to find an optimal solution to improve the quality of the predicted SR images [14, 15] . Using maximum a posteriori (MAP) can regularize a loss function to obtain a maximum probability for improving the efficiency [16] . Besides, machine learning methods can be presented to deal with SISR, according to relation of data distribution [17] . On the basis of ensuring the image SR effect, sparse-neighborembedding-based (SpNE) method via partition the training data set into a set of subsets to accelerate the speed of SR reconstruction [18] . There are also many other SR methods [17, 19] that often adopt sophisticated prior knowledge to restrict the possible solution space with an advantage of generating flexible and sharp detail. However, the performance of these methods rapidly degrades when the scale factor is increased, and these methods tend to be time-consuming [20] .

To obtain a better and more efficient SR model, a variety of deep learning methods were applied to a large-scale image dataset to solve the super-resolution tasks. For instance, Dong et al. proposed a super-resolution convolutional neural network (SRCNN) based pixel mapping that used only three layers to obtain stronger learning ability than these of some popular machine learning methods on image super-resolution [21] . Although the SRCNN had a good SR effect, it still faced problems in terms of shallow architecture and high complexity. To overcome challenges of shallow architectures, Kim et al. [22] designed a deep architecture by stacking some small convolutions to improve performance of image superresolution. Tai et al. [23] relied on recursive and residual operations in a deep network to enhance learning ability of arXiv:2204.13620v1 [eess.IV] 28 Apr 2022 a SR model. To further improve the SR effect, Lee et al. [24] used weights to adjust residual blocks to achieve better SR performance. To extract robust information, the combination of traditional machine learning methods and deep networks can restore more detailed information for SISR [25] . For instance, Wang et al. [25] embedded sparse coding method into a deep neural network to make a tradeoff between performance and efficiency in SISR. To reduce the complexity, an up-sampling operation is used in a deep layer in a deep CNN to increase the resolution of low-frequency features and produce highquality images [26] . For example, Dong et al. [26] directly exploited the given low-resolution images to train a SR model for improving training efficiency, where the SR network used a deconvolution layer to reconstructing HR images. There are also other effective SR methods. For example, Lai et al. [27] used Laplacian pyramid technique into a deep network in shared parameters to accelerate the training speed for SISR. Zhang et al. [28] guided a CNN by attention mechanisms to extract salient features for improving the performance and visual effects in image SISR.

Although the mentioned SR methods have obtained excellent effect in SISR, the obtained damaged images are insufficient in the real world, which limits the application of the above SR methods on real cameras. To address problem of small samples, generative adversarial nets (GANs) used generator and discriminator in a game-like manner to obtain good performance on image applications [29] . Specifically, the generator can generate new samples, according existing samples [30] . The discriminator is used to distinguish the samples from generator [30] . Due to their strong learning abilities, GANs become popular image super-resolution methods [31] . However, there are few studies summarizing these GANs for SISR. In this paper, we conduct a comprehensive overview of 193 papers to show their performance, pros, cons, complexity, challenges and potential research points, etc. First, we show the effects of GANs for image applications. Second, we present popular architectures for GANs in big and small samples for image applications. Third, we analyze motivations, implementations and differences of GANs based optimization methods and discriminative learning for image super-resolution in terms of supervised, semi-supervised and unsupervised manners. Fifth, we compare these GANs using experimental setting, quantitative analysis (i.e., PSNR, SSIM, complexity and running time) and qualitative analysis. Finally, we report on potential research points and existing challenges of GANs for image super-resolution. The overall architecture of this paper is shown in Fig. 1 .

The remainder of this survey is organized as follows. Section II resents the developments of GANs. Section III gives a brief introduction of basic GANs for image processing tasks. Section IV focuses on introduction of existing GANs via three ways on SISR. Section V compares performance of mentioned GANs from Section IV for SISR. Section VI offers potential directions and challenges of GANs in image super-resolution. Section VII concludes the overview. 

Traditional machine learning methods prefer to use prior knowledge to improve performance of image processing applications [32] . For instance, Sun et al. [32] proposed a gradient profile to restore more detailed information for improving performance of image super-resolution. Although machine learning methods based prior knowledge has fast execution speed, they have some drawbacks. First, they required manual setting parameters to achieve better performance on image tasks. Second, they required complex optimization methods to find optimized parameters. According to mentioned challenges, deep learning methods are developed [33] . Deep learning methods used deep networks, i.e., CNNs to automatically learn features rather than manual setting parameters to obtain effective effects in image processing tasks, i.e., image classification [33] , image inpainting [34] and image super-resolution [1] . Although these methods are effective big samples, they are limited for image tasks with small samples [29] .

To address problems above, GANs are presented in image processing [29] . GANs consist of generator network and discriminator network. The generator network is used to generate new samples, according to given samples. The discriminator network is used to determine truth of obtained new samples. When generator and discriminator is balance, a GAN model is finished. The work process of GAN can be shown in Fig. 2 , where G and D denote a generator network and discriminator network. To better understand GANs, we introduce several basic GANs as follows. To obtain more realistic effects, conditional information is fused into a GAN (CGAN) to randomly generate images, which are closer to real images [35] . CGAN improves GAN to obtain more robust data, which has an important reference value to GANs for computer vision applications. Subsequently, increasing the depth of GAN instead of the original multilayer perceptron in a CNN to improve expressive ability of GAN is developed for complex vision tasks [36] . To mine more useful information, the bidirectional generative adversarial network (BiGAN) used dual encoders to collaborate a generator and discriminator to obtain richer information for improving performance in anomaly detection, which is shown in Fig. 3 [37] . In Fig. 3 , x denotes a feature vector, E is an encoder and y expresses an image from discriminator. It is known that pretrained operations can be used to accelerate the training speed of CNNs for image recognition [38] . This idea can be treated as an energy drive. Inspired by that, Zhao et al. proposed an energy-based generative adversarial network (EBGAN) by using a pretraining operation into a discriminator to improve the performance in image recognition [39] . To keep consistency of obtained features with original images, cycle-consistent adversarial network (Cycle-GAN) relies on a cyclic architecture to achieve an excellent style transfer effect [40] as illustrated in Fig. 4 .

Although pretrained operations are useful for training efficiency of network models, they may suffer from mode collapse. To address this problem, Wasserstein GAN (WGAN) used weight clipping to enhance importance of Lipschitz constraint to improve the stability of training a GAN [41] . WGAN used weight clipping to perform well. However, it is easier to cause gradient vanishing or gradient exploding [42] . To resolve this issue, WGAN used a gradient penalty (treated as WGAN-GP) to break the limitation of Lipschitz for pursuing good performance in computer vision applications [43] . To further improve results of image generation, GAN enlarged batch size and used truncation trick as well as BIGGAN can make a tradeoff between variety and fidelity [43] . To better obtained features of different parts of an image (i.e., freckles and hair), style-based GAN (StyleGAN) uses feature decoupling to control different features and finish style transfer for image generation [44] . The architecture of StyleGAN and its generator are shown in Fig. 5 and Fig. 6 .

Real sample In recent years, GANs with good performance have been applied in the fields of image processing, natural language processing (NLP) and video processing. Also, there are other variants based on GANs for multimedia applications, such as Laplacian pyramid of GAN (LAPGAN) [45] , coupled GAN (CoupleGAN) [46] , self-attention GAN (SAGAN) [47] , losssensitive GAN (LSGAN) [48] . These methods emphasize how to generate high-quality images through various sampling mechanisms. However, researchers focused applications of GANs from 2019, i.e., FUNIT [49] , SPADE [50] and U-GAT-IT [51] . Illustrations of more GANs are shown in Table I. III. POPULAR GANS FOR IMAGE APPLICATIONS According to mentioned illustrations, it is known that variants of GANs based on properties of vision tasks are developed in Section II. To further know GANs, we show different GANs on training data, i.e., big samples and small samples for different high-and low-level computer vision tasks as shown in Fig. 7 . A. Popular GANs on big samples for image applications 1) GANs on big samples for image generation: Good performance of image generation depends on rich samples. Inspired by that, GANs are improved for image generation [30] . That is, GANs use generator to produce more samples from high-dimensional data to cooperate discriminator for promoting results of image generation. For instance, boundary equilibrium generative adversarial networks (BEGAN) used obtained loss from Wasserstein to match loss of auto-encoder in the discriminator and achieve a balance between a generator and discriminator, which can obtain more texture information than that of common GANs in image generation [52] . To control different parts of a face, StyleGAN decoupled different features to form a feature space for finishing transfer of texture information [44] . Besides, texture synthesis is another important application of image generation [53] . For instance, Markovian GANs (MGAN) can quickly capture texture date of Markovian patches to achieve function of real-time texture synthesis [53] , where Markovian patches can be obtained Ref. [30] . Periodic spatial GAN (PSGAN) [54] is a variant of spatial GAN (SGAN) [55] , which can learn periodic textures of big datasets and a single image. These methods can be summarized in Table II .

2) GANs on big samples for object detection: Object detection has wide applications in the industry, i.e., smart transportation [56] and medical diagnosis [57] , etc. However, complex environments have huge challenges for pursuing good performance of object detection methods [58] . Rich data is important for object detection. Existing methods used a data-driven strategy to collect a large-scale dataset including different object examples under different conditions to obtain an object detector. However, the obtained dataset does not contain all kinds of deformed and occluded objects, which limits effects of object detection methods. To resolve the issue, GANs are used for object detection [59, 60] . Ehsani et al. used segmentation and generation in a GANs from invisible parts in the objects to overcome occluded objects [59] . To address a challenge of small object detection on low-resolution and noisy representation, a perceptual GAN (Perceptual GAN) reduced differences of small objects and big objects to improve performance in small object detection [60] . That is, its generator converted poor perceived representation from small objects to high-resolution big objects to fool a discriminator, where mentioned big objects are similar to real big objects [60] . To obtain sufficient information of objects, an end-to-end multi-task generative adversarial network (SOD-MTGAN) used a generator to recover detailed information for generating high-quality images for achieving accurate detection [61] . Also, a discriminator transferred classification and regression losses in a back-propagated way into a generator [61] . Two operations can extract objects from backgrounds to achieve good performance in object detection [59] . More detailed information is shown in Table III. B. GANs on small samples for image applications 1) GANs on small samples for image style transfer: Makeup has important applications in the real world [62] . To save costs, visual makeup software is developed, leading to image style transfer (i.e., image-to-image) translation becoming a research hotspot in the field of computer vision in recent years [30] . GANs are good tools for style transfer on small samples, which can be used to establish mappings between given images and object images [30] . The obtained mappings are strongly related to aligned image pairs [63] . However, we found that the above mappings do not match our ideal models in terms of transfer effects [40] . Motivated by that, CycleGAN used two pairs of a generator and discriminator in a cycle consistent way to learn two mappings for achieving style transfer [40] . CycleGAN had two phases in style transfer. In the first phase, an adversarial loss [50] was used to ensure the quality of generated images. In the second phase, a cycle consistency loss [40] was utilized to guarantee that predicted images to fell into the desired domains [64] . CycleGAN had the following merits. It does not require paired training examples [64] . And it does not require that the input image and the output image have the same low-dimensional embedding space [40] . Due to its excellent properties, many variants of CycleGAN have been conducted for many vision tasks, i.e., image style transfer [40, 65] , object transfiguration [66] and image enhancement [67] , etc. More GANs on small samples for image style transfer can be found in Table IV. 2) GANs on small samples for image inpainting: Images have played important roles in human-computer interaction in the real world [69] . However, they may be damaged when they were collected by digital cameras, which has a negative impact on high-level computer vision tasks. Thus, image inpainting had important values in the real world [70] . Due to missing [33] CGAN Image classification Conditional GAN for image classification PD-GAN [34] GAN Image inpainting GAN for image inpainting and image restoration CGAN [35] GAN Image generation GAN in a supervised way for image generation DCGAN [36] GAN Image generation GAN in an unsupervised way for image generation BiGAN [37] GAN Image generation GAN with encoder in an unsupervised way for image generation EBGAN [39] GAN Image generation and training nets GAN based energy for image generation CycleGAN [40] GAN Image generation GAN with cycle-consistent for image generation WGAN-GP [42] GAN Image generation GAN with gradient penalty for image generation BIGGAN [43] GAN Image super-resolution GAN with big channels of image super-resolution StyleGAN [44] GAN Image generation GAN with stochastic variation for image generation LAPGAN [45] CGAN Image super-resolution GAN with Laplacian pyramid for image super-resolution CoupleGAN [46] GAN Image generation GAN for both up-sampling and image generation SAGAN [47] GAN Image generation Unsupervised GAN with self-attention for image generation FUNIT [49] GAN Image translation GAN in an unsupervised way for image-to-image translation SPADE [50] GAN Image generation GAN with spatially-adaptive normalization for image generation U-GAT-IT [51] GAN Image translation GAN with attention in an unsupervised way for image-to-image translation CycleGAN with U-net for image-to-image translation ECycleGAN [67] CycleGAN CycleGAN with convolutional block attention module (CBAM) for image-to-image translation pixels, image inpainting suffered from enormous challenges [71] . To overcome shortcoming above, GANs are used to generate useful information to repair damaged images based on the surrounding pixels in the damaged images [72] . For instance, GAN used a reconstruction loss, two adversarial losses and a semantic parsing loss to guarantee pixel faithfulness and local-global contents consistency for face image inpainting [73] . Although this method can generate useful information, which may cause boundary artifacts, distorted structures and blurry textures inconsistent with surrounding areas [74, 75] .

To resolve this issue, Zhang et al. embedded prior knowledge into a GAN to generate more detailed information for achieving good performance in image inpainting [74] . Yu et al. exploited a contextual attention mechanism to improve a GAN for obtaining excellent visual effect in image inpainting [75] . Typical GANs on small samples for image inpainting is summarized in Table V . GAN GAN with prior knowledge for face inpainting GFC [73] GAN GAN with autoencoder for image inpainting GIICA [75] WGAN WGAN with attention model for image inpainting IV. GANS FOR IMAGE SUPER-RESOLUTIONS According to mentioned illustrations, it is clear that GANs have many important applications in image processing. Also, image super-resolution is crucial for high-level vision tasks, i.e., medical image diagnosis and weather forecast, etc. Thus, GANs in image super-resolution have important significance in the real world. However, there are few summaries about GANs for image super-resolution. Inspired by that, we show GANs for image super-resolution, according to supervised GANs, semi-supervised GANs and unsupervised GANs for image super-resolution as shown in Fig. 8 . Specifically, supervised GANs in image super-resolution include supervised GANs based improved architectures, supervised GANs based prior knowledge, supervised GANs with improved loss functions and supervised GANs based multi-tasks for image superresolution. Semi-supervised GANs for image super-resolution contain semi-supervised GANs based improved architectures, semi-supervised GANs with improved loss functions and semi-supervised GANs based multi-tasks for image superresolution.

Unsupervised GANs for image super-resolution consists of unsupervised GANs based improved architectures, unsupervised GANs based prior knowledge, unsupervised GANs with improved loss functions and unsupervised GANs based multitasks in image super-resolution. More information of GANs on image super-resolution can be illustrated as follows.

A. Deep CNNs based group convolutions for SISR 1) Supervised GANs based improved architectures for image super-resolution: GANs in a supervised way to train image super-resolution models are very mainstream. Also, designing GANs via improving network architectures are very novel. Thus, improved GANs in a supervised way for image super-resolution are very popular. That can improve GANs by designing novel discriminator networks, generator networks, attributes of image super-resolution task, complexity and computational costs. For example, Laplacian pyramid of adversarial networks (LAPGAN) fused a cascade of convolutional networks into Laplacian pyramid network in a coarse-to-fine way to obtain high-quality images for assisting image recognition task [45] . To overcome the effect of big scales, curvature and highlight compact regions can be used to obtain a local salient map for adapting big scales in imageresolution [76] . More research on improving discriminators and generators is shown as follows.

In terms of designing novel and discriminators and generators, progressive growing generative adversarial networks (PGGAN or ProGAN) utilized different convolutional layers to progressively enlarge low-resolution images to improve image qualities for image recognition [77] . An enhanced SR-GAN (ESRGAN) used residual dense blocks into a generator without batch normalization to mine more detailed information for image super-resolution [78] . To eliminate effects of checkerboard artifacts and the unpleasing high-frequency, multi-discriminators were proposed for image super-resolution [79] . That is, a perspective discriminator was used to overcome checkerboard artifacts and a gradient perspective was utilized to address unpleasing high-frequency question in image superresolution. To improve the perceptual quality of predicted images, ESRGAN+ fused two adjacent layers in a residual learning way based on residual dense blocks in a generator to enhance memory abilities and added noise in a generator to obtain stochastic variation and obtain more details of highresolution images [80] .

Restoring detailed information may generate artifacts, which can seriously affect qualities of restored images [81] . In terms of face image super-resolution, Zhang et al. used a supervised pixel-wise GAN (SPGAN) to obtain higher-quality face images via given low-resolution face images of multiple scale factors to remove artifacts in image super-resolution [81] . In terms of remote sensing image super-resolution, Gong et al. used enlighten blocks to make a deep network achieve a reliable point and used self-supervised hierarchical perceptual loss to overcome effects of artifacts in remote sensing image super-resolution [82] . Dharejo et al. used Wavelet Transform (WT) characteristics into a transferred GAN to eliminate artifacts to improve quality of predicted remote sensing images [83] . Moustafa et al. embedded squeeze-and-excitation blocks and residual blocks into a generator to obtain more highfrequency details [84] . Besides, Wasserstein distance is used to enhance the stability of training a remote sensing superresolution model [84] . To address pseudo-textures problem, a saliency analysis is fused with a GAN to obtain a salient map that can be used to distinguish difference between a discriminator and a generator [85] .

To obtain more detailed information in image superresolution, a lot of GANs are developed [86] . Ko et al. used Laplacian idea and edge in a GAN to obtain more useful information to improve clarities of predicted face images [86] . Using tensor structures in a GAN can facilitate texture information for SR [87] . Using multiple generators in a GAN can obtain more realistic texture details, which was useful to recover high-quality images [88, 89] . To obtain better visual effects, a gradually GAN used gradual growing factors in a GAN to improve performance in SISR [90] .

To reduce computational costs and memory, Ma et al. used two-stage generator in a supervision way to extract more effective features of cytopathological images, which can reduce the cost of data acquisition and save cost [91] . Cheng et al. designed a generator by multi-scale feature aggregation and a discriminator via a PatchGAN to reduce memory consumption for a GAN on SR [92] . Besides, distilling a generator and discriminator can accelerate the training efficiency of a GAN model for SR [92] . More supervised GANs for image superresolution are shown in Table VI 2) Supervised GANs based prior knowledge for image super-resolution: It is known that combination of discriminative method and optimization can make a tradeoff between efficiency and performance [102] . Guan et al. used high-resolution image to low-resolution image network and low-resolution image to high-resolution image network with nearest neighbor down-sampling method to learn detailed information and noise prior for image super-resolution [103] . Chan GAN GAN with identity-based discriminator for face image super-resolution MLGE [86] LAPGAN LAPGAN with edge information for face image super-resolution SD-GAN [85] GAN GAN for remote sensing image super-resolution PathSRGAN [91] SRGAN SRGAN with RRDB for cytopathology image super-resolution Enlighten-GAN [82] GAN GAN with enlighten block for remote sensing image super-resolution TWIST-GAN [83] GAN GAN with wavelet transform (WT) for remote sensing image super-resolution SCSE-GAN [84] GAN GAN with SCSE block for image super-resolution MFAGAN [92] GAN GAN with multi-scale feature aggregation net for image super-resolution TGAN [87] GAN GAN with visual tracking and attention networks for image super-resolution DGAN [88] GAN GAN with disentangled representation learning and anisotropic BRDF reconstruction for image super-resolution DMGAN [89] GAN GAN with two same generators for image super-resolution G-GANISR [90] GAN GAN with gradual learning for image super-resolution SRGAN [93] GAN GAN with deep ResNet for image super-resolution RaGAN [94] GAN GAN with relativistic discriminator for image super-resolution LE-GAN [95] GAN GAN with a latent encoder for realistic hyperspectral image super-resolution NCSR [96] GAN GAN with a noise conditional layer for image super-resolution Beby-GAN [97] GAN GAN with a region-aware adversarial learning strategy for image super-resolution MA-GAN [98] GAN GAN with pyramidal convolution for image super-resolution CMRI-CGAN [99] CGAN CGAN with optical flow component for magnetic resonance image super-resolution D-SRGAN [100] SRGAN SRGAN for image super-resolution LMISR-GAN [101] GAN GAN with residual channel attention block for medical image super-resolution gradient prior into a GAN to suppress the effect of blur kernel estimation for image super-resolution [105] .

3) Supervised GANs with improved loss functions for image super-resolution: Loss function can affect performance and efficiency of a trained SR model. Thus, we analyze the combination of GANs with different loss functions in image super-resolution [106] . Zhang et al. trained a Ranker to obtain representation of perceptual metrics and used a rank-content loss in a GAN to improve visual effects in image superresolution [106] . To eliminate effect of artifacts, Zhu et al. used image quality assessment metric to implement a novel loss function to enhance the stability for image super-resolution [107] . To decrease complexity of GAN model in image superresolution, Fuoli et al. used a Fourier space supervision loss to recover lost high-frequency information to improve predicted image quality and accelerate training efficiency in SISR [108] . To enhance stability of a SR model, using residual blocks and a self-attention layer in a GAN enhances robustness of a trained SR model. Also, combining improved Wasserstein gradient penalty and perceptual Loss enhances stability of a SR model [109] . To extract accurate features, fusing a measurement loss function into a GAN can obtain more detailed information to obtain clearer images [110] .

4) Supervised GANs based multi-tasks for image superresolution: Improving image quality is important for highlevel vision tasks, i.e., image recognition [111] . Besides, devices often suffer from effects of multiple factors, i.e., device hardware, camera shakes and shooting distances, which results in collected images are damaged. That may include noise and low-resolution pixels. Thus, addressing the multitasks for GANs are very necessary [112] . For instance, Adil et al. exploited SRGAN and a denoising module to obtain a clear image. Then, they used a network to learn unique representative information for identifying a person [113] . In terms of image super-resolution and object detection, Wang et al. used multi-class cyclic super-resolution GAN to restore high-quality images, and used a YOLOv5 detector to finish object detection task [114] . Zhang et al. used a fully connected network to implement a generator for obtaining high-definition plate images and a multi-task discriminator is used to enhance super-resolution and recognition tasks [115] . The use of an adversarial learning was a good tool to simultaneously address text recognition and super-resolution [116] .

In terms of complex damaged image restoration, GANs are good choices [117] . For instance, Li et al. used a multiscale residual block and an attention mechanism in a GAN to remove noise and restore detailed information in CTA image super-resolution [117] . Nneji et al. improved a VGG19 to fine-tune two sub-networks with a wavelet technique to simultaneously address COVID-19 image denoising and superresolution problems [118] . More information is shown in Table  VII. B. Semi-supervised GANs for image super-resolution 1) Semi-supervised GANs based improved architectures for image super-resolution: For real problems with less data, semi-supervised techniques are developed. For instance, asking patients takes multiple CT scans with additional radiation doses to conduct paired CT images for training SR models in clinical practice is not realistic. Motivated by that, GANs in semi-supervised ways are used for image super-resolution [121] . For instance, by maintaining the cycle-consistency of Wasserstein distance, a mapping from noisy low-resolution GAN GAN with pre-trained models for image super-resolution I-SRGAN [105] GAN GAN with infrared prior knowledge for image super-resolution on infrared image RankSRGAN [106] SRGAN SRGAN with ranker for image super-resolution GMGAN [107] GAN GAN with a novel quality loss for image super-resolution FSLSR [108] GAN GAN with fourier space losses for image super-resolution I-WAGAN [109] GAN GAN with improved wasserstein gradient penalty and perceptual loss for image super-resolution CESR-GAN [110] GAN GAN with a feature-based measurement loss function for image super-resolution RTSRGAN [112] SRGAN SRGAN for real time image super-resolution MSSRGAN [113] ESRGAN ESRGAN with denoising module for image super-resolution RSISRGAN [114] GAN GAN for image super-resolution on RSI JPLSRGAN [115] GAN GAN for license plate recognition and image super-resolution SRR-GAN [116] GAN GAN for image super-resolution on text images MRD-GAN [117] GAN GAN with attention mechanism for image super-resolution and denoising MESRGAN+ [118] ESRGAN ESRGAN with siamese network for image super-resolution and denoising. RealESRGAN [56] ESRGAN ESRGAN with pure synthetic data for blind image super-resolution SNPE-SRGAN [119] SRGAN SRGAN with SPNE for image super-resolution SOUP-GAN [120] GAN GAN with 3D MRI for image super-resolution images to high-resolution images was built [121] . Besides, combining a convolutional neural network, residual learning operations in a GAN can facilitate more detailed information for image super-resolution [121] . To resolve super-resolution and few labeled samples, Xia et al. used soft multi-labels to implement a semi-supervised super-resolution method for person re-identification [122] . That is, first, a GAN is used to conduct a SR model. Second, a graph convolutional network is exploited to construct relationship of local features from a person. Third, some labeled samples are used to train unlabeled samples via a graph convolutional network.

2) Semi-supervised GANs with improved loss functions and semi-supervised GANs based multi-tasks for image superresolution: The combinations of semi-supervised GANs and loss functions are also effective in image super-resolution [123] . For example, Jiang et al. combined an adversarial loss, a cycle-consistency loss, an identity loss and a joint sparsifying transform loss into a GAN in a semi-supervised way to train a CT image super-resolution model [123] . Although this model made a significantly progress on some evaluation criteria, it was still disturbed by artifacts and noise.

In terms of multi-tasks, Nicolo et al. proposed to use a mixed adversarial Gaussian domain adaptation in a GAN in a semi-supervised way to obtain more useful information for implementing a 3D super-resolution and segmentation [124] . More information of semi-supervised GANs in image superresolution can be illustrated in Table VIII .

Collected images in the real world have less pairs. To address this phenomenon, unsupervised GANs are presented [125] . It can be divided into four types, i.e., improved architectures, prior knowledge, loss functions and multi-tasks in GANs in unsupervised ways for image super-resolution as follows.

1) Unsupervised GANs based improved architectures for image super-resolution: CycleGANs have obtained success in unsupervised ways in image-to-image translation applications [40] . Accordingly, the CycleGANs are extended into SISR to address unpair images (i.e., low-resolution and highresolution) in the datasets in the real world [125] . Yuan et al. used a CycleGAN for blind super-resolution over the following phases [125] . The first phase removed noise from noisy and low-resolution images. The second phase resorted to an upsampled operation in a pre-trained deep network to enhance the obtained low-resolution images. The third phase used a fine-tune mechanism for a GAN to obtain high-resolution images. To address blind super-resolution, bidirectional structural consistency was used into a GAN in an unsupervised way to train a blind SR model and construct high-quality images [126] . Alternatively, Zhang et al. exploited multiple GANs as basis components to implement an improved CycleGAN for train an unsupervised SR model [127] . To eliminate checkerboard artifacts, an upsampling module containing a bilinear interpolation and a transposed convolution was used in an unsupervised CycleGAN to improve visual effects of restored images in the real world [128] .

There are also other popular methods that use GANs in unsupervised ways for image super-resolution [129] . To improve the learning ability of a SR model in the real world, it combines an unsupervised learning and a mean opinion score in a GAN to improve perceptual quality in the real-world image super-resolution [129] . To recover more natural image characteristics, Lugmayr et al. combined unsupervised and supervised ways for blind image super-resolution [130] . The first step learned to invert the effects of bicubic down sampling operation in a GAN in an unsupervised way to extract useful information from natural images [130] . To generate image pairs in the real world, the second step used a pixel-wise network in a supervised way to obtain high-resolution images [130] . To break fixed downscaling kernel, Sefi et al. used KernelGAN [31] and Internal-GAN [131] to obtain an internal distribution of patches in the blind image super-resolution. To accelerate the training speed, a guidance module was used in a GAN to quickly seek a correct mapping from a lowresolution domain to a high-resolution domain in unpaired image super-resolution [132] . To improve the accuracy of medical diagnosis, Song et al. used dual GANs in a selfsupervised way to mine high dimensional information for PET image super-resolution [76] . Besides, other SR methods can have an important reference value for unsupervised GANs with for image super-resolution. For example, Wang et al. used an unsupervised method to translate real low-resolution images to real low-resolution images [133] . Chen et al. resorted to a supervised super-resolution method to convert obtained real low-resolution images into real high-resolution images [134] . More information of mentioned unsupervised GANs for image super-resolution can be shown in Table IX as follows. 2) Unsupervised GANs based prior knowledge for image super-resolution: Combining unsupervised GANs and prior knowledge in unsupervised GANs can better address unpair image super-resolution [136] . Lin et al. combined data error, a regular term and an adversarial loss to guarantee consistency of local-global content and pixel faithfulness in a GAN in an unsupervised way to train an image super-resolution model [136] . To better support medical diagnosis, Das et al. combined adversarial learning in a GAN, cycle consistency and prior knowledge, i.e., identity mapping prior to facilitate more useful information i.e., spatial correlation, color and texture information for obtaining cleaner high-quality images [137] . In terms of remoting sensing super-resolution, a random noise is used in a GAN to reconstruct satellite images [138] . Then, authors conducted image prior by transforming the reference [122] GAN GAN with soft multi-labels in a semi-supervised way for image super-resolution CTGAN [123] GAN GAN with four losses in a semi-supervised way for image super-resolution Gemini-GAN [124] GAN GAN with mixed adversarial Gaussian domain adaptation in a semi-supervised way for 3D super-resolution and segmentation 

Methods Key words

CinCGAN [125] GAN Unsupervised GAN for image super-resolution DNSR [126] GAN Unsupervised GAN with bidirectional structural consistency for blind image super-resolution MCinCGAN [127] CycleGAN Unsupervised GAN for image super-resolution RWSR-CycleGAN [128] CycleGAN Unsupervised GAN for image super-resolution USISResNet [129] GAN Unsupervised GAN with USISResNet for image super-resolution ULRWSR [130] GAN Unsupervised GAN with pixel wise supervision for image super-resolution KernelGAN [31] Internal-GAN Unsupervised GAN for blind image super-resolution InGAN [131] GAN Unsupervised GAN for image super-resolution FG-SRGAN [132] SRGAN Unsupervised GAN with a guided block for image super-resolution PETSRGAN [76] GAN Unsupervised GAN with a self-supervised way for PET image super-resolution TrGAN [133] GAN Unsupervised GAN for image synthesis and super-resolution CycleSR [134] GAN Unsupervised GAN with an indirect supervised path for image super-resolution UGAN-Circle [135] GAN-Circle Unsupervised GAN-Circle for image super-resolution on CT images image into a latent space [138] . Finally, they updated the noise and latent space to transfer obtained structure information and texture information for improving resolution of remote sensing images [138] .

3) Unsupervised GANs with improved loss functions for image super-resolution: Combining loss functions and GANs in an unsupervised way is useful for training image superresolution models in the real world [139] . For instance, Zhang et al. used a novel loss function based image quality assessment in a GAN to obtain accurate texture information and more visual effects [139] . Besides, an encoder-decoder architecture is embedded in this GAN to mine more structure information for pursuing high-quality images of a generator from this GAN [139] . Han et al. depended on SAGAN and L1 loss in a GAN in an unsupervised manner to act multisequence structural MRI for detecting braining anomalies [140] . Also, Zhang et al. fused a content loss into a GAN in an unsupervised manner to improve SR results of hyperspectral images [141] . Unsupervised GANs based prior knowledge and improved loss functions for image super-resolution can be summarized in Table X . [137] GAN GAN with cycle consistency and identity mapping priors for image super-resolution EIPGAN [138] GAN GAN with remote sensing image prior for image super-resolution URSGAN [139] GAN GAN with prior based image quality assessment for remote sensing image super-resolution MADGAN [140] SAGAN SAGAN with L1 loss for medical image super-resolution DLGAN [141] GAN GAN with a content loss for hyperspectral image super-resolution 4) Unsupervised GANs based multi-tasks for image superresolution: Unsupervised GANs are good tools to address multi-tasks, i.e., noisy low-resolution image super-resolution. For instance, Prajapati et al. transferred a variational autoencoder and the idea of quality assessment in a GAN to deal with image denoising and SR tasks [142] . Cui et al. relied on low-pass-filter loss and weighted MR images in a GAN in an unsupervised GAN to mine texture information for removing noise and recovering resolution of MRI images [143] . Cai et al. presented a pipeline that optimizes a periodic implicit GAN to obtain neural radiance fields for image synthesis and image super-resolution based on 3D [144] . More unsupervised GANs based multi-tasks for image super-resolution can be presented in Table XI .

To make readers conveniently know GANs in image superresolution, we compare super-resolution performance of these GANs from datasets and experimental settings to quantitative and qualitative analysis in this section. More information can be shown as follows.

Mentioned GANs can be divided into three kinds: supervised methods, semi-supervised methods and unsupervised methods for image super-resolution, which make datasets have three categories, training datasets and test datasets for supervised methods, semi-supervised methods and unsupervised methods. These datasets can be summarized as follows.

(1) Supervised GANs for image-resolution Training datasets: CIFAR10 [146] , STL [147] , LSUN [148] , ImageNet [149] , Celeb A [150] , DIV2K [151] , Flickr2K [152] , OST [153] , CAT [154] Market-1501 [155] , Duke MTMC-reID [156, 157] , GeoEye-1 satellite dataset [85] , Whole slide images (WSIs) [91] , MNIST [158] and PASCAL2 [159] .

Test datasets: CIFAR10 [146] , STL [147] , LSUN [148] , Set5 [160] , Set14 [161] , BSD100 [162] , CELEBA [150] , 

Methods Key words VAEGAN [142] GAN GAN with variational auto-encoder and quality assessment idea for image super-resolution and denoising ASLGAN [143] GAN GAN with low-pass-filter loss and weighted MR images for MRI image super-resolution and denoising Pix2NeRF [144] GAN Optimizing a periodic implicit GAN for 3D-aware image synthesis and super-resolution Pi-GAN [145] GAN GAN with periodic activation functions for 3D-aware image synthesis and image super-resolution OST300 [148] , CAT [154] , PIRM datasets [163] , Market-1501 [155] , GeoEye-1 satellite dataset [85] , WSIs [91] MNIST [158] and PASCAL2 [159] .

(2) Semi-supervised GANs for image-resolution Training datasets: Market-1501 [155] , Tibia Dataset [164] , Abdominal Dataset [165] , CUHK03 [166] , MSMT17 [167] , LUNA [168] , Data Science Bowl 2017 (DSB) [169] , UKDHP [170] , SG [170] and UKBB [171] .

Test datasets: Tibia Dataset [164] , Abdominal Dataset [165] , CUHK03 [166] , Widerface [172] , LUNA [168] , DSB [169] , SG [170] and UKBB [171] .

(3) Unsupervised GANs for image-resolution Training datasets: CIFAR10 [146] , ImageNet [149] , DIV2K [151] , DIV2K random kernel (DIV2KRK) [151] , Flickr2K [152] , Widerface [172] , NTIRE 2020 Real World SR challenge [173] , KADID-10K [174] , DPED [175] , DF2K [78] , NTIRE' 2018 Blind-SR challenge [176] , LS3D-W [177] , CELEBA-HQ [77] , LSUN-BEDROOM [178] , ILSVRC2012 [149, 179] , NTIRE 2020 [173] , 91-images [180] , Berkeley segmentation [162] , BSDS500 [181] , Training datasets of USROCTGAN [182, 183] , SD-OCT dataset [183] , UC Merced dataset [184] , NWPU-RESIS45 [185] and WHU-RS19 [186] .

Test datasets: CIFAR10 [146] , ImageNet [149] , Set5 [160] , Set14 [161] , BSD100 [162] , DIV2K [151] , DIV2KRK [151] , Urban100 [187] , Widerface [172] , NTIRE 2020 [173] , NTIRE 2020 Real-world SR Challenge [173] , NTIRE 2020 Real World SR challenge validation dataset [173] , DPED [175] , CELEBA-HQ [77] , LSUN-BEDROOM [178] , Test datasets of USROCTGAN [182, 183] , and Test datasets of USRGAN [139] .

These mentioned datasets about GANs for image superresolution can be shown in Table XII . To make readers easier understand datasets of different methods via different GANs for different training ways on image super-resolutions, we conduct Table 13 to show their detailed information.

In this section, we compare the differences of environment configurations between different GANs via different training ways (i.e., supervised, semi-supervised and unsupervised ways) for image super-resolution, which contain batch size, scaling factors, deep learning framework, learning rate and iteration. That can make readers easier to conduct experiments with GANs for image super-resolution. Their information can be listed as shown in Table XIV as follows.

To make readers understand the performance of different GANs on image super-resolution, we use quantitative analysis and qualitative analysis to evaluate super-resolution effects of these GANs. Quantitative analysis is PSNR and SSIM of different methods via three training ways on different datasets for image super-resolution, running time and complexities of different GANs on image super-resolution. Qualitative analysis is used to evaluate qualities of recovered images.

1) Quantitative analysis of different GANs for image superresolution: We use SRGAN [93] , PGGAN [77] , ESRGAN [78] , ESRGAN+ [80] , DGAN [88] , G-GANISR [90] , GM-GAN [107] , SRPGAN [93] , DNSR [126] , DULGAN [136] , CinCGAN [125] ,MCinCGAN [127] , USISResNet [129] , UL-RWSR [130] , KernelGAN [31] and CycleSR [134] in one training way from supervised, semi-supervised and unsupervised ways on a public dataset from Set14 [161] , BSD100 [162] and DIV2K [151] to test performance for different scales in image super-resolution as shown in Table XV . For instance, ESRGAN [78] outperforms SRGAN [93] in terms of PSNR and SSIM in a supervised ways on Set 14 for ×2, which shows that ESRGAN has obtained better superresolution performance for ×2. More information of these GANs can be shown in Table XV .

Running time and complexity are important indexes to evaluate performance of image super-resolution techniques in the real devices [192] . According to that, we conduct experiments of four GANs (i.e., ESRGAN [78] , PathSRGAN [91] , RankSRGAN [106] and KernelGAN [31] ) on two lowresolution images with sizes and for ×4 to test running time and compute parameters of different GANs. The conducted experiments have the following experimental environments. They can run on Ubuntu of 20.04.1, CPU of AMD EPYC ROME 7502P with 32 cores and Memory of 128G via PyTorch of 1.10.1 [189] . Besides, they depend on a NVIDIA GeFore RTX 3090 with cuda of 11.1 and cuDNN of. 8.0.4. In Table  XVI , we can see that ESRGAN [78] has slower speed than that of PathSRGAN for ×4 on image super-resolution. However, it uses less parameters than that of PathSRGAN for ×4 on image super-resolution. Thus, ESRGAN is competitive with PathSRGAN for image super-resolution. More information of different GANs for image super-resolution in terms of running time and parameters can be shown in Table XVI .

2) Qualitative analysis of different GANs for image superresolution: To test visual effects of different GANs for image super-resolution, we choose Bicubic, ESRGAN [78] , RankSRGAN [106] , KernelGAN [31] and PathSRGAN [91] to conduct experiments to obtain high-quality images for ×4. To further observe these images, we choose an area of predicted images from these GANs to amplify it as an observation area. Observation area is clearer, corresponding method has good superior SR performance. For example, ESRGAN [78] [45] CIFAR10 [146] , STL [147] , LSUN [148] CIFAR10 [146] , STL [147] , LSUN [148] SRGAN [93] ImageNet [149] Set5 [160] , Set14 [160] and BSD100 [162] PGGAN [77] CIFAR10 [146] , Celeb A [150] CELEBA [150] , LSUN [148] , CIFAR10 [146] ESRGAN [78] DIV2K [151] , Flickr2K [152] , OST [153] Set5 [160] , Set14 [161] , BSD100 [162] , Urban100 [187] RaGAN [94] CIFAR10 [146] , CAT [154] CAT [154] ESRGAN+ [80] DIV2K [151] BSD100 [162] , Urban100 [187] , OST300 [153] , Set5 [160] , Set14 [160] and the PIRM datasets [163] SPGAN [81] Market-1501 [155] , Duke MTMC-reID [156, 157] Market-1501 [155] SD-GAN [85] GeoEye-1 satellite dataset [85] GeoEye-1 satellite dataset [85] PathSRGAN [91] Whole slide images (WSIs) [91] WSIs [91] TGAN [87] MNIST [158] , PASCAL2 [159] , CIFAR10 [146] MNIST [158] , PASCAL2 [159] , CIFAR10 [146] DGAN [88] DIV2K [151] Set5 [160] , Set14 [161] , CIFAR-10 [146] , BSD100 [162] G-GANISR [90] Set5 [160] , Set14 [161] , BSD100 [162] , and Urban100 [187] Set5 [160] , Set14 [161] , and Urban100 [187] Semi-supervised ways GAN-CIRCLE [121] Tibia Dataset [164] , Abdominal Dataset [165] Tibia Dataset [164] , Abdominal Dataset [165] MSSR [122] Market-1501 [155] , CUHK03 [166] , MSMT17 [167] Market-1501 [155] , CUHK03 [166] CTGAN [123] LUNA [168] , Data Science Bowl 2017 (DSB) [169] LUNA [168] , DSB [169] Gemini-GAN [124] UKDHP [170] , SG [170] , UKBB [171] SG [170] , UKBB [171] Unsupervised ways

CinCGAN [125] DIV2K [151] DIV2K [151] DNSR [126] DIV2K [151] , Flickr2K [152] , Widerface [172] Set5 [160] , Set14 [161] , Urban100 [187] , BSD100 [162] , DIV2K [151] MCinCGAN [127] DIV2K [151] DIV2K [151] RWSR-CycleGAN [128] NTIRE 2020 Real World SR challenge [173] NTIRE 2020 Real World SR challenge [173] USISResNet [129] NTIRE-2020 Real-world SR Challenge validation dataset [173] , DIV2K [151] , Flickr2k [152] , KADID-10K [174] NTIRE-2020 Real-world SR Challenge validation dataset [173] ULRWSR [130] DPED [175] , DF2K [78] , DIV2K [151] , Flickr2K [152] DPED [175] , DIV2K [151] KernelGAN [31] NTIRE'2018 Blind-SR challenge [153] , DIV2K random kernel (DIV2KRK) [151] DIV2KRK [151] FG-SRGAN [132] LS3D-W [177] Widerface [172] TrGAN [133] CIFAR10 [146] , ImageNet [149] , CELEBA-HQ [77] , LSUN-BEDROOM [178] CIFAR10 [146] , ImageNet [149] , CELEBA-HQ [77] , LSUN-BEDROOM [178] PDCGAN [188] ImageNet [149] , ILSVRC2012 [149, 179] , CIFAR10 [146] , CIFAR-100 [146] ImageNet [149] CycleSR [134] DIV2K [151] , NTIRE 2020 [173] DIV2K [151] , NTIRE 2020 [173] DULGAN [136] 91-images [180] , Berkeley segmentation [162] , BSDS500 [181] Set5 [160] , Set14 [161] USROCTGAN [137] Training datasets of USROCTGAN [182, 183] , SD-OCT dataset [183] Test datasets of USROCTGAN [182] URSGAN [139] UC Merced dataset [184] , NWPU-RESIS45 [185] , WHU-RS19 [186] Test datasets of USRGAN [139] 

Testing time (s) Parameters ESRGAN [78] 9.7607 1.670×107 PathSRGAN [91] 9.1608 2.433×107 RankSRGAN [106] 7.3818 1.554×106 KernelGAN [31] 251.00 1.816×105

is clearer than that of PathSRGAN [91] on an image from the BSDS100 in Table XVII and Set14 in Table XVIII for ×4, which show that the ESRGAN is more effective in image super-resolution.

Variations of GANs have achieved excellent performance in image super-resolution. Accordingly, we provide an overview of GANs for image super-resolution to offer a guide for readers to understand these methods. In this section, we analyze challenges of current GANs for image super-resolution and give corresponding solutions to facilitate the development of GANs for image super-resolution.

Although GANs perform well in image super-resolution, they suffer from the following challenges. 2) Large computational resources and high memory consumption. A GAN is composed of a generator and discriminator, which may increase computational costs and memory consumption. This may lead to a higher demand on digital devices.

3) High-quality images without references. Most of existing GANs relied on paired high-quality images and low-resolution images to train image super-resolution models, which may be limited by digital devices in the real world. 4) Complex image super-resolution. Most of GANs can deal with a single task, i.e., image super-resolution and synthetic noisy image super-resolution, etc. However, collected images by digital cameras in the real world suffer from drawbacks, i.e., low-resolution and dark-lighting images, complex noisy and low-resolution images. Besides, digital cameras have higher requirement on the combination of image low-resolution and image recognition. Thus, existing GANs for image superresolution cannot effectively repair low-resolution images of mentioned conditions. 5) Metrics of GANs for image super-resolution. Most of existing GANs used PSNR and SSIM to test super-resolution performance of GANs. However, PSNR and SSIM cannot fully measure restored images. Thus, finding effective metrics is very essential about GANs for image super-resolution.

To address these problems, some potential research points about GANs for image super-resolution are stated below.

1) Enhancing a generator and discriminator extracts salient features to enhance stabilities of GANs on image superresolution. For example, using attention mechanism (i.e., Transformer [193] ), residual learning operations, concatenation operations act a generator and discriminator to extract more effective features to enhance stabilities for accelerating GAN models in image super-resolution.

2) Designing lightweight GANs for image super-resolution. Reducing convolutional kernels, group convolutions, the combination of prior and shallow network architectures can decrease the complexities of GANs for image super-resolution.

3) Using self-supervised methods can obtain high-quality reference images. 4) Combining attributes of different low-level tasks, decomposing complex low-level tasks into a single low-level task via different stages in different GANs repairs complex lowresolution images, which can help high-level vision tasks. 5) Using image quality assessment techniques as metrics evaluates quality of precited images from different GANs.

In this paper, we analyze and summarize GANs for image super-resolution. First, we introduce developments of GANs. Then, we present GANs in big samples and small samples for image applications, which can make readers easier GANs. Next, we give differences of GANs based optimization methods and discriminative learning for image super-resolution in terms of supervised, semi-supervised and unsupervised manners. Subsequently, we compare the performance of these popular GANs on public datasets via quantitative and qualitative analysis in SISR. Finally, we highlight challenges of GANs and potential research points on SISR. is currently an Associate Professor with the School of Software, Northwestern Polytechnical University. Also, he is a member of National Engineering Laboratory for Integrated Aerospace Ground-Ocean Big Data Application Technology. His research interests include image restoration and deep learning. He has published over 40 papers in academic journals and conferences, including IEEE TNNLS, IEEE TMM, IEEE TSMC, NN, Information Sciences, KBS, PRL, ICASSP, ICPR, ACPR and IJCB. He has four ESI highly-cited papers, three homepage papers of the Neural Networks, one homepage paper of the TMM, one excellent paper in 2018 and 2019 for the CAAI Transactions on Intelligence Technology. Also, his three codes are rated as the contribution codes of the GitHub 2020. His two paper techniques are integrated on the iHub and Profillic. He has obtained 2021 Shenzhen CCF Excellent Doctoral Dissertation. Besides, he is an associate editor of the Journal of Electrical and Electronic Engineering, International Journal of Image and Graphics and Journal of Artificial Intelligence and Technology, a guest editor of Mathematics and Electronics, PC Chair and Workshop Chair of MLCCIM 2022, SI Chair of ACAIT 2022, Special Session Co-Chair and Workshop Chair of ICCSI 2022, Publicity Chair of AMDS 2022, a workshop chair of ICCBDAI 2021, a reviewer of some journals and conferences, such as the IEEE TIP, the IEEE TII, the IEEE TNNLS, IEEE TCYB, the IEEE TSMC, the NN, the CVIU, the information fusion, the Neurocomputing, the Visual Computer, the PRL and the SPL, etc.

Xuanyu Zhang received the bachelor's degree from Xinjiang University, Urumqi, China, in 2020. He is currently pursuing the Master degree with the School of Software, Northwestern Polytechnical University, Xi'an, China. His research interests include image processing, deep learning on GANs model, and image super-resolution. He has applied Chinese two patents for invention and four software copyrights. Besides, he is a reviewer of the International Journal of Image and Graphics.

Deep learning for image superresolution: A survey

Deep learning for single image super-resolution: A brief review

Super resolution techniques for medical image processing

A super-resolution reconstruction algorithm for surveillance images

Bayesian deconvolution for angular super-resolution in forward-looking scanning radar

Simultaneous super-resolution and cross-modality synthesis of 3d medical images using weaklysupervised joint convolutional sparse coding

A local structure adaptive superresolution reconstruction method based on btv regularization

Super-resolution image reconstruction: a technical overview

Nearest neighbor value interpolation

New edge-directed interpolation

Cubic convolution interpolation for digital image processing

A survey on techniques and challenges in image super resolution reconstruction

High-resolution image recovery from imageplane arrays, using convex projections

Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images

Improving resolution by image registration

Superresolved surface reconstruction from multiple images

Image super-resolution using gradient profile prior

Image super-resolution with sparse neighbor embedding

Single image superresolution based on gradient profile sharpness

Super-resolution: a comprehensive survey

Image super-resolution using deep convolutional networks

Accurate image super-resolution using very deep convolutional networks

Image super-resolution via deep recursive residual network

Enhanced deep residual networks for single image super-resolution

Deep networks for image super-resolution with sparse prior

Accelerating the super-resolution convolutional neural network

Deep laplacian pyramid networks for fast and accurate super-resolution

Image superresolution using very deep residual channel attention networks

Generative adversarial nets

A review on generative adversarial networks: Algorithms, theory, and applications

Blind super-resolution kernel estimation using an internal-gan

Gradient profile prior and its applications in image super-resolution and enhancement

Fruit quality and defect image classification with conditional gan data augmentation

Pd-gan: Probabilistic diverse gan for image inpainting

Conditional generative adversarial nets

Unsupervised representation learning with deep convolutional generative adversarial networks

Adversarial feature learning

On pretrained image features and synthetic images for deep learning

Energy-based generative adversarial network

Unpaired image-to-image translation using cycle-consistent adversarial networks

Wasserstein generative adversarial networks

Improved training of wasserstein gans

Large scale gan training for high fidelity natural image synthesis

A style-based generator architecture for generative adversarial networks

Deep generative image models using a laplacian pyramid of adversarial networks

Coupled generative adversarial networks

Self-attention generative adversarial networks

Loss-sensitive generative adversarial networks on lipschitz densities

Few-shot unsupervised image-to-image translation

Semantic image synthesis with spatially-adaptive normalization

U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation

Began: Boundary equilibrium generative adversarial networks

Precomputed real-time texture synthesis with markovian generative adversarial networks

Learning texture manifolds with the periodic spatial gan

Texture synthesis with spatial generative adversarial networks

A real-time object detection solution and its application in transportation

Object detection for chest x-ray image diagnosis using deep learning with pseudo labeling

Object detection in 20 years: A survey

Segan: Segmenting and generating the invisible

Perceptual generative adversarial networks for small object detection

Sod-mtgan: Small object detection via multi-task generative adversarial network

Ramt-gan: Realistic and accurate makeup transfer with generative adversarial network

Image-to-image translation with conditional adversarial networks

Generating handwritten chinese characters using cyclegan

Arcyclegan: Improved cyclegan for style transferring of fruit images

U-net and residual-based cycle-gan for improving object transfiguration performance

Fundus image enhancement method based on cyclegan

Correlation alignment total variation model and algorithm for style transfer

Emotion recognition in human-computer interaction

Image inpainting: Overview and recent advances

Image inpainting: A review

Patch-based image inpainting with generative adversarial networks

Generative face completion

De-gan: Domain embedded gan for high quality face image inpainting

Generative image inpainting with contextual attention

Image super resolution using generative adversarial networks and local saliency maps for retinal image analysis," in International conference on medical image computing and computer-assisted intervention

Progressive growing of gans for improved quality, stability, and variation

Esrgan: Enhanced super-resolution generative adversarial networks

Multi-perspective discriminators-based generative adversarial network for image super resolution

Esrgan+: Further improving enhanced super-resolution generative adversarial network

Supervised pixel-wise gan for face superresolution

Enlighten-gan for super resolution reconstruction in midresolution remote sensing images

Twist-gan: Towards wavelet transform and transferred gan for spatio-temporal single image super resolution

Satellite imagery super-resolution using squeeze-and-excitation-based gan

Sd-gan: Saliency-discriminated gan for remote sensing image superresolution

Multi-laplacian gan with edge enhancement for face super resolution

Tgan: Deep tensor generative adversarial nets for large image generation

Diverse adversarial network for image super-resolution

Deep mutual gan for life-detection radar super resolution

G-ganisr: Gradual generative adversarial network for image super resolution

Pathsrgan: multi-supervised super-resolution for cytopathological images using generative adversarial network

Mfagan: A compression framework for memory-efficient on-device super-resolution gan

Photo-realistic single image super-resolution using a generative adversarial network

The relativistic discriminator: a key element missing from standard gan

A Latent Encoder Coupled Generative Adversarial Network (LE-GAN) for Efficient Hyperspectral Image Super-resolution

Noise conditional flow model for learning the super-resolution space

Bestbuddy gans for highly detailed image super-resolution

Multi-attention generative adversarial network for remote sensing image super-resolution

Super-resolution of cardiac mr cine imaging using conditional gans and unsupervised transfer learning

D-srgan: Dem super-resolution with generative adversarial networks

Medical image super-resolution using a relativistic average generative adversarial network

Learning deep cnn denoiser prior for image restoration

Srdgan: learning the noise prior for super resolution with dual generative adversarial networks

Glean: Generative latent bank for large-factor image super-resolution

Infrared image super resolution using gan with infrared image prior

Ranksrgan: Generative adversarial networks with ranker for image super-resolution

Ganbased image super-resolution with a novel quality loss

Fourier space losses for efficient perceptual image super-resolution

Breast cancer histopathology image super-resolution using wide-attention gan with improved wasserstein gradient penalty and perceptual loss

Proposing a novel cascade ensemble super resolution generative adversarial network (cesr-gan) method for the reconstruction of super-resolution skin lesion images

Super-low resolution face recognition using integrated efficient sub-pixel convolutional neural network (espcn) and convolutional neural network (cnn)

Rtsrgan: Real-time super-resolution generative adversarial networks

Multi scale-adaptive super-resolution person re-identification using gan

Remote sensing image super-resolution and object detection: Benchmark and state of the art

Joint license plate super-resolution and recognition in one multi-task gan framework

Srr-gan: Super-resolution based recognition with gan for low-resolved text images

Multiscale residual denoising gan model for producing super-resolution cta images

Fine-tuned siamese network with modified enhanced super-resolution gan plus based on low-quality chest x-ray images for covid-19 identification

Snpe-srgan: Lightweight generative adversarial networks for single-image superresolution on mobile using snpe framework

Soup-gan: Super-resolution mri using generative adversarial networks

Ct super-resolution gan constrained by the identical, residual, and cycle learning ensemble (gan-circle)

Real-world person re-identification via super-resolution and semi-supervised methods

A novel superresolution ct image reconstruction via semi-supervised generative adversarial network

Joint semi-supervised 3d superresolution and segmentation with mixed adversarial gaussian domain adaptation

Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks

Unsupervised degradation learning for single image super-resolution

Multiple cyclein-cycle generative adversarial networks for unsupervised image superresolution

Unsupervised real-world super resolution with cycle generative adversarial network and domain discriminator

Unsupervised single image super-resolution network (usisresnet) for real-world data using generative adversarial network

Unsupervised learning for real-world super-resolution

Ingan: Capturing and remapping the" dna" of a natural image

Fg-srgan: a feature-guided superresolution generative adversarial network for unpaired image superresolution

Transformation gan for unsupervised image synthesis and representation learning

Unsupervised image super-resolution with an indirect supervised path

Unsupervised gan-circle for high-resolution reconstruction of bone microstructure from low-resolution ct scans

Deep unsupervised learning for image super-resolution with generative adversarial network

Unsupervised super-resolution of oct images using generative adversarial network for improved agerelated macular degeneration diagnosis

Enhanced image prior for unsupervised remoting sensing super-resolution

An unsupervised remote sensing single-image super-resolution method based on generative adversarial network

Madgan: Unsupervised medical anomaly detection gan using multiple adjacent brain mri slice reconstruction

Degradation learning for unsupervised hyperspectral image super-resolution based on generative adversarial network

Unsupervised real-world super-resolution using variational auto-encoder and generative adversarial network

Unsupervised arterial spin labeling image superresolution via multiscale generative adversarial network

Pix2nerf: Unsupervised conditional pi-gan for single image to neural radiance fields translation

pigan: Periodic implicit generative adversarial networks for 3d-aware image synthesis

Learning multiple layers of features from tiny images

Unsupervised feature learning with c-svddnet

Large-scale scene understanding challenge: Room layout estimation

Imagenet large scale visual recognition challenge

Deep learning face attributes in the wild

Ntire 2017 challenge on single image super-resolution: Dataset and study

Flickr1024: A largescale dataset for stereo image super-resolution

Recovering realistic texture in image super-resolution by deep spatial feature transform

Cat head detection-how to effectively exploit shape and texture features

Scalable person re-identification: A benchmark

Performance measures and a data set for multi-target, multi-camera tracking

Unlabeled samples generated by gan improve the person re-identification baseline in vitro

Gradient-based learning applied to document recognition

The pascal visual object classes (voc) challenge

Low-complexity single-image super-resolution based on nonnegative neighbor embedding

On single image scale-up using sparse-representations

A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics

The 2018 pirm challenge on perceptual image super-resolution

Quantitative imaging of peripheral trabecular bone microarchitecture using mdct

Low dose ct image and projection data [data set

Deepreid: Deep filter pairing neural network for person re-identification

Person transfer gan to bridge domain gap for person re-identification

Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the luna16 challenge

Deep learning for lung cancer detection: tackling the kaggle data science bowl 2017 challenge

Mridb: medical image management for biobank research

The uk biobank resource with deep phenotyping and genomic data

Wider face: A face detection benchmark

Ntire 2020 challenge on real-world image super-resolution: Methods and results

Kadid-10k: A large-scale artificially distorted iqa database

Dslr-quality photos on mobile devices with deep convolutional networks

Ntire 2018 challenge on single image super-resolution: Methods and results

How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks)

Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop

Plug & play generative networks: Conditional iterative generation of images in latent space

Image super-resolution via sparse representation

Object contour detection with a fully convolutional encoder-decoder network

Fast acquisition and reconstruction of optical coherence tomography images via sparse representation

Sparsity based denoising of spectral domain optical coherence tomography images

Bag-of-visual-words and spatial extensions for land-use classification

Remote sensing image scene classification: Benchmark and state of the art

Satellite image classification via two-layer sparse coding with biased image representation

Single image super-resolution from transformed self-exemplars

cgans with projection discriminator

Pytorch: An imperative style, high-performance deep learning library

A tour of tensorflow

Unpaired image super-resolution using pseudosupervision

Deep learning on image denoising: An overview

Chunwei Tian (Member, IEEE) received his Ph.D degree in Computer Application Technique from Harbin Institute of Technology

 , IEEE TKDE, IEEE TFS, IEEE TNNLS, IEEE  TCYB, IEEE TII, IEEE TITS, IEEE TNSE, IEEE TETCI,