Image Deblurring Method Based on a Dual-Discriminator Weighted Generative Adversarial Network
-
摘要: 原始生成对抗网络(generative adversarial network, GAN)在训练过程中容易产生梯度消失及模式崩溃的问题,去模糊效果不佳。由此本文提出双判别器加权生成对抗网络(dual discriminator weighted generative adversarial network, D2WGAN)的图像去模糊方法,在GAN的基础上增加了一个判别器网络,将正向和反向KL(Kullback-Leibler)散度组合成一个目标函数,引入加权的思想调整正向和反向KL散度的比例,利用两个散度的互补特性,在学习清晰图片过程中避免不良模式的形成。实验结果表明,与现有方法相比,本文方法能更真实地恢复图像细节部分,且在评价指标峰值信噪比和图像结构相似度上有更好的表现。Abstract: The original generative adversarial network (GAN) is susceptible to the problems of vanishing gradients and mode collapse during the training process, and its deblurring effectiveness is poor. This study proposes an image deblurring method using a dual-discriminator weighted GAN. To extend the original GAN, a discriminator network is added to combine the forward and reverse Kullback–Leibler (KL) divergences into an objective function, and weights are used to adjust the ratio of forward and reverse KL divergences to leverage the complementary characteristics of the two divergences to avoid the formation of undesirable patterns in the process of learning clear pictures. Theoretical analysis proves that when an optimal discriminator is given, the difference between the forward and reverse KL divergences between real and generated data can be minimized. Experimental results demonstrate that compared to the existing methods, the proposed method can restore the details of an image more realistically and provides better performance in terms of the evaluation indexes of peak signal-to-noise ratio and structural similarity.
-
Key words:
- generation adversarial network /
- weighted /
- dual discriminator /
- image deblurring
-
表 1 判别器网络结构
Table 1. Discriminator network structure
Input (256×256×3) 4×4,64,stride=2,LeakyReLU 4×4,128,stride=2,instanceNorm2d,LeakyReLU 4×4,256,stride=2,instanceNorm2d,LeakyReLU 4×4,512,stride=1,instanceNorm2d,LeakyReLU 4×4,1,stride=1 SoftPlus 表 2 不同方法对图 6中单张图像的质量评价结果
Table 2. The quality evaluation results of the single image in Fig. 6 by different methods
Evaluation indices LR GAN DeblurGAN D2WGAN PSNR/dB 23.03 27.96 30.54 33.08 SSIM 0.79 0.83 0.88 0.91 表 3 不同方法在GOPRO验证集上的图像质量评价
Table 3. Image quality evaluation of different methods on GOPRO validation set
Evaluation indices GAN DeblurGAN D2WGAN PSNR/dB 25.82 27.15 28.98 SSIM 0.772 0.815 0.891 -
[1] 李明东, 张娟, 伍世虔, 等. 基于RANSAC变换的车牌图像去模糊算法[J]. 传感器与微系统, 2020, 39(2): 153-156, 160. https://www.cnki.com.cn/Article/CJFDTOTAL-CGQJ202002043.htmLI Mingdong, ZHANG Juan, WU Shiyu, et al. A deblurring algorithm for license plate image based on RANSAC transform[J]. Sensors and Microsystems, 2020, 39(2): 153-156, 160. https://www.cnki.com.cn/Article/CJFDTOTAL-CGQJ202002043.htm [2] 马苏欣, 王家希, 戴雅淑, 等. 监控视频下模糊车牌的去模糊与识别探析[J]. 信息系统工程, 2019(11): 111-113. doi: 10.3969/j.issn.1001-2362.2019.11.046MA Suxin, WANG Jiaxi, DAI Yashu, et al. Research on the deblurring and recognition of fuzzy license plates under surveillance video[J]. Information System Engineering, 2019(11): 111-113. doi: 10.3969/j.issn.1001-2362.2019.11.046 [3] 裴慧坤, 颜源, 林国安, 等. 基于生成对抗网络的无人机图像去模糊方法[J]. 地理空间信息, 2019, 17(12): 4-9, 155. doi: 10.3969/j.issn.1672-4623.2019.12.002FEI Huikun, YAN Yuan, LIN Guoan et al. Deblurring method of UAV image based on generative confrontation network[J]. Geospatial Information, 2019, 17(12): 4-9, 155. doi: 10.3969/j.issn.1672-4623.2019.12.002 [4] 黄允浒, 吐尔洪江, 唐泉, 等. 一种基于à trous算法的遥感图像模糊集增强算法[J]. 计算机应用与软件, 2018, 35(3): 187-192, 246. https://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ201803037.htmHUANG Yunhu, TU Erhong, TANG Quan, et al. A remote sensing image fuzzy set enhancement algorithm based on à trous algorithm[J]. Computer Applications and Software, 2018, 35(3): 187-192, 246. https://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ201803037.htm [5] 张广明, 高爽, 尹增山, 等. 基于模糊图像和噪声图像的遥感图像运动模糊复原方法[J]. 电子设计工程, 2017, 25(18): 82-86. doi: 10.3969/j.issn.1674-6236.2017.18.020ZHANG Guangming, GAO Shuang, YI Zengshan, et al. Remote sensing image motion blur restoration method based on blurred image and noise image[J]. Electronic Design Engineering, 2017, 25(18): 82-86. doi: 10.3969/j.issn.1674-6236.2017.18.020 [6] 吴庆波, 任文琦. 基于结构加权低秩近似的泊松图像去模糊[J]. 北京航空航天大学学报, 2020, 46(9): 1701-1710. https://www.cnki.com.cn/Article/CJFDTOTAL-BJHK202009010.htmWU Qingbo, REN Wenqi. Poisson image deblurring based on structure-weighted low-rank approximation[J]. Journal of Beijing University of Aeronautics and Astronautics, 2020, 46(9): 1701-1710. https://www.cnki.com.cn/Article/CJFDTOTAL-BJHK202009010.htm [7] RICHARDSON W. Bayesian-based iterative method of image restoration[J]. Journal of the Optical Society of America, 1972, 62(1): 55-59. doi: 10.1364/JOSA.62.000055 [8] LUCY B. An iterative technique for the rectification of observed distributions[J]. The Astronomical Journal, 1974, 79(6): 745-754. http://pdfs.semanticscholar.org/30ad/4474c7d6e9fd68b3e0fa2db235f1c8bc32f0.pdf [9] IAN G, JEAN P, MEHDI M, et al. Generative adversarial nets[C]//Adv. in 27th Neural Inf. Processing Syst. (NIPS), 2014: 2672-2680. [10] LEDIG C. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proc. of the IEEE Conf. on Comp. Vis. and Patt. Recog. (CVPR), 2017: 105-114. [11] LI Y, ZHAO K, ZHAO J. Research on super-resolution image reconstruction based on low-resolution infrared sensor[J]. IEEE Access, 2020(8): 69186-69199. http://ieeexplore.ieee.org/document/9052738/ [12] LI Z, WANG W, ZHAO Y. Image Translation by Domain-Adversarial Train[J]. Compu. Intel. And Neuro. , 2018: 1-11. Doi: 10.1155/2018/8974638. [13] YANG T, CHANG X, SU H, et al. Raindrop removal with light field image using image inpainting[J]. IEEE Access, 2020(8): 58416-58426. http://ieeexplore.ieee.org/document/9040628 [14] Mirza M, Osindero S. Conditional generative adversarial nets[J/OL]. arXiv preprint arXiv: 1411.1784, 2014, https://arxiv.org/abs/1411.1784. [15] Orest K, Volodymyr B, Mykola M, et al. DeblurGAN: Blind motion deblurring using conditional adversarial networks[C]//Proc. of the IEEE Conf. on Comp. Vis. And Patt. Recog., 2018: 8183-8192. [16] NGUYENT, LE T, VU H. Dual discriminator generative adversarial nets[C]//Proc. 29th Int. Conf. Neur. Inf. Pro. Sys., 2017: 2667-2677. [17] Lucas T, Aäron V, Matthias B. A note on the evaluation of generative models[J/OL]. arXiv preprint arXiv: 1511.01844, 2015. https://arxiv.org/abs/1511.01844 [18] IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[C]//ICML'15: Proceedings of the 32nd International Conference on International Conference on Machine Learning, 2015, 37: 448-456. [19] Ulyanov D, Vedaldi A, Lempitsky V. Instance normalization: the missing ingredient for fast stylization[C]//Proc. of the IEEE Conf. on Comp. Vis. and Patt. Recog. (CVPR), 2016: 1-13. [20] LI C, WAND M. Precomputed Real-time texture synthesis with markovian generative adversarial networks[C]//European Conference on Computer Vision, 2016: 702-716. [21] Maas L, Hannun Y, Ng Y. Rectifier nonlinearities improve neural network acoustic models[C]//Proc. ICML., 2013: 1-3. [22] JOHNSON J, ALAHI A, FEI L. Perceptual losses for real-time style transfer and super-resolution[C]//Proc. of European Conference on Computer Vision, 2016: 694-711. [23] SUN J, CAO W, XU Z, et al. Learning a convolutional neural network for non-uniform motion blur removal[C]//Proc. of the IEEE Conf. on Comp. Vis. and Patt. Recog. (CVPR), 2015: 769-777. [24] NAH S, KIM H, LEE M. Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//IEEE Conf. on Comp. Vis. and Patt. Recog. (CVPR), 2017: 257-265. [25] Kingma D, Ba J. Adam: A method for stochastic optimization[C]//Int. Conf. for Learning Representations (ICLR), 2015: 1-15.