基于双判别器加权生成对抗网络的图像去模糊方法

黄梦涛; 高娜; 刘宝

基于双判别器加权生成对抗网络的图像去模糊方法

西安科技大学电气与控制工程学院，陕西西安 710054

基金项目:

陕西省重点研发计划项目 2019GY-097

陕西省重点研发计划项目 2021GY-131

西安市科技计划项目 2020KJRC0068

榆林市科技计划项目 CXY-2020-037

详细信息

作者简介:
黄梦涛（1965-），女，教授，博士，主要从事基于图像的测量与识别和智能系统等方面的研究。E-mail：huangmt@xust.edu.cn

通讯作者:
刘宝（1983-），男，讲师，硕士生导师，主要从事多源信息融合、图像处理等研究。E-mail：xiaobei0077@163.com

中图分类号: TN911.7
计量
- 文章访问数: 137
- HTML全文浏览量: 23
- PDF下载量: 27
- 被引次数: 0
出版历程
- 收稿日期: 2021-01-24
- 修回日期: 2021-04-08
- 刊出日期: 2022-01-20

Image Deblurring Method Based on a Dual-Discriminator Weighted Generative Adversarial Network

College of Electrical & Control Engineering, Xi'an University of Science and Technology, Xi'an 710054, China

摘要

摘要: 原始生成对抗网络（generative adversarial network, GAN）在训练过程中容易产生梯度消失及模式崩溃的问题，去模糊效果不佳。由此本文提出双判别器加权生成对抗网络（dual discriminator weighted generative adversarial network, D2WGAN）的图像去模糊方法，在GAN的基础上增加了一个判别器网络，将正向和反向KL（Kullback-Leibler）散度组合成一个目标函数，引入加权的思想调整正向和反向KL散度的比例，利用两个散度的互补特性，在学习清晰图片过程中避免不良模式的形成。实验结果表明，与现有方法相比，本文方法能更真实地恢复图像细节部分，且在评价指标峰值信噪比和图像结构相似度上有更好的表现。
- 生成对抗网络 /
- 加权 /
- 双判别器 /
- 图像去模糊
Abstract: The original generative adversarial network (GAN) is susceptible to the problems of vanishing gradients and mode collapse during the training process, and its deblurring effectiveness is poor. This study proposes an image deblurring method using a dual-discriminator weighted GAN. To extend the original GAN, a discriminator network is added to combine the forward and reverse Kullback–Leibler (KL) divergences into an objective function, and weights are used to adjust the ratio of forward and reverse KL divergences to leverage the complementary characteristics of the two divergences to avoid the formation of undesirable patterns in the process of learning clear pictures. Theoretical analysis proves that when an optimal discriminator is given, the difference between the forward and reverse KL divergences between real and generated data can be minimized. Experimental results demonstrate that compared to the existing methods, the proposed method can restore the details of an image more realistically and provides better performance in terms of the evaluation indexes of peak signal-to-noise ratio and structural similarity.
- generation adversarial network /
- weighted /
- dual discriminator /
- image deblurring

HTML全文

图 1 D2WGAN网络模型结构

Figure 1. D2WGAN network model structure

下载: 全尺寸图片幻灯片

图 2 生成器模型结构

Figure 2. Generator model structure

下载: 全尺寸图片幻灯片

图 3 判别器模型结构

Figure 3. Discriminator model structure

下载: 全尺寸图片幻灯片

图 4 合并后的图片

Figure 4. The combined picture

下载: 全尺寸图片幻灯片

图 5 本文方法去模糊前后效果对比图

Figure 5. Comparison of the effect of the method before and after deblurring

下载: 全尺寸图片幻灯片

图 6 去模糊效果对比

Figure 6. Comparison of deblurring effect

下载: 全尺寸图片幻灯片

表 1 判别器网络结构

Table 1. Discriminator network structure

Input (256×256×3)
4×4，64，stride=2，LeakyReLU
4×4，128，stride=2，instanceNorm2d，LeakyReLU
4×4，256，stride=2，instanceNorm2d，LeakyReLU
4×4，512，stride=1，instanceNorm2d，LeakyReLU
4×4，1，stride=1
SoftPlus

下载: 导出CSV

表 2 不同方法对图 6中单张图像的质量评价结果

Table 2. The quality evaluation results of the single image in Fig. 6 by different methods

Evaluation indices	LR	GAN	DeblurGAN	D2WGAN
PSNR/dB	23.03	27.96	30.54	33.08
SSIM	0.79	0.83	0.88	0.91

下载: 导出CSV

表 3 不同方法在GOPRO验证集上的图像质量评价

Table 3. Image quality evaluation of different methods on GOPRO validation set

Evaluation indices	GAN	DeblurGAN	D2WGAN
PSNR/dB	25.82	27.15	28.98
SSIM	0.772	0.815	0.891

下载: 导出CSV

参考文献(25)

[1]	李明东, 张娟, 伍世虔, 等. 基于RANSAC变换的车牌图像去模糊算法[J]. 传感器与微系统, 2020, 39(2): 153-156, 160. https://www.cnki.com.cn/Article/CJFDTOTAL-CGQJ202002043.htm LI Mingdong, ZHANG Juan, WU Shiyu, et al. A deblurring algorithm for license plate image based on RANSAC transform[J]. Sensors and Microsystems, 2020, 39(2): 153-156, 160. https://www.cnki.com.cn/Article/CJFDTOTAL-CGQJ202002043.htm
[2]	马苏欣, 王家希, 戴雅淑, 等. 监控视频下模糊车牌的去模糊与识别探析[J]. 信息系统工程, 2019(11): 111-113. doi: 10.3969/j.issn.1001-2362.2019.11.046 MA Suxin, WANG Jiaxi, DAI Yashu, et al. Research on the deblurring and recognition of fuzzy license plates under surveillance video[J]. Information System Engineering, 2019(11): 111-113. doi: 10.3969/j.issn.1001-2362.2019.11.046
[3]	裴慧坤, 颜源, 林国安, 等. 基于生成对抗网络的无人机图像去模糊方法[J]. 地理空间信息, 2019, 17(12): 4-9, 155. doi: 10.3969/j.issn.1672-4623.2019.12.002 FEI Huikun, YAN Yuan, LIN Guoan et al. Deblurring method of UAV image based on generative confrontation network[J]. Geospatial Information, 2019, 17(12): 4-9, 155. doi: 10.3969/j.issn.1672-4623.2019.12.002
[4]	黄允浒, 吐尔洪江, 唐泉, 等. 一种基于à trous算法的遥感图像模糊集增强算法[J]. 计算机应用与软件, 2018, 35(3): 187-192, 246. https://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ201803037.htm HUANG Yunhu, TU Erhong, TANG Quan, et al. A remote sensing image fuzzy set enhancement algorithm based on à trous algorithm[J]. Computer Applications and Software, 2018, 35(3): 187-192, 246. https://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ201803037.htm
[5]	张广明, 高爽, 尹增山, 等. 基于模糊图像和噪声图像的遥感图像运动模糊复原方法[J]. 电子设计工程, 2017, 25(18): 82-86. doi: 10.3969/j.issn.1674-6236.2017.18.020 ZHANG Guangming, GAO Shuang, YI Zengshan, et al. Remote sensing image motion blur restoration method based on blurred image and noise image[J]. Electronic Design Engineering, 2017, 25(18): 82-86. doi: 10.3969/j.issn.1674-6236.2017.18.020
[6]	吴庆波, 任文琦. 基于结构加权低秩近似的泊松图像去模糊[J]. 北京航空航天大学学报, 2020, 46(9): 1701-1710. https://www.cnki.com.cn/Article/CJFDTOTAL-BJHK202009010.htm WU Qingbo, REN Wenqi. Poisson image deblurring based on structure-weighted low-rank approximation[J]. Journal of Beijing University of Aeronautics and Astronautics, 2020, 46(9): 1701-1710. https://www.cnki.com.cn/Article/CJFDTOTAL-BJHK202009010.htm
[7]	RICHARDSON W. Bayesian-based iterative method of image restoration[J]. Journal of the Optical Society of America, 1972, 62(1): 55-59. doi: 10.1364/JOSA.62.000055
[8]	LUCY B. An iterative technique for the rectification of observed distributions[J]. The Astronomical Journal, 1974, 79(6): 745-754. http://pdfs.semanticscholar.org/30ad/4474c7d6e9fd68b3e0fa2db235f1c8bc32f0.pdf
[9]	IAN G, JEAN P, MEHDI M, et al. Generative adversarial nets[C]//Adv. in 27th Neural Inf. Processing Syst. (NIPS), 2014: 2672-2680.
[10]	LEDIG C. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proc. of the IEEE Conf. on Comp. Vis. and Patt. Recog. (CVPR), 2017: 105-114.
[11]	LI Y, ZHAO K, ZHAO J. Research on super-resolution image reconstruction based on low-resolution infrared sensor[J]. IEEE Access, 2020(8): 69186-69199. http://ieeexplore.ieee.org/document/9052738/
[12]	LI Z, WANG W, ZHAO Y. Image Translation by Domain-Adversarial Train[J]. Compu. Intel. And Neuro. , 2018: 1-11. Doi: 10.1155/2018/8974638.
[13]	YANG T, CHANG X, SU H, et al. Raindrop removal with light field image using image inpainting[J]. IEEE Access, 2020(8): 58416-58426. http://ieeexplore.ieee.org/document/9040628
[14]	Mirza M, Osindero S. Conditional generative adversarial nets[J/OL]. arXiv preprint arXiv: 1411.1784, 2014, https://arxiv.org/abs/1411.1784.
[15]	Orest K, Volodymyr B, Mykola M, et al. DeblurGAN: Blind motion deblurring using conditional adversarial networks[C]//Proc. of the IEEE Conf. on Comp. Vis. And Patt. Recog., 2018: 8183-8192.
[16]	NGUYENT, LE T, VU H. Dual discriminator generative adversarial nets[C]//Proc. 29th Int. Conf. Neur. Inf. Pro. Sys., 2017: 2667-2677.
[17]	Lucas T, Aäron V, Matthias B. A note on the evaluation of generative models[J/OL]. arXiv preprint arXiv: 1511.01844, 2015. https://arxiv.org/abs/1511.01844
[18]	IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[C]//ICML'15: Proceedings of the 32nd International Conference on International Conference on Machine Learning, 2015, 37: 448-456.
[19]	Ulyanov D, Vedaldi A, Lempitsky V. Instance normalization: the missing ingredient for fast stylization[C]//Proc. of the IEEE Conf. on Comp. Vis. and Patt. Recog. (CVPR), 2016: 1-13.
[20]	LI C, WAND M. Precomputed Real-time texture synthesis with markovian generative adversarial networks[C]//European Conference on Computer Vision, 2016: 702-716.
[21]	Maas L, Hannun Y, Ng Y. Rectifier nonlinearities improve neural network acoustic models[C]//Proc. ICML., 2013: 1-3.
[22]	JOHNSON J, ALAHI A, FEI L. Perceptual losses for real-time style transfer and super-resolution[C]//Proc. of European Conference on Computer Vision, 2016: 694-711.
[23]	SUN J, CAO W, XU Z, et al. Learning a convolutional neural network for non-uniform motion blur removal[C]//Proc. of the IEEE Conf. on Comp. Vis. and Patt. Recog. (CVPR), 2015: 769-777.
[24]	NAH S, KIM H, LEE M. Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//IEEE Conf. on Comp. Vis. and Patt. Recog. (CVPR), 2017: 257-265.
[25]	Kingma D, Ba J. Adam: A method for stochastic optimization[C]//Int. Conf. for Learning Representations (ICLR), 2015: 1-15.