留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于空洞全局注意力机制的近红外图像彩色化方法

高美玲 段锦 赵伟强 胡奇

高美玲, 段锦, 赵伟强, 胡奇. 基于空洞全局注意力机制的近红外图像彩色化方法[J]. 红外技术, 2023, 45(10): 1096-1105.
引用本文: 高美玲, 段锦, 赵伟强, 胡奇. 基于空洞全局注意力机制的近红外图像彩色化方法[J]. 红外技术, 2023, 45(10): 1096-1105.
GAO Meiling, DUAN Jin, ZHAO Weiqiang, HU Qi. Near-infrared Image Colorization Method Based on a Dilated Global Attention Mechanism[J]. Infrared Technology , 2023, 45(10): 1096-1105.
Citation: GAO Meiling, DUAN Jin, ZHAO Weiqiang, HU Qi. Near-infrared Image Colorization Method Based on a Dilated Global Attention Mechanism[J]. Infrared Technology , 2023, 45(10): 1096-1105.

基于空洞全局注意力机制的近红外图像彩色化方法

基金项目: 

吉林省科技发展计划项目 20210203181SF

详细信息
    作者简介:

    高美玲(1997-),女,辽宁锦州人,博士研究生,主要研究方向:图像处理等

    通讯作者:

    段锦(1971-),男,吉林长春人,博士,教授,博士生导师,主要研究方向:图像处理与模式识别等

  • 中图分类号: TP391

Near-infrared Image Colorization Method Based on a Dilated Global Attention Mechanism

  • 摘要: 针对目前卷积神经网络未能充分提取图像的浅层特征信息导致近红外图像彩色化算法存在结果图像局部区域误着色及网络训练不稳定导致结果出现模糊问题,提出了一种新的生成对抗网络方法用于彩色化任务。首先,在生成器残差块中引入自行设计的空洞全局注意力模块,对近红外图像的每个位置理解更加充分,改善局部区域误着色问题;其次,在判别网络中,将批量归一化层替换成梯度归一化层,提升网络判别性能,改善彩色化图像生成过程带来的模糊问题;最后,将本文算法在RGB_NIR数据集上进行定性和定量对比。实验表明,本文算法与其他经典算法相比能充分提取近红外图像的浅层信息特征,在指标方面,结构相似性提高了0.044,峰值信噪比提高了0.835,感知相似度降低了0.021。
  • 图  1  本文彩色化GAN模型总体框架

    Figure  1.  Overall framework for colorized GAN models in this paper

    图  2  生成器结构

    Figure  2.  Structure of generator

    图  3  空洞全局注意力机制模块

    Figure  3.  Dilated global attention module

    图  4  空洞全局注意力机制模块网络结构

    Figure  4.  The network architecture of dilated global attention mechanism module

    图  5  空洞卷积

    Figure  5.  Dialted convolution

    图  6  不同扩张率下感受野

    Figure  6.  Receptive filed at different expansion rates

    图  7  判别模型

    Figure  7.  Discriminator model

    图  8  各个算法对比结果:(a) 近红外图像;(b) Deoldify[22]结果;(c) Wei[23]结果;(d) In2i[24]结果;(e) CycleGAN算法[7]结果;(f) 本文算法结果;(g) 可见光图像

    Figure  8.  Comparison results of each algorithm: (a) NIR images; (b) Deoldify[22] results; (c) Wei [23] results; (d) In2i[24] results; (e) CycleGAN[7] Results; (f) Our method results; and (g) Visible images

    图  9  消融实验一对比算法结果:(a) 近红外图像;(b) 实验一结果;(c) 实验二结果;(d) 实验三结果;(e) 实验四结果;(f) 可见光图像

    Figure  9.  Results of ablation exp.1 comparison algorithm: (a) NIR images; (b) Exp.1 results; (c) Exp.2 results; (d) Exp.3 results; (e) Exp.4 results and (f) Visible images

    图  10  均方误差趋势

    Figure  10.  The trend of mean square error

    图  11  峰值信噪比趋势

    Figure  11.  The trend of peak signal-to-noise ratio

    表  1  指标对比

    Table  1.   Comparison of indicators

    Algorithm Ranch image Mountain image Statue image
    SSIM PSNR/dB LPIPS SSIM PSNR/dB LPIPS SSIM PSNR/dB LPIPS
    Deoldify[22]] 0.718 17.088 0.447 0.781 16.682 0.451 0.844 17.397 0.450
    Wei[23] 0.743 17.533 0.419 0.699 16.054 0.354 0.705 17.316 0.377
    In2i[24] 0.823 16.987 0.371 0.703 18.271 0.435 0.759 17.746 0.362
    CycleGAN[7] 0.715 19.787 0.341 0.867 20.738 0.326 0.831 17.681 0.222
    Our method 0.832 21.531 0.324 0.904 20.271 0.310 0.835 18.974 0.219
    下载: 导出CSV

    表  2  消融实验一指标比对

    Table  2.   Comparison of ablation experiment 1 metrics

    Street Football Building
    IS FID IS FID IS FID
    Exp.1 8.263 14.824 7.263 10.942 9.823 14.234
    Exp.2 9.072 13.495 8.752 10.234 9.102 11.293
    Exp.3 8.273 11.293 9.528 11.293 10.583 14.245
    Exp.4 9.210 10.056 9.473 9.351 11.973 10.248
    下载: 导出CSV
  • [1] 戴康. 基于超像素提取和级联匹配的灰度图像自动彩色化[J]. 计算机与数字工程, 2019, 47(12): 3169-3172. https://www.cnki.com.cn/Article/CJFDTOTAL-JSSG201912044.htm

    DAI K. Automatic colorization of grayscale images based on superpixel extraction and cascade matching[J]. Computer and Digital Engineering, 2019, 47(12): 3169-3172. https://www.cnki.com.cn/Article/CJFDTOTAL-JSSG201912044.htm
    [2] 曹丽琴, 商永星, 刘婷婷, 等. 局部自适应的灰度图像彩色化[J]. 中国图象图形学报, 2019, 24(8): 1249-1257. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB201908004.htm

    CAO L Q, SHANG Y X, LIU T T, et al. Locally adaptive grayscale image colorization[J]. Journal of Image and Graphics, 2019, 24(8): 1249-1257. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB201908004.htm
    [3] Tomihisa Welsh, Michael Ashikhmin, Klaus Mueller. Transferring color to greyscale images[C]//ACM TOG, 2002, 21(3): 277-280.
    [4] Reinhard E, Adhikhmin M, Gooch B, et al. Color transfer between images[J]. IEEE Computer Graphics and Applications, 2001, 21(5): 34-41.
    [5] 冯佳男, 江倩, 金鑫, 等. 基于深度神经网络的遥感图像彩色化方法[J]. 计算机辅助设计与图形学学报, 2021, 33(11): 1658-1667. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJF202111003.htm

    FENG J N, JIANG Q, JINX, et al. Colorization method of remote sensing image based on deep neural network[J]. Journal of Computer-Aided Design and Computer Graphics, 2021, 33(11): 1658-1667. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJF202111003.htm
    [6] CHENG Z, YANG Q, SHENG B. Deep colorization[C]//Proceedings of the IEEE International Conference on Computer Vision, 2015: 415-423.
    [7] Isola P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1125-1134.
    [8] Deblina Bhattacharjee, Seungryong Kim, Guillaume Vizier, et al. DUNIT: detection based unsupervised image-to-image translation[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 4787-4796.
    [9] 万园园, 王雨青, 张晓宁, 等. 结合全局语义优化的对抗性灰度图像彩色化[J]. 液晶与显示, 2021, 36(9): 1305-1313. https://www.cnki.com.cn/Article/CJFDTOTAL-YJYS202109011.htm

    WAN Y Y, WANG Y Q, ZHANG X N, et al. Adversarial grayscale image colorization combined with global semantic optimization[J]. Chinese Journal of Liquid Crystals and Displays, 2021, 36(9): 1305-1313. https://www.cnki.com.cn/Article/CJFDTOTAL-YJYS202109011.htm
    [10] 左岑, 杨秀杰, 张捷, 等. 基于轻量级金字塔密集残差网络的红外图像超分辨增强[J]. 红外技术, 2021, 43(3): 251-257. http://hwjs.nvir.cn/article/id/a1540103-27df-466a-a058-1fd126ff5aec

    ZUO Q, YANG X J, ZHANG J, et al. Super-resolution enhancement of infrared images based on lightweight pyramidal dense residual networks[J]. Infrared Technology, 2021, 43(3): 251-257. http://hwjs.nvir.cn/article/id/a1540103-27df-466a-a058-1fd126ff5aec
    [11] 姜玉宁, 李劲华, 赵俊莉. 基于生成式对抗网络的图像超分辨率重建算法[J]. 计算机工程, 2021, 47(3): 249-255. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJC202103032.htm

    JIANG Y N, LI J H, ZHAO J L. Image super resolution reconstruction algorithm based on generative adversarial networks[J]. Computer Engineering, 2021, 47(3): 249-255. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJC202103032.htm
    [12] 张振江, 张宝金, 刘伟新, 等. 基于深度卷积网络的矿岩图像分割算法研究[J]. 采矿技术, 2021, 21(5): 149-152, 171. https://www.cnki.com.cn/Article/CJFDTOTAL-SJCK202105041.htm

    ZHANG Z J, ZHANG B J, LIU W X, et al. Research on mining rock image segmentation algorithm based on deep convolutional network[J]. Mining Technology, 2021, 21(5): 149-152, 171. https://www.cnki.com.cn/Article/CJFDTOTAL-SJCK202105041.htm
    [13] 姚永康. 基于对抗式迁移学习的皮肤病变图像分割方法研究[D]. 西安: 西京学院, 2021.

    YAO Y K. Research on Skin Lesion Image Segmentation Method Based on Adversarial Transfer Learning[D]. Xi'an: Xijing University, 2021.
    [14] 吴杰, 段锦, 董锁芹, 等. DFM-GAN网络在跨年龄模拟的人脸识别技术研究[J]. 计算机工程与应用, 2021, 57(10): 117-124. https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG202110015.htm

    WU J, DUAN J, TONG S Q, et al. DFM-GAN networks in cross-age simulation for face recognition[J]. Computer Engineering and Applications, 2021, 57(10): 117-124. https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG202110015.htm
    [15] 刘高天, 段锦, 范祺, 等. 基于改进RFBNet算法的遥感图像目标检测[J]. 吉林大学学报: 理学版, 2021, 59(5): 1188-1198. https://www.cnki.com.cn/Article/CJFDTOTAL-JLDX202105026.htm

    LIU G T, DUAN J, FAN Q, et al. Remote sensing image target detection based on improved RFBNet algorithm[J]. Journal of Jilin University(Science Edition), 2021, 59(5): 1188-1198. https://www.cnki.com.cn/Article/CJFDTOTAL-JLDX202105026.htm
    [16] LI C, WAN D M. Precomputed real-time texture synthesis with markovian generative adversarial networks[C]//2016 European Conference on Computer Vision of IEEE, 2016: 702-716.
    [17] LI Y, CHEN Y, WANG N, et al. Scale aware trident networks for object detection[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019: 6053-6062.
    [18] Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module[C]//2018 European Conference on Computer Vision, 2018: 3-19.
    [19] Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift[C]//2015 Proceedings of the 32nd International Conference on International Conference on Machine Learning, 2015: 448-456.
    [20] Bhaskara V S, Aumentado-Armstrong T, Jepson A D, et al. GraN-GAN: piecewise gradient normalization for generative adversarial networks[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022: 3821-3830.
    [21] Brown M, Süsstrunk S. Multi-spectral sift for scene category recognition[C]//2011 IEEE Conference on Computer Vision and Pattern Recognition, 2011: 177-184.
    [22] Jason Antic. jantic/deoldify: a deep learning based project for colorizing and restoring old images (and video!)[J/OL] [2019-10-16]https://github.com/jantic/DeOldify.
    [23] LIANG W, DING D, WEI G. An improved dual GAN for near infrared image colorization[J]. Infrared Physics & Technology, 2021, 116(4): 103764-103777.
    [24] Perera P, Abavisani M, Patel V M. In2i: Unsupervised multi-image-to-image translation using generative adversarial networks[C]//2018 International Conference on Pattern Recognition, 2018: 140-146.
  • 加载中
图(11) / 表(2)
计量
  • 文章访问数:  33
  • HTML全文浏览量:  2
  • PDF下载量:  23
  • 被引次数: 0
出版历程
  • 收稿日期:  2022-09-07
  • 修回日期:  2022-09-29
  • 刊出日期:  2023-10-20

目录

    /

    返回文章
    返回