WU Yuanyuan, WANG Zhishe, WANG Junyao, SHAO Wenyu, CHEN Yanlin. Infrared and Visible Image Fusion Using Attention- Based Generative Adversarial Networks[J]. Infrared Technology , 2022, 44(2): 170-178.
Citation: WU Yuanyuan, WANG Zhishe, WANG Junyao, SHAO Wenyu, CHEN Yanlin. Infrared and Visible Image Fusion Using Attention- Based Generative Adversarial Networks[J]. Infrared Technology , 2022, 44(2): 170-178.

Infrared and Visible Image Fusion Using Attention- Based Generative Adversarial Networks

More Information
  • Received Date: May 28, 2021
  • Revised Date: July 19, 2021
  • At present, deep learning-based fusion methods rely only on convolutional kernels to extract local features, but the limitations of single-scale networks, convolutional kernel size, and network depth cannot provide a sufficient number of multi-scale and global image characteristics. Therefore, here we propose an infrared and visible image fusion method using attention-based generative adversarial networks. This study uses a generator consisting of an encoder and decoder, and two discriminators. The multi-scale module and channel self-attention mechanism are designed in the encoder, which can effectively extract multi-scale features and establish the dependency between the long ranges of feature channels, thus enhancing the global characteristics of multi-scale features. In addition, two discriminators are constructed to establish an adversarial relationship between the fused image and the source images to preserve more detailed information. The experimental results demonstrate that the proposed method is superior to other typical methods in both subjective and objective evaluations.
  • [1]
    MA J, MA Y, LI C. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. DOI: 10.1016/j.inffus.2018.02.004
    [2]
    LI S, KANG X, FANG L, et al. Pixel-level image fusion: a survey of the state of the art[J]. Information Fusion, 2017, 33: 100-112. DOI: 10.1016/j.inffus.2016.05.004
    [3]
    LIU Y, CHEN X, WANG Z, et al. Deep learning for pixel-level image fusion: Recent advances and future prospects[J]. Information Fusion, 2018, 42: 158-173. DOI: 10.1016/j.inffus.2017.10.007
    [4]
    LI S, YANG B, HU J. Performance comparison of different multi-resolution transforms for image fusion[J]. Information Fusion, 2011, 12(2): 74-84. DOI: 10.1016/j.inffus.2010.03.002
    [5]
    ZHANG Q, LIU Y, Rick S Blum, et al. Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: a review[J]. Information Fusion, 2018, 40: 57-75. DOI: 10.1016/j.inffus.2017.05.006
    [6]
    ZHANG Xiaoye, MA Yong, ZHANG Ying, et al. Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition[J]. Journal of the Optical Society of America A Optics Image Science & Vision, 2017, 34(8): 1400-1410.
    [7]
    YU L, LIU S, WANG Z. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015, 24: 147-164. DOI: 10.1016/j.inffus.2014.09.004
    [8]
    HAN J, Pauwels E J, P De Zeeuw. Fast saliency-aware multimodality image fusion[J]. Neurocomputing, 2013, 111: 70-80. DOI: 10.1016/j.neucom.2012.12.015
    [9]
    YIN Haitao. Sparse representation with learned multiscale dictionary for image fusion[J]. Neurocomputing, 2015, 148: 600-610. DOI: 10.1016/j.neucom.2014.07.003
    [10]
    WANG Zhishe, YANG Fengbao, PENG Zhihao, et al. Multi-sensor image enhanced fusion algorithm based on NSST and top-hat transformation[J]. Optik-International Journal for Light and Electron Optics, 2015, 126(23): 4184-4190. DOI: 10.1016/j.ijleo.2015.08.118
    [11]
    CUI G, FENG H, XU Z, et al. Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition[J]. Optics Communications, 2015, 341: 199-209. DOI: 10.1016/j.optcom.2014.12.032
    [12]
    LI Q, LU L, LI Z, et al. Coupled GAN with relativistic discriminators for infrared and visible images fusion[J]. IEEE Sensors Journal, 2021, 21(6): 7458-7467. DOI: 10.1109/JSEN.2019.2921803
    [13]
    LIU Y, CHEN X, CHENG J, et al. Infrared and visible image fusion with convolutional neural networks[J]. International Journal of Wavelets, Multiresolution and Information Processing, 2018, 16(3): 1850018. DOI: 10.1142/S0219691318500182
    [14]
    LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2019, 28(5): 2614-2523. DOI: 10.1109/TIP.2018.2887342
    [15]
    XU H, MA J, JIANG J, et al. U2Fusion: A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 502-518.
    [16]
    HOU R. VIF-Net: an unsupervised framework for infrared and visible image fusion[J]. IEEE Transactions on Computational Imaging, 2020, 6: 640-651. DOI: 10.1109/TCI.2020.2965304
    [17]
    HUI L A, XJW A, JK B. RFN-Nest: An end-to-end residual fusion network for infrared and visible images[J]. Information Fusion, 2021, 73: 72-86. DOI: 10.1016/j.inffus.2021.02.023
    [18]
    MA J, WEI Y, LIANG P, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26. DOI: 10.1016/j.inffus.2018.09.004
    [19]
    JM A, Pl A, WEI Y A, et al. Infrared and visible image fusion via detail preserving adversarial learning[J]. Information Fusion, 2020, 54: 85-98. DOI: 10.1016/j.inffus.2019.07.005
    [20]
    MA J, XU H, JIANG J, et al. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995. DOI: 10.1109/TIP.2020.2977573
    [21]
    GAO S, CHENG M M, ZHAO K, et al. Res2Net: A new multi-scale backbone architecture[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(2): 652-662. DOI: 10.1109/TPAMI.2019.2938758
    [22]
    FU J, LIU J, TIAN H, et al. Dual attention network for scene segmentation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 2020: DOI: 10.1109/cvpr.2019.00326.
    [23]
    Nencini F, Garzelli A, Baronti S, et al. Alparone, remote sensing image fusion using the curvelet transform[J]. Information Fusion, 2007, 8(2): 143-156. DOI: 10.1016/j.inffus.2006.02.001
    [24]
    LIU Y, WANG Z. Simultaneous image fusion and denoising with adaptive sparse representation[J]. Image Processing Iet. , 2014, 9(5): 347-357.
    [25]
    MA J, ZHOU Z, WANG B, et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics & Technology, 2017, 82: 8-17.
    [26]
    YU Z A, YU L B, PENG S C, et al. IFCNN: A general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118. DOI: 10.1016/j.inffus.2019.07.011
  • Related Articles

    [1]JIN Dan, LIU Xiaoguang, SHI Gang, SONG Renping, ZU Mingxia. Temperature Compensation for Infrared Detection of Carbon Dioxide Concentration[J]. Infrared Technology , 2023, 45(6): 671-677.
    [2]HE Lei, HOU Bin, WANG Renhao, SI Hongli, WU Xingguang. Non-Sequential Simulation Analysis of Narcissus Effect for Cooled Infrared Optical System[J]. Infrared Technology , 2023, 45(6): 592-597.
    [3]LI Huan, HU Liang, MENG Xiangfu, LI Qi, WANG Shuang. Simulation Analysis of Thermal-Structure of an Optical Detection System[J]. Infrared Technology , 2020, 42(12): 1141-1150.
    [4]PAN Xuejuan, ZHU Youpan, PAN Chao, HAN Juan, LI Daqing, ZHANG Hao, LUO Lin, LI Zemin. Simulation Analysis of Auto Focusing Sharpness Evaluation Function for Images Based on Gradient Operator[J]. Infrared Technology , 2016, 38(11): 960-968.
    [5]PAN Xuejuan, ZHU Youpan, PU Enchang, LUO Lin, ZENG Bangze, ZHAO Deli, LI Zemin, SUN Aiping. Image Entropy Autofocus Sharpness Evaluation Function Simulation and Analysis under Matlab[J]. Infrared Technology , 2016, 38(10): 838-844.
    [6]ZHANG Dongge, FU Yutian. Stray Radiation Analysis For Mid-wave Fish Eye Infrared Lens[J]. Infrared Technology , 2016, 38(2): 117-120.
    [7]MAO Yi-fan, ZHANG Duo-lin, WANG Lu. Simulation Analysis of Ballistic Missile Detection by STSS[J]. Infrared Technology , 2015, (3): 218-223.
    [8]LI Long, SUN Hao, ZHU Xian. Simulation Analysis of Dark Current in Long Wavelength Infrared HgCdTe Photodiodes[J]. Infrared Technology , 2014, (1): 73-78.
    [9]JIN Ning, CAO Ling, CUI Qin. Analysis for Scanning Footprint of Infrared 6-face Rotating Drum System[J]. Infrared Technology , 2007, 29(5): 249-252. DOI: 10.3969/j.issn.1001-8891.2007.05.001
    [10]The Simulation of Absorptance and Response Characteristics for Pyroelectric Thin Films Infrared Sensors[J]. Infrared Technology , 2002, 24(2): 25-29. DOI: 10.3969/j.issn.1001-8891.2002.02.007
  • Cited by

    Periodical cited type(6)

    1. 刘传洋,吴一全. 基于红外图像的电力设备识别及发热故障诊断方法研究进展. 中国电机工程学报. 2025(06): 2171-2196 .
    2. 邱刚,高超,陈杰,谭笑,杨景刚,李鸿泽. 基于热成像技术的支柱瓷绝缘子实时故障诊断研究. 绝缘材料. 2023(06): 100-105 .
    3. 张玲玲,张继冉,许廒,任攀攀,丁立斌. 基于红外图像处理的建筑外窗缺陷能耗分析研究. 红外技术. 2023(09): 996-1004 . 本站查看
    4. 尹林,胡京,王文彬,贺林轩,周恺,苏朝辉,闫磊,刘若溪,王博,屠幼萍. 环境条件对基于无人机红外检测的零值绝缘子检测判据的影响研究. 电瓷避雷器. 2023(05): 171-177 .
    5. 刘新伯,李昊,陈强,梁鉴如,郭梦茹. 复杂背景下多个绝缘子串红外图像分割方法. 计算技术与自动化. 2022(02): 125-130 .
    6. 刘志宏,刘艺林. 电气设备预防性维护技术综述. 科技资讯. 2021(31): 30-32 .

    Other cited types(4)

Catalog

    Article views (187) PDF downloads (69) Cited by(10)
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return