QI Yanjie, HOU Qinhe. Infrared and Visible Image Fusion Combining Multi-scale and Convolutional Attention[J]. Infrared Technology , 2024, 46(9): 1060-1069.
Citation: QI Yanjie, HOU Qinhe. Infrared and Visible Image Fusion Combining Multi-scale and Convolutional Attention[J]. Infrared Technology , 2024, 46(9): 1060-1069.

Infrared and Visible Image Fusion Combining Multi-scale and Convolutional Attention

More Information
  • Received Date: June 20, 2023
  • Revised Date: August 21, 2023
  • A multiscale and convolutional attention-based infrared and visible image fusion algorithm is proposed to address the issues of insufficient single-scale feature extraction and loss of details, such as infrared targets and visible textures, when fusing infrared and visible images. First, an encoder network, combining a multiscale feature extraction module and deformable convolutional attention module, is designed to extract important feature information of infrared and visible images from multiple receptive fields. Subsequently, a fusion strategy based on spatial and channel dual-attention mechanisms is adopted to further fuse the typical features of infrared and visible images. Finally, a decoder network composed of three convolutional layers is used to reconstruct the fused image. Additionally, hybrid loss function constraint network training based on mean squared error, multiscale structure similarity, and color is designed to further improve the similarity between the fused and source images. The results of the experiment are compared with seven image-fusion algorithms using a public dataset. In terms of subjective and objective evaluations, the proposed algorithm exhibits better edge preservation, source image information retention, and higher fusion image quality than other algorithms.

  • [1]
    代立杨, 刘刚, 肖刚. 基于FRC框架的红外与可见光图像融合方法[J]. 控制与决策, 2021, 36(11): 2690-2698.

    DAI L Y, LIU G, XIAO G. Infrared and visible image fusion based on FRC algorithm[J]. Control and Decision, 2021, 36(11): 2690-2698.
    [2]
    MA J, MA Y, LI C. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. DOI: 10.1016/j.inffus.2018.02.004
    [3]
    LI X S, WAN W J, ZHOU F Q, et al. Medical image fusion based on sparse representation and neighbor energy activity[J]. Biomedical Signal Processing and Control, 2023, 80(2): 104353.
    [4]
    LIU G, LIN Z, YAN S, et al. Robust recovery of subspace structures by low-rank representation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 35(1): 171-184.
    [5]
    孙彬, 诸葛吴为, 高云翔, 等. 基于潜在低秩表示的红外和可见光图像融合[J]. 红外技术, 2022, 44(8): 853-862. http://hwjs.nvir.cn/article/id/7fc3a60d-61bb-454f-ad00-e925eeb54576

    SUN Bin, ZHUGE Wuwei, GAO Yunxiang, et al. Infrared and visible image fusion based on latent low-rank representation[J]. Infrared Technology, 2022, 44(8): 853-862. http://hwjs.nvir.cn/article/id/7fc3a60d-61bb-454f-ad00-e925eeb54576
    [6]
    LI H, WU X J. Multi-focus image fusion using dictionary learning and low-rank representation[C]//Proceedings of the 9th International Conference on Image and Graphics, 2017: 675-686.
    [7]
    LIU C H, QI Y, DING W R. Infrared and visible image fusion method based on saliency detection in sparse domain[J]. Infrared Physics and Technology, 2017, 83: 94-102. DOI: 10.1016/j.infrared.2017.04.018
    [8]
    GAO R, Vorobyov S A, ZHAO H. Image fusion with cosparse analysis operator[J]. IEEE Signal Processing Letters, 2017, 24(7): 943-947. DOI: 10.1109/LSP.2017.2696055
    [9]
    LI Y H, LIU G, Bavirisetti D P, et al. Infrared-visible image fusion method based on sparse and prior joint saliency detection and LatLRR-FPDE[J]. Digital Signal Processing, 2023, 134: 103910. DOI: 10.1016/j.dsp.2023.103910
    [10]
    蒋杰伟, 刘尚辉, 金库, 等. 基于FCM与引导滤波的红外与可见光图像融合[J]. 红外技术, 2023, 45(3): 249-256. http://hwjs.nvir.cn/article/id/67d60996-565d-4597-96a1-937255cc33cc

    JIANG Jiewei, LIU Shanghui, JIN Ku, et al. Infrared and visible-light image fusion based on FCM and guided filtering[J]. Infrared Technology, 2023, 45(3): 249-256. http://hwjs.nvir.cn/article/id/67d60996-565d-4597-96a1-937255cc33cc
    [11]
    李文, 叶坤涛, 舒蕾蕾, 等. 基于高斯模糊逻辑和ADCSCM的红外与可见光图像融合算法[J]. 红外技术, 2022, 44(7): 693-701. http://hwjs.nvir.cn/article/id/227ae3cd-57b4-4ec7-a248-bdc1de60993c

    LI W, YE K T, SHU L L, et al. Infrared and visible image fusion algorithm based on Gaussian fuzzy logic and adaptive dual-channel spiking cortical model[J]. Infrared Technology, 2022, 44(7): 693-701. http://hwjs.nvir.cn/article/id/227ae3cd-57b4-4ec7-a248-bdc1de60993c
    [12]
    LI S, KANG X, HU J. Image fusion with guided filtering[J]. IEEE Transactions on Image Processing, 2013, 22(7): 2864-2875. DOI: 10.1109/TIP.2013.2244222
    [13]
    霍星, 邹韵, 陈影, 等. 双尺度分解和显著性分析相结合的红外与可见光图像融合[J]. 中国图象图形学报, 2021, 26(12): 2813-2825.

    HUO X, ZOU Y, CHEN Y, et al. Dual-scale decomposition and saliency analysis based infrared and visible image fusion[J]. Journal of Image and Graphics, 2021, 26(12): 2813-2825.
    [14]
    刘明葳, 王任华, 李静, 等. 各向异性导向滤波的红外与可见光图像融合[J]. 中国图象图形学报, 2021, 26(10): 2421-2432. DOI: 10.11834/jig.200339

    LIU M W, WANG R H, LI J, et al. Infrared and visible image fusion with multi-scale anisotropic guided filtering[J]. Journal of Image and Graphics, 2021, 26(10): 2421-2432. DOI: 10.11834/jig.200339
    [15]
    LIU Y, CHEN X, WANG Z, et al. Deep learning for pixel-level image fusion: recent advances and future prospects[J]. Inf. Fusion, 2018, 42: 158-173. DOI: 10.1016/j.inffus.2017.10.007
    [16]
    MA J, WEI Y, LIANG P, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Inf. Fusion, 2019, 48: 11-26. DOI: 10.1016/j.inffus.2018.09.004
    [17]
    MA J, ZHANG H, SHAO Z, et al. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14.
    [18]
    Prabhakar K R, Srikar V S, Babu R V. DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure imagepairs[C]//IEEE International Conference on Computer Vision (ICCV), 2017: 4724-4732.
    [19]
    LI H, WU X J. DenseFuse: A fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2019, 28(5): 2614-2623. DOI: 10.1109/TIP.2018.2887342
    [20]
    ZHANG Y, LIU Y, SUN P, et al. IFCNN: a general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118. DOI: 10.1016/j.inffus.2019.07.011
    [21]
    陈永, 张娇娇, 王镇. 多尺度密集连接注意力的红外与可见光图像融合[J]. 光学精密工程, 2022, 30(18): 2253-2266. DOI: 10.37188/OPE.20223018.2253

    CHEN Yong, ZHANG Jiaojiao, WANG Zhen. Infrared and visible image fusion based on multi-scale dense attention connection network[J]. Optics and Precision Engineering, 2022, 30(18): 2253-2266. DOI: 10.37188/OPE.20223018.2253
    [22]
    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, et al. Rethinking the inception architecture for computer vision[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016: 2818-2826.
    [23]
    WOO S, PARK J, LEE J, et al. CBAM: Convolutional block attention module[C]//European Conference on Computer Vision, 2018, 06521: 3-19.
    [24]
    李霖, 王红梅, 李辰凯. 红外与可见光图像深度学习融合方法综述[J]. 红外与激光工程, 2022, 51(12): 20220125.

    LI L, WANG H M, LI C K. A review of deep learning fusion methods for infrared and visible images[J]. Infrared and Laser Engineering, 2022, 51(12): 20220125.
    [25]
    LIN T Y, Maire M, Belongie S, et al. Microsoft COCO: common objects in context[C]//Proceedings of the 13th European Conference on Computer Vision, 2014: 740-755.
    [26]
    XU H, MA J, JIANG J, et al. U2Fusion: a unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518. DOI: 10.1109/TPAMI.2020.3012548
    [27]
    ZHANG H, MA J. SDNet: a versatile squeeze-and-decomposition network for real-time image fusion[J]. International Journal of Computer Vision, 2021, 129: 2761-785. DOI: 10.1007/s11263-021-01501-8
    [28]
    LI H, WU X J, Durrani T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656. DOI: 10.1109/TIM.2020.3005230
    [29]
    TOET A. TNO image fusion dataset [EB/OL]. [2021-02-20]. https://figshare.com/articles/TN Image Fusion Dataset/1008029.
    [30]
    XU Han. Roadscene database[DB/OL]. [2020-08-07]. https://github.com/hanna-xu/RoadScene.
    [31]
    Kristan M, Matas J, Leonardis A, et al. The eighth visual object tracking VOT2020 challenge results[C]//Proceedings of the 16th European Conference on Computer Vision, 2020, 12539: 547-601.
  • Related Articles

    [1]CHEN Zhuang, HE Feng, HONG Xiaohang, ZHANG Qiran, YANG Yuyan. Embedded Platform IR Small-target Detection Based on Self-attention and Convolution Fused Architecture[J]. Infrared Technology , 2025, 47(1): 89-96.
    [2]DI Jing, LIANG Chan, REN Li, GUO Wenqing, LIAN Jing. Infrared and Visible Image Fusion Based on Multi-Scale Contrast Enhancement and Cross-Dimensional Interactive Attention Mechanism[J]. Infrared Technology , 2024, 46(7): 754-764.
    [3]ZHAO Songpu, YANG Liping, ZHAO Xin, PENG Zhiyuan, LIANG Dongxing, LIANG Hongjun. Object Detection in Visible Light and Infrared Images Based on Adaptive Attention Mechanism[J]. Infrared Technology , 2024, 46(4): 443-451.
    [4]HE Le, LI Zhongwei, LUO Cai, REN Peng, SUI Hao. Infrared and Visible Image Fusion Based on Dilated Convolution and Dual Attention Mechanism[J]. Infrared Technology , 2023, 45(7): 732-738.
    [5]CHEN Xin. Infrared and Visible Image Fusion Using Double Attention Generative Adversarial Networks[J]. Infrared Technology , 2023, 45(6): 639-648.
    [6]CHEN Yanlin, WANG Zhishe, SHAO Wenyu, YANG Fan, SUN Jing. Multi-scale Transformer Fusion Method for Infrared and Visible Images[J]. Infrared Technology , 2023, 45(3): 266-275.
    [7]WANG Tianyuan, LUO Xiaoqing, ZHANG Zhancheng. Infrared and Visible Image Fusion Based on Self-attention Learning[J]. Infrared Technology , 2023, 45(2): 171-177.
    [8]HUANG Linglin, LI Qiang, LU Jinzheng, HE Xianzhen, PENG Bo. Infrared and Visible Image Fusion Based on Multi-scale and Attention Model[J]. Infrared Technology , 2023, 45(2): 143-149.
    [9]CHEN Da, HE Quancai, DI Erzhen, DENG Zaozhu. Application of Partial Differential Segmentation Model with Adaptive Weight in Infrared Image of Substation Equipment[J]. Infrared Technology , 2022, 44(2): 179-188.
    [10]WU Yuanyuan, WANG Zhishe, WANG Junyao, SHAO Wenyu, CHEN Yanlin. Infrared and Visible Image Fusion Using Attention- Based Generative Adversarial Networks[J]. Infrared Technology , 2022, 44(2): 170-178.
  • Cited by

    Periodical cited type(1)

    1. 杨晓超,郝慧良. 矿用电缆放电监测系统研究设计. 中国煤炭. 2024(S1): 406-410 .

    Other cited types(0)

Catalog

    Article views (86) PDF downloads (36) Cited by(1)
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return