DING Huabin, DING Qiwen. Fusion Algorithm of Infrared and Visible Images Based on Semantic Loss[J]. Infrared Technology , 2023, 45(9): 941-947.
Citation: DING Huabin, DING Qiwen. Fusion Algorithm of Infrared and Visible Images Based on Semantic Loss[J]. Infrared Technology , 2023, 45(9): 941-947.

Fusion Algorithm of Infrared and Visible Images Based on Semantic Loss

More Information
  • Received Date: August 17, 2022
  • Revised Date: November 23, 2022
  • In this study, we propose an infrared and visible image fusion algorithm based on semantic loss, to ensure that the generated images contain more semantic information through semantic loss, thereby satisfying the requirements of advanced vision tasks. First, a pre-trained segmentation network is used to segment the fused image, with the segmentation result and label map determining the semantic loss. Under the joint guidance of semantic and content losses, we force the fusion network to guarantee the quality of the fused image by considering the amount of semantic information in the image, to ensure that the fused image meets the requirements of advanced vision tasks. In addition, a new feature extraction module is designed in this study to achieve feature reuse through a residual dense connection to improve detail description capability while further reducing the fusion framework, which improves the time efficiency of image fusion. The experimental results show that the proposed algorithm outperforms existing fusion algorithms in terms of subjective visual effects and quantitative metrics and that the fused images contain richer semantic information.
  • [1]
    MA J, MA Y, LI C. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. DOI: 10.1016/j.inffus.2018.02.004
    [2]
    ZHU Y, LI C, LUO B, et al. Dense feature aggregation and pruning for RGBT tracking[C]// The 27th ACM International Conference. ACM, 2019: 465-472.
    [3]
    Bhatnagar G, WU Q, ZHENG L. Directive contrast based multimodal medical image fusion in nsct domain[J]. IEEE Transactions on Multimedia, 2014, 9(5): 1014-1024.
    [4]
    Poudel R, Liwicki S, Cipolla R. Fast-SCNN: fast semantic segmentation network[J/OL]. arXiv Preprint arXiv: 1902.04502, 2019. https://doi.org/ 10.48550/arXiv.1902.04502
    [5]
    Cordts M, Omran M, Ramos S, et al. The cityscapes dataset for semantic urban scene understanding[C]//Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016: 3213-3223.
    [6]
    ZHOU W, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Trans Image Process, 2004, 13(4): 600-612. DOI: 10.1109/TIP.2003.819861
    [7]
    HA Q, Watanabe K, Karasawa T, et al. MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes[C]// IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017: 4714-4722.
    [8]
    Paszke A, Gross S, Massa F, et al. PyTorch: an imperative style, high-performance deep learning library[J/OL]. Advances in Neural Information Processing Systems, 2019, https://doi.org/10.48550/arXiv.1912.01703.
    [9]
    Prabhakar K R, Srikar V S, Babu R V. DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs[C]// IEEE International Conference on Computer Vision (ICCV). IEEE Computer Society, 2017: 4714-4722.
    [10]
    MA J, XU H, JIANG J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995. DOI: 10.1109/TIP.2020.2977573
    [11]
    LI H, WU X J, Durrani T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020(99): 1-1.
    [12]
    MA J, ZHANG H, SHAO Z, et al. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14, Doi: 10.1109/TIM.2020.3038013.
  • Cited by

    Periodical cited type(1)

    1. 张德银,黄少晗,赵志恒,李俊佟,张裕尧. 基于融合神经网络的飞机蒙皮缺陷检测的研究. 成都大学学报(自然科学版). 2023(04): 365-371 .

    Other cited types(1)

Catalog

    Article views (155) PDF downloads (49) Cited by(2)
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return