基于语义损失的红外与可见光图像融合算法

丁华彬, 丁麒文

丁华彬, 丁麒文. 基于语义损失的红外与可见光图像融合算法[J]. 红外技术, 2023, 45(9): 941-947.
引用本文: 丁华彬, 丁麒文. 基于语义损失的红外与可见光图像融合算法[J]. 红外技术, 2023, 45(9): 941-947.
DING Huabin, DING Qiwen. Fusion Algorithm of Infrared and Visible Images Based on Semantic Loss[J]. Infrared Technology , 2023, 45(9): 941-947.
Citation: DING Huabin, DING Qiwen. Fusion Algorithm of Infrared and Visible Images Based on Semantic Loss[J]. Infrared Technology , 2023, 45(9): 941-947.

基于语义损失的红外与可见光图像融合算法

详细信息
    作者简介:

    丁华彬(1998-),男,山东青岛人,硕士研究生,研究方向为图像处理,深度学习。E-mail: dhb41416@163.com

  • 中图分类号: TP391.41

Fusion Algorithm of Infrared and Visible Images Based on Semantic Loss

  • 摘要: 提出了一种基于语义损失的红外与可见光图像融合算法,通过语义损失引导生成图像包含更多语义信息,满足高级视觉任务需求。首先使用预训练的分割网络对融合图像进行分割,分割结果与标签图构成语义损失,在语义损失和内容损失的共同引导下,迫使融合网络在保证融合图像质量的前提下同时兼顾图像语义信息量,融合图像满足高级视觉任务需求。同时本文还设计了一种新的特征提取模块,通过残差密集连接实现特征重用,提高细节描述能力,进一步减轻融合框架,从而提高图像融合的时间效率。实验结果表明,本文算法在主观视觉效果和定量指标方面优于现有融合算法,且融合图像包含更丰富的语义信息。
    Abstract: In this study, we propose an infrared and visible image fusion algorithm based on semantic loss, to ensure that the generated images contain more semantic information through semantic loss, thereby satisfying the requirements of advanced vision tasks. First, a pre-trained segmentation network is used to segment the fused image, with the segmentation result and label map determining the semantic loss. Under the joint guidance of semantic and content losses, we force the fusion network to guarantee the quality of the fused image by considering the amount of semantic information in the image, to ensure that the fused image meets the requirements of advanced vision tasks. In addition, a new feature extraction module is designed in this study to achieve feature reuse through a residual dense connection to improve detail description capability while further reducing the fusion framework, which improves the time efficiency of image fusion. The experimental results show that the proposed algorithm outperforms existing fusion algorithms in terms of subjective visual effects and quantitative metrics and that the fused images contain richer semantic information.
  • 图  1   本文融合算法整体框架

    Figure  1.   The overall framework of the proposed image fusion algorithm

    图  2   融合网络架构

    Figure  2.   Fusion network architecture

    图  3   特征提取器具体设计

    Figure  3.   Specific design of the feature extractor

    图  4   “00057D”图像不同算法的融合结果

    Figure  4.   Fusion results of different algorithms for 00057D image

    图  5   “00510D”图像不同算法的融合结果

    Figure  5.   Fusion results of different algorithms for 00510D image

    图  6   “01347N”图像不同算法的融合结果

    Figure  6.   Fusion results of different algorithms for 01347N image

    图  7   不同算法融合图分割可视化图

    Figure  7.   Different algorithms merge the graph segmentation visualization diagram

    表  1   图像融合客观指标结果

    Table  1   Objective index results of fusion image

    MI SD SF EN QAB/F
    DDcGAN 1.9848 6.2586 0.0231 5.2684 0.1859
    Nestfuse 2.9764 7.5843 0.0301 6.2386 0.4862
    GANMcM 2.5130 8.2627 0.0236 6.1895 0.3238
    Ours 3.8672 7.9858 0.0458 6.5841 0.5842
    下载: 导出CSV

    表  2   融合图像分割性能指标(mIoU)

    Table  2   mIoU of fusion image

    DDcGAN Nestfuse GANMcM Ours
    mIoU 75.33 76.32 75.68 77.98
    下载: 导出CSV
  • [1]

    MA J, MA Y, LI C. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. DOI: 10.1016/j.inffus.2018.02.004

    [2]

    ZHU Y, LI C, LUO B, et al. Dense feature aggregation and pruning for RGBT tracking[C]// The 27th ACM International Conference. ACM, 2019: 465-472.

    [3]

    Bhatnagar G, WU Q, ZHENG L. Directive contrast based multimodal medical image fusion in nsct domain[J]. IEEE Transactions on Multimedia, 2014, 9(5): 1014-1024.

    [4]

    Poudel R, Liwicki S, Cipolla R. Fast-SCNN: fast semantic segmentation network[J/OL]. arXiv Preprint arXiv: 1902.04502, 2019. https://doi.org/ 10.48550/arXiv.1902.04502

    [5]

    Cordts M, Omran M, Ramos S, et al. The cityscapes dataset for semantic urban scene understanding[C]//Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016: 3213-3223.

    [6]

    ZHOU W, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Trans Image Process, 2004, 13(4): 600-612. DOI: 10.1109/TIP.2003.819861

    [7]

    HA Q, Watanabe K, Karasawa T, et al. MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes[C]// IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017: 4714-4722.

    [8]

    Paszke A, Gross S, Massa F, et al. PyTorch: an imperative style, high-performance deep learning library[J/OL]. Advances in Neural Information Processing Systems, 2019, https://doi.org/10.48550/arXiv.1912.01703.

    [9]

    Prabhakar K R, Srikar V S, Babu R V. DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs[C]// IEEE International Conference on Computer Vision (ICCV). IEEE Computer Society, 2017: 4714-4722.

    [10]

    MA J, XU H, JIANG J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995. DOI: 10.1109/TIP.2020.2977573

    [11]

    LI H, WU X J, Durrani T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020(99): 1-1.

    [12]

    MA J, ZHANG H, SHAO Z, et al. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14, Doi: 10.1109/TIM.2020.3038013.

  • 期刊类型引用(1)

    1. 张德银,黄少晗,赵志恒,李俊佟,张裕尧. 基于融合神经网络的飞机蒙皮缺陷检测的研究. 成都大学学报(自然科学版). 2023(04): 365-371 . 百度学术

    其他类型引用(1)

图(7)  /  表(2)
计量
  • 文章访问数:  160
  • HTML全文浏览量:  47
  • PDF下载量:  49
  • 被引次数: 2
出版历程
  • 收稿日期:  2022-08-17
  • 修回日期:  2022-11-23
  • 刊出日期:  2023-09-19

目录

    /

    返回文章
    返回