Fusion Algorithm of Infrared and Visible Images Based on Semantic Loss
-
摘要: 提出了一种基于语义损失的红外与可见光图像融合算法,通过语义损失引导生成图像包含更多语义信息,满足高级视觉任务需求。首先使用预训练的分割网络对融合图像进行分割,分割结果与标签图构成语义损失,在语义损失和内容损失的共同引导下,迫使融合网络在保证融合图像质量的前提下同时兼顾图像语义信息量,融合图像满足高级视觉任务需求。同时本文还设计了一种新的特征提取模块,通过残差密集连接实现特征重用,提高细节描述能力,进一步减轻融合框架,从而提高图像融合的时间效率。实验结果表明,本文算法在主观视觉效果和定量指标方面优于现有融合算法,且融合图像包含更丰富的语义信息。Abstract: In this study, we propose an infrared and visible image fusion algorithm based on semantic loss, to ensure that the generated images contain more semantic information through semantic loss, thereby satisfying the requirements of advanced vision tasks. First, a pre-trained segmentation network is used to segment the fused image, with the segmentation result and label map determining the semantic loss. Under the joint guidance of semantic and content losses, we force the fusion network to guarantee the quality of the fused image by considering the amount of semantic information in the image, to ensure that the fused image meets the requirements of advanced vision tasks. In addition, a new feature extraction module is designed in this study to achieve feature reuse through a residual dense connection to improve detail description capability while further reducing the fusion framework, which improves the time efficiency of image fusion. The experimental results show that the proposed algorithm outperforms existing fusion algorithms in terms of subjective visual effects and quantitative metrics and that the fused images contain richer semantic information.
-
Keywords:
- image fusion /
- semantic loss /
- neural network /
- infrared image /
- visible image
-
-
表 1 图像融合客观指标结果
Table 1 Objective index results of fusion image
MI SD SF EN QAB/F DDcGAN 1.9848 6.2586 0.0231 5.2684 0.1859 Nestfuse 2.9764 7.5843 0.0301 6.2386 0.4862 GANMcM 2.5130 8.2627 0.0236 6.1895 0.3238 Ours 3.8672 7.9858 0.0458 6.5841 0.5842 表 2 融合图像分割性能指标(mIoU)
Table 2 mIoU of fusion image
DDcGAN Nestfuse GANMcM Ours mIoU 75.33 76.32 75.68 77.98 -
[1] MA J, MA Y, LI C. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. DOI: 10.1016/j.inffus.2018.02.004
[2] ZHU Y, LI C, LUO B, et al. Dense feature aggregation and pruning for RGBT tracking[C]// The 27th ACM International Conference. ACM, 2019: 465-472.
[3] Bhatnagar G, WU Q, ZHENG L. Directive contrast based multimodal medical image fusion in nsct domain[J]. IEEE Transactions on Multimedia, 2014, 9(5): 1014-1024.
[4] Poudel R, Liwicki S, Cipolla R. Fast-SCNN: fast semantic segmentation network[J/OL]. arXiv Preprint arXiv: 1902.04502, 2019. https://doi.org/ 10.48550/arXiv.1902.04502
[5] Cordts M, Omran M, Ramos S, et al. The cityscapes dataset for semantic urban scene understanding[C]//Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016: 3213-3223.
[6] ZHOU W, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Trans Image Process, 2004, 13(4): 600-612. DOI: 10.1109/TIP.2003.819861
[7] HA Q, Watanabe K, Karasawa T, et al. MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes[C]// IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017: 4714-4722.
[8] Paszke A, Gross S, Massa F, et al. PyTorch: an imperative style, high-performance deep learning library[J/OL]. Advances in Neural Information Processing Systems, 2019, https://doi.org/10.48550/arXiv.1912.01703.
[9] Prabhakar K R, Srikar V S, Babu R V. DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs[C]// IEEE International Conference on Computer Vision (ICCV). IEEE Computer Society, 2017: 4714-4722.
[10] MA J, XU H, JIANG J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995. DOI: 10.1109/TIP.2020.2977573
[11] LI H, WU X J, Durrani T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020(99): 1-1.
[12] MA J, ZHANG H, SHAO Z, et al. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14, Doi: 10.1109/TIM.2020.3038013.
-
期刊类型引用(1)
1. 张德银,黄少晗,赵志恒,李俊佟,张裕尧. 基于融合神经网络的飞机蒙皮缺陷检测的研究. 成都大学学报(自然科学版). 2023(04): 365-371 . 百度学术
其他类型引用(1)