Infrared and Visible Image Fusion Based on Deep Image Decomposition
-
摘要:
红外与可见光图像融合是一种图像增强技术，其目标是为了获得保留有源图像优势的融合图像。对此本文提出了一种基于深度图像分解的红外与可见光图像融合方法。首先源图像经过编码器分解为背景特征图和细节特征图；同时编码器中引入显著性特征提取模块，突出源图像的边缘和纹理特征; 随后通过解码器获得融合图像。在训练过程中对可见光图像采用梯度系数惩罚进行正则化重建去保证纹理一致性；对图像分解，图像重建分别设计损失函数，以缩小背景特征图之间的差异，同时放大细节特征图之间的差异。实验结果表明，该方法可生成具有丰富细节和高亮目标的融合图像，在TNO和FLIR公开数据集上的主客观评价上优于其他对比方法。
Abstract:Infrared and visible light image fusion is an enhancement technique designed to create a fused image that retains the advantages of the source image. In this study, a depth image decomposition-based infrared and visible image fusion method is proposed. First, the source image is decomposed into the background feature map and detail feature map by the encoder; simultaneously, the saliency feature extraction module is introduced in the encoder to highlight the edge and texture features of the source image; subsequently, the fused image is obtained by the decoder. In the training process, a gradient coefficient penalty was applied to the visible image for regularized reconstruction to ensure texture consistency, and a loss function was designed for image decomposition and reconstruction to reduce the differences between the background feature maps and amplify the differences between the detail feature maps. The experimental results show that the method can generate fused images with rich details and bright targets. In addition, this method outperforms other comparative methods in terms of subjective and objective evaluations of the TNO and FLIR public datasets.
-
Keywords:
- image fusion /
- deep learning /
- saliency feature /
- multi-scale decomposition
-
-
![]()
图 4 TNO数据集“Kaptein_1654_Ⅱ”融合结果比对
Figure 4. Comparison of the fusion results of the TNO dataset 'Kaptein_1654_Ⅱ'
![]()
图 5 TNO数据集“Kaptein_1123_Ⅱ”融合结果比对
Figure 5. Comparison of the fusion results of the TNO dataset 'Kaptein_1123_Ⅱ'
![]()
图 6 FLIR数据集“People”融合结果比对
Figure 6. Comparison of fusion results for the FLIR dataset 'People'
![]()
图 7 FLIR数据集“Car”融合结果对比
Figure 7. Comparison of fusion results for the FLIR dataset 'Car'
![]()
图 8 求和与加权求和融合策略对比
Figure 8. Comparison of summation and weighted summation fusion strategies
表 1 本文网络配置
Table 1 Network configuration in this paper
Layers I O S Padding Activation CONV1 1 64 3 Reflection PReLU CONV2 64 64 3 0 PReLU CONV3 64 64 3 0 Tanh CONV4 64 64 3 0 Tanh SFE 64 64 1 0 Sigmoid CONV5 128 64 3 0 PReLU CONV6 64 64 3 0 PReLU CONV7 64 1 3 Reflection Sigmoid 
下载: 导出CSV
表 2 TNO数据集融合结果客观评价指标对比
Table 2 Comparison of objective evaluation indicators for TNO dataset fusion results
EN SF AG VIF PSNR MI SD Ours 7.410 13.446 5.220 0.631 62.519 2.208 44.064 DDcGAN 7.485 13.283 5.374 0.513 60.939 1.838 50.416 DIDFUSE 6.816 12.311 4.531 0.597 61.658 2.207 41.974 FusionGAN 6.468 6.453 2.488 0.410 61.319 2.213 28.634 GANMcC 6.670 6.447 2.650 0.517 62.265 2.250 32.664 RFN-NEST 6.962 6.320 2.876 0.550 63.089 2.195 37.670 SDDGAN 7.188 9.597 3.882 0.554 61.868 2.223 48.578 IFCNN 6.970 13.319 5.158 0.628 63.641 2.158 40.248 DeepFuse 5.704 13.422 4.813 0.074 61.733 0.780 15.780
下载: 导出CSV
表 3 FLIR数据集融合结果客观评价指标对比
Table 3 Comparison of objective evaluation indicators for FLIR dataset fusion results
EN SF AG VIF PSNR MI SD Ours 7.367 17.134 6.625 0.642 62.579 2.692 49.959 DDcGAN 7.593 13.148 5.182 0.393 60.112 2.452 56.133 DIDFUSE 7.376 12.563 6.032 0.558 61.950 2.683 48.315 FusionGAN 7.029 8.831 3.455 0.339 59.606 2.677 37.268 GANMcC 7.204 8.766 3.772 0.450 60.144 2.552 42.176 RFN-NEST 7.248 8.487 3.671 0.464 60.342 2.586 43.801 SDDGAN 7.499 10.852 4.648 0.455 59.976 2.873 56.163 IFCNN 7.111 16.315 6.465 0.524 62.360 2.660 38.091 DeepFuse 5.870 14.370 5.130 0.079 58.760 1.101 16.943
下载: 导出CSV
表 4 不同融合策略结果客观评价指标对比（TNO）
Table 4 Comparison of objective evaluation indicators of the results of different integration strategies (TNO)
EN SF AG VIF PSNR MI SD Ours 7.41 13.446 5.22 0.631 62.519 2.208 44.064 L1 Norm 6.614 8.142 3.062 0.602 61.651 2.821 33.609
下载: 导出CSV
表 5 不同融合策略结果客观评价指标对比（FLIR）
Table 5 Comparison of objective evaluation indicators of the results of different integration strategies (FLIR)
EN SF AG VIF PSNR MI SD Ours 7.367 17.134 6.625 0.642 62.579 2.692 49.959 L1 Norm 7.113 11.608 4.691 0.547 62.299 2.948 39.677
下载: 导出CSV
-
[1] TANG L F, YUAN J, MA J Y. Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network[J]. Information Fusion, 2022, 82: 28-42. DOI: 10.1016/j.inffus.2021.12.004
[2] 罗迪, 王从庆, 周勇军. 一种基于生成对抗网络与注意力机制的可见光和红外图像融合方法[J]. 红外技术, 2021, 43(6): 566-574. http://hwjs.nvir.cn/article/id/3403109e-d8d7-45ed-904f-eb4bc246275a LUO Di, WANG Congqing, ZHOU Yongjun. A visible and infrared image fusion method based on generative adversarial networks and attention mechanism[J]. Infrared Technology, 2021, 43(6): 566-574. http://hwjs.nvir.cn/article/id/3403109e-d8d7-45ed-904f-eb4bc246275a
[3] ZHANG X. Deep learning-based multi-focus image fusion: a survey and a comparative study[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(9): 4819-4838. http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9428544
[4] MA Jiayi, ZHOU Yi. Infrared and visible image fusion via gradientlet filter[J]. Computer Vision and Image Understanding, 2020(197-198): 103016.
[5] LI G, LIN Y, QU X. An infrared and visible image fusion method based on multi-scale transformation and norm optimization[J]. Information Fusion, 2021, 71: 109-129. DOI: 10.1016/j.inffus.2021.02.008
[6] MA J Y, YU W, LIANG P W, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26. DOI: 10.1016/j.inffus.2018.09.004
[7] 刘佳, 李登峰. 马氏距离与引导滤波加权的红外与可见光图像融合[J]. 红外技术, 2021, 43(2): 162-169. http://hwjs.nvir.cn/article/id/56484763-c7b0-4273-a087-8d672e8aba9a LIU Jia, LI Dengfeng. Infrared and visible light image fusion based on Mahalanobis distance and guided filter weighting[J]. Infrared Technology, 2021, 43(2): 162-169. http://hwjs.nvir.cn/article/id/56484763-c7b0-4273-a087-8d672e8aba9a
[8] TANG L F, YUAN J, MA J Y, et al. PIAFusion: a progressive infrared and visible image fusion network based on illumination aware[J]. Information Fusion, 2022, 83: 79-92. http://www.sciencedirect.com/science/article/pii/S156625352200032X?dgcid=rss_sd_all
[9] TANG L F, XIANG X Y, ZHANG H, et al. DIVFusion: darkness-free infrared and visible image fusion[J]. Information Fusion, 2023, 91: 477-493. DOI: 10.1016/j.inffus.2022.10.034
[10] YU F, JUN X W, Tariq D. Image fusion based on generative adversarial network consistent with perception[J]. Information Fusion, 2021, 72: 110-125. DOI: 10.1016/j.inffus.2021.02.019
[11] 何乐, 李忠伟, 罗偲, 等. 基于空洞卷积与双注意力机制的红外与可见光图像融合[J]. 红外技术, 2023, 45(7): 732-738. http://hwjs.nvir.cn/article/id/66ccbb6b-ad75-418f-ae52-cc685713d91b HE Le, LI Zhongwei, LUO Cai, et al. Infrared and visible image fusion based on cavity convolution and dual attention mechanism[J]. Infrared Technology, 2023, 45(7): 732-738. http://hwjs.nvir.cn/article/id/66ccbb6b-ad75-418f-ae52-cc685713d91b
[12] ZHANG Y, LIU Y, SUN P, et al. IFCNN: a general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118. DOI: 10.1016/j.inffus.2019.07.011
[13] MA J Y, XU H, JIANG J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion [J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995. http://www.xueshufan.com/publication/3011768656
[14] MA J, ZHANG H, SHAO Z, et al. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14. http://doc.paperpass.com/foreign/rgArti2020150972012.html
[15] Prabhakar K R, Srikar V S, Babu R V. DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs[C]// IEEE International Conference on Computer Vision (ICCV), 2017: 4724-4732.
计量
- 文章访问数: 117
- HTML全文浏览量: 21
- PDF下载量: 53
