Multi-Feature Adaptive Fusion Method for Infrared and Visible Images
-
摘要: 由于成像机理不同，红外图像以像素分布表征典型目标，而可见光图像以边缘和梯度描述纹理细节，现有的融合方法不能依据源图像特征自适应变化，造成融合结果不能同时保留红外目标特征与可见光纹理细节。为此，本文提出红外与可见光图像多特征自适应融合方法。首先，构建了多尺度密集连接网络，可以有效聚合所有不同尺度不同层级的中间特征，利于增强特征提取和特征重构能力。其次，设计了多特征自适应损失函数，采用VGG-16网络提取源图像的多尺度特征，以像素强度和梯度为测量准则，以特征保留度计算特征权重系数。多特征自适应损失函数监督网络训练，可以均衡提取源图像各自的特征信息，从而获得更优的融合效果。公开数据集的实验结果表明，该方法在主、客观评价方面均优于其他典型方法。Abstract: Owing to different imaging mechanisms, infrared images represent typical targets by pixel distribution, whereas visible images describe texture details through edges and gradients. Existing fusion methods fail to adaptively change according to the characteristics of the source images, thereby resulting in fusion results that do not retain infrared target features and visible texture details simultaneously. Therefore, a multi-feature adaptive fusion method for infrared and visible images is proposed in this study. First, a multi-scale dense connection network that can effectively reuse all intermediate features of different scales and levels, and further enhance the ability of feature extraction and reconstruction is constructed. Second, a multi-feature adaptive loss function is designed. Using the pixel intensity and gradient as measurement criteria, the multi-scale features of the source image are extracted by the VGG-16 network and the feature weight coefficients are calculated by the degree of information preservation. The multi-feature adaptive loss function can supervise network training and evenly extract the respective feature information of the source image to obtain a better fusion effect. The experimental results of public datasets demonstrate that the proposed method is superior to other typical methods in terms of subjective and objective evaluations.
-
Key words:
- image fusion /
- dense skip connection /
- adaptive loss function /
- visible image /
- infrared image
-
图 3 多特征自适应模块的定性比较结果
Figure 3. Qualitative comparison results of multi-feature adaptive modules
图 5 Roadscene数据集的定性比较结果
Figure 5. Qualitative comparison results of Roadscene datasets
表 1 密集连接的定量比较结果
Table 1. Quantitative comparison results of dense skip connection
Metrics No_dense Only_En Only_De Ours EN 6.90676 7.00322 6.98832 7.00507 NCIE 0.80460 0.80443 0.80519 0.80464 SD 35.68939 38.39500 43.99837 41.44601 SCD 1.79833 1.84037 1.78040 1.84590 VIFF 0.56981 0.60081 0.58553 0.62035 MS_SSIM 0.92346 0.92305 0.91745 0.93027 注：将最优值以黑体加粗标注，次优值以下划线标注。客观评价指标越高，表明融合性能越好。
Note: The best average value is marked in bold, and the second value is marked with an underline. The objective evaluation index is positively correlated with the fusion performance.
下载: 导出CSV
表 2 多特征自适应损失函数的定量比较结果
Table 2. Quantitative comparison results of multi-feature adaptive loss function
Metrics Fmean Finten Fgrad Ours EN 6.82654 6.95075 6.92680 7.00507 NCIE 0.80432 0.80460 0.80455 0.80464 SD 33.71341 38.96415 36.28725 41.44601 SCD 1.77467 1.83212 1.82509 1.84590 VIFF 0.55015 0.59584 0.59834 0.62035 MS_SSIM 0.91588 0.92522 0.92957 0.93027
下载: 导出CSV
表 3 TNO数据集的定量比较结果
Table 3. Quantitative comparison results of TNO datasets
Metrics MDLatLRR DenseFuse IFCNN UNFusion GANMcC U2Fusion RFN-Nest Ours EN 6.29523 6.25275 6.33795 6.98828 6.57763 6.84306 6.89803 7.00507 NCIE 0.80435 0.80451 0.80404 0.80912 0.80452 0.80392 0.80428 0.80464 SD 23.70282 22.85769 24.06712 40.93903 29.92973 33.59608 34.85373 41.44601 SCD 1.59002 1.57018 1.59052 1.70351 1.67191 1.76712 1.78875 1.84590 VIFF 0.30902 0.28295 0.34396 0.45033 0.40468 0.62469 0.53121 0.62035 MS_SSIM 0.90133 0.87696 0.90474 0.87404 0.85915 0.87283 0.91217 0.93027
下载: 导出CSV
表 4 Roadscene数据集的定量比较结果
Table 4. Quantitative comparison results of Roadscene datasets
Metrics MDLatLRR DenseFuse IFCNN UNFusion GANMcC U2Fusion RFN-Nest Ours EN 6.86973 6.82473 6.91635 7.37792 7.23683 7.41910 7.33677 7.47617 NCIE 0.80701 0.80718 0.80629 0.80998 0.80670 0.80771 0.80649 0.80658 SD 33.05830 32.22811 33.64622 50.26212 43.81169 49.15315 46.03340 51.43262 SCD 1.38030 1.36329 1.38949 1.63715 1.60983 1.76111 1.72851 1.84427 VIFF 0.39051 0.35676 0.41529 0.48462 0.45465 0.58028 0.47894 0.61286 MS_SSIM 0.88881 0.85857 0.89583 0.85420 0.85744 0.93322 0.87277 0.92484
下载: 导出CSV
表 5 OTCBVS数据集的定量比较结果
Table 5. Quantitative comparison results of OTCBVS datasets
Metrics MDLatLRR DenseFuse IFCNN UNFusion GANMcC U2Fusion RFN-Nest Ours EN 7.12984 7.07264 7.23707 7.28585 6.60437 7.35554 7.19973 7.57158 NCIE 0.80557 0.80569 0.80505 0.81011 0.80560 0.80501 0.80528 0.80578 SD 36.43416 35.31958 38.72566 44.91661 27.50886 43.12793 39.04909 50.45001 SCD 1.44604 1.41625 1.45256 1.42248 1.09337 1.62165 1.48793 1.75565 VIFF 0.27563 0.27216 0.29275 0.22372 0.16750 0.32515 0.26596 0.35130 MS_SSIM 0.83939 0.81697 0.86125 0.79352 0.67759 0.87722 0.80530 0.88162
下载: 导出CSV
-
[1] Paramanandham N, Rajendiran K. Multi sensor image fusion for surveillance applications using hybrid image fusion algorithm[J]. Multimedia Tools and Applications, 2018, 77(10): 12405-12436. doi: 10.1007/s11042-017-4895-3 [2] ZHANG Xingchen, YE Ping, QIAO Dan, et al. Object fusion tracking based on visible and infrared images: a comprehensive review[J]. Information Fusion, 2020, 63: 166-187. doi: 10.1016/j.inffus.2020.05.002 [3] TU Zhengzheng, LI Zhun, LI Chenglong, et al. Multi-interactive dual- decoder for RGB-thermal salient object detection[J]. IEEE Transactions on Image Processing, 2021, 30: 5678-5691. doi: 10.1109/TIP.2021.3087412 [4] FENG Zhanxiang, LAI Jianhuang, XIE Xiaohua. Learning modality- specific representations for visible-infrared person re-identification[J]. IEEE Transactions on Image Processing, 2020, 29: 579-590. doi: 10.1109/TIP.2019.2928126 [5] MO Yang, KANG Xudong, DUAN Puhong, et al. Attribute filter based infrared and visible image fusion[J]. Informantion Fusion, 2021, 75: 41-54. doi: 10.1016/j.inffus.2021.04.005 [6] LI Hui, WU Xiaojun, Kittle J. MDLatLRR: a novel decomposition method for infrared and visible image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4733-4746. doi: 10.1109/TIP.2020.2975984 [7] 李辰阳, 丁坤, 翁帅, 等. 基于改进谱残差显著性图的红外与可见光图像融合[J]. 红外技术, 2020, 42(11): 1042-1047. http://hwjs.nvir.cn/article/id/6e57a6fb-ba92-49d9-a000-c00e7a933365LI Chenyang, DING Kun, WENG Shuai, et al. Image fusion of infrared and visible images based on residual significance[J]. Infrared Technology, 2020, 42(11): 1042-1047. http://hwjs.nvir.cn/article/id/6e57a6fb-ba92-49d9-a000-c00e7a933365 [8] WANG Zhishe, YANG Fengbao, PENG Zhihao, et al. Multi-sensor image enhanced fusion algorithm based on NSST and top-hat transformation[J]. Optik-International Journal for Light and Electron Optics, 2015, 126(23): 4184-4190. doi: 10.1016/j.ijleo.2015.08.118 [9] LIU Yu, CHEN Xun, PENG Hu, et al. Multi-focus image fusion with a deep convolutional neural network[J]. Informantion Fusion, 2017, 36: 191-207. doi: 10.1016/j.inffus.2016.12.001 [10] WANG Zhishe; WU Yuanyuan; WANG Junyao, et al. Res2Fusion: infrared and visible image fusion based on dense Res2net and double non-local attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-12. [11] MA Jiayi, MA Yong, LI Chang. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. doi: 10.1016/j.inffus.2018.02.004 [12] Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation[C]//Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015: 234-241. [13] Toet A. Computational versus psychophysical bottom-up image saliency: a comparative evaluation study[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2011, 33(11): 2131-2146. [14] LI Hui, WU Xiaojun. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2019, 28(5): 2614-2623. doi: 10.1109/TIP.2018.2887342 [15] ZHANG Yu, LIU Yu, SUN Peng, et al. IFCNN: a general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118. doi: 10.1016/j.inffus.2019.07.011 [16] WANG Zhishe, WANG Junyao, WU Yuanyuan, et al. UNFusion: a unified multi-scale densely connected network for infrared and visible image fusion[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(6): 3360- 3374. doi: 10.1109/TCSVT.2021.3109895 [17] MA Jiayi, YU Wei, LIANG Pengwei, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26. doi: 10.1016/j.inffus.2018.09.004 [18] MA Jiayi, ZHANG Hao, SHAO Zhenfeng, et al. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14. [19] LI Hui, WU Xiaojun, Josef Kittler. RFN-Nest: an end-to-end residual fusion network for infrared and visible images[J]. Information Fusion, 2021, 73: 72-86. doi: 10.1016/j.inffus.2021.02.023 [20] TOET A. TNO Image Fusion Datase[DB/OL]. [2014-04-26]. https://figshare.com/articles/TN Image Fusion Dataset/1008029. [21] XU Han. Roadscene Database[DB/OL]. [2020-08-07]. https://github.com/hanna-xu/RoadScene. [22] Ariffin S. OTCBVS Database[DB/OL]. [2007-06]. http://vcipl-okstate.org/pbvs/bench/. [23] XU Han, MA Jiayi, JIANG Junjun, et al. U2Fusion: a unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518. doi: 10.1109/TPAMI.2020.3012548 [24] Aslantas V, Bendes E. Assessment of image fusion procedures using entropy, image quality, and multispectral classification[J]. Journal of Applied Remote Sensing, 2008(2): 1-28. [25] LIU Zheng, Blasch E, XUE Zhiyun, et al. Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 34: 94-109. [26] RAO Yunjiang. In-fibre bragg grating sensors[J]. Measurement Science and Technology, 1997(8): 355-375. [27] Aslantas V, Bendes E. A new image quality metric for image fusion: The sum of the correlations of differences[J]. AEU-Int. J. Electron. C. , 2015, 69: 1890-1896. doi: 10.1016/j.aeue.2015.09.004 [28] HAN Yu, CAI Yunze, CAO Yin, et al. A new image fusion performance metric based on visual information fidelity[J]. Information Fusion, 2013(14): 127-135. [29] MA Kede, ZENG Kai, WANG Zhou. Perceptual quality assessment for multi-exposure image fusion[J]. IEEE Trans. Image Process, 2015, 24: 3345-3356. doi: 10.1109/TIP.2015.2442920 -
点击查看大图
图(6) / 表(5)
计量
- 文章访问数: 220
- HTML全文浏览量: 74
- PDF下载量: 61
- 被引次数: 0
