M-SWF域红外与可见光图像结构相似性融合

李威; 田时舜; 刘广丽; 邹文斌

M-SWF域红外与可见光图像结构相似性融合

深圳大学电子与信息工程学院, 广东深圳 518060

基金项目:

国家自然科学基金项目 62171294

国家自然科学基金项目 62101344

广东省自然科学基金 2022A1515010159

深圳自然科学基金 JCYJ20200109105832261

深圳自然科学基金 JCYJ20190808122409660

深圳市科技计划重点项目 20220810180617001

详细信息

作者简介:
李威（1997-），男，博士研究生，主要从事图像处理的研究，E-mail：lv2881314@163.com

通讯作者:
邹文斌（1981-），男，博士，副教授，主要从事图像处理的研究，E-mail：wzou@szu.edu.cn

中图分类号: TP391
计量
- 文章访问数: 34
- HTML全文浏览量: 9
- PDF下载量: 10
- 被引次数: 0
出版历程
- 收稿日期: 2022-11-14
- 修回日期: 2023-03-08
- 刊出日期: 2024-03-20

Structural Similarity Fusion of Infrared and Visible Image in the M-SWF Domain

College of Electronics and Information Engineering, Shenzhen University, Shenzhen 518060, China

摘要

摘要: 为了解决常规滤波器组在红外与可见光图像融合领域中存在提取结构信息不充分和融合视觉效果不佳的问题，本文提出了一种基于多尺度滑动窗口滤波器（Multi-scale Sliding Window Filter，M-SWF）图像融合方法。首先，提出一种基于SWF的多尺度图像分解方法实现对源图像的结构细节层和基础层提取；其次，采用L1范数融合规则（L1-Fusion, L1F）整合结构细节层，提取图像的结构信息；然后，利用一种图像能量贡献融合规则（Energy Attribute-Fusion, EAF）整合基础层，突出显著性目标；最后，融合图像通过叠加整合后的多尺度结构细节层和基础层得到。实验首先通过分析能量贡献系数，从主客观方面得到M-SWF域内红外与可见光图像融合较为适宜的能量贡献系数；其次，在该取值下，本文提出的M-SWF融合模型与其他的融合方法相比，不仅提高了对源图像结构信息的提取能力，而且通过整合图像的能量属性，改善了融合效果不佳问题，有效地突出了显著性目标。
- 红外图像 /
- 可见光图像 /
- 图像融合 /
- 滑动窗口滤波器 /
- 结构相似性
Abstract: This study introduces a multiscale sliding window filter (M-SWF) image fusion method to address issues with traditional filter banks in infrared and visible image fusion. First, a multiscale image decomposition method based on SWF is proposed to extract the structural detail layers and base layers of the source image. Second, the L1 norm fusion rule (L1-Fusion, L1F) is used to integrate the structural detail layers, which can extract the structure of the image. Then, to highlight the salient objects, energy attribute fusion (EAF), which is a rule for fusing image energy contributions, is used to integrate the base layers, and the fusion results are obtained by stacking the integrated multiscale structure detail layers and base layers. The energy contribution coefficient was analyzed, and a suitable energy contribution coefficient was obtained for the fusion of infrared and visible images in the M-SWF domain from subjective and objective perspectives. Compared with other fusion methods, the M-SWF not only improves the ability to extract the structural information of the source image but also improves the poor fusion effect and effectively highlights salient targets by integrating the energy attributes of the image.
- infrared image /
- visible image /
- image fusion /
- side window filter /
- structural similarity

HTML全文

图 1 M-SWF图像融合框架

Figure 1. The M-SWF based image fusion framework

下载: 全尺寸图片幻灯片

图 2 不同尺寸R下的SWF滤波结果

Figure 2. The results of images by SWF with different R

下载: 全尺寸图片幻灯片

图 3 M-SWF图像分解

Figure 3. The image decomposition by M-SWF

下载: 全尺寸图片幻灯片

图 4 不同参数P的M-SWF融合结果

Figure 4. The result of images by M-SWF with different P

下载: 全尺寸图片幻灯片

图 5 不同方法在“Camp_1811”图像下的融合结果

Figure 5. The fusion results by different methods under "Camp_1811"

下载: 全尺寸图片幻灯片

图 6 不同方法在“Trees_4917”图像下的融合结果

Figure 6. The fusion results by different methods under "Trees_4917"

下载: 全尺寸图片幻灯片

图 7 不同方法在“Kaptein_1123”图像下的融合结果

Figure 7. The fusion results by different methods under "Kaptein_1123"

下载: 全尺寸图片幻灯片

表 1 不同P值下的融合定量数据

Table 1. Fusion of quantitative data under different P

	Q_abf	MIP	SCD	SSIM	MSSIM	VIFF	RT/s
Fig.4(c)	0.5007	0.9249	1.5446	0.7640	0.9071	0.3044	0.510
Fig.4(d)	0.5303	0.9329	1.7225	0.7637	0.9287	0.3311	0.513
Fig.4(e)	0.5293	0.9364	1.7823	0.7604	0.9287	0.3323	0.508
Fig.4(f)	0.5233	0.9380	1.7936	0.7565	0.9350	0.3407	0.504
Fig.4(g)	0.5163	0.9382	1.7969	0.7527	0.9181	0.3512	0.518
Fig.4(h)	0.5096	0.9369	1.7989	0.7490	0.9135	0.3626	0.506

下载: 导出CSV

表 2 客观评价数据

Table 2. Objective evaluation data

		Q_abf	MIP	SCD	SSIM	MSSIM	VIFF	RT/s
Camp_1811	GFF	0.4241	0.8979	1.1453	0.6664	0.7186	0.2724	0.0990
	CBF	0.3920	0.8701	1.3044	0.6238	0.7438	0.2839	3.4876
	FFIF	0.4562	0.9104	1.2300	0.7257	0.8442	0.2933	0.3520
	Resnet	0.3392	0.8823	1.4860	0.7777	0.8707	0.3203	12.3780
	MLEPF	0.3640	0.8463	1.2691	0.6230	0.6878	0.2791	8.9630
	M-SWF	0.4802	0.8857	1.7003	0.7476	0.9249	0.4410	0.2520
Trees_4917	GFF	0.2526	0.8807	0.9299	0.7559	0.8560	0.2651	0.1160
	CBF	0.3296	0.8086	1.2718	0.5963	0.6850	0.5599	3.1175
	FFIF	0.2271	0.8928	0.6808	0.7627	0.7782	0.0877	0.4260
	Resnet	0.3847	0.8492	1.2224	0.8462	0.9249	0.2975	11.8230
	MLEPF	0.4963	0.7815	0.4175	0.5972	0.6365	0.3880	9.1730
	M-SWF	0.5431	0.8626	1.6195	0.8095	0.9597	0.4528	0.2370
Kaptein_1123	GFF	0.1079	0.8486	0.4842	0.4425	0.2967	0.3722	0.2830
	CBF	0.4315	0.8645	1.4597	0.5370	0.6509	0.3280	8.9544
	FFIF	0.3053	0.8954	1.2110	0.6758	0.8472	0.3671	0.4770
	Resnet	0.3303	0.8911	1.6111	0.7467	0.8582	0.3024	40.4150
	MLEPF	0.4110	0.8171	1.3662	0.5523	0.5958	0.1951	20.7530
	M-SWF	0.5778	0.8955	1.8601	0.7183	0.9423	0.4522	0.4840

下载: 导出CSV

表 3 TNO数据集上的平均客观评价数据

Table 3. Average objective evaluation data on TNO

	Q_abf	MIP	SCD	SSIM	MSSIM	VIFF	RT/s
GFF	0.2899	0.9020	1.1450	0.6736	0.7554	0.2488	0.2118
CBF	0.6054	0.8897	1.1488	0.6679	0.7864	0.2895	9.7087
FFIF	0.2881	0.9016	1.1914	0.7098	0.8050	0.2271	0.4828
Resnet	0.4093	0.8683	1.2537	0.6625	0.7516	0.3145	27.2588
MLEPF	0.4325	0.8514	1.3164	0.6006	0.6582	0.2634	20.2127
M-SWF	0.5691	0.9084	1.8071	0.7476	0.9317	0.3827	0.3944

下载: 导出CSV

参考文献(20)

[1]	MA Jiayi, MA Yong, LI Chang. Infrared and visible image fusion methods and applications: a survey [J]. Information Fusion, 2019, 45: 153-178. DOI: 10.1016/j.inffus.2018.02.004。
[2]	ZHAO Z, XU S, ZHANG C, et al. Bayesian fusion for infrared and visible images[J]. Signal Processing, 2020, 177: 165-168. DOI: 10.1016/ j.sigpro.2020.107734
[3]	HUAN Kewei, LI Xiangyang, CAO Yutong, et al. Infrared and visible image fusion of convolutional neural network and NSST[J]. Infrared and Laser Engineering, 2022, 51(3): 20210139. DOI: 10.3788/IRLA20210139.
[4]	CHENG Boyang, LI Ting, WANG Yulin. Fusion of infrared and visible light images based on visual saliency weighting and maximum gradient singular value[J]. Chinese Optics, 2022, 15(4): 675-688. DOI: 10.37188/CO.2022-0124
[5]	李威, 李忠民. 一种基于EASSF的红外与可见光图像视觉保真度融合[J]. 红外技术, 2022, 44(7): 686-692. http://hwjs.nvir.cn/article/id/2c60451f-34b1-47a3-bab6-629774a73a0c LI Wei, LI Zhongmin. Visual fidelity fusion of infrared and visible image using edge-aware smoothing-sharpening filter[J]. Infrared Technology, 2022, 44(7): 686-692. http://hwjs.nvir.cn/article/id/2c60451f-34b1-47a3-bab6-629774a73a0c
[6]	李永萍, 杨艳春, 党建武, 等. 基于变换域VGGNet19的红外与可见光图像融合[J]. 红外技术, 2022, 44(12): 1293-1300. http://hwjs.nvir.cn/article/id/48843dec-a48b-468f-b743-4a00c345f406 LI Yongping, YANG Yanchun, DANG Jianwu, et al. Infrared and visible image fusion based on transform domain VGGNet19[J]. Infrared Technology, 2022, 44(12): 1293-1300. http://hwjs.nvir.cn/article/id/48843dec-a48b-468f-b743-4a00c345f406
[7]	雷大江, 杜加浩, 张莉萍, 等. 联合多流融合和多尺度学习的卷积神经网络遥感图像融合方法[J]. 电子与信息学报, 2022, 44(1): 237-244. Doi: 10.11999/JEIT200792. LEI Dajiang, DU Jiahao, ZHANG Liping, et al. Multi-stream architecture and multi-scale convolutional neural network for remote sensing image fusion [J]. Journal of Electronics & Information Technology, 2022, 44(1): 237-244. Doi: 10.11999/JEIT200792.
[8]	马梁, 苟于涛, 雷涛, 等. 基于多尺度特征融合的遥感图像小目标检测[J]. 光电工程, 2022, 49(4): 49-65. https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC202204005.htm MA Liang, GOU Yutao, LEI Tao, et al. Small object detection based on multi-scale feature fusion using remote sensing images[J]. Opto-Electron Eng, 2022, 49(4): 49-65. https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC202204005.htm
[9]	钱金卓, 马骏, 李峰, 等. 面向CMOS遥感相机的多曝光图像融合方法[J]. 遥感信息, 2022, 37(4): 51-57. https://www.cnki.com.cn/Article/CJFDTOTAL-YGXX202204008.htm QIAN Jinzhuo, MA Jun, LI Feng, et al. Multi-exposure image fusion method for CMOS remote sensing camera [J]. Remote Sensing Information, 2022, 37(4): 51-57. https://www.cnki.com.cn/Article/CJFDTOTAL-YGXX202204008.htm
[10]	LI Shutao, KANG Xudong, HU Jianwen. Image fusion with guided filtering[J]. IEEE Transactions on Image Processing, 2013, 22(7): 2864-2875. Doi: 10.1109/TIP.2013.2244222.
[11]	Shreyamsha Kumar B K. Image fusion based on pixel significance using cross bilateral filter[J]. Signal Image Video Process, 2015, 9(5): 1193-1204. Doi: 10.1007/s11760-013-0556-9.
[12]	ZHAN K, XIE Yuange, MIN Yufang. Fast filtering image fusion[J]. J. Electron. Imaging, 2017, 26(6): 063004. Doi: 10.1117/1.JEI.26.6.063004.
[13]	LI Hui, WU Xiaojun, Tariq S Durrani. Infrared and visible image fusion with ResNet and zero-phase component analysis[J]. Infrared Physics & Technology, 2019, 102: 1030390. Doi: 10.1016/j.infrared.2019.-103039.
[14]	TAN Wei, Thitn W, XIANG P, et al. Multi-modal brain image fusion based on multi-level edge-preserving filtering[J]. Biomedical Signal Processing and Control, 2021, 64: 102280. Doi: 10.1016/j. bspc.2020.102280.
[15]	YIN Hui, GONG Yuanhao, QIU Guoping. Side window guided filtering[J]. Signal Process, 2019, 165: 315-330. Doi: 10.1016/j.sigpro. 2019.07.026.
[16]	LI Xiaosong, ZHOU Fuqiang, TAN Haishu, et al. Multimodal medical image fusion based on joint bilateral filter and local gradient energy[J]. Information Sciences, 2021, 569: 302-325.
[17]	LI Hui, WU Xiaojun. DenseFuse: a fusion approach to infrared and visible images [J]. IEEE Trans. Image Process, 2019, 28(5): 2614-2623. Doi: 10.1109/TIP.2018.2887342.
[18]	Toet A. TNO Image Fusion Dataset[EB/OL]. [2022-10-30]. http://figshare.com/articles/-TNO_Image_Fusion_Dataset/1008029.
[19]	Xydeas C S, Petrovic V. Objective image fusion performance measure [J]. Electron. Lett, 2000, 36(4): 308-309. Doi: 10.1109/ICCV.2005.175.
[20]	Aslantas V, Bendes E. A new image quality metric for image fusion: the sum of the correlations of differences[J]. AEU- Int. J. Electron. Commune, 2015, 69(12): 1890-1896. Doi: 10.1016/j.aeue.2015.09.004.