基于卷积神经网络结合NSCT的红外与可见光图像融合

曹宇彤; 宦克为; 薛超; 韩丰地; 李向阳; 陈笑

基于卷积神经网络结合NSCT的红外与可见光图像融合

长春理工大学物理学院, 吉林长春 130022

基金项目:

国家自然科学基金 61905026

吉林省科技发展计划项目 20210101158JC

详细信息

作者简介:
曹宇彤（1996-），男，硕士研究生，研究方向：红外成像技术

通讯作者:
宦克为（1982-），男，博士，副教授，研究方向：红外成像技术，近红外光谱分析技术。E-mail：huankewei@126.com

中图分类号: TP391
计量
- 文章访问数: 156
- HTML全文浏览量: 61
- PDF下载量: 60
- 被引次数: 0
出版历程
- 收稿日期: 2021-08-07
- 修回日期: 2021-09-13
- 刊出日期: 2023-04-20

Infrared and Visible Image Fusion Based on CNN with NSCT

College of Physics, Changchun University of Science and Technology, Changchun 130022, China

摘要

摘要: 传统红外与可见光融合图像在复杂环境下存在细节缺失，特征单一导致目标模糊等问题，本文提出一种基于卷积神经网络结合非下采样轮廓波变换（non-subsampled contourlet transform，NSCT）的红外与可见光图像进行融合的方法。首先，通过卷积神经网络提取红外与可见光目标特征信息，同时利用NSCT对源图像进行多尺度分解，得到源图像的高频系数与低频系数；其次，结合目标特征图利用自适应模糊逻辑与局部方差对比度分别对源图像高频子带与低频子带进行融合；最后，通过逆NSCT变换得到融合图像并与其他5种传统算法进行对比；实验结果表明，本文方法在多个客观评价指标上均有所提高。
- 图像融合 /
- 卷积神经网络 /
- 非下采样轮廓波变换 /
- 模糊逻辑
Abstract: Traditional infrared and visible fused images suffer from missing details and blurred targets owing to single features in complex environments. This study presents a method for fusing infrared and visible images based on a convolution neural network(CNN) combined with a non-subsampled contourlet transform (NSCT). Firstly, the infrared and visible target feature information is extracted by CNN, and the source image is decomposed by the NSCT at multiple scales to obtain its high-frequency coefficients and low-frequency coefficients. Secondly, the high-frequency sub-bands and low-frequency sub-bands of the source image are fused separately using adaptive fuzzy logic and local variance contrast in combination with the target feature image. Finally, the fused image is obtained by inverse NSCT transformation. We conducted a comparative analysis with five other traditional algorithms. The experimental results show that the proposed method performs better in several objective evaluation indicators.
- image fusion /
- convolutional neural network /
- NSCT /
- fuzzy logic

HTML全文

图 1 卷积神经网络结构

Figure 1. Convolutional neural network structure

下载: 全尺寸图片幻灯片

图 2 损失函数

Figure 2. Loss function

下载: 全尺寸图片幻灯片

图 3 红外图像“Nato_camp”及多种方法提取显著性图像：(a) 红外图像“Nato_camp”；(b) 图像“Nato_camp”标准分割图；(c) FT方法；(d) AC方法；(e) LC方法；(f) CNN方法

Figure 3. Infrared image ''Nato_camp'' and images after saliency extraction by various methods: (a) Infrared image "Nato_camp"; (b) Standard segmentation of image "Nato_camp"; (c) FT method; (d) AC method; (e) LC method; (f) CNN method

下载: 全尺寸图片幻灯片

图 4 红外图像“Kaptein”及多种方法提取显著性图像：(a) 红外图像“Kaptein”；(b) 图像“Kaptein”标准分割图；(c) FT方法；(d) AC方法；(e) LC方法；(f) CNN方法

Figure 4. Infrared image ''Kaptein'' and images after saliency extraction by various methods: (a) Infrared image "Kaptein"; (b) Standard segmentation of image "Kaptein"; (c) FT method; (d) AC method; (e) LC method; (f) CNN method

下载: 全尺寸图片幻灯片

图 5 神经网络结合NSCT图像融合模型

Figure 5. Image fusion model based on convolutional neural network and NSCT

下载: 全尺寸图片幻灯片

图 6 “Nato_camp”红外和可见光图像以及融合结果：(a) 红外图像；(b) 可见光图像；(c) LP方法；(d) DWT方法；(e) BEMD方法；(f) NSST方法；(g) NSCT方法；(h) 本文方法

Figure 6. ''Nato_camp'' infrared and visible images and fusion results: (a) Infrared image; (b) Visible image; (c) LP method; (d) DWT method; (e) BEMD method; (f) NSST method; (g) NSCT method; (h) Proposed method

下载: 全尺寸图片幻灯片

图 7 “Kaptein”红外和可见光图像以及融合结果：(a) 红外图像；(b) 可见光图像；(c) LP方法；(d) DWT方法；(e) BEMD方法；(f) NSST方法；(g) NSCT方法；(h) 本文方法

Figure 7. ''Kaptein'' infrared and visible images and fusion results: (a) Infrared image; (b) Visible image; (c) LP method (d) DWT method; (e) BEMD method; (f) NSST method; (g) NSCT method; (h) Proposed method

下载: 全尺寸图片幻灯片

图 8 “iron”红外和可见光图像以及融合结果：(a) 红外图像；(b) 可见光图像；(c) LP方法；(d) DWT方法；(e) BEMD方法；(f) NSST方法；(g) NSCT方法；(h)本文方法

Figure 8. ''iron'' infrared and visible images and fusion results: (a) Infrared image; (b) Visible image; (c) LP method (d) DWT method; (e) BEMD method; (f) NSST method; (g) NSCT method; (h) Proposed method

下载: 全尺寸图片幻灯片

表 1 目标显著性提取评价指标MAE

Table 1. Target significance extraction evaluation index MAE

Method	AC	FT	LC	CNN
MSE₁	1429.6117	1816.0338	1377.7657	9.8357
MSE₂	672.0769	1379.5997	1965.2603	17.2382

下载: 导出CSV

表 2 红外与可见光图像融合效果评价

Table 2. Infrared and visible image fusion effect evaluation

Image	Image fusion method		IE	AG	SF	MI	CEN
“Nato_camp”	LP		6.6747	5.5365	15.9373	1.7365	1.4858
	DWT		6.9908	6.6575	17.4658	1.6712	0.6218
	BEMD		6.6029	6.2838	17.1804	1.3671	1.7880
	NSST		6.8419	5.8138	16.7266	1.3348	1.5568
	NSCT		6.6224	6.3133	17.3085	2.0859	0.3409
	Ours		7.1934	4.7142	17.5647	2.0883	0.3206
“Kaptein”	LP		6.7174	4.0659	12.8846	3.2774	1.3173
	DWT		7.1419	6.2009	18.9346	3.8891	1.0452
	BEMD		6.7497	5.9790	19.4699	3.0136	1.2275
	NSST		6.8063	4.4414	15.8489	3.9370	1.1409
	NSCT		6.9281	6.3012	19.2932	3.6070	1.6858
	Ours		7.2729	4.5076	20.1209	3.9387	1.0457
“iron”	LP		6.4638	7.1542	20.2285	3.1355	0.5064
	DWT		6.6377	11.7327	33.4574	3.3785	0.6087
	BEMD		6.5765	9.4525	23.3567	3.4273	0.5408
	NSST		6.7624	14.7945	39.4278	3.4058	0.5075
	NSCT		6.7352	15.6254	40.4253	3.1668	0.4772
	Ours	6.7648		14.7857	40.8547	3.8173	0.4117

下载: 导出CSV

参考文献(15)

[1]	MA J Y, MA Yong. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. doi: 10.1016/j.inffus.2018.02.004
[2]	董安勇, 杜庆治. 基于卷积神经网络的红外与可见光图像融合[J]. 红外技术, 2020, 42(7): 660-669. http://hwjs.nvir.cn/article/id/hwjs202007009 DONG Anyong, DU Qingzhi. Infrared and visible image fusion based on vonvolutional neural network[J]. Infrared Technology, 2020, 42(7): 660-669. http://hwjs.nvir.cn/article/id/hwjs202007009
[3]	杨孙运, 奚峥皓, 汪汉东, 等. 基于NSCT和最小化-局部平均梯度的图像融合[J]. 红外技术, 2021, 43(1): 13-20. http://hwjs.nvir.cn/article/id/144252d1-978c-4c1e-85ad-e0b8d5e03bf6 YANG Sunyun, XI Zhenghao, WANG Handong, et al. Image fusion based on NSCT and minimum-local mean gradient[J]. Infrared Technology, 2021, 43(1): 13-20. http://hwjs.nvir.cn/article/id/144252d1-978c-4c1e-85ad-e0b8d5e03bf6
[4]	ZHANG Shuang, LIU Feng. Infrared and visible image fusion based on non-subsampled shearlet transform, regional energy, and co-occurrence filtering[J]. Electronics Letters, 2020, 56(15): 761-764. doi: 10.1049/el.2020.0557
[5]	LIU Yu, LIU Shuping, WANG Zengfu. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015, 24: 147-164. doi: 10.1016/j.inffus.2014.09.004
[6]	CUI Y, DU H, MEI W. Infrared and visible image fusion using detail enhanced channel attention network[J]. IEEE Access, 2019, 7: 182185-182197. doi: 10.1109/ACCESS.2019.2959034
[7]	XIA Jingming, YI Lu. Intelligent fusion of infrared and visible image data based on convolutional sparse representation and improved pulse-coupled neural network[J]. CMC-Computers Materials & Continua, 2021, 67(1): 613-624.
[8]	肖中杰. 基于NSCT红外与可见光图像融合算法优化研究[J]. 红外技术, 2017, 39(12): 1127-1130. http://hwjs.nvir.cn/article/id/hwjs201712010 XIAO Zhongjie. Improved infrared and visible light image fusion algorithm based on NSCT[J]. Infrared Technology, 2017, 39(12): 1127-1130. http://hwjs.nvir.cn/article/id/hwjs201712010
[9]	MA Jinlei, ZHOU Zhiqiang. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics and Technology, 2017, 82: 8-17. doi: 10.1016/j.infrared.2017.02.005
[10]	傅志中, 王雪, 李晓峰, 等. 基于视觉显著性和NSCT的红外与可见光图像融合[J]. 电子科技大学学报, 2017, 46(2): 357-362. doi: 10.3969/j.issn.1001-0548.2017.02.007 FU Zhizhong, WANG Xue, LI Xiaofeng, et al. Infrared and visible image fusion based on visual saliency and NSCT[J]. Journal of University of Electronic Science and Technology of China, 2017, 46(2): 357-362. doi: 10.3969/j.issn.1001-0548.2017.02.007
[11]	闫利, 向天烛. NSCT域内结合边缘特征和自适应PCNN的红外与可见光图像融合[J]. 电子学报, 2016, 44(4): 761-766. doi: 10.3969/j.issn.0372-2112.2016.04.002 YAN Li, XIANG Tianzhu. Fusion of infrared and visible images based on edge feature and adaptive PCNN in NSCT domain[J]. Acta Electronica Sinica, 2016, 44(4): 761-766. doi: 10.3969/j.issn.0372-2112.2016.04.002
[12]	陈震, 杨小平. 基于补偿机制的NSCT域红外与可见光图像融合[J]. 仪器仪表学报, 2016, 37(4): 860-870. doi: 10.3969/j.issn.0254-3087.2016.04.019 CHEN Zhen, YANG Xiaoping. Infrared and visible image fusion based on the compensation mechanism in NSCT domain[J]. Chinese Journal of Scientific Instrument, 2016, 37(4): 860-870. doi: 10.3969/j.issn.0254-3087.2016.04.019
[13]	周渝人, 耿爱辉, 张强. 基于压缩感知的红外与可见光图像融合[J]. 光学精密工程, 2015, 23(3): 855-863. https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201503032.htm ZHOU Yuren, GENG Aihui, ZHANG Qiang. Fusion of infrared and visible images based on compressive sensing[J]. Optics and Precision Engineering, 2015, 23(3): 855-863. https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201503032.htm
[14]	REN Long, PAN Zhibin. Infrared and visible image fusion based on edge-preserving guided filter and infrared feature decomposition[J]. Signal Processing, 2021, 186: 108108.
[15]	FU Zhizhong, WANG Xue. Infrared and visible images fusion based on RPCA and NSCT[J]. Infrared Physics and Technology, 2016, 77: 114-123.