Infrared and Visible Image Fusion Based on CNN with NSCT
-
摘要: 传统红外与可见光融合图像在复杂环境下存在细节缺失,特征单一导致目标模糊等问题,本文提出一种基于卷积神经网络结合非下采样轮廓波变换(non-subsampled contourlet transform,NSCT)的红外与可见光图像进行融合的方法。首先,通过卷积神经网络提取红外与可见光目标特征信息,同时利用NSCT对源图像进行多尺度分解,得到源图像的高频系数与低频系数;其次,结合目标特征图利用自适应模糊逻辑与局部方差对比度分别对源图像高频子带与低频子带进行融合;最后,通过逆NSCT变换得到融合图像并与其他5种传统算法进行对比;实验结果表明,本文方法在多个客观评价指标上均有所提高。Abstract: Traditional infrared and visible fused images suffer from missing details and blurred targets owing to single features in complex environments. This study presents a method for fusing infrared and visible images based on a convolution neural network(CNN) combined with a non-subsampled contourlet transform (NSCT). Firstly, the infrared and visible target feature information is extracted by CNN, and the source image is decomposed by the NSCT at multiple scales to obtain its high-frequency coefficients and low-frequency coefficients. Secondly, the high-frequency sub-bands and low-frequency sub-bands of the source image are fused separately using adaptive fuzzy logic and local variance contrast in combination with the target feature image. Finally, the fused image is obtained by inverse NSCT transformation. We conducted a comparative analysis with five other traditional algorithms. The experimental results show that the proposed method performs better in several objective evaluation indicators.
-
Keywords:
- image fusion /
- convolutional neural network /
- NSCT /
- fuzzy logic
-
-
图 3 红外图像“Nato_camp”及多种方法提取显著性图像:(a) 红外图像“Nato_camp”;(b) 图像“Nato_camp”标准分割图;(c) FT方法;(d) AC方法;(e) LC方法;(f) CNN方法
Figure 3. Infrared image ''Nato_camp'' and images after saliency extraction by various methods: (a) Infrared image "Nato_camp"; (b) Standard segmentation of image "Nato_camp"; (c) FT method; (d) AC method; (e) LC method; (f) CNN method
图 4 红外图像“Kaptein”及多种方法提取显著性图像:(a) 红外图像“Kaptein”;(b) 图像“Kaptein”标准分割图;(c) FT方法;(d) AC方法;(e) LC方法;(f) CNN方法
Figure 4. Infrared image ''Kaptein'' and images after saliency extraction by various methods: (a) Infrared image "Kaptein"; (b) Standard segmentation of image "Kaptein"; (c) FT method; (d) AC method; (e) LC method; (f) CNN method
图 6 “Nato_camp”红外和可见光图像以及融合结果:(a) 红外图像;(b) 可见光图像;(c) LP方法;(d) DWT方法;(e) BEMD方法;(f) NSST方法;(g) NSCT方法;(h) 本文方法
Figure 6. ''Nato_camp'' infrared and visible images and fusion results: (a) Infrared image; (b) Visible image; (c) LP method; (d) DWT method; (e) BEMD method; (f) NSST method; (g) NSCT method; (h) Proposed method
图 7 “Kaptein”红外和可见光图像以及融合结果:(a) 红外图像;(b) 可见光图像;(c) LP方法;(d) DWT方法;(e) BEMD方法;(f) NSST方法;(g) NSCT方法;(h) 本文方法
Figure 7. ''Kaptein'' infrared and visible images and fusion results: (a) Infrared image; (b) Visible image; (c) LP method (d) DWT method; (e) BEMD method; (f) NSST method; (g) NSCT method; (h) Proposed method
图 8 “iron”红外和可见光图像以及融合结果:(a) 红外图像;(b) 可见光图像;(c) LP方法;(d) DWT方法;(e) BEMD方法;(f) NSST方法;(g) NSCT方法;(h)本文方法
Figure 8. ''iron'' infrared and visible images and fusion results: (a) Infrared image; (b) Visible image; (c) LP method (d) DWT method; (e) BEMD method; (f) NSST method; (g) NSCT method; (h) Proposed method
表 1 目标显著性提取评价指标MAE
Table 1 Target significance extraction evaluation index MAE
Method AC FT LC CNN MSE1 1429.6117 1816.0338 1377.7657 9.8357 MSE2 672.0769 1379.5997 1965.2603 17.2382 表 2 红外与可见光图像融合效果评价
Table 2 Infrared and visible image fusion effect evaluation
Image Image fusion method IE AG SF MI CEN “Nato_camp” LP 6.6747 5.5365 15.9373 1.7365 1.4858 DWT 6.9908 6.6575 17.4658 1.6712 0.6218 BEMD 6.6029 6.2838 17.1804 1.3671 1.7880 NSST 6.8419 5.8138 16.7266 1.3348 1.5568 NSCT 6.6224 6.3133 17.3085 2.0859 0.3409 Ours 7.1934 4.7142 17.5647 2.0883 0.3206 “Kaptein” LP 6.7174 4.0659 12.8846 3.2774 1.3173 DWT 7.1419 6.2009 18.9346 3.8891 1.0452 BEMD 6.7497 5.9790 19.4699 3.0136 1.2275 NSST 6.8063 4.4414 15.8489 3.9370 1.1409 NSCT 6.9281 6.3012 19.2932 3.6070 1.6858 Ours 7.2729 4.5076 20.1209 3.9387 1.0457 “iron” LP 6.4638 7.1542 20.2285 3.1355 0.5064 DWT 6.6377 11.7327 33.4574 3.3785 0.6087 BEMD 6.5765 9.4525 23.3567 3.4273 0.5408 NSST 6.7624 14.7945 39.4278 3.4058 0.5075 NSCT 6.7352 15.6254 40.4253 3.1668 0.4772 Ours 6.7648 14.7857 40.8547 3.8173 0.4117 -
[1] MA J Y, MA Yong. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. DOI: 10.1016/j.inffus.2018.02.004
[2] 董安勇, 杜庆治. 基于卷积神经网络的红外与可见光图像融合[J]. 红外技术, 2020, 42(7): 660-669. http://hwjs.nvir.cn/article/id/hwjs202007009 DONG Anyong, DU Qingzhi. Infrared and visible image fusion based on vonvolutional neural network[J]. Infrared Technology, 2020, 42(7): 660-669. http://hwjs.nvir.cn/article/id/hwjs202007009
[3] 杨孙运, 奚峥皓, 汪汉东, 等. 基于NSCT和最小化-局部平均梯度的图像融合[J]. 红外技术, 2021, 43(1): 13-20. http://hwjs.nvir.cn/article/id/144252d1-978c-4c1e-85ad-e0b8d5e03bf6 YANG Sunyun, XI Zhenghao, WANG Handong, et al. Image fusion based on NSCT and minimum-local mean gradient[J]. Infrared Technology, 2021, 43(1): 13-20. http://hwjs.nvir.cn/article/id/144252d1-978c-4c1e-85ad-e0b8d5e03bf6
[4] ZHANG Shuang, LIU Feng. Infrared and visible image fusion based on non-subsampled shearlet transform, regional energy, and co-occurrence filtering[J]. Electronics Letters, 2020, 56(15): 761-764. DOI: 10.1049/el.2020.0557
[5] LIU Yu, LIU Shuping, WANG Zengfu. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015, 24: 147-164. DOI: 10.1016/j.inffus.2014.09.004
[6] CUI Y, DU H, MEI W. Infrared and visible image fusion using detail enhanced channel attention network[J]. IEEE Access, 2019, 7: 182185-182197. DOI: 10.1109/ACCESS.2019.2959034
[7] XIA Jingming, YI Lu. Intelligent fusion of infrared and visible image data based on convolutional sparse representation and improved pulse-coupled neural network[J]. CMC-Computers Materials & Continua, 2021, 67(1): 613-624.
[8] 肖中杰. 基于NSCT红外与可见光图像融合算法优化研究[J]. 红外技术, 2017, 39(12): 1127-1130. http://hwjs.nvir.cn/article/id/hwjs201712010 XIAO Zhongjie. Improved infrared and visible light image fusion algorithm based on NSCT[J]. Infrared Technology, 2017, 39(12): 1127-1130. http://hwjs.nvir.cn/article/id/hwjs201712010
[9] MA Jinlei, ZHOU Zhiqiang. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics and Technology, 2017, 82: 8-17. DOI: 10.1016/j.infrared.2017.02.005
[10] 傅志中, 王雪, 李晓峰, 等. 基于视觉显著性和NSCT的红外与可见光图像融合[J]. 电子科技大学学报, 2017, 46(2): 357-362. DOI: 10.3969/j.issn.1001-0548.2017.02.007 FU Zhizhong, WANG Xue, LI Xiaofeng, et al. Infrared and visible image fusion based on visual saliency and NSCT[J]. Journal of University of Electronic Science and Technology of China, 2017, 46(2): 357-362. DOI: 10.3969/j.issn.1001-0548.2017.02.007
[11] 闫利, 向天烛. NSCT域内结合边缘特征和自适应PCNN的红外与可见光图像融合[J]. 电子学报, 2016, 44(4): 761-766. DOI: 10.3969/j.issn.0372-2112.2016.04.002 YAN Li, XIANG Tianzhu. Fusion of infrared and visible images based on edge feature and adaptive PCNN in NSCT domain[J]. Acta Electronica Sinica, 2016, 44(4): 761-766. DOI: 10.3969/j.issn.0372-2112.2016.04.002
[12] 陈震, 杨小平. 基于补偿机制的NSCT域红外与可见光图像融合[J]. 仪器仪表学报, 2016, 37(4): 860-870. DOI: 10.3969/j.issn.0254-3087.2016.04.019 CHEN Zhen, YANG Xiaoping. Infrared and visible image fusion based on the compensation mechanism in NSCT domain[J]. Chinese Journal of Scientific Instrument, 2016, 37(4): 860-870. DOI: 10.3969/j.issn.0254-3087.2016.04.019
[13] 周渝人, 耿爱辉, 张强. 基于压缩感知的红外与可见光图像融合[J]. 光学精密工程, 2015, 23(3): 855-863. https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201503032.htm ZHOU Yuren, GENG Aihui, ZHANG Qiang. Fusion of infrared and visible images based on compressive sensing[J]. Optics and Precision Engineering, 2015, 23(3): 855-863. https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201503032.htm
[14] REN Long, PAN Zhibin. Infrared and visible image fusion based on edge-preserving guided filter and infrared feature decomposition[J]. Signal Processing, 2021, 186: 108108.
[15] FU Zhizhong, WANG Xue. Infrared and visible images fusion based on RPCA and NSCT[J]. Infrared Physics and Technology, 2016, 77: 114-123.
-
期刊类型引用(3)
1. 刘文强,姜迈,乔顺利,李宏达. 面向双模态夜视图像的混合尺度融合算法. 兵器装备工程学报. 2024(05): 291-298 . 百度学术
2. 王金生,周元元,陈珺. 基于残差Swin Transformer模块的红外与可见光图像融合研究. 苏州市职业大学学报. 2024(02): 55-62 . 百度学术
3. 陈彦林,王志社,邵文禹,杨帆,孙婧. 红外与可见光图像多尺度Transformer融合方法. 红外技术. 2023(03): 266-275 . 本站查看
其他类型引用(2)