结合改进显著性检测与NSST的红外与可见光图像融合方法

叶坤涛; 李文; 舒蕾蕾; 李晟

结合改进显著性检测与NSST的红外与可见光图像融合方法

江西理工大学理学院，江西赣州 341000

基金项目:

江西省教育厅科学技术研究项目 GJJ170526

详细信息

作者简介:
叶坤涛（1972-），男，博士，副教授，硕士生导师，目前研究方向有MEMS、信号处理等。E-mail：mems_123@126.com

中图分类号: TP391.41
计量
- 文章访问数: 214
- HTML全文浏览量: 184
- PDF下载量: 55
- 被引次数: 0
出版历程
- 收稿日期: 2021-04-02
- 修回日期: 2021-04-13
- 刊出日期: 2021-12-20

Infrared and Visible Image Fusion Method Based on Improved Saliency Detection and Non-subsampled Shearlet Transform

School of Science, Jiangxi University of Science and Technology, Ganzhou 341000, China

摘要

摘要: 针对当前基于显著性检测的红外与可见光图像融合方法存在目标不够突出、对比度低等问题，本文提出了一种结合改进显著性检测与非下采样剪切波变换（non-subsampled shearlet transform, NSST）的融合方法。首先，使用改进最大对称环绕（maximum symmetric surround, MSS）算法提取出红外图像的显著性图，并进一步通过改进伽马校正进行增强，同时应用同态滤波增强可见光图像。然后，对红外图像与增强的可见光图像进行NSST分解，利用显著性图指导低频部分进行融合；同时设定区域能量取大规则指导高频部分融合。最后，通过NSST逆变换重构融合图像。实验结果表明，本文方法在平均梯度、信息熵、空间频率和标准差上远优于其他7种融合方法，可以有效突出红外目标，提高融合图像的对比度和清晰度，并保留可见光图像的丰富背景信息。
- 图像融合 /
- 显著性检测 /
- 非下采样剪切波变换 /
- 最大对称环绕 /
- 同态滤波
Abstract: To address the problems in the current infrared and visible image fusion method wherein targets are not prominent and contrast is low based on saliency detection, this paper proposes a fusion method by combining improved saliency detection and non-subsampled shearlet transform (NSST). First, the improved maximum symmetric surround algorithm is used to extract the saliency map of an infrared image, the improved gamma correction method is utilized to enhance the map, and the visible image is enhanced through homomorphic filtering. Second, the infrared and enhanced visible images are decomposed into low-and high-frequency parts through NSST, and the saliency map is used to guide the fusion of the low-frequency parts. Simultaneously, the rule of maximum region energy selection is used to guide the fusion of the high-frequency parts. Finally, the fusion image is reconstructed using the inverse NSST. The experimental results show that the proposed method is far superior to other seven fusion methods in terms of average gradient, information entropy, spatial frequency, and standard deviation. Thus, proposed method can effectively highlight the infrared target, improve the contrast and definition of fused images, and preserve rich background information of visible images.
- image fusion /
- saliency detection /
- non-subsampled shearlet transform /
- maximum symmetric surround /
- homomorphicfiltering

HTML全文

图 1 红外图像的显著性图

Figure 1. Saliency maps of infrared image

下载: 全尺寸图片幻灯片

图 2 可见光图像的增强结果

Figure 2. The enhancement results of visible image

下载: 全尺寸图片幻灯片

图 3 源图像的3级NSST分解过程

Figure 3. Three level NSST decomposition process of source image

下载: 全尺寸图片幻灯片

图 4 本文方法流程框图

Figure 4. Schematic of our proposed method

下载: 全尺寸图片幻灯片

图 5 “Camp”图像的融合结果

Figure 5. Fusion results on "Camp" image

下载: 全尺寸图片幻灯片

图 6 “Kaptein”图像的融合结果

Figure 6. Fusion results on "Kaptein" image

下载: 全尺寸图片幻灯片

图 7 “Tree”图像的融合结果

Figure 7. Fusion results on "Tree" image

下载: 全尺寸图片幻灯片

图 8 “APC”图像的融合结果

Figure 8. Fusion results on "APC" image

下载: 全尺寸图片幻灯片

图 9 “Marne”图像的融合结果

Figure 9. Fusion results on "Marne" image

下载: 全尺寸图片幻灯片

图 10 “Steamboat”图像的融合结果

Figure 10. Fusion results on "Steamboat" image

下载: 全尺寸图片幻灯片

表 1 前三组融合图像的客观评价结果

Table 1. The objective evaluation results of the first three groups of fused images

Images	Methods	MI	AG	IE	SF	SD
Camp	LP	1.4778	6.4761	6.6515	12.3629	28.9089
	CVT	1.3857	6.3399	6.5356	11.9980	26.7685
	NSCT	1.4565	6.3190	6.5447	12.0755	26.9155
	NSST-PCNN	1.9196	6.0255	6.6738	11.0564	28.8536
	NSST-PAPCNN	2.0762	5.8494	6.7943	10.5186	29.5982
	NSST-FT	2.4885	6.7244	7.0753	12.6604	38.0868
	Refs ^[6]	2.5042	6.6892	7.0785	12.6110	38.2066
	Proposed	2.7722	6.8739	7.2089	12.7387	39.1209
Kaptein	LP	1.5623	5.8712	6.6301	11.4779	35.3726
	CVT	1.4951	5.6699	6.5105	11.0531	30.6265
	NSCT	1.5836	5.7224	6.4981	11.2409	31.2731
	NSST-PCNN	2.1031	5.4520	6.7022	10.5732	38.4590
	NSST-PAPCNN	2.5149	4.8540	6.9353	9.5255	41.3301
	NSST-FT	3.3999	6.0008	7.2952	11.6395	56.2293
	Refs ^[6]	3.4299	6.0415	7.2943	11.7266	56.3160
	Proposed	3.9293	7.0654	7.6067	13.0026	65.3635
Tree	LP	1.5992	5.0152	6.0166	8.7768	16.4923
	CVT	1.5299	4.8874	5.9432	8.5367	15.3844
	NSCT	1.5883	4.8566	5.9525	8.5571	15.5355
	NSST-PCNN	1.9811	4.7939	6.2575	8.3593	19.8322
	NSST-PAPCNN	1.4902	4.8654	6.4536	8.5262	21.7829
	NSST-FT	2.0784	5.1505	6.3483	8.9234	20.8631
	Refs ^[6]	2.1290	5.1189	6.3475	8.8783	20.8498
	Proposed	3.1750	6.9352	7.1273	11.9791	36.2389

下载: 导出CSV

表 2 后三组融合图像的客观评价结果

Table 2. The objective evaluation results of the last three groups of fused images

Images	Methods	MI	AG	EN	SF	SD
APC	LP	0.6861	5.6418	5.8989	9.6969	15.0333
	CVT	0.5984	5.5327	5.7615	9.5127	13.5490
	NSCT	0.6326	5.5686	5.8252	9.5973	14.1751
	NSST-PCNN	1.3097	5.2874	5.9710	9.1733	15.6671
	NSST-PAPCNN	1.0539	3.5190	6.0989	6.4778	16.9964
	NSST-FT	2.0421	5.7441	6.4358	9.8360	21.0700
	Refs ^[6]	2.0316	5.7494	6.4383	9.8406	21.1113
	Proposed	2.4917	6.9622	6.7164	11.7056	25.5275
Marne	LP	2.1164	3.9831	6.7132	7.2543	27.6995
	CVT	1.8075	4.0716	6.6968	7.2306	27.4600
	NSCT	2.1117	3.8934	6.5826	7.0751	25.3114
	NSST-PCNN	2.9627	3.6828	6.9520	6.6598	36.0598
	NSST-PAPCNN	4.8694	3.2487	6.9969	6.2804	37.1140
	NSST-FT	2.7102	4.0981	7.1696	7.4645	45.7950
	Refs ^[6]	2.7344	4.1054	7.1787	7.5188	45.9412
	Proposed	3.3513	4.8461	7.4630	8.4996	73.7589
Steamboat	LP	1.6304	2.7601	5.3071	7.0341	14.0743
	CVT	1.4169	2.7281	5.2087	6.9052	12.4699
	NSCT	1.5171	2.7462	5.1657	6.9943	12.6583
	NSST-PCNN	3.3886	2.3807	5.7938	6.5980	18.9923
	NSST-PAPCNN	2.5101	2.7789	6.1287	6.9999	21.0319
	NSST-FT	2.7227	2.8126	5.9811	7.0718	18.4518
	Refs ^[6]	2.7219	2.8231	5.9639	7.1179	18.5114
	Proposed	3.0865	2.9878	6.2459	7.3507	25.2450

下载: 导出CSV

参考文献(20)

[1]	CHEN J, LI X, LUO L, et al. Infrared and visible image fusion based on target-enhanced multiscale transform decomposition[J]. Information Sciences, 2020, 508(1): 64-78. http://www.sciencedirect.com/science/article/pii/S0020025519308163
[2]	崔晓荣, 沈涛, 黄建鲁, 等. 基于BEMD改进的视觉显著性红外和可见光图像融合[J]. 红外技术, 2020, 42(11): 1061-1071. http://hwjs.nvir.cn/article/id/c89c0447-6d07-4a75-99f6-1bf8681cf588 CUI Xiaorong, SHEN Tao, HUANG Jianlu, et al. Infrared and visble image fusion based on BEMD and improved visual saliency[J]. Infrared Technology, 2020, 42(11): 1061-1071. http://hwjs.nvir.cn/article/id/c89c0447-6d07-4a75-99f6-1bf8681cf588
[3]	CHENG B, JIN L, LI G. Infrared and visual image fusion using LNSST and an adaptive dual-channel PCNN with triple-linking strength[J]. Neurocomputing, 2018, 310(10): 135-147. http://www.onacademic.com/detail/journal_1000040423651710_9d04.html
[4]	郑伟, 赵成晨, 郝冬梅. NSST和改进PCNN相结合的甲状腺图像融合[J]. 光电工程, 2016, 43(10): 42-48, 55. https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC201610008.htm ZHENG Wei, ZHAO Chengchen, HAO Dongmei. Thyroid image fusion based on NSST and improved PCNN[J]. Opto-Electronic Engineering, 2016, 43(10): 42-48, 55. https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC201610008.htm
[5]	Borji A, Itti L. State-of-the-art in visual attention modeling[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1): 185-207. doi: 10.1109/TPAMI.2012.89
[6]	傅志中, 王雪, 李晓峰, 等. 基于视觉显著性和NSCT的红外与可见光图像融合[J]. 电子科技大学学报, 2017, 46(2): 357-362. doi: 10.3969/j.issn.1001-0548.2017.02.007 FU Zhizhong, WANG Xue, LI Xiaofeng, et al. Infrared and visible image fusion based on visual saliency and NSCT[J]. Journal of University of Electronic Science and Technology of China, 2017, 46(2): 357-363. doi: 10.3969/j.issn.1001-0548.2017.02.007
[7]	林子慧, 魏宇星, 张建林, 等. 基于显著性图的红外与可见光图像融合[J]. 红外技术, 2019, 41(7): 640-645. http://hwjs.nvir.cn/article/id/hwjs201907008 LIN Zihui, WEI Yuxing, ZHANG Jianlin, et al. Image fusion of infrared and visible images based on saliency map[J]. Infrared Technology, 2019, 41(7): 640-645. http://hwjs.nvir.cn/article/id/hwjs201907008
[8]	安影, 范训礼, 陈莉, 等. 结合FABEMD和改进的显著性检测的图像融合[J]. 系统工程与电子技术, 2020, 42(2): 292-300. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYD202002006.htm AN Ying, FAN Xunli, CHEN Li, et al. Image fusion combining with FABEMD and improved saliency detection[J]. Systems Engineering and Electronics, 2020, 42(2): 292-300. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYD202002006.htm
[9]	Achanta R, Hemami S, Estrada F, et al. Frequency-tuned salient region detection[C]//IEEE Conference on Computer Vision & Pattern Recognition, 2009: 1597-1604.
[10]	Achanta R, Susstrunk S. Saliency detection using maximum symmetric surround[C]//Proc. of IEEE International Conference on Image Processing, 2010: 2653-2656.
[11]	冯维, 吴贵铭, 赵大兴, 等. 多图像融合Retinex用于弱光图像增强[J]. 光学精密工程, 2020, 28(3): 736-744. https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM202003024.htm FENG Wei, WU Guiming, ZHAO Daxing, et al. Multi images fusion Retinex for low light image enhancement[J]. Optics and Precision Engineering, 2020, 28(3): 736-744. https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM202003024.htm
[12]	吴京辉, 唐林波, 赵保军. 局部特征与全局信息联合的自适应图像增强算法[J]. 北京理工大学学报, 2014, 34(9): 955-960. https://www.cnki.com.cn/Article/CJFDTOTAL-BJLG201409015.htm WU Jinghui, TANG Linbo, ZHAO Baojun. Local-to-global adaptive image enhancement algorithm[J]. Transactions of Beijing Institute of Technology, 2014, 34(9): 955-960. https://www.cnki.com.cn/Article/CJFDTOTAL-BJLG201409015.htm
[13]	ZHAI Y, SHAH M. Visual attention detection in video sequences using spatiotemporal cues[C]//Proceedings of the 14th ACM International Conference on Multimedia, 2006: 815-824.
[14]	DONG S, MA J, SU Z, et al. Robust circular marker localization under non-uniform illuminations based on homomorphic filtering[J]. Measurement, 2021, 170(1): 108700. http://www.sciencedirect.com/science/article/pii/S0263224120312070
[15]	焦姣, 吴玲达, 于少波, 等. 混合多尺度分析和改进PCNN相结合的图像融合方法[J]. 计算机辅助设计与图形学学报, 2019, 31(6): 988-996. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJF201906015.htm JIAO Jiao, WU Lingda, YU Shaobo, et al. Image fusion method using multi-scale analysis and improved PCNN[J]. Journal of Computer-Aided Design & Computer Graphics, 2019, 31(6): 988-996. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJF201906015.htm
[16]	Burt P J, Adelson E H. The laplacian pyramid as a compact image code[J]. IEEE Transactions on Communications, 1983, 31(4): 532-540. doi: 10.1109/TCOM.1983.1095851
[17]	Nencini F, Garzelli A, Baronti S, et al. Remote sensing image fusion using the curvelet transform[J]. Information Fusion, 2007, 8(2): 143-156. doi: 10.1016/j.inffus.2006.02.001
[18]	ZHANG Q, GUO B. Multifocus image fusion using the nonsubsampled contourlet transform[J]. Signal Process, 2009, 89(7): 1334-1346. doi: 10.1016/j.sigpro.2009.01.012
[19]	YIN M, LIU X, LIU Y, et al. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain[J]. IEEE Transactions on Instrumentation and Measurement, 2019, 68(1): 49-64. doi: 10.1109/TIM.2018.2838778
[20]	Figshare. TNO Image Fusion Dataset[EB/OL]. [2014-04-26]. https://figshare.com/articles/TNO_Image_Fusion_Dataset/1008029.