Infrared and Visible Image Fusion Method Based on Improved Saliency Detection and Non-subsampled Shearlet Transform
-
摘要: 针对当前基于显著性检测的红外与可见光图像融合方法存在目标不够突出、对比度低等问题,本文提出了一种结合改进显著性检测与非下采样剪切波变换(non-subsampled shearlet transform, NSST)的融合方法。首先,使用改进最大对称环绕(maximum symmetric surround, MSS)算法提取出红外图像的显著性图,并进一步通过改进伽马校正进行增强,同时应用同态滤波增强可见光图像。然后,对红外图像与增强的可见光图像进行NSST分解,利用显著性图指导低频部分进行融合;同时设定区域能量取大规则指导高频部分融合。最后,通过NSST逆变换重构融合图像。实验结果表明,本文方法在平均梯度、信息熵、空间频率和标准差上远优于其他7种融合方法,可以有效突出红外目标,提高融合图像的对比度和清晰度,并保留可见光图像的丰富背景信息。Abstract: To address the problems in the current infrared and visible image fusion method wherein targets are not prominent and contrast is low based on saliency detection, this paper proposes a fusion method by combining improved saliency detection and non-subsampled shearlet transform (NSST). First, the improved maximum symmetric surround algorithm is used to extract the saliency map of an infrared image, the improved gamma correction method is utilized to enhance the map, and the visible image is enhanced through homomorphic filtering. Second, the infrared and enhanced visible images are decomposed into low-and high-frequency parts through NSST, and the saliency map is used to guide the fusion of the low-frequency parts. Simultaneously, the rule of maximum region energy selection is used to guide the fusion of the high-frequency parts. Finally, the fusion image is reconstructed using the inverse NSST. The experimental results show that the proposed method is far superior to other seven fusion methods in terms of average gradient, information entropy, spatial frequency, and standard deviation. Thus, proposed method can effectively highlight the infrared target, improve the contrast and definition of fused images, and preserve rich background information of visible images.
-
表 1 前三组融合图像的客观评价结果
Table 1. The objective evaluation results of the first three groups of fused images
Images Methods MI AG IE SF SD Camp LP 1.4778 6.4761 6.6515 12.3629 28.9089 CVT 1.3857 6.3399 6.5356 11.9980 26.7685 NSCT 1.4565 6.3190 6.5447 12.0755 26.9155 NSST-PCNN 1.9196 6.0255 6.6738 11.0564 28.8536 NSST-PAPCNN 2.0762 5.8494 6.7943 10.5186 29.5982 NSST-FT 2.4885 6.7244 7.0753 12.6604 38.0868 Refs [6] 2.5042 6.6892 7.0785 12.6110 38.2066 Proposed 2.7722 6.8739 7.2089 12.7387 39.1209 Kaptein LP 1.5623 5.8712 6.6301 11.4779 35.3726 CVT 1.4951 5.6699 6.5105 11.0531 30.6265 NSCT 1.5836 5.7224 6.4981 11.2409 31.2731 NSST-PCNN 2.1031 5.4520 6.7022 10.5732 38.4590 NSST-PAPCNN 2.5149 4.8540 6.9353 9.5255 41.3301 NSST-FT 3.3999 6.0008 7.2952 11.6395 56.2293 Refs [6] 3.4299 6.0415 7.2943 11.7266 56.3160 Proposed 3.9293 7.0654 7.6067 13.0026 65.3635 Tree LP 1.5992 5.0152 6.0166 8.7768 16.4923 CVT 1.5299 4.8874 5.9432 8.5367 15.3844 NSCT 1.5883 4.8566 5.9525 8.5571 15.5355 NSST-PCNN 1.9811 4.7939 6.2575 8.3593 19.8322 NSST-PAPCNN 1.4902 4.8654 6.4536 8.5262 21.7829 NSST-FT 2.0784 5.1505 6.3483 8.9234 20.8631 Refs [6] 2.1290 5.1189 6.3475 8.8783 20.8498 Proposed 3.1750 6.9352 7.1273 11.9791 36.2389 表 2 后三组融合图像的客观评价结果
Table 2. The objective evaluation results of the last three groups of fused images
Images Methods MI AG EN SF SD APC LP 0.6861 5.6418 5.8989 9.6969 15.0333 CVT 0.5984 5.5327 5.7615 9.5127 13.5490 NSCT 0.6326 5.5686 5.8252 9.5973 14.1751 NSST-PCNN 1.3097 5.2874 5.9710 9.1733 15.6671 NSST-PAPCNN 1.0539 3.5190 6.0989 6.4778 16.9964 NSST-FT 2.0421 5.7441 6.4358 9.8360 21.0700 Refs [6] 2.0316 5.7494 6.4383 9.8406 21.1113 Proposed 2.4917 6.9622 6.7164 11.7056 25.5275 Marne LP 2.1164 3.9831 6.7132 7.2543 27.6995 CVT 1.8075 4.0716 6.6968 7.2306 27.4600 NSCT 2.1117 3.8934 6.5826 7.0751 25.3114 NSST-PCNN 2.9627 3.6828 6.9520 6.6598 36.0598 NSST-PAPCNN 4.8694 3.2487 6.9969 6.2804 37.1140 NSST-FT 2.7102 4.0981 7.1696 7.4645 45.7950 Refs [6] 2.7344 4.1054 7.1787 7.5188 45.9412 Proposed 3.3513 4.8461 7.4630 8.4996 73.7589 Steamboat LP 1.6304 2.7601 5.3071 7.0341 14.0743 CVT 1.4169 2.7281 5.2087 6.9052 12.4699 NSCT 1.5171 2.7462 5.1657 6.9943 12.6583 NSST-PCNN 3.3886 2.3807 5.7938 6.5980 18.9923 NSST-PAPCNN 2.5101 2.7789 6.1287 6.9999 21.0319 NSST-FT 2.7227 2.8126 5.9811 7.0718 18.4518 Refs [6] 2.7219 2.8231 5.9639 7.1179 18.5114 Proposed 3.0865 2.9878 6.2459 7.3507 25.2450 -
[1] CHEN J, LI X, LUO L, et al. Infrared and visible image fusion based on target-enhanced multiscale transform decomposition[J]. Information Sciences, 2020, 508(1): 64-78. http://www.sciencedirect.com/science/article/pii/S0020025519308163 [2] 崔晓荣, 沈涛, 黄建鲁, 等. 基于BEMD改进的视觉显著性红外和可见光图像融合[J]. 红外技术, 2020, 42(11): 1061-1071. http://hwjs.nvir.cn/article/id/c89c0447-6d07-4a75-99f6-1bf8681cf588CUI Xiaorong, SHEN Tao, HUANG Jianlu, et al. Infrared and visble image fusion based on BEMD and improved visual saliency[J]. Infrared Technology, 2020, 42(11): 1061-1071. http://hwjs.nvir.cn/article/id/c89c0447-6d07-4a75-99f6-1bf8681cf588 [3] CHENG B, JIN L, LI G. Infrared and visual image fusion using LNSST and an adaptive dual-channel PCNN with triple-linking strength[J]. Neurocomputing, 2018, 310(10): 135-147. http://www.onacademic.com/detail/journal_1000040423651710_9d04.html [4] 郑伟, 赵成晨, 郝冬梅. NSST和改进PCNN相结合的甲状腺图像融合[J]. 光电工程, 2016, 43(10): 42-48, 55. https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC201610008.htmZHENG Wei, ZHAO Chengchen, HAO Dongmei. Thyroid image fusion based on NSST and improved PCNN[J]. Opto-Electronic Engineering, 2016, 43(10): 42-48, 55. https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC201610008.htm [5] Borji A, Itti L. State-of-the-art in visual attention modeling[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1): 185-207. doi: 10.1109/TPAMI.2012.89 [6] 傅志中, 王雪, 李晓峰, 等. 基于视觉显著性和NSCT的红外与可见光图像融合[J]. 电子科技大学学报, 2017, 46(2): 357-362. doi: 10.3969/j.issn.1001-0548.2017.02.007FU Zhizhong, WANG Xue, LI Xiaofeng, et al. Infrared and visible image fusion based on visual saliency and NSCT[J]. Journal of University of Electronic Science and Technology of China, 2017, 46(2): 357-363. doi: 10.3969/j.issn.1001-0548.2017.02.007 [7] 林子慧, 魏宇星, 张建林, 等. 基于显著性图的红外与可见光图像融合[J]. 红外技术, 2019, 41(7): 640-645. http://hwjs.nvir.cn/article/id/hwjs201907008LIN Zihui, WEI Yuxing, ZHANG Jianlin, et al. Image fusion of infrared and visible images based on saliency map[J]. Infrared Technology, 2019, 41(7): 640-645. http://hwjs.nvir.cn/article/id/hwjs201907008 [8] 安影, 范训礼, 陈莉, 等. 结合FABEMD和改进的显著性检测的图像融合[J]. 系统工程与电子技术, 2020, 42(2): 292-300. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYD202002006.htmAN Ying, FAN Xunli, CHEN Li, et al. Image fusion combining with FABEMD and improved saliency detection[J]. Systems Engineering and Electronics, 2020, 42(2): 292-300. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYD202002006.htm [9] Achanta R, Hemami S, Estrada F, et al. Frequency-tuned salient region detection[C]//IEEE Conference on Computer Vision & Pattern Recognition, 2009: 1597-1604. [10] Achanta R, Susstrunk S. Saliency detection using maximum symmetric surround[C]//Proc. of IEEE International Conference on Image Processing, 2010: 2653-2656. [11] 冯维, 吴贵铭, 赵大兴, 等. 多图像融合Retinex用于弱光图像增强[J]. 光学精密工程, 2020, 28(3): 736-744. https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM202003024.htmFENG Wei, WU Guiming, ZHAO Daxing, et al. Multi images fusion Retinex for low light image enhancement[J]. Optics and Precision Engineering, 2020, 28(3): 736-744. https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM202003024.htm [12] 吴京辉, 唐林波, 赵保军. 局部特征与全局信息联合的自适应图像增强算法[J]. 北京理工大学学报, 2014, 34(9): 955-960. https://www.cnki.com.cn/Article/CJFDTOTAL-BJLG201409015.htmWU Jinghui, TANG Linbo, ZHAO Baojun. Local-to-global adaptive image enhancement algorithm[J]. Transactions of Beijing Institute of Technology, 2014, 34(9): 955-960. https://www.cnki.com.cn/Article/CJFDTOTAL-BJLG201409015.htm [13] ZHAI Y, SHAH M. Visual attention detection in video sequences using spatiotemporal cues[C]//Proceedings of the 14th ACM International Conference on Multimedia, 2006: 815-824. [14] DONG S, MA J, SU Z, et al. Robust circular marker localization under non-uniform illuminations based on homomorphic filtering[J]. Measurement, 2021, 170(1): 108700. http://www.sciencedirect.com/science/article/pii/S0263224120312070 [15] 焦姣, 吴玲达, 于少波, 等. 混合多尺度分析和改进PCNN相结合的图像融合方法[J]. 计算机辅助设计与图形学学报, 2019, 31(6): 988-996. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJF201906015.htmJIAO Jiao, WU Lingda, YU Shaobo, et al. Image fusion method using multi-scale analysis and improved PCNN[J]. Journal of Computer-Aided Design & Computer Graphics, 2019, 31(6): 988-996. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJF201906015.htm [16] Burt P J, Adelson E H. The laplacian pyramid as a compact image code[J]. IEEE Transactions on Communications, 1983, 31(4): 532-540. doi: 10.1109/TCOM.1983.1095851 [17] Nencini F, Garzelli A, Baronti S, et al. Remote sensing image fusion using the curvelet transform[J]. Information Fusion, 2007, 8(2): 143-156. doi: 10.1016/j.inffus.2006.02.001 [18] ZHANG Q, GUO B. Multifocus image fusion using the nonsubsampled contourlet transform[J]. Signal Process, 2009, 89(7): 1334-1346. doi: 10.1016/j.sigpro.2009.01.012 [19] YIN M, LIU X, LIU Y, et al. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain[J]. IEEE Transactions on Instrumentation and Measurement, 2019, 68(1): 49-64. doi: 10.1109/TIM.2018.2838778 [20] Figshare. TNO Image Fusion Dataset[EB/OL]. [2014-04-26]. https://figshare.com/articles/TNO_Image_Fusion_Dataset/1008029.