基于NSST-DWT-ICSAPCNN的多模态图像融合算法

王晓娜; 潘晴; 田妮莉

基于NSST-DWT-ICSAPCNN的多模态图像融合算法

广东工业大学信息工程学院, 广东广州 510006

基金项目:

国家自然科学基金项目 61901123

详细信息

作者简介:
王晓娜（1997-），女，硕士研究生，主要研究方向为图像处理、模式识别。E-mail：717057123@qq.com

通讯作者:
潘晴（1975-），男，副教授，主要研究方向为图像处理、信号处理、模式识别等。E-mail：pangqing@gdut.edu.cn

中图分类号: TP391
计量
- 文章访问数: 184
- HTML全文浏览量: 73
- PDF下载量: 30
- 被引次数: 0
出版历程
- 收稿日期: 2021-09-02
- 修回日期: 2021-11-24
- 刊出日期: 2022-05-20

Multi-modality Image Fusion Algorithm Based on NSST-DWT-ICSAPCNN

Faculty of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China

摘要

摘要: 为了增加融合图像的信息量，结合非下采样剪切波变换（Non-Subsampled Shearlet Transform, NSST）和离散小波变换（Discrete Wavelet Transform, DWT）的互补优势，提出了改进的多模态图像融合方法。采用NSST对两幅源图像进行多尺度、多方向的分解，得到相应的高频子带和低频子带；利用DWT将低频子带进一步分解为低频能量子带和低频细节子带，并利用最大值选择规则融合能量子带；采用改进连接强度的自适应脉冲耦合神经网络（Improved Connection Strength Adaptive Pulse Coupled Neural Network, ICSAPCNN）分别融合细节子带和高频子带，并对能量子带和细节子带进行DWT逆变换，得到融合的低频子带；采用NSST逆变换重构出细节信息丰富的融合图像。实验证明，提出的算法在主观视觉和客观评价方面均优于其他几种算法，且能同时适用于红外与可见光源图像、医学源图像的融合。
- 多模态图像 /
- 图像融合 /
- 离散小波变换 /
- 自适应脉冲耦合神经网络 /
- 非下采样剪切波变换
Abstract: To increase the information of the fused image, this paper proposes an improved multi-modality image fusion algorithm that combines the complementary advantages of the non-subsampled shearlet transform (NSST) and discrete wavelet transform (DWT). NSST was used to decompose the two source images in multiscale and multi-direction to obtain the corresponding high-frequency and low-frequency sub-bands. The low-frequency sub-bands were further decomposed into low-frequency energy sub-bands and low-frequency detail sub-bands by the DWT, and the low-frequency energy sub-bands were fused by the maximum selection rules. An adaptive pulse-coupled neural network with improved connection strength (ICSAPCNN) was used to fuse the detailed sub-bands and high-frequency sub-bands, and the energy sub-bands and detailed sub-bands were fused by inverse DWT to obtain the fused low-frequency sub-bands. The NSST inverse transform was used to reconstruct the fusion image with rich details. The experimental results verified that the proposed algorithm is superior to the other algorithms in both subjective vision and objective evaluation and can be applied to the fusion of both infrared and visible source images and medical source images.
- multi-modality images /
- image fusion /
- discrete wavelet transform /
- adaptive pulse coupled neural network /
- non-subsampled shearlet transform

HTML全文

图 1 NSST二级分解过程

Figure 1. The two-level decomposition process of NSST

下载: 全尺寸图片幻灯片

图 2 DWT分解过程

Figure 2. The decomposition process of DWT

下载: 全尺寸图片幻灯片

图 3 基于NSST-DWT-ICSAPCNN的融合流程图

Figure 3. The fusion diagram based on NSST-DWT-ICSAPCNN

下载: 全尺寸图片幻灯片

图 4 “road”红外和可见光图像以及融合结果

Figure 4. The "road" infrared and visible source images and fusion results

下载: 全尺寸图片幻灯片

图 5 “tree”红外和可见光图像以及融合结果

Figure 5. The "tree" infrared and visible source images and fusion results

下载: 全尺寸图片幻灯片

图 6 致死性脑卒中CT和MRI图像以及融合结果

Figure 6. The fatal stroke CT and MRI source images and fusion results

下载: 全尺寸图片幻灯片

图 7 脑膜瘤CT和MRI图像以及融合结果

Figure 7. The meningoma CT and MRI source images and fusion results

下载: 全尺寸图片幻灯片

表 1 两组红外与可见光图像客观评估指标值

Table 1. Values of objective evaluation index for 2 groups of infrared and visible images

Images	Metrics	ASR^[7]	CNN^[8]	NSCT-APCNN^[9]	NSST-APCNN^[10]	NSST-DWT-ICSAPCNN
Road	Q_EN	7.1339	7.4964	7.3703	7.331	7.4247
	Q_MI	3.0046	3.2051	3.0786	3.2336	3.0167
	Q_SD	38.3922	48.4964	45.5887	44.5039	51.7009
	Q_VIFF	0.4469	0.5842	0.5206	0.5078	0.6275
	Q_IE	0.8055	0.8054	0.8052	0.8053	0.8062
	Q_TE	0.5749	0.5207	0.5401	0.5454	0.5886
Tree	Q_EN	6.3464	7.1022	6.9596	6.9152	7.1043
	Q_MI	1.2234	1.1755	1.3188	1.7535	2.1287
	Q_SD	24.3398	37.2648	32.8565	31.4357	34.8227
	Q_VIFF	0.3177	0.4706	0.3822	0.3798	0.4261
	Q_IE	0.8033	0.8043	0.8035	0.8035	0.8040
	Q_TE	0.4090	0.2861	0.2981	0.3279	0.3282

下载: 导出CSV

表 2 六组红外与可见光图像客观评估指标平均值

Table 2. Average values of objective evaluation index for 6 groups of infrared and visible images

Metrics	ASR^[7]	CNN^[8]	NSCT-APCNN^[9]	NSST-APCNN^[10]	NSST-DWT-ICSAPCNN
Q_EN	6.2345	6.8978	6.9633	6.9094	7.0247
Q_MI	2.8656	3.2917	3.6756	4.1826	4.3438
Q_SD	24.7236	38.7514	37.0670	35.4332	38.6467
Q_VIFF	0.3761	0.5399	0.5445	0.5032	0.5514
Q_IE	0.8063	0.8076	0.8086	0.8090	0.8097
Q_TE	0.7311	0.6582	0.6534	0.6971	0.6841

下载: 导出CSV

表 3 两组医学图像客观评估指标值

Table 3. Values of objective evaluation index for 2 groups of medical images

Images	Metrics	ASR^[7]	CNN^[8]	NSCT-APCNN^[9]	NSST-APCNN^[10]	NSST-DWT-ICSAPCNN
fatal stroke	Q_EN	4.5440	4.8244	5.0632	4.8747	5.1693
	Q_MI	2.5170	2.8593	2.7118	2.8665	2.7618
	Q_SD	72.3351	90.2448	90.0339	84.2365	88.4652
	Q_VIFF	0.2691	0.3333	0.3259	0.3100	0.3131
	Q_IE	0.8051	0.8055	0.8054	0.8051	0.8054
	Q_TE	0.6663	0.7252	0.7277	0.7102	0.7896
meningoma	Q_EN	4.1794	4.2013	4.3485	4.6852	4.6013
	Q_MI	2.5408	2.9163	2.9516	3.0001	3.0665
	Q_SD	72.0789	88.7470	92.8914	90.2904	91.3901
	Q_VIFF	0.4940	0.6192	0.6279	0.5624	0.6292
	Q_IE	0.8056	0.8059	0.8062	0.8064	0.8064
	Q_TE	0.7907	0.7923	0.8445	0.8733	0.8804

下载: 导出CSV

表 4 八组医学图像客观评估指标平均值

Table 4. Average values of objective evaluation index for 8 groups of medical images

Metrics	ASR^[7]	CNN^[8]	NSCT-APCNN^[9]	NSST-APCNN^[10]	NSST-DWT-ICSAPCNN
Q_EN	4.3242	4.6515	4.7943	4.7715	4.8254
Q_MI	2.6843	2.9002	2.8697	2.8998	2.9848
Q_SD	66.6290	83.3568	85.9244	85.7634	85.7755
Q_VIFF	0.3561	0.4417	0.4562	0.4491	0.4647
Q_IE	0.8057	0.8061	0.8061	0.8062	0.8062
Q_TE	0.7033	0.7494	0.7608	0.7593	0.7818

下载: 导出CSV

参考文献(18)

[1]	YANG Y, QUE Y, HUANG S, et al. Multimodal sensor medical image fusion based on type-2 fuzzy logic in NSCT domain[J]. IEEE Sensors Journal, 2016, 16(10): 3735-3745. doi: 10.1109/JSEN.2016.2533864
[2]	LI G, LIN Y, QU X. An infrared and visible image fusion method based on multi-scale transformation and norm optimization[J]. Information Fusion, 2021, 71: 109-129. doi: 10.1016/j.inffus.2021.02.008
[3]	LI X, ZHOU F, TAN H. Joint image fusion and denoising via three-layer decomposition and sparse representation[J]. Knowledge-Based Systems, 2021, 224: 107087. doi: 10.1016/j.knosys.2021.107087
[4]	XU H, MA J. EMFusion: An unsupervised enhanced medical image fusion network[J]. Information Fusion, 2021, 76: 177-186. . doi: 10.1016/j.inffus.2021.06.001
[5]	Bulanon D M, Burks T F, Alchanatis V. Image fusion of visible and thermal images for fruit detection[J]. Biosystems Engineering, 2009, 103(1): 12-22. doi: 10.1016/j.biosystemseng.2009.02.009
[6]	ZHAN L, ZHUANG Y, HUANG L. Infrared and visible images fusion method based on discrete wavelet transform[J]. J. Comput. , 2017, 28(2): 57-71.
[7]	LIU Y, WANG Z. Simultaneous image fusion and denoising with adaptive sparse representation[J]. IET Image Processing, 2015, 9(5): 347-357. doi: 10.1049/iet-ipr.2014.0311
[8]	LIU Y, CHEN X, CHENG J, et al. Infrared and visible image fusion with convolutional neural networks[J]. International Journal of Wavelets, Multiresolution and Information Processing, 2018, 16(3): 1850018. doi: 10.1142/S0219691318500182
[9]	ZHU Z, ZHENG M, QI G, et al. A phase congruency and local Laplacian energy based multi-modality medical image fusion method in NSCT domain[J]. IEEE Access, 2019, 7: 20811-20824. doi: 10.1109/ACCESS.2019.2898111
[10]	ZHANG L, ZENG G, WEI J, et al. Multi-modality image fusion in adaptive-parameters SPCNN based on inherent characteristics of image[J]. IEEE Sensors Journal, 2019, 20(20): 11820-11827.
[11]	张蕾. 采用改进平均梯度与自适应PCNN的图像融合[J]. 计算机应用与软件, 2021, 38(3): 218-223. doi: 10.3969/j.issn.1000-386x.2021.03.033 ZHANG Lei. Image fusion using improved average gradient and adaptive PCNN[J]. Computer Application and Software, 2021, 38(3): 218-223. doi: 10.3969/j.issn.1000-386x.2021.03.033
[12]	YIN M, LIU X, LIU Y, et al. Medical image fusion with parameter- adaptive pulse coupled neural network in nonsubsampled shearlet transform domain[J]. IEEE Transactions on Instrumentation and Measurement, 2018, 68(1): 49-64.
[13]	Diwakar M, Singh P, Shankar A. Multi-modal medical image fusion framework using co-occurrence filter and local extrema in NSST domain[J]. Biomedical Signal Processing and Control, 2021, 68: 102788. doi: 10.1016/j.bspc.2021.102788
[14]	邓辉, 王长龙, 胡永江, 等. 脉冲耦合神经网络在图像融合中的应用研究[J]. 电光与控制, 2019, 26(11): 19-24. https://www.cnki.com.cn/Article/CJFDTOTAL-DGKQ201911006.htm DENG Hui, WANG Changlong, HU Yongjiang, et al. Application of pulse coupled neural network in image fusion[J]. Electronics Options & Contral, 2019, 26(11): 19-24. https://www.cnki.com.cn/Article/CJFDTOTAL-DGKQ201911006.htm
[15]	杨风暴, 董安冉, 张雷, 等. DWT、NSCT和改进PCA协同组合红外偏振图像融合[J]. 红外技术, 2017, 39(3): 201-208. http://hwjs.nvir.cn/article/id/hwjs201703001 YANG Fengbao, DONG Aran, ZHANG Lei, et al. Infrared Polarization Image fusion using the synergistic combination of DWT, NSCT and improved PCA[J]. Infrared Technology, 2017, 39(3): 201-208. http://hwjs.nvir.cn/article/id/hwjs201703001
[16]	TAN W, Tiwari P, Pandey H M, et al. Multimodal medical image fusion algorithm in the era of big data[J]. Neural Computing and Applications, 2020: 1-21.
[17]	JIANG L, ZHANG D, CHE L. Texture analysis-based multi-focus image fusion using a modified Pulse-Coupled Neural Network (PCNN)[J]. Signal Processing: Image Communication, 2021, 91: 116068. doi: 10.1016/j.image.2020.116068
[18]	LIU Z, Blasch E, XUE Z, et al. Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 34(1): 94-109.