基于MSPCNN与FCM的红外与可见光图像融合

邸敬; 王国栋; 马帅; 廉敬

基于MSPCNN与FCM的红外与可见光图像融合

兰州交通大学电子与信息工程学院, 甘肃兰州 730070

基金项目:

甘肃省科技技术资助 22JR5RA360

国家自然科学基金 62061023

国家自然科学基金 61941109

甘肃省高等学校创新能力提升项目 2019B-052

甘肃省杰出青年基金 21JR7RA345

详细信息

作者简介:
邸敬（1979-），女，甘肃兰州人，硕士，副教授，硕士生导师，主要从事图像检测识别、信号处理技术和宽带无线通信方面的研究。E-mail: 46891771@qq.com

中图分类号: TP391.9
计量
- 文章访问数: 133
- HTML全文浏览量: 32
- PDF下载量: 42
- 被引次数: 0
出版历程
- 收稿日期: 2022-06-29
- 修回日期: 2023-01-12
- 刊出日期: 2023-01-20

Infrared and Visible Image Fusion Based on MSPCNN and FCM

School of Electronic and Information Engineering, Lanzhou Jiaotong University, Lanzhou 730070, China

摘要

摘要: 针对红外和可见光图像融合存在的轮廓信息不全、边缘及纹理细节信息缺失等问题，提出一种改进简化脉冲耦合神经网络（Improved Simplified Pulse Coupled Neural Network, MSPCNN）和模糊C-均值（Fuzzy C-mean, FCM）图像融合算法。首先，将红外和可见光图像用非下采样剪切波算法（Non-Subsampled Shearlet Transform，NSST）分解为高低频子带；然后对分解后的高频子带采用MSPCNN融合，用一种高斯分布权重矩阵进行处理，增强细节信息和对比度；接着，将得到的低频子带图像使用FCM聚类算法进行聚类中心提取，设置聚类中心近似阈值简化过程，实现背景分类提取；最后利NSST进行逆变换，从而完成红外和可见光的图像融合过程。通过客观评价指标计算，本文所提方法在平均梯度、标准差、平均相似度等参考指标上相对于其他同类型算法均有改善提高，由于模型参数的简化，算法运行速度相对于其他算法得到提升，算法更适用于复杂场景。
- 图像融合 /
- 非下采样剪切波算法 /
- 脉冲耦合神经网络 /
- 模糊C-均值
Abstract: Aiming at the problems of incomplete contour information, missing edge and texture details in infrared and visible image fusion, A Improved Simplified Pulse Coupled Neural Network (MSPCNN) and Fuzzy C-mean (FCM) image fusion algorithm is proposed. First, infrared and visible images were decomposed into high and low frequency sub-bands using the Non-Subsampled Shearlet Transform (NSST).Then MSPCNN is used to fuse the decomposed high frequency subband, and a Gaussian distribution weight matrix is used for processing to enhance the detail information and contrast. Then, the obtained low-frequency sub-band images were extracted by using FCM clustering algorithm, and the approximate threshold of clustering center was set to simplify the process to achieve background classification extraction.Finally, the inverse transformation of NSST is carried out to complete the infrared and visible image fusion process.Through objective evaluation index calculation, compared with other algorithms of the same type, the method proposed in this paper has been improved in terms of average gradient, standard deviation, average similarity and other reference indexes. As the running speed of simplified algorithm of model parameters has been improved, the timeliness of the algorithm in this paper has been improved compared with other algorithms, and the algorithm is more suitable for complex scenarios.
- image fusion /
- non-subsampled shearlet transform /
- pulse coupled neural network /
- fuzzy C-mean

HTML全文

图 1 NSST变换分解过程

Figure 1. NSST transform decomposition process

下载: 全尺寸图片幻灯片

图 2 MSPCNN模型

Figure 2. Model diagram of MSPCNN

下载: 全尺寸图片幻灯片

图 3 MSPCNN与FCM结合的图像融合模型

Figure 3. Image fusion model based on MSPCNN and FCM

下载: 全尺寸图片幻灯片

图 4 不同算法在6种不同场景下的融合图像

Figure 4. Fusion images of different algorithms in six different scenes

下载: 全尺寸图片幻灯片

表 1 不同方法的客观评价指标均值

Table 1. Objective evaluation index mean value of different methods

Image	Algorithm	AVG	SSIM	Q^AB/F	PSN	SF	FD
First	PCA	3.4672	0.7171	0.4166	19.5587	8.1563	4.2357
	NSCT-PCNN	2.6693	0.7345	0.3013	22.4713	5.8531	3.0063
	NSST-PCNN	3.3602	0.7281	0.4503	19.5606	8.0915	4.0816
	PCNN-IFS	3.3544	0.3994	0.5144	12.3504	8.0694	4.0632
	NSST-FCM	4.8366	0.7226	0.4541	18.9521	12.6139	6.0441
	NSST-PA-PCNN	4.3941	0.7196	0.4776	19.0621	11.0428	5.1949
	Proposed algorithm	4.9783	0.7357	0.4673	19.5632	11.9764	5.8945
Second	PCA	2.2481	0.6639	0.4406	12.7854	5.3698	2.7438
	NSCT-PCNN	1.7512	0.6406	0.3292	18.6211	3.9054	1.9935
	NSST-PCNN	2.4183	0.6599	0.4638	12.7745	5.8367	2.9133
	PCNN-IFS	2.4097	0.4415	0.5566	11.9401	5.8081	2.8891
	NSST-FCM	3.2193	0.6478	0.4973	14.7370	8.1806	4.3017
	NSST-PA-PCNN	2.6837	0.6437	0.4906	12.6591	6.6781	3.2303
	Proposed algorithm	3.3542	0.6546	0.5156	13.6431	8.3459	3.9543
Third	PCA	3.1381	0.522	0.2667	16.6712	12.5317	5.5287
	NSCT-PCNN	2.1691	0.5998	0.3712	18.7813	6.0813	2.4056
	NSST-PCNN	2.6825	0.6185	0.4206	16.8078	9.7897	3.8225
	PCNN-IFS	2.6759	0.4484	0.4881	16.7297	9.7805	3.8034
	NSST-FCM	3.6972	0.6182	0.6442	17.2496	13.2931	4.6566
	NSST-PA-PCNN	3.2976	0.6164	0.5943	16.4798	11.6122	4.0373
	Proposed algorithm	3.6932	0.6245	0.6358	18.9682	12.3589	4.3589
Fourth	PCA	2.7559	0.7871	0.5614	16.8755	7.7808	3.3122
	NSCT-PCNN	1.5552	0.763	0.2586	16.9802	4.4557	1.8387
	NSST-PCNN	2.7275	0.7872	0.6003	16.8603	9.1856	3.5813
	PCNN-IFS	2.7188	0.4403	0.6549	12.5775	9.1723	3.5592
	NSST-FCM	3.1382	0.7685	0.5462	17.0442	10.7773	4.0853
	NSST-PA-PCNN	2.9165	0.7659	0.5529	16.7003	9.8139	3.6201
	Proposed algorithm	3.3584	0.7756	0.5641	17.1298	9.9821	4.1269
Fifth	PCA	4.9907	0.6288	0.4748	19.5355	9.9664	6.0269
	NSCT-PCNN	3.0046	0.6506	0.2351	21.5392	5.9493	3.5833
	NSST-PCNN	4.9841	0.6214	0.4186	19.4975	10.9044	6.5557
	PCNN-IFS	4.9805	0.3504	0.4423	11.1055	10.8874	6.5449
	NSST-FCM	6.2671	0.6559	0.4191	19.1803	13.5097	7.9851
	NSST-PA-PCNN	5.3543	0.6463	0.4451	19.2727	11.4391	6.6561
	Proposed algorithm	6.1596	0.7521	0.4563	19.6581	12.6985	6.7539
Sixth	PCA	3.9552	0.5015	0.5236	19.7263	10.9258	4.1901
	NSCT-PCNN	2.8944	0.4914	0.3599	21.4056	9.4389	2.9698
	NSST-PCNN	3.5472	0.5064	0.5285	19.7583	10.5539	3.7219
	PCNN-IFS	3.5657	0.4371	0.6022	16.0777	10.5725	3.7328
	NSST-FCM	5.2416	0.5077	0.6883	18.5657	14.9875	5.5393
	NSST-PA-PCNN	4.7394	0.5021	0.6371	19.0715	14.2598	4.9865
	Proposed algorithm	5.3684	0.5129	0.6752	20.5691	13.5214	5.6321

下载: 导出CSV

表 2 不同算法在6种不同场景下的融合时间

Table 2. The fusion time of different algorithms in six different scenarios s

Algorithm	Scenes of fusion image
Algorithm	First	Second	Third	Fourth	Fifth	Sixth
PCA	4.603	1.515	4.744	3.646	2.564	2.464
NSCT-PCNN	10.423	5.305	4.459	4.482	4.417	4.667
NSST-PCNN	4.319	2.851	2.917	2.902	3.819	2.301
PCNN-IFS	5.429	1.226	3.194	2.631	2.197	2.903
NSST-FCM	6.061	4.102	5.269	3.462	4.556	4.321
NSST-PA-PCNN	3.589	3.452	3.454	4.839	3.543	3.455
Proposed Algorithm	4.063	2.091	2.658	2.286	2.981	2.2136

下载: 导出CSV

参考文献(20)

[1]	WANG Z G, WEI W, SU B. Multi-sensor Image fusion algorithm based on multiresolution analysis[J]. International Journal of Online Engineering, 2018, 14(6): 44. doi: 10.3991/ijoe.v14i06.8697
[2]	SHEN Yu, CHEN Xiaopeng, YUAN Yubin, et al. Infrared and visible image fusion based on significant matrix and neural network[J]. Laser & Optoelectronics Progress, 2020, 57(20): 56.
[3]	González José R, Damião Charbel, Moran Maira, et al. A computational study on the role of parameters for identification of thyroid nodules by infrared images (and comparison with real data) [J]. Sensors, 2021, 21(13): 4459. doi: 10.3390/s21134459
[4]	XING Xiaoxue, LIU Cheng, LUO Cong, et al. Infrared and visible image fusion based on nonlinear enhancement and NSST decomposition[J]. EURASIP Journal on Wireless Communications and Networking, 2020(1): 125.
[5]	ZHANG Kang, HUANG Yongdong, YUAN Xia, et al. Infrared and visible image fusion based on intuitionistic fuzzy sets[J]. Infrared Physics and Technology, 2020(4): 85.
[6]	LIU Zhanwen, FENG Yan, CHEN Hang, et al. A fusion algorithm for infrared and visible based on guided filtering and phase congruency in NSST domain[J]. Optics and Lasers in Engineering, 2017, 97(6): 23.
[7]	王丽丽, 刘辉, 郭肇禄. 图像融合在域内的算法研究进展[J]. 计算机技术与发展, 2021, 31(11): 8-14. https://www.cnki.com.cn/Article/CJFDTOTAL-WJFZ202111002.htm WANG Lili, LIU Hui, GUO Zhaolu. Research progress of image fusion algorithm in domain [J]. Computer Technology and Development, 2021, 31(11): 8-14. https://www.cnki.com.cn/Article/CJFDTOTAL-WJFZ202111002.htm
[8]	徐慧娴, 黄坤超, 陈明举, 等. 基于深层次多分类生成对抗网络的红外与可见光图像融合方法[J/OL]. 电子测量技术, 2022, 3: 1-10. XU Huixian, HUANG Kunchao, CHEN Mingju, et al. Infrared and visible image fusion method based on deep multi classification to generate countermeasure network [J/OL]. Electronic Measurement Technology, 2022, 3: 1-10
[9]	许丽娜, 周丹丹, 高翾, 等. 一种结合GIHS、NSST和PCA的全色多光谱影像融合方法[J/OL]. 武汉大学学报: 信息科学版, 2022, 3: 1-17. XU Lina, ZHOU Dandan, GAO Zhen, et al. A panchromatic multispectral image fusion method combining GHIS, NSST and PCA [J/OL]. Journal of Wuhan University: Information Science Edition, 2022, 3: 1-17.
[10]	李向阳, 曹宇彤, 陈笑, 等. 基于自适应NSST-PCNN的红外与可见光图像融合方法研究[J]. 长春理工大学学报: 自然科学版, 2021, 44(5): 12-18. doi: 10.3969/j.issn.1672-9870.2021.05.003 LI Xiangyang, CAO Yutong, CHEN Xiao, et al. Research on infrared and visible image fusion method based on adaptive nsst-pcnn [J]. Journal of Changchun University of Technology: Natural Science Edition, 2021, 44(5): 12-18. doi: 10.3969/j.issn.1672-9870.2021.05.003
[11]	SHEN Yu, YUAN Yubin, PENG Jing. Research on near infrared and color visible fusion based on PCNN in transform domain[J]. Spectroscopy and Spectral Analysis, 2021, 41(7): 65.
[12]	刘帅, 王磊, 郝永平, 等. 基于NSCT与改进PCNN的红外与可见光图像融合方法研究[J]. 光电技术应用, 2021, 36(4): 60-65. https://www.cnki.com.cn/Article/CJFDTOTAL-GDYG202104017.htm LIU Shuai, WANG Lei, HAO Yongping, et al. Research on infrared and visible image fusion method based on NSCT and improved PCNN [J]. Optoelectronic Technology Application, 2021, 36(4): 60-65. https://www.cnki.com.cn/Article/CJFDTOTAL-GDYG202104017.htm
[13]	戴文战, 王琪. 基于PCNN与IFS的可见光与红外图像融合方法[J]. 光电子·激光, 2020, 31(7): 738-744. https://www.cnki.com.cn/Article/CJFDTOTAL-GDZJ202007012.htm DAI Wenzhan, WANG Qi. Visible and infrared image fusion method based on PCNN and IFS [J]. Optoelectronics and Laser, 2020, 31(7): 738-744. https://www.cnki.com.cn/Article/CJFDTOTAL-GDZJ202007012.htm
[14]	WU Chenwen, MA Ning, JIANG Yufan. Weighted FCM clustering algorithm based on jeffrey divergence similarity measure[J]. Laser & Optoelectronics Progress, 2021, 58(8): 67
[15]	巩稼民, 刘爱萍, 张晨, 等. 基于FCM与ADSCM的红外与可见光图像融合[J]. 激光与光电子学进展, 2020, 57(20): 222-230. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202020025.htm GONG Jiamin, LIU Aiping, ZHANG Chen, et al. Infrared and visible image fusion based on FCM and adscm[J]. Progress in Laser and Optoelectronics, 2020, 57(20): 222-230. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202020025.htm
[16]	ZHANG Hongna, YAN Wei, ZHANG Chunyou, et al. Research on image fusion algorithm based on NSST frequency division and improved LSCN[J]. Mobile Networks and Applications, 2021, 26(5): 53.
[17]	GUO K, Labate D. Optimally sparse multidimensional representation using shearlets[J]. SIAM Journal on Mathematical Analysis, 2007, 39(1): 298-318.
[18]	Easley G, Labate D, LIM W Q. Sparse directional image representation using discrete shearlet transform[J]. Applied and Computation Harmonic Analysis, 2008, 25(1): 25-46.
[19]	YAN Peizhou, ZOU Jiancheng, LI Zhengzheng, et al. Infrared and visible image fusion based on NSST and RDN[J]. Intelligent Automation & Soft Computing, 2021, 28(1): 35. http://www.xueshufan.com/publication/3147917438
[20]	TAN Wei, ZHANG Jiajia, XIANG Pei, et al. Infrared and visible image fusion via NSST and PCNN in multiscale morphological gradient domain[J]. Optics, Photonics And Digital Technologies For Imaging Applications Ⅵ, 2021, 11353: 54.