基于多尺度和注意力模型的红外与可见光图像融合

黄玲琳; 李强; 路锦正; 贺贤珍; 彭波

基于多尺度和注意力模型的红外与可见光图像融合

黄玲琳^1,,
李强^{1, 2, ,},
路锦正¹,
贺贤珍¹,
彭波^{1, 2}

1.
西南科技大学信息工程学院, 四川绵阳 621010
2.
西南科技大学特殊环境机器人技术四川省重点实验室, 四川绵阳 621010

基金项目:

国家重点研发计划项目 2019YFB1705100

四川省科技计划项目 2019JDTD0019

详细信息

作者简介:
黄玲琳（1997-），女，硕士，主要从事图像融合等图像处理技术的研究。E-mail：1551769402@qq.com

通讯作者:
李强（1982-），男，博士，教授，主要从事物联网与智能信息处理的研究。E-mail：liqiangsir@swust.edu.cn

中图分类号: TP751.1
计量
- 文章访问数: 139
- HTML全文浏览量: 75
- PDF下载量: 47
- 被引次数: 0
出版历程
- 收稿日期: 2021-02-25
- 修回日期: 2021-04-12
- 刊出日期: 2023-02-20

Infrared and Visible Image Fusion Based on Multi-scale and Attention Model

HUANG Linglin^1
,,
LI Qiang^{1, 2
, ,},
LU Jinzheng¹,
HE Xianzhen¹,
PENG Bo^{1, 2}

1.
School of Information Engineering, Southwest University of Science and Technology, Mianyang 621010, China
2.
Robot Technology Used for Special Environment Key Laboratory of Sichuan Province, Southwest University of Science and Technology, Mianyang 621010, China

摘要

摘要: 针对红外与可见光图像在融合后容易出现伪影，小目标轮廓不清晰等问题，提出一种基于多尺度特征与注意力模型相结合的红外与可见光图像融合算法。通过5次下采样提取源图像不同尺度的特征图，再将同一尺度的红外与可见光特征图输入到基于注意力模型的融合层，获得增强的融合特征图。最后把小尺度的融合特征图进行5次上采样，再与上采样后同一尺度的特征图相加，直到与源图像尺度一致，实现对特征图的多尺度融合。实验对比不同融合框架下融合图像的熵、标准差、互信息量、边缘保持度、小波特征互信息、视觉信息保真度以及融合效率，本文方法在多数指标上优于对比算法，且融合图像目标细节明显轮廓清晰。
- 图像融合 /
- 多尺度特征融合 /
- 注意力模型 /
- 红外图像
Abstract: Aiming at the problems that infrared and visible images are prone to artifacts and unclear outlines of small targets after fusion, an infrared and visible images fusion algorithm based on the combination of multi-scale features and attention model is proposed. The feature maps of different scales of the source image are extracted through five times of down-sampling, and then the infrared and visible image feature maps of the same scale are input to the fusion layer based on the attention model to obtain an enhanced fusion feature map. Finally, the small-scale fusion feature map is up-sampled five times, and then added to the feature map of the same scale after up-sampling, until the scale is consistent with the source image, and the multi-scale fusion of the feature map is realized. Experiments compare the entropy, standard deviation, mutual information, edge retention, wavelet feature mutual information, visual information fidelity, and fusion efficiency of fused images under different fusion frameworks. The method in this paper is superior to the comparison algorithm in most indicators, and the target details are obvious and the outline are clear in the fused images.
- image fusion /
- multi-scale feature fusion /
- attention model /
- infrared images

HTML全文

图 1 多尺度融合框架FPNFuse

Figure 1. Multi-scale image fusion framework FPNFuse

下载: 全尺寸图片幻灯片

图 2 模型训练损失曲线图：（左）结构相似性损失（右）像素损失

Figure 2. The loss curve during model training: (Left) SSIM loss; (Right): Pixel loss

下载: 全尺寸图片幻灯片

图 3 “房子”图像不同算法融合结果

Figure 3. The fusion results of the same image of different fusion algorithms

下载: 全尺寸图片幻灯片

图 4 近红外与热红外的融合图像

Figure 4. The fusion image of near and thermal infrared images

下载: 全尺寸图片幻灯片

表 1 编码网络和解码网络的设置

Table 1. The setting of encoder and decoder networks

	Layer	Size	Stride	Channel(input)	Channel(output)
Encoder	C1	3	1	1	16
	ECB10	-	-	16	64
	ECB20	-	-	64	112
	ECB30	-	-	112	160
	ECB40	-	-	160	208
	ECB50	-	-	208	256
Decoder	DCB41	-	-	464	208
	DCB31	-	-	368	160
	DCB21	-	-	272	112
	DCB11	-	-	176	64
	C2	1	1	64	1
ECB	Conv	3	1	N_in	16
	Conv	1	1	16	N_out
	max-pooling	-	-	-	-
DCB	conv	3	1	N_in	16
DCB	conv	1	1	16	N_out

下载: 导出CSV

表 2 不同融合策略下融合图像质量评价均值

Table 2. The mean value of image quality evalution under different fusion strategies

Method		EN	SD	MI	Q_ab/f	FMI_w	VIF
DenseFuse	add	6.8558	35.6741	13.7116	0.3987	0.3567	0.6756
DenseFuse	ours	7.0173	42.5361	14.0346	0.4361	0.3651	0.8019
FPNFuse	add	6.8312	36.6245	13.6625	0.463	0.4184	0.6818
FPNFuse	ours	7.0672	44.5546	14.13447	0.5181	0.4394	0.8263

下载: 导出CSV

表 3 不同算法融合图像质量度量均值

Table 3. The mean value of image quality evalution under different fusion algorithms

Method		EN	SD	MI	Q_ab/f	FMI_w	VIF	AT/s
WLS		6.6861	34.4462	13.3723	0.5210	0.3630	0.6656	1.1688
DeepFuse	add	6.8135	36.9112	13.6270	0.4536	0.4150	0.6908	0.2916
DenseFuse	add	6.8558	35.6741	13.7116	0.3987	0.3567	0.6756	0.4611
DenseFuse	avg	7.0173	42.5361	14.0346	0.4536	0.3651	0.8019	0.5237
Ours	max	7.0327	43.3592	14.0655	0.5173	0.4338	0.7894	0.0298
	avg	7.0672	44.5546	14.1345	0.5181	0.4394	0.8263	0.0248
	nuclear	7.0576	44.6828	14.1152	0.5212	0.4374	0.8044	0.0335

下载: 导出CSV

参考文献(17)

[1]	赵立昌, 张宝辉, 吴杰, 等. 基于灰度能量差异性的红外与可见光图像融合[J]. 红外技术, 2020, 42(8): 775-782. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS202008012.htm ZHAO Lichang, ZHANG Baohui, WU Jie, et al. Fusion of infrared and visible images based on gray energy difference[J]. Infrared Technology, 2020, 42(8): 775-782. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS202008012.htm
[2]	白玉, 侯志强, 刘晓义, 等. 基于可见光图像和红外图像决策级融合的目标检测算法[J]. 空军工程大学学报(自然科学版), 2020, 21(6): 53-59, 100. doi: 10.3969/j.issn.1009-3516.2020.06.009 BAI Yu, HOU Zhiqiang, LIU Xiaoyi, et al. An object detection algorithm based on decision-level fusion of visible light image and infrared image[J]. Journal of Air Force Engineering University(Natural Science Editon), 2020, 21(6): 53-59, 100. doi: 10.3969/j.issn.1009-3516.2020.06.009
[3]	董安勇, 杜庆治, 苏斌, 等. 基于卷积神经网络的红外与可见光图像融合[J]. 红外技术, 2020, 42(7): 660-669. http://hwjs.nvir.cn/article/id/hwjs202007009 DONG Anyong, DU Qingzhi, SU Bin, et al. Infrared and visible image fusion based on convolutional neural network[J]. Infrared Technology, 2020, 42(7): 660-669. http://hwjs.nvir.cn/article/id/hwjs202007009
[4]	陈卓, 方明, 柴旭, 等. 红外与可见光图像融合的U-GAN模型[J]. 西北工业大学学报, 2020, 38(4): 904-912. doi: 10.3969/j.issn.1000-2758.2020.04.027 CHEN Zhuo, FANG Ming, CHAI Xu, et al. Infrared and visible image fusion of U-GAN model[J]. Journal of Northwestern Polytechnical University, 2020, 38(4): 904-912. doi: 10.3969/j.issn.1000-2758.2020.04.027
[5]	陈潮起, 孟祥超, 邵枫, 等. 一种基于多尺度低秩分解的红外与可见光图像融合方法[J]. 光学学报, 2020, 40(11): 72-80. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB202011008.htm CHEN Chaoqi, MENG Xiangchao, SHAO Feng, et al. Infrared and visible image fusion method based on multiscale low-rank decomposition [J]. Acta Optica Sinica, 2020, 40(11): 72-80. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB202011008.htm
[6]	林子慧. 基于多尺度变换的红外与可见光图像融合技术研究[D]. 成都: 中国科学院大学(中国科学院光电技术研究所), 2019. LIN Zihui. Research on Infrared and Visible Image Fusion Based on Multi-scale Trandform[D]. Chengdu: The Chinese Academy of Sciences(The Institute of Optics and Electronics), 2019.
[7]	马旗, 朱斌, 张宏伟. 基于VGG网络的双波段图像融合方法[J]. 激光与红外, 2019, 49(11): 1374-1380. doi: 10.3969/j.issn.1001-5078.2019.11.018 MA Qi, ZHU Bin, ZHANG Hongwei. Dual-band image fusion method based on VGGNet[J]. Laser & Infrared, 2019, 49(11): 1374-1380. doi: 10.3969/j.issn.1001-5078.2019.11.018
[8]	LI H, WU X, Durrani T S. Infrared and visible image fusion with ResNet and zero-phase component analysis[J]. Infrared Physics & Technology, 2019, 102: 103039.
[9]	LIN T, Dollár P, Girshick R, et al. Feature pyramid networks for object detection[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017: 936-944.
[10]	Woo S, Park J, Lee J, et al. CBAM: Convolution-al Block Attention Module[C]//ECCV, 2018: 3-19.
[11]	LI H, WU X J, Durrani T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models [J]. IEEE Transactions on Instrumentation and Measurement, 2020, 12(69): 9645-9656.
[12]	杨艳春, 李娇, 王阳萍. 图像融合质量评价方法研究综述[J]. 计算机科学与探索, 2018, 12(7): 1021-1035. https://www.cnki.com.cn/Article/CJFDTOTAL-KXTS201807002.htm YANG Yanchun, LI Jiao, WANG Yangping. Review of image fusion quality evaluation methods[J]. Journal of Frontiers of Computer Science and Technology, 2018, 12(7): 1021-1035. https://www.cnki.com.cn/Article/CJFDTOTAL-KXTS201807002.htm
[13]	LIN T Y, Maire M, Belongie S, et al. Microsoft coco: common objects in context[C]//ECCV, 2014: 3-5.
[14]	Toet A. TNO Image Fusion Dataset. figshare. Dataset[DB/OL]. https://doi.org/10.6084/m9.figshare.1008029.v2, 2014.
[15]	LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Trans. Image Process, 2019, 28(5): 2614-2623.
[16]	Prabhakar K R, Srikar V S, Babu R V. DeepFuse: a deep unsuper-vised approach for exposure fusion with extreme exposure image pairs[C]//2017 IEEE International Conference on Computer Vision (ICCV), 2017: 4724-4732.
[17]	MA J, ZHOU Z, WANG B. et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics & Technology, 2017, 82: 8-17.