Infrared and Visible Image Fusion Based on Multi-scale and Attention Model
-
摘要: 针对红外与可见光图像在融合后容易出现伪影，小目标轮廓不清晰等问题，提出一种基于多尺度特征与注意力模型相结合的红外与可见光图像融合算法。通过5次下采样提取源图像不同尺度的特征图，再将同一尺度的红外与可见光特征图输入到基于注意力模型的融合层，获得增强的融合特征图。最后把小尺度的融合特征图进行5次上采样，再与上采样后同一尺度的特征图相加，直到与源图像尺度一致，实现对特征图的多尺度融合。实验对比不同融合框架下融合图像的熵、标准差、互信息量、边缘保持度、小波特征互信息、视觉信息保真度以及融合效率，本文方法在多数指标上优于对比算法，且融合图像目标细节明显轮廓清晰。Abstract: Aiming at the problems that infrared and visible images are prone to artifacts and unclear outlines of small targets after fusion, an infrared and visible images fusion algorithm based on the combination of multi-scale features and attention model is proposed. The feature maps of different scales of the source image are extracted through five times of down-sampling, and then the infrared and visible image feature maps of the same scale are input to the fusion layer based on the attention model to obtain an enhanced fusion feature map. Finally, the small-scale fusion feature map is up-sampled five times, and then added to the feature map of the same scale after up-sampling, until the scale is consistent with the source image, and the multi-scale fusion of the feature map is realized. Experiments compare the entropy, standard deviation, mutual information, edge retention, wavelet feature mutual information, visual information fidelity, and fusion efficiency of fused images under different fusion frameworks. The method in this paper is superior to the comparison algorithm in most indicators, and the target details are obvious and the outline are clear in the fused images.
-
Keywords:
- image fusion /
- multi-scale feature fusion /
- attention model /
- infrared images
-
红外无损检测技术是一门跨学科、跨应用领域的创新性无损检测技术，具有非接触、检测速度快、检测精度与分辨率高、可靠性高等突出优点，已被广泛应用于航空、航天、风电、石化、电力等领域的工业材料与装备检测。近年来，人工智能、计算机科学、电子信息等科学技术的快速发展，不仅推动红外无损检测技术取得了巨大进步，也促使红外无损检测技术向着多样化、智能化、集成化等方向发展。
为了促进我国红外无损检测技术的创新发展，2023年10期，《红外技术》推出了“红外无损检测新技术”专栏，共收录7篇学术论文，内容涉及红外热成像技术在FRP复合材料热障涂层无损检测应用中的研究现状与进展，超声激励红外热成像研究现状与进展，基于YOLO v5的带涂层钢结构亚表面缺陷脉冲涡流热成像智能检测，基于脉冲红外热成像技术的锂电池端盖焊接质量检测，线激光扫描热成像无损检测参数仿真研究，滚动轴承红外热成像故障诊断与状态监测等，涉及内容广泛。旨在集中反映报道红外无损检测技术的最新动态和发展趋势，为我国相关科研人员和广大读者提供学术参考，为红外无损检测技术的创新发展提供一些新思路和新手段。
最后，感谢专栏论文所有作者和各位审稿专家的卓越贡献。
——郑凯 -
![]()
图 2 模型训练损失曲线图：（左）结构相似性损失（右）像素损失
Figure 2. The loss curve during model training: (Left) SSIM loss; (Right): Pixel loss
![]()
图 3 “房子”图像不同算法融合结果
Figure 3. The fusion results of the same image of different fusion algorithms
表 1 编码网络和解码网络的设置
Table 1 The setting of encoder and decoder networks
Layer Size Stride Channel(input) Channel(output) Encoder C1 3 1 1 16 ECB10 - - 16 64 ECB20 - - 64 112 ECB30 - - 112 160 ECB40 - - 160 208 ECB50 - - 208 256 Decoder DCB41 - - 464 208 DCB31 - - 368 160 DCB21 - - 272 112 DCB11 - - 176 64 C2 1 1 64 1 ECB Conv 3 1 Nin 16 Conv 1 1 16 Nout max-pooling - - - - DCB conv 3 1 Nin 16 conv 1 1 16 Nout 
下载: 导出CSV
表 2 不同融合策略下融合图像质量评价均值
Table 2 The mean value of image quality evalution under different fusion strategies
Method EN SD MI Qab/f FMI_w VIF DenseFuse add 6.8558 35.6741 13.7116 0.3987 0.3567 0.6756 ours 7.0173 42.5361 14.0346 0.4361 0.3651 0.8019 FPNFuse add 6.8312 36.6245 13.6625 0.463 0.4184 0.6818 ours 7.0672 44.5546 14.13447 0.5181 0.4394 0.8263
下载: 导出CSV
表 3 不同算法融合图像质量度量均值
Table 3 The mean value of image quality evalution under different fusion algorithms
Method EN SD MI Qab/f FMI_w VIF AT/s WLS 6.6861 34.4462 13.3723 0.5210 0.3630 0.6656 1.1688 DeepFuse add 6.8135 36.9112 13.6270 0.4536 0.4150 0.6908 0.2916 DenseFuse add 6.8558 35.6741 13.7116 0.3987 0.3567 0.6756 0.4611 avg 7.0173 42.5361 14.0346 0.4536 0.3651 0.8019 0.5237 Ours max 7.0327 43.3592 14.0655 0.5173 0.4338 0.7894 0.0298 avg 7.0672 44.5546 14.1345 0.5181 0.4394 0.8263 0.0248 nuclear 7.0576 44.6828 14.1152 0.5212 0.4374 0.8044 0.0335
下载: 导出CSV
-
[1] 赵立昌, 张宝辉, 吴杰, 等. 基于灰度能量差异性的红外与可见光图像融合[J]. 红外技术, 2020, 42(8): 775-782. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS202008012.htm ZHAO Lichang, ZHANG Baohui, WU Jie, et al. Fusion of infrared and visible images based on gray energy difference[J]. Infrared Technology, 2020, 42(8): 775-782. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS202008012.htm
[2] 白玉, 侯志强, 刘晓义, 等. 基于可见光图像和红外图像决策级融合的目标检测算法[J]. 空军工程大学学报(自然科学版), 2020, 21(6): 53-59, 100. DOI: 10.3969/j.issn.1009-3516.2020.06.009 BAI Yu, HOU Zhiqiang, LIU Xiaoyi, et al. An object detection algorithm based on decision-level fusion of visible light image and infrared image[J]. Journal of Air Force Engineering University(Natural Science Editon), 2020, 21(6): 53-59, 100. DOI: 10.3969/j.issn.1009-3516.2020.06.009
[3] 董安勇, 杜庆治, 苏斌, 等. 基于卷积神经网络的红外与可见光图像融合[J]. 红外技术, 2020, 42(7): 660-669. http://hwjs.nvir.cn/article/id/hwjs202007009 DONG Anyong, DU Qingzhi, SU Bin, et al. Infrared and visible image fusion based on convolutional neural network[J]. Infrared Technology, 2020, 42(7): 660-669. http://hwjs.nvir.cn/article/id/hwjs202007009
[4] 陈卓, 方明, 柴旭, 等. 红外与可见光图像融合的U-GAN模型[J]. 西北工业大学学报, 2020, 38(4): 904-912. DOI: 10.3969/j.issn.1000-2758.2020.04.027 CHEN Zhuo, FANG Ming, CHAI Xu, et al. Infrared and visible image fusion of U-GAN model[J]. Journal of Northwestern Polytechnical University, 2020, 38(4): 904-912. DOI: 10.3969/j.issn.1000-2758.2020.04.027
[5] 陈潮起, 孟祥超, 邵枫, 等. 一种基于多尺度低秩分解的红外与可见光图像融合方法[J]. 光学学报, 2020, 40(11): 72-80. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB202011008.htm CHEN Chaoqi, MENG Xiangchao, SHAO Feng, et al. Infrared and visible image fusion method based on multiscale low-rank decomposition [J]. Acta Optica Sinica, 2020, 40(11): 72-80. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB202011008.htm
[6] 林子慧. 基于多尺度变换的红外与可见光图像融合技术研究[D]. 成都: 中国科学院大学(中国科学院光电技术研究所), 2019. LIN Zihui. Research on Infrared and Visible Image Fusion Based on Multi-scale Trandform[D]. Chengdu: The Chinese Academy of Sciences(The Institute of Optics and Electronics), 2019.
[7] 马旗, 朱斌, 张宏伟. 基于VGG网络的双波段图像融合方法[J]. 激光与红外, 2019, 49(11): 1374-1380. DOI: 10.3969/j.issn.1001-5078.2019.11.018 MA Qi, ZHU Bin, ZHANG Hongwei. Dual-band image fusion method based on VGGNet[J]. Laser & Infrared, 2019, 49(11): 1374-1380. DOI: 10.3969/j.issn.1001-5078.2019.11.018
[8] LI H, WU X, Durrani T S. Infrared and visible image fusion with ResNet and zero-phase component analysis[J]. Infrared Physics & Technology, 2019, 102: 103039.
[9] LIN T, Dollár P, Girshick R, et al. Feature pyramid networks for object detection[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017: 936-944.
[10] Woo S, Park J, Lee J, et al. CBAM: Convolution-al Block Attention Module[C]//ECCV, 2018: 3-19.
[11] LI H, WU X J, Durrani T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models [J]. IEEE Transactions on Instrumentation and Measurement, 2020, 12(69): 9645-9656.
[12] 杨艳春, 李娇, 王阳萍. 图像融合质量评价方法研究综述[J]. 计算机科学与探索, 2018, 12(7): 1021-1035. https://www.cnki.com.cn/Article/CJFDTOTAL-KXTS201807002.htm YANG Yanchun, LI Jiao, WANG Yangping. Review of image fusion quality evaluation methods[J]. Journal of Frontiers of Computer Science and Technology, 2018, 12(7): 1021-1035. https://www.cnki.com.cn/Article/CJFDTOTAL-KXTS201807002.htm
[13] LIN T Y, Maire M, Belongie S, et al. Microsoft coco: common objects in context[C]//ECCV, 2014: 3-5.
[14] Toet A. TNO Image Fusion Dataset. figshare. Dataset[DB/OL]. https://doi.org/10.6084/m9.figshare.1008029.v2, 2014.
[15] LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Trans. Image Process, 2019, 28(5): 2614-2623.
[16] Prabhakar K R, Srikar V S, Babu R V. DeepFuse: a deep unsuper-vised approach for exposure fusion with extreme exposure image pairs[C]//2017 IEEE International Conference on Computer Vision (ICCV), 2017: 4724-4732.
[17] MA J, ZHOU Z, WANG B. et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics & Technology, 2017, 82: 8-17.
计量
- 文章访问数: 176
- HTML全文浏览量: 109
- PDF下载量: 59
