Infrared and Visible Image Fusion Based on Multi-scale and Attention Model
-
摘要: 针对红外与可见光图像在融合后容易出现伪影,小目标轮廓不清晰等问题,提出一种基于多尺度特征与注意力模型相结合的红外与可见光图像融合算法。通过5次下采样提取源图像不同尺度的特征图,再将同一尺度的红外与可见光特征图输入到基于注意力模型的融合层,获得增强的融合特征图。最后把小尺度的融合特征图进行5次上采样,再与上采样后同一尺度的特征图相加,直到与源图像尺度一致,实现对特征图的多尺度融合。实验对比不同融合框架下融合图像的熵、标准差、互信息量、边缘保持度、小波特征互信息、视觉信息保真度以及融合效率,本文方法在多数指标上优于对比算法,且融合图像目标细节明显轮廓清晰。Abstract: Aiming at the problems that infrared and visible images are prone to artifacts and unclear outlines of small targets after fusion, an infrared and visible images fusion algorithm based on the combination of multi-scale features and attention model is proposed. The feature maps of different scales of the source image are extracted through five times of down-sampling, and then the infrared and visible image feature maps of the same scale are input to the fusion layer based on the attention model to obtain an enhanced fusion feature map. Finally, the small-scale fusion feature map is up-sampled five times, and then added to the feature map of the same scale after up-sampling, until the scale is consistent with the source image, and the multi-scale fusion of the feature map is realized. Experiments compare the entropy, standard deviation, mutual information, edge retention, wavelet feature mutual information, visual information fidelity, and fusion efficiency of fused images under different fusion frameworks. The method in this paper is superior to the comparison algorithm in most indicators, and the target details are obvious and the outline are clear in the fused images.
-
Keywords:
- image fusion /
- multi-scale feature fusion /
- attention model /
- infrared images
-
-
表 1 编码网络和解码网络的设置
Table 1 The setting of encoder and decoder networks
Layer Size Stride Channel(input) Channel(output) Encoder C1 3 1 1 16 ECB10 - - 16 64 ECB20 - - 64 112 ECB30 - - 112 160 ECB40 - - 160 208 ECB50 - - 208 256 Decoder DCB41 - - 464 208 DCB31 - - 368 160 DCB21 - - 272 112 DCB11 - - 176 64 C2 1 1 64 1 ECB Conv 3 1 Nin 16 Conv 1 1 16 Nout max-pooling - - - - DCB conv 3 1 Nin 16 conv 1 1 16 Nout 表 2 不同融合策略下融合图像质量评价均值
Table 2 The mean value of image quality evalution under different fusion strategies
Method EN SD MI Qab/f FMI_w VIF DenseFuse add 6.8558 35.6741 13.7116 0.3987 0.3567 0.6756 ours 7.0173 42.5361 14.0346 0.4361 0.3651 0.8019 FPNFuse add 6.8312 36.6245 13.6625 0.463 0.4184 0.6818 ours 7.0672 44.5546 14.13447 0.5181 0.4394 0.8263 表 3 不同算法融合图像质量度量均值
Table 3 The mean value of image quality evalution under different fusion algorithms
Method EN SD MI Qab/f FMI_w VIF AT/s WLS 6.6861 34.4462 13.3723 0.5210 0.3630 0.6656 1.1688 DeepFuse add 6.8135 36.9112 13.6270 0.4536 0.4150 0.6908 0.2916 DenseFuse add 6.8558 35.6741 13.7116 0.3987 0.3567 0.6756 0.4611 avg 7.0173 42.5361 14.0346 0.4536 0.3651 0.8019 0.5237 Ours max 7.0327 43.3592 14.0655 0.5173 0.4338 0.7894 0.0298 avg 7.0672 44.5546 14.1345 0.5181 0.4394 0.8263 0.0248 nuclear 7.0576 44.6828 14.1152 0.5212 0.4374 0.8044 0.0335 -
[1] 赵立昌, 张宝辉, 吴杰, 等. 基于灰度能量差异性的红外与可见光图像融合[J]. 红外技术, 2020, 42(8): 775-782. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS202008012.htm ZHAO Lichang, ZHANG Baohui, WU Jie, et al. Fusion of infrared and visible images based on gray energy difference[J]. Infrared Technology, 2020, 42(8): 775-782. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS202008012.htm
[2] 白玉, 侯志强, 刘晓义, 等. 基于可见光图像和红外图像决策级融合的目标检测算法[J]. 空军工程大学学报(自然科学版), 2020, 21(6): 53-59, 100. DOI: 10.3969/j.issn.1009-3516.2020.06.009 BAI Yu, HOU Zhiqiang, LIU Xiaoyi, et al. An object detection algorithm based on decision-level fusion of visible light image and infrared image[J]. Journal of Air Force Engineering University(Natural Science Editon), 2020, 21(6): 53-59, 100. DOI: 10.3969/j.issn.1009-3516.2020.06.009
[3] 董安勇, 杜庆治, 苏斌, 等. 基于卷积神经网络的红外与可见光图像融合[J]. 红外技术, 2020, 42(7): 660-669. http://hwjs.nvir.cn/article/id/hwjs202007009 DONG Anyong, DU Qingzhi, SU Bin, et al. Infrared and visible image fusion based on convolutional neural network[J]. Infrared Technology, 2020, 42(7): 660-669. http://hwjs.nvir.cn/article/id/hwjs202007009
[4] 陈卓, 方明, 柴旭, 等. 红外与可见光图像融合的U-GAN模型[J]. 西北工业大学学报, 2020, 38(4): 904-912. DOI: 10.3969/j.issn.1000-2758.2020.04.027 CHEN Zhuo, FANG Ming, CHAI Xu, et al. Infrared and visible image fusion of U-GAN model[J]. Journal of Northwestern Polytechnical University, 2020, 38(4): 904-912. DOI: 10.3969/j.issn.1000-2758.2020.04.027
[5] 陈潮起, 孟祥超, 邵枫, 等. 一种基于多尺度低秩分解的红外与可见光图像融合方法[J]. 光学学报, 2020, 40(11): 72-80. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB202011008.htm CHEN Chaoqi, MENG Xiangchao, SHAO Feng, et al. Infrared and visible image fusion method based on multiscale low-rank decomposition [J]. Acta Optica Sinica, 2020, 40(11): 72-80. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB202011008.htm
[6] 林子慧. 基于多尺度变换的红外与可见光图像融合技术研究[D]. 成都: 中国科学院大学(中国科学院光电技术研究所), 2019. LIN Zihui. Research on Infrared and Visible Image Fusion Based on Multi-scale Trandform[D]. Chengdu: The Chinese Academy of Sciences(The Institute of Optics and Electronics), 2019.
[7] 马旗, 朱斌, 张宏伟. 基于VGG网络的双波段图像融合方法[J]. 激光与红外, 2019, 49(11): 1374-1380. DOI: 10.3969/j.issn.1001-5078.2019.11.018 MA Qi, ZHU Bin, ZHANG Hongwei. Dual-band image fusion method based on VGGNet[J]. Laser & Infrared, 2019, 49(11): 1374-1380. DOI: 10.3969/j.issn.1001-5078.2019.11.018
[8] LI H, WU X, Durrani T S. Infrared and visible image fusion with ResNet and zero-phase component analysis[J]. Infrared Physics & Technology, 2019, 102: 103039.
[9] LIN T, Dollár P, Girshick R, et al. Feature pyramid networks for object detection[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017: 936-944.
[10] Woo S, Park J, Lee J, et al. CBAM: Convolution-al Block Attention Module[C]//ECCV, 2018: 3-19.
[11] LI H, WU X J, Durrani T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models [J]. IEEE Transactions on Instrumentation and Measurement, 2020, 12(69): 9645-9656.
[12] 杨艳春, 李娇, 王阳萍. 图像融合质量评价方法研究综述[J]. 计算机科学与探索, 2018, 12(7): 1021-1035. https://www.cnki.com.cn/Article/CJFDTOTAL-KXTS201807002.htm YANG Yanchun, LI Jiao, WANG Yangping. Review of image fusion quality evaluation methods[J]. Journal of Frontiers of Computer Science and Technology, 2018, 12(7): 1021-1035. https://www.cnki.com.cn/Article/CJFDTOTAL-KXTS201807002.htm
[13] LIN T Y, Maire M, Belongie S, et al. Microsoft coco: common objects in context[C]//ECCV, 2014: 3-5.
[14] Toet A. TNO Image Fusion Dataset. figshare. Dataset[DB/OL]. https://doi.org/10.6084/m9.figshare.1008029.v2, 2014.
[15] LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Trans. Image Process, 2019, 28(5): 2614-2623.
[16] Prabhakar K R, Srikar V S, Babu R V. DeepFuse: a deep unsuper-vised approach for exposure fusion with extreme exposure image pairs[C]//2017 IEEE International Conference on Computer Vision (ICCV), 2017: 4724-4732.
[17] MA J, ZHOU Z, WANG B. et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics & Technology, 2017, 82: 8-17.
-
期刊类型引用(20)
1. 邱祥彪,杨晓明,孙建宁,王健,丛晓庆,金戈,曾进能,张正君,潘凯,陈晓倩. 高空间分辨微通道板现状及发展. 红外技术. 2024(04): 460-466 . 本站查看
2. 刘峰阁,苏天宁,刘倍宏,成帅,朱荣胜,姬明,肖杰,赵航,张理淞,常乐. 基于离散系数与Harris角点的微光像增强器闪烁噪声测试方法. 红外技术. 2024(10): 1154-1161 . 本站查看
3. 马俊卉,王岭雪,胡婧玲,郑德智,闻浩诚,王兵,蔡毅. 紫外像增强器在航空发动机燃烧诊断中的应用. 红外技术. 2024(10): 1107-1118 . 本站查看
4. 王子薇,于逸凡,范兴玥,赵树伟. 微光像增强器低照度分辨力分析. 光电子技术. 2024(04): 350-354 . 百度学术
5. 杨敏杰,钱芸生,严毅赟,吴胜. 基于CMOS图像传感器的像增强器闪烁噪声测试系统. 激光与光电子学进展. 2023(02): 325-330 . 百度学术
6. 冯丹青,郭欣达,拜晓锋,张琴,党小刚,张姝丽,杨书宁,李琦,韩坤. 三代微光像增强器亮度增益对像质影响的分析. 红外技术. 2023(02): 188-194 . 本站查看
7. 刘宇,时荔蕙. 像增强器性能梯次及发展路线研究. 红外与毫米波学报. 2023(04): 427-433 . 百度学术
8. 孙磊,金东东,纪春恒,裴崇雷,安鸿波,段恩悦. 基于增强型CCD探测器的距离选通三维成像不均匀性补偿方法. 兵工学报. 2023(08): 2495-2502 . 百度学术
9. 司俊杰,曾伟明,孔渊,庄迁政,宋文明. 新一代飞行头盔夜视用传感器分析. 半导体光电. 2023(06): 817-826 . 百度学术
10. 赵静,冯琤,覃翠,郭婧. AlGaAs基光电阴极光学性质计算模型分析. 南京工程学院学报(自然科学版). 2023(03): 14-20 . 百度学术
11. 李晓峰,杜木林,徐传平,黄丽书,陈俊宇,常乐. 影响超二代像增强器最高增益的因数分析. 光子学报. 2022(03): 1-12 . 百度学术
12. 张琴,拜晓锋,程宏昌,焦岗成,李周奎,韩坤,李琦. 基于夜天光光谱匹配的微光像增强器信噪比研究. 光子学报. 2022(03): 173-181 . 百度学术
13. 李亚情,周盛涛,王光凡,褚祝军,杜培德,朱文锦,李晓露,左加宁,朱世聪. 普通高压电源超二代微光像增强器亮度增益温度特性研究. 红外技术. 2022(08): 804-810 . 本站查看
14. 李晓峰,何雁彬,常乐,王光凡,徐传平. 超二代与三代像增强器性能的比较研究. 红外技术. 2022(08): 764-777 . 本站查看
15. 俞兵. 国防光学计量技术的发展与展望. 应用光学. 2022(04): 565-576 . 百度学术
16. 周二瑞,李刚,杨少华,严明,李斌康,郭明安,刘璐,白琼,张雪莹. 高时间分辨sCMOS成像系统. 现代应用物理. 2022(03): 23-29 . 百度学术
17. 李晓峰,何雁彬,徐传平,李金沙,张勤东. 欧洲超二代像增强器技术的选择及进一步发展. 红外技术. 2022(12): 1249-1263 . 本站查看
18. 李晓峰,常乐,赵恒,邱永生,陈俊宇,张彦云. 超二代与三代像增强器低照度分辨力的比较. 光子学报. 2021(09): 268-275 . 百度学术
19. 李晓峰,赵恒,张彦云,张勤东. 高性能超二代像增强器及发展. 红外技术. 2021(09): 811-816 . 本站查看
20. 程宏昌,石峰,李周奎,师宏立,拜晓锋,闫磊,姚泽,王淼鑫,李燕红. 微光夜视器件划代方法初探. 应用光学. 2021(06): 1092-1101 . 百度学术
其他类型引用(7)