融合视觉显著性的红外航拍行人检测

张兴平; 邵延华; 梅艳莹; 张晓强; 楚红雨

融合视觉显著性的红外航拍行人检测

张兴平¹,
邵延华^1, ,,
梅艳莹¹,
张晓强¹,
楚红雨^{1, 2}

1.
西南科技大学信息工程学院, 四川绵阳 621010
2.
西南科技大学四川天府新区创新研究院, 四川成都 610299

基金项目:

国家自然科学基金资助项目 61601382

四川省自然科学基金资助项目 2023NSFSC1388

详细信息

作者简介:
张兴平（1999-），女，四川泸州人，硕士研究生，主要研究方向为目标检测与模型加速

通讯作者:
邵延华（1982-），男，四川绵阳人，博士，副教授，硕导，主要研究方向为计算机视觉与模式识别。E-mail: syh@alu.cqu.edu.cn

中图分类号: TP391
计量
- 文章访问数: 41
- HTML全文浏览量: 12
- PDF下载量: 19
出版历程
- 收稿日期: 2023-01-09
- 修回日期: 2023-03-13
- 刊出日期: 2024-09-19

Aerial Infrared Pedestrian Detection with Saliency Map Fusion

1.
School of Information Engineering, Southwest University of Science & Technology, Mianyang 621010, China
2.
Tianfu Institute of Research and Innovation, Southwest University of Science & Technology, Chengdu 610299, China

摘要

摘要:
目标检测是计算机视觉的基本任务之一，无人机搭载红外相机为夜间侦察、监视等提供便利。针对红外航拍场景检测目标小、图像纹理信息少、对比度弱以及红外目标检测中传统算法精度有限，深度算法依赖算力及功耗不友好等问题，提出了一种融合显著图的红外航拍场景行人检测方法。首先，采用U2-Net从原始热红外图像中提取显著图对原始图像进行增强；其次分析了像素级加权融合和图像通道替换融合两种方式的影响；再次，重聚类先验框，以提高算法对航拍目标场景的适应性。实验结果表明：像素级视觉显著性加权融合效果更优，对典型YOLOv3、YOLOv3-tiny和YOLOv4-tiny平均精度分别提升了6.5%、7.6%和6.2%，表明所设计的融合视觉显著性方法的有效性。
- 红外行人检测 /
- 图像增强 /
- 显著图 /
- YOLOv4
Abstract:
Object detection is a fundamental task in computer vision. Drones equipped with infrared cameras facilitate nighttime reconnaissance and surveillance. To realize small target detection, slight texture information, weak contrast in infrared aerial photography scenes, limited accuracy of traditional algorithms, and heavy dependence on computing power and power consumption in infrared object detection, a pedestrian detection method for infrared aerial photography scenes that integrates salient images is proposed. First, we use U2-Net to generate saliency maps from the original thermal infrared images for image enhancement. Second, we analyze the impact of two fusion methods, pixel-level weighted fusion, and replacement of image channels as image-enhancement schemes. Finally, to improve the adaptability of the algorithm to the target scene, the prior boxes are reclustered. The experimental results show that pixel-level weighted fusion presents excellent results. This method improves the average accuracy of typical YOLOv3, YOLOv3-tiny, and YOLOv4-tiny algorithms by 6.5%, 7.6%, and 6.2%, respectively, demonstrating the effectiveness of the designed fused visual saliency method.
- infrared pedestrian detection /
- salient map /
- image enhancement /
- YOLOv4

HTML全文

图 1 YOLOv4-tiny网络结构

Figure 1. The network structure of YOLOv4-tiny

下载: 全尺寸图片幻灯片

图 2 U2-Net网络结构

Figure 2. The network of U2-Net

下载: 全尺寸图片幻灯片

图 3 RSU-L模块网络

Figure 3. The network of RSU-L module

下载: 全尺寸图片幻灯片

图 4 航拍热红外图像显著图生成

Figure 4. Aerial thermal infrared image salient map generation

下载: 全尺寸图片幻灯片

图 5 显著图替换伪彩色图像RGB通道

Figure 5. Salient maps replacement of infrared image channels

下载: 全尺寸图片幻灯片

图 6 像素级加权融合

Figure 6. The pixel-level weighted fusion

下载: 全尺寸图片幻灯片

图 7 ComNet数据集实例

Figure 7. Instances of ComNet dataset

下载: 全尺寸图片幻灯片

图 8 多场景行人检测推理实例

Figure 8. The inference instances of multi-scene pedestrian detection

下载: 全尺寸图片幻灯片

表 1 K-Means锚框聚类结果

Table 1 The results of anchor box re-clustered

Models	Anchor Box	IOU/%
YOLOv4-tiny	(10, 14) (23, 27) (37, 58) (81, 82) (135, 169) (344, 319)	49.32%
Ours	(14, 39) (21, 69) (26, 95) (38, 85) (33, 113) (45, 130)	82.08%

下载: 导出CSV

表 2 与BASNet^[14]显著图提取方法对比

Table 2 Comparison with BASNet^[14] salient maps generation method

Data fusion method	Salient map extraction method	AP₅₀/%
Data fusion method	Salient map extraction method	YOLOv3	YOLOv4-tiny
Original infrared image	-	88.8	88.6
Salient map	-	77.1	75.6
R channel replaced	BASNet	92.7	91.7
R channel replaced	U2-Net	94.7	93.7
G channel replaced	BASNet	93.8	91.1
G channel replaced	U2-Net	94.2	94.7
B channel replaced	BASNet	90.5	91.5
B channel replaced	U2-Net	92.4	92.9
Weighted fusion	BASNet	94.4	92.0
Weighted fusion	U2-Net	95.3	94.8

下载: 导出CSV

表 3 与先进目标检测器的对比

Table 3 Comparison of advanced object detectors

Model	AP₅₀/%	Model size/MB
IRA-YOLOv3^[23]	84.4	10.7
YOLOv3^[8]	88.8	235
YOLOv3+WF	95.3	235
ZHAO, et al.^[14]	90.3	97
YOLOv3-tiny	86.3	28.1
YOLOv3-tiny+WF	93.9	28.1
YOLOv4-tiny^[1]	88.6	23.1
YOLOv4-tiny+WF	94.8	23.1

下载: 导出CSV

参考文献(23)

[1]	Bochkovskiy Alexey, WANG Chienyao, LIAO Hongyuan. YOLOv4: Optimal speed and accuracy of object detection[EB/OL]. [2020-8-28]. https://arxiv.org/abs/2004.10934.
[2]	顾佼佼, 李炳臻, 刘克, 等. 基于改进Faster R-CNN的红外舰船目标检测算法[J]. 红外技术, 2021, 43(2): 170-178. http://hwjs.nvir.cn/article/id/6dc47229-7cdb-4d62-ae05-6b6909db45b9 GU J J, LI B Z, LIU K, et al. Infrared ship target detection algorithm based on improved faster R-CNN[J]. Infrared Technology, 2021, 43(2): 170-178. http://hwjs.nvir.cn/article/id/6dc47229-7cdb-4d62-ae05-6b6909db45b9
[3]	杨蜀秦, 刘江川, 徐可可, 等. 基于改进CenterNet的玉米雄蕊无人机遥感图像识别[J]. 农业机械学报, 2021, 52(9): 206-212. YANG S Q, LIU J C, XU K K, et al. Remote sensing image recognition of corn stamens based on improved CenterNet for unmanned aerial vehicles[J]. Transactions of the Chinese Society for Agricultural Machinery, 2021, 52(9): 206-212.
[4]	Miezianko Roland, Pokrajac Dragoljub. People detection in low resolution infrared videos [C]//Proc of the 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008: 1-6.
[5]	REN Shaoqing, HE Kaiming, Girshick Ross, et al. Faster R-CNN: towards real-time object detection with region proposal networks [J]. IEEE Trans on Pattern Analysis, 2016, 39(6): 1137-1149.
[6]	Redmon Joseph, Divvala Santosh, Girshick Ross, et al. You only look once: unified, real-time object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 779-788.
[7]	Redmon Joseph, Farhadi Ali. YOLO9000: better, faster, stronger[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 7263-7271.
[8]	Redmon Joseph, Farhadi Ali. Yolov3: An incremental improvement[EB/OL]. [2018-04-08], https://arxiv.org/abs/1804.02767v1.
[9]	LIU Wei, Anguelov Dragomir, Erhan Dumitru, et al. SSD: Single shot multibox detector[C]// Proceedings of the European Conference on Computer Vision, 2017: 21-37.
[10]	LI Chengyang, SONG Dan, TONG Ruofeng, et al. Illumination-aware faster R-CNN for robust multispectral pedestrian detection[J]. Pattern Recognition, 2019, 85: 161-171. DOI: 10.1016/j.patcog.2018.08.005
[11]	仇国庆, 杨海静, 王艳涛, 等. 基于视觉特征融合的机载红外弱小目标检测[J]. 激光与光电子学进展, 2020, 57(18): 79-86. QIU G Q, YANG H J, WANG Y T, et al. Airborne infrared dim small target detection based on visual feature fusion[J]. Laser & Optoelectronics Progress, 2020, 57(18): 79-86.
[12]	李婉蓉, 徐丹, 史金龙, 等. 显著性物体检测研究综述: 方法、应用和趋势[J/OL]. 计算机应用研究, https://doi.org/10.19734/j.issn.1001-3695.2021.12.0645. LI W R, XU D, SHI J L, et al. Review of salient object detection research: methods, applications and trends[J/OL]. Computer Application Research, https://doi.org/10.19734/j.issn.1001-3695.2021.12.0645.
[13]	LIU Yixiu, ZHANG Yunzhou, Coleman Sonya, et al. A new patch selection method based on parsing and saliency detection for person re-identification[J]. Neurocomputing, 2020, 374: 86-99. DOI: 10.1016/j.neucom.2019.09.073
[14]	赵兴科, 李明磊, 张弓, 等. 基于显著图融合的无人机载热红外图像目标检测方法[J]. 自动化学报, 2021, 47(9): 2120-2131. ZHAO X K, LI M L, ZHANG G, et al. Object detection method based on saliency map fusion for UAV-borne thermal images[J]. Acta Automatica Sinice, 2021, 47(9): 2120-2131.
[15]	QIN Xuebin, ZHANG Zichen, HUANG Chenyang, et al. U2-Net: Going deeper with nested U-structure for salient object detection[J]. Pattern Recognition, 2020, 106: 107404. DOI: 10.1016/j.patcog.2020.107404
[16]	刘若阳, 艾斯卡尔·艾木都拉. 基于局部协方差矩阵判别模型的红外小目标检测方法[J]. 激光与红外, 2020, 50(6): 761-768. DOI: 10.3969/j.issn.1001-5078.2020.06.019 LIU R Y, Aiskar Aimudu. Infrared small target detection method based on local covariance matrix discriminant model[J]. Laser & Infrared, 2020, 50(6): 761-768. DOI: 10.3969/j.issn.1001-5078.2020.06.019
[17]	袁明, 宋延嵩, 张梓祺, 等. 基于增强局部对比度的红外弱小目标检测方法[J]. 激光与光电子学进展, https://kns.cnki.net/kcms/detail/31.1690.tn.20220524.1403.002.html. YUAN M, SONG Y S, ZHANG Z Q, et al. Infrared small target detection method based on enhanced local contrast[J]. Laser and Optoelectronics Progress, https://kns.cnki.net/kcms/detail/31.1690.tn.20220524.1403.002.html.
[18]	CHEN Yunfan, Hyunchul Shin. Pedestrian detection at night in infrared images using an attention-guided encoder-decoder convolutional neural network [J]. Applied Sciences, 2020, 10(3): 809. DOI: 10.3390/app10030809
[19]	代牮, 赵旭, 李连鹏, 等. 基于改进YOLOv5的复杂背景红外弱小目标检测算法[J]. 红外技术, 2022, 44(5): 504-512. http://hwjs.nvir.cn/article/id/f71aa5f4-92b0-4570-9056-c2abd5506021 DAI J, ZHAO X, LI L P, et al. Infrared small target detection algorithm in complex background based on improved YOLOv5[J]. Infrared Technology, 2022, 44(5): 504-512. http://hwjs.nvir.cn/article/id/f71aa5f4-92b0-4570-9056-c2abd5506021
[20]	罗会兰, 陈鸿坤. 基于深度学习的目标检测研究综述[J]. 电子学报, 2020, 48(6): 1230-1239. LUO H L, CHEN H K. A review of object detection based on deep learning[J]. Chinese Journal of Electronics, 2020, 48(6): 1230-1239.
[21]	赵鹏鹏, 李庶中, 李迅, 等. 融合视觉显著性和局部熵的红外弱小目标检测[J]. 中国光学, 2022, 15(2): 267-275. ZHAO P P, LI S Z, LI X, et al. Infrared weak and small target detection combining visual saliency and local entropy[J]. China Optics, 2022, 15(2): 267-275.
[22]	LI Minglei, ZHAO Xingke, LI Jiasong, et al. ComNet: combinational neural network for object detection in UAV-Borne thermal images [J]. IEEE Trans on Geoscience and Remote Sensing, 2021, 59(8): 6662-6673. DOI: 10.1109/TGRS.2020.3029945
[23]	SHAO Yanhua, ZHANG Xingping, CHU Hongyu, et al. AIR-YOLOv3: aerial infrared pedestrian detection via an improved YOLOv3 with network pruning[J]. Applied Sciences, 2022, 12(7): 3627. DOI: 10.3390/app12073627

施引文献(15)

期刊类型引用(9)

1.	曲宏杨. 基于激光测距的机械手臂防碰撞自动控制技术研究. 中国设备工程. 2023(03): 37-39 . 百度学术
2.	乔俊福，郭晋秦，张健. 自主移动机器人激光视觉测距误差补偿方法研究. 激光杂志. 2023(05): 219-223 . 百度学术
3.	鄂晶晶，杨丽华，冯锋. 基于激光扫描的杂散光数据传输优化方法. 激光杂志. 2023(06): 172-176 . 百度学术
4.	刘丽，李晴，王颖. 移动机器人激光雷达测距误差自适应修正研究. 激光杂志. 2023(07): 261-265 . 百度学术
5.	梅开锋，朱超. 基于三相自搜寻比较的电气设备过热故障识别. 电子设计工程. 2023(23): 22-25+30 . 百度学术
6.	李佳卉，李佐艳，杨冲，袁浩波，冯彦，白晓磊. 提高脉冲激光测距精度的方法. 内蒙古科技与经济. 2023(23): 117-122 . 百度学术
7.	李相敏，康壮. 基于正交混频的相位激光测距实验分析系统. 激光杂志. 2022(06): 170-174 . 百度学术
8.	张海庆，韩军，吕德华. 分步式短脉冲激光时间分布特征快速识别模型构建. 激光杂志. 2021(03): 86-90 . 百度学术
9.	吴培鹏，蔡文郁，唐国栋，王志强，朱张峰. 激光测距动态多阈值误差修正技术研究. 电子测量与仪器学报. 2021(07): 170-177 . 百度学术