| Citation: |
LI Ruihong, FU Zhitao, ZHANG Shaochen, ZHANG Jian, WANG Leiguang. Nighttime Object Detection in Infrared and Visible Images Based on Multi-Attention Mechanism[J]. Infrared Technology , 2024, 46(12): 1371-1379.
|
Object detection has long been a research hotspot in the field of computer vision, and the YOLO series of object detection models is widely used in numerous fields. However, most current image data for object detection are based on a single type of sensor, which makes it difficult to fully characterize the imaging scene. The detected objects contain limited useful information, especially under conditions of low illumination, night, rain, and fog. To improve nighttime object detection, our study proposed a multi-attention mechanism for infrared and visible images. This mechanism combines the CBAM attention mechanism with a Transformer to obtain rich local and contextual information and reduce false detections. To verify the effectiveness of the method, five current mainstream object detection algorithms were selected and tested on a public infrared object detection dataset. The mAP of the proposed method improved from 62.6% to 71.5% compared to the original YOLOv7. This study also produced an infrared–visible fusion dataset for nighttime object detection. On this dataset, the mAP improved significantly from 79.90% to 94.80% compared to the original YOLOv7.
| [1] |
Hafiz A M, Bhat G M. A survey on instance segmentation: state of the art[J]. International Journal of Multimedia Information Retrieval, 2020, 9(3): 171-189. DOI: 10.1007/s13735-020-00195-x
|
| [2] |
ZHANG D, Islam M M, LU G. A review on automatic image annotation techniques[J]. Pattern Recognition, 2012, 45(1): 346-362. DOI: 10.1016/j.patcog.2011.05.013
|
| [3] |
Souza É L, Nakamura E F, Pazzi R W. Object tracking for sensor networks: a survey[J]. ACM Computing Surveys (CSUR), 2016, 49(2): 1-31.
|
| [4] |
YAO H, ZHANG Y, JIAN H, et al. Nighttime pedestrian detection based on fore-background contrast learning[J]. Knowledge-Based Systems, 2023, 275: 110719. DOI: 10.1016/j.knosys.2023.110719
|
| [5] |
Polukhin A, Gordienko Y, Jervan G, et al. Object detection for rescue operations by high-altitude infrared thermal imaging collected by unmanned aerial vehicles[C]//Iberian Conference on Pattern Recognition and Image Analysis. Cham: Springer Nature Switzerland, 2023: 490-504.
|
| [6] |
MA J, MA Y, LI C. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. DOI: 10.1016/j.inffus.2018.02.004
|
| [7] |
MA J, TANG L, XU M, et al. STDFusionNet: An infrared and visible image fusion network based on salient object detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-13.
|
| [8] |
CHEN J, XU X, ZHANG J, et al. Ship target detection algorithm based on decision-level fusion of visible and SAR images[J]. IEEE Journal on Miniaturization for Air and Space Systems, 2023, 4(3): 242-249. DOI: 10.1109/JMASS.2023.3269434
|
| [9] |
WANG C Y, Bochkovskiy A, LIAO H Y M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 7464-7475.
|
| [10] |
WANG C Y, LIAO H Y M. Yolov4: Optimal speed and accuracy of object detection[J]. arXiv preprint arXiv: 2004.10934, 2020.
|
| [11] |
Woo S, Park J, Lee J Y, et al. Cbam: Convolutional lock attention module[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 3-19.
|
| [12] |
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5998-6008.
|
| [13] |
MA J, YU W, LIANG P, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26. DOI: 10.1016/j.inffus.2018.09.004
|
| [14] |
MA J, CHEN C, LI C, et al. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31: 100-109. DOI: 10.1016/j.inffus.2016.02.001
|
| [15] |
XU H, MA J, JIAGN J, et al. U2Fusion: a unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 502-518.
|
| [16] |
LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623.
|
| [17] |
REN S, HE K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39(6): 1137-1149.
|
| [18] |
LIU W, Anguelov D, Erhan D, et al. Ssd: single shot multibox detector[C]//Computer Vision–ECCV 2016: 14th European Conference, Proceedings, Part Ⅰ 14. Springer International Publishing, 2016: 21-37.
|
| [19] |
ZHU X, LYU S, WANG X, et al. TPH-YOLOv5: Improved YOLOv5 based on transformer prediction head for object detection on drone-captured scenarios[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 2778-2788.
|
| [1] | LIU Lei, QIAN Yunsheng. A Low Illumination Image Acquisition and Processing System Based on FPGA[J]. Infrared Technology , 2022, 44(5): 462-468. |
| [2] | SUN Shaowei, YANG Yuetao, YANG Bingwei, WAN Anjun, ZHONG Hailin. Research and Implementation of Infrared Lens Auto-focus Technology Based on Field Programmable Gate Array[J]. Infrared Technology , 2021, 43(5): 464-472. |
| [3] | CHI Linhui, QIAN Yunsheng, JI Yuhao. Verification Protocol for Improving Communication Stability Between FPGAs[J]. Infrared Technology , 2020, 42(11): 1022-1027. |
| [4] | High-speed Spectrum Inversion System Based on FPGA[J]. Infrared Technology , 2019, 41(6): 535-539. |
| [5] | LIU Yuan, LI Qing, LIANG Yanju. Implementation of Infrared Target Detection System Based on FPGA[J]. Infrared Technology , 2019, 41(6): 521-526. |
| [6] | LIU Jiangping, XUE Heru. High-speed Spectrum Acquisition and Processing System Based on FPGA[J]. Infrared Technology , 2018, 40(11): 1042-1046. |
| [7] | ZHANG Chenghong, LI Fanming, YANG Long. Real Time Infrared Video Capture and Display System Based on FPGA[J]. Infrared Technology , 2017, 39(2): 143-146. |
| [8] | LIU Rui-qiang, WANG Yong-xin. Based on FPGA Real-time Spectrum Obtained of Static Fourier Spectrometer[J]. Infrared Technology , 2011, 33(8): 465-469. DOI: 10.3969/j.issn.1001-8891.2011.08.008 |
| [9] | GONG Man-man, CHEN Qian, GU Guo-hua, SUI Xiu-bao. FPGA-Based Realization of Second-Order Newton Interpolation of Infrared Image[J]. Infrared Technology , 2010, 32(12): 723-726. DOI: 10.3969/j.issn.1001-8891.2010.12.009 |
| [10] | Improved Canny Edge Detection Algorithm and Implementation in FPGA[J]. Infrared Technology , 2010, 32(2): 93-96. DOI: 10.3969/j.issn.1001-8891.2010.02.008 |
| 1. |
朱强，周维虎，陈晓梅，石俊凯，李冠楠. 高速实时近红外弱信号检测系统. 光学精密工程. 2022(24): 3116-3127 .
|
Supported by: Beijing Renhe Information Technology Co., Ltd. 
DownLoad: