YE Zhihui, WU Jian, ZHAO Xiaozhong, WANG Wenjuan, SHAO Xinguang. Multimodal Object Detection Based on Feature Interaction and Adaptive Grouping Fusion[J]. Infrared Technology , 2025, 47(4): 468-474.
Citation: YE Zhihui, WU Jian, ZHAO Xiaozhong, WANG Wenjuan, SHAO Xinguang. Multimodal Object Detection Based on Feature Interaction and Adaptive Grouping Fusion[J]. Infrared Technology , 2025, 47(4): 468-474.

Multimodal Object Detection Based on Feature Interaction and Adaptive Grouping Fusion

More Information
  • Received Date: October 24, 2023
  • Revised Date: November 20, 2023
  • To improve the performance of object detection methods in complex scenes, a multimodal object detection model based on feature interaction and adaptive grouping fusion is proposed by combining deep learning algorithms with multimodal information fusion technology. The model uses infrared and visible object images as inputs, constructs a symmetrical dual-branch feature extraction structure based on the PP-LCNet network, and introduces a feature interaction module to ensure complementary information between different modal object features during the extraction process. Secondly, a binary grouping attention mechanism was designed. Global pooling combined with the sign function was used to group the output features of the interaction module into their respective object categories, and spatial attention mechanisms were used to enhance the object information in each group of features. Finally, based on the group-enhanced features, similar feature groups at different scales were extracted, and multi-scale fusion was carried out through adaptive weighting from deep to shallow. Object prediction was then achieved based on the fused features at each scale. The experimental results show that the proposed method significantly improves multimodal feature interaction, key feature enhancement, and multi-scale fusion. Moreover, in complex scenarios, the model exhibits higher robustness and can be better applied to different scenarios.

  • [1]
    孙涵, 刘译善, 林昱涵. 基于深度学习的显著性目标检测综述[J]. 数据采集与处理, 2023, 38(1): 21-50.

    SUN Han, LIU Yishan, LIN Yuhan. A review of salient object detection based on deep learning[J]. Data Collection and Processing, 2023, 38(1): 21-50.
    [2]
    KANG J, Tariq S, Oh H, et al. A survey of deep learning-based object detection methods and datasets for overhead imagery[J]. IEEE Access, 2022, 10: 20118-20134. DOI: 10.1109/ACCESS.2022.3149052
    [3]
    张静, 农昌瑞, 杨智勇. 基于卷积神经网络的目标检测算法综述[J]. 兵器装备工程学报, 2022, 43(6): 37-47.

    ZHANG Jing, NONG Changrui, YANG Zhiyong. Overview of object detection algorithms based on convolutional neural networks[J]. Journal of Weapon Equipment Engineering, 2022, 43(6): 37-47.
    [4]
    JIAO L, ZHANG F, LIU F, et al. A survey of deep learning-based object detection[J]. IEEE Access, 2019, 7: 128837-128868. DOI: 10.1109/ACCESS.2019.2939201
    [5]
    汪鹏, 张大蔚. 基于多源信息和时空约束的移动目标检测算法[J]. 信息技术与信息化, 2022(11): 82-85.

    WANG Peng, ZHANG Dawei. Mobile object detection algorithm based on multi-source information and spatiotemporal constraints[J]. Information Technology and Informatization, 2022(11): 82-85.
    [6]
    YAO X, ZHAO S, XU P, et al. Multi-source domain adaptation for object detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 3273-3282.
    [7]
    陶洋, 祝小钧, 杨柳. 基于皮尔逊相关系数和信息熵的多传感器数据融合[J]. 小型微型计算机系统, 2023, 44(5): 1075-1080.

    TAO Yang, ZHU Xiaojun, YANG Liu. Multi sensor data fusion based on Pearson correlation coefficient and information entropy[J]. Small Microcomputer System, 2023, 44(5): 1075-1080.
    [8]
    LI H, WU X J, Kittler J. RFN-Nest: an end-to-end residual fusion network for infrared and visible images[J]. Information Fusion, 2021, 73: 72-86. DOI: 10.1016/j.inffus.2021.02.023
    [9]
    YANG Y, LIU J, HUANG S, et al. Infrared and visible image fusion via texture conditional generative adversarial network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(12): 4771-4783. DOI: 10.1109/TCSVT.2021.3054584
    [10]
    WU J, SHEN T, WANG Q, et al. Local adaptive illumination-driven input-level fusion for infrared and visible object detection[J]. Remote Sensing, 2023, 15(3): 660. DOI: 10.3390/rs15030660
    [11]
    解宇敏, 张浪文, 余孝源, 等. 可见光–红外特征交互与融合的YOLOv5目标检测算法[J]. 控制理论与应用, 2024, 41(5): 914-922.

    XIE Yumin, ZHANG Langwen, YU Xiaoyuan, et al. YOLOv5 object detection algorithm with visible-infrared feature interaction and fusion[J]. Control Theory and Technology, 2024, 41(5): 914-922.
    [12]
    宁大海, 郑晟. 可见光和红外图像决策级融合目标检测算法[J]. 红外技术, 2023, 45(3): 282-291. http://hwjs.nvir.cn/cn/article/id/5340b616-c317-4372-9776-a7c81ca2c729

    NING Dahai, ZHENG Sheng. Decision level fusion target detection algorithm for visible light and infrared images[J]. Infrared Technology, 2023, 45(3): 282-291. http://hwjs.nvir.cn/cn/article/id/5340b616-c317-4372-9776-a7c81ca2c729
    [13]
    吴泽, 缪小冬, 李伟文, 等. 基于红外可见光融合的低能见度道路目标检测算法[J]. 红外技术, 2022, 44(11): 1154-1160. http://hwjs.nvir.cn/cn/article/id/4bac684b-eed1-4894-900f-ed97489995e6

    WU Ze, MIAO Xiaodong, LI Weiwen, et al. Low visibility road target detection algorithm based on infrared visible light fusion[J]. Infrared Technology, 2022, 44(11): 1154-1160. http://hwjs.nvir.cn/cn/article/id/4bac684b-eed1-4894-900f-ed97489995e6
    [14]
    BAO C, CAO J, HAO Q, et al. Dual-YOLO architecture from infrared and visible images for object detection[J]. Sensors, 2023, 23(6): 2934.
    [15]
    CUI C, GAO T, WEI S, et al. PP-LCNet: a lightweight CPU convolutional neural network[J]. arXiv preprint arXiv, 2021: 2109.15099.
    [16]
    WANG Q, WU B, ZHU P, et al. ECA-Net: efficient channel attention for deep convolutional neural networks[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 11534-11542.
    [17]
    Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision, 2018: 3-19.
  • Related Articles

    [1]SONG Hongwei, LIU Su, LI Haiying, YU Hongyou, SHI Shengbing, GUO Xiuli, LI Haolan, ZHANG Yaping, WANG Xiangqian. Reliability Verification Test of Mid-Wave Infrared 640×512(25 μm) Detector Assembly[J]. Infrared Technology , 2024, 46(8): 879-882.
    [2]MAO Jingxiang, KONG Jincheng, KONG Linglei, GONG Xiaodan, HUANG Junbo, MA Yingting. Theoretical Calculation and Test Analysis of Noise Equivalent Temperature Difference of Photovoltaic Infrared FPA Detectors[J]. Infrared Technology , 2024, 46(8): 872-878.
    [3]MAO Jingxiang, GUO Jianhua, LI Lihua, KONG Linglei, WANG Zhengkai. Calculation of Parameters for Long Wave Infrared FPA Detectors Applied in Low-temperature Background[J]. Infrared Technology , 2023, 45(5): 553-559.
    [4]LI Junbin, LI Dongsheng, WU Shengjuan, ZHOU Xuchang, LI Yanhui, YANG Chunzhang, YANG Wen, JIANG Zhi, CHANG Chao, REN Yang. The Research Progress in Type Ⅱ Superlattices Infrared Focal Plane Array Detectors[J]. Infrared Technology , 2021, 43(11): 1034-1043.
    [5]WANG Qiaofang, ZHENG Wanxiang, WANG Chongwen, LIU Jian, LUO Rui, ZHAO Yuanrong. Preliminary Study on Storage Life Distribution of Semiconductor Device Based on Weibull Distribution[J]. Infrared Technology , 2020, 42(11): 1077-1080.
    [6]CUI Kun, CHEN Fansheng, SU Xiaofeng, LIANG Qinghua, TANG Yujun. A Fast Screening Method of Pixels with Unstable Response Rate in IRFPA[J]. Infrared Technology , 2017, 39(2): 130-135.
    [7]ZHAO Jun, YANG Yu-lin, LI Yan-hui, YANG Chun-zhang, TAN Ying, QI Hang, HAN Fu-zhong, XING Yi-shan, WANG Yu, WANG Xiao-xuan, JI Rong-bin, KONG Jin-cheng. Analysis of MW 320×256 IRFPA Detector Based on MCT/Ge[J]. Infrared Technology , 2014, (6): 439-442,456.
    [8]LU Jian-ming, CAI Yi. The Developing Situation About HgCdTe IRFPA in Russia[J]. Infrared Technology , 2009, 31(5): 303-309. DOI: 10.3969/j.issn.1001-8891.2009.05.015
    [9]CAO Yang, JIN Wei-qi, WANG Xia, XU Chao. Development in Shortwave Infrared Focal Plane Array and Application[J]. Infrared Technology , 2009, 31(2): 63-68. DOI: 10.3969/j.issn.1001-8891.2009.02.001
    [10]Indium Bump Reflow in Flip Chip Inter-connection of Infrared Focal Plane Array Detectors[J]. Infrared Technology , 2007, 29(2): 96-98. DOI: 10.3969/j.issn.1001-8891.2007.02.009
  • Cited by

    Periodical cited type(1)

    1. 袁钱图,邵丽萍,白忠臣. 傅里叶变换红外光谱仪量化误差仿真分析. 智能计算机与应用. 2019(05): 111-114+120 .

    Other cited types(2)

Catalog

    Article views (58) PDF downloads (20) Cited by(3)
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return