高级检索
ENGLISH

红外与可见光图像配准技术研究综述

李云红, 刘宇栋, 苏雪平, 罗雪敏, 姚兰

李云红, 刘宇栋, 苏雪平, 罗雪敏, 姚兰. 红外与可见光图像配准技术研究综述[J]. 红外技术, 2022, 44(7): 641-651.
引用本文: 李云红, 刘宇栋, 苏雪平, 罗雪敏, 姚兰. 红外与可见光图像配准技术研究综述[J]. 红外技术, 2022, 44(7): 641-651.
shu
LI Yunhong, LIU Yudong, SU Xueping, LUO Xuemin, YAO Lan. Review of Infrared and Visible Image Registration[J]. Infrared Technology , 2022, 44(7): 641-651.
Citation: LI Yunhong, LIU Yudong, SU Xueping, LUO Xuemin, YAO Lan. Review of Infrared and Visible Image Registration[J]. Infrared Technology , 2022, 44(7): 641-651.
shu

红外与可见光图像配准技术研究综述

  • 西安工程大学 电子信息学院, 陕西 西安 710048

基金项目: 

国家自然科学基金 61902301

陕西省科技厅自然科学基础研究重点项目 2022JZ-35

国家级大学生创新创业训练计划项目 S202110709002

详细信息
    作者简介:

    李云红（1974-），女，教授，硕士生导师，研究方向为红外热像技术、数字图像处理和信号与信息处理技术。E-mail：hitliyunhong@163.com

  • 中图分类号: TP391

Review of Infrared and Visible Image Registration

  • School of Electronics and Information, Xi'an Polytechnic University, Xi'an 710048, China

摘要

    摘要
  • 摘要: 多模态图像配准能提供比单模态图像配准更加丰富和全面的信息，红外与可见光图像配准作为一种常见的多模态配准类型，在电力、遥感、军事以及人脸识别等领域具有重要的应用价值。首先介绍了红外与可见光图像配准的相关技术并阐述了配准中存在的难点与挑战，然后详细分析和总结了基于区域、基于特征和基于深度学习3种红外与可见光图像配准方法，并分别阐述了不同配准方法的优缺点，之后概述了红外与可见光图像配准技术的实际应用，最后对红外与可见光图像配准未来的发展趋势进行讨论。
    Abstract: Multi-modal image registration can provide richer and more comprehensive information than single-modal image registration. Among them, infrared and visible image registration, which is a common multi-modal form of registration, has important application value in fields such as electric power, remote sensing, military, and face recognition. In this paper, the correlation technique of infrared and visible image registration is introduced, and the existing difficulties and challenges involved in registration are analyzed. Subsequently, the advantages and disadvantages of different registration methods are evaluated in detail the three types based on area, feature, and deep learning, and a practical application of infrared and visible image registration technology is presented. Finally, the future development trend of infrared and visible image registration is discussed.
    • HTML全文

  • 图  1   基于区域的图像配准算法步骤

    Figure  1.   The steps of region-based image registration algorithm

    下载: 全尺寸图片 幻灯片

    图  2   基于特征的图像配准算法步骤

    Figure  2.   The steps of feature-based image registration algorithm

    下载: 全尺寸图片 幻灯片

    图  3   基于深度学习的图像配准算法步骤

    Figure  3.   Image registration algorithm principle based on deep learning

    下载: 全尺寸图片 幻灯片

    图  4   基于互信息的PSO-Powell图像配准算法及实验结果：算法流程图(a)；红外与可见光图像对(b)和配准结果(c)[20]

    Figure  4.   Pso-powell image registration algorithm based on mutual information and experimental results: (a)Flowchart of the PSO–Powell algorithm; (b)A pair of infrared and visual images and (c)Registration results[20]

    下载: 全尺寸图片 幻灯片

    图  5   基于点特征的图像配准：(a)红外与可见光图像配准；(b)融合结果

    Figure  5.   Image registration based on point feature: (a)Infrared and visible image registration; (b)Fusion result

    下载: 全尺寸图片 幻灯片

    图  6   基于形态学梯度和C_SIFT的实时自适应可见光和红外图像配准及其结果：(a)算法流程图；(b)待配准图像对；(c)待配准图像的形态梯度图像；(d)红外与可见光图像配准结果[24]

    Figure  6.   Real-time adaptive visible and infrared image registration based on morphological gradient and C_SIFT and result: (a)The flowchart of proposed algorithm; (b)Image pairs to be registered; (c)The morphology gradient image of visible and infrared images; (d)The registration results of visible and infrared images[24]

    下载: 全尺寸图片 幻灯片

    图  7   高斯加权形状上下文原理与配准结果：(a)由原始SC提供的对应点的实例；(b)改进的GWSC配准结果[42]

    Figure  7.   Principle and registration results of Gaussian weighted shape Context (GWSC): (a)An example of point correspondence by the original SC; (b)Qualitative registration results of GWSC[42]

    下载: 全尺寸图片 幻灯片

    图  8   RegiNet的网络架构。

    Figure  8.   Network architecture of RegiNet

    下载: 全尺寸图片 幻灯片

    图  9   基于GAN的变换参数预测框架

    Figure  9.   Transform parameter prediction framework based on GAN

    下载: 全尺寸图片 幻灯片

    图  10   变压器对抗网络（TAN）的红外与可见光图像配准框架及部分实验结果：变压器对抗网络框架(a)与配准结果(b)[45]

    Figure  10.   Infrared and visible image registration framework using transformer adversarial network (TAN) and some experimental results: (a) The framework of the proposed Transformer Adversarial Network and (b) registration results of TAN[45]

    下载: 全尺寸图片 幻灯片
    • 参考文献(53)
  • [1]

    LI Y, YU F Y, CAI Q, et al. Image fusion of fault detection in power system based on deep learning[J]. Cluster Computing the Journal of Networks Software Tools and Applications, 2019, 22(4): 9435-9443.
    [2]

    MA J, ZHAO J, MA Y, et al. Non-rigid visible and infrared face registration via regularized Gaussian fields criterion[J]. Pattern Recognition, 2015, 48(3): 772-784. DOI: 10.1016/j.patcog.2014.09.005
    [3] 韩静, 柏连发, 张毅, 等. 基于改进配准测度的红外与可见光图像配准[J]. 红外技术, 2011, 33(5): 271-274. DOI: 10.3969/j.issn.1001-8891.2011.05.006

    HAN J, BO L F, ZHANG Y, et al. A registration algorithm of infrared and visible image based on revised measure function[J]. Infrared Technology, 2011, 33(5): 271-274. DOI: 10.3969/j.issn.1001-8891.2011.05.006
    [4]

    Sarvaiya J N, Patnaik S, Bombaywala S. Image registration by template matching using normalized cross-correlation[C]//2009 International Conference on Advances in Computing, Control, and Telecommunication Technologies. IEEE, 2009: 819-822.
    [5]

    YANG Z, SHEN G, WANG W, et al. Spatial-spectral cross correlation for reliable multispectral image registration[C]//2009 IEEE Applied Imagery Pattern Recognition Workshop (AIPR 2009), 2009: 1-8.
    [6]

    MA J, JIANG X, FAN A, et al. Image matching from handcrafted to deep features: A survey[J]. International Journal of Computer Vision, 2021, 129(1): 23-79. DOI: 10.1007/s11263-020-01359-2
    [7]

    Stone H S, Tao B, McGuire M. Analysis of image registration noise due to rotationally dependent aliasing[J]. Journal of Visual Communication and Image Representation, 2003, 14(2): 114-135. DOI: 10.1016/S1047-3203(03)00002-6
    [8]

    LI Z, YANG J, LI M, et al. Estimation of large scalings in images based on multilayer pseudopolar fractional Fourier transform[J]. Mathematical Problems in Engineering, 2013, 2013: 179489.
    [9]

    DONG Y, JIAO W, LONG T, et al. An extension of phase correlation-based image registration to estimate similarity transform using multiple polar Fourier transform[J]. Remote Sensing, 2018, 10(11): 1719. DOI: 10.3390/rs10111719
    [10]

    Fujisawa T, Ikehara M. High-accuracy image rotation and scale estimation using radon transform and sub-pixel shift estimation[J]. IEEE Access, 2019, 7: 22719-22728. DOI: 10.1109/ACCESS.2019.2899390
    [11]

    Fouda Y, Ragab K. An efficient implementation of normalized cross-correlation image matching based on pyramid[C]//2013 International Joint Conference on Awareness Science and Technology & Ubi-Media Computing, IEEE, 2013: 98-103.
    [12]

    YANG M Y, QIANG Y, Rosenhahn B. A global-to-local framework for infrared and visible image sequence registration[C]//2015 IEEE Winter Conference on Applications of Computer Vision, 2015: 381-388.
    [13]

    BAI L F, HAN J, ZHANG Y, et al. Registration algorithm of infrared and visible images based on improved gradient normalized mutual information and particle swarm optimization[J]. Infrared Laser Engineering, 2012, 41(1): 248-254.
    [14]

    CHEN S J, SHEN H L, LI C, et al. Normalized total gradient: A new measure for multispectral image registration[J]. IEEE Transactions on Image Processing, 2017, 27(3): 1297-1310.
    [15]

    YANG T J, TANG Q, LI L. Nonrigid registration of medical image based on adaptive local structure tensor and normalized mutual information[J]. Journal of Applied Clinical Medical Physics, 2019, 20(6): 99-110. DOI: 10.1002/acm2.12612
    [16] 赵洪山, 张则言. 基于文化狼群算法的电力设备红外和可见光图像配准[J]. 光学学报, 2020, 40(16): 51-64. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB202016007.htm

    ZHAO H, ZHANG Z. Power equipment infrared and visible images registration based om cultural wolf pack algorithm[J]. Acta Optica Sinica, 2020, 40(16): 51-64. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB202016007.htm
    [17] 孙凤杰, 赵孟丹, 刘威, 等. 基于方向场的输电线路间隔棒匹配定位算法[J]. 中国电机工程学报, 2014, 34(1): 206-213. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGDC201401025.htm

    SUN F, ZHAO M, LIU W, et al. Spacer matching and localizetion algorithm for transmission line video image based on directional field[J]. Proceedings of the CSEE, 2014, 34(1): 206-213. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGDC201401025.htm
    [18] 刘刚, 周珩, 梁晓庚, 等. 非下采样轮廓波域红外与可见光图像配准算法[J]. 计算机科学, 2016, 43(11): 313-316. DOI: 10.11896/j.issn.1002-137X.2016.11.061

    LIU G, ZHOU H, LIANG X, et al. Image registration algorithm for infrared and visible light based on non-subsamoled Contourlet transform[J]. Computer Science, 2016, 43(11): 313-316. DOI: 10.11896/j.issn.1002-137X.2016.11.061
    [19] 吴延海, 张程, 张烨. 基于梯度信息和区域互信息的图像配准[J]. 广西大学学报: 自然科学版, 2017, 42(2): 720-727. https://www.cnki.com.cn/Article/CJFDTOTAL-GXKZ201702040.htm

    WU Y, ZHANG C, ZHANG Y. Image Registration based on gradient and regional mutual information[J]. Journal of Guangxi University: Natural Science Edition, 2017, 42(2): 720-727. https://www.cnki.com.cn/Article/CJFDTOTAL-GXKZ201702040.htm
    [20]

    ZHUANG Y, GAO K, MIU X, et al. Infrared and visual image registration based on mutual information with a combined particle swarm optimization–Powell search algorithm[J]. Optik, 2016, 127(1): 188-191. DOI: 10.1016/j.ijleo.2015.09.199
    [21]

    LI Y, WANG J, YAO K. Modified phase correlation algorithm for image registration based on pyramid[J]. Alexandria Engineering Journal, 2022, 61(1): 709-718. DOI: 10.1016/j.aej.2021.05.004
    [22]

    Morel J M, YU G. ASIFT: A new framework for fully affine invariant image comparison[J]. SIAM Journal on Imaging Sciences, 2009, 2(2): 438-469. DOI: 10.1137/080732730
    [23]

    YANG N, YANG Y, LI P, et al. Research on infrared and visible image registration of substation equipment based on Multi-scale Retinex and ASIFT features[C]//Sixth International Workshop on Pattern Recognition. International Society for Optics and Photonics, 2021, 11913: 1191303.
    [24]

    ZENG Q, ADU J, LIU J, et al. Real-time adaptive visible and infrared image registration based on morphological gradient and C_SIFT[J]. Journal of Real-Time Image Processing, 2020, 17(5): 1103-1115. DOI: 10.1007/s11554-019-00858-x
    [25]

    JIANG Q, LIU Y, YAN Y, et al. A contour angle orientation for power equipment infrared and visible image registration[J]. IEEE Transactions on Power Delivery, 2020, 36(4): 2559-2569.
    [26] 李伟, 王军, 俞跃. 基于可见光匹配矩阵的电气部件故障红外自动识别算法[J]. 红外技术, 2019, 41(11): 1047-1056. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS201911009.htm

    LI W, WANG J, YU Y. An infrared automatic fault recognition method for electrical parts based on visible matching matrix[J]. Infrared Technology, 2019, 41(11): 1047-1056. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS201911009.htm
    [27]

    CHEN Y, DAI J, MAO X, et al. Image registration betwe en visible and infrared images for electrical equipment inspection robots based on quadrilateral features[C]// 2nd International Conference on Robotics and Automation Engineering (ICRAE), 2017: 126-130.
    [28] 戴进墩, 刘亚东, 毛先胤, 等. 基于NSCT域FAST角点检测的电气设备红外与可见光图像配准[J]. 电测与仪表, 2019, 56(1): 108-114. https://www.cnki.com.cn/Article/CJFDTOTAL-DCYQ201901018.htm

    DAI J, LIU Y, MAO X, et al. Registration based on NSCT-domain FAST corner detection for infrared and visible images of electrical equipment[J]. Electrical Measurement & Instrumentation, 2019, 56(1): 108-114. https://www.cnki.com.cn/Article/CJFDTOTAL-DCYQ201901018.htm
    [29] 江泽涛, 刘小艳, 王琦. 基于显著性和ORB的红外和可见光图像配准算法[J]. 激光与红外, 2019, 49(2): 251-256. DOI: 10.3969/j.issn.1001-5078.2019.02.022

    JIANG Z, LIU X, WANG Q. Visible and infrared image registration algorithm based on saliency and ORB[J]. Laser & Infrared, 2019, 49(2): 251-256. DOI: 10.3969/j.issn.1001-5078.2019.02.022
    [30] 占祥慧, 徐智勇, 张建林. 结合滚动引导滤波和相位信息的红外与可见光图像配准[J]. 半导体光电, 2021, 42(5): 726-732. https://www.cnki.com.cn/Article/CJFDTOTAL-BDTG202105023.htm

    ZHANG X, XU Z, ZHANG J. Infrared and visible images registration using rolling guided filter and phase information[J]. Semiconductor Optoelectronics, 2021, 42(5): 726-732. https://www.cnki.com.cn/Article/CJFDTOTAL-BDTG202105023.htm
    [31]

    LI Q, HAN G, LIU P, et al. An infrared-visible image registration method based on the constrained point feature[J]. Sensors, 2021, 21(4): 1188. DOI: 10.3390/s21041188
    [32]

    CHEN X, LIU L, ZHANG J, et al. Registration of multimodal images with edge features and scale invariant PIIFD[J]. Infrared Physics & Technology, 2020, 111: 103549.
    [33]

    LIU X, LI J B, PAN J S, et al. An advanced gradient texture feature descriptor based on phase information for infrared and visible image matching[J]. Multimedia Tools and Applications, 2021, 80(11): 16491-16511. DOI: 10.1007/s11042-020-10213-z
    [34] 李云红, 罗雪敏, 苏雪平, 等. 基于改进CSS的电力设备红外与可见光图像配准[J/OL]. 激光与光电子学进展: 1-14. [2021-9-9]. http://www.cnki.com.cn/Article/CJFDTotal-JGDJ20210909004.htm.

    LI Y, LUO X, SU X, et al. Registration method for power equipment infrared and visible images based on improved CSS[J/OL]. Laser & Optoelectronics Progress: 1-14. [2021-9-9]. http://www.cnki.com.cn/ Article/CJFDTotal-JGDJ20210909004.htm.
    [35]

    CHENG T, GU J, ZHANG X, et al. Multimodal image registration for power equipment using clifford algebraic geometric invariance[J]. Energy Reports, 2022, 8: 1078-1086. DOI: 10.1016/j.egyr.2022.02.192
    [36] 陈亮, 周孟哲, 陈禾. 一种结合边缘区域和互相关的图像配准方法[J]. 北京理工大学学报, 2016, 36(3): 320-325. https://www.cnki.com.cn/Article/CJFDTOTAL-BJLG201603018.htm

    CHEN L, ZHOU M, CHEN H. A method for image registration combined by edge region and cross correlation[J]. Transactions of Beijing Institute of Technology, 2016, 36(3): 320-325. https://www.cnki.com.cn/Article/CJFDTOTAL-BJLG201603018.htm
    [37]

    LIU G, LIU Z, LIU S. et al. Registration of infrared and visible light image based on visual saliency and scale invariant feature transform[J]. J. Image Video Proc. , 2018: 45. https://doi.org/10.1186/s13640-018-0283-9.
    [38] 廉蔺, 李国辉, 张军, 等. 基于边缘最优映射的红外和可见光图像自动配准算法[J]. 自动化学报, 2012, 38(4): 570-581. https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201204009.htm

    LIAN L, LI G, ZHANG J, et al. An automatic registration algorithm of infrared and visible images based on optimal mapping of edges[J]. Acta Automatica Sinica, 2012, 38(4): 570-581. https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201204009.htm
    [39]

    MA J, ZHAO J, MA Y, et al. Non-rigid visible and infrared face registration via regularized Gaussian fields criterion[J]. Pattern Recognition, 2015, 48(3): 772-784. DOI: 10.1016/j.patcog.2014.09.005
    [40] 李巍, 董明利, 吕乃光, 等. 基于T分布混合模型的多光谱人脸图像配准[J]. 光学学报, 2019, 39(7): 56-66. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB201907007.htm

    LI W, DONG M L, LYU N G, et al. Multispectral face image registration based on t-distribution mixture model[J]. Acta Optica Sinica, 2019, 39(7): 56-66. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB201907007.htm
    [41]

    MIN C, GU Y, LI Y, et al. Non-rigid infrared and visible image registration by enhanced affine transformation[J]. Pattern Recognition, 2020, 106: 107377. DOI: 10.1016/j.patcog.2020.107377
    [42]

    MIN C, GU Y, YANG F, et al. Non-rigid registration for infrared and visible images via Gaussian weighted shape context and enhanced affine transformation[J]. IEEE Access, 2020, 8: 42562-42575. DOI: 10.1109/ACCESS.2020.2976767
    [43]

    ZHAO Z, ZHAO L, QI Y, et al. Infrared and visible image registration based on hypercolumns[C]//CCF Chinese Conference on Computer Vision, 2017: 529-539.
    [44]

    WEI Z, JUNG C, SU C. RegiNet: Gradient guided multispectral image registration using convolutional neural networks[J]. Neurocomputing, 2020, 415: 193-200. DOI: 10.1016/j.neucom.2020.07.066
    [45]

    WANG L, GAO C, ZHAO Y, et al. Infrared and visible image registration using transformer adversarial network[C]// 25th IEEE International Conference on Image Processing (ICIP). IEEE, 2018: 1248-1252.
    [46]

    Kumari K, Krishnamurthi G. GAN-based End-to-End Unsupervised Image Registration for RGB-Infrared Image[C]// 3rd International Conference on Intelligent Autonomous Systems (ICoIAS). IEEE, 2020: 62-66.
    [47]

    MAO Y, HE Z. Dual-Y network: infrared-visible image patches matching via semi-supervised transfer learning[J]. Applied Intelligence, 2021, 51(4): 2188-2197. DOI: 10.1007/s10489-020-01996-7
    [48] 杨冰超, 王鹏, 李晓艳, 等. 基于模态转换结合鲁棒特征的红外图像和可见光图像配准[J]. 激光与光电子学进展, 2022, 59(4): 180-189. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202204017.htm

    YANG B, WANG P, LI X, et al. Infrared and visible light image registration based on model conversion and robust features[J]. Laser & Optoelectronics Progress, 2022, 59(4): 180-189. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202204017.htm
    [49]

    YU K, MA J, HU F, et al. A grayscale weight with window algorithm for infrared and visible image registration[J]. Infrared Physics & Technology, 2019, 99: 178-186.
    [50]

    LUO W, HAO X, XU C, et al. Coarse-to-fine registration for infrared and visible images of power grid[C]// 4th International Conference on Systems and Informatics (ICSAI), 2017: 1181-1185.
    [51] 姜骞, 刘亚东, 方健, 等. 基于轮廓特征的电力设备红外和可见光图像配准方法[J]. 仪器仪表学报, 2020, 41(11): 252-260. https://www.cnki.com.cn/Article/CJFDTOTAL-YQXB202011028.htm

    JIANG Q, LIU Y D, FANG J, et al. Registration method for power equipment infrared and visible images based on contour feature[J]. Chinese Journal of Scientific Instrument, 2020, 41(11): 252-260. https://www.cnki.com.cn/Article/CJFDTOTAL-YQXB202011028.htm
    [52] 马露凡, 罗凤, 严江鹏, 等. 深度医学图像配准研究进展: 迈向无监督学习[J]. 中国图象图形学报, 2021, 26(9): 2037-2057. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202109002.htm

    MA L F, LUO F, YAN J P, et al. Deep-learning based medical image registration pathway: towards unsupervised learning[J]. Journal of Image and Graphics, 2021, 26(9): 2037-2057. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202109002.htm
    [53] 刘欢, 肖根福, 欧阳春娟, 等. 初-精结合和多特征融合的多源遥感图像配准[J]. 遥感信息, 2018, 33(6): 61-70. DOI: 10.3969/j.issn.1000-3177.2018.06.009

    LIU H, XIAO G F, OUYANG C J, et al. Initial-fine combination and multi-feature fusion for multi-source remote sensing image registration[J]. Remote Sensing Information, 2018, 33(6): 61-70. DOI: 10.3969/j.issn.1000-3177.2018.06.009
    • 相关文章
    • 施引文献(11)

  • 期刊类型引用(5)

    1. 王刚，万洵，崔志超，谢良平. 基于动态温控的光纤陀螺高温工作控制方案. 应用光学. 2023(05): 1153-1156 . 百度学术
    2. 王斌华，黄迟航，胡桥，孔军，陈平. 加速度场下陀螺光纤环形变的影响分析. 应用光学. 2021(02): 360-370 . 百度学术
    3. 何柏青. 四频激光陀螺抗强磁场干扰数字电路优化设计. 激光杂志. 2021(08): 183-186 . 百度学术
    4. 孙继泽，杜金其，王伟，甘怡红，游清清. 电力光纤通信网络实时安全风险量化参数优化算法. 激光杂志. 2021(12): 176-180 . 百度学术
    5. 朱锐. 干涉式光纤陀螺电路串扰的智能辨识技术. 激光杂志. 2020(10): 149-152 . 百度学术

    其他类型引用(6)

    • 资源附件(0)
图(10)
计量
  • 文章访问数:  648
  • HTML全文浏览量:  258
  • PDF下载量:  262
  • 被引次数: 11
出版历程
  • 收稿日期:  2022-04-12
  • 修回日期:  2022-05-23
  • 刊出日期:  2022-07-19

目录

摘要
HTML全文
    参考文献(53)
    相关文章
    施引文献(11)
    资源附件(0)

    /

    返回文章
    返回

    版权所有 © 《红外技术》编辑部滇ICP备12000354号-3

    本系统由 北京仁和汇智信息技术有限公司 开发  百度统计