Infrared and Visible Image Registration for Power Equipments Based on Deep Learning
-
摘要: 针对现有电力设备红外与可见光图像配准难度大、配准时间长等问题，提出一种基于深度学习的电力设备红外与可见光图像配准融合的方法。本文将特征提取与特征匹配联合在深度学习框架中，直接学习图像块对与匹配标签之间的映射关系，用于后续的配准。此外为了缓解训练时红外图像样本不足的问题，提出一种利用红外图像及其变换图像学习映射函数的自学习方法，同时采用迁移学习来减少训练时间，加速网络框架。实验结果表明：本文方法与其他4种配准算法相比性能指标均有显著提升，本文平均准确率为89.909，同其余4种算法相比分别提高了2.31%、3.36%、2.67%、0.82%，本文平均RMSE（Root Mean Square Error）为2.521，同其余4种配准算法相比分别降低了14.68%、15.24%、4.90%、1.04%，算法平均用时为5.625 s，较其余4种算法分别降低了5.57%、6.82%、2.45%、1.75%，有效提高了电力设备红外与红外可见光图像配准的效率。Abstract: A registration fusion method of infrared and visible images of power equipment based on deep learning is proposed that aims at problems with difficult and long registration time of infrared and visible images of existing power equipment. In this study, feature extraction and feature matching are combined in a deep learning framework to directly learn the mapping relationship between image block pairs and matching labels for subsequent registration. In addition, a self-learning method using infrared image and its transform image to learn the mapping function is proposed to alleviate the problem of insufficient infrared image samples during training Simultaneously, transfer learning is used to reduce the training time and accelerate the network framework. The experimental results show that the performance index of this method is significantly improved compared with the other four registration algorithms. The average accuracy of this method is 89.909, which is 2.31%, 3.36%, 2.67%, and 0.82% higher than that of the other four algorithms, respectively. The average RMSE of this method is 2.521. Compared with the other four registration algorithms, the algorithm is reduced by 14.68%, 15.24%, 4.90%, and 1.04%, respectively. The average time of the algorithm is 5.625 s, which is reduced by 5.57%, 6.82%, 2.45%, and 1.75% respectively. The efficiency of infrared and visible image registration of the power equipment must be effectively improved.
-
Keywords:
- image registration /
- deep learning /
- self-study /
- deep neural network /
- power equipment
-
-
![]()
图 2 自学习策略中训练样本的生成方案
Figure 2. Generation scheme of training samples in self-learning strategy
![]()
图 3 改进前后RANSAC算法比较
Figure 3. Comparison of RANSAC algorithm before and after improvement
![]()
图 5 电力设备红外与可见光图像配准融合
Figure 5. Registration and fusion of infrared and visible images of power equipment
表 1 不同图像大小对模型的影响结果
Table 1 Influence results of different image sizes on model %
Image block ACC P R F-measure 18×18 87.62 87.88 87.28 87.58 34×34 92.27 95.21 89.01 92.01 50×50 92.19 98.21 85.94 91.67 66×66 94.10 97.92 90.10 93.85 
下载: 导出CSV
表 2 不同配准算法配准结果对比
Table 2 Comparison of registration results of different registration algorithms
Methods SIFT Improvement CSS DNN SURF Ours Text 1 ACC 89.142 88.652 88.149 90.011 90.178 RMSE 2.117 2.032 1.713 1.649 1.642 Text 2 ACC 84.492 79.635 80.542 87.799 87.826 RMSE 4.893 5.126 3.854 3.601 3.584 Text 3 ACC 83.816 82.873 85.361 83.931 86.507 RMSE 4.023 3.893 3.901 3.878 3.852 Text 4 ACC 88.920 88.972 88.688 88.998 89.717 RMSE 2.494 2.613 2.598 2.501 2.450 Text 5 ACC 93.028 94.813 95.099 95.158 95.316 RMSE 1.247 1.208 1.189 1.108 1.077
下载: 导出CSV
表 3 不同匹配算法时间对比
Table 3 Time comparison of different matching algorithms
s Time SIFT CSS DNN SURF OURS Test 1 6.021 6.634 6.127 6.031 5.827 Test 2 7.138 7.041 6.732 6.644 6.521 Test 3 9.152 9.037 8.802 8.668 8.646 Test 4 6.071 6.083 5.853 5.973 5.837 Test 5 1.401 1.386 1.317 1.309 1.293
下载: 导出CSV
-
[1] 张旭, 魏娟, 赵冬梅, 等. 电网故障诊断的研究历程及展望[J]. 电网技术, 2013, 37(10): 2745-2553. https://www.cnki.com.cn/Article/CJFDTOTAL-DWJS201310011.htm ZHANG Xu, WEI Juan, ZHAO Dongmei, et al. Research history and prospect of power grid fault diagnosis [J]. Power Grid Technology, 2013, 37(10): 2745-2553. https://www.cnki.com.cn/Article/CJFDTOTAL-DWJS201310011.htm
[2] 刘晓康, 万曦, 涂文超, 等. 基于红外可见光图像配准的电力设备分割算法[J]. 计算机与现代化, 2020(8): 26-30. https://www.cnki.com.cn/Article/CJFDTOTAL-JYXH202008005.htm LIU Xiaokang, WAN Xi, TU Wenchao, et al. Power equipment segmentation algorithm based on infrared and visible image registration[J]. Computer and modernization, 2020(8): 26-30. https://www.cnki.com.cn/Article/CJFDTOTAL-JYXH202008005.htm
[3] 王鲲鹏, 徐一丹, 于起峰. 红外与可见光图像配准方法分类及现状[J]. 红外技术, 2009, 31(5): 270-274. DOI: 10.3969/j.issn.1001-8891.2009.05.007 WANG Kunpeng, XU Yidan, YU Qifeng. Classification and status quo of infrared and visible image registration methods[J]. Infrared Technology, 2009, 31(5): 270-274. DOI: 10.3969/j.issn.1001-8891.2009.05.007
[4] 赵洪山, 张则言. 基于文化狼群算法的电力设备红外和可见光图像配准[J]. 光学学报, 2020, 40(16): 1-14. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB202016007.htm ZHAO Hongshan, ZHANG Zeyan Infrared and visible image registration of power equipment based on cultural wolf swarm algorithm[J]. Acta Optica Sinica, 2020, 40(16): 1-14. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB202016007.htm
[5] MA J Y, JIANG X Y, FAN A X, et al. Image matching from handcrafled to deep features: A survey [J]. International Journal of Computer Vision, 2021, 129(1): 23-79. DOI: 10.1007/s11263-020-01359-2
[6] 徐全飞, 冯旗. 基于SURF和矩阵乘法的超大规模遥感图像亚像素配准算法研究[J]. 红外技术, 2017, 39(1): 44-52. http://hwjs.nvir.cn/article/id/hwjs201701009 XU Quanfei, FENG Qi Research on subpixel registration algorithm of super large scale remote sensing image based on surf and matrix multiplication[J]. Infrared Technology, 2017, 39(1): 44-52. http://hwjs.nvir.cn/article/id/hwjs201701009
[7] WU G R, KIM M, WANG Q, GAO Y Z, et al. Unsupervised deep feature learning for deformable registration of MR brain image[C]//Proceedings of the 16th International Conference on Medical Image Computing and Computer-Assisted Intervention, 2013: 649-656.
[8] CHENG X, ZHANG L, ZHENG Y F. Deep similarity learning for multimodal medical images[J]. Computer Methods in Biomechanics and Biomedical Engineering: Imaging and Visualization, 2018, 6(3): 248-252.
[9] Simonovsky M, Gulierrez-Becker B, Mateus D, et al. A deep metric for multimodal registration[C]//Proceedings of the 19th International Conference on Medical Image Computing and Computer Assisted Intervention, 2016: 10-18.
[10] Kabus S, Netsch T, Fischer B, et al. B-spline registration of 3D images with Levenberg-Marquardt optimization[C]//Proceedings of SPIE, Medical Imaging, Image Processing, 2004, 5370: 304-313.
[11] TANG L S, Hamarneh G, Celler A. Co-registration of bone CT and SPECT images using mutual information[C]//IEEE International Symposium on Signal Processing and Information Technology, 2006: 116-121.
[12] MIAO S, WANG Z J, LIAO R. A CNN regression approach for real-time 2D/3D registration[J]. IEEE Transactions on Medical Imaging, 2016, 35(5): 1352-1363.
[13] HAN Xufeng, LEUNG T, JIA Y, et al. MatchNet: Unifying feature and metric learning for patch-based matching[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015: 3279-3286, Doi: 10.1109/CVPR.2015.7298948.
[14] Krebs J, Mansi T, Delingette H, et al. Robust non-rigid registration through agent-based action learning[C]// Proceedings of the 20th International Conference on Medical Image Computing and Computer-Assisted Intervention, 2017: 344-35.
[15] Dosovitskiy A, Fischer P, Ilg E, et al. FlowNet: learning optical flow with convolutional networks[C]// Proceedings of IEEE International Conference on Computer Vision, 2015: 2758-2766.
[16] LI H W, WANG B H, LI L. Research on the infrared and visible power-equipment image fusion for inspection robots[C]//2010 1st International Conference on Applied Robotics for the Power Industry, 2010: 1-5.
[17] 李寒, 王库, 刘韶军. 基于灰度冗余和SURF算法的电气设备红外和可见光图像配准[J]. 电力系统保护与控制, 2011, 39(11): 111-115, 123. https://www.cnki.com.cn/Article/CJFDTOTAL-JDQW201111021.htm LI Han, WANG Ku, LIU Shaojun. Infrared and visible image registration of electrical equipment based on gray redundancy and surf algorithm[J]. Power System Protection and Control, 2011, 39(11): 111-115, 123. https://www.cnki.com.cn/Article/CJFDTOTAL-JDQW201111021.htm
[18] 姜骞, 刘亚东, 方健, 等. 基于轮廓特征的电力设备红外和可见光图像配准方法[J]. 仪器仪表学报, 2020, 41(11): 252-260. https://www.cnki.com.cn/Article/CJFDTOTAL-YQXB202011028.htm JIANG Qian, LIU Yadong, FANG Jian, et al. Infrared and visible image registration method of power equipment based on contour feature[J]. Journal of Instrumentation, 2020, 41(11): 252-260. https://www.cnki.com.cn/Article/CJFDTOTAL-YQXB202011028.htm
[19] 李云红, 罗雪敏, 苏雪平, 等. 基于改进CSS的电力设备红外与可见光图像配准[J]. 激光与光电子学进展, 2021, 11(24): 1-15. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202212014.htm LI Yunhong, LUO Xuemin, SU Xueping, et al. Infrared and visible image registration of power equipment based on improved CSS [J]. Progress in Laser and Optoelectronics, 2021, 11(24): 1-15. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202212014.htm
[20] Taylor S, Drummond T. Binary histogrammed intensity patches for efficient and robust matching[J]. International Journal of Computer Vision, 2011, 94(2): 241-265.
[21] MA W, ZHANG J, WU Y, et al. A novel two-step registration method for remote sensing images based on deep and local features[J]. IEEE Trans Geosci Remote Sen, 2019, 57: 4834-4843.
[22] YE F, SU Y, XIAO H, et al. Remote sensing image registration using convolutional neural network features[J]. IEEE Geosci. Remote Sens Lett. , 2018, 15: 232-236.
[23] LIU X L, JIANG D S, WANG M N, et al. Image synthesis based multi-modal image registration framework by using deep fully convolutional networks[J]. Medical & Biological Engineering & Computing, 2019, 57(5): 1037-1048.
[24] 周微硕, 安博文, 赵明, 等. 基于几何不变性和局部相似特征的异源遥感图像配准算法[J]. 红外技术, 2019, 41(6): 561-571. http://hwjs.nvir.cn/article/id/hwjs201906012 ZHOU Weishuo, AN Bowen, ZHAO Ming, et al A registration algorithm for heterogeneous remote sensing images based on geometric invariance and local similarity [J]. Infrared Technology, 2019, 41(6): 561-571. http://hwjs.nvir.cn/article/id/hwjs201906012
[25] Salakhutdinov R, Hinton G E. Deep Boltzmann machines[C]// Proceedings of the 12th International Conference on Artificial Intelligence and Statistics, 2009: 448-455.
[26] 杨琼楠, 马天力, 杨聪锟, 等. 基于优化采样的RANSAC图像匹配算法[J]. 激光与光电子学进展, 2020, 57(10): 259-266. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202010029.htm YANG Qiongnan, MA Tianli, YANG Congkun, et al. RANSAC image matching algorithm based on optimized sampling[J]. Progress in Laser and Optoelectronics, 2020, 57(10): 259-266. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202010029.htm
[27] 李航. 统计学习方法[M]. 北京: 清华大学出版社, 2012. LI Hang. Statistical Learning Method[M]. Beijing: Tsinghua University Press, 2012.
[28] 朱英宏, 李俊山, 汤雨. 基于CSS角点提取的红外与可见光图像匹配算法[J]. 系统工程与电子技术, 2011, 33(11): 2540-2545. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYD201111037.htm ZHU Yinghong, LI Junshan, TANG Yu. Infrared and visible image matching algorithm based on CSS corner extraction[J]. Systems Engineering and Electronic Technology, 2011, 33(11): 2540-2545. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYD201111037.htm
[29] 岳国华, 邢晓利. 基于卷积神经网络和校正网络相结合的遥感图像配准方法研究[J]. 计算机应用与软件, 2021, 38(11): 185-190. https://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ202111031.htm YUE Guohua, XING Xiaoli. Research on remote sensing image registration method based on convolution neural network and correction network[J]. Computer Applications and Software, 2021, 38(11): 185-190. https://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ202111031.htm
[30] FU J, LI W S, XU L M. DSAGAN: A generative adversarial network based on dual-stream attention mechanism for anatomical and functional image fusion [J]. Information Sciences, 2021, 576: 484-506.
[31] 王少杰, 武文波, 徐其志. VGG与DoG结合的光学遥感影像精确配准方法[J]. 航天返回与遥感, 2021, 42(5): 76-84. https://www.cnki.com.cn/Article/CJFDTOTAL-HFYG202105010.htm WANG Shaojie, WU Wenbo, XU Qizhi. An accurate registration method of optical remote sensing images based on VGG and DoG[J]. Aerospace Return and Remote Sensing, 2021, 42(5): 76-84. https://www.cnki.com.cn/Article/CJFDTOTAL-HFYG202105010.htm
-
期刊类型引用(2)
1. 张利欣，南清荣，曾慧，王粉花. 多尺度特征融合的钢坯红外成像测温系统实验设计. 实验室研究与探索. 2023(12): 15-19 . 
百度学术
2. 张鸣华，刘持标，陈玉叶，陈杞榕. 基于CCD三波段发射率迭代的加热炉内温度场软测量研究. 冶金设备. 2022(05): 36-41 . 百度学术
其他类型引用(3)
计量
- 文章访问数: 359
- HTML全文浏览量: 74
- PDF下载量: 65
- 被引次数: 5
