基于多源图像融合的光伏面板缺陷检测

闫号; 戴佳佳; 龚小溪; 吴宇祥; 汪俊

基于多源图像融合的光伏面板缺陷检测

南京航空航天大学机电学院, 江苏南京 210006

详细信息

作者简介:
闫号（1998-），男，硕士研究生，研究方向为图像处理、数字化设计与制造。E-mail：864650996@qq.com

通讯作者:
汪俊（1989-），男，博士，教授，研究方向为数字化检测技术、深度学习。E-mail：wjun@nuaa.edu.cn

中图分类号: TP181
计量
- 文章访问数: 284
- HTML全文浏览量: 63
- PDF下载量: 56
- 被引次数: 0
出版历程
- 收稿日期: 2022-08-17
- 修回日期: 2022-09-13
- 刊出日期: 2023-05-20

Defect Detection of Photovoltaic Panel Based on Multisource Image Fusion

College of Mechanical & Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210006, China

摘要

摘要: 传统光伏面板缺陷检测任务以人工目视方法为主，存在效率低、精度差、成本高等问题。提出基于深度学习的融合光伏面板可见光图像与红外图像的缺陷检测网络，即多源图像融合网络（Multi-source Fusion Network, MF-Net），实现光伏面板的缺陷检测。MF-Net以YOLOv3 tiny为主干结构，并针对光伏面板缺陷特征进行网络结构改进。其中包括：在特征提取模块中增加网络深度并融入密集块结构，使得MF-Net能够融合更多高层语义信息的同时增强特征的选择；将双尺度检测增加为三尺度检测，以提高网络对不同尺寸缺陷的适用度。此外，提出自适应融合模块，使特征图融合过程中可以根据像素邻域信息自适应分配融合系数。实验结果表明，相比基于YOLOv3 tiny的融合网络，改进后的融合检测网络mAP提高7.41%；自适应融合模块使mAP进一步提升2.14%，且自适应融合模块能够有效提高特征的显著性；在与单一图像（仅有可见光图像或红外图像）的检测网络及其他融合图像检测网络的对比实验中，所提出的网络F1 score最高（F1＝0.86）。
- 光伏面板 /
- 图像融合 /
- 缺陷检测 /
- 可见光图像 /
- 红外图像
Abstract: This study proposed a multisource fusion network (MF-Net) that combines visible and infrared images for the inspection of a photovoltaic panel to achieve photovoltaic panel defect detection, defect classification, and localization. The limitations of the traditional methods include low efficiency, low accuracy, and high cost. In this study, a defect detection network was designed based on the backbone of YOLOv3-tiny. Deep layers are added to the network, constituting a dense block structure to augment semantic information on fused feature maps. The detection scale of the network was extended to improve its applicability at different scales. In addition, an adaptive weight fusion strategy was proposed to achieve feature map fusion, where the fusion coefficients can be allocated according to the pixel neighborhood information. Compared with the backbone, the results show that the mAP of our network improved by 7.41%. The performance improves (by approximately 2.14% mAP) when the weighted fusion strategy is replaced with ours, and the significance of the features can be effectively improved. Relative to other networks, the proposed network that takes the fused images as the input has the highest performance in terms of the F1 score (F1=0.86).
- photovoltaic panel /
- image fusion /
- defect detection /
- visible image /
- infrared image

HTML全文

图 1 红外图像热斑

Figure 1. Hot spots on the infrared image

下载: 全尺寸图片幻灯片

图 2 MF-Net框架

Figure 2. Framework for MF-Net

下载: 全尺寸图片幻灯片

图 3 特征提取模块

Figure 3. Feature extraction module

下载: 全尺寸图片幻灯片

图 4 密集块结构

Figure 4. Dense block structure

下载: 全尺寸图片幻灯片

图 5 自适应融合策略

Figure 5. Adaption fusion strategy

下载: 全尺寸图片幻灯片

图 6 硬件平台与数据采集

Figure 6. Hardware platform and data collection

下载: 全尺寸图片幻灯片

图 7 图像获取与配准

Figure 7. Image acquisition and registration

下载: 全尺寸图片幻灯片

图 8 五种缺陷在不同融合检测网络中的P-R测试曲线

Figure 8. P-R curves of five kinds of defects in different fusion detection networks

下载: 全尺寸图片幻灯片

图 9 基于单一图像或多源图像对不同缺陷的检测结果对比

（(a)基于可见光图像检测结果；(b)基于红外图像检测结果；(c)本文方法检测结果；(d)实际缺陷类型及位置）

Figure 9. Comparison of detection results of different defects based on single image or multi-source image

((a) Detection results based on visible image; (b) Based on infrared image detection results; (c) Detection results of this method; (d) Actual defect type and location)

下载: 全尺寸图片幻灯片

表 1 改进前后网络模型对比

Table 1. Comparison parameters of network model before and after improvement

Network model	Model size/Mb	FLOPS/Gb
YOLOv3	245.3	63.20
YOLOv3 tiny	33.4	5.56
MF-Net	35.6	6.02

下载: 导出CSV

表 2 数据集分配统计

Table 2. Distributions of our dataset

	Number of images/pairs	Crazing	Shadow	Covering	Bubble	Internal
Filtered images	659	103	398	367	96	403
Enhanced images	1000	544	552	534	537	565
Training set	700	366	363	349	356	371
Validation set	100	60	63	61	59	64
Test set	200	118	126	124	122	130

下载: 导出CSV

表 3 网络训练参数

Table 3. Network training parameters

Parameter	Value	Parameter	Value
Input size	416×416	Epoch	200
Batch size	2	Momentum	0.9
Learning rate	0.0005	Confidence	0.65

下载: 导出CSV

表 4 消融实验结果

Table 4. Results of ablation experiences

S/N	Replace Conv.	Three scale detection	Dense Block	mAP	FPS
1				75.93%	11.23
2	√			82.02%(+6.09%)	8.13(-3.10)
3		√		76.21%(+0.28%)	9.43(-1.80)
4			√	80.96%(+5.03%)	11.36(+0.13)
5	√	√	√	83.34%(+7.41%)	7.41(-3.82)

下载: 导出CSV

表 5 不同网络框架对比评估

Table 5. Comparative evaluation of different networks

S/N	Model	Input	Correct detection	Correct classification	FPS	F1 score
1	Improved YOLOv3 tiny	Visible image	320	297	12.61	0.53
2	Improved YOLOv3 tiny	Infrared image	497	103	12.63	0.17
3	MF-Net and channel addition strategy	Multi-source image	553	516	7.41	0.82
4	SSD	Multi-source image	512	446	1.31	0.72
5	Ours	Multi-source image	559	538	7.39	0.86

下载: 导出CSV

参考文献(23)

[1]	Alves R H F, Deus Júnior G A, Marra E G, et al. Automatic fault classification in photovoltaic modules using Convolutional Neural Networks[J]. Renewable Energy, 2021, 179: 502-516. doi: 10.1016/j.renene.2021.07.070
[2]	闫冯渊. 实用化光伏组件监测电路研制[D]. 北京: 华北电力大学, 2021. YAN Fengyuan. Development of Practical Monitoring Circuit For Photovoltaic Modules[D]. Beijing: North China Electric Power University, 2021.
[3]	林剑春, 杨爱军, 沈熠辉. 电致发光缺陷检测仪的成像性能评估[J]. 光学精密工程, 2017, 25(6): 1418-1424. https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201706003.htm LIN Jianchun, YANG Aijun, SHEN Yihui. Evaluation of imaging performance for electroluminescence defect detector[J]. Optics and Precision Engineering, 2017, 25(6): 1418-1424. https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201706003.htm
[4]	LI X, LI W, YANG Q, et al. An unmanned inspection system for multiple defects detection in photovoltaic plants[J]. IEEE Journal of Photovoltaics, 2020, 10(2): 568-576. doi: 10.1109/JPHOTOV.2019.2955183
[5]	Espinosa A R, Bressan M, Giraldo L F. Failure signature classification in solar photovoltaic plants using RGB images and convolutional neural networks – Science Direct[J]. Renewable Energy, 2020, 162: 249-256. doi: 10.1016/j.renene.2020.07.154
[6]	邓堡元, 何赟泽, 王洪金, 等. 光伏电池图像序列的深度学习检测方法[J]. 机械工程学报, 2021, 57(8): 98-106. https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB202108009.htm DENG Baoyuan, HE Yunze, WANG Hongjin, et al. Deep learning inspection for photovoltaic cell image sequence[J]. Journal of Mechanical Engineering, 2021, 57(8): 98-106. https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB202108009.htm
[7]	孙智权, 周奇, 陈震, 等. 基于CMOS图像传感器的太阳能电池缺陷检测系统设计[J]. 仪表技术与传感器, 2018(1): 60-63. doi: 10.3969/j.issn.1002-1841.2018.01.015 SUN Zhiquan, ZHOU Qi, CHEN Zhen, et al. Design of solar cell defects detection system based on CMOS image sensor[J]. Instrument Technique and Sensor, 2018(1): 60-63. doi: 10.3969/j.issn.1002-1841.2018.01.015
[8]	HE Y, DU B, HUANG S. Noncontact electromagnetic induction excited infrared thermography for photovoltaic cells and modules inspection[J]. IEEE Transactions on Industrial Informatics, 2018, 14(12): 5585-5593. doi: 10.1109/TII.2018.2822272
[9]	郭梦浩, 徐红伟. 基于Faster RCNN的红外热图像热斑缺陷检测研究[J]. 计算机系统应用, 2019, 28(11): 265-270. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY201911039.htm GUO Menghao, XU Hongwei. Hot spot defect detection based on infrared thermal image and Faster RCNN[J]. Computer Systems & Applications, 2019, 28(11): 265-270. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY201911039.htm
[10]	谢春宇. 基于红外与可见光双模图像融合的目标检测跟踪技术研究[D]. 南京: 东南大学, 2020. XIE Chunyu. Research on target detection and tracking technology based on infrared and visible dual mode image fusion[D]. Nanjing: Southeast University, 2020.
[11]	SONG H, LIU Z, DU H, et al. Depth-aware salient object detection and segmentation via multiscale discriminative saliency fusion and bootstrap learning[J]. IEEE Transactions on Image Processing, 2017, 26(9): 4204-4216. doi: 10.1109/TIP.2017.2711277
[12]	郑欣悦, 赖际舟, 吕品, 等. 基于红外视觉/激光雷达融合的目标识别与定位方法[J]. 导航定位与授时, 2021, 8(3): 34-41. https://www.cnki.com.cn/Article/CJFDTOTAL-DWSS202103007.htm ZHENG Xinyue, LAI Jizhou, LV Pin, et al. Object detection and positioning method based on infrared vision/lidar fusion[J]. Navigation Positioning & Timing, 2021, 8(3): 34-41. https://www.cnki.com.cn/Article/CJFDTOTAL-DWSS202103007.htm
[13]	康硕, 柯臻铮, 王璇, 等. 基于红外和可见光图像融合的铺丝缺陷检测方法[J]. 航空学报, 2021, 42(12): 425187. https://www.cnki.com.cn/Article/CJFDTOTAL-HKXB202203045.htm KANG Shuo, KE Zhenzheng, WANG Xuan, et al. Detection method of defects in automatic fiber placement based on infrared and visible image fusion[J]. Acta Aeronautica et Astronautica Sinica, 2021, 42(12): 425187. https://www.cnki.com.cn/Article/CJFDTOTAL-HKXB202203045.htm
[14]	LIU D, ZHOU D, NIE R, et al. Infrared and visible image fusion based on convolutional neural network model and saliency detection via hybrid l0-l1 layer decomposition[J]. Journal of Electronic Imaging, 2018, 27(6): 063036. http://www.researchgate.net/publication/329955538_Infrared_and_visible_image_fusion_based_on_convolutional_neural_network_model_and_saliency_detection_via_hybrid_l0-l1_layer_decomposition
[15]	马旗, 朱斌, 程正东, 等. 基于双通道的快速低空无人机检测识别方法[J]. 光学学报, 2019, 39(12): 105-115. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB201912012.htm MA Qi, ZHU Bin, CHENG Zhengdong, et al. Detection and recognition method of fast low-altitude unmanned aerial vehicle based on dual channel[J]. Acta Optica Sinica, 2019, 39(12): 105-115. https://www.cnki.com.cn/Article/CJFDTOTAL-GXXB201912012.htm
[16]	白玉, 侯志强, 刘晓义, 等. 基于可见光图像和红外图像决策级融合的目标检测算法[J]. 空军工程大学学报: 自然科学版, 2020, 21(6): 53-59, 100. https://www.cnki.com.cn/Article/CJFDTOTAL-KJGC202006009.htm BAI Yu, HOU Zhiqiang, LIU Xiaoyi, et al. An object detection algorithm based on decision-level fusion of visible light image and infrared image[J]. Journal of Air Force Engineering University: Natural Science Edition, 2020, 21(6): 53-59, 100. https://www.cnki.com.cn/Article/CJFDTOTAL-KJGC202006009.htm
[17]	唐聪, 凌永顺, 杨华, 等. 基于深度学习的红外与可见光决策级融合检测(英文)[J]. 红外与激光工程, 2019, 48(6): 456-470. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ201906045.htm TANG Cong, LING Yongshun, YANG Hua, et al. Decision-level fusion detection for infrared and visible spectra based on deep learning[J]. Infrared and Laser Engineering, 2019, 48(6): 456-470. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ201906045.htm
[18]	刘家银, 印杰, 牛博威, 等. 海量网站中博彩类违法网站的捕获方法[J]. 数据采集与处理, 2021, 36(5): 1050-1061. https://www.cnki.com.cn/Article/CJFDTOTAL-SJCJ202105021.htm LIU Jiayin, YIN Jie, NIU Bowei, et al. Capture methods of gambling related illegal websites in massive websites[J]. Journal of Data Acquisition and Processing, 2021, 36(5): 1050-1061. https://www.cnki.com.cn/Article/CJFDTOTAL-SJCJ202105021.htm
[19]	DING S, LONG F, FAN H, et al. A novel YOLOv3-tiny network for unmanned airship obstacle detection[C]// IEEE 8th Data Driven Control and Learning Systems Conference (DDCLS), 2019: 277-281.
[20]	HUANG G, LIU Z, Van Der Maaten L, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4700-4708.
[21]	Pech-Pacheco J L, Cristóbal G, Chamorro-Martinez J, et al. Diatom autofocusing in bright field microscopy: a comparative study[C]//Proceedings 15th International Conference on Pattern Recognition, 2000, 3: 314-317.
[22]	JIANG Q, LIU Y, YAN Y, et al. A contour angle orientation for power equipment infrared and visible image registration[J]. IEEE Transactions on Power Delivery, 2020, 36(4): 2559-2569. http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9149795
[23]	DENG J, DONG W, Socher R, et al. Imagenet: A large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009: 248-255.