Calibration Between Sparse LIDAR and Visible/Infrared Imaging Systems
-
摘要: 激光雷达与成像系统之间的位姿标定是激光点云与图像像素进行融合的前提。目前主流的离线标定方法中,普通棋盘格标定板用于64线及以上的激光雷达时效果较好,而用于16线激光雷达时由于其数据稀疏而导致误差较大。而且,涉及红外成像系统的标定时,需要特制的棋盘格来获得发射率差异。本文针对稀疏激光雷达点云数据较少的问题,研究了可以同时标定激光雷达与可见光、红外成像系统的方法,设计了菱形九孔标定板,并提出几何约束损失函数来优化特征点的坐标。最后,分别使用红外和可见光成像系统与16线激光雷达进行标定,实验结果表明,平均重投影误差均在3个像素之内,取得了较好的效果。本文方法还能用于稀疏激光雷达与可见光-红外多波段成像系统的标定。Abstract: Pose estimation between LIDAR and imaging system is the prerequisite for the data fusion. Among current mainstream off-line calibration methods, common checkerboard is generally effective for 64-line and above LIDAR, but not for 16-line LIDAR due to its sparse data and will lead to large error. Furthermore, when involving calibration of infrared imaging system, specially-made checkerboard is needed to produce difference of emissivity. Aiming at the problem of less information provided by sparse LIDARs, we propose a new calibration method that can jointly calibrate LIDAR and visible/infrared imaging systems. A novel diamond-shaped nine-hole calibration board is designed, and a geometric constraint loss function is proposed to optimize the coordinates of feature points. Finally, the infrared and visible light imaging systems are used respectively, to calibrate with 16-line LIDAR. Good results are achieved and show that, all the average reprojection error is within 3 pixels. The proposed method can also be used in calibration of multi-band imaging systems that include sparse LIDAR, visible imaging system and infrared imaging system.
-
Key words:
- sparse LIDAR /
- pose estimation /
- calibration /
- calibration board /
- infrared imaging
-
表 1 两种方法初值误差对比
Table 1. Comparison of initial value errors of two methods
3D Constraint Method (a) PnP Method (b) Number Rotary axis error(a) Angular error (a) Translation error (a) Rotary axis error(b) Angular error (b) Translation error (b) 1 6.0×10−5 5×10−5 2.3×10−5 0.890 0.165 0.376 2 6.4×10−6 8×10−5 3.1×10−5 0.593 0.067 0.434 3 2.9×10−5 8×10−5 5.3×10−5 0.283 0.038 0.387 4 6.1×10−5 3×10−5 4.9×10−5 0.733 0.048 0.218 5 5.7×10−5 8×10−5 7.7×10−5 0.710 0.070 0.520 6 3.4×10−5 1×10−4 7.2×10−5 0.904 0.022 0.615 7 5.9×10−5 7×10−5 5.5×10−5 0.278 0.036 0.497 8 3.9×10−5 3×10−5 3.9×10−5 0.820 0.021 0.581 9 5.3×10−5 9×10−6 2.9×10−5 0.718 0.027 0.480 Average 4.4×10−5 6×10−6 4.7×10−5 0.658 0.054 0.456 表 2 本文方法的仿真结果
Table 2. The simulation results of our method
Number Lidar loss
before optimizationLidar loss
after optimizationCamera loss
before optimizationCamera loss
after optimizationRotary axis error Angular error Translation error 1 4.9377 0.0147 4.3755 0.0010 5.9×10−3 0.0025 3.6×10−3 2 4.6482 0.0333 4.6580 0.0030 4.2×10−3 0.0001 6.9×10−3 3 4.1757 0.0054 4.2307 0.0052 5.310−3 0.0003 5.8×10−3 4 4.4703 0.0009 4.6283 0.0034 4.3×10−3 0.0046 8.9×10−3 5 4.7989 0.0010 4.4958 0.0011 7.1×10−3 0.0082 79×10−3 6 4.8751 0.0054 4.1330 0.0018 7.1×10−3 0.0162 6.9×10−3 7 4.2961 0.0035 4.4130 0.0095 8.4×10−3 0.0224 5.1×10−3 8 4.4903 0.0013 4.4346 0.0073 8.9×10−3 0.0043 1.1×10−2 9 4.4642 0.0018 4.5932 0.0153 7.3×10−3 0.0142 9.9×10−3 Average 4.5729 0.0075 4.4402 0.0053 6.5×10−3 0.0081 7.3×10−3 表 3 可见光系统实验结果
Table 3. Experimental results of visible imaging system
No. Rotation matrix Rlc (c) Translation vector tlc(c) Reprojection error (c)/pixel Rotation matrix Rlc(d) Translation vector tlc(d) Reprojection error (d)/pixel 1 $ \left[ {\begin{array}{*{20}{c}} {0.9905}&{ - 0.1364}&{ - 0.0192} \\ { - 0.0138}&{0.0403}&{ - 0.9991} \\ {0.1371}&{0.9898}&{0.0381} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0172} \\ {0.0660} \\ {0.0116} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:4.7595} \\ {y:2.7523} \end{array} $ $ \left[ {\begin{array}{*{20}{c}} {0.9882}&{ - 0.1524}&{ - 0.0156} \\ { - 0.0132}&{0.0166}&{ - 0.9998} \\ {0.1526}&{0.9882}&{0.0144} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0172} \\ {0.0660} \\ {0.0116} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:2.3598} \\ {y:2.0767} \end{array} $ 2 $ \left[ {\begin{array}{*{20}{c}} {0.9906}&{ - 0.1358}&{ - 0.0111} \\ { - 0.0104}&{0.0063}&{ - 0.9999} \\ {0.1359}&{0.9907}&{0.0049} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0165} \\ {0.0188} \\ {0.0200} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:3.5315} \\ {y:5.4321} \end{array} $ $ \left[ {\begin{array}{*{20}{c}} {0.9859}&{ - 0.1661}&{ - 0.0205} \\ { - 0.0149}&{0.0350}&{ - 0.9993} \\ {0.1667}&{0.9855}&{0.0320} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0165} \\ {0.0188} \\ {0.0200} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:2.6225} \\ {y:2.0955} \end{array} $ 3 $ \left[ {\begin{array}{*{20}{c}} {0.9924}&{ - 0.1224}&{ - 0.0114} \\ { - 0.0100}&{0.0119}&{ - 0.9998} \\ {0.1225}&{0.9924}&{0.0106} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0233} \\ {0.0664} \\ {0.0262} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:3.2698} \\ {y:4.5639} \end{array} $ $ \left[ {\begin{array}{*{20}{c}} {0.9883}&{ - 0.1513}&{ - 0.0204} \\ { - 0.0179}&{0.0176}&{ - 0.9997} \\ {0.1516}&{0.9883}&{0.0147} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0233} \\ {0.0663} \\ {0.0262} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:2.2155} \\ {y:2.0061} \end{array} $ 4 $ \left[ {\begin{array}{*{20}{c}} {0.9904}&{ - 0.1377}&{ - 0.0141} \\ { - 0.0115}&{0.0199}&{ - 0.9997} \\ {0.1379}&{0.9903}&{0.0182} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0219} \\ {0.0636} \\ {0.0244} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:3.0629} \\ {y:4.1338} \end{array} $ $ \left[ {\begin{array}{*{20}{c}} {0.9883}&{ - 0.1512}&{ - 0.0176} \\ { - 0.0151}&{0.0183}&{ - 0.9997} \\ {0.1515}&{0.9883}&{0.0158} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0219} \\ {0.0635} \\ {0.0244} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:2.0246} \\ {y:1.9887} \end{array} $ 5 $ \left[ {\begin{array}{*{20}{c}} {0.9905}&{ - 0.1374}&{ - 0.0092} \\ { - 0.0104}&{0.0079}&{ - 0.9999} \\ {0.1373}&{0.9905}&{0.0093} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0130} \\ {0.0569} \\ {0.0258} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:3.4974} \\ {y:4.9327} \end{array} $ $ \left[ {\begin{array}{*{20}{c}} {0.9881}&{ - 0.1537}&{ - 0.0057} \\ { - 0.0027}&{0.0195}&{ - 0.9998} \\ {0.1538}&{0.9879}&{0.0189} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0130} \\ {0.0569} \\ {0.0257} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:2.3178} \\ {y:2.0198} \end{array} $ Average reprojection error $ \begin{array}{*{20}{c}} {x:3.6242} \\ {y:4.3629} \end{array} $ Average reprojection error $ \begin{array}{*{20}{c}} {x:2.3080} \\ {y:2.0374} \end{array} $ 表 4 红外系统实验结果
Table 4. Experimental results of infrared imaging system
No. Rotation matrix Rlc (c) Translation vector tlc (c) Reprojection error (c)/pixel Rotation matrix Rlc (d) Translation vector tlc(d) Reprojection error (d)/pixel 1 $ \left[ {\begin{array}{*{20}{c}} {0.6866}&{0.7269}&{0.0072} \\ {0.0169}&{ - 0.0060}&{ - 0.9998} \\ { - 0.7268}&{0.6866}&{ - 0.0164} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0395} \\ {0.0140} \\ {0.0235} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:3.0472} \\ {y:3.0035} \end{array} $ $ \left[ {\begin{array}{*{20}{c}} {0.6358}&{0.7717}&{0.0134} \\ {0.1654}&{ - 0.1193}&{ - 0.9789} \\ { - 0.7539}&{0.6247}&{ - 0.2034} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0395} \\ {0.0140} \\ {0.0235} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:2.4367} \\ {y:2.1505} \end{array} $ 2 $ \left[ {\begin{array}{*{20}{c}} {0.6701}&{0.7339}&{ - 0.0379} \\ {0.0534}&{0.0342}&{ - 0.9979} \\ { - 0.0503}&{ - 0.0796}&{ - 1.0010} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.2037} \\ { - 0.3452} \\ { - 0.0601} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:3.5839} \\ {y:4.3602} \end{array} $ $ \left[ {\begin{array}{*{20}{c}} {0.6327}&{0.7743}&{0.0100} \\ {0.1600}&{ - 0.1181}&{ - 0.9800} \\ { - 0.7577}&{0.6216}&{ - 0.1986} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0486} \\ {0.0016} \\ {0.0297} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:2.6007} \\ {y:2.2385} \end{array} $ 3 $ \left[ {\begin{array}{*{20}{c}} {0.6765}&{0.7363}&{0.0092} \\ {0.0168}&{ - 0.0029}&{ - 0.9998} \\ { - 0.7362}&{0.6766}&{ - 0.0143} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0803} \\ { - 0.0819} \\ {0.0745} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:3.3312} \\ {y:3.3278} \end{array} $ $ \left[ {\begin{array}{*{20}{c}} {0.6229}&{0.7822}&{0.0100} \\ {0.1317}&{ - 0.0922}&{ - 0.9870} \\ { - 0.7711}&{0.6161}&{ - 0.1604} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0803} \\ { - 0.0819} \\ {0.0745} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:2.8646} \\ {y:2.1971} \end{array} $ 4 $ \left[ {\begin{array}{*{20}{c}} {0.6776}&{0.7354}&{0.0112} \\ {0.0343}&{ - 0.0163}&{ - 0.9992} \\ { - 0.7346}&{0.6775}&{ - 0.0363} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.2447} \\ { - 0.0381} \\ {0.0815} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:3.2049} \\ {y:3.1438} \end{array} $ $ \left[ {\begin{array}{*{20}{c}} {0.5524}&{0.8332}&{0.0249} \\ {0.1678}&{ - 0.0819}&{ - 0.9824} \\ { - 0.8165}&{0.5469}&{ - 0.1851} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.2447} \\ { - 0.0381} \\ {0.0815} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:2.4126} \\ {y:2.3031} \end{array} $ 5 $ \left[ {\begin{array}{*{20}{c}} {0.6686}&{0.7436}&{0.0077} \\ { - 0.0017}&{0.0118}&{ - 0.9999} \\ { - 0.7436}&{0.6685}&{0.0092} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0673} \\ {0.0042} \\ {0.0303} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:3.0379} \\ {y:3.7253} \end{array} $ $ \left[ {\begin{array}{*{20}{c}} {0.6255}&{0.7802}&{0.0075} \\ {0.1605}&{ - 0.1192}&{ - 0.9798} \\ { - 0.7635}&{0.6141}&{ - 0.1998} \end{array}} \right] $ $ \left[ {\begin{array}{*{20}{c}} { - 0.0673} \\ {0.0042} \\ {0.0303} \end{array}} \right] $ $ \begin{array}{*{20}{c}} {x:2.6443} \\ {y:2.1625} \end{array} $ Average reprojection error $ \begin{array}{*{20}{c}} {x:3.2410} \\ {y:3.5121} \end{array} $ Average reprojection error $ \begin{array}{*{20}{c}} {x:2.5918} \\ {y:2.2103} \end{array} $ -
[1] Fremont V, Bonnifait P. Extrinsic calibration between a multi-layer lidar and a camera[C]//International Conference on Multisensor Fusion and Integration for Intelligent Systems of IEEE, 2008: 214-219. [2] 陈东. 三维激光和单目视觉间的联合标定与数据融合[D]. 大连: 大连理工大学, 2009.CHEN Dong. Calibration and Data Fusion between 3D-Laser Scanner and Monocular Vision[D]. Dalian: Dalian University of Technology, 2009. [3] ZHOU L, DENG Z. Extrinsic calibration of a camera and a lidar based on decoupling the rotation from the translation[C]//Intelligent Vehicles Symposium of IEEE, 2012: 642-648. . [4] ZHOU L, LI Z, Kaess M. Automatic extrinsic calibration of a camera and a 3d lidar using line and plane correspondences[C]//International Conference on Intelligent Robots and Systems (IROS)of IEEE, 2018: 5562-5569. [5] Geiger A, Moosmann F, Car O , et al. Automatic camera and range sensor calibration using a single shot[C]//Proceedings of IEEE International Conference on Robotics and Automation, 2012: 3936-3943. [6] Horaud R, Dornaika F. Hand-eye calibration[J]. The International Journal of Robotics Research, 1995, 14(3): 195-210. [7] Ishikawa R, Oishi T, Ikeuchi K. Lidar and camera calibration using motions estimated by sensor fusion odometry[C]//International Conference on Intelligent Robots and Systems(IROS) of IEEE, 2018: 7342-7349. [8] Schneider N, Piewak F, Stiller C, et al. RegNet: multimodal sensor registration using deep neural networks[C]//Intelligent Vehicles Symposium (IV) of IEEE, 2017: 1803-1810. [9] Iyer G, Ram R K, Murthy J K, et al. CalibNet: geometrically supervised extrinsic calibration using 3D spatial transformer networks[C]// International Conference on Intelligent Robots and Systems (IROS) of IEEE, 2018: 1110-1117. [10] ZHANG Z. A flexible new technique for camera calibration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11): 1330-1334.