Automatic targetless Lidar-camera calibration

Sensor fusion is one of the main challenges in self driving and robotics applications. In this paper we propose an automatic, online and target-less camera-Lidar extrinsic calibration approach. We adopt a structure from motion (SfM) method to generate 3D point clouds from the camera data which can be matched to the Lidar point clouds; thus, we address the extrinsic calibration problem as a registration task in the 3D domain. The core step of the approach is a two-stage transformation estimation: First, we introduce an object level coarse alignment algorithm operating in the Hough space to transform the SfM-based and the Lidar point clouds into a common coordinate system. Thereafter, we apply a control point based nonrigid transformation refinement step to register the point clouds more precisely. Finally, we calculate the correspondences between the 3D Lidar points and the pixels in the 2D camera domain. We evaluated the method in various real-life traffic scenarios in Budapest, Hungary. The results show that our proposed extrinsic calibration approach is able to provide accurate and robust parameter settings on-the-fly.


Worklow of the proposed automatic Lidar-camera calibration approach [1]


Demonstation of the calibration results:

Camera-lidar fusion and multi-object tracking from a moving platform equipped with 4 cameras and a Velodyne HDL64 sensor @ SZTAKI

Demonstration of Lidar-only object tracking from a moving platform:

Object tracking in Velodyne HDL-64 point clouds @ SZTAKI



[1]B. Nagy and Cs. Benedek: ”On-the-Fly Camera and Lidar Calibration,” Remote Sensing, vol 12, no. 7, article 1137, 2020, IF: 4.848 Open Access

[2]B. Nagy, L.A. Kovács and Cs. Benedek: "SFM and Semantic Information Based Online Targetless Camera-Lidar Self-Calibration", IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 22-25 September, 2019

[3]B. Nagy, L.A. Kovács and Cs. Benedek: "Online Targetless End-to-End Camera-LIDAR Self-calibration", International Conference on Machine Vision Applications (MVA), Tokyo, Japan, 27-31 May, 2019


Geo-Information Computing @ Machine Perception Lab.

GeoComp Research Group

Machine Perception Laboratory

SZTAKI main page

GeoComp Demos:

Demo page



GeoComp Group leader: Dr. Csaba Benedek

i4D project manager: Dr. Zsolt Jankó

Head of MPLab: Prof. Tamás Szirányi

MPLab administration: Anikó Vágvölgyi


Kende utca 13-17
H-1111 Budapest, Hungary
Tel: +36 1 279 6194
Fax: +36 1 279 6292

Flag Counter