Abstract

Instead of visualizing LiDAR scanners output just via a 2D screen, developing a web-based application that displays scanner data, collected from a vehicle surrenderingenvironment, and allows data exploration both in web and web virtual reality isinevitable. For better visualization and processing of LiDAR data from multiple LiDARscanners, point cloud registration is necessary. This thesis serves three purposes: 1)To present a point-based rendering approach to visualize and process LiDAR data inreal-time on the web and web virtual reality, 2) 3D parameterizing of object trackingalgorithm, and 3) point cloud registration of tracked objects. The idea is to createinfrastructure for 3D parametrization of existing object tracking implementation whichis used for objects localization and objects approximation with bounding boxes. Fromtracked objects, we would generate targets for registration to bring all the objects to thesame coordinate system. Instead of using artificial targets or selecting point areas, ourapplication is capable of doing the registration automatically.

Project Description

The first prerequisite of point cloud processing and analyzing is point cloud visualization. In order to visualize the point cloud, a web-based solution is proposed in this work. The proposed web application should be able to stream point cloud from LiDAR sensors. Instead of creating 3D models or 2.5D models from point clouds and visualize them via a 2D screen, we proposed to visualize the streamed point cloud in virtual reality. And we had combined it with web application and the point could is visualized in web virtual reality.

A web application that supports web virtual reality is not a new solution to visualize the point cloud. One of the most common issues is how to integrate multiple LiDAR sensors. In some scenarios where the scanners accurate position and orientation are not known and there are not any other external sensors such as GNSS available. Therefore, generated point clouds from sensors should be brought into the one global coordinate system. This process is called point cloud registration. In order to automatize the registration process, the proposed method is to use dynamic objects as targets instead of using artificial markers to bring all the objects to the same coordinate system.

Input data for the registration process is created by the object tracking and detection algorithm. In order to make the point cloud registration more robust the object tracking and detection algorithm should be parameterized.

Implementation

Point Cloud Visualization

The purpose of this section is to create a smooth interactive three-dimensional point cloud visualization experience. The primary technologies used for our approach are WebGL (via Three.js) and Robot Operating System (ROS). In order to visualize the point clouds in the web browsers, Three.js is used. Because, by using a 3D web engine based on high-level JavaScript classes, there is no need for low-level programming and it allows scene navigation, GPU based rendering, and scene-management system. In a scene management system, each component of the scene is added to a scene graph and accessed using a hierarchical function call system. A prerequisite for creating a point cloud in Three.js is receiving all the point coordinates and characteristics from the scanner. After a LiDAR generates the point cloud of a scene, it will serialize the details of generated points with protocol buffers. An external library is used to read the protocol buffer message and create a ROS message from it. On the web application, Roslibjs is used, to receive and interact with ROS functionality. Roslibjs is a JavaScript library for interacting with ROS in web browsers. In order to continuously read data from the LiDAR and stream the point cloud, the web application uses a ROS subscriber. After receiving data in the ROS subscriber, the web application should render the point clouds.

Point cloud registration

Point cloud registration workflow is as following. Two LiDAR sensors are mounted in an environment and scan a scene. With the assumption of having a reasonable overlapping area between two scans, the registration process tries to find the same object in both scans and use that object to register two LiDAR sensors. In order to find the object, the registration process starts to analyze all the dynamic objects in both scenes that have been detected at the same time in each fame in both LiDAR. The trajectory of every dynamic object with previous criteria will be created. Whenever a corresponding trajectory in two LiDAR sensors is found, two trajectories will be aligned and a transformation of alignment is created. The transformation of trajectory alignment will be used to register two point clouds.

3D Parameterization

The detection algorithm creates a list of all the dynamic objects as the output and the tracking algorithm receives that list and assigns an ID to each object for tracking them in next frames. In order to create a resizable tracking area, one or multiple bounding boxes will be created. Subsampling or filtering a point cloud with a bounding box is one of the traditional solutions which has been used continually. There are three parameters to parameterize a bounding box, a 3D vector for the center of a bounding box indicating its position, a 3D vector for the orientation of the bounding box, and the bounding box size for calculating its boundary area. After placing the configured bounding box in an area, all the points outside the bounding box will be ignored and a new point cloud from the points inside the box will be created. This process is called cropping a point cloud.

Results

Point Cloud Visualization

We were not able to mount more than 6 LiDAR sensors because of hardware limitations. Therefore, we performed tests on the web application with simulated massive point clouds. We have recorded a scene with 4 LiDAR sensors and generate approximately 161,000 points with a LiDAR frame rate of 2Hz and we saved it as a rosbag file. In order to create more than 160,000 points, the same point cloud was added to the previous one therefore the number of points will be doubled. The point cloud is colored based on the points ranges to add more constraints in the point cloud processing. The test was performed on Mozilla Firefox.

Point Cloud Registration

Two LiDAR sensors are mounted at a height of 1.5 meters and facing each other in a parking garage. The distance between the two LiDAR sensors is 20 meters.

A person indicated by a blue point cloud walked from one side to the other side. The red dots indicating the center of the bounding box of the person when the person is detected in both LiDAR sensors at the same time. The trajectory of the person will be created by connecting all the red in each point cloud. There is only one dynamic object in the scene and the trajectory of the person is not complex to be analyzed. A complex trajectory to be analyzed in this thesis has following characteristics, 1) LiDAR  loses tracking of an object for 3 or more frames, will cause the trajectory of the object separated into two or more parts, 2) Object occlusion can affect the estimation accuracy of an object position, therefore affecting the accuracy of generated trajectory too, and 3) Noisy point cloud can affect the accuracy of the object bounding box. In the attached thesis, there is another test case containing a complex trajectory. The transformation trajectory alignment is created with RMSE equal to 0.2972 and the right figure is the result of trajectory alignment.

Following image presents the result of transforming the source point cloud with trajectory alignment transformation. The green point cloud is the transformed source point cloud and the white point cloud is the target point cloud. It is the top view of two point clouds. The plane of two point clouds has not been perfectly registered. Because the person started to walk in a straight line and a small area with a constant height and the trajectory alignment is done with one reference point that is the center of the bounding box. Therefore, the trajectory is acting as a line and the point cloud is like a plane that can rotate around the line.

Following image depicts the result of the registration process after two times ICP optimization one for the point cloud segments of person and the other is for two uniformed subsampled raw point clouds. The registration of a plane of two point clouds is solved by ICP. This figure also presented fewer points for two point clouds in comparison with the other point clouds. Because uniformed subsampled has been performed to reduce the number of points.

Conclusion

• Using a point-based rendering with a fixed point size, without any smoothing algorithm will result in some holes in a point cloud.
• Lack of octree in Three.js. One limitation of Three.js for point cloud visualization is the lack of support for out-of-core construction of octree for the rendering of a huge point cloud. Therefore, the web application would not be able to visualize the combined registered point clouds of 10 or more LiDAR sensors in real-time. Because of the lack of octree implementation, it can only be used to visualize point cloud in real-time with less than approximately 1.1 million points.
• An automatic marker-free registration is implemented.
• Parameterize the tracking algorithm with three 3D vectors.

[ PDF] 

[ Slides Kickoff/Final (optional)]