Vision-LiDAR fusion method and system based on deep canonical correlation analysis

US11532151B2 · kind B2 · utility

Assignee

• TSINGHUA UNIVERSITY · CN

Inventors

• Xinyu Zhang · Changzhou City, CN
• Li Wang · Beijing, CN
• Jun Li · Cranbury, US
• Lijun Zhao · Beijing, CN
• Zhiwei Li · Lo Wu, CN
• Shiyan Zhang · Beijing, CN
• Lei Yang · Shanghai, CN
• Xingang Wu · Beijing, CN
• Hanwen Gao · Beijing, CN
• Lei Zhu · Beijing, CN
• Tianlei Zhang · Beijing, CN

Key dates

• Filing date: Apr 29, 2022
• Grant date: Dec 20, 2022
• Expiry date: Apr 29, 2042

Classification

• Technology area (CPC G): Physics
• CPC primary: G06T2207/20084
• WIPO field: Computer technology
• WIPO sector: Electrical engineering

Abstract

A vision-LiDAR fusion method and system based on deep canonical correlation analysis are provided. The method comprises: collecting RGB images and point cloud data of a road surface synchronously; extracting features of the RGB images to obtain RGB features; performing coordinate system conversion and rasterization on the point cloud data in turn, and then extracting features to obtain point cloud features; inputting point cloud features and RGB features into a pre-established and well-trained fusion model at the same time, to output feature-enhanced fused point cloud features, wherein the fusion model fuses RGB features to point cloud features by using correlation analysis and in combination with a deep neural network; and inputting the fused point cloud features into a pre-established object detection network to achieve object detection. A similarity calculation matrix is utilized to fuse two different modal features.