Multi-view deep neural network for LiDAR perception

US11915493B2 · kind B2 · utility

Assignee

• NVIDIA Corporation · US

Inventors

• Nikolai Smolyanskiy · Seattle, US
• Ryan Oldja · Redmond, US
• Ke Chen · Sunnyvale, US
• Alexander Popov · Kirkland, US
• Joachim Pehserl · Lynnwood, US
• Ibrahim Eden · Redmond, US
• Tilman Wekel · Santa Clara, US
• David Wehr · Redmond, US
• Ruchi Bhargava · Redmond, US
• David Nister · Bellevue, US

Key dates

• Filing date: Aug 25, 2022
• Grant date: Feb 27, 2024
• Expiry date: Aug 25, 2042

Classification

• Technology area (CPC —): General

Abstract

A deep neural network(s) (DNN) may be used to detect objects from sensor data of a three dimensional (3D) environment. For example, a multi-view perception DNN may include multiple constituent DNNs or stages chained together that sequentially process different views of the 3D environment. An example DNN may include a first stage that performs class segmentation in a first view (e.g., perspective view) and a second stage that performs class segmentation and/or regresses instance geometry in a second view (e.g., top-down). The DNN outputs may be processed to generate 2D and/or 3D bounding boxes and class labels for detected objects in the 3D environment. As such, the techniques described herein may be used to detect and classify animate objects and/or parts of an environment, and these detections and classifications may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.