Method and system for scene-aware interaction

US11635299B2 · kind B2 · utility

Assignee

• Mitsubishi Electric Research Laboratories, Inc. · US

Inventors

• Chiori Hori · Koganei, JP
• Anoop Cherian · Belmont, US
• Siheng Chen · Boston, US
• Tim Marks · Newton, US
• Jonathan Le Roux · Somerville, US
• Takaaki Hori · Lexington, US
• Bret Harsham · Newton, US
• Anthony Vetro · Arlington, US
• Alan Sullivan · Middleton, US

Key dates

• Filing date: Feb 6, 2020
• Grant date: Apr 25, 2023
• Expiry date: Feb 6, 2040

Classification

• Technology area (CPC G): Physics
• CPC primary: G01S19/485
• WIPO field: Measurement
• WIPO sector: Instruments

Abstract

A navigation system for providing driving instructions to a driver of a vehicle traveling on a route is provided. The driving instructions are generated by executing a multimodal fusion method that comprises extracting features from sensor measurements, annotating the features with directions for the vehicle to follow the route with respect to objects sensed by the sensors, and encoding the annotated features with a multimodal attention neural network to produce encodings. The encodings are transformed into a common latent space, and the transformed encodings are fused using an attention mechanism producing an encoded representation of the scene. The method further comprises decoding the encoded representation with a sentence generation neural network to generate a driving instruction and submitting the driving instruction to an output device.