Multimodal fine-grained mixing method and system, device, and storage medium

US11436451B2 · kind B2 · utility

Assignees

• Harbin Institute of Technology (Shenzhen) (Shenzhen Institute of Science and Technology Innovation, Harbin Institute of Technology) · CN
• Dongguan University of Technology · CN

Inventors

• Qing Liao · Lo Wu, CN
• Ye Ding · Cambridge, US
• Binxing Fang · 流花街道, CN
• Xuan Wang · Xuzhou, CN

Key dates

• Filing date: Jan 17, 2022
• Grant date: Sep 6, 2022
• Expiry date: Jan 17, 2042

Classification

• Technology area (CPC G): Physics
• CPC primary: G06V2201/10
• WIPO field: Computer technology
• WIPO sector: Electrical engineering

Abstract

The present disclosure provides a multimodal fine-grained mixing method and system, a device, and a storage medium. The method includes: extracting data features from multimodal graphic and textual data, and obtaining each composition of the data features, the data features including a visual regional feature and a text word feature; performing fine-grained classification on modal information of each composition of the data features, to obtain classification results; and performing inter-modal and intra-modal information fusion on each composition according to the classification results, to obtain a fusion feature. The method enables a multimodal model to utilize a complementary characteristic of the multimodal data, with no influence by irrelevant information.