Unified optimization for convolutional neural network model inference on integrated graphics processing units

US11797876B1 · kind B1 · utility

Assignee

• AMAZON TECHNOLOGIES, INC. · US

Inventors

• Leyuan Wang · Davis, US
• Yida Wang · Palo Alto, US
• Mu Li · Beijing, CN
• Zhi Chen · Shanghai, CN
• Yizhi Liu · Fremont, US
• Yao-Ting Wang · New Taipei, TW

Key dates

• Filing date: Jun 26, 2019
• Grant date: Oct 24, 2023
• Expiry date: Aug 24, 2042

Classification

• Technology area (CPC G): Physics
• CPC primary: G06N3/08
• WIPO field: Computer technology
• WIPO sector: Electrical engineering

Abstract

Techniques for optimizing and deploying convolutional neural network (CNN) machine learning models for inference using integrated graphics processing units are described. A model compilation system optimizes CNN models using optimized vision-specific operators as well as both graph-level tuning and tensor-level tuning to explore the optimization space for achieving heightened performance. The model compilation system may also implement a heuristic-based two-stage technique for falling back certain operators of CNN models to use CPUs when needed or otherwise beneficial.