Patent · US Active

Inter-training of pre-trained transformer-based language models using partitioning and classification

US12093645B2 · kind B2 · utility

0Cited by

0References

14Claims

0Family size

Assignee

International Business Machines Corporation · US

Inventors

Eyal Shnarch · Nes Ziona, IL
Ariel Gera · Haifa, IL
Alon Halfon · Rishon LeZion, IL
Lena Dankin · Tel Aviv-Yafo, IL
Leshem Choshen · Haifa, IL
Ranit Aharonov · Ramat HaSharon, IL
Noam Slonim · Jerusalem, IL

Key dates

Filing date	Sep 14, 2021
Grant date	Sep 17, 2024
Priority date	—
Expiry date	Dec 1, 2042

Classification

Technology area (CPC G)Physics
CPC primaryG10L25/30
WIPO fieldComputer technology
WIPO sectorElectrical engineering

Abstract

An example system includes a processor to pre-train a transformer-based language model on a general domain. The processor can inter-train the pre-trained transformer-based language model using partitioning and classification to generate an inter-trained transformer-based pre-trained language model. The processor can then fine-tune the inter-trained transformer-based pre-trained language model on a target task to generate a fine-tuned transformer-based language model.

Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.