Coding activity task (CAT) evaluation for source code generators
US12254293B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Oct 6, 2023 |
| Grant date | Mar 18, 2025 |
| Priority date | — |
| Expiry date | Oct 6, 2043 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F11/3688
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Solutions for evaluating source code generators use offline and online evaluation stages. Offline evaluation includes separating each of a plurality of input passages of software code into a plurality of constituent blocks. Each code generator (of a plurality of code generators) generates an equivalent block corresponding to each constituent block. A coding score is determined for each equivalent block (for each code generator), and the coding scores are aggregated across the equivalent blocks to provide an aggregate score for each code generator. A ranking of the aggregate scores is used to down-select to a fewer number of code generators for online evaluation. For this stage, the code generators output passages of software code, and user acceptance of the code generators' outputs may be used for further ranking and down-selection. Some examples weight the coding score according to a code utility estimate of the constituent blocks for which equivalent blocks are generated.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.