Systems and methods for an end-to-end evaluation and testing framework for task-oriented dialog systems
US12112138B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Jun 2, 2022 |
| Grant date | Oct 8, 2024 |
| Priority date | — |
| Expiry date | May 17, 2043 |
Classification
- Technology area (CPC H)Electricity
- CPC primaryH04L51/02
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Embodiments provide a software framework for evaluating and troubleshooting real-world task-oriented bot systems. Specifically, the evaluation framework includes a generator that infers dialog acts and entities from bot definitions and generates test cases for the system via model-based paraphrasing. The framework may also include a simulator for task-oriented dialog user simulation that supports both regression testing and end-to-end evaluation. The framework may also include a remediator to analyze and visualize the simulation results, remedy some of the identified issues, and provide actionable suggestions for improving the task-oriented dialog system.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.