Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks

US8554605B2 · kind B2 · utility

Assignee

• CrowdFlower, Inc. · US

Inventors

• David Trotter Oleson · San Francisco, US
• Alexander Sorokin · Foster City, US
• Greg Laughlin · San Francisco, US
• Vaughn Hester · San Francisco, US
• John Le · Anaheim, US
• Christopher R. Van Pelt · San Francisco, US
• Lukas Biewald · Cambridge, US

Key dates

• Filing date: Oct 17, 2011
• Grant date: Oct 8, 2013
• Expiry date: Oct 17, 2031

Classification

• Technology area (CPC G): Physics
• CPC primary: G09B19/00
• WIPO field: Control
• WIPO sector: Instruments

Abstract

Systems and methods for evaluating a worker in performing crowd sourced tasks and providing in-task training are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, for selecting a job distributed through a job distribution platform for workers to work on, for use to generate a test task, the job being associated with a known correct result, associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task, and/or presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker. The job distribution platform crowd sources tasks online to workers to work on via their respective computing devices.