Decision-Theoretic Control of Crowd-Sourced Workflows

This 2010 article focuses on crowd-sourced information and particularly the utility of artificial intelligence (AI) in crowdsourcing workflows. The authors raise the following question for AI: Could an autonomous agent control these workflows without human intervention, yielding better results than today’s state of the art methods as a fixed control program? The article describes a planner called TURKONTROL and formulates workflow control as a decision-theoretical optimization problem. The authors problematize that workflows trade off the implicit quality of a solution artifact against the cost of human workers to achieve the desired results. The authors lay the mathematical framework to govern the various decisions at each point in a popular class of workflows. Based on their analysis, the authors implement their workflow algorithm and present experiments that demonstrate that TURKONTROL obtains much higher utilities than popular fixed policies.

This article will be of use to PVE researchers and practitioners interested in crowd-sourced information and particularly the application of artificial intelligence to crowdsourcing. The authors argue that crowdsourcing is a framework in which human intelligence tasks are outsourced to a crowd of unknown people or “workers” as an open call. The use of crowd-sourced information has become increasingly popular with employers or “requesters”, who use it to solve a wide variety of jobs, such as dictation transcription, content screening, and more. In order to achieve quality results, requesters often subdivide a large task into a chain of bite-sized subtasks that are combined into a complex, iterative workflow in which workers check and improve each other’s results.