DOI

Workers of microtask crowdsourcing marketplaces strive to find a balance between the need for monetary income and the need for high reputation. Such balance is often threatened by poorly formulated tasks, as workers attempt their execution despite a sub-optimal understanding of the work to be done. In this paper we highlight the role of clarity as a characterising property of tasks in crowdsourcing. We surveyed 100 workers of the CrowdFlower platform to verify the presence of issues with task clarity in crowdsourcing marketplaces, reveal how crowd workers deal with such issues, and motivate the need for mechanisms that can predict and measure task clarity. Next, we propose a novel model for task clarity based on the goal and role clarity constructs. We sampled 7.1K tasks from the Amazon mTurk marketplace, and acquired labels for task clarity from crowd workers. We show that task clarity is coherently perceived by crowd workers, and is affected by the type of the task. We then propose a set of features to capture task clarity, and use the acquired labels to train and validate a supervised machine learning model for task clarity prediction. Finally, we perform a long-term analysis of the evolution of task clarity on Amazon mTurk, and show that clarity is not a property suitable for temporal characterisation.

Original languageEnglish
Title of host publicationHT'17 Proceedings of the 28th ACM Conference on Hypertext and Social Media
Place of PublicationNew York
PublisherAssociation for Computing Machinery (ACM)
Pages5-14
Number of pages10
ISBN (Electronic)978-1-4503-4708-2
DOIs
StatePublished - 2017
Event28th ACM Conference on Hypertext and Social Media, HT 2017 - Prague, Czech Republic
Duration: 4 Jul 20177 Jul 2017

Conference

Conference28th ACM Conference on Hypertext and Social Media, HT 2017
CountryCzech Republic
CityPrague
Period4/07/177/07/17

    Research areas

  • Crowd Workers, Crowdsourcing, Goal Clarity, Microtasks, Performance, Prediction, Role Clarity, Task Clarity

ID: 33898366