CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing

Shaoyang Fan, Ujwal Gadiraju, Alessandro Checco, Gianluca Demartini

Research output: Contribution to journalArticleScientificpeer-review

13 Citations (Scopus)
23 Downloads (Pure)

Abstract

Paid micro-task crowdsourcing has gained in popularity partly due to the increasing need for large-scale manually labelled datasets which are often used to train and evaluate Artificial Intelligence systems. Modern paid crowdsourcing platforms use a piecework approach to rewards, meaning that workers are paid for each task they complete, given that their work quality is considered sufficient by the requester or the platform. Such an approach creates risks for workers; their work may be rejected without being rewarded, and they may be working on poorly rewarded tasks, in light of the disproportionate time required to complete them. As a result, recent research has shown that crowd workers may tend to choose specific, simple, and familiar tasks and avoid new requesters to manage these risks.
In this paper, we propose a novel crowdsourcing reward mechanism that allows workers to share these risks and achieve a standardized hourly wage equal for all participating workers. Reward-focused workers can thereby take up challenging and complex HITs without bearing the financial risk of not being rewarded for completed work. We experimentally compare different crowd reward schemes and observe their impact on worker performance and satisfaction. Our results show that 1) workers clearly perceive the benefits of the proposed reward scheme, 2) work effectiveness and efficiency are not impacted as compared to those of the piecework scheme, and 3) the presence of slow workers is limited and does not disrupt the proposed cooperation-based approaches.
Original languageEnglish
Article number132
Number of pages24
JournalProceedings of ACM Human-Computer Interaction (CSCW)
Volume4
DOIs
Publication statusPublished - 2020

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

  • crowdsourcing
  • fairness
  • human computation
  • reward sharing
  • worker behavior

Fingerprint

Dive into the research topics of 'CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing'. Together they form a unique fingerprint.

Cite this