Standard

Clarity is a worthwhile quality : On the role of task clarity in microtask crowdsourcing. / Gadiraju, Ujwal; Yang, Jie; Bozzon, Alessandro.

HT'17 Proceedings of the 28th ACM Conference on Hypertext and Social Media. New York : Association for Computing Machinery (ACM), 2017. p. 5-14.

Research output: Scientific - peer-reviewConference contribution

Harvard

Gadiraju, U, Yang, J & Bozzon, A 2017, Clarity is a worthwhile quality: On the role of task clarity in microtask crowdsourcing. in HT'17 Proceedings of the 28th ACM Conference on Hypertext and Social Media. Association for Computing Machinery (ACM), New York, pp. 5-14, 28th ACM Conference on Hypertext and Social Media, HT 2017, Prague, Czech Republic, 4/07/17. DOI: 10.1145/3078714.3078715

APA

Gadiraju, U., Yang, J., & Bozzon, A. (2017). Clarity is a worthwhile quality: On the role of task clarity in microtask crowdsourcing. In HT'17 Proceedings of the 28th ACM Conference on Hypertext and Social Media (pp. 5-14). New York: Association for Computing Machinery (ACM). DOI: 10.1145/3078714.3078715

Vancouver

Gadiraju U, Yang J, Bozzon A. Clarity is a worthwhile quality: On the role of task clarity in microtask crowdsourcing. In HT'17 Proceedings of the 28th ACM Conference on Hypertext and Social Media. New York: Association for Computing Machinery (ACM). 2017. p. 5-14. Available from, DOI: 10.1145/3078714.3078715

Author

Gadiraju, Ujwal ; Yang, Jie ; Bozzon, Alessandro. / Clarity is a worthwhile quality : On the role of task clarity in microtask crowdsourcing. HT'17 Proceedings of the 28th ACM Conference on Hypertext and Social Media. New York : Association for Computing Machinery (ACM), 2017. pp. 5-14

BibTeX

@inbook{8392e9e6408446188e3a9de086e8f5bf,
title = "Clarity is a worthwhile quality: On the role of task clarity in microtask crowdsourcing",
abstract = "Workers of microtask crowdsourcing marketplaces strive to find a balance between the need for monetary income and the need for high reputation. Such balance is often threatened by poorly formulated tasks, as workers attempt their execution despite a sub-optimal understanding of the work to be done. In this paper we highlight the role of clarity as a characterising property of tasks in crowdsourcing. We surveyed 100 workers of the CrowdFlower platform to verify the presence of issues with task clarity in crowdsourcing marketplaces, reveal how crowd workers deal with such issues, and motivate the need for mechanisms that can predict and measure task clarity. Next, we propose a novel model for task clarity based on the goal and role clarity constructs. We sampled 7.1K tasks from the Amazon mTurk marketplace, and acquired labels for task clarity from crowd workers. We show that task clarity is coherently perceived by crowd workers, and is affected by the type of the task. We then propose a set of features to capture task clarity, and use the acquired labels to train and validate a supervised machine learning model for task clarity prediction. Finally, we perform a long-term analysis of the evolution of task clarity on Amazon mTurk, and show that clarity is not a property suitable for temporal characterisation.",
keywords = "Crowd Workers, Crowdsourcing, Goal Clarity, Microtasks, Performance, Prediction, Role Clarity, Task Clarity",
author = "Ujwal Gadiraju and Jie Yang and Alessandro Bozzon",
year = "2017",
doi = "10.1145/3078714.3078715",
pages = "5--14",
booktitle = "HT'17 Proceedings of the 28th ACM Conference on Hypertext and Social Media",
publisher = "Association for Computing Machinery (ACM)",
address = "United States",

}

RIS

TY - CHAP

T1 - Clarity is a worthwhile quality

T2 - On the role of task clarity in microtask crowdsourcing

AU - Gadiraju,Ujwal

AU - Yang,Jie

AU - Bozzon,Alessandro

PY - 2017

Y1 - 2017

N2 - Workers of microtask crowdsourcing marketplaces strive to find a balance between the need for monetary income and the need for high reputation. Such balance is often threatened by poorly formulated tasks, as workers attempt their execution despite a sub-optimal understanding of the work to be done. In this paper we highlight the role of clarity as a characterising property of tasks in crowdsourcing. We surveyed 100 workers of the CrowdFlower platform to verify the presence of issues with task clarity in crowdsourcing marketplaces, reveal how crowd workers deal with such issues, and motivate the need for mechanisms that can predict and measure task clarity. Next, we propose a novel model for task clarity based on the goal and role clarity constructs. We sampled 7.1K tasks from the Amazon mTurk marketplace, and acquired labels for task clarity from crowd workers. We show that task clarity is coherently perceived by crowd workers, and is affected by the type of the task. We then propose a set of features to capture task clarity, and use the acquired labels to train and validate a supervised machine learning model for task clarity prediction. Finally, we perform a long-term analysis of the evolution of task clarity on Amazon mTurk, and show that clarity is not a property suitable for temporal characterisation.

AB - Workers of microtask crowdsourcing marketplaces strive to find a balance between the need for monetary income and the need for high reputation. Such balance is often threatened by poorly formulated tasks, as workers attempt their execution despite a sub-optimal understanding of the work to be done. In this paper we highlight the role of clarity as a characterising property of tasks in crowdsourcing. We surveyed 100 workers of the CrowdFlower platform to verify the presence of issues with task clarity in crowdsourcing marketplaces, reveal how crowd workers deal with such issues, and motivate the need for mechanisms that can predict and measure task clarity. Next, we propose a novel model for task clarity based on the goal and role clarity constructs. We sampled 7.1K tasks from the Amazon mTurk marketplace, and acquired labels for task clarity from crowd workers. We show that task clarity is coherently perceived by crowd workers, and is affected by the type of the task. We then propose a set of features to capture task clarity, and use the acquired labels to train and validate a supervised machine learning model for task clarity prediction. Finally, we perform a long-term analysis of the evolution of task clarity on Amazon mTurk, and show that clarity is not a property suitable for temporal characterisation.

KW - Crowd Workers

KW - Crowdsourcing

KW - Goal Clarity

KW - Microtasks

KW - Performance

KW - Prediction

KW - Role Clarity

KW - Task Clarity

UR - http://www.scopus.com/inward/record.url?scp=85026407379&partnerID=8YFLogxK

U2 - 10.1145/3078714.3078715

DO - 10.1145/3078714.3078715

M3 - Conference contribution

SP - 5

EP - 14

BT - HT'17 Proceedings of the 28th ACM Conference on Hypertext and Social Media

PB - Association for Computing Machinery (ACM)

ER -

ID: 33898366