Standard

Leveraging Crowdsourcing Data for Deep Active Learning An Application : Learning Intents in Alexa. / Yang, Jie; Drake, Thomas; Damianou, Andreas; Maarek, Yoelle.

Proceedings of the 2018 Worls Wide Web Conference, WWW 2018. Geneva, 2018. p. 23-32.

Research output: Scientific - peer-reviewConference contribution

Harvard

Yang, J, Drake, T, Damianou, A & Maarek, Y 2018, Leveraging Crowdsourcing Data for Deep Active Learning An Application: Learning Intents in Alexa. in Proceedings of the 2018 Worls Wide Web Conference, WWW 2018. Geneva, pp. 23-32, WWW 2018, Lyon, France, 23/04/18. DOI: 10.1145/3178876.3186033

APA

Yang, J., Drake, T., Damianou, A., & Maarek, Y. (2018). Leveraging Crowdsourcing Data for Deep Active Learning An Application: Learning Intents in Alexa. In Proceedings of the 2018 Worls Wide Web Conference, WWW 2018 (pp. 23-32). Geneva. DOI: 10.1145/3178876.3186033

Vancouver

Yang J, Drake T, Damianou A, Maarek Y. Leveraging Crowdsourcing Data for Deep Active Learning An Application: Learning Intents in Alexa. In Proceedings of the 2018 Worls Wide Web Conference, WWW 2018. Geneva. 2018. p. 23-32. Available from, DOI: 10.1145/3178876.3186033

Author

Yang, Jie ; Drake, Thomas ; Damianou, Andreas ; Maarek, Yoelle. / Leveraging Crowdsourcing Data for Deep Active Learning An Application : Learning Intents in Alexa. Proceedings of the 2018 Worls Wide Web Conference, WWW 2018. Geneva, 2018. pp. 23-32

BibTeX

@inbook{9cefcd37758743bdbb3456924bd1382a,
title = "Leveraging Crowdsourcing Data for Deep Active Learning An Application: Learning Intents in Alexa",
abstract = "This paper presents a generic Bayesian framework that enables any deep learning model to actively learn from targeted crowds. Ourframework inherits from recent advances in Bayesian deep learning, and extends existing work by considering the targeted crowdsourcing approach, where multiple annotators with unknown expertise contribute an uncontrolled amount (often limited) of annotations. Our framework leverages the low-rank structure in annotations to learn individual annotator expertise, which then helps to infer the true labels from noisy and sparse annotations. It provides a unified Bayesian model to simultaneously infer the true labels and train the deep learning model in order to reach an optimal learning efficacy. Finally, our framework exploits the uncertainty of the deep learning model during prediction as well as the annotators’ estimated expertise to minimize the number of required annotations and annotators for optimally training the deep learning model. We evaluate the effectiveness of our framework for intent classification in Alexa (Amazon’s personal assistant), using both synthetic and real-world datasets. Experiments show that our framework can accurately learn annotator expertise, infer true labels, and effectively reduce the amount of annotations in model training as compared to state-of-the-art approaches. We further discuss the potential of our proposed framework in bridging machine learning and crowdsourcing towards improved human-in-the-loop systems.",
keywords = "Deep Active Learning, Targeted Crowdsourcing, Bayesian Method",
author = "Jie Yang and Thomas Drake and Andreas Damianou and Yoelle Maarek",
note = "Track: Crowdsourcing and Human Computation for the Web",
year = "2018",
doi = "10.1145/3178876.3186033",
pages = "23--32",
booktitle = "Proceedings of the 2018 Worls Wide Web Conference, WWW 2018",

}

RIS

TY - CHAP

T1 - Leveraging Crowdsourcing Data for Deep Active Learning An Application

T2 - Learning Intents in Alexa

AU - Yang,Jie

AU - Drake,Thomas

AU - Damianou,Andreas

AU - Maarek,Yoelle

N1 - Track: Crowdsourcing and Human Computation for the Web

PY - 2018

Y1 - 2018

N2 - This paper presents a generic Bayesian framework that enables any deep learning model to actively learn from targeted crowds. Ourframework inherits from recent advances in Bayesian deep learning, and extends existing work by considering the targeted crowdsourcing approach, where multiple annotators with unknown expertise contribute an uncontrolled amount (often limited) of annotations. Our framework leverages the low-rank structure in annotations to learn individual annotator expertise, which then helps to infer the true labels from noisy and sparse annotations. It provides a unified Bayesian model to simultaneously infer the true labels and train the deep learning model in order to reach an optimal learning efficacy. Finally, our framework exploits the uncertainty of the deep learning model during prediction as well as the annotators’ estimated expertise to minimize the number of required annotations and annotators for optimally training the deep learning model. We evaluate the effectiveness of our framework for intent classification in Alexa (Amazon’s personal assistant), using both synthetic and real-world datasets. Experiments show that our framework can accurately learn annotator expertise, infer true labels, and effectively reduce the amount of annotations in model training as compared to state-of-the-art approaches. We further discuss the potential of our proposed framework in bridging machine learning and crowdsourcing towards improved human-in-the-loop systems.

AB - This paper presents a generic Bayesian framework that enables any deep learning model to actively learn from targeted crowds. Ourframework inherits from recent advances in Bayesian deep learning, and extends existing work by considering the targeted crowdsourcing approach, where multiple annotators with unknown expertise contribute an uncontrolled amount (often limited) of annotations. Our framework leverages the low-rank structure in annotations to learn individual annotator expertise, which then helps to infer the true labels from noisy and sparse annotations. It provides a unified Bayesian model to simultaneously infer the true labels and train the deep learning model in order to reach an optimal learning efficacy. Finally, our framework exploits the uncertainty of the deep learning model during prediction as well as the annotators’ estimated expertise to minimize the number of required annotations and annotators for optimally training the deep learning model. We evaluate the effectiveness of our framework for intent classification in Alexa (Amazon’s personal assistant), using both synthetic and real-world datasets. Experiments show that our framework can accurately learn annotator expertise, infer true labels, and effectively reduce the amount of annotations in model training as compared to state-of-the-art approaches. We further discuss the potential of our proposed framework in bridging machine learning and crowdsourcing towards improved human-in-the-loop systems.

KW - Deep Active Learning

KW - Targeted Crowdsourcing

KW - Bayesian Method

U2 - 10.1145/3178876.3186033

DO - 10.1145/3178876.3186033

M3 - Conference contribution

SP - 23

EP - 32

BT - Proceedings of the 2018 Worls Wide Web Conference, WWW 2018

ER -

ID: 36754455