Chatbots are text-based conversational agents. Natural Language Understanding (NLU) models are used to extract meaning and intention from user messages sent to chatbots. The user experience of chatbots largely depends on the performance of the NLU model, which itself largely depends on the initial dataset the model is trained with. The training data should cover the diversity of real user requests the chatbot will receive. Obtaining such data is a challenging task even for big corporations. We introduce a generic approach to generate training data with the help of crowd workers, we discuss the approach workflow and the design of crowdsourcing tasks assuring high quality. We evaluate the approach by running an experiment collecting data for 9 different intents. We use the collected training data to train a natural language understanding model. We analyse the performance of the model under different training set sizes for each intent. We provide recommendations on selecting an optimal confidence threshold for predicting intents, based on the cost model of incorrect and unknown predictions.
Original languageEnglish
Title of host publicationWeb Engineering - 18th International Conference, ICWE 2018, Proceedings
PublisherSpringer Verlag
Pages114-128
Number of pages15
Volume10845 LNCS
ISBN (Print)9783319916613
DOIs
Publication statusPublished - 2018
Event18th International Conference on Web Engineering, ICWE 2018 - Caceres, Spain
Duration: 5 Jun 20188 Jun 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10845 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference18th International Conference on Web Engineering, ICWE 2018
CountrySpain
CityCaceres
Period5/06/188/06/18

    Research areas

  • Conversational agents, Crowdsourcing, Natural language understanding

ID: 45467561