Standard

Validation methodology for expert-annotated datasets : Event annotation case study. / Inel, Oana; Aroyo, Lora.

2nd Conference on Language, Data and Knowledge, LDK 2019. ed. / Gerard de Melo; Bettina Klimek; Christian Fath; Paul Buitelaar; Milan Dojchinovski; Maria Eskevich; John P. McCrae; Christian Chiarcos. Vol. 70 Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing, 2019. p. 1-15 12.

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Harvard

Inel, O & Aroyo, L 2019, Validation methodology for expert-annotated datasets: Event annotation case study. in G de Melo, B Klimek, C Fath, P Buitelaar, M Dojchinovski, M Eskevich, JP McCrae & C Chiarcos (eds), 2nd Conference on Language, Data and Knowledge, LDK 2019. vol. 70, 12, Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing, pp. 1-15, 2nd Conference on Language, Data and Knowledge, LDK 2019, Leipzig, Germany, 20/05/19. https://doi.org/10.4230/OASIcs.LDK.2019.12

APA

Inel, O., & Aroyo, L. (2019). Validation methodology for expert-annotated datasets: Event annotation case study. In G. de Melo, B. Klimek, C. Fath, P. Buitelaar, M. Dojchinovski, M. Eskevich, J. P. McCrae, ... C. Chiarcos (Eds.), 2nd Conference on Language, Data and Knowledge, LDK 2019 (Vol. 70, pp. 1-15). [12] Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing. https://doi.org/10.4230/OASIcs.LDK.2019.12

Vancouver

Inel O, Aroyo L. Validation methodology for expert-annotated datasets: Event annotation case study. In de Melo G, Klimek B, Fath C, Buitelaar P, Dojchinovski M, Eskevich M, McCrae JP, Chiarcos C, editors, 2nd Conference on Language, Data and Knowledge, LDK 2019. Vol. 70. Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing. 2019. p. 1-15. 12 https://doi.org/10.4230/OASIcs.LDK.2019.12

Author

Inel, Oana ; Aroyo, Lora. / Validation methodology for expert-annotated datasets : Event annotation case study. 2nd Conference on Language, Data and Knowledge, LDK 2019. editor / Gerard de Melo ; Bettina Klimek ; Christian Fath ; Paul Buitelaar ; Milan Dojchinovski ; Maria Eskevich ; John P. McCrae ; Christian Chiarcos. Vol. 70 Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing, 2019. pp. 1-15

BibTeX

@inproceedings{4ede1172ac7a415dbc66d1ec8fe3bd19,
title = "Validation methodology for expert-annotated datasets: Event annotation case study",
abstract = "Event detection is still a difficult task due to the complexity and the ambiguity of such entities. On the one hand, we observe a low inter-annotator agreement among experts when annotating events, disregarding the multitude of existing annotation guidelines and their numerous revisions. On the other hand, event extraction systems have a lower measured performance in terms of F1-score compared to other types of entities such as people or locations. In this paper we study the consistency and completeness of expert-annotated datasets for events and time expressions. We propose a data-agnostic validation methodology of such datasets in terms of consistency and completeness. Furthermore, we combine the power of crowds and machines to correct and extend expert-annotated datasets of events. We show the benefit of using crowd-annotated events to train and evaluate a state-of-the-art event extraction system. Our results show that the crowd-annotated events increase the performance of the system by at least 5.3{\%}.",
keywords = "Crowdsourcing, Event extraction, Human-in-the-loop, Time extraction",
author = "Oana Inel and Lora Aroyo",
year = "2019",
month = "5",
day = "1",
doi = "10.4230/OASIcs.LDK.2019.12",
language = "English",
volume = "70",
pages = "1--15",
editor = "{de Melo}, Gerard and Bettina Klimek and Christian Fath and Paul Buitelaar and Milan Dojchinovski and Maria Eskevich and McCrae, {John P.} and Christian Chiarcos",
booktitle = "2nd Conference on Language, Data and Knowledge, LDK 2019",
publisher = "Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing",

}

RIS

TY - GEN

T1 - Validation methodology for expert-annotated datasets

T2 - Event annotation case study

AU - Inel, Oana

AU - Aroyo, Lora

PY - 2019/5/1

Y1 - 2019/5/1

N2 - Event detection is still a difficult task due to the complexity and the ambiguity of such entities. On the one hand, we observe a low inter-annotator agreement among experts when annotating events, disregarding the multitude of existing annotation guidelines and their numerous revisions. On the other hand, event extraction systems have a lower measured performance in terms of F1-score compared to other types of entities such as people or locations. In this paper we study the consistency and completeness of expert-annotated datasets for events and time expressions. We propose a data-agnostic validation methodology of such datasets in terms of consistency and completeness. Furthermore, we combine the power of crowds and machines to correct and extend expert-annotated datasets of events. We show the benefit of using crowd-annotated events to train and evaluate a state-of-the-art event extraction system. Our results show that the crowd-annotated events increase the performance of the system by at least 5.3%.

AB - Event detection is still a difficult task due to the complexity and the ambiguity of such entities. On the one hand, we observe a low inter-annotator agreement among experts when annotating events, disregarding the multitude of existing annotation guidelines and their numerous revisions. On the other hand, event extraction systems have a lower measured performance in terms of F1-score compared to other types of entities such as people or locations. In this paper we study the consistency and completeness of expert-annotated datasets for events and time expressions. We propose a data-agnostic validation methodology of such datasets in terms of consistency and completeness. Furthermore, we combine the power of crowds and machines to correct and extend expert-annotated datasets of events. We show the benefit of using crowd-annotated events to train and evaluate a state-of-the-art event extraction system. Our results show that the crowd-annotated events increase the performance of the system by at least 5.3%.

KW - Crowdsourcing

KW - Event extraction

KW - Human-in-the-loop

KW - Time extraction

UR - http://www.scopus.com/inward/record.url?scp=85068074385&partnerID=8YFLogxK

U2 - 10.4230/OASIcs.LDK.2019.12

DO - 10.4230/OASIcs.LDK.2019.12

M3 - Conference contribution

VL - 70

SP - 1

EP - 15

BT - 2nd Conference on Language, Data and Knowledge, LDK 2019

A2 - de Melo, Gerard

A2 - Klimek, Bettina

A2 - Fath, Christian

A2 - Buitelaar, Paul

A2 - Dojchinovski, Milan

A2 - Eskevich, Maria

A2 - McCrae, John P.

A2 - Chiarcos, Christian

PB - Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing

ER -

ID: 55545400