Standard

An axiomatic approach to diagnosing neural IR models. / Rennings, Daniël; Moraes, Felipe; Hauff, Claudia.

Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings. ed. / Norbert Fuhr; Philipp Mayr; Benno Stein; Djoerd Hiemstra; Leif Azzopardi; Claudia Hauff. Springer Verlag, 2019. p. 489-503 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11437 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Harvard

Rennings, D, Moraes, F & Hauff, C 2019, An axiomatic approach to diagnosing neural IR models. in N Fuhr, P Mayr, B Stein, D Hiemstra, L Azzopardi & C Hauff (eds), Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11437 LNCS, Springer Verlag, pp. 489-503, 41st European Conference on Information Retrieval, ECIR 2019, Cologne, Germany, 14/04/19. https://doi.org/10.1007/978-3-030-15712-8_32

APA

Rennings, D., Moraes, F., & Hauff, C. (2019). An axiomatic approach to diagnosing neural IR models. In N. Fuhr, P. Mayr, B. Stein, D. Hiemstra, L. Azzopardi, & C. Hauff (Eds.), Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings (pp. 489-503). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11437 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-15712-8_32

Vancouver

Rennings D, Moraes F, Hauff C. An axiomatic approach to diagnosing neural IR models. In Fuhr N, Mayr P, Stein B, Hiemstra D, Azzopardi L, Hauff C, editors, Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings. Springer Verlag. 2019. p. 489-503. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-15712-8_32

Author

Rennings, Daniël ; Moraes, Felipe ; Hauff, Claudia. / An axiomatic approach to diagnosing neural IR models. Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings. editor / Norbert Fuhr ; Philipp Mayr ; Benno Stein ; Djoerd Hiemstra ; Leif Azzopardi ; Claudia Hauff. Springer Verlag, 2019. pp. 489-503 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

BibTeX

@inproceedings{f0e04c03d4be498db1d64b7ee637f9e6,
title = "An axiomatic approach to diagnosing neural IR models",
abstract = "Traditional retrieval models such as BM25 or language models have been engineered based on search heuristics that later have been formalized into axioms. The axiomatic approach to information retrieval (IR) has shown that the effectiveness of a retrieval method is connected to its fulfillment of axioms. This approach enabled researchers to identify shortcomings in existing approaches and “fix” them. With the new wave of neural net based approaches to IR, a theoretical analysis of those retrieval models is no longer feasible, as they potentially contain millions of parameters. In this paper, we propose a pipeline to create diagnostic datasets for IR, each engineered to fulfill one axiom. We execute our pipeline on the recently released large-scale question answering dataset WikiPassageQA (which contains over 4000 topics) and create diagnostic datasets for four axioms. We empirically validate to what extent well-known deep IR models are able to realize the axiomatic pattern underlying the datasets. Our evaluation shows that there is indeed a positive relation between the performance of neural approaches on diagnostic datasets and their retrieval effectiveness. Based on these findings, we argue that diagnostic datasets grounded in axioms are a good approach to diagnosing neural IR models.",
author = "Dani{\"e}l Rennings and Felipe Moraes and Claudia Hauff",
year = "2019",
doi = "10.1007/978-3-030-15712-8_32",
language = "English",
isbn = "978-3-030-15711-1",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "489--503",
editor = "Norbert Fuhr and Philipp Mayr and Benno Stein and Djoerd Hiemstra and Leif Azzopardi and Claudia Hauff",
booktitle = "Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings",

}

RIS

TY - GEN

T1 - An axiomatic approach to diagnosing neural IR models

AU - Rennings, Daniël

AU - Moraes, Felipe

AU - Hauff, Claudia

PY - 2019

Y1 - 2019

N2 - Traditional retrieval models such as BM25 or language models have been engineered based on search heuristics that later have been formalized into axioms. The axiomatic approach to information retrieval (IR) has shown that the effectiveness of a retrieval method is connected to its fulfillment of axioms. This approach enabled researchers to identify shortcomings in existing approaches and “fix” them. With the new wave of neural net based approaches to IR, a theoretical analysis of those retrieval models is no longer feasible, as they potentially contain millions of parameters. In this paper, we propose a pipeline to create diagnostic datasets for IR, each engineered to fulfill one axiom. We execute our pipeline on the recently released large-scale question answering dataset WikiPassageQA (which contains over 4000 topics) and create diagnostic datasets for four axioms. We empirically validate to what extent well-known deep IR models are able to realize the axiomatic pattern underlying the datasets. Our evaluation shows that there is indeed a positive relation between the performance of neural approaches on diagnostic datasets and their retrieval effectiveness. Based on these findings, we argue that diagnostic datasets grounded in axioms are a good approach to diagnosing neural IR models.

AB - Traditional retrieval models such as BM25 or language models have been engineered based on search heuristics that later have been formalized into axioms. The axiomatic approach to information retrieval (IR) has shown that the effectiveness of a retrieval method is connected to its fulfillment of axioms. This approach enabled researchers to identify shortcomings in existing approaches and “fix” them. With the new wave of neural net based approaches to IR, a theoretical analysis of those retrieval models is no longer feasible, as they potentially contain millions of parameters. In this paper, we propose a pipeline to create diagnostic datasets for IR, each engineered to fulfill one axiom. We execute our pipeline on the recently released large-scale question answering dataset WikiPassageQA (which contains over 4000 topics) and create diagnostic datasets for four axioms. We empirically validate to what extent well-known deep IR models are able to realize the axiomatic pattern underlying the datasets. Our evaluation shows that there is indeed a positive relation between the performance of neural approaches on diagnostic datasets and their retrieval effectiveness. Based on these findings, we argue that diagnostic datasets grounded in axioms are a good approach to diagnosing neural IR models.

UR - http://www.scopus.com/inward/record.url?scp=85064855409&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-15712-8_32

DO - 10.1007/978-3-030-15712-8_32

M3 - Conference contribution

SN - 978-3-030-15711-1

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 489

EP - 503

BT - Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Proceedings

A2 - Fuhr, Norbert

A2 - Mayr, Philipp

A2 - Stein, Benno

A2 - Hiemstra, Djoerd

A2 - Azzopardi, Leif

A2 - Hauff, Claudia

PB - Springer Verlag

ER -

ID: 53627755