Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)

Ryan Clancy; Nicola Ferro; Claudia Hauff; Jimmy Lin; Tetsuya Sakai; Ze Zhong Wu

Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)

Ryan Clancy, Nicola Ferro, Claudia Hauff, Jimmy Lin, Tetsuya Sakai, Ze Zhong Wu

Web Information Systems

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

5 Citations (Scopus)

109 Downloads (Pure)

Abstract

The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by gathering a community of researchers to jointly develop a common Docker specification and build Docker images that encapsulate a diversity of systems and retrieval models. We articulate the goals of this workshop and describe the "jig" that encodes the Docker specification. In total, 13 teams from around the world submitted 17 images, most of which were designed to produce retrieval runs for the TREC 2004 Robust Track test collection. This exercise demonstrates the feasibility of orchestrating large, community-based replication experiments with Docker technology. We envision OSIRRC becoming an ongoing community-wide effort to ensure experimental replicability and sustained progress on standard test collections.

Original language	English
Title of host publication	OSIRRC 2019 The Open-Source IR Replicability Challenge
Subtitle of host publication	Proceedings of the Open-Source IR Replicability Challenge (OSIRRC 2019) co-located with 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019)
Editors	Ryan Clancy, Nicola Ferro, Claudia Hauff, Jimmy Lin, Tetsuya Sakai , Ze Zhong Wu
Publisher	CEUR-WS
Pages	1-7
Number of pages	7
Publication status	Published - 2019
Event	2019 Open-Source IR Replicability Challenge, OSIRRC 2019 - Paris, France Duration: 25 Jul 2019 → 25 Jul 2019

Publication series

Name	CEUR Workshop Proceedings
Volume	2409
ISSN (Print)	1613-0073

Conference

Conference	2019 Open-Source IR Replicability Challenge, OSIRRC 2019
Country/Territory	France
City	Paris
Period	25/07/19 → 25/07/19

Access to Document

invited01Final published version, 1.59 MBLicence: CC BY

http://ceur-ws.org/Vol-2409/

Cite this

Clancy, R., Ferro, N., Hauff, C., Lin, J., Sakai, T., & Wu, Z. Z. (2019). Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019). In R. Clancy, N. Ferro, C. Hauff, J. Lin, T. Sakai , & Z. Z. Wu (Eds.), OSIRRC 2019 The Open-Source IR Replicability Challenge: Proceedings of the Open-Source IR Replicability Challenge (OSIRRC 2019) co-located with 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019) (pp. 1-7). (CEUR Workshop Proceedings; Vol. 2409). CEUR-WS. http://ceur-ws.org/Vol-2409/

Clancy, Ryan ; Ferro, Nicola ; Hauff, Claudia et al. / Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019). OSIRRC 2019 The Open-Source IR Replicability Challenge: Proceedings of the Open-Source IR Replicability Challenge (OSIRRC 2019) co-located with 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019). editor / Ryan Clancy ; Nicola Ferro ; Claudia Hauff ; Jimmy Lin ; Tetsuya Sakai ; Ze Zhong Wu. CEUR-WS, 2019. pp. 1-7 (CEUR Workshop Proceedings).

@inproceedings{e2d15ec57d044b03964e4de873c9284d,

title = "Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)",

abstract = "The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by gathering a community of researchers to jointly develop a common Docker specification and build Docker images that encapsulate a diversity of systems and retrieval models. We articulate the goals of this workshop and describe the {"}jig{"} that encodes the Docker specification. In total, 13 teams from around the world submitted 17 images, most of which were designed to produce retrieval runs for the TREC 2004 Robust Track test collection. This exercise demonstrates the feasibility of orchestrating large, community-based replication experiments with Docker technology. We envision OSIRRC becoming an ongoing community-wide effort to ensure experimental replicability and sustained progress on standard test collections.",

author = "Ryan Clancy and Nicola Ferro and Claudia Hauff and Jimmy Lin and Tetsuya Sakai and Wu, {Ze Zhong}",

year = "2019",

language = "English",

series = "CEUR Workshop Proceedings",

publisher = "CEUR-WS",

pages = "1--7",

editor = "Ryan Clancy and Nicola Ferro and Claudia Hauff and Lin, {Jimmy } and {Sakai }, {Tetsuya } and Wu, {Ze Zhong }",

booktitle = "OSIRRC 2019 The Open-Source IR Replicability Challenge",

note = "2019 Open-Source IR Replicability Challenge, OSIRRC 2019 ; Conference date: 25-07-2019 Through 25-07-2019",

}

Clancy, R, Ferro, N, Hauff, C, Lin, J, Sakai, T & Wu, ZZ 2019, Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019). in R Clancy, N Ferro, C Hauff, J Lin, T Sakai & ZZ Wu (eds), OSIRRC 2019 The Open-Source IR Replicability Challenge: Proceedings of the Open-Source IR Replicability Challenge (OSIRRC 2019) co-located with 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019). CEUR Workshop Proceedings, vol. 2409, CEUR-WS, pp. 1-7, 2019 Open-Source IR Replicability Challenge, OSIRRC 2019, Paris, France, 25/07/19. <http://ceur-ws.org/Vol-2409/>

Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019). / Clancy, Ryan; Ferro, Nicola; Hauff, Claudia et al.
OSIRRC 2019 The Open-Source IR Replicability Challenge: Proceedings of the Open-Source IR Replicability Challenge (OSIRRC 2019) co-located with 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019). ed. / Ryan Clancy; Nicola Ferro; Claudia Hauff; Jimmy Lin; Tetsuya Sakai ; Ze Zhong Wu. CEUR-WS, 2019. p. 1-7 (CEUR Workshop Proceedings; Vol. 2409).

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)

AU - Clancy, Ryan

AU - Ferro, Nicola

AU - Hauff, Claudia

AU - Lin, Jimmy

AU - Sakai, Tetsuya

AU - Wu, Ze Zhong

PY - 2019

Y1 - 2019

N2 - The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by gathering a community of researchers to jointly develop a common Docker specification and build Docker images that encapsulate a diversity of systems and retrieval models. We articulate the goals of this workshop and describe the "jig" that encodes the Docker specification. In total, 13 teams from around the world submitted 17 images, most of which were designed to produce retrieval runs for the TREC 2004 Robust Track test collection. This exercise demonstrates the feasibility of orchestrating large, community-based replication experiments with Docker technology. We envision OSIRRC becoming an ongoing community-wide effort to ensure experimental replicability and sustained progress on standard test collections.

AB - The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by gathering a community of researchers to jointly develop a common Docker specification and build Docker images that encapsulate a diversity of systems and retrieval models. We articulate the goals of this workshop and describe the "jig" that encodes the Docker specification. In total, 13 teams from around the world submitted 17 images, most of which were designed to produce retrieval runs for the TREC 2004 Robust Track test collection. This exercise demonstrates the feasibility of orchestrating large, community-based replication experiments with Docker technology. We envision OSIRRC becoming an ongoing community-wide effort to ensure experimental replicability and sustained progress on standard test collections.

UR - http://www.scopus.com/inward/record.url?scp=85070667049&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85070667049

T3 - CEUR Workshop Proceedings

SP - 1

EP - 7

BT - OSIRRC 2019 The Open-Source IR Replicability Challenge

A2 - Clancy, Ryan

A2 - Ferro, Nicola

A2 - Hauff, Claudia

A2 - Lin, Jimmy

A2 - Sakai , Tetsuya

A2 - Wu, Ze Zhong

PB - CEUR-WS

T2 - 2019 Open-Source IR Replicability Challenge, OSIRRC 2019

Y2 - 25 July 2019 through 25 July 2019

ER -

Clancy R, Ferro N, Hauff C, Lin J, Sakai T, Wu ZZ. Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019). In Clancy R, Ferro N, Hauff C, Lin J, Sakai T, Wu ZZ, editors, OSIRRC 2019 The Open-Source IR Replicability Challenge: Proceedings of the Open-Source IR Replicability Challenge (OSIRRC 2019) co-located with 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019). CEUR-WS. 2019. p. 1-7. (CEUR Workshop Proceedings).

Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this