The Treatment of Ties in AP Correlation

Julián Urbano, Mónica Marrero

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

6 Citations (Scopus)
117 Downloads (Pure)

Abstract

The Kendall tau and AP correlation coefficients are very commonly use to compare two rankings over the same set of items. Even though Kendall tau was originally defined assuming that there are no ties in the rankings, two alternative versions were soon developed to account for ties in two different scenarios: measure the accuracy of an observer with respect to a true and objective ranking, and measure the agreement between two observers in the absence of a true ranking. These two variants prove useful in cases where ties are possible in either ranking, and may indeed result in very different scores. AP correlation was devised to incorporate a top-heaviness component into Kendall tau, penalizing more heavily if differences occur between items at the top of the rankings, making it a very compelling coefficient in Information Retrieval settings. However, the treatment of ties in AP correlation remains an open problem. In this paper we fill this gap, providing closed analytical formulations of AP correlation under the two scenarios of ties contemplated in Kendall tau. In addition,we developed an R package that implements these coefficients.

Original languageEnglish
Title of host publicationProceedings of the 2017 ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2017
Place of PublicationNew York, NY
PublisherAssociation for Computing Machinery (ACM)
Pages321-324
Number of pages4
ISBN (Electronic)978-1-4503-4490-6
DOIs
Publication statusPublished - 2017
EventICTIR 2017: 3rd ACM International Conference on the Theory of Information Retrieval - Amsterdam, Netherlands
Duration: 1 Oct 20174 Oct 2017
Conference number: 3
http://sigir.org/ictir2017/

Conference

ConferenceICTIR 2017
Country/TerritoryNetherlands
CityAmsterdam
Period1/10/174/10/17
Internet address

Bibliographical note

Best Short Paper
Accepted author manuscript

Keywords

  • Average Precision
  • Correlation
  • Evaluation
  • Kendall
  • Ties

Fingerprint

Dive into the research topics of 'The Treatment of Ties in AP Correlation'. Together they form a unique fingerprint.

Cite this