Matching images and text with multi-modal tensor fusion and re-ranking

Tan Wang, Alan Hanjalic, Xing Xu*, Heng Tao Shen, Yang Yang, Jingkuan Song

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

69 Citations (Scopus)
99 Downloads (Pure)

Abstract

A major challenge in matching images and text is that they have intrinsically different data distributions and feature representations. Most existing approaches are based either on embedding or classification, the first one mapping image and text instances into a common embedding space for distance measuring, and the second one regarding image-text matching as a binary classification problem. Neither of these approaches can, however, balance the matching accuracy and model complexity well. We propose a novel framework that achieves remarkable matching performance with acceptable model complexity. Specifically, in the training stage, we propose a novel Multi-modal Tensor Fusion Network (MTFN) to explicitly learn an accurate image-text similarity function with rank-based tensor fusion rather than seeking a common embedding space for each image-text instance. Then, during testing, we deploy a generic Cross-modal Re-ranking (RR) scheme for refinement without requiring additional training procedure. Extensive experiments on two datasets demonstrate that our MTFN-RR consistently achieves the state-of-the-art matching performance with much less time complexity.

Original languageEnglish
Title of host publicationMM 2019 - Proceedings of the 27th ACM International Conference on Multimedia
PublisherAssociation for Computing Machinery (ACM)
Pages12-20
Number of pages9
ISBN (Electronic)9781450368896
DOIs
Publication statusPublished - 15 Oct 2019
Event27th ACM International Conference on Multimedia, MM 2019 - Nice, France
Duration: 21 Oct 201925 Oct 2019

Conference

Conference27th ACM International Conference on Multimedia, MM 2019
Country/TerritoryFrance
CityNice
Period21/10/1925/10/19

Bibliographical note

Accepted author manuscript

Keywords

  • Cross-modal re-ranking
  • Image-text matching
  • Tensor fusion

Fingerprint

Dive into the research topics of 'Matching images and text with multi-modal tensor fusion and re-ranking'. Together they form a unique fingerprint.

Cite this