Learning a robust shared representation space is critical for effective multimedia retrieval, and is increasingly important as multimodal data grows in volume and diversity. The labeled datasets necessary for learning such a space are limited in size and also in coverage of semantic concepts. These limitations constrain performance: a shared representation learned on one dataset may not generalize well to another. We address this issue by building on the insight that, given limited data, it is easier to optimize the semantic structure of a space within a modality, than across modalities. We propose a two-stage shared representation learning framework with intra-modal optimization and subsequent cross-modal transfer learning of semantic structure that produces a robust shared representation space. We integrate multi-task learning into each step, making it possible to leverage multiple datasets, annotated with different concepts, as if they were one large dataset. Large-scale systematic experiments demonstrate improvements over previously reported state-of-the-art methods on cross-modal retrieval tasks.

Original languageEnglish
Title of host publicationProceedings - 2019 IEEE 5th International Conference on Multimedia Big Data, BigMM 2019
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages10
ISBN (Electronic)9781728155272
Publication statusPublished - 1 Sep 2019
Event5th IEEE International Conference on Multimedia Big Data, BigMM 2019 - Singapore, Singapore
Duration: 11 Sep 201913 Sep 2019


Conference5th IEEE International Conference on Multimedia Big Data, BigMM 2019

    Research areas

  • Cross-modal retrieval, Image retrieval, Multi-task learning, Video retrieval

ID: 68565245