Mutually Orthogonal Softmax Axes for Cross-Domain Retrieval

In this paper, we introduce and address a crucial but less accentuated problem in cross-domain retrieval task. We first highlight the challenge caused by diversity of inter-class similarities across different domains. For example, bear and teddy bear classes have almost the same instances in a sketch domain whereas they are greatly different in photos. In other words, two classes can be mixed together or even far away in the feature space depending on the modality. A key component of the problem, mapping feature representations of different domains to a common space while preserving semantic similarities, is a tough task due to the domain gap. In order to address such diversity, we introduce a novel feature learning approach by forcing the decision axes of a softmax-based classifier to be mutually orthogonal. Since the feature spaces of different domains share a regularized structure thanks to the orthogonal decision axes, we formulate the mapping to a common space as a simple linear transformation. Our proposed method is extensively evaluated on four different cross-domain retrieval benchmarks consisting of various data modalities, and we found that our method significantly outperforms the state-of-the-art techniques.