Self-supervised label augmentation via input transformations

Self-supervised learning, which learns by con_structing artificial labels given only the input signals, has recently gained considerable at_tention for learning representations with unla_beled datasets, i.e., learning without any human_annotated supervision. In this paper, we show that such a technique can be used to significantly improve the model accuracy even under fully_labeled datasets. Our scheme trains the model to learn both original and self-supervised tasks, but is different from conventional multi-task learn_ing frameworks that optimize the summation of their corresponding losses. Our main idea is to learn a single unified task with respect to the joint distribution of the original and self-supervised labels, i.e., we augment original labels via self_supervision of input transformation. This simple, yet effective approach allows to train models eas_ier by relaxing a certain invariant constraint dur_ing learning the original and self-supervised tasks simultaneously. It also enables an aggregated inference which combines the predictions from different augmentations to improve the predic_tion accuracy. Furthermore, we propose a novel knowledge transfer technique, which we refer to as self-distillation, that has the effect of the aggre_gated inference in a single (faster) inference. We demonstrate the large accuracy improvement and wide applicability of our framework on various fully-supervised settings, e.g., the few-shot and imbalanced classification scenarios.