Bridging the Gap between Click and Relevance for Learning-to-Rank with Minimal Supervision

Recently, unbiased learning-to-rank models have been widely studied to learn a better ranker by eliminating the biases from click data. Toward this goal, existing work mainly focused on estimating the propensity weight to design a specific bias type from click data. From a different perspective, we propose a simple-yet-effective ranking model, namely wLambdaMART, which estimates the confidence of click data with a few labeled data, instead of learning the propensity weight to reduce the bias from click data. We first train a confidence estimator to bridge the gap between biased click data and unbiased relevance. Then, we infer confidence weights for all click data and apply them to LambdaMART to learn a debiased ranker. Practically, since it is found that learning the confidence estimator only requires a few labeled data, it does not incur high labeling costs. Our experimental results show that wLambdaMART outperforms state-of-the-art click models and unbiased learning-to-rank models on the real-world click datasets collected from a commercial search engine.