Does transparency matter when an AI system meets performance expectations? An experiment with an online dating site

Powered by sophisticated algorithms, artificial intelligence (AI) systems are notorious for being opaque, prompting demands for more algorithmic transparency. However, empirical evidence reveals inconsistent effects of transparency on user trust, suggesting critical boundary conditions. Drawing on the Expectation Confirmation Model and Heuristic-Systematic Model, we propose that the effectiveness of providing algorithmic transparency depends on whether the AI system's performance meets, exceeds, or falls short of users’ expectations. We tested this through a 3 (Algorithmic transparency: high vs. low vs. absent) × 3 (Expectation confirmation: positive vs. simple vs. negative) between-subjects online experiment (N = 227), in which participants interacted with an online dating site driven by date-matching algorithms. Results revealed that detailed algorithmic explanations enhanced perceived understanding and trust when system performance either exceeded or fell short of expectations, while concise explanations proved more effective when performance met expectations. These findings advance our theoretical understanding of algorithmic transparency and provide practical implications for designing adaptive explainable AI systems that adjust transparency levels based on performance outcomes.