CAFTTA: Mitigating Unseen Class Forgetting in Test-Time Adaptation with Knowledge Fusion

In real-world applications, deployed deep learning models often encounter test data from domains significantly different from their training data. This domain shift typically results in performance degradation for conventional deep learning models. Test-time adaptation (TTA) has emerged as a promising research direction to address this challenge by adapting models during test-time to unpredictable domains. However, in real-world scenarios, data could exhibit different class distributions as well as domains. Consequently, models are deployed in environments where they temporarily cannot observe data of a particular class. Under these conditions, test-time adaptation methods exhibit catastrophic forgetting of unseen class, significantly compromising their ability to generalize. To address this critical issue, we propose Class Anti-Forgetting Test-Time Adaptation (CAFTTA). Our method shares domain knowledge by weight sharing between the original source model and the test-time adaptation model, and preserves knowledge of unseen classes through entropy-based knowledge fusion. Our method minimizes performance degradation on seen classes while preventing unseen class forgetting. Experimentally, we demonstrate that our method significantly mitigates the unseen class forgetting problem faced by existing test time adaptation methods in both static and continuous domain scenarios.