Self-Supervised Pretraining Strategies for Robust Transfer Learning under Domain and Distributional Shifts

Abstract

Transfer learning has become a pivotal approach in modern machine learning pipelines, particularly when labeled data is limited. However, its robustness under domain and distributional shifts remains a significant challenge. This study explores self-supervised pretraining strategies to enhance transferability across diverse downstream tasks and environments. We compare contrastive, generative, and clustering-based self-supervised objectives in scenarios with synthetic and natural domain gaps. Empirical results on three benchmark datasets show that contrastive pretraining yields an average +8.3% improvement in target-domain accuracy compared to supervised pretraining under heavy distributional shift. The findings underscore the importance of pretext task design, representational invariance, and semantic alignment in transfer learning robustness.