DESIGN AND OPTIMIZATION OF CLOUD-NATIVE DATA PROCESSING PIPELINES USING AUTOML FOR DYNAMIC WORKLOAD ADAPTATION AND COST MINIMIZATION
Abstract
The rising demand for scalable and efficient data processing has driven the adoption of cloud-native architectures. However, designing pipelines that adapt automatically to fluctuating workloads while minimizing cost remains a complex challenge. This study proposes an AutoML-powered framework for dynamically optimizing data processing pipelines in cloud-native environments. The system automates configuration tuning, workload scaling, and cost optimization across compute resources. Our results demonstrate significant improvements in cost-efficiency and processing latency across varied workload patterns when compared with static rule-based systems.
Keywords
Details