Paper Title

DESIGN AND OPTIMIZATION OF CLOUD-NATIVE DATA PROCESSING PIPELINES USING AUTOML FOR DYNAMIC WORKLOAD ADAPTATION AND COST MINIMIZATION

Keywords

  • Cloud-Native
  • AutoML
  • Data Pipelines
  • Cost Minimization
  • Dynamic Workload
  • Kubernetes
  • Resource Optimization
  • Serverless

Article Type

Research Article

Journal

International Journal of Computer Science and Engineering Research and Development External link

Publication Info

Volume: 15 | Issue: 3 | Pages: 14-21

Published On

May, 2025

Downloads

CITATION
COPY LINK

Abstract

The rising demand for scalable and efficient data processing has driven the adoption of cloud-native architectures. However, designing pipelines that adapt automatically to fluctuating workloads while minimizing cost remains a complex challenge. This study proposes an AutoML-powered framework for dynamically optimizing data processing pipelines in cloud-native environments. The system automates configuration tuning, workload scaling, and cost optimization across compute resources. Our results demonstrate significant improvements in cost-efficiency and processing latency across varied workload patterns when compared with static rule-based systems.

View more »