AI-Powered Feature Engineering in Data Science Pipelines Using Automated Feature Selection and Embedding Techniques

Abstract

Feature engineering is a crucial component in data science pipelines, enhancing the performance of machine learning models by transforming raw data into meaningful representations. Traditional feature selection methods are often manual and time-intensive, limiting scalability and efficiency. AI-powered feature engineering leverages automated feature selection, deep learning embeddings, and meta-learning frameworks to streamline feature extraction. This paper explores recent advancements in AI-driven feature selection techniques, compares traditional and automated approaches, and evaluates their impact on model performance and computational efficiency.