Paper Title
BUILDING SCALABLE AI-POWERED ANALYTICS PIPELINES USING DELTA LIVE TABLES: A CYBERSECURITY-FIRST APPROACH
Keywords
- ai security
- threat detection
- delta lake
- mlops
- cloud security
- pyspark
- mlflow
- security analytics
- real-time detection
Publication Info
Volume: 14 | Issue: 2 | Pages: 301-314
Published On
May, 2023
Downloads
FULL PDF
Abstract
The intersection of artificial intelligence, cybersecurity, and data engineering has created new paradigms for building robust analytics pipelines. This article explores the development of AI-powered analytics pipelines on Databricks using Delta Live Tables, with a particular emphasis on cybersecurity applications. By integrating advanced machine learning and deep learning models with Databricks' cloud-native architecture, organizations can build scalable threat detection and response systems that operate in near real-time. We examine the end-to-end process of building security-focused AI applications, from secure data collection and preprocessing to model training and deployment of threat predictions. The research demonstrates how modern data pipelines can adaptively respond to evolving threat landscapes through continuous learning mechanisms. Furthermore, we discuss specific techniques for optimizing performance when processing large security datasets while maintaining the confidentiality, integrity, and availability requirements inherent to cybersecurity operations. This approach presents a comprehensive framework for security teams looking to leverage AI capabilities within their security operations centers (SOCs).
View more »