Optimizing Neural Network Training With Gradient Boosting Techniques

Abstract

This study introduces two novel training frameworks—Gradient Boosted Recurrent Neural Network (GB-RNN) and Gradient Boosted Deep Neural Network (GB-DNN)—that synergize the principles of ensemble learning with deep learning models. By employing a stage-wise refinement strategy inspired by boosting, these models incrementally construct layered architectures that address limitations often seen in conventional neural networks, such as excessive model complexity, unstable gradients, and susceptibility to overfitting. Each layer in the network is trained to improve upon the residual shortcomings of previous iterations, enabling more efficient learning. Experimental validation is conducted using the CIFAR-10 image dataset, and performance is assessed using comprehensive classification metrics, demonstrating the effectiveness of the proposed approach. In this context, this project proposes two specialized neural architectures: Gradient Boosted Recurrent Neural Network (GB-RNN) and Gradient Boosted Deep Neural Network (GB-DNN). These models are built upon the philosophy of modular learning, where additional components are introduced only when they yield measurable improvements. The GB-RNN framework is designed for data with inherent sequential or spatial ordering, such as images or signals, while GB-DNN is tailored for structured datasets where deep fully connected layers suffice. Both models leverage a layered refinement mechanism where each new block addresses the residual errors left by previous layers. Early trained components are frozen to preserve learned patterns, and fine-tuning is selectively applied to ensure stability and efficiency. This approach not only enhances the learning process but also reduces redundancy, accelerates convergence, and improves generalization across data types.