Case Study: Gradient Boosting Iris

This case study demonstrates Gradient Boosting Machine (GBM) on the Iris dataset for binary classification, comparing with other TOP 10 algorithms.

Running the Example

cargo run --example gbm_iris

Results Summary

Test Accuracy: 66.7% (4/6 correct predictions on binary Setosa vs Versicolor)

###Comparison with Other TOP 10 Classifiers

Note: GBM's 66.7% accuracy reflects this simplified implementation using classification trees for residual fitting. Production GBM implementations use regression trees and achieve state-of-the-art results.

Hyperparameter Effects

Number of Estimators (Trees)

Insight: Consistent accuracy suggests algorithm has converged.

Learning Rate (Shrinkage)

Guideline: Lower learning rates (0.01-0.1) with more trees typically generalize better.

Tree Depth

Guideline: Shallow trees (3-8) prevent overfitting in boosting.

Implementation

use aprender::tree::GradientBoostingClassifier;
use aprender::primitives::Matrix;

// Load data
let (x_train, y_train, x_test, y_test) = load_binary_iris_data()?;

// Train GBM
let mut gbm = GradientBoostingClassifier::new()
    .with_n_estimators(50)
    .with_learning_rate(0.1)
    .with_max_depth(3);

gbm.fit(&x_train, &y_train)?;

// Predict
let predictions = gbm.predict(&x_test)?;
let probabilities = gbm.predict_proba(&x_test)?;

// Evaluate
let accuracy = compute_accuracy(&predictions, &y_test);
println!("Accuracy: {:.1}%", accuracy * 100.0);

Probabilistic Predictions

Sample  Predicted  P(Setosa)  P(Versicolor)
────────────────────────────────────────────
   0     Setosa       0.993      0.007
   1     Setosa       0.993      0.007
   2     Setosa       0.993      0.007
   3     Setosa       0.993      0.007
   4     Versicolor   0.007      0.993

Observation: High confidence predictions (>99%) despite moderate accuracy.

Why Gradient Boosting

Advantages

✓ Sequential learning: Each tree corrects previous errors
✓ Flexible: Works with any differentiable loss function
✓ Regularization: Learning rate and tree depth control overfitting
✓ State-of-the-art: Dominates Kaggle competitions
✓ Handles complex patterns: Non-linear decision boundaries

Disadvantages

✗ Sequential training: Cannot parallelize tree building
✗ Hyperparameter sensitive: Requires careful tuning
✗ Slower than Random Forest: Trees built one at a time
✗ Overfitting risk: Too many trees or high learning rate

Algorithm Overview

1. Initialize with constant prediction (log-odds)
2. For each iteration:
   • Compute negative gradients (residuals)
   • Fit weak learner (shallow tree) to residuals
   • Update predictions: F(x) += learning_rate * h(x)
3. Final prediction: sigmoid(F(x))

Hyperparameter Guidelines

n_estimators (50-500)

• More trees = better fit but slower
• Risk of overfitting with too many
• Use early stopping in production

learning_rate (0.01-0.3)

• Lower = better generalization, needs more trees
• Higher = faster convergence, risk of overfitting
• Typical: 0.1

max_depth (3-8)

• Shallow trees (3-5) prevent overfitting
• Deeper trees capture complex interactions
• GBM uses "weak learners" (shallow trees)

Comparison: GBM vs Random Forest

When to Use GBM

✓ Tabular data (not images/text)
✓ Need maximum accuracy
✓ Have time for hyperparameter tuning
✓ Moderate dataset size (<1M rows)
✓ Feature engineering done

Related Examples

• examples/random_forest_iris.rs - Random Forest comparison
• examples/naive_bayes_iris.rs - Naive Bayes comparison
• examples/svm_iris.rs - SVM comparison

TOP 10 Milestone

Gradient Boosting completes the TOP 10 most popular ML algorithms (100%)!

All industry-standard algorithms are now implemented in aprender:

1. ✅ Linear Regression
2. ✅ Logistic Regression
3. ✅ Decision Tree
4. ✅ Random Forest
5. ✅ K-Means
6. ✅ PCA
7. ✅ K-Nearest Neighbors
8. ✅ Naive Bayes
9. ✅ Support Vector Machine
10. ✅ Gradient Boosting Machine