Gradient boosting classifiers in Scikit-Learn and Caret

install.packages('gbm')
install.packages('caret')
install.packages('palmerpenguins')

library(gbm)
library(caret)  
library(palmerpenguins)

head(penguins) # head() returns the top 6 rows of the dataframe
summary(penguins) # prints a statistical summary of the data columns

The createDataPartition function from the caret package to split the original dataset into a training and testing set and split data into training (70%) and testing set (30%).

dim(penguins)

# get rid of any NA

penguins <- na.omit(penguins)
parts = caret::createDataPartition(penguins$species, p = 0.7, list = F)

train = penguins[parts, ]
test = penguins[-parts, ]

Now you're ready to train and test.

The train method from the caret library uses R formulas, where the dependent variable (often also called a target) is on the left side of a tilde '~.' The independent variables (often also called a feature) are on the right side of the '~'. For instance:

height ~ age

This step predicts height based on age.

To caret train, you pass the formula, the training data and the method to be used. The caret library provides methods for many different types of training, so setting the method as "gbm" is where you'll specify to use gradient boosting. The next parameter configures the training process. The "repeatedcv" method performs X-fold cross-validation on subsamples of the training set data points. Here, you specify 3 repeats of 5-fold cross-validation by using a different set of folds for each cross-validation.

model_gbm <- caret::train("species ~ .",
                          data = train,
                          method = "gbm", # gbm for gradient boosting machine
                          trControl = trainControl(method = "repeatedcv", 
                                                   number = 5, 
                                                   repeats = 3, 
                                                   verboseIter = FALSE),
                          verbose = 0)

Now you can use the predictive model to make predictions on test data:

pred_test = caret::confusionMatrix(
  data = predict(model_gbm, test),
  reference = test$species
)

print(pred_test)

This step prints:

Confusion Matrix and Statistics
           Reference
Prediction  Adelie Chinstrap Gentoo
  Adelie        42         0      0
  Chinstrap      0        20      0
  Gentoo         1         0     35

Overall Statistics
                                          
               Accuracy : 0.9898          
                 95% CI : (0.9445, 0.9997)
    No Information Rate : 0.4388          
    P-Value [Acc > NIR] : < 2.2e-16       
                  Kappa : 0.984           

 Mcnemar's Test P-Value : NA              

Statistics by Class:
                     Class: Adelie Class: Chinstrap Class: Gentoo
Sensitivity                 0.9767           1.0000        1.0000
Specificity                 1.0000           1.0000        0.9841
Pos Pred Value              1.0000           1.0000        0.9722
Neg Pred Value              0.9821           1.0000        1.0000
Prevalence                  0.4388           0.2041        0.3571
Detection Rate              0.4286           0.2041        0.3571
Detection Prevalence        0.4286           0.2041        0.3673
Balanced Accuracy           0.9884           1.0000        0.9921

Due to the nature of cross validation with folds, the sensitivity and specificity for each class can be slightly different than what is observed here, although the accuracy will be the same. The accuracy is quite good, even with the Chinstrap penguin, which makes up 20% of the training dataset.

This step installs the libraries that you'll use to train and test your gradient boosting classifier. The training itself is done with scikit-learn and the data comes from the palmerpenguins library.

!pip install seaborn pandas scikit-learn palmerpenguins

Now install the libraries into the notebook environment:

import pandas as pd
import seaborn as sns
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import classification_report
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from palmerpenguins import load_penguins

As in the R code, there are some NAs in the penguins dataset that needs to be removed. This code snippet loads the dataset, removes any NA rows, and then splits the data into features and target.

# Load the penguins
penguins = load_penguins() #initialize the dataset
penguins = penguins.dropna()
X = penguins.drop("species", axis=1)
y = penguins["species"]

Now create a training and testing split of the dataset, with 70% of the data pulled for training and 30% reserved for testing.

X_train, X_test, y_train, y_test = train_test_split(
   X, y, test_size=0.3, random_state=42
)

Next, you'll gather two lists of the column names, one for the categorical features of X and another for the numerical features, e.g. float64 or int64. Then, use ColumnTransformer from scikit-learn to apply different preprocessing to different column types. A OneHotEncoder is applied to categorical features to convert them into binary vectors. A StandardScaler is applied to numerical features to standardize them around a mean f 0 and a variance of 1.

# Define categorical and numerical features
categorical_features = X.select_dtypes(
   include=["object"]
).columns.tolist()

numerical_features = X.select_dtypes(
   include=["float64", "int64"]
).columns.tolist()

preprocessor = ColumnTransformer(
   transformers=[
       ("cat", OneHotEncoder(), categorical_features),
       ("num", StandardScaler(), numerical_features),
   ]
)

Now that you've created the feature sets and the prepocessor, you can create a pipeline to train the model. That pipeline uses the preprocessor on input data and then passes it to the Gradient Boosting algorithm to build the classifier. With a larger or more complex dataset, there are other training parameters that you might want to configure. For instance, max_features, which sets the number of features to consider when looking for the best split or max_depth, which limits the number of nodes in the tree. This code snippet sets the criterion parameter, which measures the quality of a split for training. In this case, we’re using the mean squared error with improvement score by Jerry Friedman (resides outside of IBM)

pipeline = Pipeline([
       ("preprocessor", preprocessor),
       ("classifier", GradientBoostingClassifier(random_state=42, 
         criterion='friedman_mse', max_features=2)),
   ])

Next, perform cross-validation to evaluate how well your machine learning pipeline performs on the training data. Calling the fit method of the pipeline you created trains the model. The loss function uses Mean Squared Error (or MSE) by default.

# Perform 5-fold cross-validation
cv_scores = cross_val_score(pipeline, X_train, y_train, cv=5)

# Fit the model on the training data
pipeline.fit(X_train, y_train)

Now that the model has been trained, predict the test set and check the performance:

# Predict on the test set
y_pred = pipeline.predict(X_test)

# Generate classification report
report = classification_report(y_test, y_pred)

Print the results:

print(f"Mean Cross-Validation Accuracy: {cv_scores.mean():.4f}")
print("\nClassification Report:")
print(report)

This step prints the following results:

Mean Cross-Validation Accuracy: 0.9775
Classification Report:
              precision    recall  f1-score   support
      Adelie       1.00      1.00      1.00        31
   Chinstrap       1.00      1.00      1.00        18
      Gentoo       1.00      1.00      1.00        18
    accuracy                           1.00        67
   macro avg       1.00      1.00      1.00        67
weighted avg       1.00      1.00      1.00        67

These results are very close to the accuracy reported by the R methods in the first part of this tutorial.