Classification of Wines (Red or White) using Tree Based Methods

We have used this data set earlier in the story Classification of Red and White Wines using Logistic Regression and LDA.

The feature variables are:

• fixed acidity
• volatile acidity
• citric acid
• residual sugar
• chlorides
• free sulfur dioxide
• total sulfur dioxide
• density
• pH
• sulphates
• alcohol

All of these 11 variables are continuous variables.

In this article, we will not show the pre-processing steps or data exploration, but we will start looking into tree-based methods for classification.

A Single Decision Tree Classifier

Tree-based methods are simple and useful for interpretation. In building a decision tree, we divide the feature space — that is, the set of possible values for 𝑋1,𝑋2,…,𝑋𝑝 — into 𝐽 distinct and non-overlapping regions, 𝑅1,𝑅2,…,𝑅𝐽. For a classification tree, we predict that each observation belongs to the most commonly occurring class of training observations in the region to which it belongs.

In theory, the regions could have any shape. However, we choose to divide the feature space into high-dimensional rectangles, or boxes, for simplicity and for ease of interpretation of the resulting predictive model. Unfortunately, it is computationally infeasible to consider every possible partition of the feature space into 𝐽 boxes. For this reason, we take a top-down, greedy approach that is known as recursive binary splitting.

The approach is top-down because it begins at the top of the tree and then successively splits the predictor space; each split is indicated via two new branches further down on the tree. It is greedy because at each step of the tree-building process, the best split is made at that particular step, rather than looking ahead and picking a split that will lead to a better tree in some future step.

A large decision tree constructed in this way may produce good predictions on the training set, but is likely to overfit the data, leading to poor test set performance. To overcome this problem, a better strategy is to grow a very large tree 𝑇, and then prune it back in order to obtain a subtree.

Cost complexity pruning — also known as weakest link pruning — is used to do this, where we consider a sequence of trees indexed by a nonnegative tuning parameter C. For a given value of C, we find the subtree which minimizes a cost complexity function based on it predictive accuracy and the size of the tree.

Common cost complexity functions for classficatiuon problems are given by Gini index and cross-entropy.

from sklearn.tree import DecisionTreeClassifier # Import Decision Tree Classifier
from sklearn import metrics

# Create Decision Tree classifer object
tree_model = DecisionTreeClassifier()

# Train Decision Tree Classifer
tree_model = tree_model.fit(X_tr_scaled,y_train)

Visualize the fitted tree

Using graphviz and pydotplus libraries, we visualize the fitted tree with each node showing the splitting criterion, the value of the Gini index for that node and the predicted class for that node.

from sklearn.tree import export_graphviz
from six import StringIO  
from IPython.display import Image  
import pydotplus

dot_data = StringIO()
export_graphviz(tree_model, out_file=dot_data,  
                filled=True, rounded=True,
                special_characters=True,feature_names = features,class_names=['Red','White'])
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  
graph.write_png('winetree_1.png')
Image(graph.create_png())

Figure 1: The single decision tree for the wine data

We observe that the tree is too big and the simplicity is lost. We can change the criterion from the Gini index to croos-entropy and also restrict the tree to a maximum depth of 4.

# Create Decision Tree classifer object
tree_model = DecisionTreeClassifier(criterion="entropy", max_depth=4)

# Train Decision Tree Classifer
tree_model = tree_model.fit(X_tr_scaled,y_train)

dot_data = StringIO()
export_graphviz(tree_model, out_file=dot_data,  
                filled=True, rounded=True,
                special_characters=True,feature_names = features,class_names=['Red','White'])
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  
graph.write_png('winetree_2.png')
Image(graph.create_png())

Figure 2: The pruned Decision Tree of depth 4

Now the tree is easier to understand. Let us check the confusion matrix for this model.

#Predict the response for test dataset
y_te_pr = tree_model.predict(X_te_scaled)

#Visualise the confusion matrix
cnf_matrix = metrics.confusion_matrix(y_test, y_te_pr)

#Create a heat map to visualize
labels = np.unique(y_train)
fig, ax = plt.subplots()

# create heatmap
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, cmap="YlGnBu" ,fmt='g',
           xticklabels=labels, yticklabels=labels)
ax.xaxis.set_label_position("top")
plt.tight_layout()
plt.title('Confusion matrix', y=1.1)
plt.ylabel('Actual label')
plt.xlabel('Predicted label')

plt.show()

Figure 3: Confusion matrix

There are only 3 White wines in the test data, which were misclassified, but there are 17 Red wines which were misclassified. Let us have a look into the accuracy metrics.

from sklearn.metrics import classification_report

print(classification_report(y_test, y_te_pr, target_names=labels))

              precision    recall  f1-score   support

         Red       0.99      0.95      0.97       311
       White       0.98      1.00      0.99       989

    accuracy                           0.98      1300
   macro avg       0.99      0.97      0.98      1300
weighted avg       0.98      0.98      0.98      1300

Still we have 98% accuracy for the test data, which is slightly lower than the accuracy of the logistic regression and linear discriminant analysis models.

Cross-Validation

We have observed that the accuracy of the tree depends on the max_depth parameter of the tree. To prune the tree to the right size, we need to use cross-validation to find the optimal max_depth of the tree. Let us 5-fold cross-validation and visualize the cross-validation errors.

from sklearn.model_selection import cross_val_score
cv = 5 #5-fold cross-validation

tree_depths = np.arange(1,15)

cv_scores_mean = []
cv_scores_std = []
accuracy_scores = []
for depth in tree_depths:
    tree_model = DecisionTreeClassifier(max_depth=depth)
    cv_scores = cross_val_score(tree_model, X_tr_scaled, y_train, cv=cv, scoring='accuracy')
    cv_scores_mean.append(cv_scores.mean())
    cv_scores_std.append(cv_scores.std())
    accuracy_scores.append(tree_model.fit(X_tr_scaled, y_train).score(X_tr_scaled, y_train))
cv_scores_mean = np.array(cv_scores_mean)
cv_scores_std = np.array(cv_scores_std)
accuracy_scores = np.array(accuracy_scores)

fig, ax = plt.subplots(1,1, figsize=(16,6))
ax.plot(tree_depths, cv_scores_mean, '-o', label='Mean cross-validation accuracy', alpha=0.9)
ax.fill_between(tree_depths, cv_scores_mean-2*cv_scores_std, cv_scores_mean+2*cv_scores_std, alpha=0.2)
ylim = plt.ylim()
ax.plot(tree_depths, accuracy_scores, '-*', label='Train accuracy', alpha=0.9)
ax.set_title('Cross-Validation Accuracy Scores for DecisionTreeClassifier', fontsize=16)
ax.set_xlabel('Tree depth', fontsize=14)
ax.set_ylabel('Accuracy', fontsize=14)
ax.set_ylim(ylim)
ax.set_xticks(tree_depths)
ax.legend(loc='lower right')
plt.show()

Figure 4: Cross-validation accuracy with tree depth

From 5-fold cross-validation, it appears that max_depth=5 might be a good choice.

Bagging

Bootstrap aggregation, or bagging, is a general-purpose procedure for reducing the variance of a statistical learning method; we introduce it here because it is particularly useful and frequently used in the context of decision trees.

It is an ensemble algorithm that combines the predictions from many decision trees. In this approach, we generate 𝐵 different bootstrapped training data sets. We then train our method on the 𝑏-th bootstrapped training set in order to get the predicted class from that tree. We record the class predicted by each of the 𝐵 trees, and take a majority vote: the overall prediction is the most commonly occurring class among the 𝐵 predictions.

from sklearn.ensemble import BaggingClassifier

# define the model

bag_model = BaggingClassifier(base_estimator=DecisionTreeClassifier(max_depth=4),
                              n_estimators=10, random_state=21)
bag_model.fit(X_tr_scaled, y_train)

#Predict the response for test dataset
y_te_pr = bag_model.predict(X_te_scaled)

print(classification_report(y_test, y_te_pr, target_names=labels))

              precision    recall  f1-score   support

         Red       0.99      0.94      0.96       311
       White       0.98      1.00      0.99       989

    accuracy                           0.98      1300
   macro avg       0.98      0.97      0.98      1300
weighted avg       0.98      0.98      0.98      1300

In bagging, n_estimators or the number of bootstrap trees is a hyper-parameter and the performance or prediction accuracy may depend on that. We can again use cross-validation to find out the optimal number of bootstrap samples to be used.

cv = 5 #5-fold cross-validation

n_trees = [1,5,10,20,30,40,50,60,70,80,90,100]

cv_scores_mean = []
cv_scores_std = []
accuracy_scores = []
for n_tree in n_trees:
    bag_cv = BaggingClassifier(estimator=DecisionTreeClassifier(max_depth=4),
                              n_estimators=n_tree, random_state=21)
    cv_scores = cross_val_score(bag_cv, X_tr_scaled, y_train, cv=cv, scoring='accuracy')
    cv_scores_mean.append(cv_scores.mean())
    cv_scores_std.append(cv_scores.std())
    accuracy_scores.append(bag_cv.fit(X_tr_scaled, y_train).score(X_tr_scaled, y_train))
cv_scores_mean = np.array(cv_scores_mean)
cv_scores_std = np.array(cv_scores_std)
accuracy_scores = np.array(accuracy_scores)

fig, ax = plt.subplots(1,1, figsize=(16,6))
ax.plot(n_trees, cv_scores_mean, '-o', label='Mean cross-validation accuracy', alpha=0.9)
ax.fill_between(n_trees, cv_scores_mean-2*cv_scores_std, cv_scores_mean+2*cv_scores_std, alpha=0.2)
ylim = plt.ylim()
ax.plot(n_trees, accuracy_scores, '-*', label='Train accuracy', alpha=0.9)
ax.set_title('Cross-Validation Accuracy Scores for Bagging', fontsize=16)
ax.set_xlabel('Number of Trees', fontsize=14)
ax.set_ylabel('Accuracy', fontsize=14)
ax.set_ylim(ylim)
ax.set_xticks(n_trees)
ax.legend(loc='lower right')
plt.show()

Figure 5: Cross-Validation accuracy scores for bagging

From 5-fold cross-validation scores, it appears that n_estimators=30 might be the best choice for the number of trees.

RandomForests

Random forests provide an improvement over bagged trees by way of a small tweak that decorrelates the trees. This reduces the variance when we average the trees. As in bagging, we build a number of decision trees on bootstrapped training samples. But when building these decision trees, each time a split in a tree is considered, a random selection of 𝑚 predictors is chosen as split candidates from the full set of 𝑝 predictors. The split is allowed to use only one of those 𝑚 predictors. A fresh selection of 𝑚 predictors is taken at each split, and typically we choose 𝑚≈√p, that is, the number of predictors considered at each split is approximately equal to the square root of the total number of predictors

In this case, we have 11 predictors, so an optimal choice would be 𝑚=3.

from sklearn.ensemble import RandomForestClassifier

# define the model

rf_model = RandomForestClassifier(n_estimators=100, random_state=21)
rf_model.fit(X_tr_scaled, y_train)

#Predict the response for test dataset
y_te_pr = rf_model.predict(X_te_scaled)

print(classification_report(y_test, y_te_pr, target_names=labels))

We observe that RandomForestClassifier attains 99% accuracy similar to the logistic regression models.

We can see the effect of the number of features on the accuracy by plotting the accuracy of the test data for different values of 𝑚 in the RandomForestClassifier.

mtry = np.arange(1,len(features)+1)

rf_acc = []
for m in mtry:
    rf_model = RandomForestClassifier(n_estimators=100, max_depth=4,max_features=m, random_state=21)
    rf_model.fit(X_tr_scaled, y_train)
    y_te_pr = rf_model.predict(X_te_scaled)

    rf_acc.append(metrics.accuracy_score(y_test, y_te_pr))

#plt.figure(figsize=(12,8))
plt.plot(mtry, rf_acc, 'bo-' )
plt.xlabel('$m$')
plt.ylabel('Test Accuracy')
plt.show()

Figure 6: Accuracy of the RandomForestClassifier for different values of m

The best value of 𝑚 with maximum test accuracy is observed for 𝑚=5 in this example. However, one can again use cross-validation to select the optimal values for the max_depth of each tree, number of trees, n_estimators and the max_features to be used for the split.

For the RandomForestClassifier, we extract the imprtance of individual features, which will give us some idea about which feature is more imprtant in this classification.

The higher, the more important the feature. The importance of a feature is computed as the (normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance.

rf_model = RandomForestClassifier(n_estimators=100, max_depth=4,max_features=5, random_state=21)
rf_model.fit(X_tr_scaled, y_train)

importance = rf_model.feature_importances_

plt.bar(features, importance)
plt.xticks(rotation=90)
plt.show()

Figure 7: Imprtance of features in the RandomForest Classfier

Most important features are chlorides and total sulfur dioxide.

In our next study, we will use support vector classifiers for this problem.

Contact: biman.pph@gmail.com