Understanding Decision Trees using the Palmer Penguins Dataset

Photo credit: Simplilearn

Introduction

Decision Tree is one of the easiest Machine Learning algorithms to understand. It is conceptually simple yet powerful [1]. A Decision Tree (DT) is similar to how human beings make choices in life depending on several factors present. Decision Trees remain one of the most widely used Machine Learning (ML) algorithms and have many real-world applications including customer segmentation, health care, fraud detection (banking), insurance and education.

Decision Tree — what it is?

Decision Tree is a supervised ML algorithm that uses a tree-like flowchart structure. Since it is supervised, we need to provide labelled data to fit the model. DT can be used both for classification and regression problems. As depicted in Figure 1, the process starts at root node with all sample points, and then depending on a scenario (such as whether the flipper length is less than 200 mm), it moves to the next node. This continues until a leaf node is reached, where the final prediction is made.

Figure 1: A simple conceptual Decision Tree created by the author.

There are several advantages of DT:

• DTs are simple to understand and visualise.
• Even if many features (independent variables) can be present, a DT can select a feature automatically that will provide better class separation.
• Feature scaling is generally not necessary since DTs are based on threshold-based splits rather than distance calculations. Thus, Decision Trees can often reduce the need for extensive feature selection or feature scaling.
• They often perform well on structured/tabular datasets.

There are more sophisticated ML algorithms like Random Forest, XGBoost where multiple DTs are used to predict the outcome.

In this article, we will discuss only the basics of DT and how it can be used in a simple classification problem using Palmer Penguin dataset.

Classification using Decision Tree

Dataset used and description of data

We will implement DT on a dataset called Palmer Archipelago (Antarctica) penguin dataset [3] that is available on public domain and widely used for data exploration and visualisation. Out of total features we will use only 4 features, namely blength (bill_length_mm), bdepth (bill_depth_mm), flength (flipper_length_mm) and mass (body_mass_g). This is to keep things simple. After pre-processing, the dataset contains 342 samples belonging to three species, namely Adelie, Chinstrap and Gentoo.

We will use Python and Scikit library to implement the algorithm.

Figure 2: Decision Tree generated using DecisionTreeClassifier on the Palmer Penguins dataset.

# Import DecisionTreeClassifier.
from sklearn.tree import DecisionTreeClassifier

# Create a decision tree classifier.
clf = DecisionTreeClassifier(max_depth=3, random_state=42)

# Train the classifier, or fit the model.
clf.fit(X_train, y_train)

Gini index

The Gini Index is used by the CART (Classification and Regression Trees) algorithm, which is the default algorithm used by Scikit-learn for Decision Trees. It measures how “pure” a group of samples is. A Gini value of 0 means the group contains only one class (perfect purity). Higher Gini values indicate a mixture of different classes.

For binary classification:

• p1= probability of class 1
• p2 = probability of class 2

Suppose a node contains:

· 80% Yes

· 20% No

Then:

This indicates that the node is reasonably pure.

As can be seen from Figure 2, the root node contains 273 samples, of which 121 are Adelie, 54 of which is Chinstrap and rest from Gentoo variety. It starts with flength but uses other features at other decision levels. The tree stops splitting when the node impurity becomes very low or reaches 0.

Decision Trees can produce different structures when randomness is involved, for example during train-test splitting or when random_state is not fixed.

Evaluation of Model Performance

There are many metrics used to evaluate a model’s performance. The most widely used ones are Precision, Recall, F1-score and Accuracy. The definitions of these are given below.

where:

TP = correctly predicted positive

TN = correctly predicted negative

FP = falsely predicted positive

FN = falsely predicted negative

Confusion Matrix

A confusion matrix is used to evaluate the performance of a classifier. It gives a record of TP, TN, FP and FN.

Figure 3: Confusion Matrix

In the Figure 3, TP is 30 for Adelie, meaning all samples are classified correctly. This is also true for Gentoo species where all 25 samples are correctly classified. One Chinstrap sample is incorrectly classified as Adelie.

A Decision Tree can sometimes become too complex and memorise the training data, leading to overfitting.

Key Takeaways

• DT provides a simple way to classify data. It is beginner friendly.
• DT can nicely classify penguins in the Palmer dataset.
• Overfitting: DTs can do very well on training data, but poorly on test data.
• Difficulty in capturing complex relationships — where features are related in a complex way.
• Decision Trees closely resemble human decision-making, making them an excellent introduction to Machine Learning.

References

1. The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman, 2nd edition.

2. The Hundred-Page Machine Learning Book by Andriy Burkov (2019).

3. https://archive.ics.uci.edu/dataset/690/palmer+penguins-3.

Feel free to access the GitHub repo. Feedback and comments are welcome.