Categorical Feature in Decision Tree Classifier

Able to handle both numerical and categorical data. However, the scikit-learn implementation does not support categorical variables for now

is the statement we got from scikit-learn documentation.

Yes, decision trees are capable of handling both numerical and categorical data. This is true in theory, but in implementation we should try one hot encoding for categorical features before training or testing the model. Always keep in mind that the ML model understands nothing but numbers.

One hot encoding is a method used in machine learning models to encode categorical variables as numerical values. It constructs new binary columns of every possible value from the original data. Let’s take an example.

The values in the original data are Sunny, Rain and Wind. We create a separate column for each possible value. Wherever the original value was Sunny, we put a 1 in the Sunny column.

Let’s see how this is done in code.

import pandas as pd
from sklearn.tree import DecisionTreeClassifier

data = pd.DataFrame()
data['A'] = ['sunny','rain','sunny','wind']
data['B'] = ['b','b','a','a']
data['C'] = [1, 0, 1, 0]
data['Class'] = ['n','y','n','n']

tree = DecisionTreeClassifier()

one_hot_data = pd.get_dummies(data[['A','B','C']],drop_first=True)
tree.fit(one_hot_data, data['Class'])

If we hot encode a variable A with three options ‘sunny’, ‘rain’, ‘wind’ into three binary variables x1, x2, x3, having the decision rule A == ‘sunny’ is basically the same as x1 <= 0.5. It is computationally more expensive to do the hot encoding, but as you’ll notice, you will get good performance.

And that’s all! Good luck to us!