Supervised Learning — Classification

Contents

1. Logistic Regression
2. Naive Bayes Classifier
3. Decision Tree Classifier

Logistic Regression

Though the name has regression in it , Logistic regression is a classification algorithm that is used to perform binary classification tasks . (0 or 1) , (Yes or No ) , (True or False).

e.g. Predicting whether a customer will buy a product (Yes/No) based on their profile is an example of Binomial Logistic Regression.

Why Use Logistic Regression:

Bounded output (0 to 1)

• Uses the sigmoid (logistic) function to ensure outputs are between 0 and 1.
• Output can be interpreted as a probability of the positive class.

Natural thresholding

• We can set a threshold (e.g., 0.5) to decide the class: If above 0.5 → positive class , else → negative class

Better loss function

• Logistic regression uses log loss (cross-entropy), which is ideal for classification tasks, unlike the squared error in linear regression.

Naive Bayes Classifier

Uses Bayes algorithm to classify data based on probabilities. ‘Naive’ → the presence of one feature does not affect the others.

Assumptions

1. Feature independence
2. All features are equally important
3. No missing data

Bayes Theorem

e.g. Golf Playing prediction

Data to be classified: X = (outlook =Sunny, Temperature = Mild, Humidity = Windy = False)

Naive Bayes

Since P(C1|X) > P(C2|X ) , play_golf = Yes

Decision Tree

A Decision Tree Classifier is a supervised learning algorithm used for both classification (categorical targets) and regression (continuous targets) tasks. It builds a model in the shape of a tree structure, where data is split into branches based on feature values, leading to a series of decisions that ultimately assign a class label or value to each data point.

CART: Classification and Regression Trees

CART (Classification and Regression Trees) is a popular framework for constructing decision trees:

• Classification Trees: Used when the target variable is categorical (e.g., classifying emails as spam or not spam).
• Regression Trees: Used when the target variable is continuous (e.g., predicting house prices).

Decision Tree Structure

A decision tree consists of:

• Root Node: The initial node representing the entire dataset.
• Internal Nodes: Nodes where decisions are made based on feature values.
• Branches: Outcomes of decisions, leading to further nodes.
• Leaf Nodes: Terminal nodes that assign a class label (for classification) or a value (for regression).

Steps to Build a Decision Tree

1. Select the Best Feature to Split

• For classification, use impurity measures such as Gini Impurity or Entropy to evaluate how well a feature separates the classes.
• For regression, use metrics like Mean Squared Error (MSE) to assess splits.

2. Split the Data

• Partition the data into subsets based on the chosen feature and its value(s).

3. Repeat the Process Recursively

• For each subset, repeat the process: select the next best feature and split again, forming a recursive, top-down tree structure.

4. Assign Labels to Leaf Nodes

• Once further splitting no longer improves homogeneity (purity) or a stopping criterion is met, assign the majority class (classification) or average value (regression) to the leaf node.

Impurity Measures for Splitting

• Gini Impurity: Measures the probability of misclassifying a randomly chosen element.
• Entropy: Measures the amount of information disorder or unpredictability.