From Basics to Smart Predictions: A Journey Through Machine Learning(Part 1)

Machine learning is like teaching a computer to make decisions. But before jumping into complex models, it’s important to understand the basics.

Nearest Neighbor(NN) Algorithm

Imagine you move to a new city. You want to find a hospital. You ask, “Which hospital is closest to where I am?”

• This is like Nearest Neighbor (NN).
• The computer looks for the closest data point to make a decision.
• Simple, but it only works well when the data is clean and small.

Disadvantage: Only one neighbor may be misleading. Sensitive to noise and scales of features. Processing: Compute distance to all points, pick the closest.

I had implemented the simple algorithm for hospital finding.

# Problem Statement:
# You are building a location-based app. A user opens the app and wants to find the nearest hospital from their location.
# Input: User’s coordinates (x, y)
# Dataset: A list of hospitals with their coordinates and names.
# Task: Find and return the nearest hospital name.

import math
hospitals = [
    ("City Hospital", (2, 3)),
    ("Green Valley Clinic", (5, 4)),
    ("Lakeside Hospital", (9, 6)),
    ("Sunrise Health Center", (4, 7)),
    ("Riverdale Hospital", (8, 1))
]

# Example user location:
userLocation = (6, 9)

def findNearestHospital(userLocation, hospitals):
    nearestHospital = None
    nearestDistance = float('inf')

    for hospital in hospitals:
        name, location = hospital
        xA, yA = userLocation
        xB, yB = location
        distance = math.sqrt((xB - xA) ** 2 + (yB - yA) ** 2)
        if distance < nearestDistance:
            nearestDistance = distance
            nearestHospital = name
    print(f"Nearest hospital: {nearestHospital}")

findNearestHospital(userLocation, hospitals)

We saw how Nearest Neighbor finds the closest point. But what if one neighbor is wrong? This is where K-nearest neighbors come in. K Nearest Neighbors improves the decision by considering multiple neighbors

K Nearest Neighbors(Crowd Wisdom)

Sometimes, one neighbor is wrong. You ask five people instead.

• This is K Nearest Neighbors (KNN).
• The computer checks the K closest points and chooses the most common answer. Disadvantage:
• Slow if the dataset is large, because distances must be computed for all points.
• Choice of K affects results. Processing: Compute distances to all points, sort them, take top K, then do majority vote.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

data = pd.read_csv("../../data/KNNAlgorithmDataset.csv")

featureColumns = data.columns.drop('diagnosis')

diagnosisColumn = data['diagnosis']

classes = data['diagnosis'].unique()
# Features and Target
X = data[featureColumns].values
Y = data['diagnosis'].values

np.random.seed(12)
indexes = np.random.permutation(len(X))

# Split the data into training and testing sets
splitIdx = int(0.8 * len(X))
trainIdx = indexes[:splitIdx]
testIdx = indexes[splitIdx:]

# Fill the missing values with the mean of the column
X_Filled = X.copy()

# Drop columns that are all NaN
X_Filled = X_Filled[:, ~np.all(np.isnan(X_Filled), axis=0)]

# Fill the missing values with the mean of the column
for col in range(X_Filled.shape[1]):
    colMean = np.nanmean(X_Filled[:, col])

    X_Filled[np.isnan(X_Filled[:, col]), col] = colMean

# Training and Testing Sets
X_train = X_Filled[trainIdx]
Y_train = Y[trainIdx]
X_test = X_Filled[testIdx]
Y_test = Y[testIdx]

# Majority Voting Function
def majorityVoting(neighbor_labels):
    unique_labels = np.unique(neighbor_labels)
    counts = []
    for label in unique_labels:
        counts.append(np.sum(neighbor_labels == label))
    return unique_labels[np.argmax(counts)]

def KNN(X_train, Y_train, X_test, Y_test, k):
    distances = np.zeros((len(X_test), len(X_train)))
    predicted_labels = []
    for i in range(len(X_test)):
        for j in range(len(X_train)):
            dist = np.sqrt(np.sum((X_test[i] - X_train[j]) ** 2))
            distances[i, j] = dist  

        nearest_idx = np.argsort(distances[i])[:k]
        neighbor_labels = Y_train[nearest_idx]

        # majority voting
        predicted_label = majorityVoting(neighbor_labels)
        predicted_labels.append(predicted_label)

    return predicted_labels

accuracies = []
k_values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
for k in k_values:
    predicted_labels = KNN(X_train, Y_train, X_test, Y_test, k)
    accuracies.append(np.mean(predicted_labels == Y_test))

plt.plot(k_values, accuracies)
plt.xlabel('k')
plt.ylabel('Accuracy')
plt.title('Accuracy vs k')
plt.show()

NN and KNN rely on distances and similarity. Now let’s explore a different approach: probability. Bayes’ theorem helps the computer make predictions based on evidence

Bayes' Theorem (Thinking Probabilistically)

Now, imagine you want to guess whether the email is spam or not.

• Bayes’ theorem helps combine prior knowledge with new evidence.
• It answers: “Given this evidence, what is the probability of an event?”
• Works well with single, clear features.

# Dataset
emails = [
    ("win a free lottery ticket", "spam"),
    ("claim your free prize now", "spam"),
    ("let’s meet for lunch", "ham"),
    ("are you coming to the meeting", "ham")
]

totalEmails = len(emails)
classes = set(email[1] for email in emails)

# feature engineering
featuredWord="lottery"

# Bayes theorem
# P(Class|Feature) = P(Feature|Class) * P(Class) / P(Feature)

# first calculate the probabiliy of each class -> P(Class)
emails_in_class = {}
P_Class = {}
for cls in classes:
    emails_in_class[cls] = 0
    for text,label in emails:
        if label == cls:
            emails_in_class[cls] += 1
    P_Class[cls] = emails_in_class[cls] / totalEmails

print("P(Class):",P_Class)

# second calculate the probabiliy of feature -> P(Feature)
# P(Feature) = No. of emails with feature / No. of total emails
No_of_emails_with_feature = 0
for text,label in emails:
    if featuredWord in text:
        No_of_emails_with_feature += 1
P_Feature = No_of_emails_with_feature / totalEmails
print("P(Feature):",P_Feature)

# third calculate the probabiliy of feature -> P(Feature|Class)
# P(Feature|Class) = P(Feature,Class) / No. of emails in this class

P_Feature_Given_Class = {}
for cls in classes:
    P_Feature_Given_Class[cls] = 0
    for text,label in emails:
        if label == cls and featuredWord in text:
            P_Feature_Given_Class[cls] += 1
    P_Feature_Given_Class[cls] = P_Feature_Given_Class[cls] / emails_in_class[cls]
print("P(Feature|Class):",P_Feature_Given_Class)

# fourth calculate the probabiliy of class given feature -> P(Class|Feature)
# P(Class|Feature) = P(Feature|Class) * P(Class) / P(Feature)
P_Class_Given_Feature = {}
for cls in classes:
    P_Class_Given_Feature[cls] = P_Feature_Given_Class[cls] * P_Class[cls] / P_Feature
print("P(Class|Feature):",P_Class_Given_Feature)

Naive Bayes — Many Features, Simple Assumption

Earlier, we used just one word: ‘lottery’. Now, we look at all the words in the email to decide if it is spam.

• Naive Bayes assumes all events/features are independent.
• The computer multiplies probabilities of all features.
• Simple, fast, and surprisingly effective for tasks like spam detection.

# Dataset
emails = [
    ("win a free lottery ticket", "spam"),
    ("claim your free prize now", "spam"),
    ("let’s meet for lunch", "ham"),
    ("are you coming to the meeting", "ham")
]

totalEmails = len(emails)
classes = set(email[1] for email in emails)

# Vocabulary set (all unique words)
vocab = set()
for text, _ in emails:
    for word in text.split():
        vocab.add(word)

# Count emails in each class
emails_in_class = {}
P_Class = {}

for cls in classes:
    count = 0
    for text, label in emails:
        if label == cls:
            count += 1
    emails_in_class[cls] = count
    P_Class[cls] = count / totalEmails  # Prior probability

print("Emails in each class:", emails_in_class)
print("P(Class):", P_Class)

# Count of each word per class
word_count_per_class = {}
for cls in classes:
    word_count_per_class[cls] = {}
    for word in vocab:
        count = 0
        for text, label in emails:
            if label == cls and word in text:
                count += 1
        word_count_per_class[cls][word] = count

print("Word count per class:", word_count_per_class)

# Calculate P(word|Class)
P_Word_Given_Class = {}
for cls in classes:
    P_Word_Given_Class[cls] = {}
    for word in vocab:
        count = word_count_per_class[cls][word]
        P_Word_Given_Class[cls][word] = (count + 1) / (emails_in_class[cls] + len(vocab))
print("P(word|Class):", P_Word_Given_Class)

new_Email = "let’s win lottery hello"

# classify the email with spam or ham
def classify_email(email):
    scores = {}
    for cls in classes:
        P_Class_Given_Email = P_Class[cls]
        for word in email.split():
            if word in vocab:
                P_Class_Given_Email *= P_Word_Given_Class[cls][word]
        scores[cls] = P_Class_Given_Email
    return max(scores, key=scores.get)

print("Classified email:", classify_email(new_Email))

Naive Bayes assumes all words are independent. In reality, words like ‘free lottery’ are related. Still, this assumption works surprisingly well.

Final Remarks

NN

• Sensitive to noise, scales, and only one neighbor
• May overfit if the single neighbor is noisy.

KNN:

• Slow for large datasets, the choice of K matters.
• Small K can overfit, large K can underfit.

**Bayes:

• **Only single feature, zero probability if unseen.
• May underfit if the feature is insufficient.

Naive Bayes:

• Assumes independence, may ignore word relationships.
• simpler model, often reduces overfitting, but may underfit correlated features.

We do not need to train these models because they can make predictions directly using their processing modes. Even so, overfitting and underfitting remain important considerations. Overfitting happens when a model is too complex. In KNN, choosing a very small K can overfit, and in neural networks or Bayesian models, very specific rules can do the same. Underfitting happens when a model is too simple and cannot capture the patterns in the data.

Here is the GitHub link to follow the progress. https://github.com/engrdeveloper/Machine-Learning-Algorithm-Implementation

References

https://aiml.com/what-is-overfitting/ https://www.kaggle.com/code/walidmohd1/classification-using-k-nearest-neighbour