Support Vector Machine 02: Non Linear Data

While linear SVMs are effective for simple datasets, non-linear SVMs can handle more complex datasets with non-linear relationships. In this blog, we will explore the concept of non-linear SVMs, their advantages, and how they work.

What is Non-Linear SVM?

Non-linear SVMs are an extension of linear SVMs that can handle datasets with non-linear relationships between features. They use kernel functions to transform the data into a higher-dimensional space, where the data becomes linearly separable.

Let’s Discuss about what exactly is kernel functions

Kernel Function

A Kernel function is a mathematical function that transforms the data from the original feature space into a higher-dimensional space, known as the feature space or kernel space. This transformation enables the SVM to learn non-linear decision boundaries.

Types of Non-Linear SVM

• Polynomial Kernel SVM: Uses a polynomial kernel function to transform the data.
• RBF Kernel SVM: Uses a radial basis function kernel to transform the data.

Polynomial Kernel SVM

The Polynomial Kernel function is a popular kernel function used in Support Vector Machines (SVMs) to handle non-linearly separable data. It is defined as:

K(a, b) = (a * b+ c)^d

where:

• a and b are the input vectors
• c is a constant term
• d is the degree of the polynomial

Let’s Imagine we got a data like this as image shown below, off course data is not linear. In this case we cannot able to use simple linear SVM. we should use a polynomial kernel SVM.

after transforming the data points with degree of polynomial 2, then we get a graph something will look like image below.

Transformed data points after being transformed into two dimensional

now by just using simple SVM we can able to plot a boundary decision which will look like the image shown below. if any new points appears then we can easily able to classify data and our model will be reliable.

In sklearn SVM library we just need to give a degree of polynomial and rest of the things SVM model handles very well.

Code for Polynomial SVM

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split

# Load Iris dataset
iris = load_iris(as_frame=True)
df = iris.frame
X = df[['sepal length (cm)','petal width (cm)']].values  
y = df.target

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create a non-linear kernel SVM classifier
svm = SVC(kernel='poly',degree=3, random_state=42)
svm.fit(X_train, y_train)

# Plot the decision boundary
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.02),
                     np.arange(y_min, y_max, 0.02))
Z = svm.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

plt.figure(figsize=(8, 6))
plt.contourf(xx, yy, Z, alpha=0.5, cmap='plasma')
plt.scatter(X[:, 0], X[:, 1], c=y, edgecolors='k', cmap='plasma')
plt.xlabel('Sepal Length (cm)')
plt.ylabel('Petal Width (cm)')
plt.title('Iris Classification using Non-linear Kernel SVM')
plt.show()

Radial Basis Function

RBF kernels are the most generalized form of kernelization and is one of the most widely used kernels due to its similarity to the Gaussian distribution. The RBF kernel function for two points X₁ and X₂ computes the similarity or how close they are to each other. This kernel can be mathematically represented as follows:

if you want to deep dive into the math check out this link radial basis function

Let’s have a code intuition to understand better

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.axes._axes import _log as matplotlib_axes_logger
from mpl_toolkits import mplot3d
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from matplotlib.colors import ListedColormap

from sklearn.datasets.samples_generator import make_circles
X, y = make_circles(100, factor=.1, noise=.1)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='bwr')

Let’s transform the original datapoint with radial basis function.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20)

def plot_3d_plot(X, y):
    r = np.exp(-(X ** 2).sum(1))
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=100, cmap='bwr')
    ax.set_xlabel('X1')
    ax.set_ylabel('X2')
    ax.set_zlabel('y')
    return ax

plot_3d_plot(X,y)

what you can do is, you can separate the data by drawing a plane.

rbf_classifier = SVC(kernel="rbf")
rbf_classifier.fit(X_train, y_train)
y_pred = rbf_classifier.predict(X_test)

accuracy_score(y_test, y_pred)

1.0

zero_one_colourmap = ListedColormap(('blue', 'red'))
def plot_decision_boundary(X, y, clf):
    X_set, y_set = X, y
    X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, 
                                 stop = X_set[:, 0].max() + 1, 
                                 step = 0.01),
                       np.arange(start = X_set[:, 1].min() - 1, 
                                 stop = X_set[:, 1].max() + 1, 
                                 step = 0.01))

    plt.contourf(X1, X2, clf.predict(np.array([X1.ravel(), 
                                             X2.ravel()]).T).reshape(X1.shape),
               alpha = 0.75, 
               cmap = zero_one_colourmap)
    plt.xlim(X1.min(), X1.max())
    plt.ylim(X2.min(), X2.max())
    for i, j in enumerate(np.unique(y_set)):
        plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
                c = (zero_one_colourmap)(i), label = j)
    plt.title('SVM Decision Boundary')
    plt.xlabel('X1')
    plt.ylabel('X2')
    plt.legend()
    return plt.show()

plot_decision_boundary(X, y, rbf_classifier)

That’s the magic of using kernel trick in SVM. if you liked my blog please give some clap to support my content.