Principal Component Analysis (PCA)

Principal Component Analysis (PCA) is one of the most popular unsupervised machine learning techniques for dimensionality reduction, data visualization, noise reduction, and feature extraction. It transforms high-dimensional data into a lower-dimensional space while preserving as much variance (information) as possible.

1. What is PCA?

PCA finds new axes (called principal components) in the data that:

• Are orthogonal (perpendicular) to each other.
• Capture the maximum possible variance in the data.
• The first principal component (PC1) explains the most variance.
• The second (PC2) explains the next highest, and so on.

Goal: Reduce the number of features while losing as little information as possible.

2. Why Do We Need PCA?

• Curse of Dimensionality: High-dimensional data leads to sparsity, overfitting, and slow computation.
• Visualization: Plot data in 2D or 3D.
• Speed up ML models: Fewer features = faster training.
• Remove multicollinearity (highly correlated features).
• Noise reduction.

Trade-off: You lose some interpretability of original features.

3. Key Mathematical Concepts (Easy Explanation)

Variance

Measures how spread out the data is along one direction. PCA maximizes this.

Covariance

Shows how two features move together:

• Positive → both increase/decrease together.
• Negative → opposite directions.
• Zero → independent.

Covariance Matrix (for p features): A p × p symmetric matrix.

Eigenvalues & Eigenvectors

For a matrix A:

• Eigenvector (v): Direction that doesn’t change when transformed by A (only scaled).
• Eigenvalue (λ): The scaling factor for that eigenvector.

In PCA:

• Eigenvectors of the covariance matrix = directions of principal components.
• Eigenvalues = amount of variance explained by each component.

Higher eigenvalue → more important component.

4. Step-by-Step Process of PCA

• Standardize the Data (very important!)

Subtract mean and divide by standard deviation for each feature.

Formula: X_std = (X — mean) / std

Why? Features with larger scales would dominate otherwise.

• Compute the Covariance Matrix of standardized data.
• Compute Eigenvalues and Eigenvectors of the covariance matrix.
• Sort Eigenvectors by decreasing eigenvalues.
• Select Top k Components (based on explained variance or desired dimensions).
• Transform Data: Project original data onto the new axes.

X_new = X_std × W (where W = matrix of selected eigenvectors).

• (Optional) Reconstruct original data approximately to check information loss.

Explained Variance Ratio = eigenvalue / sum of all eigenvalues. Helps decide how many components to keep (e.g., keep 95% variance).

Numerical Example (2D Data)

Imagine 2 features that are highly correlated.

I ran a small simulation

import numpy as np

np.random.seed(42)
X = np.random.rand(100, 2) * 10
X[:, 1] = X[:, 0] * 1.5 + np.random.randn(100) * 0.5  # Correlated features

# Standardize
mean = np.mean(X, axis=0)
std = np.std(X, axis=0)
X_std = (X - mean) / std

cov = np.cov(X_std.T)
eigvals, eigvecs = np.linalg.eig(cov)

# Sort
idx = np.argsort(eigvals)[::-1]
eigvals = eigvals[idx]
eigvecs = eigvecs[:, idx]

print("Covariance:\n", np.round(cov, 3))
print("Sorted Eigenvalues:", np.round(eigvals, 3))

Output (approximate)

• Covariance shows high correlation (~1.0).
• Eigenvalues: ~[2.01, 0.006] → First component captures almost all variance!

PC1 explains >99% variance. You can safely reduce to 1 dimension.

Complete Code Example: Iris Dataset

Iris Dataset: 150 samples, 4 features (sepal/petal length/width), 3 species.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt

iris = load_iris()
X = iris.data
y = iris.target

# Standardize
scaler = StandardScaler()
X_std = scaler.fit_transform(X)

# PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_std)

print("Explained Variance Ratio:", pca.explained_variance_ratio_)

# Plot
plt.figure(figsize=(8,6))
plt.scatter(X_pca[:,0], X_pca[:,1], c=y, cmap='viridis', s=50)
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.title('PCA with scikit-learn')
plt.colorbar(label='Species')
plt.grid(True)
plt.show()

Output

Typical Result: First two components explain ~95%+ variance. Species are nicely separated.

How to Choose Number of Components?

• Scree Plot: Plot explained variance vs. component number. Look for “elbow”.
• Cumulative Explained Variance: Keep enough for 90–95%.
• Cross-validation for downstream tasks (classification/regression).

Advantages & Disadvantages

Advantages:

• Reduces overfitting and computation time.
• Uncorrelated features in new space.
• Handles multicollinearity.
• Easy to implement and interpret variance.

Disadvantages:

• Linear method (misses non-linear relationships → use Kernel PCA or t-SNE/UMAP).
• New components are linear combinations → hard to interpret biologically.
• Sensitive to scaling (always standardize!).
• Assumes data is somewhat Gaussian-like.

Real-World Applications

• Image compression (Eigenfaces).
• Genomics (gene expression).
• Finance (portfolio optimization).
• Recommender systems.
• Preprocessing before SVM, KNN, Neural Nets.