Clustering

(Here we will learn about K-Mean Clustering with basics)

Types of Machine Learning:

1. Supervised Learning

Definition: A technique where the training data includes both inputs (features) and their corresponding outputs (labels).

Goal: Learn a mapping function from input → output.

Examples:

• Predicting house prices (Regression).
• Classifying emails as spam or not spam (Classification).

2. Unsupervised Learning

Definition: A technique where the training data has inputs only (no labels). The model tries to find patterns, structure, or groupings in the data.

Goal: Discover hidden patterns or clusters in data.

Examples:

• Customer segmentation in marketing (Clustering).
• Reducing data dimensions using PCA (Dimensionality Reduction).

3. Reinforcement Learning (RL)

Definition: A technique where an agent learns to make decisions by interacting with an environment and receiving feedback in the form of rewards or penalties.

Goal: Maximize the total reward over time by learning the best actions (policy).

Examples:

• Training a robot to walk.
• Game playing AI (e.g., AlphaGo beating humans in Go).

Unsupervised Learning

Clustering:

Definition: Grouping a large amount of unlabeled data into clusters such that data points in the same group are more similar to each other than to those in other groups.

Goal: Discover hidden groupings/patterns.

Examples of Algorithms:

1. K-Means(partition data into k clusters.
2. DBSCAN(Density — Based Spatial Clustering, useful for irregular shapes & noise)
3. Gaussian Mixture Models(GMMs) (probabilistic clustering).

Association:

Definition: A rule-based method of discovering interesting relationships, associations, or co-occurrence patterns between variables in large datasets.

Goal: Find rules like “If X happens, Y is likely to happen too.”

Examples of Algorithms:

1. Apriori Algorithm
2. FP-Growth (Frequent Pattern Growth)

Dimensionality Reduction:

Definition: The process of reducing the number of input features (dimensions) while retaining the most important information in the data.

Goal: Simplify data, remove redundancy/noise, and make analysis/visualization easier.

Examples of Algorithms:

1. PCA (Principal Component Analysis) — projects data into fewer dimensions with maximum variance.
2. t-SNE (t-distributed Stochastic Neighbor Embedding) — mainly for visualization of high-dimensional data.

Anomaly Detection:

Definition: The process of identifying rare, unusual, or abnormal data points that do not follow the general pattern of the dataset.

Goal: Detect outliers that may indicate critical information such as fraud, faults, or errors.

Examples of Algorithms:

1. Isolation Forest

Language Model:

Definition: A model that learns the probability distribution of words/sequences in a language. It predicts the next word (or sequence of words) given the previous context.

Goal: Understand and generate human-like text based on statistical patterns or learned representations.

Examples of Algorithms/Models:

1. Statistical Models:

• n-gram models (predict next word using previous n–1 words).

2. Neural Network Models:

• RNN (Recurrent Neural Networks)
• LSTM (Long Short-Term Memory)
• GRU (Gated Recurrent Unit)

3. Transformer-based Models:

• GPT (Generative Pre-trained Transformer)
• BERT (Bidirectional Encoder Representations from Transformers)

Applications of Clustering

1. Customer Segmentation: Suppose you are working in a product-based company and a major sale is about to begin. Since this sale features premium branded products with higher price ranges, it would not be cost-effective to notify every customer through SMS. Instead, it is wiser to identify and target a specific cluster of customers who have shown interest in such premium shopping experiences. By doing so, the company can optimize communication costs while ensuring that the right customers receive personalized notifications, ultimately leading to higher engagement and better conversion rates.
2. Data Analysis: In order to analyze the data more effectively, it is considered a good practice to study the clusters. Analyzing clusters allows us to identify patterns, similarities, and differences within groups of data, which leads to deeper insights and ultimately better results.
3. Semi Supervised Learning: A great example of clustering in real life is Google Photos. It automatically groups photos of the same person into clusters using facial recognition. Later, by asking the user to confirm or provide names, it labels these folders, making photo organization more efficient and personalized.
4. Image Segmentation: It is a technique of labeling similar objects or features in photos with the same tag or color or pixels , so that visually or contextually related items are grouped together.

Types of Clustering

Partitional Clustering

Basic Concept: Partitioning clustering algorithm divide a dataset into a set of non-overlapping subgroups or clusters, where each data point belongs to exactly one cluster.

Examples: The most famous partitioning clustering algorithm is K-means. It assigns data points to clusters in such a way that each point belongs to the cluster with the closest mean, which serves as a prototype of the cluster.

Defining Number of Clusters: A key requirements is pre-specifying the number of clusters (k). The selection of k significantly affects the outcome and quality of the clustering.

Iterative Process: These algorithms typically use an iterative refinement technique. For instance, in K-means, the process involves repeatedly assigning points to the nearest cluster centroid and then recalculating the centroid.

Objective Function Optimization: They aim to optimize an objective function, such as minimizing the total within-cluster variance or the sum of squared distances between data points and their respective cluster centroids.

Suitability for Certain Data Shapes: Partitioning methods are most effective when clusters are spherical or globular in shape. They assume homogeneity in cluster shapes and sizes.

Sensitivity to Initial Conditions: These algorithms can be sensitive to the initial starting conditions (like initial cluster centroids in k-means). Different initializations can lead to different clustering results.

Handling of Outliers: Partitioning algorithms can be influenced by outliers, as these can significantly skew the mean or centroid of a cluster.

Scalability and Efficiency: They are generally more scalable and efficient for larger datasets compared to hierarchical clustering, making them suitable for many practical applications.

Use Cases and Limitations: While widely used in various fields like market research, pattern recognition, and image processing, these algorithms have limitations in handling non-spherical clusters, varying cluster sizes, and noisy datasets. Advanced version and variations of partitioning algorithm have been developed to address some of these limitations.

Hierarchical Clustering

Nature of Clustering: Hierarchical clustering builds a hierarchy of clusters either by successively merging smaller clusters into larger ones(aggiomerative approach) or by successively splitting larger clusters into smaller ones (divisive approach).

No Need to Specify Number of Clusters: Unlike partitioning algorithm like K-means, hierarchical clustering does not require pre-specifying the number of clusters. The number of clusters can be determined by analyzing the dendrogram.

Dendrogram Visualization: It provides a tree-like diagram called a dendrogram, which is a visual representation of the clustering process showing the order of cluster combination and the distance at which clusters are merged.

Distance Metrics and Linkage Criteria: Hierarchical clustering uses various distance metrics (like Euclidean or Manhattan distance) and linkage criteria(like single linkage, complete linkage, average linkage, and Ward’s method) to decide which clusters to merge or split.

Flexibility in Identifying Cluster Shapes: Hierarchical clustering can identify clusters with various shapes and sizes, unlike partitioning methods that generally assume spherical clusters.

Computational Complexity: It is generally more computationally intensive than partitioning methods, especially for large datasets, due to the need to compute and store distances between all pairs of points.

Sensitivity to Noise and Outliers: The method can be sensitive to noise and outliers, as these can influence the formation of clusters and the structure of the dendrogram.

Applications: It is widely used in fields like biology (for gene and protein sequencing), social science, and linguistics , and is particularly useful for exploratory data analysis where understanding the hierarchical relationship between objects is important.

Density Based Clustering

Principle: Density- based clustering groups data points based on the density of data points in a region. It defines clusters as areas of high density separated by areas of low density. The algorithm identifies clusters as regions where data points are densely packed together, with areas of low density or noise between them.

Examples: DBSCAN is one of the most popular density-based clustering algorithms. It is known for its efficiency and ability to find clusters of arbitrary shapes.

No Need to specify Number of Clusters: Unlike partitioning methods, density based clustering doesn’t require pre-specifying the number of clusters.

Handling Noise and Outliers: It is robust to outliers and noise, as these are typically not part of the dense regions that form clusters.

Ability to Find Arbitrary Shapes: Density-based clustering can discover clusters of arbitrary shapes, unlike methods like K-means which are biased towards spherical clusters.

Parameter Sensitivity: The performance of these algorithms is sensitive to the input parameters, like the radius of neighborhood and the minimum number of points required to form a dense region (MinPts in DBSCAN).

Scalability Issues: Some density-based algorithms may struggle with very large datasets due to computational and memory constraints.

Applications: Widely used in fields such as anomaly detection, geospatial data analysis(like identifying geographic regions of interest), and image processing, especially where the shape of the clusters is not known in advance or the data contains noise.

Distribution/Model based Clustering

Statistical Distribution Models: The central concept of distribution-based clustering is that data points in a cluster follow a certain statistical distribution, most commonly Gaussian or normal distributions.

Parameter Estimations: These algorithms focus on estimating the parameters(like mean, variance) of the assumed distributions for each clusters. The fit of these parameters to the actual data determines the quality of the clustering.

Expectation-Maximization (EM) Algorithm: A key algorithm used in distributions-based clustering is EM, which alternates between assigning data points to the most likely distribution (expectation step) and updating the distribution parameters to maximize data fit(maximization step).

Handling of Complex Cluster Shapes: Unlike methods such as K-means, distribution-based clustering can identify clusters of various shapes and sizes, making it more flexible in handling real-world data complexities.

Computational Intensity: The process of estimating distribution parameters and assigning data points can be computationally demanding, especially for large datasets and when the number of features (dimensions) is high.

Handling of Outliers: These methods can be more robust to outliers, as outliers are less likely to significantly affect the parameters of the overall distribution.

Scalability Issues: While effective for small to medium-sized datasets, scalability to very large datasets can be challenging due to the computational complexity of the algorithms and the need for more sophisticated optimization techniques.

K-Means

This is the objective function in K-Means clustering.

• It measures how far the data points are from their cluster centroids.
• The goal of K-Means is to minimize J, meaning we want each point to be as close as possible to its cluster’s centroid.

Breaking it down

• Outer Sum (∑j=1 to k) Goes over each cluster j. → We calculate the error for all clusters.
• Inner Sum (∑i=1 to n) Goes over each data point i that belongs to cluster j. → We calculate the error for all points inside that cluster.
• Term ∥xi^(j)−cj∥^2 This is the squared Euclidean distance between:
• xi^(j)→ the i-th data point in cluster j
• cj​ → the centroid of cluster j
• Squaring ensures distances are always positive and gives more weight to points that are far from the centroid.

That total sum = inertia (Within-Cluster Sum of Squares).

This function is non-convex, which means it has many local minima. If we try to solve it using standard optimization techniques like gradient descent, we can easily get stuck in a local minimum. In fact, finding the global minimum is an NP-hard problem — practically impossible to solve exactly for large datasets.

K-Means Clustering — Steps(Lloyd’s Algorithm)

1. Decide the number of clusters (k): Choose how many clusters you want to divide the data into.
2. Initialize centroids: Randomly select k points as the initial centroids.
3. Iterate until convergence:

• Assign clusters: Assign each data point to the nearest centroid (using distance measures like Euclidean distance).
• Move centroids: Recalculate the centroid of each cluster by taking the mean of all data points assigned to it.
• Check for stopping condition:
• If the centroids do not change (or the change is very small), stop.
• Otherwise, repeat the assign → move steps again.

How do we decide the correct value of K

Elbow Method

The Elbow Method works on the concept of inertia, also called WCSS (Within-Cluster Sum of Squared Distances).

It can be seen in terms of variance.

Steps:

• For different values of k (k = 1, 2, 3 …), run K-Means and calculate the WCSS.
• Plot a graph with:
• X-axis → Number of clusters (k).
• Y-axis → WCSS.
• The curve typically decreases steeply at first and then starts flattening, creating a shape like an “elbow.”
• The elbow point (where the decrease slows down significantly) is considered the optimal value of k.

Intuition:

• Too few clusters → High WCSS (points are far from centroids).
• Too many clusters → WCSS keeps decreasing but gives little real improvement.
• The elbow balances accuracy vs simplicity.

Limitations of Elbow Method

Subjectivity in identifying the Elbow: The biggest challenge with the elbow method is the subjective nature of identifying the “elbow” point. The point where the inertia starts decreasing at a slower rate can be open to interpretation and may not be clear-cut , especially in datasets where the decrease in inertia is gradual.

Not suitable for all datasets: The method does not work well if the data is not very clustered or if the clusters have an irregular shape. In such cases, the elbow might not be distinct, leading to ambiguity in choosing the right number of clusters.

Performance with large number of features: The elbow method can become less effective as the number of features in the dataset increases. High-dimensional data can make the identification of a clear elbow more difficult.

Doesn’t consider cluster quality: The elbow method focuses solely on the variance within the clusters and does not take into account the quality of the clusters formed. It’s possible to choose a k where clusters are not meaningful or well-separated.

Sensitivity to scaling: Like k-means clustering itself, the results of the elbow method can be sensitive to the scale of the data. Features with larger scales can dominate the result, potentially leading to suboptimal choices of k.

code:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs

n_samples=300
n_features=2
centers=4

#generate data
x,y =make_blobs(n_samples=n_samples,n_features=n_features,centers=centers,cluster_std=1.5,random_state=42)

plt.scatter(x[:,0],x[:,1],s=50,cmap='viridis')
plt.title('Generated Dataset for K-Means Clustering')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.show()

from sklearn.cluster import KMeans
WCSS=[]
for i in range(1,11):
  kmeans=KMeans(n_init=10,n_clusters=i)
  kmeans.fit(x)
  WCSS.append(kmeans.inertia_)

plt.plot(range(1,11),WCSS)

kmeans=KMeans(n_init=10,n_clusters=4)
y_kmeans=kmeans.fit_predict(x)
plt.scatter(x[y_kmeans==0,0],x[y_kmeans==0,1],s=60,c='red',label='cluster1')
plt.scatter(x[y_kmeans==1,0],x[y_kmeans==1,1],s=60,c='blue',label='cluster2')
plt.scatter(x[y_kmeans==2,0],x[y_kmeans==2,1],s=60,c='green',label='cluster3')
plt.scatter(x[y_kmeans==3,0],x[y_kmeans==3,1],s=60,c='yellow',label='cluster4')
plt.scatter(kmeans.cluster_centers_[:,0],kmeans.cluster_centers_[:,1],s=100,c='black',label='centroids')
plt.legend()

Assumptions of K Means

Spherical Cluster Shape: K-means assumes that the clusters are spherical and isotropic, meaning they are uniform in all directions. Consequently, the algorithm works best when the actual clusters in the data are circular (in 2D) or spherical {in higher dimensions}.

Similar Cluster Size: The algorithm tends to perform better when all clusters of approximately the same size. If one cluster is much larger than others, K-means might struggle to correctly assign the points to the appropriate cluster.

Equal Variance of Cluster: K-means assumes that all clusters have similar variance. The algorithm uses the Euclidean distance metric, which can bias the clustering towards clusters with lower variance.

Clusters are Well Separated: The algorithm works best when the clusters are well separated from each other. If clusters are overlapping or intertwined, K-means might not be able to distinguish them effectively.

Number of Clusters (k) is predefined: K-means requires the number of clusters (k) to be specified in advance. Choosing the right value of k is crucial, but it is not always straightforward and typically requires domain knowledge or additional methods like the Elbow method or Silhouette analysis.

Large n, Small k: K-means is generally more efficient and effective when the dataset is large (large n)and the number of clusters is small (small k).

Limitations of K Means

except the above limitations there are few more.

Vulnerability to Outlier: Outliers can significantly distort the mean value of a cluster, leading to misleading results.

Hard Clustering: Each data point is forced into exactly one cluster, which may not be suitable for all applications, especially where data can belong to multiple clusters.

High-Dimensional Challenges: In very high-dimensional spaces, the distance between data points can become less meaningful, affecting the performance of K Means.

Sensitive to Scale: The measure is sensitive to the scale of the features. Hence, feature scaling (like standardization) is often recommended before applying K-means.

Assignment:

Solution:

link of the kaggle notebook

Silhouette Score

Cohesion(intra cluster distance)

Definition: cohesion refers to the degree to which elements in the same cluster are close to each other. It measures how tightly grouped the data points in a cluster are.

Ideal Scenario: Low cohesion means that data points in a cluster are similar or near to each other, indicating good clustering where each cluster is good and meaningful.

Measurement: Cohesion can be quantify using metric such as sum of squared distances of data points from their respective cluster centroid. In K-means, this is often referred to as inertia or within cluster sum of square.

Separation(inter cluster distance)

Definition: Separation, on the other hand, refers to how distinct or well-parted different clusters from each other. It measures the extant to which cluster are different and distant from each other.

Ideal Scenario: High separation means that clusters are well-differentiated and far apart, indicating that the algorithm has done a good job in distinguishing between different groups in the data.

Measurements: Separation can be quantified by metric such as distance between cluster centroid, or more complex measures like silhouette score, which consider both cohesion and separation.

Silhouette Score

The Silhouette Score is a measure used to assess the quality of clusters created by a clustering algorithm. It provides a succinct graphical representation of how well each data point lies within its cluster, which is a combination of both cohesion and separation. The value of the silhouette score ranges from -1 to 1, where a high value indicated that the object is well matched to its own cluster and poorly matched to neighboring clusters.

For a data point i:

• a(i) = average distance of point i from all other points in the same cluster (measures how well it is assigned to its own cluster → intra-cluster cohesion).
• b(i) = the minimum average distance of point i from all points in any other cluster (measures how far away it is from the “next best” cluster → inter-cluster separation).

Interpretation

Close to +1: indicates that the data point is far away from the neighboring clusters.

Close to 0: indicates that the data point is on or very close to the decision boundary between two neighboring clusters.

Close to -1: indicates that the data point may have been assigned to the wrong cluster.

Silhouette Graph

A great way to visualize silhouette scores is by plotting them in a histogram.

• Each bar in the histogram represents the number of points that have a silhouette score in that range.
• If most of the bars are toward the right (0.5–1.0), your clusters are well-formed.
• If a lot of bars are near 0, clusters are overlapping.
• If you see many negative scores, some points may be wrongly assigned.

The Red Vertical Line

In the silhouette histogram, you’ll often see a red vertical line.

• This line represents the average silhouette score across the entire dataset.
• This single value summarizes the overall clustering quality.
• The closer it is to 1, the better. A value around 0 indicates weak clustering, and negative values indicate poor separation.

K means Hyperparameters

KMeans(n_clusters=8, **, init=’k-means++’, n_init=’auto’, max_iter=300, tol=0.0001, verbose=0, random_state=None, copy_x=True, algorithm=’lloyd’*)

Number of Clusters(k):

Description: This is the number of clusters you want the algorithm to form, as well as the number of centroids to generate.

importance: Choosing the right number of clusters is crucial as it significantly influences the clustering results. Too many clusters can overfit the data, while too few can miss important patterns.

Initialization Method:

Description: This parameter specifies the method for initializing the centroids. Common methods include ‘random’ (randomly choosing k data points as initial centroids) and ‘k-means++’ (a smarter way of initializing centroids to improve convergence).

Importance: Good initialization can lead to faster convergence and better clustering. K-means++ is generally preferred over random initialization.

Number of Initialization Runs(n_init):

Description: This is the number of times the KMeans algorithm will be run with different centroid initializations. The final results will be the best output of n_init consecutive runs in terms of inertia.

Importance: Multiple initializations can prevent the algorithm from falling into sub-optimal solutions, but increase computational cost.

Maximum Iteration(max_iter):

Description:

The parameter max_iter in K-Means defines the maximum number of times the algorithm will repeat its update cycle:

1. Assigning each data point to the nearest centroid.
2. Recomputing the centroids as the average of points in each cluster.

This process continues until the centroids stop moving significantly (convergence) or until the algorithm has run for max_iter iterations — whichever happens first.

Importance:

Prevents infinite loops: If convergence is slow or doesn’t happen, max_iter ensures the algorithm still stops.

Tolerance Level(tol):

Description: The parameter tol sets the convergence threshold for the K-Means algorithm.

• After each iteration, new centroids are computed.
• If the movement (distance) between the old centroids and the new centroids is greater than tol, the algorithm continues.
• If the movement is less than or equal to tol for all centroids, the algorithm assumes convergence and stops early, even if max_iter has not been reached.

Random State(random_state)

Description: The parameter random_state is used to fix the randomness in selecting the initial centroids of clusters. Since K-Means starts by randomly choosing centroids, setting a random_state ensures that the same centroids are chosen every time you run the algorithm, leading to consistent and reproducible results.

Importance: Without fixing random_state, every run of the algorithm may give slightly different clusters due to random initialization. By setting it, we make results stable and comparable, which is especially important in research, reporting, or debugging.

**algorithm**{“lloyd”, “elkan”}, default=”lloyd”

K-means algorithm to use. The classical EM-style algorithm is "lloyd". The "elkan" variation can be more efficient on some datasets with well-defined clusters, by using the triangle inequality. However it’s more memory intensive due to the allocation of an extra array of shape (n_samples, n_clusters).

**verbose(**int, default=0)

Description: The verbose parameter in K-Means controls the amount of output printed during the clustering process. When you set verbose to a value greater than 0, the algorithm prints detailed logs of each step and iteration while running.

For example:

• verbose = 0 → No output (default, silent mode).
• verbose = 1 (or higher) → Prints details about the initialization of centroids and progress of each iteration.
• Higher values (like verbose = 2) provide even more detailed debugging information.

KMeans++

It is an algorithm for choosing the initial values( or ‘seeds’) for the KMeans clustering algorithm. The standard KMeans algorithm is sensitive to the initial starting points (centroids), and KMeans++ provides a way to overcome this problem by specifying a procedure to initialize the centroids before proceeding with the standard KMeans iterative algorithm.

Here’s a simplified overview of how KMeans++ works:

Initial Centroid Selection: The first centroid is chosen uniformly at random from the data points that are being clustered.

Distance Calculation: Calculate the distance of each data point from the nearest, previously chosen centroid.

probabilistic Selection of Next Centroids: Choose the next centroid from the data points with a probability proportional to the square of the distance from the point to its nearest centroid. This step biases the algorithm to select data points that are far from the existing centroids.

Repeat until K Centroids: Repeat steps 2 and 3 until k centroids have been chosen.

Proceed with Standard KMeans: Once the initial centroids are chosen, proceed with the standard KMeans clustering algorithm.

This method tends to spread out the initial centroids, which can lead to better clustering results compared to selecting the initial centroids randomly, as the standard KMeans algorithm does. By doing so, KMeans++ can often lead to faster convergence and better clustering.

K-Means Time and Space Complexity

Time Complexity

O(nkd*i)

where:

• n -> number of rows( number of data points).
• k -> number of centroids.
• d -> dimension of the data points (number of features).
• i -> number of iterations.

Space Complexity

O(nd + kd’)

where:

• n -> number of rows(number of data points).
• d -> features/columns of data points.
• k -> number of centroids.
• d’(d) -> features/columns of data points of the centroids.

Mini Batch Kmeans

One of the limitations of the standard K-Means algorithm is that it requires all data points to be used for every centroid update. While this works fine for small datasets, it becomes computationally expensive and slow for very large datasets.

To solve this problem, we use Mini-Batch K-Means. Instead of using the entire dataset in each iteration, Mini-Batch K-Means:

1. Randomly selects a small subset (mini-batch) of the data.
2. Uses only this mini-batch to compute the new centroid positions.
3. Updates the centroids incrementally based on the mini-batch.
4. Repeats the process until one of the following occurs:

• The centroids stabilize (stop changing significantly).
• The maximum number of iterations is reached.
• A defined convergence threshold is met.

Mini Batch Algorithm

We’re given:

• X → dataset (all data points / rows)
• k → number of clusters
• b → mini-batch size (number of random data points chosen at each iteration)
• t → number of iterations

1. Initialization

• Choose initial centroids for each cluster either *randomly or using K-Means++*.
• Create a vector v of size k, initialized to zeros. This vector will keep track of how many points have been assigned to each centroid.

2. Main Loop (Iterate for t steps)

For each iteration from 1 to t:

(a) Mini-Batch Selection

• Randomly pick b examples from the dataset X.
• This subset is our mini-batch for the current iteration.

(b) Assign Points to Closest Centroid (Cache Step)

• For each data point in the mini-batch.
• Find its nearest centroid.
• Store this information in a dictionary (or array) that maps each point to its closest centroid.

3. Update Centroids

Now, loop through each data point in the mini-batch again:

Step 1 → Identify Centroid Retrieve the nearest centroid for the data point from the cached dictionary.

Step 2 → Update Counts Increase the count for that centroid in vector v.

• Example: If the point is closest to centroid 2, then increment v[2].

Step 3 → Compute Learning Rate The learning rate is defined as:

• η=1/v[c]
• where v[c] is the number of points seen so far for centroid c.

Step 4 → Update Centroid Position Update the centroid using the weighted average formula:

• c ← (1−η)⋅c+η⋅x
• Here, c is the old centroid,
• x is the current data point,
• η balances how much influence the new point has compared to the centroid’s past updates.

4. Repeat Until Convergence

• Continue for t iterations, or until centroids stop moving significantly.

When to use Mini Batch

• When to use big dataset, for faster speed.
• When to do online Machine Learning(Chat GPT).

DBSCAN (Density Based Spatial Clustering of Applications with Noise)

MinPts & Epsilon (Hyperparameters)

How to measure density around a point?

To define density around a point, we need three things:

1. Neighborhood (area or radius) — a region around the point.

2. Number of points within that neighborhood — how many data points fall inside it.

3. Threshold — a cutoff value that helps determine whether the region is considered dense or sparse.

MinPts stands for “minimum points”, is a parameter that specifies the minimum number of points required to form a dense region, which is considered a cluster.

Epsilon(ε) is a key parameter that defines the radius of the neighborhood around a given data point, Specifically , ε is the maximum distance between two points for them to be considered as part of the same neighborhood. This parameter is crucial in determining whether points are close enough to be included in a cluster.

Core Points, Border Points & Noise Points

Core Points

A point is considered a Core Point if it has a minimum number of other points (specified by MinPts) within a given radius ε of itself.

Border Points

A border point is defined as follows:

• Not a core point: A border point does not meet the criteria to be a core point. It has fewer than MinPts within its ε-neighborhood.
• Neighbor of a core point: A border point is within the ε distance of one or more core points. In other words, it lies on the edge of a cluster, within the radius ε of at least one core point.

Noise Points

A noise point is a data point which can neither a core point nor a border point.

Density Connected Points

If the points are density connected, they will fall in the same cluster. Before going for density connected points, we should see directly density reachable points.

Directly Density Reachable

A point P is directly density-reachable from a point Q given Eps, MinPts if:

1. P is the Eps-neighborhood of Q.
2. Both P and Q are core points.

Density Connected Points

A point P is density connected to Q given Eps, MinPts if there is a chain of points p1,p2,p3….pn , p1=P and pn=Q such that pi+1 is directly density reachable from pi.

Simplified DBSCAN Algorithm

Step 1- Identify all points as either core point, border point or noise point.

Step 2- For all of the unclustered core points

• Step 2a- Create a new cluster
• Step 2b- add all the points that are unclustered and density connected to the current point into this cluster.

Step 3- For each unclustered border point assign it to the cluster of nearest core point.

Step 4- Leave all the noise points as it is.

Note: DBSCAN is not a predictive algorithm; it is used only for clustering. While the fit_predict method is available, it simply reassigns all the data points to clusters rather than making predictions on new, unseen data.

Advantages

1. Robust to outliers
2. No need to specify clusters
3. Can find arbitrary shaped clusters
4. Only 2 hyperparameters to tune

Limitations

1. Sensitivity to hyperparameters
2. Difficulty with varying density clusters
3. Does not predict

Application Areas

1. Spatial Data Analysis: DBSCAN is particularly well-suited for spatial data clustering due to its ability to find clusters of arbitrary shapes, which is common in geographic data. It’s used in applications like identifying regions of similar land use in satellite images or grouping locations with similar activities in GIS (Geographic Information Systems).
2. Anomaly Detection: The algorithm’s effectiveness in distinguishing noise or outliers from core clusters makes it useful in anomaly detection tasks, such as detecting fraudulent activities in banking transactions or identifying unusual patterns in network traffic.
3. Image Processing: In image analysis, DBSCAN can be used for tasks like object recognition and image segmentation, where the goal is to group pixels or features that form meaningful structures.
4. Bioinformatics: DBSCAN is applied in bioinformation for tasks such as gene expression data analysis, where it helps to identify groups of genes with similar expression patterns, which might indicate a functional relationship.
5. Customer Segmentation: In marketing and business analytics, DBSCAN can be used for customer segmentation by identifying clusters of customers with similar buying behavior or preferences.
6. Astronomy: The algorithm is employed in astronomy for tasks like star cluster identification, where it groups stars based on their physical proximity or other attributes.
7. Environmental Studies: DBSCAN can be used in environmental monitoring, for example, to cluster areas based on pollution levels or to identify regions with similar environmental characteristics.
8. Traffic Analysis: In traffic and transportation studies, DBSCAN is useful for identifying hotspots of traffic congestion or for clustering routes with similar traffic patterns.
9. Machine learning and Data Mining: More broadly, in the fields of machine learning and data mining, DBSCAN is employed for exploratory data analysis, helping to uncover natural structures or patterns in data that might not be apparent otherwise.
10. Social Network Analysis: The algorithm can be used to detect communities or groups within social networks based on interaction patterns or shared interests.

Code:

from sklearn.cluster import DBSCAN
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_circles

x,_=make_circles(n_samples=500,factor=.5,noise=0.03,random_state=4)

dbscan=DBSCAN(eps=0.1,min_samples=5)
clusters=dbscan.fit_predict(x)

plt.scatter(x[:,0],x[:,1],c=clusters,cmap='viridis',marker='o')
plt.title('DBSCAN clustering of concentric circles')
plt.xlabel('Feature 0')
plt.ylabel('Feature 1')
plt.show()

Hierarchical Clustering

Hierarchical Clustering is a method of cluster analysis used in data mining. It seeks to build a hierarchy of clusters in a step-by-step manner. There are two main types of hierarchical clustering:

1. Agglomerative(Bottom-Up Approach):

• Initial Step: Starts by treating each data point as a separate cluster. So, if there are N data points, you begin with N clusters.
• Clustering Process: In each step, the algorithm merges the two clusters that are closest to each other until all the clusters are merged into one big cluster containing all data points.
• Dendrogram: The result can be represented in a tree-like structure called a dendrogram, which shows the arrangement of the clusters and their proximity.

2. Divisive(Top-Down Approach):

• Initial Step: Begins with all data points in a single cluster.
• Clustering Process: At each step, the algorithm splits the cluster until each cluster contains only one data point.
• Top-Down Splitting: This is less common compared to agglomerative clustering and is computationally more intensive.

Algorithm

1. Initialization:

• Treat each data point as a separate cluster. Thus, if you have N data points, you start with N clusters, each containing just one data point.

2. Compute Distance Matrix:

• Calculate the distance between each pair of clusters. Common distance metrics include Euclidean, Manhattan, and Cosine distances. The choice of distance metric can significantly affect the outcome of the clustering.
• This results in an N*N distance matrix, where the distance between a cluster and itself is zero and create a proximity matrix.

3. Find the Closest Clusters:

• Identify the two clusters that are closest to each other based on the distance matrix.

4. Merge Clusters:

• Combine the two closest clusters into a single cluster.
• This step reduces the total number of clusters by one.

5. Update Distance Matrix:

• Recalculate the distances between the new cluster and all the existing clusters.
• The method of recalculating the distance depends on the linkage criterion used.

common linkage criteria include:

• Single Linkage: Distance between two clusters is defined as the shortest distance between any two points in the clusters.

• Complete Linkage: Distance is the longest distance between any two points in the clusters.

• Average Linkage: Distance is the average distance between all pair of points in the clusters.

• Ward’s Method: Distance is calculated as the increase in the total within cluster variance after merging the clusters.

6. Repeat:

• Repeat steps 3 to 5 until all data points are merged into a single cluster.

Hack to calculate the number of ideal clusters using a dendrogram

1. Draw the dendrogram from hierarchical clustering.
2. Look at all the horizontal lines (linkages) in the tree. Each horizontal line represents the distance (or dissimilarity) at which clusters are merged.
3. Find the longest vertical gap between two successive horizontal lines (i.e., the maximum height difference without any merge).
4. Draw a horizontal cut through this gap.
5. The number of vertical lines intersected by your cut = the number of clusters.

code:

import matplotlib.pyplot as plt
import pandas as pd
%matplotlib inline
import numpy as np

customer_data=pd.read_csv('hierarchical-clustering-with-python-and-scikit-learn-shopping-data.csv')
customer_data.head()

#selecting the annual income and spending score columns
data=customer_data.iloc[:,3:5].values

import scipy.cluster.hierarchy as sch

plt.figure(figsize=(10,7))
plt.title('customer Dendogram')
dend=sch.dendrogram(sch.linkage(data,method='ward'))

Applied the hack discussed above and calculate the number of clusters.

from sklearn.cluster import AgglomerativeClustering

cluster=AgglomerativeClustering(n_clusters=5,linkage='ward', metric='euclidean')
labels_=cluster.fit_predict(data)

plt.figure(figsize=(10,7))
plt.scatter(data[:,0],data[:,1],c=labels_,cmap='rainbow')
plt.title('Clusters of customers')
plt.xlabel('Annual Income')
plt.ylabel('Spending Score')

Linkage

In hierarchical clustering, linkage is the criterion that determines the distance between sets of observations as a function of the pairwise distance between observations. It’s essentially the algorithm used to decide the proximity of clusters. There are several linkage methods, each defining the distance between clusters differently.

1. Single Linkage(Nearest Point Algorithm):

• The distance between two clusters is defined as the shortest distance from any member of one cluster to any member of the other cluster.
• Capable of detecting non-elliptical shapes in the data.
• Works well for datasets where the clusters are well-separated.
• May not perform well when clusters are close together or overlap as it is sensitive to outliers.

2. Complete Linkage (Farthest Point Algorithm):

• The distance between two clusters is defined as the longest distance from any member of one cluster to any member of the other cluster.
• Less susceptible to noise and outliers compared to single linkage.
• Can struggle with elongated clusters with uneven size or non-convex shapes.

3. Average Linkage (Average Link Algorithm):

• The distance between two clusters is defined as the average distance between each member of one cluster to every member of the other cluster.

4. Ward’s Method:

• Objective: The main goal of Ward’s method is to find the pair of clusters that, when merged, will increase the total within-cluster variance as little as possible. This is like trying to keep the clusters as compact as possible.
• Within-Cluster Variance: This is a measure of how spread out the points are within a cluster. A lower within-cluster variance means the points are closer to each other, and therefore, the cluster is more compact.
• How it Works: At each step of the algorithm, Ward’s method looks at all possible pairs of clusters and calculates how much the within-cluster variance would increase if those two clusters were merged. It then merges the two clusters that result in the smallest increase in variance.
• Resulting Clusters: Because Ward’s method tries to keep the within-cluster variance low, it tends to create that are compact and roughly spherical in shape. This can be particularly effective if the natural groups in your data are also compact and spherical.

Space Complexity: O(N*N)

Time Complexity: O(NNN)

Advantages

1. Discovery of Hierarchical Structure: The algorithm reveals the hierarchy and nested structure within the data, which can be informative for understanding complex relationships.
2. Useful for Any Distance Measure: The method can be used with any distance measure, which is beneficial for different types of data, such as genomic data or mixed data types.
3. Does Not Assume Clusters as Spherical: Unlike K-means, agglomerative clustering does not assume that clusters are spherical in shape, which can result in more natural cluster shapes.
4. Easy to Implement and Understand: The algorithm is conceptually simple and can be easily implemented, making it accessible for users with varying levels of expertise.
5. Robust to Noise and Outliers: With the appropriate choise of linkage criteria, hierarchical clustering can be relatively robust to noise and outliers, as these will typically be merged into clusters at later stages of the process.

Disadvantages

1. Computational Complexity: One of the biggest drawbacks is its computational cost. The algorithm has a time complexity of O(n³) and space complexity of O(n²) for the simplest implementations, making it impractical for large datasets.
2. Sensitivity to Noise and Outliers: Certain linkage criteria, such as single linkage, can be highly sensitive to noise and outliers, which can lead to misleading results. Outliers can cause clusters to merge prematurely, distorting the true structure of the data.
3. Difficulty in Identifying the Number of Clusters: While the dendrogram can provide insights into the potential number of clusters, there is often subjectivity involved in interpreting where to ‘cut’ the dendrogram to define the clusters.
4. Arbitrary Decisions in Linkage Criteria: The choice of linkage criteria can significantly affect the results, and there is no definitive rule for choosing the best method, which can make the process somewhat arbitrary.
5. No Global Objective Function: Unlike K-means, which minimizes within-cluster variance, there’s no clear global objective in hierarchical clustering, which can make it difficult to assess the quality of the resulting clusters.

Gaussian Mixture Model

A Gaussian Mixture Model (GMM) is a probabilistic model based clustering technique where we assume that data is generated from a mixture of several Gaussian distributions, each with its own mean and covariance. GMMs are used extensively in statistics, machine learning, and data analysis for tasks like clustering, density estimation, and pattern recognition.

1. Clustering Capabilities: Unlike hard clustering methods like K-means, GMMs provide soft clustering, assigning probabilities of membership to each cluster. This is particularly useful in scenarios where data points do not distinctly belong to one group or another but share properties with multiple clusters. It can also cluster data with non-spherical shape.
2. Density Estimation: GMMs are used for density estimation, which is the process of estimating the probability distribution underlying a dataset. Understanding this distribution is crucial in many areas, including risk assessment in finance, image processing in computer vision, and anomaly detection in various domains.
3. Handling correlated and Complex Data: With their ability to model different covariance structures, GMMs can handle complex datasets where features are correlated, unlike algorithms that assume feature independence.
4. Anomaly Detection and Novelty Detection: GMMs can be used to identify data points that do not fit well into any of the Gaussian components, which is useful for anomaly or outlier detection.

EM Algorithm

Initialization

1. Choose the Number of Components(k): Decide on the number of Gaussian distributions (components) to use in the model.
2. Initialize Parameters:

• Initialize the means(μ1,μ2,….μk), covariances(Σ1,Σ2,…..Σk) and mixture weights(π1,π2…..,πk) for each components.
• These can be initialized randomly or based on some heuristic.

Expectation-Maximization Algorithm

Repeat the following steps until convergence (i.e. Until the change in the parameters becomes negligible or a maximum number of iterations is reached).

Expectation Step(E-Step)

1. Calculate Responsibilities:

• For each data point xi and each component k, calculate the responsibility yk(xi), using the formula:

Maximization Step(M-Step)

1. Update Parameters: Update the parameters for each gaussian component based on the responsibilities calculated in E-Step.
2. Updated Mean(μk):

3. Updated Covariance(Σk):

4. Updated Mixture weight(πk):

Convergence Check

• After each iteration, check if the change in the parameters(mean, covariance, weight) is below the certain threshold.
• Alternative, check if the log-likelihood of data given the model is not increased significantly, which is the sign of convergence.

Output

The final model parameters(mean, covariance, weight) after convergence are used to represent the data as a mixture of gaussian distribution.

Covariance Type

1. Spherical:

• In this type, each Gaussian component has its own single variance.
• All dimensions are assumed to have the same variance, which means that the component has a spherical shape in the feature space.
• This is the simplest covariance type, leading to equally sized clusters across all dimensions.

2. Diagonal:

• Each Gaussian component has its own diagonal covariance matrix.
• This means that the variances can differ across dimensions, but covariances between dimensions(off-diagonal elements) are assumed to be zero.
• The resulting clusters can have different sizes but are not rotated-i.e., they are aligned with the coordinate axes.

3. Tied:

• All Gaussian components share the same general covariance matrix.
• This means that all components have the same shape, size, and orientation, but they can have different means(centers).
• This type can capture complex shapes but assumes that the same shape applies to all clusters in the data.

4. Full:

• Each Gaussian component has its own general covariance matrix.
• This is the most flexible type, as it allows each cluster to have its own shape, size and orientation in the feature space.
• While it can model the data most accurately, it also requires the most parameters and can be prone to overfitting, especially in cases of limited data or high dimensionality.

How to decide n_components?

Likelihood formula for a gaussian mixture model

Given a GMM with K gaussian components, each with its own parameters, the likelihood for datasets is:

where p(xi) is the probability of observing the data point xi under the GMM, which is the weighted sum of the gaussian components:

• Here, N(xi|μk,Σk) is the probability density of xi under the kth gaussian distribution, which can be calculated using the multivariate normal density function.
• πk is the weight of the kth gaussian component in the mixture.

Akaike Information Criterion(AIC):

The AIC is calculated using the following formula:

AIC=2k-2ln(L)

where:

• k is the number of estimated parameters in the model.
• L is the maximum value of the likelihood function for the model.

The AIC balances the complexity of the model against the gooness of fit. A lower AIC value suggest the best model. When comparing models, the one with the lower AIC is generally preferred.

Bayesian Information Criterion(BIC):

The BIC is calculated using the formula:

BIC=ln(n)k-2ln(L)

where:

• n is the number of datapoints.
• k is the number of estimated parameters in the model.
• L is the maximum likelihood of the model.

Like the AIC, the BIC balances model complexity against fit, but it include a penalty term for the number of datapoints, making it more stringent against complex model as the sample size increases. A lower BIC value indicate a better model.

Code:

# Try GMMs with 1 to N components
N = 10
aics = []
bics = []
for i in range(1, N+1):
    gmm = GaussianMixture(n_components=i).fit(X)
    aics.append(gmm.aic(X))
    bics.append(gmm.bic(X))

# Plotting the AIC and BIC
plt.plot(range(1, N+1), aics, label='AIC')
plt.plot(range(1, N+1), bics, label='BIC')
plt.legend()
plt.xlabel('Number of Components')
plt.ylabel('Information Criterion')
plt.show()

T-SNE

What is T-SNE

t-SNE, or t-Distributed Stochastic Neighbor Embedding, is a statistical method for visualizing high-dimensional data by reducing it to lower-dimensional spaces, typically two or three dimensions. This makes it easier to visualize and interpret the data, especially when dealing with complex datasets like those in machine learning and data science.

Why learn T-SNE

Although PCA can also be used to reduce high-dimensional data into 2D or 3D for visualization, T-SNE is often preferred because:

• If the data has non-linear relationships, PCA may struggle to capture the structure, whereas T-SNE can handle such complexity effectively.
• T-SNE produces more meaningful and visually interpretable graphs, making patterns and clusters in the data easier to observe.

Geometric Intuition

For simplification, let’s assume the high-dimensional data is 2D and the reduced data is 1D. T-SNE focuses on preserving the local structure of the data rather than the global structure. In other words, it emphasizes the relative distances between nearby points (whether they are close or far) or focuses on the neighborhood instead of trying to preserve the exact global distances among all points.

We calculate pairwise similarities for all points to determine which points are neighbors and which are far apart. A natural first idea is to use the Euclidean distance as a measure of similarity. However, in high-dimensional spaces, distances become less reliable due to the curse of dimensionality.

To address this, T-SNE models the similarity of points using a Gaussian distribution centered at each point. The mean is set to the point itself, and the variance (σ) is chosen based on the local density around that point:

• If the density is high, σ is small (narrow Gaussian).
• If the density is low, σ is large (wider Gaussian).

Each point’s similarity to others is then defined by the probability density function (PDF) of this Gaussian, evaluated at the distances of other points. This probability represents how likely two points are to be considered neighbors.

By calculating the pairwise similarity of each point with every other point, we obtain a matrix of conditional probabilities. However, note that p(xj∣xi) is generally not equal to p(xi∣xj), because the variance (σi​) used in the Gaussian distribution is specific to each point. This means the neighborhood definition is asymmetric — point xi​ may consider xj as a close neighbor, but the reverse may not hold with the same probability.

Steps to work with T-SNE:

1. Random initialization: Place all the points randomly on a 1D line (low-dimensional space).
2. Compute similarities: Based on this arrangement, calculate a similarity matrix in the low-dimensional space.
3. Compare with high-dimensional similarities: Now we have two matrices — one from the high-dimensional space and one from the low-dimensional space. Compare them row by row.
4. Update positions: Calculate the differences between the two matrices, sum them up, and move the points in the low-dimensional space according to these differences.
5. Iterate: Repeat steps 2–4 until the low-dimensional structure closely reflects the local structure of the high-dimensional data.

Mathematical Formulation

1. High-Dimensional Similarities:

• Given a set of N points in a high-dimensional space, {x1,x2…..xN}, the similarity of datapoint xj to datapoint xi is represented as a conditional probability Pj|i, which is the probability that xi would pick xj as its neighbor.
• This probability is given by the Gaussian Distribution centered on xi:

• Here, ||xi-xj|| is the Euclidean distance between xi and xj, and σi is the variance of the Gaussian that is centered on datapoint xi. The value of σi is chosen such that the perplexity of the conditional distribution equals a predefined perplexity.

The probabilities are symmetrized using:

2. Low-Dimensional Similarities:

• In the low-dimensional space, for a corresponding set of points {y1,y2…..yN}, the similarity of datapoint yj to datapoint yi is given by a similar conditional probability, but using a Student’s t-distribution (with one degree of freedom, equivalent to the cauchy distribution) instead of the Gaussian distribution.

• This heavy-tailed distribution in the low-dimensional space is what helps t-SNE to alleviate the crowding problem.

3. Cost Function(Kullback-Leibler Divergence):

• The t-SNE algorithm aims to minimize the difference between these two probability distributions P and Q, which is quantified by the Kullback-Leibler(KL) divergence:

• This is a measure of how one probability distribution diverges from a second, expected probability distribution.

4. Optimization:

• The positions of points yi in the low-dimensional space are optimized (usually through gradient descent) to minimize this KL divergence.
• The gradient of the KL divergence with respect to the point yi can be computed, and this gradient is used to update the positions of the points in the low-dimensional map.

Why use gaussian distribution to calculate similarity in high dimension?

1. control of density with the variance parameter.
2. Graceful handling of distance.
3. Distance to probability output.
4. Differentiable.

How is variance calculated for each gaussian distribution?

1. Define Perplexity

• Perplexity: is a measure set by the user that indirectly controls the number of effective nearest neighbors. It reflects the expected density around a point in the high dimensional space.

2. Initialize Variances

• For each point in the dataset, initialize a variance for the Gaussian distribution that will be used to calculated probabilities between this point in the high dimensional space.

3. Calculate Conditional Probabilities

• For each point i, calculate the conditional probability pj|i that point i would pick point j as its neighbor. This is done using the Gaussian distribution centered on point i with variance.

• These probabilities are normalized so that they sum to 1 for each point i.

4. Calculate Shannon Entropy and Perplexity

• Compute the Shannon entropy H(Pi) of the conditional probability distribution for each point i:

• The perplexity is then calculated as 2^(H(Pi)), which represents the effective number of neighbors around point i.

5. Adjust Variance to Match User-Specified Perplexity

• For each point, adjust the variance so that the calculated perplexity from the conditional probabilities matches the user-specified perplexity, This involves:

• Binary Search: Iteratively adjust variance through binary search. If the calculated perplexity is too high, decrease variance. If too low, increase variance.

• Convergence: Continue adjusting until the calculated perplexity closely matches the user-specified perplexity. This ensures that the local structure around each point consistent with the user’s expectations.

Why use T-distribution in low dimension?

1. The “Crowding Problem”

• When you project high-dimensional data into a low-dimensional space, points that were moderately far apart in high dimensions tend to collapse together.
• If you use a Gaussian distribution in both spaces, the low-dimensional representation cannot give enough “room” for moderately distant points → everything gets crowded in the center.

2. Heavy Tails of the t-distribution

• The Student’s t-distribution with 1 degree of freedom (Cauchy distribution) has much heavier tails than a Gaussian.
• This means:
• Nearby points stay close (good for preserving local structure).
• Distant points are pushed far away (because the probability decays slowly).
• This spreads out the points better in 2D/3D and avoids crowding.

3. Matching Similarities Between Spaces

• In high-dimensional space: pairwise similarities are modeled with Gaussian probabilities (since distances concentrate well in high-dim).
• In low-dimensional space: we need a distribution that allows more flexibility to represent both close neighbors and distant non-neighbors.
• The t-distribution’s heavy tails ensure that dissimilar points in high-dim remain far apart in the low-dim map.

Hyperparameters

Perplexity:

• Perplexity is perhaps the most important hyperparameter in t-SNE. It can be thought of a measure of the effective number of neighbors for each point.
• The value of perplexity affects the balance between local and global aspects of your data. A small perplexity emphasizes local structure, while a larger perplexity brings more of the global structures into play.
• Typical values for perplexity range between 5 and 50, but this can vary depending on the dataset. It’s often recommended to experiment with different values to see how they affect the results.

Learning Rate:

• The learning rate determines the step size at each iteration while moving toward a minimum of the cost function.
• A too high learning rate might cause the algorithm to oscillate and miss the global minimum, while a too low learning rate can result in a long training process that might get stuck in a local minimum.
• Common values for the learning rate are between 10 and 1000. Again, experimenting with different values is key to finding the best setting for a given dataset.

Number of Iterations:

• This hyperparameter controls how many iterations the algorithm runs before it terminates.
• If the number is too low, the algorithm might not fully converge. If it’s too high, you might waste computational resources without gaining much in terms of the quality of the embedding.
• The default number of iterations is often set to a value like 1000, but this might need to be increased for larger datasets.

Points to remember

1. Interpreting Clusters: t-SNE can reveal clusters and local structures very effectively. However, the distance between clusters or the relative position of clusters in the plot may not have a meaningful interpretation. Avoid over-interpreting global relationships.
2. Axes have no meaning: Axes in T-sne has no interpretable meaning.
3. Perplexity Matters: Perplexity is a crucial hyperparameter in t-SNE. It roughly corresponds to the number of effective nearest Neighbors. There’s no one-size-fits-all value; different values can reveal different structures, so experiment with a range of values. Common values are between 5 and 50.
4. Reproducibility: t-SNE starts with a random initialization, leading to different results each time you run it. If reproducibility is important, set a random seed. Also, multiple runs with different initializations can give a fuller picture of your data’s structure.
5. Scaling the Data: Pre-processing steps like scaling or normalizing your data, especially if features are on different scales, can have a significant impact on the results of t-SNE.
6. Curse of Dimensionality: t-SNE can mitigate but not completely overcome the curse of dimensionality. Very high-dimensional data might require other steps, like initial dimensionality reduction with PCA, before applying t-SNE.
7. Learning Rate and Number of Iterations: Beyond perplexity, other parameters like the learning rate and the number of iterations also impact the results. A learning rate that’s too high or too low can lead to poor embeddings, and insufficient iterations might mean the algorithm doesn’t fully converge.
8. It’s Not a Silver Bullet: While t-SNE is a powerful tool, it’s not suitable for every kind of dataset or analysis. Sometimes other dimensionality reduction techniques like PCA, UMAP, or MDS might be more appropriate.

Advantages and Disadvantages

Advantages

• If done correctly, can give very intuitive visualizations as it preserves the local structure of the data in the lower dimensions.

Disadvantages

• Computationally Expensive
• Not very good at preserving global structure
• Sensitive to hyperparameters
• Can get stuck in local minima
• Interpretation is challenging