UMAP and Clustering: When Dimensionality Reduction Becomes Self-Affirming

The Problem: Self-Affirming Clusters

UMAP (Uniform Manifold Approximation and Projection) is widely used to reduce high-dimensional data into 2D or 3D embeddings for visualization or preprocessing. Clustering algorithms — DBSCAN, OPTICS, Hierarchical Clustering (HCA) — then operate on these embeddings to detect groups.

At first glance, this seems natural: reduce the dimensionality, then cluster. But here’s the catch:

UMAP is designed to preserve local structure, not global distances. When you cluster on its output, you are effectively “forcing” clusters to appear where the embedding thinks they exist.

This can lead to self-affirming bias: the embedding makes clusters more separable than they actually are, and the clustering algorithm happily confirms them.

Why This Happens: Distance Distortion

UMAP constructs a high-dimensional graph of nearest neighbors, then optimizes a low-dimensional representation. Hyper-parameters like n_neighbors and min_dist control:

• How local vs. global structure is preserved
• How tightly points are packed in the embedding

Small min_dist → points collapse tightly, producing visually distinct clusters Large min_dist → clusters spread, with less pronounced separation

In other words, the embedding itself can artificially amplify or merge clusters. Any downstream clustering algorithm inherits this bias.

Experiment 1: Three Datasets, Three UMAP Configurations

We tested three datasets with strong cluster-invading noise:

1. Blobs: simple Gaussian clusters
2. Classification: overlapping clusters with redundancy
3. Anisotropic Gaussian mixture: clusters with different covariance shapes

We applied UMAP embeddings (min_dist = 0.0, 0.1, 0.9) and DBSCAN clustering.

Experiment 2: Hyper-parameter Exploration

We ran a grid search across:

• UMAP min_dist: 0.1 → 0.9
• DBSCAN eps: 0.1 → 0.9
• OPTICS max_eps: 0.1 → 0.9
• HCA distance threshold: 2 → 10

DBSCAN and HCA maintain relative stability across embeddings, while OPTICS deteriorates rapidly as embeddings become aggressive. Frequency plots confirm these trends: DBSCAN concentrates near zero Δ, HCA remains moderately stable, and OPTICS shows a wide spread.

Frequency and Correlation Analysis

DBSCAN seems to be working the best in our experiments, especially with a higher epsilon, followed by HCA. OPTICS seems not to work well at all.

The correlations and their significance point to an interesting hypothesis:

We can remediate performance by making the clustering algorithms “less eager” to cluster.

This is shown by:

• DBSCAN: Increasing eps makes the algorithm less likely to split clusters caused by local density fluctuations. Small eps creates over-fragmentation; larger eps merges nearby points more readily, reducing false clusters.
• OPTICS: Reducing max_eps or controlling reachability distances makes the algorithm less prone to merging clusters too aggressively.
• HCA (Agglomerative Clustering): Increasing the distance threshold allows clusters to grow more before being split, making the algorithm less sensitive to minor variations in embedding distances.

Conclusion

The results of clustering with UMAP embeddings are heavily dependent on:

• The hyperparameters of UMAP, those being n_neighbors and min_dist.
• The clustering algorithm that is used, e.g. DBSCAN, OPTICS and HCA
• We can (seemingly) remediate the performance of the clustering with hyperparameter tuning by reducing the eagerness to cluster

The takeaway is clear:

UMAP and clustering are self-affirming; don't use them together if you don't have to.

Sources:

• Source Code: https://github.com/psmgeelen/UMAPandClustering
• Scikit-Learn Clustering Basics: https://scikit-learn.org/stable/modules/clustering.html
• UMAP documentation: https://umap-learn.readthedocs.io/en/latest/