Dimensionality Reduction: A Beginner’s Guide to Principal Components Analysis (PCA)

Dimensionality reduction is actually a very easy concept to understand.

The idea is that if you have a machine learning dataset with a lot of features (meaning many columns of data), it is often beneficial to reduce the number of features you are working with without losing too much useful information.

Instead of simply deleting features, dimensionality reduction tries to combine related features into new ones that still preserve most of the important information in the data.

For example, imagine you are working with a dataset about students. You might have features such as homework scores, test scores, exam scores, attendance records, classroom participation, and project grades. Some of these features are likely closely related to one another. Do you really need all of them?

The GPA Analogy

A great real-life example of dimensionality reduction in action is a student’s GPA (Grade Point Average).

If you are hiring and want to assess a candidate’s academic performance, you could ask them to send over their full transcripts, covering every homework score, test result, exam grade, attendance record, and project mark. Or, you could simply ask for their GPA. The GPA is essentially all of those things boiled down into one number that gives you a reliable general sense of how the student performed academically.

That is exactly what dimensionality reduction does. It looks at all the features in your dataset and figures out how to compress them into a smaller number of features while still preserving most of the useful information. You are not throwing data away carelessly. You are distilling it.

What Is PCA?

One of the most common dimensionality reduction algorithms is Principal Components Analysis, usually shortened to PCA.

The name may seem crude at first, but the idea behind it is fairly intuitive. Of all the components in your data, which are the principal ones, the most important ones? And if the information across all of them matters, how do you intelligently boil them down into, say, five components?

PCA answers this by creating entirely new features called principal components. These components are designed to capture as much variation in the data as possible while using fewer dimensions. But this raises an important question: how does PCA decide what is important and what can be compressed away?

The answer lies in something called variance.

Understanding Variance

In statistics and machine learning, variance refers to how spread out the data points are. Features with higher variance tend to carry more information because their values differ meaningfully across observations. Features with very little variance often contribute less because the values barely change from one row to the next.

PCA finds the directions in your data where variation is highest. These directions become the principal components.

Here is a concrete way to picture it. Suppose you have data about people’s heights and weights. Since taller people often weigh more, if you plotted this data on a graph, the points would not scatter randomly. They would form a diagonal band running from the bottom-left to the top-right of the chart.

PCA looks at that cloud of points and identifies the direction where the data spreads out the most. In this case, that is roughly the diagonal line running through the middle of the cluster. That direction becomes the first principal component, because it captures the largest amount of variation in the dataset.

The second principal component then captures the next largest amount of variation, while being completely independent of the first. The process continues from there, with each new component capturing whatever variation the previous ones left behind.

PCA Does Not Simply Select Features

This is worth emphasising: PCA is not picking the “best” original features from your dataset and discarding the rest. It is creating entirely new transformed features that are combinations of the originals.

This is one reason PCA is so powerful. The principal components it creates often summarise the data more efficiently than any single original feature could on its own.

In practice, the benefits are significant. Reducing dimensions can make machine learning models train faster, reduce storage requirements, and help remove noise from data. It is also extremely useful for visualisation. Humans cannot easily interpret data beyond three dimensions, but PCA can compress a dataset with 100 features down to just 2 or 3 principal components, making it possible to plot and explore the data visually. That alone can surface patterns you would never spot otherwise.

PCA as an Unsupervised Learning Algorithm

PCA is also classified as an unsupervised learning algorithm.

In supervised learning, models learn from labelled data, meaning data where the correct answers are already provided. A model predicting house prices, for example, is trained on past sales where the prices are already known.

Unsupervised learning is different. There are no labels. The goal is to discover hidden patterns, structures, or relationships within the data itself. Clustering, association rule learning, and dimensionality reduction all fall under this umbrella:

• K-Means is a clustering algorithm
• Apriori is an association rule learning algorithm
• PCA is a dimensionality reduction algorithm

Final Thoughts

Despite the fact that PCA actually involves real mathematics behind the scenes (specifically the computation of eigenvectors and eigenvalues, which identify those high-variance directions in the data), the core idea is surprisingly simple:

Take a large number of features and compress them into a smaller number of meaningful components, while preserving as much useful information as possible.

If you are curious to go deeper, eigenvectors and eigenvalues are the natural next steps, but that’s linear algebra. I won’t be writing about maths for a while. I recommend Mathematics for Machine Learning by Imperial College on Coursera, or Mathematics for Machine Learning and Data Science by DeepLearning.AI on Coursera. The essence of linear algebra by 3Blue1Brown on YouTube is also a superb way to learn linear algebra foundations. All of that said, even without diving into the maths, understanding what PCA is doing and why is already a meaningful step forward.

See you in the next article, cheers!