Understanding Correlation: Pearson, Spearman, and More

Correlation analysis is an essential tool in statistics and data science to measure the relationship between two variables. It helps us understand whether changes in one variable can be associated with changes in another, and if so, how strong and in what direction that association is. In this blog, we will dive into popular correlation measures: Pearson, Spearman, and Normalized Mutual Information. We’ll also explore Kendall’s Tau, Point-Biserial Correlation, and Distance Correlation to cover a range of methods for various data types and scenarios.

1. Pearson Correlation Coefficient (r)

Pearson’s correlation coefficient is a measure of the linear relationship between two continuous variables.

Range: -1 to +1

• +1: Perfect positive correlation (as X increases, Y increases)
• -1: Perfect negative correlation (as X increases, Y decreases)
• 0: No linear relationship

Assumptions:

1. Both variables are continuous and normally distributed.
2. The relationship is linear.
3. No significant outliers.

Why use Pearson Correlation?

It is the most straightforward way to measure linear association.

Often used in finance (e.g., stock price correlations) and scientific studies.

Example:

Let’s say you are measuring the relationship between study hours and exam scores. If Pearson’s r = 0.85, it means that as study hours increase, exam scores tend to increase too, with a strong positive linear relationship.

2. Spearman Rank Correlation (ρ)

Spearman’s correlation is a non-parametric measure of the monotonic relationship between two variables. It works by ranking the data points and then calculating the Pearson correlation of the ranks.

Range: Same as Pearson, from -1 to +1, but it measures monotonic relationships.

where di is the difference between the ranks of corresponding values.

Source: Laerd

When to use Spearman Correlation?

When the data is ordinal or not normally distributed.

When dealing with ranked data (e.g., survey responses on a scale of 1 to 10).

When the relationship is non-linear but monotonic.

Why use Spearman Correlation?

It can detect non-linear relationships.

It is robust to outliers and works well with ranked data.

Example:

Suppose you rank 10 movies by their box office revenue and user ratings. If Spearman’s ρ = 0.75, it suggests that higher box office revenue tends to align with higher user ratings, though not perfectly.

3. Normalized Mutual Information (NMI)

NMI is a measure of the mutual dependence between two variables. It quantifies how much information the variables share relative to their entropy.

Range: 0 (no shared information) to 1 (perfect alignment).

Why use NMI?

Useful in clustering validation to measure how well two clustering structures align.

Works for categorical data and is robust to non-linear dependencies.

Example:

Imagine you have two clusterings of customers — one based on purchase patterns and the other based on demographics. NMI can quantify the alignment between these two clusterings, helping you understand if one reflects the other.

4. Kendall’s Tau (τ)

Kendall’s Tau is another non-parametric measure of the strength and direction of a relationship between two ranked variables. It measures the concordance between pairs of observations.

Range: -1 to +1, with higher values indicating better agreement between ranks.

where C is the number of concordant pairs, and D is the number of discordant pairs.

When to use Kendall’s Tau?

For small datasets with tied ranks.

When you want a measure less sensitive to data errors than Spearman’s ρ.

Example

Voting Patterns: If two politicians rank public policy priorities similarly, Kendall’s Tau will give a high score.

5. Point-Biserial Correlation

This is a special case of Pearson’s correlation used when one variable is binary (0 or 1) and the other is continuous.

Range: 0 (no dependency) to 1 (strong dependency).

Why use Point-Biserial Correlation?

Helps measure the relationship between a binary and continuous variable.

When to Use Distance Correlation?

When dealing with complex, non-linear relationships.

Works well in scientific experiments where relationships may not follow predictable patterns.

Example

It can determine the correlation between gender (0 = Male, 1 = Female) and test scores.

6. Distance Correlation

Distance correlation is a measure that can detect non-linear relationships between variables, unlike Pearson and Spearman correlations, which assume specific types of relationships.

Why use Distance Correlation?

It works for all types of dependencies (linear and non-linear).

Useful for complex data relationships.

When to Use Point-Biserial Correlation?

For A/B testing, when comparing results of two groups (binary variable).

Works with binary classification models to see how continuous predictions align with ground truth labels.

Which Correlation Metric Should You Use?

Summary

Choosing the right correlation metric is essential for accurate insights. Pearson’s correlation works for simple linear relationships, Spearman helps with non-linear but monotonic relationships, and NMI comes into play for categorical data and clustering validation. Meanwhile, Kendall’s Tau gives insights into rank agreement, Distance Correlation detects non-linear patterns, and Point-Biserial bridges binary and continuous data.

By carefully selecting the appropriate correlation metric, you can extract meaningful insights from your data and improve decision-making — whether you’re analyzing stock prices, customer clusters, or experiment results.