Kolmogorov–Smirnov Test. Image Source : Walid Soula

How and when to use Kolmogorov–Smirnov Test

In the last article I covred Shapiro-Wilk test , a statistical test that helps us assess whether data follows a normal distribution and guides the choice between parametric and non-parametric approaches

But what if we want something more general, to check whether the data fits any theoretical distribution you specify (normal, uniform, exponential, …) ? That’s what the Kolmogorov–Smirnov test does! It’s a more general test that provides flexible options !

As always, if you find my articles interesting, don’t forget to clap and follow 👍🏼 These articles take time and effort to create!

How Kolmogorov–Smirnov Test works ?

As an engineer or a scientist, you surely want to understand how things work and not just be an integrator, right? In a nutshell, the K-S Test compares two things:

• Your sample’s empirical distribution function (EDF): the cumulative distribution of your observed data
• Theoretical cumulative distribution function (CDF): what the data would look like if it truly followed the distribution you’re testing against

The test then checks the maximum distance (D statistic) between these two curves

Kolmogorov–Smirnov test. Image Source: Wikipedia

And as you might think right now:

• If the distance is small => your data is close to the target distribution
• If the distance is large => your data likely does not follow that distribution

Of course, since it’s a statistical test, the p-value tells you whether the difference is statistically significant!

The math behind the test is as follows:

• Suppose you have a dataset of size n
• X1​,X2​,…,Xn​ (where each Xi is an observed data point, like “transaction amount”)

1 — Empirical Distribution Function (EDF) : The EDF tells you, for any value x, what proportion of your sample is less than or equal to x. Think of Fn(x) as the “cumulative percentage” from your data

Empirical Distribution Function (EDF). Image Source : Soula Walid

• x : a threshold value you pick (eg : 120 if you’re testing “transactions ≤ 120”)
• Fn(x) : the fraction of your dataset below that threshold

2 — Theoretical Cumulative Distribution Function (CDF) : F(x) has no single formula because it depends entirely on the distribution you’re testing against !

3 — Kolmogorov–Smirnov statistic : The test computes the maximum vertical gap between your data’s EDF and the theoretical CDF !

Kolmogorov–Smirnov statistic. Image Source : Walid Soula

D : the KS statistic, the largest observed difference

• If D is small, your data aligns well with the theoretical distribution
• If D is large, your data deviates too much

Let’s have an example to apply what you know from the article ! I will put multiple scenarios to fit real life application starting from the less likely

1 — One-sample KS test : you have a sample and the theoretical distribution and its parameters are known (rare in practice, but straightforward)

Formulate the hypothesis :

• 1 — Null hypothesis H0 : The data sample comes from the specified theoretical distribution F(x)
• 2 — The data does not come from the specified distribution

I will simulate a data of 100 values with a mean of 100 and a standard deviation of 20

import numpy as np
from scipy import stats

np.random.seed(0)
data = np.random.normal(loc=100, scale=20, size=100)

• I define a theoretical distribution parameters (In this ideal scenario, we know the exact parameters of the distribution under the null hypothesis where the data follows N(100,20))

mu_known = 100
sigma_known = 20

• Perform one-sample KS test : I will compare the empirical distribution function (EDF) of the sample with the CDF of the theoretical distribution

D, p_value = stats.kstest(data, 'norm', args=(mu_known, sigma_known))
print(f"KS statistic: {D:.4f}, p-value: {p_value:.4f}")
# KS statistic: 0.0582, p-value: 0.8668

At the 5% significance level, we fail to reject H0 (p-value: 0.8668). The data is consistent with coming from the specified normal distribution N(100,20)

You can add vizualisations too :

• We start by order the values of observations and compute the number of observations

xs = np.sort(data) 
n = len(xs)

• Building the empirical CDF and compute the theoretical CDF

Fn = np.arange(1, n+1) / n
F_theo = stats.norm.cdf(xs, loc=mu_known, scale=sigma_known)

• Measure the vertical gaps : At each observed value, compute how far the empirical CDF is from the theoretical CDF (Check just before the step and just after the step : Keep the biggest one, that’s the KS statistic)

gaps_before = np.abs((np.arange(0, n)/n) - F_theo)   # just before jump
gaps_after  = np.abs(Fn - F_theo)                    # just after jump
gaps = np.concatenate([gaps_before, gaps_after])

imax = np.argmax(gaps)
D_value = gaps[imax]
x_for_gap = xs[imax % n]   # map index back to x (where the largest gap happens)

• Plot empirical and theoretical CDFs, then mark the largest gap (KS Statistic)

# Plot empirical and theoretical CDFs
plt.step(xs, Fn, where='post', label="Empirical CDF", color="blue")
plt.plot(xs, F_theo, label="Theoretical CDF", color="black")

# Mark the largest gap (KS statistic)
plt.vlines(x_for_gap,
           F_theo[imax % n],
           (np.arange(0, n+1)/n)[imax % (n+1)],
           colors="red", linestyles="--",
           label=f"KS statistic D = {D_value:.3f}")

plt.xlabel("x")
plt.ylabel("Cumulative probability")
plt.title("Kolmogorov–Smirnov Test: Visualization of D")
plt.legend()
plt.show()

KS TEST. Image Source : Walid Soula

2 — One-sample KS test (unknown parameters) : You have a sample, but the theoretical distribution parameters are unknown (more realistic)

• H0: The data comes from a normal distribution (parameters estimated from the data)
• H1: The data does not come from a normal distribution.

I will state by simulate the data and estimate parameters from the sample, then perform the test

np.random.seed(0)
data = np.random.normal(loc=100, scale=20, size=100)
mu_est = np.mean(data)
sigma_est = np.std(data, ddof=1)
D, p_value = stats.kstest(data, 'norm', args=(mu_est, sigma_est))
print(f"KS statistic: {D:.4f}, p-value: {p_value:.4f}")
# KS statistic: 0.0643, p-value: 0.7778 fail to reject the null hypothesis

Finaly, vizualisation (same stuff)

xs = np.sort(data)
n = len(xs)

Fn = np.arange(1, n+1) / n
F_theo = stats.norm.cdf(xs, loc=mu_est, scale=sigma_est)

gaps_before = np.abs((np.arange(0, n)/n) - F_theo)
gaps_after  = np.abs(Fn - F_theo)
gaps = np.concatenate([gaps_before, gaps_after])

imax = np.argmax(gaps)
D_value = gaps[imax]
x_for_gap = xs[imax % n]

plt.step(xs, Fn, where='post', label="Empirical CDF", color="blue")
plt.plot(xs, F_theo, label="Theoretical CDF", color="black")
plt.vlines(x_for_gap,
           F_theo[imax % n],
           (np.arange(0, n+1)/n)[imax % (n+1)],
           colors="red", linestyles="--",
           label=f"KS statistic D = {D_value:.3f}")
plt.xlabel("x")
plt.ylabel("Cumulative probability")
plt.title("KS Test (Unknown Parameters)")
plt.legend()
plt.show()

KS TEST. Image Source : Walid Soula

3 — Two-sample KS test : Checking whether two samples come from the same continuous distribution, let’s formulate first the hypothesis :

• H0 ​: Both samples come from the same distribution (their CDFs are equal)
• H1 : The distributions differ

Next, I will run the test

res = stats.ks_2samp(a, b, alternative='two-sided', mode='auto')
D = res.statistic
p_value = res.pvalue
print(f"Two-sample KS: D = {D:.4f}, p-value = {p_value:.4f}")
# Two-sample KS: D = 0.1717, p-value = 0.0349

Since the p value <= 0.05, we reject the null hypothesis. The next step would be the same sorting stuff for both distribution to end with the vizualisation

# Prepare combined grid for plotting EDFs
xs = np.sort(np.concatenate([a, b]))
n_a = len(a)
n_b = len(b)

# Empirical CDF values at each xs for sample A and B
Fn_a = np.searchsorted(np.sort(a), xs, side='right') / n_a
Fn_b = np.searchsorted(np.sort(b), xs, side='right') / n_b

gaps = np.abs(Fn_a - Fn_b)
imax = np.argmax(gaps)
D_value = gaps[imax]
x_for_gap = xs[imax]

plt.step(xs, Fn_a, where='post', label='EDF A (sample a)', linewidth=1)
plt.step(xs, Fn_b, where='post', label='EDF B (sample b)', linewidth=1)

# Mark max gap
plt.vlines(x_for_gap,
           min(Fn_a[imax], Fn_b[imax]),
           max(Fn_a[imax], Fn_b[imax]),
           colors='red', linestyles='--', label=f"KS D = {D_value:.3f}")

plt.xlabel('x')
plt.ylabel('Cumulative probability')
plt.title('Two-sample KS: EDF A vs EDF B')
plt.legend()
plt.show()

Two-sample KS: EDF A vs EDF B. Image Source : Walid Soula

So, the Kolmogorov–Smirnov test helps you validate whether your data follows a given distribution by measuring the maximum gap between the empirical and theoretical cumulative distributions. If there’s a specific topic you’d like me to explore, please don’t hesitate to let me know. Your input helps shape the direction of my content and keeps it relevant and engaging 😀

Resources :

[embed]How to use Shapiro-Wilk test Learn how the Shapiro–Wilk test assesses normality and guides the choice between parametric and non-parametric methodsmedium.com

[embed]Data Science Digest Explore my curated playlist of essential Data science articles, covering topics from machine learning to data…medium.com

[embed]Dr. Walid Soula - Medium Read writing from Dr. Walid Soula on Medium. AI & Marketing Expert in Healthcare industriesmedium.com

Please consider the following and subscribe to the newsletter for more articles about business, data science, machine learning, and extended reality, it’s FREE! You can find my lists in the following links :

• Data Science Digest : https://medium.com/@soulawalid/list/statistics-data-science-65305693779d
• Generative AI : https://medium.com/@soulawalid/list/generative-ai-ee31117869a9
• Programming with Python : https://medium.com/@soulawalid/list/programming-c0a3ef000f5f
• Linguistic AI Lab : https://medium.com/@soulawalid/list/linguistic-ai-lab-9eb7d30369c1
• Strategic Business Intelligence : https://medium.com/@soulawalid/list/strategic-business-intelligence-1528f08575a7
• AI for Health Professionals : https://medium.com/@soulawalid/list/ai-for-health-professionals-f8b87eeab19f
• The Neuroscience of Consumer Behavior : https://medium.com/@soulawalid/list/the-neuroscience-of-consumer-behavior-8f94149e3c73
• Beyond Reality : https://medium.com/@soulawalid/list/beyond-reality-bf03607b0b80
• Quantum Leap : https://medium.com/@soulawalid/list/quantum-leap-be0b06f7a986

If you have any questions, you can ask me on LinkedIn, here is my profile: https://www.linkedin.com/in/oualid-soula/ Let’s connect!