Normal Distribution, Standard Normal Distribution, Statistical Moments, Skewness and Kurtosis

Normal Distribution (Gaussian Distribution)

The **Normal Distribution, also known as the Bell Curve, is the most important probability distribution function (PDF) in statistical analysis. It is a continuous probability distribution that is **perfectly symmetrical around its center, creating a distinct bell-shaped curve. Most of the data points cluster near the middle, with very few points found far away. The curve features two floating “tails” that are asymptotic in nature — meaning they stretch towards infinity in both directions but theoretically never touch the horizontal axis. Many natural phenomena naturally follow a normal distribution, making it incredibly useful. Examples include human heights, the weights of objects produced by a machine, measurement errors, and IQ scores. Here, f(x) is probability density.

The normal distribution is characterized by just two parameters:

• The Mean, μ: Represents the center or peak of the distribution
• The Standard Deviation, σ: Represents the spread or width of the distribution.

Normal Distribution Denoted by

PDF Equation of Normal Distribution

Normal Distribution Graph

Standard Normal Distribution (Standard Normal Variate)

What if we want to compare two completely different normal distributions? E.g., comparing SAT scores to ACT scores?

We use the Standard Normal Distribution, denoted by Z. It is a standardized version of the normal distribution where the mean is exactly 0 and the standard deviation is exactly 1. Standardizing allows us to compare different datasets on the same scale and easily calculate probabilities using a using standardized normal tables like *Z-table*. We convert a regular normal distribution to a standard normal distribution using the Z-score formula:

Standard Normal Distribution Denoted by

PDF Equation of Normal Distribution

PDF of Normal and Standard Normal Distributions

A Z-value (or Z-score) tells you exactly how many standard deviations a raw data point (X) is above or below the population mean. A positive Z-score means the value is above the mean, while a negative Z-score means it is below.

The Z-table tells the probability from negative infinite to the z-score.

Question: Suppose the heights of adult males in a population follow a normal distribution with a mean of 68 inches and a standard deviation of 3 inches. What is the probability that a randomly selected male is taller than 72 inches?

Answer:

1. Calculate the Z-score: Z = (72 - 68) / 3 = 4 / 3 = 1.33
2. Look up 1.33 on a Z-table. The table tells us the probability from negative infinity up to that point. The area for 1.33 is approximately 0.9082 (or 90.82%).
3. This means there is a 90.82% chance a person is shorter than 72 inches.
4. To find the probability they are taller, subtract from 1: 1 - 0.9082 = 0.0918.
5. Result: There is a 9.18% probability the person is taller than 72 inches.

Question: For a Normal Distribution X ~ (μ, σ²), what percentage of the population lies exactly between the mean and 1, 2, and 3 standard deviations to the right?

Answer: Since the mean has a Z-score of 0, exactly 50% of the data lies below it.

• 1st Standard Deviation (Z = 1): The Z-table value is 0.8413. The population lying between the mean and 1 SD is 84.13% - 50% = 34.13%.
• 2nd Standard Deviation (Z = 2): The Z-table value is 0.9772. The population between the mean and 2 SD is 97.72% - 50% = 47.72%.
• 3rd Standard Deviation (Z = 3): The Z-table value is 0.9986. The population between the mean and 3 SD is 99.86% - 50% = 49.86%.

Alternative Solution,

• z-value for mean = (μ-μ)/σ² = 0/σ² = 0. so, population less than mean = 50%,
• z-value for 1st std = (μ + σ² — μ)/σ² = σ²/σ² = 1. so, population less than 1st std = 84.13%,
• z-value for 2nd std = (μ + 2*σ² — μ)/σ²* = 2σ²/σ²** = 2. so, population less than 2nd std = 97.72% and
• z-value for 3rd std = (μ + 3*σ² — μ)/σ²* = 3σ²/σ²** = 3. so, population less than 3rd std = 99.86%.
• So, population lie between mean and 1st std = 84.13% - 50% = 34.13%,
• population lie between mean and 2nd std = 97.72% - 50% = 47.72% and
• population lie between mean and 3rd std = 99.86% - 50% = 49.86%.

Properties of the Normal Distribution

• Symmetricity: The left side of the curve is a perfect mirror image of the right side.

Symmetricity of Normal Distribution

• All Measures of Central Tendencies are Equal: The Mean, Median, and Mode all fall at the exact same point (the highest peak of the curve).
• The Empirical Rule (68–95–99.7 Rule): If we look at both sides of the mean, we find a universal rule for normal distributions:

1. ~68% of the data falls within +/- 1 standard deviation.
2. ~95% of the data falls within +/- 2 standard deviations.
3. ~99.7% of the data falls within +/- 3 standard deviations.

Empirical Rule

• Area Under the Curve: The total area under the probability density curve is exactly equal to 1 (or 100%).

CDF of Normal and Standard Normal Distributions

CDF of Normal and Standard Normal Distributions

How We Use Standard Normal Distribution in Data Science?

• Outlier Detection: Because 99.73% of data falls within 3 standard deviations of the mean, any value beyond 3 standard deviations (a Z-score > 3 or < -3) is exceptionally rare (0.27% chance) and is standardly treated as an outlier.
• Machine Learning Algorithms: Algorithms like Gaussian Mixture Models (GMM) explicitly rely on normal distributions. Additionally, while Linear Regression strictly assumes the residuals (errors) are normally distributed, transforming your input features to be normally distributed often yields better model performance.
• Hypothesis Testing: Many parametric statistical tests (like t-tests and ANOVA) assume the underlying sample data is normally distributed.
• Central Limit Theorem (CLT): The CLT does not assume data is normally distributed. In fact, it is powerful for the exact opposite reason! The CLT states that no matter what the original distribution of your data is, if you take large enough samples, the distribution of the sample means will always approximate a normal distribution.

The 4 Statistical Moments

To fully describe the shape of any dataset mathematically, statisticians use “Moments.”

1. First Moment (Mean): A measure of location or central tendency.
2. Second Moment (Variance): A measure of spread or dispersion (Standard Deviation is the square root of Variance).
3. Third Moment (Skewness): A measure of asymmetry.
4. Fourth Moment (Kurtosis): A measure of “tailedness” or how heavy/fat the tails of the distribution are compared to a normal distribution.

The 4 Statistical Moments of Data Distribution

Skewness: When Data Isn’t Normal

Skewness indicates that the data is not symmetrical, meaning it is not normally distributed. It is a statistical measure of the asymmetry of a probability distribution or say degree to which a dataset deviates from the normal distribution. In a symmetrical distribution, the Mean = Median = Mode. In a skewed distribution, these three metrics are not equal, and the distribution develops a longer “tail” on one side.

• Positive (Right) Skewness: The tail extends further to the right. The Mean is pulled to the right by large outliers, making it greater than the Median (Mean > Median > Mode). Example: Wealth distribution.
• Negative (Left) Skewness: The tail extends further to the left. The Mean is pulled down by small outliers, making it less than the Median (Mean < Median < Mode). Example: Age of retirement.
• Zero Skewness: A perfectly symmetrical distribution (like the Normal Distribution Mean = Median = Mode).

Types of Skewness

Measuring Skewness in Python: df['column_name'].skew().

• -0.5 to 0.5: Almost symmetrical. and can be considered symmetrical to practical purpose.
• -1.0 to -0.5 OR 0.5 to 1.0: Moderately skewed.
• Less than -1.0 OR Greater than 1.0: Highly skewed.

NOTE: The greater the skewness, the greater the distance between mean, median and mode.

Sample Skew Formula used by pandas pd.skew()

Note: Just because data has a skewness near 0 does not automatically guarantee it is perfectly Normally Distributed (it could be a uniform distribution or bimodal, etc.). It simply means it is symmetrical. Thus also look for normal distribution of PDF.

Symmetric and Skew but not Normal

Kurtosis: tells about tails!

In probability theory and statistics, Kurtosis (from the Greek word for “curved” or “arching”) is the 4th statistical moment. It is a measure of the “tailedness” of the PDF of a real-valued random variable. QQ plots are better diagnostic tools for kurtosis analysis.

Note: Historically, many people incorrectly taught that kurtosis measured the “peakedness” (how sharp or flat the top of the curve is). Modern statisticians agree: kurtosis is a measure of “tailedness” only. It tells us about the extremities (outliers) of the distribution, not the center!

Example: Why do we care about tails? In finance, kurtosis risk refers to the risk associated with the possibility of extreme outcomes or “fat tails” in the distribution of an asset’s returns. If a distribution has high kurtosis, there is a higher likelihood of extreme events occurring — either massive positive gains or devastating negative losses — compared to a normal distribution. Investors must pay close attention to kurtosis risk; if an asset has “fat tails,” standard risk models (which often assume a normal distribution) might drastically underestimate the chance of a market crash.

Sample Kurtosis formula

Excess Kurtosis

*A standard Normal Distribution has a baseline kurtosis of exactly 3.* Because it’s easier to compare things to zero, statisticians usually look at Excess Kurtosis, which is simply **calculated by subtracting 3 from the actual kurtosis (sample kurtosis coefficient). It is a measure of how much more peaked or flat a distribution is compared to a normal distribution, which is considered to have a kurtosis of 0.

Excess Kurtosis Formula

Three types of Excess Kurtosis are:

Leptokurtic (Excess Kurtosis > 0)

• “Lepto” means “slender” or “thin.”
• A leptokurtic distribution has a positive excess kurtosis, meaning it has fatter, heavier tails than a normal distribution means there are more extreme values or outliers in the data.
• Example: Assets with leptokurtic returns are highly volatile and riskier. They are prone to sudden, aggressive price movements that result in massive gains or “black swan” losses.

Platykurtic (Excess Kurtosis < 0)

• “Platy” means “broad” or “flat” (think of a platypus).
• A platykurtic distribution has a negative excess kurtosis, meaning it has thinner, lighter tails than a normal distribution means there are fewer extreme values or outliers. The data is heavily concentrated around the mean without wild deviations.
• Example: Assets with platykurtic returns are generally safer and less volatile. They experience more gradual price movements and are very unlikely to produce extreme surprises.

Mesokurtic (Excess Kurtosis = 0)

• “Meso” means “middle.”
• A mesokurtic distribution has an excess kurtosis of exactly 0. It has the exact same baseline “tailedness” and outlier probability as a standard normal distribution.
• Example: In finance, a mesokurtic distribution is often considered the baseline for standard portfolio models, representing a predictable, standard balance between risk and return.

Types of Excess Kurtosis

How to Find if a Given Distribution is Normal?

Before applying machine learning algorithms that assume normality (like Linear Regression), you must test your data. Here are the four primary ways to do it:

1. Visual Inspection (Histograms & KDE): One of the easiest ways to check for normality is to visually inspect a histogram or a density plot of the data. A normal distribution has a bell-shaped curve, meaning the majority of the data falls in the middle, and the tails taper off symmetrically.

2. The Skewness Function: You can calculate the mathematical asymmetry of your data using pandas in Python: pd['column_name'].skew(). If the value is close to 0 (typically between -0.5 and 0.5), the data is highly symmetrical, which is a strong indicator (though not a guarantee) of normality.

3. QQ Plot (Quantile-Quantile Plot): A QQ plot is a powerful graphical tool used to assess if your dataset follows a specific theoretical distribution (like the Normal distribution i.e. called Normal Probability Plot). It assess the similarity of the distribution of two sets of data, first is given data and second is theoretical data with known distribution like normal.

How it works: It sorts your data and calculates its quantiles, then plots those exact quantiles on the Y-axis against the theoretical quantiles of a perfect Normal distribution on the X-axis.

• If your data is perfectly Normal, the points on the QQ plot will fall exactly along a straight 45-degree diagonal line.
• If the points curve away or deviate significantly from that straight line, your data is not normal. (Note: QQ plots can also be configured to test against Uniform or Pareto distributions, not just Normal!)

PDF and QQ Plot of Normal Distribution

QQ PLOT Types

QQ PLOT Types

4. Statistical Tests: There are several strict statistical tests used to test for normality, such as Shapiro-Wilk test (Best for small datasets), Kolmogorov-Smirnov test, Anderson-Darling test, etc.

These tests compare your observed data to the expected values of a normal distribution and provide a p-value. If the p-value is less than your significance level (usually 0.05), you reject the null hypothesis, meaning your data is not normal.