Non-Normal Distributions and Transformations

Non-Gaussian Distribution (Non-Normal Distribution):

i. Continuous Non-Gaussian: **Continuous Uniform, Log normal and Parato **etc.

ii. **Discrete Non-Gaussian: **Discrete Uniform, Bernoulli, Binomial, Poisson etc.

Continuous Non-Gaussian

Continuous Uniform Distribution:

Uniform Distribution

In probability theory, a **Uniform Distribution is a distribution where **all outcomes are equally likely within a given range. If you select a random value from this range, any value is exactly as likely to be chosen as any other. Because it is a flat block, its skewness is always exactly 0.

Types of Uniform Distribution:

• **Discrete Uniform Distribution:** Rolling a fair 6-sided die. Getting a 1, 2, 3, 4, 5, or 6 all have the exact same 1/6 probability.
• **Continuous Uniform Distribution:** A random number generator picking a decimal between 0.0 and 1.0.

Examples of Continuous Uniform Distribution:

• Waiting for a subway train that arrives exactly every 20 minutes. If you show up at a random time, your wait time (anywhere from 0.00 to 19.99 minutes) is uniformly distributed.
• The exact coordinate where a raindrop lands on a perfectly flat, rectangular roof.
• The height of a person randomly selected from a group of individuals whose heights range from 5'6" to 6'0" would follow a continuous uniform distribution.
• The time it takes for a machine to produce a product, where the production time ranges from 5 to 10 minutes, would follow a continuous uniform distribution.
• The distance that a randomly selected car travels on a tank of gas, where the distance ranges from 300 to 400 miles, would follow a continuous uniform distribution.
• The weight of a randomly selected apple from a basket of apples that weighs between 100 and 200 grams, would follow a continuous uniform distribution.

Applications of Uniform Distribution in Data Science:

• Random Initialization: In algorithms like Neural Networks and K-Means clustering, starting parameters are often initialized uniformly to ensure no bias toward a specific starting value.
• Sampling: If you have a dataset with an equal number of samples from each class, you can use uniform distribution to randomly select a subset of the data that is representative of all the classes.
• Data augmentation: In some cases, you may want to artificially increase the size of your dataset by generating new examples that are similar to the original data. Uniform distribution can be used to generate new data points that are within a specified range of the original data.

PDF and QQ Plot of Continuous Uniform Distribution

• Hyperparameter Tuning: In random grid searches, defining a uniform prior distribution allows the algorithm to explore the hyperparameter space evenly.

Log-Normal Distribution:

A Log-Normal distribution is a heavy-tailed, continuously right-skewed distribution. It gets its name from a very specific mathematical rule: if you take the natural logarithm (log) of your data points, the resulting transformed data will form a perfect Normal Distribution. A right skewed distribution is log normal distribution only if logarithm of the distribution is normally distributed. Examples:

• The length of comments posted in internet discussion forums.
• Users’ dwell time on online articles.
• The distribution of wealth/income among 97%–99% of the population. etc.

Note: All log-normal distributions are right-skewed, but not all right-skewed distributions are log-normal.

How to check if a random variable is log normally distributed?

• Take the log of your variable. If the new histogram or kde looks like a bell curve (and passes a QQ plot test), the original variable was log-normally distributed.

Log Normal Distribution Denoted

PDF & CDF of Log Normal Distribution

Pareto Distribution

The Pareto distribution is a heavily skewed probability distribution used to model variables that exhibit Power-Law behavior. It has one parameter i.e. alpha, α. If α is smaller than tail is thicker and peak will be smaller. If α is infinite then y willl be straight line at Xm. Y is maximum at Xm. Example: The sizes of human settlements (few cities, many hamlets/villages), File size distribution of Internet traffic which uses the TCP protocol (many smaller files, few larger ones), etc.

Pareto Distribution Denoted

In mathematics, a Power Law is a functional relationship where one variable is proportional to a power of the other i.e. *y = kx^α**.

Vilfredo Pareto originally used this to describe the allocation of wealth, leading to the famous Pareto Principle or “80–20 Rule” (e.g., 20% of the population controls 80% of the wealth; 20% of your customers generate 80% of your sales).

Note: The strict 80–20 rule only occurs when the Pareto distribution’s shape parameter (α) is approximately 1.16.

Pareto distribution

PDF & CDF of Pareto distribution

How to detect a Pareto Distribution:

• Method 1: Plot the log(x) against the log(y) i.e. Log-Log Plot. If it forms a downward-sloping straight line, it follows a power law. here, y = pdf of x.

Log-Log Plot

• Method 2: Use a QQ plot configured for a Pareto distribution.

QQ Plot

Discrete Non-Gaussian

Bernoulli Distribution

Bernoulli distribution is a probability distribution that models a binary outcome, where the outcome can be either success (represented by the value 1) or failure (represented by the value 0). The Bernoulli distribution is named after the Swiss mathematician Jacob Bernoulli, who first introduced it in the late 1600s. The Bernoulli distribution is characterized by a single parameter, which is the probability of success, denoted by p. Also, 1 — p is probability of failure.

PMF of Bernoulli Distribution

The Bernoulli distribution is commonly used in machine learning (classification problem) for modelling binary outcomes, such as whether a customer will make a purchase or not, whether an email is spam or not, or whether a patient will have a certain disease or not.

PMF of Bernoulli Distribution

Binomial Distribution

Binomial distribution is a probability distribution that describes the number of successes in a fixed number of independent Bernoulli trials with two possible outcomes (often called “success” and “failure”), where the probability of success is constant for each trial. The binomial distribution is characterized by two parameters: the number of trials, n and the probability of success, p. Criteria for binomail distribution are :

• The process consists of n trials
• Only 2 exclusive outcomes are possible, a success and a failure.
• P(success) = p and P(failure) = 1 — p and it is fixed for all trails.
• The trials are independent.

PMF of Binomial Distribution

The Probability of getting head on a coin toss is 0.5. What is the probability that:

• No-one out of 3 will be head: 1/8
• 1 out of 3 will be head: 3/8
• 2 out of 3 will be head: 3/8
• 3 out of 3 will be head: 1/8

PMF of Binomial Distribution

Applications of Binomial Distribution:

• Binary classification problems: In binary classification problems, we often model the probability of an event happening as a binomial distribution. For example, in a spam detection system, we may model the probability of an email being spam or not spam using a binomial distribution.
• Hypothesis testing: In statistical hypothesis testing, we use the binomial distribution to calculate the probability of observing a certain number of successes in a given number of trials, assuming a null hypothesis is true. This can be used to make decisions about whether a certain hypothesis is supported by the data or not.
• Logistic regression: Logistic regression is a popular machine learning algorithm used for classification problems. It models the probability of an event happening as a logistic function of the input variables. Since the logistic function can be viewed as a transformation of a linear combination of inputs, the output of logistic regression can be thought of as a binomial distribution.
• A/B testing: A/B testing is a common technique used to compare two different versions of a product, web page, or marketing campaign. In A/B testing, we randomly assign individuals to one of two groups and compare the outcomes of interest between the groups. Since the outcomes are often binary (e.g., click-through rate or conversion rate), the binomial distribution can be used to model the distribution of outcomes and test for differences between the groups.

Transformations

What happens if your data is non-Gaussian, but your Machine Learning algorithm (like Linear Regression) performs best with normal data? You apply a Transformation. Transformations mathematically converts non-gaussian (non-normal distribution) to gaussian (normal distribution).

1. Function Transformations:

• Log Transformation (base 10 or natural 2): Taking the log of the data. Excellent for fixing right-skewed data. Cannot be applied to zero or negative values.

Log Transformation on Right Skewed Data (PDF & Histogram)

Log Transformation on Right Skewed Data (QQ Plot)

• Reciprocal Transformation (1/x): Turns smaller values into bigger ones and vice versa.
• Power Transformation (Square/Square Root): Often used to fix left-skewed data by squaring the values.
• Custom Transformation: any mathematical formula we can apply on values.

2. Power Transformations:

These are algorithmic transformations that generally perform better than basic function transformations because they mathematically search for the optimal curve.

• Box-Cox Transformation: Searches across an exponent parameter (lambda) from -5 to 5 to find the exact value that makes the data as normal as possible using Maximum Likelihood. Cannot be applied to* zero or negative values.*

Box-Cox Transformation Formula

Before and After of Box-Cox Transformation

Before and After of Box-Cox Transformation

• Yeo-Johnson Transformation: A modern variation of Box-Cox that can successfully handle zero and negative numbers. It is standard practice in modern pipelines and it works better than Box-Cox transformation.

Yeo-Johnson Transformation

Before and After of Yeo-Johnson Transformation

Before and After of Yeo-Johnson Transformation

3. Quantile Transformation:

This is a non-parametric transformation that maps the data’s probability distribution directly to a Normal distribution using the cumulative distribution function (CDF). It forces the data into a perfect bell curve, but it distorts linear correlations, so it should be used carefully!