Frequentist Null Hypothesis Testing

Let’s demystify Frequentist Null Hypothesis Testing. It’s a cornerstone of classical statistics and the framework behind familiar terms like “p-values” and “statistically significant.” We’ll focus on the core intuition using a simple, practical example.

Image source: From interesting article, must read https://towardsdatascience.com/a-story-of-frequentist-statistical-inference-12d38a7bcd77/

1. The Core Idea: Innocent Until Proven Guilty

Imagine you are a juror in a courtroom. The legal system operates on a powerful default assumption: the defendant is presumed innocent.

The prosecutor’s job is not to prove the defendant is “probably guilty.” Their job is to present evidence so overwhelming that the idea of the defendant being innocent becomes ridiculously unlikely. Only then can the jury reject the “innocent” assumption and declare them guilty.

Null Hypothesis Testing is the scientific equivalent of this courtroom.

The Null Hypothesis (H₀): This is your “presumption of innocence.” It’s the boring, default state of the world where nothing interesting is happening. It always contains a statement of equality or “no effect.”

• Courtroom Analogy: “The defendant is innocent.” (There is no guilt).
• Scientific Example: “This new drug has no effect on recovery time.”

The Alternative Hypothesis (H₁ or Hₐ): This is what the researcher (the “prosecutor”) is trying to prove. It’s the interesting, exciting claim that an effect exists.

• Courtroom Analogy: “The defendant is guilty.”
• Scientific Example: “This new drug does have an effect on recovery time.”

The Data (The Evidence): This is the data you collect from your experiment.

• Courtroom Analogy: Fingerprints, witness testimony, etc.

The P-value (The “Shock Factor”): This is the most important and often misunderstood part. The p-value is a probability that answers a very specific question:

• “If the null hypothesis were true (i.e., the defendant is innocent), what is the probability of seeing evidence this extreme or more extreme?”

The Verdict (The Conclusion):

• If the p-value is very small (typically < 0.05), it means your evidence is very “shocking” or surprising if you assume innocence. It’s so unlikely that you feel confident rejecting the null hypothesis in favor of the alternative. You are saying, “The idea that this drug has no effect is so improbable given my data that I’m going to conclude it does have an effect.”
• If the p-value is large (> 0.05), your evidence is not surprising. It’s perfectly plausible that you’d see this data even if the drug had no effect. Therefore, you fail to reject the null hypothesis. This is like a “not guilty” verdict. It doesn’t mean you’ve proven the defendant is innocent; it just means the prosecutor didn’t bring enough evidence to convince you otherwise.

2. A Simple Example: The “Miracle Grow” Fertilizer

Let’s put this into practice.

1. The Setup

Our Goal: We work for a company that created a new fertilizer, “Miracle Grow.” We want to know if it actually makes plants grow taller.

The Experiment: We take 20 tomato plants. We give 10 of them regular water (the “control” group) and 10 of them water with Miracle Grow (the “treatment” group). After a month, we measure the height of all 20 plants.

2. State the Hypotheses

Null Hypothesis (H₀): The “innocent” assumption. The fertilizer does nothing.

• “There is no difference in the average height between plants that receive Miracle Grow and plants that receive regular water.” (μ_miracle = μ_control)

Alternative Hypothesis (H₁): The exciting claim we want to prove.

• “There is a difference in the average height between the two groups.” (μ_miracle ≠ μ_control)

3. Collect the Data (The Evidence) We run the experiment and get our results. Let’s say the average heights are:

Control Group (Water Only): Average height = 25 cm

Treatment Group (Miracle Grow): Average height = 28 cm

It looks like there’s a difference! But is this 3 cm difference real, or could it just be due to random chance (some plants just happened to be stronger)? This is the question our statistical test will answer.

4. Perform the Statistical Test (Calculate the P-value) Since we are comparing the means of two independent groups with a small sample size, the correct tool is an independent t-test. We feed our data into the test.

The t-test will calculate our “shock factor,” the p-value. Let’s say the result is:

p-value = 0.03

5. Make a Decision (The Verdict)

The Threshold (Significance Level, α): We need to pre-define our threshold for what we consider “shocking.” By convention, this is almost always set to α = 0.05.

The Comparison: We compare our p-value to our threshold.

• 0.03 < 0.05

The Conclusion: Our p-value is smaller than our significance level. This means that if the fertilizer truly had no effect, there’s only a 3% chance we would have seen a height difference of 3 cm or more. That’s a very low chance!

The Final Statement: Because the result is so unlikely under the null hypothesis, we reject the null hypothesis. We conclude that there is a statistically significant difference in height between the two groups, and our Miracle Grow fertilizer likely had an effect.

3. Simple Python Code to Explain It

This code simulates our Miracle Grow experiment and runs the t-test.

import numpy as np
from scipy import stats

# --- 1. The Setup ---
# Define our significance level (the threshold for our "verdict")
alpha = 0.05
print(f"Significance Level (alpha): {alpha}\n")

# --- 2. The Hypotheses ---
print("--- Hypotheses ---")
print("Null Hypothesis (H₀): The average height of the Miracle Grow group is EQUAL to the control group.")
print("Alternative Hypothesis (H₁): The average height of the Miracle Grow group is NOT EQUAL to the control group.\n")

# --- 3. Collect the Data (Simulated) ---
# We create two sets of data. Notice the means are different.
# np.random.normal(mean, standard_deviation, sample_size)
control_group_heights = np.random.normal(25, 2, 10)  # Avg height = 25cm
miracle_grow_heights = np.random.normal(28, 2, 10) # Avg height = 28cm

print("--- The Evidence (Our Data) ---")
print(f"Control Group Heights (cm): {np.round(control_group_heights, 1)}")
print(f"Miracle Grow Heights (cm): {np.round(miracle_grow_heights, 1)}")
print(f"Average Control Height: {np.mean(control_group_heights):.2f} cm")
print(f"Average Miracle Grow Height: {np.mean(miracle_grow_heights):.2f} cm\n")

# --- 4. Perform the Statistical Test ---
# Use an independent t-test because we have two separate groups.
t_statistic, p_value = stats.ttest_ind(miracle_grow_heights, control_group_heights)

print("--- The Test ---")
print(f"Calculated P-value: {p_value:.4f}\n")

# --- 5. Make a Decision ---
print("--- The Verdict ---")
if p_value < alpha:
    print(f"Since the p-value ({p_value:.4f}) is less than our alpha ({alpha}), the result is 'shocking'.")
    print("We REJECT the Null Hypothesis.")
    print("Conclusion: There is a statistically significant difference. The fertilizer likely works!")
else:
    print(f"Since the p-value ({p_value:.4f}) is greater than our alpha ({alpha}), the result is not surprising.")
    print("We FAIL TO REJECT the Null Hypothesis.")
    print("Conclusion: We do not have enough evidence to say the fertilizer has an effect.")

Significance Level (alpha): 0.05

--- Hypotheses ---
Null Hypothesis (H₀): The average height of the Miracle Grow group is EQUAL to the control group.
Alternative Hypothesis (H₁): The average height of the Miracle Grow group is NOT EQUAL to the control group.

--- The Evidence (Our Data) ---
Control Group Heights (cm): [28.8 23.8 25.1 25.9 26.1 21.7 25.6 24.1 24.1 20.1]
Miracle Grow Heights (cm): [29.4 30.8 28.5 30.9 27.9 28.1 27.2 22.9 25.7 28. ]
Average Control Height: 24.54 cm
Average Miracle Grow Height: 27.94 cm

--- The Test ---
Calculated P-value: 0.0053

--- The Verdict ---
Since the p-value (0.0053) is less than our alpha (0.05), the result is 'shocking'.
We REJECT the Null Hypothesis.
Conclusion: There is a statistically significant difference. The fertilizer likely works!

Happy learning !!!