Chi Square Distribution, Goodness of Fit Test & Test for Independence

Chi Square Distribution, Goodness of Fit Test and Test for Independence Explained

Photo by Jorgen Hendriksen on Unsplash

⬅️ Previous: *[Hypothesis Testing & Rejection Region Approach Explained]*

Chi Square Distribution (χ2 distribution)

It is a continuous probability distribution that is widely used in statistical hypothesis testing, particularly in the context of goodness-of-fit tests and tests for independence in contingency tables. It arises when the sum of the squares of independent standard normal random variables follows this distribution. The Chi-Square distribution has a single parameter, the degrees of freedom (df), which influences the shape and spread of the distribution. The degrees of freedom are typically associated with the number of independent variables or constraints in a statistical problem.

The Chi-Square distribution is used in various statistical tests, such as the Chi-Square goodness-of-fit test, which evaluates whether an observed frequency distribution fits an expected theoretical distribution, and the Chi-Square test for independence, which checks the association between categorical variables in a contingency table.

Chi Square Distribution

Some key properties of the Chi-Square distribution are:

1. It is a continuous distribution, defined for non-negative values.
2. It is positively skewed, with the degree of skewness decreasing as the degrees of freedom increase.
3. The mean of the Chi-Square distribution is equal to its degrees of freedom i.e. (𝜇 = k), and its variance is equal to twice the degrees of freedom i.e. (σ² = 2k).
4. As the degrees of freedom increase, the Chi-Square distribution approaches the normal distribution in shape.

Chi Square Tests

It is a statistical hypothesis test used to (determine if there is a significant association between categorical variables) or (if an observed distribution of categorical data differs from an expected theoretical distribution). It is based on the Chi-Square (χ2) distribution, and it is commonly applied in two main scenarios:

1. Chi-Square Goodness-of-Fit Test: This test is used to determine if the observed distribution of a single categorical variable matches an expected theoretical distribution. It is often applied to check if the data follows a specific probability distribution, such as the uniform or binomial distribution.
2. Chi-Square Test for Independence (Test for Association): This test is used to determine whether there is a significant association between two categorical variables in a sample.

Chi-Square Goodness of Fit Test

The Chi-Square Goodness-of-Fit test is a statistical hypothesis test used to determine if the observed distribution of a single categorical variable matches an expected theoretical distribution. It helps to evaluate whether the data follows a specific probability distribution, such as uniform, binomial, or Poisson distribution, among others. This test is particularly useful when you want to assess if the sample data is consistent with an assumed distribution or if there are significant deviations from the expected pattern.The Chi-Square Goodness-of-Fit test is a non-parametric test. Non-parametric tests do not assume that the data comes from a specific probability distribution or make any assumptions about population parameters like the mean or standard deviation. In the Chi-Square Goodness-of-Fit test, we compare the observed frequencies of the categorical data to the expected frequencies based on a hypothesized distribution. The test doesn’t rely on any assumptions about the underlying distribution’s parameters. Instead, it focuses on comparing observed counts to expected counts, making it a non-parametric test.

The Chi-Square Goodness-of-Fit test involves the following steps:

• Define the null hypothesis (H0) and the alternative hypothesis (H1): H0: The observed data follows the expected theoretical distribution. H1: The observed data does not follow the expected theoretical distribution.
• Calculate the expected frequencies for each category based on the theoretical distribution and the sample size.

• Compute the Chi-Square test statistic (χ²) by comparing the observed and expected frequencies. The test statistic is calculated as:
• where Oi is the observed frequency in category i, Ei is the expected frequency in category i, and the summation is taken over all categories.
• Determine the degrees of freedom (df), which is typically the number of categories minus one (df = k — 1), where k is the number of categories.
• Calculate the p-value for the test statistic using the Chi-Square distribution with the calculated degrees of freedom.
• Compare the test statistic to the critical value or the p-value

Assumptions:

1. Independence: The observations in the sample must be independent of each other. This means that the outcome of one observation should not influence the outcome of another observation.
2. Categorical data: The variable being analysed must be categorical, not continuous or ordinal. The data should be divided into mutually exclusive and exhaustive categories.
3. Expected frequency: Each category should have an expected frequency of at least 5. This guideline helps ensure that the Chi-Square distribution is a reasonable approximation for the distribution of the test statistic. Having small expected frequencies can lead to an inaccurate estimation of the Chi-Square distribution, potentially increasing the likelihood of a Type I error (incorrectly rejecting the null hypothesis) or a Type II error (incorrectly failing to reject the null hypothesis).
4. Fixed distribution: The theoretical distribution being compared to the observed data should be specified before the test is conducted. It is essential to avoid choosing a distribution based on the observed data, as doing so can lead to biased results.

Example 1: Suppose we have a six-sided fair die, and we want to test if the die is indeed fair. We roll the die 60 times and record the number of times each side comes up. We’ll use the Chi-Square Goodness-of-Fit test to determine if the observed frequencies are consistent with a fair die (i.e., a uniform distribution of the sides). alpha = 0.05 and observed frequencies are :

○ Side 1: 12 times ○ Side 2: 8 times ○ Side 3: 11 times ○ Side 4: 9 times ○ Side 5: 10 times ○ Side 6: 10 times

Chi-Square Goodness of Fit Test | Uniform

Alternative, Calculated χ² Score, 1 < Critical Value χ² Score, 11.07.

Example 2: Suppose a marketing team at a retail company wants to understand the distribution of visits to their website by day of the week. They have a hypothesis that visits are uniformly distributed across all days of the week, meaning they expect an equal number of visits on each day. They collected data on website visits for four weeks and want to test if the observed distribution matches the expected uniform distribution. Observed frequencies (number of website visits per day of the week for four weeks): • Monday: 420 • Tuesday: 380 • Wednesday: 410 • Thursday: 400 • Friday: 410 • Saturday: 430 • Sunday: 390

Ho : follow uniform distribution, Ha : don’t follow uniform distribution

Chi-Square Goodness of Fit Test | Uniform

Example: A survey of 800 families in a village with 4 children each revealed the following distribution: • 0 boys : 32 familes • 1 boys : 178 familes • 2 boys : 290 familes • 3 boys : 236 familes • 4 boys : 64 familes

Is this data consistent with the result that male and female births are equally probable?

Ho : P(m) = P(f) = 1/2 and H1 : P(m)=!P(f) and events are 5 (0, 1, 2, 3, 4 boys) while n = 4.

Theoritical Probability

• 0 boys : 32 familes : 50 familes • 1 boys : 178 familes : 200 familes • 2 boys : 290 familes : 300 familes • 3 boys : 236 familes : 200 familes • 4 boys : 64 familes : 50 familes

Chi-Square Goodness of Fit Test | Binomial Distribution

NOTE: rejecting Null Hypothesis means given data don’t have equal male and female probability of Binomial Distribution. But still it can be Binomial Distribution of different Probability that 0.5.

Chi-Square Test for Independence (Test for Association)

It is a statistical test used to determine whether there is a significant association between two categorical variables in a sample. It helps to identify if the occurrence of one variable is dependent on the occurrence of the other variable, or if they are independent of each other. The test is based on comparing the observed frequencies in a contingency table (a table that displays the frequency distribution of the variables) with the frequencies that would be expected under the assumption of independence between the two variables.

Test for Independence

where Oij is the observed frequency in each cell and Eij is the expected frequency.

Degrees of Freedom Chi-Square Test for Independence

The Chi-Square Test for Independence involves the following steps:

1. State the null hypothesis (H0) and alternative hypothesis (H1): ○ H0: There is no association between the two categorical variables (they are independent). ○ H1: There is an association between the two categorical variables (they are dependent).
2. Create a contingency table with the observed frequencies for each combination of the categories of the two variables.
3. Calculate the expected frequencies for each cell in the contingency table assuming that the null hypothesis is true (i.e., the variables are independent).
4. Compute the Chi-Square test statistic
5. Determine the degrees of freedom
6. Obtain the critical value or p-value using the Chi-Square distribution table or a statistical software/calculator with the given degrees of freedom and significance level (commonly α = 0.05).
7. Compare the test statistic to the critical value or the p-value to the significance level to decide whether to reject or fail to reject the null hypothesis. If the test statistic is greater than the critical value, or if the p-value is less than the significance level, we reject the null hypothesis and conclude that there is a significant association between the two variables.

Assumptions:

1. Independence of observations: The observations in the sample should be independent of each other. This means that the occurrence of one observation should not affect the occurrence of another observation. In practice, this usually implies that the data should be collected using a simple random sampling method.
2. Categorical variables: Both variables being tested must be categorical, either ordinal or nominal. The Chi-Square test for independence is not appropriate for continuous variables.
3. Adequate sample size: The sample size should be large enough to ensure that the expected frequency for each cell in the contingency table is sufficient. A common rule of thumb is that the expected frequency for each cell should be at least 5. If some cells have expected frequencies less than 5, the test may not be valid, and other methods like Fisher’s exact test may be more appropriate.
4. Fixed marginal totals: The marginal totals (the row and column sums of the contingency table) should be fixed before the data is collected. This is because the Chi-Square test for independence assesses the association between the two variables under the assumption that the marginal totals are fixed and not influenced by the relationship between the variables.

Example: A researcher wants to investigate if there is an association between the level of education (categorical variable) and the preference for a particular type of exercise (categorical variable) among a group of 150 individuals. The researcher collects data and creates the following contingency table.

Chi-Square Test for Independence

Applications in Machine Learning:

1. Feature selection: Chi-Square test can be used as a filter-based feature selection method to rank and select the most relevant categorical features in a dataset. By measuring the association between each categorical feature and the target variable, you can eliminate irrelevant or redundant features, which can help improve the performance and efficiency of machine learning models.
2. Evaluation of classification models: For multi-class classification problems, the Chi-Square test can be used to compare the observed and expected class frequencies in the confusion matrix. This can help assess the goodness of fit of the classification model, indicating how well the model’s predictions align with the actual class distributions.
3. Analysing relationships between categorical features: In exploratory data analysis, the Chi-Square test for independence can be applied to identify relationships between pairs of categorical features. Understanding these relationships can help inform feature engineering and provide insights into the underlying structure of the data.
4. Discretization of continuous variables: When converting continuous variables into categorical variables (binning), the Chi-Square test can be used to determine the optimal number of bins or intervals that best represent the relationship between the continuous variable and the target variable.
5. Variable selection in decision trees: Some decision tree algorithms, such as the CHAID (Chi-squared Automatic Interaction Detection) algorithm, use the Chi-Square test to determine the most significant splitting variables at each node in the tree. This helps construct more effective and interpretable decision trees.

➡️ Next Up: *[F — Distribution and ANOVA Explained]*

If you found value in this content, I invite you to stay connected for more insightful posts. Your time and interest are greatly appreciated. Thank you for reading!