Generalized Linear Models (GLM)

A generalized linear model (GLM) is an extension of ordinary linear regression that allows for the dependent variable Y to have error distributions other than the normal distribution. In addition, a GLM does NOT assume a linear relationship between the response variable and the explanatory variables. As a result, GLMs can model different types of data including counts or binary outcomes.

A GLM specifies a distribution for the response variable Y and a link function that relates a linear model of the predictors to the expected value of the response variable Y. GLM does assume a linear relationship between the transformed expected response in terms of the link function and the explanatory variables.

The general form of a Generalized linear model (GLM) can be expressed as follows:

Here, μ is the mean response E[Y|X] and g is the link function:

Note that the independent variables can be nonlinear transformations of some original variables.

The Model Parameters in GLM are estimated using Maximum Likelihood Estimation (MLE).

Primary Assumptions for GLM:

GLM relies on several assumptions to ensure valid inference and predictions:

1- Independent Observations: There is no correlation between the errors or the response variables across observations (no autocorrelation).

2- No multicollinearity (predictors are not highly correlated): As discussed before for Linear Regression, multicollinearity occurs when two or more predictors in the model are highly correlated, leading to unreliable estimates of the regression coefficients.

3- Exogeneity of the predictors: The predictors are exogenous, meaning they are not correlated with the error term.

4- Correct distribution of the response variable: The dependent variable Y_i does NOT need to be normally distributed, but it typically assumes a distribution from an exponential family (e.g. binomial, Poisson, multinomial, normal, etc.). The specified distribution of the response variable is correct.

5- Correct specification of the link function: The relationship between the mean of the dependent variable μ and the linear predictor β’X should be appropriately modeled by the chosen link function.

Note that, unlike Linear Regression, the homogeneity of variance (homoscedasticity: constant variance) does NOT need to be satisfied. In fact, it is not even possible in many cases given the model structure.

Generalized linear models unify diverse statistical models:

(1) Linear Regression is a GLM with normally distributed response variable Y and an identity link function g, called identity link:

Recall (homoscedasticity) that for the Linear Regression, we have

(2) Poisson Regression is a GLM to model count data (e.g., number of events) and the log link function. In Poisson Regression, the response variable Y is assumed to follow a Poisson distribution, with arrival rate λ:

In Poisson Regression, the link function is a log link function,

By definition, the variance of a Poisson random variable is equal to its mean. For the Poisson Regression, we have

(3) Gamma Regression is a GLM to model positively skewed continuous data (e.g., waiting times). In Gamma Regression, the response variable Y≥0 is assumed to follow a Gamma Distribution with mean E[Y|X]=μ and shape parameter ν:

with a log-link function:

Thus, a plot of xij versus log(Yi) should be close to linear.

The Gamma Regression implies

(4) Logistic Regression is a GLM where the dependent variable Y follows a Bernoulli distribution with parameter p:

where p is the probability that Y=1. Note that for this Bernoulli distribution, E[Y|X]=μ=p.

In Logistic Regression, the link function is the logit function, the inverse of the logistic function:

(5) Negative Binomial Regression: The dependent variable is a count variable (non-negative integers) representing the number of occurrences of an event (e.g., number of doctor visits, number of accidents).

Similar to Poisson regression, Negative Binomial regression typically uses a log-link function to relate the mean of the response variable to the linear predictors:

Note on Overdispersion:

Overdispersion happens when the data show more variability than the statistical model expects. If the observed data have, i.e., Variance > Mean. This issue commonly arises in count data.

Example: Suppose you are modeling the number of customer complaints per day using a Poisson model. The Poisson model assumes that the mean and variance are equal.

• Model assumption: mean = 5 complaints/day → variance should also be 5.
• Actual data: mean = 5, but variance = 18.

Since 18 > 5, we have overdispersion. This means the Poisson model is underestimating variability.

Negative Binomial regression accounts for overdispersion. The Negative Binomial distribution can be thought of as a mixture of Poisson distributions, where the rate parameter itself follows a gamma distribution.

Advantages of GLMs:

• We can select any link function fitting many data problems (bounded to positive, bounded to [0,1], …)
• We can specify the data distribution (think of counts, success/failure, rates, …)
• GLMs do not need to transform the response to have a normal distribution.
• The choice of link is separate from the choice of random component, giving us more flexibility in modeling.
• The models are fitted via maximum likelihood estimation, so likelihood functions and parameter estimates benefit from asymptotic normal and chi-square distributions.
• Similar inference tools and model checking apply for all GLMs too; e.g., Wald and Likelihood ratio tests, deviance, residuals, confidence intervals, and overdispersion.

Example

Let the response variable be the number of doctor visits in past two weeks, and independent variables include gender, age, income, illness, name of days of reduced activity, general health score:

Y: number of doctor visits in past two weeks

X: gender, age, income, illness, general health score

import pandas as pd
import statsmodels.api as sm

# Data: 
data = {
    'doctor_visits': [9,8,8,8,8,8,7],  # Response variable
    'gender': [1,0,1,1,0,1,1],        # binary: 1=Male, 0=Female
    'age': [62,19,52,57,62,67,22],    # numeric feature
    'income': [0.25,0.25,0.25,0.01,1.5,0.25,0.00],  # numeric feature
    'illness': [5,2,5,1,1,2,1],       # numerica feature
    'health_score': [10,0,7,4,1,5,1]   # numeric feature
}

df = pd.DataFrame(data)

y = df['doctor_visits']
X = df[['gender', 'age', 'income', 'illness', 'health_score']]
# Add intercept 
X = sm.add_constant(X)  

# Fit the Poisson regression model (GLM with log-link function)
glm_poisson = sm.GLM(y, X, family=sm.families.Poisson())
result = glm_poisson.fit()

print(result.summary())

The output would be:

Summary of Poisson Regression

Next, we would like to include an additional feature, number of days of reduced activity with values

reduced={14,7,14,9,14,14,14},

in the model. This results in overdispersion when implementing Poisson Regression and thus the above code would not have any result summary.

Note: **Overdispersion **refers to a situation in statistical modeling, particularly in count data analysis (like Poisson regression), where the observed variability in the data is greater than what the model expects. In other words, the variance of the response variable is greater than the mean. This phenomenon can lead to inefficient estimates and misleading conclusions if not addressed.

Since our data with the new feature “reduced={14,7,14,9,14,14,14}” results in overdispersion, we use Negative Binomial Regression rather than Poisson Regression:

import pandas as pd
import statsmodels.api as sm

# Data: 
data = {
    'doctor_visits': [9, 8, 8, 8, 8, 8, 7],  # Response variable
    'gender': [1, 0, 1, 1, 0, 1, 1],        # binary: 1=Male, 0=Female
    'age': [62, 19, 52, 57, 62, 67, 22],    # numeric feature
    'income': [0.25, 0.25, 0.25, 0.01, 1.5, 0.25, 0.00],  # numeric feature
    'illness': [5, 2, 5, 1, 1, 2, 1],       # numeric feature
    'days_reduced_activity': [14, 7, 14, 9, 14, 14, 14],  # numeric feature
    'health_score': [10, 0, 7, 4, 1, 5, 1]   # numeric feature
}

df = pd.DataFrame(data)

y = df['doctor_visits']
X = df[['age', 'income', 'illness', 'days_reduced_activity', 'health_score']]
# Add intercept
X = sm.add_constant(X)  

# Fit the Negative Binomial regression model
nb_model = sm.GLM(y, X, family=sm.families.NegativeBinomial())
result = nb_model.fit()

print(result.summary())

The output is

Summary of Negative Binomial Regression

For another detailed example with R implementation , see the example from Geography by Humboldt-Universität zu Berlin in [1]

[1] https://pages.cms.hu-berlin.de/EOL/gcg_quantitative-methods/Lab08_GLM1.html