Linear Regression Model—Comprehensive Overview

Linear regression models the relationship between a continuous scalar dependent variable (response) and one or more independent variables (predictors) by fitting a linear model to the observed data. The core assumption is that the relationship is linear in the parameters.

Definition: Linear regression models the relationship between a scalar dependent variable y ∈ ℝ and an input feature vector x ∈ ℝⁿ, assuming a linear form plus error:

• y = β₀ + β₁ x₁ + β₂ x₂ + … + βₙ xₙ + ε* where,

• β ∈ ℝⁿ are regression coefficients,
• ε is random noise (often assumed ε ∼ 𝒩(0,σ²)).

There are two basic types of linear regression models.

1. Simple Linear Regression (One predictor):

yᵢ = β₀ + β₁ xᵢ + εᵢ

2. Multiple Linear Regression (p predictors):

yᵢ = β₀ + β₁ xᵢ₁ + β₂ xᵢ₂ + … + βₚ xᵢₚ + εᵢ

Now we will explore each type of linear regression in detail.

Simple Linear Regression (One predictor):

Simple linear regression is a statistical method we use to model the relationship between one independent variable (predictor, x) and one dependent variable (outcome, y). The goal is to find a straight line that best fits the data points on a scatter plot.

This best-fit line allows us to:

1. Understand the relationship: How does a change in x affect y?
2. Predict: For a new value of x, what is the most likely value of y?

The model is represented by a simple straight-line equation you might remember:

y = β₀ + β₁ xᵢ + εᵢ

Where,

• y: The Dependent Variable (the thing we want to predict or explain).
• x: The independent variable (the thing we use to make the prediction).
• β₀: The y-intercept that predicts the value of y when x is zero.
• β₁: The slope or coefficient that adjusts the value of the independent variable x and determines the strength and direction of the relationship.
• ε: The error term or residual (ŷ) that tells the difference between the actual observed value of y and the predicted value ŷ. It represents the variation in y that the model cannot explain.

We don’t just draw a line that looks good. We use a precise mathematical method called Ordinary Least Squares (OLS). The goal of OLS is to find the values of β₀ and β₁ that make the line minimize the sum of the squared errors. The error for each prediction is calculated by,

Squaring the errors ensures that all errors are positive (so a negative error doesn’t cancel out a positive one). Moreover, squaring penalizes larger errors more heavily, which helps create a better overall fit. The residual sum of squares over the all data points can be defined as,

The derivative of RSS(β₀,β₁) w.r.t. β₀ and β₁ are,

and setting these derivatives to zero

yields the following equations:

From the above equation 1, we can derive the estimate for the intercept:

From the equation 2, we can derive the estimate for the slope:

Note that the numerator can be rewritten as,

and that the denominator can be rewritten as,

With equation 5 and 6, the estimate from equation 4 can be simplified as follows:

Together, equation 3 and 7 constitute the ordinary least squares parameter estimates for simple linear regression. Let’s create a concrete example with a scatter plot and a best-fit line.

House Size vs. Rental Price

Scenario: A estate agent wants to understand the relationship between the size of in house and the rental prize of the house in the local vicinity. Data Collected:

[embed]

First, we plot this data to visualize the relationship. Each point represents one house.

Scatter plot: Housing rental price VS. Size.

Our goal is to find the straight line that minimizes the sum of squared errors. The equation is 𝑦̂ᵢ = 𝛽₀ + 𝛽₁𝑥ᵢ . Let’s calculate the slope (𝛽₁) and intercept (𝛽₀) manually using the equations 7 and 4 respectively. For simplicity, we’ll use the data points directly.

Suppose x = size and y = prize, Mean of x:

and Mean of y:

Now calculate the slop 𝛽₁ using the values as provided in the following Table,

[embed]

Now calculate the slop 𝛽₀ using the values of 𝑥̄,𝑦̄, and 𝛽₁ as calculated above,

Put the values of 𝛽₀ and 𝛽₁ in regression equation, suppose the value of εᵢ = 0,

Now we put 𝑥ᵢ value in above equation and draw a line graph on a previous scatter plot.

The above graph shows all predicted points (red color) on a regression line. The difference between predicted point (red color) and actual point (blue color) on graph is called residual ε.

Now, its time to check the model performance by computing sum of residual squared error.

[embed]

This straight line is called best fit line that gives minimum sum of square distance between the actual points and predicted points on a graph.

The python code for above regression problem is given below.

[embed]

In the next post we will explore the concept of Multivariate Regression model that contains multiple independent variables (predictors) and one dependent variable (response).