Linear Regression — Only Math(4 Methods, NO Code)

Multiple approaches (calculus, statistics, geometry, and linear algebra) and all lead to the same regression line.

Why this Article on Linear Regression?

Most machine learning learners(specially beginners) know linear regression as:

LinearRegression().fit(X, y) and it’s implemented using y = wx+b

But if u think how the weights are calculated if there are no initaial weights given, it becomes complex?

What makes linear regression beautiful is that it can be derived from multiple mathematical perspectives, each revealing a different intuition:

• Optimization (Calculus)
• Statistics (Mean–Variance–Covariance)
• Linear Algebra (Matrix Projection)
• Numerical Methods (Curve Fitting)- I think some of u may know this.

In this article, I will solve the same regression problem using four different mathematical methods — *without writing a single line of code*. I will not discuss how a line is fitted geometrically, I think everyone understands that.**

Let’s consider a simple dataset: {(1,2),(2,3),(3,5),(4,4),(5,6)}

Our goal is to find the best fitting line: y=wx+b

We minimize squared error :

All four methods solve this same objective function — only the reasoning differs.

Method 1 — Calculus Approach (Optimization View)

Idea- Choose w and b that minimize total squared error.

method1- solution steps

we get model y=0.9x+1.3

Method 2 — Mean–Variance–Covariance Approach (Statistical View)

we get model y=0.9x+1.3

For sample case, replace n with n−1.

Method 3 — Matrix Approach (Linear Algebra View)

consider 𝛽 is weight vector and our input points are x1, x2, x3, x4,…… then X is our input data with 1’s augmented.

we get model y=0.9x+1.3

Linear regression is Projection of Y onto the column space of X.

Method 4 — Curve Fitting / Geometric View

we get model y=0.9x+1.3

All Methods Give the Same Answer. They *try to *find the closest possible predictions to observed data under squared error.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Liked this ! Follow me, I plan to explain other topics like

• Multicollinearity effect and it’s solutions
• Gradient descent in linear regression
• Beauty of Diagonalisation and SVD
• Eigen value decomposition in linear regression, ridge regression
• Regularisation and it’s effect
• Why linear classification methods fails and logistic regression passes?
• Logistic loss
• Naive Bayes MAP rule
• Zero Probality Problem in Naive Bayes
• Choice of logarithm Base in Entropy formula
• Decision tree creates rectangular regions, how?
• How Bias and varience become proxy to training and testing error?
• How Overfitting tends to unstablity?
• Overfitting and Rote Learning
• what is Kernel Trick?
• Hard margin vs Soft margin svm : complete math
• How RBF kernel tends to Overfit?
• Does Kernel Trick always work?
• The way of writing Chain Rule
• LDA as Gaussian Naive Bayes with MAP and Equal Covarience assumption
• Decision Boundary in LDA
• Error of prediction in LDA
• 4 Approaches of describing PCA
• Agglomerative clustering is Greedy
• Outliers, Noise effects
• Curses in ML like curse of dimensionality, data not following assumption of models
• Data Hungry ML Models
• Entropy is symmetrical, how?