Loss Function for Logistic Regression

The reason we were not reaching the optimal solution using perceptron and even sigmoid because points are being selected randomly and each time we will get a slightly different solution and each time we cant say which is the better solution.

So to solve this problem we use the generic machine learning way that is to find a loss function/ error function. This helps us quantify the mistakes of the model and then we minimize this loss function and reach minima, at this point the error of the model will be the lower and then we can say we reached the optimal solution.

Loss function will help us decide between model 1 and model 2.

Maximum Likelihood

Now suppose we have these two models

By looking at it we can intuitively say that the model 2 correctly classifies all the points and is better than model 1.

But now talking in context of maximum likelihood

Probabilities of the points are given (basically yi_hat).

Maximum likelihood tells us to multiply all the correct probabilities for all the points. More the maximum likelihood the better the model.

For eg. in model 1

and model 2

So we can now see that model 2 has more maximum likelihood than 1 so it is better model than model 2.

Now a question arises why to multiply the probabilities and not just add it to save us from the hassel, but we are using a concept called Bernoulli probabilities.

Basically we take the union of all the probabilities which makes it such that all the events are happening together.

Also adding the probabilities comes with a problem explained in the pic below

But for now we considered only 4 points but in real world datasets these numbers range to millions, so our value for maximum likelihood will be very small and comparision at that scale is not valid, so we need to take the maths one step further to simplify our work.

We convert this product to summation using Log.

We know Log(AB) = Log A + Log B

But we do negative log as the values of our probabilities range from 0–1 and log of numbers less than 1 is -ve so to get our value positive we make these logs -ve.

So the thing we reached at is called Cross Entropy — The summation of negative logs of maximum likelihood.

Cross entropy needs to be minimized for our model to perform better.

Now converting all of this into a formula which we can implement in code

This is the final Log Loss (Binary Cross-Entropy) error function used in logistic regression.

It evaluates the probability of the actual class by using both p and 1−p, heavily penalizing confident wrong predictions.

But this is not a closed form solution this is just the loss function which we will have to minimize using gradient descent in order to find the optimal coefficients for our decision boundary.