What is Gradient Descent?
A walk thru of the core concepts and resources to help solidify understanding
What is Gradient Descent?
A walk thru of the core concepts and resources to help solidify understanding
Photo by Clémence Bergougnoux on Unsplash
TL;DR: Gradient descent is a way to optimize model parameters to minimize error between predicted and actual results.
Why is Gradient Descent Import to Machine Learning?
Machine learning models are similar to writing a draft blog post. Your initial draft may be good, however, after iterating over the text to emphasize certain sections and correcting grammatical mistakes, your blog is ready to go in a much better state than when you stated. The true art in machine learning is selecting the correct features and adapting the parameters to improve a model for a specific task. Selecting the optimal value for parameters is accomplished by an optimization algorithm like gradient descent. Optimal parameters in a machine learning model will increase the accuracy of prediction on new data.
🚨 Jargon Alert! 🚨
Important Terms in Gradient Descent (Jargon Alert)
Loss Function: Defines a good predictive model by quantifying how well an algorithm models given the data. An example of a Loss Function is mean square error (MSE).
Learning Rate: A value less than 1 to used to adjust Step Size. Typically set as 0.1, 0.01, or 0.001. A larger learning rate converges faster whereas a smaller learning rate takes more steps.
Step Size: The size of the step related to the slope of the Loss Function. Step Size is calculated using the slope * Learning Rate. If the calculated Step Size is too small or the number of steps is above a threshold, the gradient descent algorithm will stop.
Global Minimum: The lowest point in an error graph. The goal of gradient descent is to find the parameters that localize to the Global Minimum in a derivative curve of the Loss Function.
Objective of Gradient Descent
The goal of gradient descent is to find the Global Minimum of a Loss Function derivative (best fitting model parameters).
The color of the line in the figure on the left corresponds to the color of the square on the right. The position of the squares in the right figure correspond to the error of the Loss Function.
The gradient descent algorithm takes larges steps at first then smaller steps to converge at the Global Minimum.
Gradient Descent Algorithm: A 6-step Algorithm
Step 1: Take the derivative of the Loss Function for each parameter
Step 2: Pick random values for parameters
Step 3: Plug the parameter values into the derivatives to get slope
Step 4: Calculate the Step Size:
Step Size = slope * Learning Rate (i.e. 0.01)
Step 5: Calculate new parameters:
New parameter = old parameter — Step Size
Step 6: Repeat steps 3–5 until Step Size is small (<0.0001) or maximum number of steps taken (1K)
Problems with Gradient Descent
Two possible problems in gradient descent. A non-convex Loss Function graph as shown on the left can cause the algorithm to find local minima rather than the Global Minimum. The graph on the right shows a large number of data points in the initial regression. The time to calculate the Loss Function of large datasets can become intractable. Images by author.
Two major problems with gradient descent are shown in the figures above. First, Loss Function derivatives may not always be convex in shape and non-convex shapes may result in the algorithm discovering a local minima rather than the Global Minimum. This local minimum may result in errors due to overfitting when the model is exposed to new data. Overfitting is explained well by Farhad Malik in The Problem Of Overfitting And How To Resolve It.
Second, the time to calculate the Loss Function scales with the amount of data to fit a model. A large dataset may make the calculation and recalculation of loss cumbersome and intractable.
Overcoming the Problems of Gradient Descent
- Non-convex shapes
Random Start Sites: Run gradient descent with different start sites and see if the final solution is the same amongst all runs.
- Large amount of data points (high computational cost)
Stochastic Gradient Descent: Take a random subset of data and run gradient descent.
Gradient Descent Explained
Gradient descent is used to adjust model parameters to find the Global Minimum of the Loss Function. The Global Minimum is found by adjusting the initial parameters by a Step Size calculated from slope * Learning Rate. Gradient descent improves model accuracy.
Additional Resources
Some excellent additional resources include StatQuest with Josh Starmer for a clear explanation of core concepts including the mathematics of the algorithm. Another great resource is a browser game called *Gradient Descent*. The game involves trying to find a hidden global minimum against a CPU that engages in gradient descent. My name is Cody Glickman and I can be found on LinkedIn. Be sure to check out some of my other articles for projects spanning a wide range of data science and machine learning topics.
메타데이터
- post_id
- 2e10095f0801
- slug
- what-is-gradient-descent-2e10095f0801
- url
- https://medium.com/byte-sized-machine-learning/what-is-gradient-descent-2e10095f0801
- canonical_url
- https://medium.com/byte-sized-machine-learning/what-is-gradient-descent-2e10095f0801
- author_url
- https://medium.com/@glickmancody
- status
- ok
- fetched_at
- 2026-06-15 20:49:13
