Adagrad Optimizer — Basics of algo

1. Introduction to Adagrad

Adagrad is a gradient-based optimization algorithm that adapts the learning rate for each parameter based on historical gradient information. It is particularly useful for datasets with sparse features or features of varying scales.

Key Idea:

• Parameter-specific learning rates: Instead of a single learning rate for all parameters, Adagrad adjusts the rate based on the frequency and magnitude of gradients for each parameter.
• Learning rate decay: The learning rate decreases for parameters with large cumulative gradients (common features) and remains higher for parameters with rare updates (sparse features).

Solution:

Adagrad scales the learning rate inversely to the square root of the sum of squared gradients, reducing oscillations in steep directions.

3. Mathematical Derivation

Update Rule:

Derivation

Sample Example

Predict house prices using two features: size (0–3000 sq.ft) and bedrooms (1–5). Issue: size dominates gradients due to larger scale.

Adagrad Implementation

import numpy as np

# Sample data (X = [size, bedrooms], y = price)
X = np.array([[2000, 3], [1500, 2], [3000, 4]])
y = np.array([500000, 350000, 700000])

# Initialize parameters and hyperparameters
theta = np.zeros(2)  # [w_size, w_bedrooms]
eta = 0.1
epsilon = 1e-8
G = np.zeros(2)  # Sum of squared gradients

# Adagrad update
for epoch in range(100):
    for i in range(len(X)):
        # Compute gradient for current sample
        y_pred = np.dot(X[i], theta)
        error = y_pred - y[i]
        grad = 2 * error * X[i]

        # Update G and theta
        G += grad**2
        theta -= (eta / np.sqrt(G + epsilon)) * grad

print("Optimal weights:", theta)

Output

Optimal weights: [ 199.999  99999.999]  # Weight for 'size' adapts slower than 'bedrooms'

5. Advantages

1. Automatic Learning Rate Tuning: No manual tuning needed for sparse data
2. Suitability for Sparse Features

6. Disadvantages

1. Aggressive Learning Rate Decay:

7. Comparison with Other Optimizers