Understanding Exponentially Weighted Averages

ML Quickies #43

Optimization algorithms in deep learning like Adam or RMSprop, or when trying to smooth out noisy data, we encounter the concept of exponentially weighted averages (EWAs). Despite their prevalence in machine learning, the intuition behind them isn’t always clear. Let’s try to really understand this elegant idea and see why it’s become such a cornerstone of modern ML.

At its core, an exponentially weighted average is a way to compute a running average that gives more weight to recent observations while still incorporating historical data. Unlike a simple moving average that treats all values in a window equally, an EWA gradually “forgets” older values in an exponential fashion.

The EWA formula is deceptively simple:

Where:

• V_t is the weighted average at time t
• θ_t is the current observation (at time t)
• β is the decay parameter (typically between 0.9 and 0.999)
• V_{t-1} is the previous weighted average (at time t-1)

Well, why is it “exponential”?

The term “exponential” comes from how past observations decay in influence. If we expand the recursive formula, we see that the current average is actually a weighted sum of all past observations, where each weight decreases exponentially:

Notice how each coefficient is multiplied by an additional factor of β, creating exponential decay. An observation from 10 time steps ago contributes β¹⁰ times less than the current observation.

The β parameter controls how quickly we forget the past. β = 0.9 means we’re roughly averaging over the last 10 observations, while β = 0.99 averages over approximately the last 100. The rule of thumb is that we’re averaging over roughly 1/(1-β) time steps. Thus, higher the value of β, the more weightage to historical data.

Why EWAs?

Machine learning practitioners use exponentially weighted averages for several compelling reasons:

• Memory efficiency: Unlike a moving average that requires storing a window of past values, an EWA only needs to remember a single number (the previous average). This makes it perfect for streaming data or large-scale applications.
• Adaptive smoothing: The exponential decay naturally adapts to trends. Recent changes have more influence, so the average can track shifting patterns while still filtering out noise.
• Computational simplicity: Computing an EWA requires just one multiplication, one subtraction, and one addition per update. This is incredibly cheap compared to recalculating averages over windows.

Choosing the Right Beta

Selecting β depends on your application. Smaller values (0.8–0.9) respond quickly to changes but retain more noise. Larger values (0.95–0.99) provide smoother estimates but lag behind rapid changes. In practice, β = 0.9 is a common default that balances responsiveness and smoothing.

Applications in Machine Learning

You’ll find exponentially weighted averages throughout modern ML:

• In optimization algorithms, momentum-based methods like SGD with momentum use EWAs to accumulate gradients, helping the optimizer build velocity in consistent directions while dampening oscillations. Adam and RMSprop take this further by maintaining EWAs of both gradients and squared gradients to adaptively adjust learning rates.
• For time series analysis and forecasting, EWAs provide a simple but effective way to smooth noisy signals and identify underlying trends without the lag of traditional moving averages.
• In reinforcement learning, exponentially weighted averages help smooth reward signals and track moving statistics of states and actions, making learning more stable.
• When monitoring training metrics, EWAs smooth out noisy loss curves and accuracy plots, making it easier to spot genuine trends versus random fluctuations.

A Simple Python Implementation

Let’s implement a basic exponentially weighted average from scratch (using classes), and then demonstrate it on some synthetic noisy data:

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

class ExpWAvg:
    def __init__(self, beta=0.9, bias_correction=True):
        self.beta = beta
        self.bias_correction = bias_correction
        self.v = 0  # running average
        self.t = 0  # time step counter

    def update(self, observation): # update the average with a new observation
        self.t += 1
        self.v = self.beta * self.v + (1 - self.beta) * observation

        # bias correction
        if self.bias_correction:
            return self.v / (1 - self.beta ** self.t)
        return self.v

    def reset(self):
        self.v = 0
        self.t = 0

# Generate noisy synthetic data
rng = np.random.default_rng(69)
true_signal = np.sin(np.linspace(0, 4*np.pi, 200))
noise = rng.normal(0, 0.4, 200)
noisy_data = true_signal + noise

# apply EWA with different beta values
betas = [0.6, 0.8, 0.9, 0.98]
results = {}

for beta in betas:
    ewa = ExpWAvg(beta=beta)
    smoothed = [ewa.update(x) for x in noisy_data]
    results[beta] = np.array(smoothed)

# results vizualization
plt.figure(figsize=(10, 4))
plt.plot(noisy_data, alpha=0.7, label='Original Data', color='black')

colors = ['blue', 'green', 'red', 'purple']
for beta, color in zip(betas, colors):
    sns.lineplot(x=range(len(results[beta])), y=results[beta], label=f'EWA (β={beta})', color=color, linewidth=1.5)

plt.xlabel('Time Step')
plt.ylabel('Value')
plt.title('Exponentially Weighted Average with Different Beta Values', pad=10)
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

A Critical Detail : Bias Correction

One subtlety worth noting is bias correction. When we initialize V_0 = 0, the early estimates are biased toward zero. This is why optimization algorithms like Adam apply bias correction by dividing by (1 — β^t). This bias correction is especially important in the initial iterations since as t grows large, β^t approaches zero and the correction becomes negligible.

Exponentially weighted averages represent one of those elegant ideas that punch well above their weight in machine learning. With minimal computational cost and memory footprint, they are deceptively simply and extremely beautiful at the same time.

Until next time:)