Before the Forecast: Exploring and Preparing Time Series Data

Forecasting is everywhere, from predicting tomorrow’s electricity demand to estimating the number of online shoppers next weekend. In all these scenarios, one thing is common: data is captured over time. This makes Time Series Analysis a powerful tool to model such trends, understand seasonality, and generate future predictions with accuracy.

In this blog, we will walk through the complete process of time series analysis, from understanding key components and identifying stationarity to applying smoothing techniques, preparing data for forecasting in the next part.

Table of Contents

1. Introduction to Time Series Analysis
2. Use Cases of Time Series Analysis
3. Key components of a Time Series
4. Stationarity
5. Exploratory Data Analysis
6. Smoothing techniques
7. Conclusion

Introduction to Time Series Analysis:

Time Series Analysis is the process of studying data points collected or recorded at specific time intervals, whether yearly, monthly, quarterly or even hourly, to identify patterns, trends, and seasonality, with the ultimate goal of forecasting future values.

Unlike typical datasets, time is a crucial component in time series data. The order of the data matters, and many values are influenced by previous ones.

Whether it’s predicting stock prices, anticipating electricity usage, or forecasting product demand, time series analysis enables:

• Better Decision-Making: By understanding how values evolve over time, businesses and researchers can make informed decisions.
• Forecasting: Estimating future values based on historical data.
• Pattern Recognition: Detecting trends, cycles, and seasonal fluctuations.
• Anomaly Detection: Identifying unusual events or data points (e.g., fraud, system failure).

Use Cases of Time Series Analysis:

In the financial sector, time series models are used to predict the future prices of stocks, currencies, or commodities by analyzing historical price movements. This helps traders and investors make informed decisions.

Power companies rely on time series analysis to forecast monthly or seasonal electricity demand. These forecasts are crucial for resource planning, preventing blackouts, and optimizing power generation.

In healthcare, time series data like heart rate or glucose levels collected over time is analyzed to detect patterns and anomalies. This enables early diagnosis and personalized treatment.

Retailers track weekly or monthly product sales to identify seasonal trends and forecast future demand. This helps with inventory management, pricing strategies, and marketing decisions.

Components of Time Series:

1. Trend:

A trend is the long-term movement or direction in the data over time. It can be increasing, decreasing, or flat.

Example: • Increasing electricity consumption over years due to population growth. • Gradual rise in global temperatures over decades.

Increasing trend in Time Series data

2. Seasonality:

Seasonality refers to regular, periodic fluctuations in the data that repeat over a known, fixed period such as daily, weekly, monthly, or yearly.

Example: • Ice cream sales peaking every summer. • Power usage spiking in winter and summer due to heating/cooling needs.

Seasonality in Time Series data

3. Cyclic patterns:

Cycle consists of long-term, irregular fluctuations that do not follow a fixed calendar pattern. Cycles often occur due to economic or external factors and can last for years.

Example: • Economic recessions or booms. • Business investment cycles.

4. Irregular Component

Also called residual or noise, this is the unpredictable, random variation in a time series that cannot be attributed to trend, seasonality, or cycles. It is what is left after modeling all known structures.

When analyzing time series data, it is crucial to understand how different components; Trend (T), Seasonality (S), and Irregularity (I) combine to form the observed series. This combination can be additive or multiplicative, depending on how the data behaves over time.

Additive Model

Equation: Y(t)=T(t)+S(t)+I(t)

• The total value at time t, Y(t), is the sum of trend, seasonality, and random noise.
• Assumes seasonal fluctuations remain constant regardless of the level of the trend.
• The ups and downs in the data are similar in magnitude across the series.

Multiplicative Model

Equation: Y(t)=T(t)×S(t)×I(t)

• The total value at time t, Y(t), is the product of trend, seasonality, and random noise.
• Seasonal patterns become larger or smaller depending on the current level of the trend.
• The magnitude of seasonal fluctuations changes over time.

Additive and Multiplication relationship

Stationarity:

Stationarity is a key concept in time series analysis that refers to the stability of statistical properties of the time series data over time.

A time series is said to be stationary if its mean, variance, and autocorrelation remain constant over time. Many forecasting models (like ARIMA) assumes stationarity. If the data is non-stationary, the model’s predictions may be unreliable.

Without stationarity:

• Mean and variance drift with time
• Model assumptions break
• Forecasts become inaccurate

What Causes Non-Stationarity in Time Series?

Time series data becomes non-stationary due to the presence of underlying patterns and irregularities that cause its statistical properties to change over time. The most common causes include:

• Trend: A consistent upward or downward movement in the data over time shift the mean and violates stationarity.
• Seasonality: Regular and repeating patterns occurring at fixed intervals leads to periodic fluctuations in the mean.
• Changing Variance (Heteroscedasticity): When the spread of the data increases or decreases over time, the variance is no longer constant, breaking a key assumption of stationarity.

Statistical tests to check for non-stationarity:

1. Augmented Dickey-Fuller (ADF) Test:

The ADF test is one of the most widely used statistical tests to detect non-stationarity in time series data. It focuses on whether the data has a unit root, which is a strong sign of non-stationarity.

Hypotheses:

Null Hypothesis (H₀): The time series has a unit root, implies it is non-stationary.

Alternative Hypothesis (H₁): The time series does not have a unit root, implies it is stationary.

Interpretation:

• If the p-value is less than 0.05, we reject the null hypothesis, implies the series is stationary.
• If the p-value is greater than 0.05, we fail to reject the null hypothesis, implies the series is non-stationary.

from statsmodels.tsa.stattools import adfuller

result = adfuller(series)
print('ADF Statistic:', result[0])
print('p-value:', result[1])

2. Kwiatkowski-Phillips-Schmidt-Shin (KPSS) Test:

The KPSS test does the opposite of ADF. While ADF looks for unit roots (non-stationarity), KPSS tests if the series is stationary around a trend or level.

Hypotheses:

• Null Hypothesis (H₀): The time series is stationary.
• Alternative Hypothesis (H₁): The time series is not stationary.

Interpretation:

• If the p-value is less than 0.05, we reject the null hypothesis, implies the series is non-stationary.
• If the p-value is greater than 0.05, we fail to reject the null hypothesis, implies the series is stationary.

from statsmodels.tsa.stattools import kpss

result = kpss(series, regression='c')  
print('KPSS Statistic:', result[0])
print('p-value:', result[1])

Exploratory Data Analysis (EDA):

EDA in time series is crucial to understanding the temporal structure and characteristics of your data before modeling.

Goals of EDA in Time Series:

• Understand the trend, seasonality, and cycles.
• Detect stationarity and autocorrelation.
• Identify missing values, outliers, and shifts.
• Make decisions about transformation, differencing, or smoothing.
• Prepare for modeling (ARIMA, LSTM, Prophet, etc.).

1. Plot the Time Series

The first step is always visual inspection to identify the patterns in the time series.

import matplotlib.pyplot as plt

plt.figure(figsize=(12, 4))
plt.plot(df['Date'], df['Value'])
plt.title('Time Series Plot')
plt.xlabel('Time')
plt.ylabel('Value')
plt.grid(True)
plt.show()

2. Summary Statistics

Summary statistics help to understand the central tendency, spread, and shape of the time series data.

Start with descriptive statistics to understand the mean, standard deviation, minimum, maximum, quartiles, median of the data.

df['Value'].describe()

Rolling (or Moving Window) Statistics:

Rolling statistics involve calculating metrics like mean or standard deviation over a fixed-size sliding window (e.g., every 12 months) as it moves across the time series.

Rolling statistics allows to track how mean and variance evolve and spot trends and shifts in level.

df['rolling_mean'] = df['Value'].rolling(window=12).mean()
df['rolling_std'] = df['Value'].rolling(window=12).std()
df[['Value', 'rolling_mean', 'rolling_std']].plot(figsize=(12, 5))

If the rolling mean or rolling standard deviation is increasing or decreasing over time, it indicates a trend, and the series may be non-stationary.

3. Decompose the Time Series

Time Series Decomposition is the process of breaking down a time series into its individual components to better understand underlying patterns and behaviors.

from statsmodels.tsa.seasonal import seasonal_decompose

decompose_result = seasonal_decompose(df['Value'], model='additive')
decompose_result.plot()
plt.show()

Time Series decomposition

4. Check for Stationarity

Stationarity can be checked using statistical tests like ADF test and KPSS tests, discussed above.

If a series is found to be non-stationary, we need to transform it before modeling. Common techniques include:

• Detrending: Removing linear or nonlinear trends.
• Differencing: Involves subtracting the current observation from the previous one. It removes trend and seasonality by focusing on changes rather than absolute values.
• Deseasonalizing: removes the seasonal component from the time series to isolate the underlying trend and noise.

5. Autocorrelation & Partial Autocorrelation

ACF Plot shows the correlation between a time series and its lagged values (previous observations).

It tells us repeating patterns and seasonality in the data, whether the series is autocorrelated at different lags and helps decide if ARMA, ARIMA, or Seasonal ARIMA models are suitable.

PACF Plot shows the correlation between the series and its lagged values, after removing the effect of intermediate lags.

It tells the direct relationship between observations and their lags and helps determine the order of the AR (AutoRegressive) part in ARIMA.

from statsmodels.graphics.tsaplots import plot_acf, plot_pacf

plot_acf(df['Value'], lags=30)
plot_pacf(df['Value'], lags=30)
plt.show()

6. Distribution of Values

Understanding the distribution of values helps to get a sense of the underlying shape of the data, whether normal or skewed. Helps to identify presence of outliers and whether transformations (like log, Box-Cox) are needed to stabilize variance.

import seaborn as sns
sns.histplot(df['Value'], kde=True)

7. Check for Missing or Duplicate Timestamps

In time series data, timestamps must be consistent and unique, because each value should correspond to a specific point in time. If timestamps are missing or duplicated, it can lead to incorrect modeling, inaccurate forecasting, and misleading analysis.

#Checking for missing timestamps

df['Date'].isnull().sum()

#Checking for duplicated timestamps

df.duplicated('Date').sum()

How to Fill Missing Values:

1. Forward Fill (ffill): Assumes latest known value continues.

df['Value'] = df['Value'].ffill()

2. Interpolate: Uses linear interpolation between known points.

df['Value'] = df['Value'].interpolate(method='linear')

Handling Duplicate Timestamps:

1. Aggregate duplicates by taking the mean or sum.

df = df.groupby('Date').mean().reset_index()

2. Drop duplicates, keeping the first.

df.drop_duplicates(subset='Date', keep='first', inplace=True)

8. Resample the Data

Resampling is the process of changing the frequency of the time series data; either downsampling (e.g., daily to monthly) or upsampling (e.g., monthly to daily). This is especially useful when working with noisy data, irregular time intervals, or when aggregating data for long-term trends.

#Dowsampling from daily to monthly

monthly_df = df.resample('M').mean()

9. Detect Outliers and Anomalies

In time series analysis, outliers and anomalies are data points that significantly deviate from the general pattern of the data.

Rolling Statistics (IQR Method):

Detects outliers using moving window and IQR (Interquartile Range).

rolling_median = df['Value'].rolling(window=12).median()
rolling_std = df['Value'].rolling(window=12).std()

threshold = 2.5
outliers = df[np.abs(df['Value'] - rolling_median) > threshold * rolling_std]

You can also use boxplot.

import seaborn as sns
sns.boxplot(data=df['Value'])

Smoothing Techniques:

Once we have explored our time series data through Exploratory Data Analysis (EDA), the next crucial step is to reduce noise and enhance patterns in the data using smoothing techniques. These methods help reveal the underlying trend and seasonality more clearly, making it easier to model and forecast the series effectively.

1. Moving Average (Rolling Mean)

The moving average is one of the simplest and most widely used smoothing methods. It calculates the average of the values over a fixed-size sliding window. It smooths out short-term fluctuations and highlights longer-term trends.

import pandas as pd
import matplotlib.pyplot as plt

df['rolling_mean_3'] = df['Value'].rolling(window=3).mean()
df['rolling_mean_12'] = df['Value'].rolling(window=12).mean()

plt.figure(figsize=(12,6))
plt.plot(df['Value'], label='Original', color='blue')
plt.plot(df['rolling_mean_3'], label='3-Month Rolling Mean', color='orange')
plt.plot(df['rolling_mean_12'], label='12-Month Rolling Mean', color='green')
plt.legend()
plt.show()

2. Exponential Moving Average

Unlike the simple moving average, the Exponential Moving Average (EMA) assigns more weight to recent observations. This makes EMA more responsive to recent changes in the time series.

df['ema_3'] = df['Value'].ewm(span=3, adjust=False).mean()
df['ema_12'] = df['Value'].ewm(span=12, adjust=False).mean()

plt.figure(figsize=(12,6))
plt.plot(df['Value'], label='Original', color='blue')
plt.plot(df['ema_3'], label='3-Month EMA', color='red')
plt.plot(df['ema_12'], label='12-Month EMA', color='purple')
plt.grid(True)
plt.show()

3. Simple Exponential Smoothing

Simple Exponential Smoothing (SES) is a foundational forecasting technique that assigns exponentially decreasing weights to past observations. Unlike moving averages, SES can be used directly for forecasting future values based on a single smoothing parameter.

from statsmodels.tsa.holtwinters import SimpleExpSmoothing

model = SimpleExpSmoothing(df['Value']).fit(smoothing_level=0.2, optimized=False)
df['ses'] = model.fittedvalues

plt.figure(figsize=(12,6))
plt.plot(df['Value'], label='Original', color='blue')
plt.plot(df['ses'], label='Simple Exponential Smoothing', color='green')
plt.legend()
plt.show()

#smoothing_level=0.2 determines how much weight is given to the most recent observation.

Conclusion:

We have come a long way in this post, starting with the basics of time series, diving into its key components, visualizing insights with EDA, and finally smoothing the series to reveal clearer trends. These preprocessing steps are essential before we apply forecasting models.

In the upcoming blog, we will explore popular modeling techniques and evaluation metrics that help us predict future values with confidence.

If you found this blog helpful or have any suggestions, feel free to reach out! I’d love to connect.

LinkedIn: Charu Agarwal

Email: agarwalcharu987@gmail.com

Github: github.com/Charu-Agarwal-01