Common Marketing Mix Modeling Misunderstanding, Episode 1: Measuring Media Performance More Granularly Than the Channel Level

A frequent question asked in Marketing Mix Modeling (MMM) is about the granularity of model learnings. Many people assume that MMM can provide insights at any level of granularity without issues — whether at the media channel level, partner level, or even down to individual creatives. However, measuring media performance beyond the channel level is problematic for several reasons, the most critical being multi-collinearity. Let’s explore why MMM is best suited for channel-level analysis and the challenges of moving beyond this level.

People often look for a one-size-fits-all solution for measuring media effectiveness. However, MMM, Multi-Touch Attribution (MTA), and experimentation each serve different purposes, and it is essential to know their strengths and limitations to use them effectively. While MMM excels at providing strategic-level insights at the channel level, MTA offers granular insights into individual user paths and the contributions of various touchpoints. On the other hand, experimentation helps to validate causality and measure lift accurately. I will discuss MTA and Incrementality Testing in the upcoming articles.

What is Multi-Collinearity?

Multi-collinearity occurs when two or more predictors in a regression model are highly correlated, which means they provide overlapping information, making it difficult for the model to distinguish the individual effect of each variable. This problem can arise among media variables or between media variables and relevant external factors.

For instance, imagine you have a product with high seasonal demand. Logically, you increase media spend during peak periods. When multiple channels (e.g., TV, OLV, and Audio) all have synchronized spending patterns, it becomes nearly impossible for the model to determine the individual contribution of each channel, as all variables follow the same trend.

Think of it as a tug-of-war scenario: if three people pull in the same direction and the team wins, how do you determine the contribution of each person without prior knowledge of their strength? This is similar to what happens in MMM without clear separation between channels. One potential approach is to incorporate experimentation results, such as geo-based incrementality tests, as Bayesian priors to improve the robustness of the model.

Example in Python to Illustrate Multi-Collinearity

We will demonstrate how synchronized media spending leads to biased coefficients in a Marketing Mix Model (MMM). Let’s break down this example into three sections:

1. Simulating the Data

In this step, we simulate weekly data for three channels: TV, OLV, and Audio. We know the true coefficients for these channels, but the simulated data will be highly correlated to mimic real-world conditions.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
import pandas as pd

# Simulating weekly TV, OLV, and Audio spends in hundreds of thousands (2024 data)
np.random.seed(150)
weeks = np.arange(1, 53)
tv_spend = np.random.normal(300_000, 50_000, 52)  # Mean of 300k, std dev of 50k
olv_spend = tv_spend * 0.85 + np.random.normal(0, 20_000, 52)  # Strong correlation with TV spend
audio_spend = tv_spend * 0.65 + np.random.normal(0, 15_000, 52)  # Correlated with TV spend

# Simulating sales with known coefficients
sales = 0.08 * tv_spend + 0.10 * olv_spend + 0.05 * audio_spend + np.random.normal(0, 30_000, 52)

# Prepare the dataset
X = np.column_stack((tv_spend, olv_spend, audio_spend))

# Plotting the synchronized spending patterns
plt.figure(figsize=(12, 6))
plt.plot(weeks, tv_spend, label='TV Spend', color='blue')
plt.plot(weeks, olv_spend, label='OLV Spend', color='orange')
plt.plot(weeks, audio_spend, label='Audio Spend', color='green')
plt.title("Synchronized Media Spending Patterns (Weekly Data - 2024)")
plt.xlabel("Weeks in 2024")
plt.ylabel("Spend Amount (in hundreds of thousands)")
plt.legend()
plt.show()

Note: Here, the actual coefficients are [0.08, 0.10, 0.05].

2. Checking Correlations

Next, we calculate the correlation matrix to observe the high correlation between these media channels. This demonstrates the real-world problem of multi-collinearity.

# Checking correlations
correlation_matrix = pd.DataFrame(X, columns=["TV Spend", "OLV Spend", "Audio Spend"]).corr()
print("Correlation Matrix:\n", correlation_matrix)

3. Running the Regression and Comparing Coefficients

Finally, we run a linear regression to estimate the coefficients and compare them with the actual values. This step reveals whether the model can accurately retrieve the true coefficients.

# Running a regression
model = LinearRegression()
model.fit(X, sales)

# Print the actual and estimated coefficients
print("Actual Coefficients: [0.08, 0.10, 0.05]")
print(f"Estimated Coefficients: {model.coef_}")

Bar Chart: Actual vs. Estimated Coefficients

To better illustrate the differences between actual and estimated coefficients, we can add a bar chart:

# Bar chart for actual vs. estimated coefficients
labels = ['TV Spend', 'OLV Spend', 'Audio Spend']
actual = [0.08, 0.10, 0.05]
estimated = model.coef_

x = np.arange(len(labels))
width = 0.35

fig, ax = plt.subplots()
rects1 = ax.bar(x - width/2, actual, width, label='Actual')
rects2 = ax.bar(x + width/2, estimated, width, label='Estimated')

ax.set_ylabel('Coefficient Values')
ax.set_title('Actual vs. Estimated Coefficients')
ax.set_xticks(x)
ax.set_xticklabels(labels)
ax.legend()
plt.show()

Key findings:

The correlation matrix indicates high correlations between the channels, which is common in real-world media spending data. However, when running the regression, we find that the estimated coefficients differ significantly from the actual values, with one coefficient even flipping signs. This highlights the challenge of interpreting results at a granular level without addressing multi-collinearity.

Why MMM Should Not Go Beyond the Channel Level

Media planning teams often leverage their understanding of demand patterns to allocate budgets, strategically increasing spend across multiple channels during peak periods. While this approach aligns with business objectives, it inadvertently introduces multi-collinearity. A common practice among marketers is to proportionally upweight or downweight spend across channels on a weekly or monthly basis to drive specific outcomes. However, this synchronized adjustment leads to high correlations between channels, making it challenging for MMM to isolate and accurately measure the individual impact of each channel.

If some media channels are already highly correlated, the spending patterns of partners within each channel tend to be even more correlated. For example, consider OLV (Online Video) as a channel with key partners such as YouTube, Hulu, and Vimeo. The spend across these platforms is likely to exhibit strong correlations due to shared strategies or synchronized campaign timings. By keeping MMM at the channel level, the model manages a relatively limited number of variables, which allows for the application of techniques like media transformation, regularization, or hierarchical modeling to mitigate multi-collinearity effectively.

However, if we expand MMM beyond the channel level to include 20 or more highly correlated variables, the complexity of the model increases exponentially. In such cases, no statistical method can fully resolve the multi-collinearity issues. For instance, if you have only 5 or 6 spend data points for each partner, and those partners’ spending patterns are highly correlated, it becomes nearly impossible to isolate and measure their individual impact accurately. So, if someone claims they can reliably do this — well, you might want to be skeptical of their model!

Note on Bayesian Priors

Some might argue that if Bayesian priors are set for media variables at the partner level, they could still obtain reasonable and positive coefficients. However, this approach is only effective if the Bayesian priors are backed by solid experimental evidence. Without experimentation to substantiate these priors, the approach is akin to making educated guesses. While it may look good on paper, it lacks the rigor and accuracy needed for reliable insights.

The Role of MTA and Experimentation

When deeper insights are required at a more granular level, Multi-Touch Attribution (MTA) can complement MMM. In this context, we refer to probabilistic models like those based on Markov chains, not the older rule-based MTA. A probabilistic MTA model analyzes user paths and the “removal effect” of media channels. For instance, the Markov chain-based model calculates the likelihood of each path and the marginal effect of removing a channel. Unlike MMM, which is built on regression models, MTA allows for a more granular assessment and is less affected by multi-collinearity.

Combining MMM with MTA provides a powerful way to understand media effectiveness across different levels of granularity. For the best results, adding experimentation, such as incrementality testing, to this framework creates a comprehensive measurement system. However, no tool is one-size-fits-all — MTA and incrementality testing have their limitations too! Together, MMM, MTA, and incrementality testing provide a holistic view of media effectiveness, ensuring both strategic insights and tactical guidance.