How a Leading Retail Chain Unified Its Campaign Data Platform on Google Cloud

The Problem: Fragmented Campaign Data Holding Back the Business

A leading retail chain operating thousands of marketing campaigns across digital, in-store, and omnichannel touchpoints hit a wall that many large retailers know too well — their data couldn’t keep up with their ambitions.

Marketing and analytics teams were drowning in a sea of disconnected data sources. Campaign performance data was siloed across dozens of systems, arriving in different formats — CSV, JSON, XML, AVRO — with no standardized ingestion layer and no single source of truth.

Answering a simple question like “Which campaigns are driving in-store conversions this week?” took days, not hours. The business was flying blind during the moments that mattered most. Here’s what the team was dealing with:

• No unified campaign data hub — each channel had its own data store with no cross-channel visibility
• Manual, error-prone ETL processes with no automated scheduling or failure alerting
• Multi-format raw data arriving with no standardized ingestion or validation layer
• No real-time or near-real-time dashboards for the marketing team
• No CI/CD discipline in the data team — deployments were manual and risky
• No data quality enforcement before data reached downstream consumers

The result: budget misallocation, missed campaign signals, and a data team that spent more time firefighting than delivering insights.

The Solution: A Modern ELT Data Hub on Google Cloud Platform

The answer was to build a Campaign Central Data Hub — a cloud-native, fully automated ELT architecture on GCP that became the single source of truth for all campaign performance data. The approach followed a medallion-style layered architecture:

Each layer added structure, quality, and business semantics — replacing fragmented pipelines with an automated, monitored, and governed data ecosystem.

How We Solved It — Step by Step

Step 01

Centralized Raw Data Landing in GCS

All source campaign data — digital ad platforms, in-store POS systems, CRM exports, and third-party marketing tools — landed into dedicated Google Cloud Storage (GCS) buckets organized by source system and date partition. This created a raw, immutable landing zone. Nothing gets deleted. Everything is traceable.

Step 02

ELT Transformation with Dataform

Dataform became the transformation engine. We built modular SQLX workflows for each layer:

• Staging tables — cleansed and standardized raw data, enforced schema consistency
• Core tables — applied business rules, deduplication, and joins across source systems
• Mart tables — aggregated, analytics-ready datasets served to reporting and BI consumers

Step 03

Orchestration and Monitoring with Cloud Composer (Airflow)

Cloud Composer orchestrated and scheduled every pipeline. DAGs were designed to be idempotent — safe to re-run without duplicating data — with automated retries, SLA-based alerting, and data quality gate tasks that blocked downstream loads if upstream checks failed.

Step 04

CI/CD with GitHub Actions

One of the most impactful improvements wasn’t even a data tool — it was engineering discipline. GitHub Actions pipelines automated Dataform SQLX compilation, code review gates before merges, and automated deployment across Dev and Prod environments. The data team went from manual, anxiety-inducing deployments to push-button releases with full audit trails.

Step 05

Business-Facing Dashboards via Looker Studio

With mart-layer data flowing reliably into BigQuery, Looker Studio dashboards were connected directly — giving marketing and business teams self-serve access to campaign performance metrics, updated on an automated, reliable schedule.

Services Used

Results and Business Impact