[Free Github Repo] An End-to-End Data Stack on Your Computer — The Practice of Data

Clone the github repo here: https://github.com/practiceofdata/northstar-data-stack

Intro

Most tutorials teach one tool. Real pipelines are never one tool.

You learn dbt. You write a model. It works. But you have no idea where the data came from, or what runs it on a schedule, or what happens after.

You learn Airflow. Every DAG prints “hello world.” You learn the syntax. You never move real data.

That’s the gap. Tutorials isolate one tool and treat everything around it as a black box. Real teams work a chain: data lands somewhere, gets transformed, gets tested, gets scheduled, gets reported. Learning one link doesn’t teach you the chain.

So here’s a repo that wires the whole chain together. Free. Open source. Runs on your machine. No cloud account.

It’s called NorthStar Commerce — a fictional e-commerce company’s full analytics stack. Clone it, run one command, and watch raw data turn into a tested, query-ready warehouse.

The stack, four tools, four jobs:

Faker → DuckDB → dbt → Dagster

Faker generates the data. DuckDB stores it. dbt transforms it. Dagster runs it on a schedule. One command — python run_pipeline.py - does all of it in about five minutes.

This isn’t meant to be read once. It’s a sandbox. Run it. Break something on purpose. Modify a model. Re-run it. See what changes.

Business Context

What business is this data even about?

NorthStar Commerce sells consumer electronics online. A mid-sized e-commerce business, asking the questions every e-commerce business asks: What’s our revenue? Which customers are about to churn? What products are driving growth?

Six tables, each standing in for a real source system:

None of it is real. But it isn’t random noise, either. It’s generated with Faker, using a fixed seed. That means two things: it’s reproducible — you and I get the same dataset — and it’s shaped like real behavior: tier-weighted purchases, realistic margins, geographic spread.

Clone the Repo

Here’s how to clone the repo:

git clone https://github.com/practiceofdata/northstar-data-stack

Then make sure you’re inside it:

Then go ahead and create a virtual environment, activate it, install the requirements, then run the pipeline script:

That last command runs everything:

• Generates data,
• Loads it into DuckDB,
• Runs dbt models,
• Runs dbt tests,
• Builds the KPI summary

It prints progress as it goes, and when it’s done, you have a live warehouse at warehouse/northstar.duckdb.

After You Run It

You’ll have data populated in your project:

• data/raw/ - six CSVs, ~135,000+ rows total
• warehouse/northstar.duckdb - a live database with three schemas: raw (loaded CSVs), staging (cleaned views), marts_* (finance, marketing, product)
• raw_audit.loads - tracks every load, with row counts and timestamps
• transform/target/ - dbt's compiled SQL, manifest, test results
• reports/summary.md - an auto-generated KPI summary

You went from nothing to a fact-and-dimension warehouse — revenue rollups, customer lifetime value, product performance — cleaned, joined, and tested. You didn’t write a single line of SQL.

How It Works

The pipeline is a straight line:

Source data → Ingestion → Warehouse → Transform → Test → Orchestrate → Report (Faker) (Python) (DuckDB) (dbt) (dbt) (Dagster) (SQL)

Inside the warehouse, dbt organizes tables into three layers:

• Staging (views) — one-to-one with raw sources, renamed, type-cast, lightly filtered
• Intermediate (ephemeral, never materialized) — joins and business logic, like gross profit per order or RFM scores
• Marts (tables) — the final analytics tables, split by domain: finance (fct_orders, mart_daily_revenue), marketing (dim_customers, mart_customer_cohorts), product (dim_products, mart_product_performance)

Dagster wraps this whole flow as a graph of assets — raw data, warehouse load, staging, intermediate, marts, tests, the summary report — each aware of what it depends on, all runnable on a daily schedule.

The Four Tools

Four tools, four jobs. No overlap. That’s why they work together.

Faker generates realistic fake data — names, emails, addresses, prices. It’s seeded for reproducibility and weighted (tiers, purchase frequency, margins) so the output looks like real e-commerce behavior, not noise.

DuckDB is the warehouse. In-process, file-based, columnar. No server, no connection string. Just a file on disk you query directly. Fast for the aggregation-heavy work this stack does. Real teams increasingly use it as a lightweight local warehouse, not a toy.

dbt is the transform layer. You write SQL SELECT statements. dbt turns them into version-controlled, tested, documented tables. It enforces the staging → intermediate → marts pattern, runs schema tests, and builds a lineage graph. The SQL you write here works the same on Snowflake or BigQuery.

Dagster is the orchestrator. It schedules and sequences the pipeline. Airflow thinks in tasks — run this, then that. Dagster thinks in assets — the tables themselves — and models the dependencies explicitly. It knows dim_customers depends on stg_customers, and only re-runs what needs to re-run.

Try These First

Once it’s running, try running these 3 queries.

1. Total revenue

This hits fct_orders, built by joining orders, line items, and products, with profit computed per order. You didn't write that join. dbt did.

2. Best customers

Every customer is scored on Recency, Frequency, and Monetary value, then labeled — Champion, Loyal, At Risk, Dormant. The logic lives in plain SQL, in transform/models/intermediate/int_customer_metrics.sql. No black box.

3. Product velocity

Products are bucketed by units sold. This shows how concentrated revenue is in a few fast-movers versus a long tail. It usually is.

Go Build Something

You now have a working end-to-end data stack, on your machine. The same pattern — staging to marts, tested transforms, Dagster asset-based orchestration.

Clone it. Run the quick wins. Play with it. Then try these: trace a number back to its raw source, break a test on purpose, watch the Dagster asset graph run, build a new mart.

The repo: **github.com/practiceofdata/northstar-data-stack**

Originally published at https://practiceofdata.com.