How I Built a Real-Time Data Pipeline to Track Crypto Prices Using Python + Streamlit

Part 3 of the “Stop Doing This” series. Live dashboard, auto-refreshing every few seconds — no Kafka, no Airflow, no cloud bill.

Everyone thinks “real-time data pipeline” means Kafka, Spark, a Kubernetes cluster, and a $400 cloud invoice.

It doesn’t. Last weekend I built a live crypto dashboard that pulls fresh prices, computes rolling indicators with Polars, and redraws charts every few seconds — all in one Python file running on my laptop. Deploy cost: $0.

Here’s the whole build, top to bottom. By the end you’ll have a portfolio piece you can point a recruiter at.

“Real-time” doesn’t mean “distributed.” For 99% of dashboards it means “refreshes while you watch.” That’s a single file, not a cluster.

What We’re Building

A live dashboard that:

1. Pulls recent price data from a free public crypto API.
2. Transforms it with Polars — rolling moving average, returns, volatility.
3. Renders an auto-refreshing line chart + live metrics in Streamlit.
4. Deploys free on Streamlit Community Cloud.

The architecture is refreshingly boring:

Public API  →  Polars transform  →  Streamlit UI (auto-refresh)

No message queue. No orchestrator. Just a fetch, a transform, and a render, on a loop.

Step 1 — Fetch the Data

Use any free price API (CoinGecko’s public endpoint works without a key). We wrap it so a failed request never crashes the dashboard:

import requests

PI = "https://api.coingecko.com/api/v3/coins/bitcoin/market_chart"
def fetch_prices(days: int = 1) -> list[dict]:
    try:
        r = requests.get(API, params={"vs_currency": "usd", "days": days}, timeout=10)
        r.raise_for_status()
        prices = r.json()["prices"]          # [[ms_timestamp, price], ...]
        return [{"timestamp": int(t / 1000), "price": float(p)} for t, p in prices]
    except requests.RequestException:
        return []                            # fail soft; UI shows "no data"

That try/except is the difference between a dashboard that survives a flaky network and one that white-screens in front of your audience.

Step 2 — Transform With Polars

Here’s where Polars shines. From a flat list of {timestamp, price} we derive everything the chart needs — in one expression chain:

import polars as pl

def transform(payload: list[dict]) -> pl.DataFrame:
    return (
        pl.DataFrame(payload)
        .with_columns(
            pl.from_epoch("timestamp", time_unit="s").alias("time"),
        )
        .with_columns(
            pl.col("price").rolling_mean(window_size=20).alias("ma_20"),
            pl.col("price").pct_change().alias("returns"),
        )
        .with_columns(
            (pl.col("returns").rolling_std(window_size=20) * (60 ** 0.5))
            .alias("volatility"),
        )
        .drop_nulls()
    )

I ran this on 500 simulated ticks to confirm it works end to end:

latest price: 64327.75
20-pt MA:     64421.86
volatility:   0.0056

Rolling mean, percentage returns, and a rolling-volatility estimate — three financial indicators, zero loops, fully vectorized.

Step 3 — Build the Streamlit UI

This is the entire app. Save it as app.py:

import time
import streamlit as st
import polars as pl
# from the snippets above:
from pipeline import fetch_prices, transform

st.set_page_config(page_title="BTC Live", layout="wide")
st.title("₿ Bitcoin - Live Tracker")
REFRESH_SECONDS = 5
placeholder = st.empty()
while True:
    df = transform(fetch_prices(days=1))
    if df.is_empty():
        st.warning("No data right now - retrying…")
    else:
        latest = df.tail(1).to_dicts()[0]
        with placeholder.container():
            c1, c2, c3 = st.columns(3)
            c1.metric("Price (USD)", f"${latest['price']:,.0f}")
            c2.metric("20-pt MA", f"${latest['ma_20']:,.0f}")
            c3.metric("Volatility", f"{latest['volatility']:.2%}")
            chart_df = df.select(["time", "price", "ma_20"]).to_pandas()
            st.line_chart(chart_df, x="time", y=["price", "ma_20"])
    time.sleep(REFRESH_SECONDS)

Run it locally:

pip install streamlit polars requests
streamlit run app.py

That while True + st.empty() placeholder is the "real-time" trick — Streamlit repaints the same container every 5 seconds, so the chart and metrics update live without a page reload.

Note: st.line_chart wants a Pandas frame, so we call .to_pandas() at the very last step. Do all the heavy lifting in Polars; convert only the tiny final slice you render.

Step 4 — Deploy for Free

1. Push app.py, pipeline.py, and a requirements.txt to a public GitHub repo.
2. Go to share.streamlit.io, connect the repo, pick app.py.
3. Click deploy.

A minute later you have a public URL you can drop straight into your portfolio or LinkedIn. No server to manage, no bill.

Why This Architecture Is the Right Default

Senior engineers reach for Kafka and Airflow when the problem actually demands them — fan-out to many consumers, guaranteed delivery, complex DAG scheduling. A single dashboard polling an API needs none of that.

Reach for the heavy stack when you have:

• multiple independent consumers of the same stream,
• strict exactly-once delivery requirements, or
• orchestrated multi-step jobs with retries and backfills.

Until then, a fetch + a transform + a render loop is not a toy — it’s the correct, boring, maintainable choice.

The Takeaway

A “real-time pipeline” is mostly three honest steps: get fresh data, transform it cleanly, show it live. Polars handles the transform in a single vectorized chain; Streamlit handles the live UI in a while loop; the cloud part is free. The complexity people associate with streaming is usually complexity they don't need yet.

Build this once, swap Bitcoin for whatever you actually care about — your app’s metrics, your server load, your store’s orders — and you’ve got a portfolio piece that moves.

This is Part 3 of the Stop Doing This series. Part 2 showed how to refactor 200 lines of Pandas into 20 lines of Polars. Next up: why modern teams are quietly moving away from complex data lakes.

What would you point this dashboard at? Tell me in the responses — I’ll suggest the transform. 👇