Tracking London’s Pulse: A Real-Time TfL Map with Apache Flink, PostGIS & Grafana

London’s red buses are iconic, but their digital footprints are often a game of “now you see me, now you don’t.” While the TfL Unified API provides precise arrival predictions, the actual live coordinates of the vehicles are often hidden between the lines of data — and needs a stack that handles spatial-temporal data without breaking a sweat.

The Stack

End-to-end pipeline that ingests every London bus stop’s arrival predictions from the TfL Unified API, interpolates “ghost” vehicle positions between stops, persists them in DB, and visualizes the live fleet — Apache Flink, PostGIS, and Grafana — all on docker. Everything runs as a Docker Compose stack.

• Stream Processor: Apache Flink 1.18.1 (Java 11)
• Storage: PostgreSQL 16 + PostGIS 3.4
• Visualization: Grafana 11 (Geomap Panel)
• Source: A Python-based TCP “tick” service triggering polls every 30 seconds.

                         +------------------+
                         |   tick-feed      |  python TCP :9999, "tick\n" every 30s
                         +---------+--------+
                                   |
                                   v
+-----------+    +-----------+    +------------------+    +-------------+
| postgres  |<-->| jobmanager|<-->| taskmanager (x1) |--->| postgres    |
| (postgis) |    | (Flink)   |    |  Flink workers   |    |  bus_positions
+-----------+    +-----+-----+    +------------------+    +-------------+
       ^               |
       |               v
       |         +-----+------+
       |         | submit-job | runs `flink run -c TflBusStreamingJob`
       |         +------------+
       |
       |       +-----------+
       +-----> |  grafana  | provisioned PostGIS datasource + TfL dashboard
               +-----------+

Start by registering an account at https://api-portal.tfl.gov.uk/ and generate your own key. We will have to create a Subscription first to generate key for our API requests which allows 500 requests/min. Now, before we can track a bus, we have to know where it’s going. The pipeline begins with HubSync. It pages through the TfL canonical endpoint (/StopPoint/Mode/bus), ingesting only major 52 hub stations for our initial test. We will later scale up the sync for all possible stops to evaluate Flink’s compute power.

Phase 1: The Flink Streaming Job

Using a 30-second tick as a trigger, the PollArrivalsFlatMap executes concurrent /Arrivals calls. With a parallelism of 48, we hit a throughput of ~633 RPS. This allows us to sweep the entire city's arrival predictions within our 30-second budget without exceeding TfL rate limits.

We initialize our postGIS DB with appropriate schema and index to prepare for geo attributes:

CREATE TABLE IF NOT EXISTS hub_stops (
    naptan_id       TEXT PRIMARY KEY,
    common_name     TEXT,
    stop_type       TEXT,
    lat             DOUBLE PRECISION NOT NULL,
    lon             DOUBLE PRECISION NOT NULL,
    hub_naptan_code TEXT,
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX IF NOT EXISTS idx_hub_stops_stop_type ON hub_stops (stop_type);

CREATE TABLE IF NOT EXISTS bus_positions (
    time                 TIMESTAMPTZ NOT NULL DEFAULT now(),
    vehicle_id           TEXT NOT NULL,
    route                TEXT NOT NULL,
    location             geometry(Point, 4326) NOT NULL,
    bearing              REAL,
    next_naptan_id       TEXT,
    prev_naptan_id       TEXT,
    raw_time_to_station  INTEGER,
    ghost_fraction       REAL
);

CREATE INDEX IF NOT EXISTS idx_bus_positions_time ON bus_positions (time DESC);
CREATE INDEX IF NOT EXISTS idx_bus_positions_vehicle_time ON bus_positions (vehicle_id, time DESC);
CREATE INDEX IF NOT EXISTS idx_bus_positions_location ON bus_positions USING GIST (location);

Phase 2: The Ghost in the Machine (Stateful Interpolation)

This is where it gets technical. Using Flink’s KeyedProcessFunction (GhostEnrichFunction), we track each vehicle_id.

• State: We store the previous stop, the next stop, and the time of the last update.
• Interpolation: We calculate a “Linear Relationship” (lerp) between two points. If a bus is 60 seconds away from a stop on a 300-second leg, Flink calculates its likely spatial coordinate in real-time.

The Spatial Sink

The enriched BusPosition POJOs are sent to a PostgisSink. Instead of raw strings, we use ST_MakePoint to transform coordinates into active geometry, indexed via GIST for lightning-fast spatial queries. In terms of Idempotency: hub_stops upsert uses ON CONFLICT (naptan_id) DO UPDATE, so re-running the sync is safe.

Phase 3: Visualizing the Fleet

In Grafana, we use the Geomap panel. By running a DISTINCT ON (vehicle_id) query against our PostGIS table, we pull the most recent "interpolated" position for every bus in the city.

Pro Tip: We set the Grafana refresh rate to 10s and the time window to the last 15 minutes to keep the map fluid and responsive.

Accessible via UI: http://localhost:3000 (admin/admin)

Grafana vehicle live location tracker

Just for fun, I later included all possible bus stop locations — and of course Flink and Grafana was easily able to populate them without a hiccup.

All vehicle locations

Summary

By combining the stateful processing power of Apache Flink with the spatial intelligence of PostGIS + Grafana, we’ve turned a simple arrival API into a living, breathing digital twin of London’s bus network.