From Notebook to Cloud: Building a Self-Updating Data Pipeline as a Bootcamp Project

A career changer’s walkthrough of scraping Wikipedia, calling three APIs, and deploying everything to Google Cloud — without a CS degree, and without dying in the logs.

Imagine you run an e-scooter company across 22 European cities. Every morning, you need to know: where will tomorrow’s weather be good? Where are flights landing? Where are concerts happening this weekend? That’s the data problem behind Gans, the fictional company my bootcamp asked us to build a pipeline for. I’m Abdul Rahman. I’m a career changer in Berlin, finishing up the Data Science programme at WBS Coding School. Before this, I spent five years in dialog marketing and leadership — different world entirely. When my bootcamp introduced the cloud project, it was the first time I’d ever deployed code that ran without my laptop being open. This is the story of how I got there, what broke along the way, and what I’d tell someone about to start the same project.

The shape of the system

At the end of all this work, the architecture looks like this:

Five moving pieces: A Python pipeline that scrapes Wikipedia for city facts, then calls three APIs (OpenWeather, AeroDataBox, Ticketmaster) for fresh weather, flights, and events data. A MySQL database with five related tables, holding around 3,400 rows of live data at any moment. A Cloud Run function that bundles the pipeline as Python code and lives on Google’s servers. A Cloud Scheduler job that pings the function once every morning. And, lurking behind all of it, a handful of permission settings and connection details that took me longer than I’d like to admit. The local version runs on my laptop. The cloud version runs on its own, refreshes itself, and keeps the database fresh without me touching anything. Both use exactly the same Python code. Only the trigger and the connection details change. Stage 1: Scraping Wikipedia The pipeline starts with 22 European cities. For each one, I needed country, population, area, elevation, and coordinates. Wikipedia has all of this, but it’s wrapped in messy HTML. I wrote a small get_city_data function using BeautifulSoup:

def get_city_data(city):
    url = f"https://en.wikipedia.org/wiki/{city}"
    headers = {"User-Agent": "Chrome/134.0.0.0"}
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.content, "html.parser")

    geo = soup.find("span", class_="geo").get_text()
    lat, lon = [float(x) for x in geo.split(";")]

    country = get_label_value(soup, ["Country", "Sovereign state"])
    population = extract_number(get_label_value(soup, "Population"))

The trickiest part wasn’t the scraping itself — it was the cleanup. Wikipedia infobox values look like “3,769,495 (2023)[1]”. Pulling a usable integer out of that took a small parser that walks character by character, collecting digits and decimal points, stopping at the first parenthesis or footnote. Capital cities use the label “Sovereign state” while other cities use “Country”, so the lookup function had to accept multiple label options.

This was my first lesson: most of data engineering is cleanup, not collection. Get the messy thing, clean it carefully, then everything downstream is easy.

Stage 2: A relational schema, not a giant spreadsheet

A common beginner instinct (mine included) is to stuff everything into one wide table. The proper way is to split it into related tables and let foreign keys do the joining work.

I ended up with five tables:

cities — the static stuff (name, country, area, coordinates).

populations — separated because the population of a city changes over time. Each row is a snapshot with a recorded_on date.

weather — 5-day, 3-hour-interval forecasts per city.

flights — tomorrow’s airport arrivals.

events — upcoming concerts and events.

Every table links back to cities.city_id. That key is what stops the database from collecting “weather for a city that doesn’t exist.”

The pattern that comes back over and over: I’d scrape data with city names, but the DB wanted city IDs. So between the scraper and the loader, I’d run a small SQL query, do a pandas merge to swap names for IDs, then load. Once you’ve seen it three times in three different stages, you stop having to think about it.

Stage 3: Three APIs, three slightly-different shapes

The three APIs all return JSON, but each one expects authentication differently:

OpenWeather takes the API key as a URL parameter.

AeroDataBox takes it as a request header.

Ticketmaster takes it as a URL parameter again, but with a different naming convention.

Reading each API’s docs carefully was non-negotiable. Skip that, you spend an hour wondering why the server keeps saying 401.

The pattern I leaned on heavily was defensive nested access:

flight.get("movement", {}).get("scheduledTime", {}).get("local")

Read aloud, that’s: “Get movement; if it’s missing, use an empty dict. From that, get scheduledTime; if missing, empty dict. From that, get local.” The alternative — flight[“movement”][“scheduledTime”][“local”] — crashes the moment any single field is missing. With around 3,000 flight records coming in daily, something is always missing. Writing the defensive version once means the pipeline never breaks because one airline forgot to populate one field.

Stage 4: From localhost to the cloud

Once the pipeline worked locally, the question was: how do I make it run when my laptop is closed?

I’d vaguely heard the word “serverless” before this project. Now I think I actually understand it: there are servers, they’re just not mine. Google runs them, I send them code, they bill me by the second.

The first cloud step was the database. I created a Cloud SQL instance in europe-west1 (Belgium, closest cheap European region to Berlin), 1 vCPU, single zone, no auto-backups — the cheapest viable configuration. Around €5 to €10 per month, comfortably covered by Google’s €300 free trial credits.

The most satisfying realization: pointing my notebook at the cloud database required changing exactly one thing. The connection URL has five ingredients:

mysql+pymysql://root:password@127.0.0.1:3306/gans

Swap 127.0.0.1 (my laptop) for 34.62.139.31 (Google’s server in Belgium), and the same notebook now writes to the cloud. I wrapped this in a toggle:

USE_CLOUD = True

if USE_CLOUD:
    host = os.getenv("CLOUD_HOST")
    password = os.getenv("CLOUD_PASSWORD")
else:
    host = "127.0.0.1"
    password = os.getenv("MYSQL_PASSWORD")

One boolean. Two worlds.

Stage 5: Putting the code itself in the cloud

A database in the cloud is one thing. Code in the cloud is another. I needed a place to put the Python pipeline so Google could run it when poked.

That’s Cloud Run Functions. You write a main.py, declare your dependencies in requirements.txt, and Cloud Run wraps it in a container, gives you a URL, and runs it whenever the URL is hit. The function I built routes by URL parameter:

The bare URL runs the daily refresh of weather, flights, and events.

Adding ?action=add_city&city=Hamburg scrapes and adds a new city to the database.

Adding ?action=test runs a sanity-check insert into a test table.

One function, three behaviors, depending on the URL.

The bugs that taught me the most

This is the part I’d actually want a hiring manager to read.

Bug 1: The reserved-word SQL error.

SELECT 'cities' AS table_name, COUNT(*) AS rows FROM cities

The error message was generic and confusing. Turns out rows is a reserved word in MySQL 8. Renaming the column alias to row_count fixed it instantly. Lesson learned: when SQL “should obviously work” but throws a syntax error, the first thing to suspect is a reserved word.

Bug 2: ImportError: libsqlite3.so.0.

This one took me an hour to read properly. My cloud function would deploy fine, but every URL hit returned a 500. The stack trace blamed pandas.to_sql. The real cause, ten lines deeper in the trace, was that Google’s Python 3.13 container shipped without the underlying libsqlite3 library. Pandas does an import sqlite3 probe even when you’re using MySQL, so the missing library crashed the whole thing.

The fix was three lines at the top of main.py:

import pysqlite3
import sys
sys.modules["sqlite3"] = pysqlite3

This tells Python “when anyone tries to import sqlite3, give them pysqlite3 instead.” Pysqlite3 ships with its own copy of the underlying library, so it works in the container. My code never actually touches sqlite — but pandas’ probe is silently redirected to a library that works.

The bigger lesson: cloud runtimes have quirks that don’t show up locally. And when reading a stack trace, the truth is usually at the bottom, not the middle.

Bug 3: The connection that wasn’t.

Even with the sqlite shim, the function still 500'd. New trace, new mystery: it couldn’t reach the database, despite my open-to-the-world 0.0.0.0/0 firewall rule.

The cause: Cloud Run blocks direct outbound connections to Cloud SQL by IP, even when you added “Cloud SQL connections” in the function config. That config doesn’t magically rewrite your Python — it just opens a special Unix socket inside your container at /cloudsql/<project:region:instance>. Your code has to use that path, not the public IP.

The fix was reshaping the connection URL:

socket_path = f"/cloudsql/gans-pipeline:europe-west1:wbs-mysql-db"
url = f"mysql+pymysql://{user}:{pw}@/{schema}?unix_socket={socket_path}"

No host. No port. A socket path instead. Two days of confusion ended when I read one paragraph of Google’s docs more carefully.

Lesson: cloud consoles have “Connect” buttons that open a door. Your code still has to walk through it.

Stage 6: The alarm clock

The final piece is Cloud Scheduler. This is just a managed cron job — Google’s “alarm clock as a service.” I set up a job called wbs-scheduler with the cron expression 0 6 * and timezone Europe/Berlin. Every morning at 6 AM Berlin time, Scheduler sends an HTTP GET to my function’s URL, which triggers the full daily refresh.

The pipeline is now fully autonomous. My laptop is closed. By the time I wake up, the cloud database has refreshed weather, flights, and events for the next day.

What I’d tell someone starting this project

Five things:

The cloud is not a thing — it’s services that need wiring. When something doesn’t work, find which service is failing, find which connection is broken, and fix that one thing. Always one thing.

Read stack traces from the bottom up. Python prints the call chain newest-first. The actual cause is usually the last line. I wasted hours staring at the middle.

Defensive code is calm code. Using .get(“a”, {}).get(“b”) looks paranoid until the day an API returns half a response and your code keeps running instead of crashing at 3 AM.

One variable can save a project. A single USE_CLOUD = True let me develop locally and deploy with the same notebook untouched. Identical code, two environments.

You will hit bugs that have nothing to do with your code. The libsqlite3 issue wasn’t my fault — it was a missing system library in Google’s container. Knowing that some bugs are environmental, not yours, is half the battle of staying sane.

What’s next

A few things on my list:

A small dashboard to actually look at the data — probably Streamlit or Looker Studio.

More cities, easier to add now that I built the ?action=add_city URL.

Better cost monitoring — I’m still on free trial credits, but I want to understand the actual cost breakdown before I’m not.

If you’re a bootcamp student halfway through a similar project: keep going. The first cloud deploy is the hardest because everything is new at once. After that, every additional cloud service is just one more “Enable API” button and one more YAML field.

If you’re a recruiter or hiring manager: I’m looking for junior data analyst, marketing analyst, or BI roles in Berlin. The code for this project is on my GitHub and you can find me on LinkedIn.

If you spotted a mistake or have a question: leave a comment. I’m still learning. Pointing things out doesn’t hurt my feelings — it speeds up my learning.

“my GitHub”

[“LinkedIn”](http://www.linkedin.com/in/ abdul-rahman-abssi-b64bb640b)

Thanks to my instructors at WBS Coding School for designing this project, and to everyone in my cohort who debugged 500 errors alongside me.