Enrich Your Movie Dataset with TMDB Metadata using Python

This is the first post in a 3-part series on creating a dashboard for your movie watching habits. We’ll start by enriching your dataset with TMDB metadata.

Photo by Donovan Silva on Unsplash

Watching movies has always been one of my favorite pastimes. But as a data geek, I couldn’t help but wonder: what patterns are hiding in my movie-watching habits? Which directors, genres, or countries dominate my watching history?

Movie-tracking platforms (Letterboxd, IMDb, or Trakt) are great for logging what you watched — but the exported CSVs are fairly minimal. They usually include a title, year, rating, and date watched, but barely any real metadata.

In this post, I’ll show how to enrich a movie dataset with TMDB metadata using Python and a caching system. With this enriched data, you can start asking questions like:

• Which director’s movies do I watch most often?
• Do I prefer English-language films, or do I watch a lot of world cinema?
• What are my favorite eras of film history?

Using a small and robust API function and a local caching mechanism, this workflow avoids repeated TMDB calls and prepares the dataset for analysis or dashboarding. The approach mirrors the structure in my movie-dashboard repo.

Prerequisite: A clean dataset with TMDB IDs

The only assumption for this post is that you have a CSV with fields like: title, release year, and tmdb_id

The TMDB ID is the key here: it lets you fetch metadata without fuzzy title-matching.

If your dataset does not contain TMDB IDs yet, here are some options to retrieve them:

• Use TMDB’s /search/movie endpoint based on title + year
• Use IMDb IDs → TMDB IDs via /find/{imdb_id}
• Use an existing dataset (e.g., from Kaggle)

But in this tutorial, we begin with TMDB IDs ready to go.

Project structure (relevant parts)

movie-dashboard/
│
├── src/
│   ├── tmdb_api.py            # API function for TMDB
│   └── ...
│
├── scripts/
│   └── update_cache.py        # Script to build or refresh local cache
│
└── data/
    ├── your_movies.csv    # provided by user 
    └── tmdb_cache.csv     # Generated on first run

The TMDB API Function

My project uses a single function: search_movie(). It can either:

• Search TMDB by title (and optionally year), or
• Fetch directly by tmdb_id if available.

It returns a dictionary with all relevant metadata, including directors, cast, screenwriters, cinematographers, genres, runtime, spoken languages, and production countries.

Here’s the simplified usage pattern:

from src.tmdb_api import search_movie

# Search by title and year
movie_data = search_movie(title="Pulp Fiction", year=1994)
# Fetch directly by TMDB ID
movie_data = search_movie(tmdb_id=680)
print(movie_data)

How search_movie() works

1. Check TMDB ID first If a tmdb_id is provided, the function skips the search and fetches details directly.
2. Search by title/year If no TMDB ID exists, the function calls /search/movie to find the best match.

• If nothing is found for the given year, it retries without the year.
• The first search result is selected as the match.

Note that this approach is not optimal if there are multiple movies with the same name in the same year

3. Fetch detailed metadata Using /movie/{id}?append_to_response=credits, it retrieves:

• Runtime
• Genres
• Production countries
• Spoken languages
• Director(s)
• Screenwriter(s)
• Cinematographer(s)
• Cast

4. Return a clean dictionary All fields are returned as strings (comma-separated for multiple entries) or numeric values (e.g., runtime).

{
  "tmdb_id": 680,
  "title": "Pulp Fiction",
  "release_date": "1994-09-10",
  "actors": "John Travolta, Samuel L. Jackson, Uma Thurman, Bruce Willis, ...",
  "directors": "Quentin Tarantino",
  "screenwriters": "Quentin Tarantino, Roger Avary",
  "cinematographers": "Andrzej Sekuła",
  "runtime": 154,
  "genres": "Crime, Drama",
  "spoken_languages": "English",
  "production_countries": "United States"
}

Handling API Errors Gracefully

When working with external APIs like TMDB, requests can fail for many reasons: network issues, rate limits, or missing data. The search_movie() function includes robust error handling to the pipeline continues without crashing. Here is how it works:

1. Network issues: ConnectionError

except ConnectionError as conn_err: 
    print(f"Connection Error for '{title}': {conn_err}")

Raised if the API server is unreachable or the network fails.

2. HTTP errors: HTTPError

r.raise_for_status()  # triggers this if status_code >= 400

• Any 4xx (client error) or 5xx (server error) triggers an HTTPError.
• Example: a bad TMDB ID or invalid API key.
• The except HTTPError block logs the issue and skips that movie.

3. Request timeouts: Timeout

except Timeout as timeout_err: 
    print(f"Timeout Error for '{title}': {timeout_err}")

• If the request takes longer than the timeout specified (5 seconds in my function), a Timeout is raised.
• Prevents the script from hanging indefinitely on a slow or stalled response.

4. Unexpected issues: generic Exception

except Exception as e: 
    print(f"Unexpected error for '{title}': {e}")

Catches any other exceptions, such as parsing errors or missing fields.

5. Safe return value

If any error occurs, or if no matching movie is found, the function returns None. This design allows that the metadata loop skips problematic entries without stopping. Failed movies can be retried later.

Fetching and caching metadata

Fetching metadata for hundreds of movies can be slow and may hit TMDB rate limits, so I’ve implemented a robust caching mechanism

1. Load movies data and existing cache

• Load the local CSV file containing the TMDB IDs.
• If a local CSV exists that already contains metadata (tmdb_data.csv), it is loaded into memory (existing_df).
• All tmdb_ids already fetched are stored in a set (existing_ids) to skip them in the next fetch.

import pandas as pd
from pathlib import Path
from tqdm import tqdm
import time
from random import uniform
from src.tmdb_api import search_movie

TMDB_DATA_FILE = Path("data/tmdb_data.csv")
MOVIES_CSV = Path("your_movies.csv")
# load movie data
movies_df = pd.read_csv(MOVIES_CSV)
# check cache
if TMDB_DATA_FILE.exists():
    existing_df = pd.read_csv(TMDB_DATA_FILE)
    existing_ids = set(existing_df["tmdb_id"].astype(int))
    print(f"Loaded {len(existing_ids)} cached movies")
else:
    existing_df = pd.DataFrame()
    existing_ids = set()
    print("No cache found - starting fresh")

2. Identify movies with missing metadata:

• The script compares your dataset’s tmdb_ids with existing_ids.
• Only the IDs not yet in the cache are fetched (remaining_ids).

This ensures incremental enrichment and prevents repeated API calls.

all_ids = [int(i) for i in movies_df["tmdb_id"]]
remaining_ids = [i for i in all_ids if i not in existing_ids]
print(f"{len(remaining_ids)} movies need metadata")

3. Fetch metadata with delays

• Metadata is retrieved using the search_movie() function.
• If metadata is found, appends it to new records
• Each request is followed by a random delay (2.5–5s) to avoid hitting TMDB rate limits.
• Progress is displayed with tqdm, making it easy to monitor long runs.

new_records = []

for idx, tmdb_id in enumerate(tqdm(remaining_ids, desc="Fetching TMDb metadata"), 1):
    # Fetch metadata
    movie_data = search_movie(tmdb_id=tmdb_id)
    if movie_data:
        new_records.append(movie_data)

    # incremental saving logic here (step 4)

    # Random delay between requests to respect TMDB rate limits
    time.sleep(uniform(2.5, 5))

4. Incremental saving

• Every 25 movies, the newly fetched records are merged with the existing cache and written to CSV.
• Duplicates are removed based on tmdb_id.

This allows resuming the script safely if interrupted.

# Incremental save every 25 movies
    if idx % 25 == 0 and new_records:
        new_df = pd.DataFrame(new_records)
        combined_df = pd.concat([existing_df, new_df], ignore_index=True)
        combined_df.drop_duplicates(subset="tmdb_id", inplace=True)
        combined_df.to_csv(TMDB_DATA_FILE, index=False)
        existing_df = combined_df
        new_records.clear()

5. Final merge and inspection

• After all remaining IDs are processed, any leftover records are merged and saved.
• The cached CSV can then be inspected to confirm columns and total entries.

# Save any remaining new records
if new_records:
    new_df = pd.DataFrame(new_records)
    combined_df = pd.concat([existing_df, new_df], ignore_index=True)
    combined_df.drop_duplicates(subset="tmdb_id", inplace=True)
    combined_df.to_csv(TMDB_DATA_FILE, index=False)

The cache is stored locally, so repeated runs do not hit the API unnecessarily. This keeps the enrichment process fast and reliable.

Why this approach works well:

• Efficient: only missing metadata is fetched.
• Safe: randomized delays and batch-saving protect against rate limits.
• Idempotent: Re-running the script won’t duplicate records or overwrite good data.

Takeaways

With your dataset enriched with TMDB metadata, you now have a foundation for analysis or building your own dashboards. You can explore viewing habits by genre, runtime, country, language, or even by director and cast.

Key takeaways from this tutorial:

• Stable IDs simplify data retrieval
• Proper error handling keeps the enrichment pipeline robust
• Local caching prevents redundant API calls and allows safe incremental updates.

In the next post, we’ll take this enriched dataset and create visualizations for your Streamlit dashboards.

This is blog 1 of 3 in a series on creating a dashboard for your movie watching habits. In the next posts, we’ll visualize insights from your enriched dataset and build the dashboard itself. Check out the other two blogs here: Blog 2: Visualize Your Movie Dataset with Plotly (Optimized for Streamlit) and Blog 3: Build Your Movie Dashboard with Streamlit.