Harvesting Trending Indonesian and Global Movie Data in 2025 at Scale: From TMDB Async API to Looker Studio

How I optimized fetching 10,000 movie records in parallel using Python AsyncIO and built a streamlined ETL pipeline to launch it on Google Looker Studio.

In the world of Data Engineering and Data Science, fetching data from APIs is our daily bread and butter. But what happens when you need to retrieve thousands of pages of data? If you stick to traditional synchronous methods, you might find yourself waiting hours for a script to finish.

For this portfolio project, I challenged myself to fetch weekly Trending Movies data from TMDB (The Movie Database). The goal wasn’t just to get the data, but to do it fast, efficiently, clean it, and load it into Google BigQuery for further analysis then I visualize it in Google Looker Studio.

Here is my journey in building this high-performance async pipeline.

1. The Setup: Why Async?

Typically, when we request data using the standard Python requests library, the computer works sequentially: Send Request 1 -> Wait for Response -> Send Request 2 -> Wait for Response, and so on. This “blocking” behavior is incredibly slow when you need to scrape 500 pages of pagination.

The solution? Asynchronous Programming.

By leveraging the aiohttp and asyncio libraries, we can fire off multiple requests simultaneously without waiting for the previous ones to complete. Think of it as opening 4 checkout lanes at a supermarket instead of forcing everyone through a single line.

Here, I set up the environment, loaded my API keys securely, and prepared the base fetching function.

2. The Engine: Parallel Data Fetching

This is the core of the project. I created a function called fetch_trending_movie_week_async. However, simply blasting the API with 500 simultaneous requests is a bad idea — it’s a quick way to get your IP banned for resembling a DDoS attack.

In accordance for tackle the DDos attack is to solve it, which I implemented a Semaphore.

What is a Semaphore? Think of it as a “bouncer” at a club. I set the concurrency=4, meaning only 4 active requests are allowed at any single moment. This keeps the script respectful of TMDB’s rate limits while still being exponentially faster than a synchronous loop.

The flow is:

1. Fetch Page 1 to determine the total_pages (which turned out to be 500 pages!).
2. Prepare tasks for pages 2 through 500.
3. Execute all tasks using asyncio.gather.

The result? I successfully retrieved 500 pages (approx. 10,000 movie records) in a fraction of the time.

3. Data Cleaning: Handling Duplicates

Once the data was collected, I encountered a classic data engineering problem: Dirty Data.

Because the “Trending” list on TMDB is dynamic (rankings can shift in real-time), it’s possible for the same movie to appear on different pages while the scraping script is running. I discovered about 200 duplicate records in the dataset.

I cleaned this by dropping duplicates based on the unique movie id and resetting the index.

I also added a helper function, make_poster_url, to convert the partial image paths provided by the API into fully accessible URLs.

4. Enriching Data: Genre Mapping

The TMDB API provides genres as a list of IDs (e.g., [28, 12]) rather than human-readable text (e.g., “Action”, “Adventure”). While efficient for computers, this isn’t great for analysis.

I built an additional async function, get_genre_mapping_async, to fetch the official genre reference list from TMDB.

I then mapped these IDs in my main DataFrame to create a readable “genres” column.

5. Segmentation: Local vs. Global Markets

A key part of my analysis plan was to compare the performance of domestic (Indonesian) films against the global market.

I split the cleaned DataFrame into two segments based on original_language:

1. Indonesian Market: Movies where the language is ‘id’.
2. Global Market: All other movies.

Interestingly, out of 10,000 trending movies this week, about 53 Indonesian films made the cut, competing alongside thousands of global blockbusters!

6. Query Processing: Google BigQuery

The next step is the Load phase of the ETL process. Clean, structured data is useless if it sits in a Jupyter Notebook. I utilized pandas_gbq and a Google Cloud Service Account to upload the data directly into BigQuery.

By doing this, the data is securely stored in a cloud data warehouse, ready to be connected to visualization tools like Google Looker Studio for real-time dashboarding.

7. The Result: Bringing Data to Life with Looker Studio

Clean data is useless if it doesn’t “speak” to us. The final step of this ETL pipeline is Visualization.

Since the data was already safely stored in BigQuery, I connected it directly to Google Looker Studio. This allowed me to build an interactive dashboard to monitor weekly movie trends without writing any extra code.

Looking at the dashboard, all the previous data cleaning and transformation steps pays off:

• Data Integrity: The dashboard displays a Total Movies count of 9,785. This matches our scraping results (post-deduplication), validating that our pipeline is reliable.
• Segmentation in Action: Remember when I split the data into “Indo” vs. “Global”? Here, we can visualize them separately. The movie “Agak Laen: Menyala Pantiku!” leads the local Indonesian chart, while global blockbusters like “Bureau 749” dominate the global chart with massive popularity scores.
• Genre Insights: The bar chart confirms that Drama and Comedy are currently the most dominant genres in the market.
• Poster Visualization: Recall the make_poster_url function from the cleaning stage? It proves vital here. The movie posters render perfectly at the bottom of the dashboard, transforming a boring dataset into an engaging visual experience.

Key Takeaways

Building this module taught me several valuable lessons:

1. The Power of AsyncIO: The speed difference between synchronous and asynchronous execution is massive for I/O-bound tasks like API fetching. It is a must-have skill for modern data engineers.
2. Rate Limiting Awareness: Speed is good, but respecting the data provider is better. Using Semaphores is the ethical and safe way to scrape data without getting blocked.
3. Data Quality Control: Never trust raw API data blindly. Always implement checks for duplicates, missing values, and data types before storage.
4. Cloud Integration: Integrating Python with Google BigQuery is seamless with the right libraries, opening the door to scalable data analytics.
5. End-to-End Flow: This project demonstrates the complete data engineering lifecycle: from fetching raw data (API), processing it (Python), storing it (Cloud Warehouse), to finally presenting visual insights (Dashboard).

This project proves that with the right code architecture, we can build robust, scalable data pipelines efficiently.

Thanks for reading!