Tableau Bridge: Concurrency & Queuing,and Job Management Issues: Part 2

The Orchestration App Architecture, Database Design, and the Python/API Logic.

Photo by Vitaly Gariev on Unsplash

I am a Senior Sales Solution Architect at Snowflake with over 14 years of data consulting, analytics, and architecture experience.

The views expressed here are mine alone and do not necessarily reflect the views of my current, former, or future employers.

Where do we start?

Following up from our saga of failures and frustrations in Part 1, let’s talk about the solution we built to bring order to the chaos.

Our orchestration app was designed to do two things:

1. Ensure Tableau Cloud jobs are processed as quickly as possible.
2. Prevent our Tableau Bridge clients from collapsing into an endless stream of job failures.

In our particular case, we used Apache Airflow (running DAGs that call .py files) as the orchestrator and Postgres as the database for processing transactions. While these tools worked for us, the architecture is flexible—you can swap these out for whatever stack fits your environment.

Before we dive deep into the technical weeds, here’s our high-level approach:

• Ingest: Using Tableau’s API, we query all data sources from our environment and ingest the metadata into a Postgres table.
• Audit Prep: We insert these records into our audit table, with status columns initially set to NULL.
• The “Sliding Window”: We send N number of jobs to Tableau Cloud to be refreshed, where N is the concurrency limit of your Tableau Bridge environment (minus a buffer for the unexpected).
• Polling: Every 5 minutes, we check the status of these active jobs, looking for one of three categories: Success, Failure, or Pending.
• Update: If a job is a success or a failure, we capture the timestamp and update our audit table.
• Replenish: We calculate how many slots have opened up (Success + Failure count) and send that many new jobs to Tableau Cloud.
• Repeat: This process loops until the queue is empty.
• Retry: For failed jobs, we have a separate process that runs after the initial processing to attempt a single re-refresh.

Simple, right? Let’s dive into each area and talk about the key considerations.

Building Your Tables

In our implementation, we use two primary tables: tableau_datasources and tableau_datasources_audit.

The tableau_datasources table is our inventory. We truncate and reload this table every morning to capture the current state of the environment. To acquire both published and embedded data sources, we use two specific endpoints:

GET /api/api-version/sites/site-id/datasources
GET /api/api-version/sites/site-id/workbooks/workbook-id

The first endpoint provides a list of all published data sources. You can simply loop through the IDs to get the relevant metadata. In our environment, this accounted for about 90% of our data.

However, it is trickier to get the data sources embedded within workbooks. For these, you need to query the workbook connections:

GET /api/api-version/sites/site-id/workbooks/workbook-id/connections

Practically, this tells us that a single workbook might have one or many data sources embedded within it.

So, how do we store this? For published data sources, we insert the metadata directly into the tableau_datasources table. For embedded data sources, even though a workbook might return multiple rows (one for each connection), we chose to insert just a single row per workbook_id to simplify the refresh logic.

Pro Tip: You’ll likely want to use pageSize and pageNumber in your API calls. Tableau limits the size of API responses, and handling pagination is best practice to manage memory and avoid timeouts.

The Audit Table

The next step is to move this data into tableau_datasource_audit. To ensure our app is idempotent, we delete all records inserted TODAY() before running the insert.

This audit table is the brain of the operation. As the orchestration app runs, it updates these records to track state. You will want to add several columns to track the lifecycle of the job:

• **row_inserted_timestamp*: Timestamp* (Current time when the record is created)
• **datasource_refresh_created_timestamp*: Timestamp* (When we sent the "Run Now" command to Tableau)
• **datasource_refresh_start_timestamp*: Timestamp* (When Tableau actually started the refresh)
• **datasource_completed_timestamp*: Timestamp* (Success time)
• **datasource_failed_timestamp*: Timestamp* (Failure time)
• **tableau_refresh_id*: String* (The Task ID returned by the API)
• **failure_rescheduled_timestamp*: Timestamp* (Used if the job failed and we triggered a retry)

These three columns are for a more complex set of orchestrating where we utilize hourly, weekly, or monthly schedules. You can get by as a V1 without using these, but our use cases required we implement them.

• min_scheduled_timestamp: set to a timestamp value controlled by your app — The minimum time a job can kick off. This is useful for controlling data sources that should refresh at least after 8:00 am.
• max_scheduleded_timestamp: set to a timestamp value controlled by your app — The maximum time a job can kick off. This is useful for controlling data sources that should refresh no later than a certain time.
• schedule_interval: defaults to Daily, but can be controlled to hourly, weekly, or monthly or any other interval you choose.

Advanced Scheduling (V2)

You can get by with the columns above for a “V1” implementation. However, our use case required more complex orchestration, such as handling specific hourly, weekly, or monthly requirements. To handle this, we added three control columns:

• **min_scheduled_timestamp*: Controlled by the app. The earliest* time a job can kick off (e.g., "Don't refresh this until after 8:00 AM").
• **max_scheduled_timestamp*: Controlled by the app. The latest* time a job can kick off.
• **schedule_interval**: Defaults to 'Daily', but allows for custom intervals.

For jobs with a schedule_interval less than daily (e.g., hourly), you must control when they are inserted into the audit table. The audit table drives the API script. If a job needs to run multiple times a day, your script needs to duplicate the rows in the audit table, adjusting the min and max timestamps accordingly.

Before you get too much further, I highly encourage you to create a tableau_utils.py file to handle your common functions—you're going to need them.

But before we dive into the actual job processing logic, take a break and enjoy a nice cup of coffee.

Photo by Christian Bass on Unsplash

The Pipeline and Engine

As a developer, you want to think of this in three distinct phases:

1. Kick-off: Setting the initial batch of jobs.
2. Status Check: Monitoring jobs for updates.
3. Replenishment: Sending more jobs when processing slots open up.

Phase 1: Kick It Off!

My recommendation is to create a script that runs once a day to trigger your morning series of scripts. Let’s call this .py file tableau_ds_lets_kick_em_off.py.

The main function of this script is to know that it needs to process N number of jobs. Again, N is defined as the maximum number of jobs your Bridge clients can handle, minus a buffer. For our example, let’s assume we have the base requirements suggested by Tableau, meaning our Bridge client can handle 15 jobs at a time. We are conservative, so we subtract 3 from that number, making our N=12.

(N should be defined in your tableau_utils.py file, as it will be used universally across scripts.)

Your script will pick up both the data source ID and workbook ID and attempt to schedule a refresh task.

For data sources:

PUT /api/api-version/sites/site-id/datasources/datasource-id

For workbooks:

PUT /api/api-version/sites/site-id/workbooks/workbook-id

There are a couple of call outs which are worth pointing out. 1) We are not using the api to “run now” or “run extract refresh now”, we are instead creating a schedule for the next 5 minute increment in the future (8:05, 8:10, 8:15, etc.) The primary purpose for this decision is to ensure we do not exceed the api hourly limits set by Tableau.

Why try both endpoints? As a rule, embedded data sources cannot be refreshed with the data source endpoint, and published data sources cannot be refreshed with the workbook endpoint. We found that trying both endpoints was the best method to catch anything that might fall through the cracks — essentially a brute-force safety net to prevent Type II errors. This, however, was arguably an overly conservative decision, and can likely be removed.

After you have configured the correct API calls, you need to throttle your jobs to not exceed N. You must ensure your script counts toward N only when a refresh extract is successfully created on Tableau Cloud, not merely when the request was sent. If a refresh is unable to be scheduled, we update the audit record to 9999-12-31 to alert engineers that an issue exists with that particular asset.

When a job is successfully sent, log the datasource_refresh_created_timestamp and update the tableau_refresh_id with your newly created task ID. You’ll need this task ID for everything moving forward.

Note: Consider adding a forced refresh interval to your authentication token management. You don’t want your token expiring mid-process.

Phase 2: The Schedule Checker

The next step is to build a script that monitors the progression of your processing jobs. Let’s call this one tableau_ds_gotta_check_em_all.py.

At a high level, this script completes the following tasks:

1. Checks to see if your task refresh jobs have completed (success or failure).
2. Updates the Postgres table with appropriate timestamps.
3. Handles logic for Bridge jobs vs. regular Cloud jobs.

This script should run as often as you feel comfortable; for us, every 5 minutes was adequate.

The “Check” Dilemma We initially hoped to use a single API method to check statuses, but we found that the required information isn’t always accessible depending on the job type. We had to settle on using four different methods to triangulate the truth:

1. Jobs Endpoint: GET /api/api-version/sites/site-id/jobs/job-id (Gets detailed info, including finish codes and timestamps).
2. Tasks Endpoint: GET /api/api-version/sites/site-id/tasks/extractRefreshes/task-id (Gets info on scheduled refreshes).
3. GraphQL: http://<tableau-server-name>/api/metadata/graphql (Useful for published data source metadata).
4. Data Source Endpoint: GET /api/api-version/sites/site-id/datasources/datasource-id (Gets the updatedAt timestamp).

The Decision Tree When checking for status, first look for failures. We break this down into primary checks. First, check the consecutiveFailedCount.

• If consecutiveFailedCount > 0, we know for certain the job has failed.
• If consecutiveFailedCount = 0, we look at finishCode.

From the jobs endpoint, we get finishCode and notes. The notes field tells us if the job is a Bridge job or a regular job. This is critical because finish codes mean different things for different job types.

The “Finish Code” Gotcha:

**Bridge Jobs:**

• 0 = Still processing (assigned to a client)
• 3 = Success
• 1 or 2 = Failure

**Regular Cloud Jobs:**

• 0 = Success
• 1 or 2 = Failure

It’s not an author’s mistake that regular jobs and Bridge jobs have contradictory definitions for the code 0. Your guess is as good as mine as to why this exists, but knowing it will save you hours of debugging.

Phase 3: The Grand Finale (Replenishment)

The last step is to send more jobs back to Tableau Cloud to be processed. Let’s call this file:

tableau_ds_what_is_my_purpose_i_send_jobs_to_tableau_oh_my_god.py.

This file’s purpose is very similar to the kick-off script, with a few key differences:

1. Trigger: It runs immediately after tableau_ds_gotta_check_em_all.py finishes successfully.
2. Dynamic Batch Size: It calculates how many slots have opened up. Logic: $N - (Current_Running_Jobs).
3. Loop: It runs as frequently as you deem appropriate until all rows in tableau_datasource_audit are processed.

Handling Failures My hunch is that once you have this up and running, the failures you see will be a result of things outside Bridge’s control (source DDL changes, credentials, Cloud outages). However, it is still appropriate to build a final script that kicks off once all jobs have been attempted.

This script essentially grabs a list of all jobs with a datasource_failed_timestamp and attempts a single retry. If successfully sent, update the failure_rescheduled_timestamp.

Summary

You made it through!

Building a custom orchestration app to manage Tableau Bridge jobs is a highly effective way to overcome the limitations of a fragmented data architecture. By designing a system with a central source of truth (the audit table), you can move beyond simple scheduling and dynamically manage the workload in real time.

The core components — a Job Initiator, a Status Checker, a Dynamic Job Sender, and a Failure Handler — work together to create a resilient system that brings order to the chaos of Tableau Bridge.