GCP-PDE — The Pipeline That Runs Itself — Designing Automation and Repeatability (Section 5.2)

A pipeline that needs a human to press start every morning isn’t automated. It’s just convenient manual work.

I once inherited a data platform where the “automated” nightly pipeline had a sticky note on the on-call engineer’s monitor that said “remember to trigger the Spark job at midnight.” The job had been running this way for eight months. The engineer who wrote the sticky note had left the company. Their replacement followed the note dutifully without understanding what the job did or why it needed to be triggered manually.

That’s not automation. That’s human-dependent scheduling with extra steps.

True automation means the pipeline runs correctly, every time, without anyone pressing a button — and when something goes wrong, it tells the right person rather than silently failing until an executive notices a missing dashboard number.

Section 5.2 is the exam’s way of asking: can you design Cloud Composer DAGs and scheduling patterns that make Cymbal’s pipelines self-operating, self-healing, and self-documenting — so that no sticky note is ever needed?

We’ll follow Cymbal’s Pipeline Automation Initiative — the project to convert twelve manually triggered batch pipelines into fully automated, monitored, and repeatable workflows that run without human intervention.

01 — The Head Chef’s Recipe Book: Creating DAGs for Cloud Composer

A DAG (Directed Acyclic Graph) is the recipe that Cloud Composer follows. It defines which tasks need to run, in what order, under what conditions, with what retry behaviour, and what happens when things go wrong. Writing a good DAG is equal parts engineering and operational thinking — you’re not just describing what happens when everything goes right, you’re designing for every failure mode too.

Think of a DAG like a flight plan. The pilot doesn’t decide the route in the air — the route is planned on the ground, validated, filed with air traffic control, and then followed. When something unexpected happens (weather, a closed runway), the plan has contingencies. A well-designed DAG is the flight plan for your data pipeline.

DAG anatomy — the essential elements

Every Cloud Composer DAG is a Python file that defines:

• DAG object — the container for all tasks and configuration. Defines the schedule, start date, catchup behaviour, default arguments, and tags.
• Tasks — individual units of work. Each task is an instance of an operator — Dataflow, BigQuery, Dataform, Bash, Python, Email, etc.
• Dependencies — the arrows in the graph. task_b.set_upstream(task_a) means task B only runs after task A succeeds. Equivalently: task_a >> task_b.
• Trigger rules — conditions under which a task runs relative to its upstream tasks.

A complete DAG example — Cymbal’s nightly order pipeline

from airflow import DAG
from airflow.providers.google.cloud.operators.dataflow import DataflowStartFlexTemplateOperator
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from airflow.providers.google.cloud.operators.dataform import DataformCreateCompilationResultOperator
from airflow.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensor
from airflow.operators.email import EmailOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'data-engineering',
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
    'retry_exponential_backoff': True,
    'email_on_failure': True,
    'email': ['data-alerts@cymbal.com'],
    'on_failure_callback': alert_slack,
}

with DAG(
    dag_id='cymbal_nightly_order_pipeline',
    schedule_interval='0 1 * * *',  # 1 AM daily
    start_date=datetime(2025, 1, 1),
    catchup=False,
    default_args=default_args,
    tags=['orders', 'nightly', 'critical'],
    max_active_runs=1,
) as dag:

    wait_for_source_file = GCSObjectExistenceSensor(
        task_id='wait_for_source_file',
        bucket='cymbal-raw-orders',
        object='daily/{{ ds }}/orders.parquet',
        timeout=3600,
        poke_interval=60,
    )

    run_dataflow_cleanse = DataflowStartFlexTemplateOperator(
        task_id='run_dataflow_cleanse',
        body={'launchParameter': {
            'jobName': 'order-cleanse-{{ ds_nodash }}',
            'containerSpecGcsPath': 'gs://cymbal-templates/order-cleanse-latest',
            'parameters': {'processing_date': '{{ ds }}'},
        }},
        project_id='cymbal-data-prod',
        location='europe-west1',
    )

    run_dataform = DataformCreateCompilationResultOperator(
        task_id='run_dataform_transform',
        project_id='cymbal-data-prod',
        region='europe-west1',
        repository_id='cymbal-transforms',
    )

    validate_row_count = BigQueryInsertJobOperator(
        task_id='validate_row_count',
        configuration={
            'query': {
                'query': '''
                    SELECT IF(COUNT(*) > 0, 'PASS', 'FAIL') AS result
                    FROM `cymbal.orders.orders_curated`
                    WHERE DATE(order_timestamp) = '{{ ds }}'
                ''',
                'useLegacySql': False,
            }
        },
    )

    notify_success = EmailOperator(
        task_id='notify_success',
        to='analytics@cymbal.com',
        subject='Nightly Order Pipeline Complete — {{ ds }}',
        html_content='Pipeline completed successfully for {{ ds }}.',
    )

    # Dependency chain
    wait_for_source_file >> run_dataflow_cleanse >> run_dataform >> validate_row_count >> notify_success

Key DAG configuration decisions the exam tests

• **schedule_interval* — when the DAG runs. Accepts cron expressions (`'0 1 '= 1 AM daily), presets ('@daily','@hourly','@weekly'), orNone` for manually-triggered only.
• **catchup** — when True, Airflow runs all missed intervals since start_date. With a start date of January 1st and catchup=True, activating the DAG in April triggers 90 daily runs immediately. Set catchup=False for new DAGs unless historical backfill is explicitly needed — the exam loves the catchup trap.
• **max_active_runs** — maximum number of DAG runs that can execute simultaneously. Set to 1 for pipelines that should not overlap — prevents two nightly runs executing at the same time if one runs late.
• **default_args** — default settings applied to all tasks. Retries, retry delay, email on failure, owner. Tasks can override individual defaults.
• **tags** — labels for organising DAGs in the Airflow UI. Critical for finding the right DAG in a large deployment.

Template variables — the exam’s most tested DAG feature

• Airflow’s Jinja templating engine is available in all operator parameters that support templates.
• {{ ds }} — the DAG run's logical date in YYYY-MM-DD format. For the 1 AM daily run on June 15th, {{ ds }} is 2025-06-14 (Airflow uses the period start date, not the execution date).
• {{ ds_nodash }} — the same date without dashes: 20250614. Used for naming files and BigQuery jobs.
• {{ execution_date }} — the full execution datetime as a Python pendulum.DateTime object.
• {{ next_ds }} — the next scheduled run's logical date.
• {{ prev_ds }} — the previous run's logical date.
• Why templates matter for the exam: Cymbal’s pipeline processes “yesterday’s orders” on each run. Without templates, you’d hardcode a date that becomes wrong immediately. With {{ ds }}, each run automatically processes its own logical date — making the pipeline idempotent and repeatable.

Operators the exam tests

• **DataflowStartFlexTemplateOperator** — submits a Dataflow Flex Template job and optionally waits for completion. The async version (do_xcom_push=True) returns the job ID and lets a DataflowJobStateSensor wait for completion separately.
• **BigQueryInsertJobOperator** — executes a BigQuery SQL query or load job. Used for validation queries, transformation SQL, or loading data from Cloud Storage.
• **DataformCreateCompilationResultOperator and `DataformCreateWorkflowInvocationOperator`** — compile and execute Dataform workspace SQL models.
• **DataprocCreateClusterOperator / `DataprocSubmitJobOperator** /DataprocDeleteClusterOperator` — the ephemeral cluster lifecycle trio.
• **GCSObjectExistenceSensor** — polls Cloud Storage for a file's existence. The DAG waits until the file appears before proceeding.
• **DataflowJobStateSensor** — polls a Dataflow job until it reaches a specified state (JOB_STATE_DONE, JOB_STATE_FAILED).
• **BigQueryTableExistenceSensor** — waits until a BigQuery table exists.
• **BranchPythonOperator** — executes a Python function that returns the task_id(s) to run next. Used for conditional branching — "if today is Monday, run the weekly summary; otherwise skip it."
• **TriggerDagRunOperator** — triggers another DAG from within a DAG. Used for cross-DAG dependencies.

Trigger rules — controlling task execution conditions

• **ALL_SUCCESS** (default) — runs only if all upstream tasks succeeded. The standard sequential dependency.
• **ALL_DONE** — runs when all upstream tasks have completed, regardless of success or failure. Used for notification tasks that should always fire — "send completion report whether the pipeline succeeded or failed."
• **ONE_FAILED** — runs if at least one upstream task failed. Used for failure-specific alerting.
• **ONE_SUCCESS** — runs if at least one upstream task succeeded. Used when parallel branches feed into a merger and only one branch needs to succeed.
• **NONE_FAILED** — runs if no upstream tasks failed (they may have been skipped). More permissive than ALL_SUCCESS.

XComs — passing data between tasks

• XComs allow tasks to share small values within a DAG run. A row count from a validation task, a job ID from a Dataflow submission, a flag from a conditional check.
• Task A: context['task_instance'].xcom_push(key='row_count', value=count) or simply return count (Airflow auto-pushes the return value as return_value).
• Task B: context['task_instance'].xcom_pull(task_ids='task_a', key='row_count').
• XComs are stored in the Airflow metadata database — keep them small (a few KB maximum). Never push large DataFrames or entire datasets through XComs.

💡 Exam Tip:

“DAG activates and triggers 90 historical runs” → **catchup=True trap — set catchup=False**

“Each DAG run processes its own date automatically” → **{{ ds }} Jinja template**

“Task runs whether upstream succeeded or failed” → **trigger_rule=ALL_DONE**

“Task runs only if an upstream task failed” → **trigger_rule=ONE_FAILED**

“Pass a row count from validation task to alerting task” → XComs

“Wait for a Cloud Storage file before proceeding” → GCSObjectExistenceSensor

“Wait for a Dataflow job to complete” → DataflowJobStateSensor

“Conditional branching in a DAG” → BranchPythonOperator

“Trigger another DAG from within a DAG” → TriggerDagRunOperator

02 — Running Like Clockwork: Scheduling and Orchestrating Repeatable Jobs

Scheduling is about when — when does the pipeline run, how often, and what triggers it. Orchestration is about how — in what order, with what dependencies, with what retry behaviour. Both must be designed for repeatability: the ability to run the same pipeline 365 times a year and get consistent, correct results every time.

Think of scheduling and orchestration like a train timetable. The timetable defines when trains run. The signal system (orchestration) ensures trains don’t collide — Train A clears the section before Train B enters. The combination of schedule + signal creates a reliable, repeatable transportation system. Without the signal system, a timetable alone is just an optimistic list of departure times.

Scheduling patterns in Cloud Composer

• Time-based scheduling — the most common pattern. A cron expression defines the exact schedule.
• '0 2 * * *' — 2 AM every day.
• '0 6 * * 1' — 6 AM every Monday.
• '0 0 1 * *' — midnight on the first day of every month.
• '*/15 * * * *' — every 15 minutes.
• Airflow uses UTC by default. Be explicit about timezone in the DAG definition when business SLAs are expressed in local time.
• Event-driven scheduling — a pipeline triggered by an event rather than a clock.
• Cloud Storage file arrival — Eventarc routes a GCS object.finalized event to a Cloud Run job or Cloud Function that calls the Airflow REST API to trigger a DAG run.
• Pub/Sub message — a Pub/Sub message triggers a Cloud Function that triggers a DAG run. Used when an upstream system signals completion.
• Dataset-aware scheduling — Airflow 2.4+ supports Dataset-aware DAGs. A DAG can declare that it depends on a Dataset (an abstract data asset). When another DAG writes to that Dataset, the dependent DAG is automatically triggered — no manual dependency wiring needed.
• Manual triggering — some pipelines should never run automatically. Historical reprocessing jobs, one-off data corrections, and emergency procedures. schedule_interval=None configures a DAG to run only when manually triggered via the UI, CLI, or REST API.

Idempotency — the foundation of repeatability

• Every pipeline step must be safe to run more than once. If a step runs twice (due to retry, catchup, or manual re-trigger), the result must be identical.
• Why it matters: retries are expected. Catchup creates multiple runs. Operators can fail mid-task and retry. A non-idempotent pipeline produces duplicates or corrupted state on retry.
• BigQuery patterns for idempotency:
• CREATE OR REPLACE TABLE — replaces the table on every run. The second run produces the same result as the first.
• TRUNCATE + INSERT — truncates the table before inserting. Equivalent to CREATE OR REPLACE for append pipelines.
• MERGE — upserts based on a unique key. Safe to run multiple times — duplicates are updated, not re-inserted.
• Partitioned inserts with WRITE_TRUNCATE disposition — overwrites a specific partition. Running twice overwrites the same partition with the same data.
• Avoid: INSERT INTO without a deduplication check — running twice doubles the rows.
• Dataflow patterns for idempotency:
• Name Dataflow jobs with the logical date: order-cleanse-{{ ds_nodash }}. A job with the same name cannot be submitted twice simultaneously — the second attempt fails with a name collision, preventing double-processing.
• Use WRITE_TRUNCATE in BigQuery IO transforms for the final write — the last run wins, no accumulation.

Dependency management across DAGs

• Complex data platforms have dozens of DAGs — some with interdependencies. The nightly customer DAG must complete before the weekly cohort analysis DAG can run.
• **TriggerDagRunOperator** — a task in DAG A that triggers DAG B when DAG A reaches that point. Simple and explicit — the dependency is visible in DAG A's task graph.
• **ExternalTaskSensor** — a task in DAG B that waits for a specific task in DAG A to reach a success state before proceeding. The dependency is managed from DAG B's side.
• Dataset-aware scheduling — DAG A declares that it produces Dataset orders_curated. DAG B declares that it depends on Dataset orders_curated. When DAG A completes, Airflow automatically triggers DAG B. Decoupled, declarative, self-documenting.
• Which to use: TriggerDagRunOperator is push-based (A controls when B runs). ExternalTaskSensor is pull-based (B controls when it checks A). Dataset-aware is event-based (the data asset drives the trigger). Dataset-aware is the most modern pattern and is increasingly tested in the 2026 exam.

Retry and failure handling

• Every production DAG must have a clear retry and failure strategy. What happens when a Dataflow job fails? How many times does it retry? What’s the delay between retries? Who gets notified?
• Retry configuration:
• retries=3 — retry the task up to 3 times before marking it as failed.
• retry_delay=timedelta(minutes=5) — wait 5 minutes between retries.
• retry_exponential_backoff=True — double the wait time on each retry (5 min, 10 min, 20 min). Prevents thundering herd when a downstream service is overloaded.
• max_retry_delay=timedelta(hours=1) — cap the maximum retry delay.
• Failure callbacks:
• on_failure_callback — a Python function called when a task fails. Cymbal's pipelines call a function that sends a Slack message with the DAG ID, task ID, and log URL.
• on_success_callback — called on task success. Used for audit logging and metrics.
• on_retry_callback — called on each retry attempt. Used for escalating alert frequency.
• SLA miss callbacks:
• sla=timedelta(hours=4) on a task — if the task hasn't completed within 4 hours of the DAG run's start, an SLA miss is recorded and the sla_miss_callback fires. The nightly pipeline's validation task has a 4-hour SLA — if it hasn't completed by 5 AM, the data operations team is alerted before the 6 AM executive dashboard loads.

Deploying and updating DAGs

• Cloud Composer environments mount a Cloud Storage bucket as the DAGs folder. Files placed in the dags/ subfolder are automatically loaded by Composer within a few minutes.
• CI/CD for DAGs — from Section 2.3: Cloud Build deploys DAG files to the Composer GCS bucket automatically on merge to main. Tests run first: DAG import validation (does the Python file parse without errors?), circular dependency check, operator configuration validation.
• DAG versioning — use semantic versioning in the DAG ID or filename when making breaking changes. cymbal_nightly_order_pipeline_v2 runs alongside cymbal_nightly_order_pipeline during the transition period — migrate and decommission cleanly.

Cloud Workflows for simple automation

• For orchestration scenarios that don’t need the full complexity of Airflow — no sensors, no complex dependency graphs, no backfill — Cloud Workflows provides a simpler serverless alternative.
• Cloud Workflows is YAML/JSON-based, serverless (pay per execution), and integrates natively with GCP APIs.
• Use Cloud Workflows when: the workflow is a linear sequence of API calls (fetch from API → write to Cloud Storage → trigger BigQuery load → send notification), the team wants zero infrastructure management, and the workflow doesn’t need Airflow’s operator library, sensor polling, or backfill.
• Use Cloud Composer when: the workflow has complex dependencies, branching logic, sensor-based waiting, cross-DAG dependencies, or backfill requirements.

💡 Exam Tip:

“Pipeline run processes its own date, safe to re-run” → **{{ ds }} template + idempotent writes (MERGE or WRITE_TRUNCATE)**

“Avoid double-inserting on retry” → **WRITE_TRUNCATE or MERGE in BigQuery writes**

“Trigger DAG B when DAG A completes” → TriggerDagRunOperator or ExternalTaskSensor or Dataset-aware scheduling

“Dataset-aware DAG trigger” → DAG A produces a Dataset; DAG B depends on it; Airflow triggers B when A writes to the Dataset

“Alert if pipeline hasn’t completed by a specific time” → SLA miss callback

“Simple linear API call sequence, serverless” → Cloud Workflows

“Complex dependencies, sensors, backfill” → Cloud Composer

“Deploy DAG file automatically on code merge” → Cloud Build copies to Composer GCS bucket

The exam will describe a scheduling or orchestration scenario and ask which tool or pattern is correct — match the complexity, trigger type, and dependency structure to the right answer.

Practice Questions

Q1 — Catchup Trap

Cymbal creates a new Cloud Composer DAG on April 1st with start_date=datetime(2025, 1, 1), schedule_interval='0 2 * * *' (daily at 2 AM), and catchup=True. The DAG is activated immediately. What happens?

• A. The DAG runs once for April 1st and schedules future runs
• B. The DAG immediately queues approximately 90 runs — one for each day from January 1st to March 31st — causing a backlog that can overwhelm the Composer environment
• C. The DAG fails to activate because the start_date is in the past
• D. The DAG runs for the most recent 7 days only — Airflow limits catchup to 7 intervals

Answer: B

• catchup=True with a start_date 90 days in the past triggers 90 historical DAG runs immediately on activation. This can exhaust Composer worker capacity, fill the metadata database, and delay other DAGs. Fix: set catchup=False unless historical backfill is explicitly required, and set start_date to a recent date when creating new DAGs.

Q2 — Jinja Templates for Idempotency

Cymbal’s nightly Dataflow pipeline processes orders from the previous day. Currently the processing date is hardcoded as '2025-06-14' in the pipeline parameters. When the date changes, an engineer manually updates the DAG. What is the correct automation pattern?

• A. Use a Cloud Scheduler job to update the hardcoded date every night
• B. Use Airflow’s {{ ds }} template variable in the pipeline parameter — each DAG run automatically uses its own logical date without any manual changes
• C. Use a Python datetime.today() call in the DAG definition — computes the current date dynamically
• D. Store the processing date in an Airflow Variable and update it manually before each run

Answer: B

• {{ ds }} is evaluated at task execution time to the DAG run's logical date. Each nightly run automatically processes its own date — no manual updates, no Cloud Scheduler, no human intervention. datetime.today() in the DAG definition (C) is evaluated at DAG parse time (when Airflow loads the file) — it would capture the date when the DAG was last loaded, not the run date. Manual Variable updates (D) reintroduces human dependency — the exact problem being solved.

Q3 — Trigger Rules

Cymbal’s DAG has five tasks in sequence. Task 3 (a Dataflow job) fails. Tasks 4 and 5 must be skipped automatically. However, Task 6 — a notification task that sends a pipeline status report — must always run, regardless of whether tasks 3, 4, and 5 succeeded or failed. Which trigger rule is correct for Task 6?

• A. trigger_rule='ALL_SUCCESS' — runs only if all upstream tasks succeeded
• B. trigger_rule='ONE_FAILED' — runs only if at least one upstream task failed
• C. trigger_rule='ALL_DONE' — runs when all upstream tasks have completed in any state (success, failure, or skipped)
• D. trigger_rule='NONE_FAILED' — runs if no upstream tasks failed

Answer: C

• ALL_DONE fires when all upstream tasks have reached a terminal state — success, failure, or skipped. Task 6 runs whether the pipeline succeeded completely or failed at task 3. ALL_SUCCESS (A) would skip Task 6 if Task 3 failed. ONE_FAILED (B) only fires when a task fails — wouldn't run on success. NONE_FAILED (D) is more permissive than ALL_SUCCESS (allows skipped) but still fails to fire when an upstream task failed.

Q4 — Cross-DAG Dependency

Cymbal has two DAGs: orders_daily (runs nightly, produces curated order data) and cohort_analysis (runs nightly, must process only after orders_daily completes). The cohort analysis has been running on a fixed 3 AM schedule, but sometimes orders_daily overruns and cohort analysis starts before the orders data is ready. Which pattern resolves the timing dependency most reliably?

• A. Schedule cohort_analysis to run at 4 AM — giving orders_daily an extra hour of buffer
• B. Use ExternalTaskSensor in cohort_analysis — it waits for a specific task in orders_daily to reach success state before cohort_analysis proceeds
• C. Merge both DAGs into one — eliminating the cross-DAG dependency entirely
• D. Add a 60-minute TimeDeltaSensor at the start of cohort_analysis

Answer: B

• ExternalTaskSensor polls orders_daily until the specified task reaches success. cohort_analysis only proceeds when orders_daily has genuinely completed — no fixed time buffer needed. A time buffer (A) is fragile — if orders_daily takes even longer, the timing issue recurs. Merging DAGs (C) creates a single monolithic workflow that's harder to manage, monitor, and retry independently. A fixed time delta sensor (D) is the same problem as option A — it waits a fixed amount of time, not until actual completion.

Q5 — Idempotent Pipeline Design

Cymbal’s Dataform transformation task writes results to a BigQuery table using INSERT INTO orders_summary SELECT .... The task is configured with retries=3. On a Thursday run, the task fails after writing 50,000 rows, then retries successfully — writing another 50,000 rows. The table now has 100,000 rows instead of 50,000. What is the correct fix?

• A. Set retries=0 — disable retries to prevent duplicate writes
• B. Add a deduplication query that runs after every INSERT to remove duplicate rows
• C. Replace INSERT INTO with CREATE OR REPLACE TABLE AS SELECT or use a BigQuery write disposition of WRITE_TRUNCATE — the retry overwrites the same result rather than appending duplicates
• D. Use a unique constraint on the target table to reject duplicate rows

Answer: C

• CREATE OR REPLACE TABLE is idempotent — running it twice produces the same final table. WRITE_TRUNCATE in a BigQuery load or query write truncates the target before writing. Either approach means the retry produces the correct 50,000-row result, not 100,000. Setting retries to 0 (A) removes fault tolerance — a transient failure now fails the pipeline permanently. A post-run deduplication query (B) adds complexity and still has a window where duplicates exist. BigQuery doesn't support unique constraints that reject duplicate rows (D).

Q6 — Dataset-Aware Scheduling

Cymbal’s inventory_update DAG produces a curated inventory dataset in BigQuery every night. Three downstream DAGs — demand_forecast, store_allocation, and executive_report — all need to run after inventory_update completes, but they currently each use ExternalTaskSensor pointed at inventory_update. The team wants a cleaner, more decoupled approach that automatically triggers all three when the inventory data is ready, without modifying the downstream DAGs each time a new consumer is added. Which Airflow feature achieves this?

• A. TriggerDagRunOperator in inventory_update — explicitly trigger each downstream DAG
• B. Dataset-aware scheduling — inventory_update declares it produces the inventory_curated Dataset; demand_forecast, store_allocation, and executive_report each declare they depend on inventory_curated; Airflow automatically triggers all three when inventory_update writes to the Dataset
• C. Merge all four DAGs into one large DAG with a parallel branch structure
• D. Use Cloud Scheduler to trigger all four DAGs at fixed times with sufficient gaps

Answer: B

• Dataset-aware scheduling decouples producers from consumers. inventory_update declares it produces a Dataset — it doesn't know or care which DAGs consume it. Each consumer declares its dependency on the Dataset — they're triggered automatically when data is ready. Adding a new consumer (a fourth downstream DAG) requires only declaring its Dataset dependency — inventory_update needs no changes. TriggerDagRunOperator (A) requires modifying inventory_update every time a new consumer is added — tight coupling. Merging DAGs (C) creates a monolithic workflow. Fixed-time scheduling (D) reintroduces the timing fragility problem.

Q7 — SLA Miss

Cymbal’s nightly pipeline must have curated order data ready in BigQuery by 6 AM for the executive dashboard. The pipeline DAG runs at 1 AM. The data operations team needs to be automatically alerted at 5:30 AM if the pipeline hasn’t completed — giving them 30 minutes to investigate before the dashboard load. Which Airflow feature achieves this?

• A. Set retry_delay=timedelta(hours=4.5) on the final task — it will fail and alert at 5:30 AM
• B. Configure sla=timedelta(hours=4, minutes=30) on the final pipeline task — if it hasn't completed 4.5 hours after the DAG run started (5:30 AM), an SLA miss fires and the sla_miss_callback alerts the team
• C. Create a separate Cloud Monitoring alert that checks if the BigQuery table exists at 5:30 AM
• D. Set execution_timeout=timedelta(hours=4, minutes=30) on the DAG — the DAG fails at 5:30 AM

Answer: B

• Airflow’s SLA mechanism is designed exactly for this: sla=timedelta(hours=4, minutes=30) on the final task means if the task hasn't succeeded within 4.5 hours of the DAG run's start time (1 AM + 4.5 hours = 5:30 AM), an SLA miss is recorded and the sla_miss_callback fires — alerting the team proactively. retry_delay (A) controls time between retries, not SLA alerting. A separate Cloud Monitoring alert (C) works but duplicates functionality Airflow provides natively. execution_timeout (D) kills the task at the timeout — that's a failure, not an SLA alert.

Q8 — Cloud Composer vs Cloud Workflows

Cymbal needs to automate two new workflows. Workflow X: daily import from a Marketing SaaS API → write response JSON to Cloud Storage → trigger BigQuery load → send Slack notification on completion. Three linear steps, no dependencies on other DAGs, no file sensing, no branching. Workflow Y: nightly data quality pipeline that waits for 5 different source files to arrive in Cloud Storage (via sensors), runs validation tasks in parallel, branches based on quality scores, triggers a Dataform compilation only if all validations pass, and sends different alerts depending on which validations failed. Which tool is correct for each?

• A. Cloud Composer for both — it handles all workflow types
• B. Cloud Workflows for Workflow X (simple linear API sequence, serverless, minimal overhead); Cloud Composer for Workflow Y (sensors, parallel tasks, conditional branching, complex dependencies)
• C. Cloud Workflows for both — simpler and cheaper than Composer
• D. Cloud Composer for Workflow X; Cloud Workflows for Workflow Y

Answer: B

• Workflow X is exactly the Cloud Workflows use case: 3–4 linear API calls, serverless, no sensors, no branching, pay-per-execution. Cloud Workflows handles this natively in YAML without any Composer infrastructure. Workflow Y requires sensors (wait for files), parallel execution, conditional branching (BranchPythonOperator), and complex failure alerting — these are Composer/Airflow strengths that Cloud Workflows doesn’t natively support. Cloud Composer for all (A) over-engineers Workflow X. Cloud Workflows for all © can’t express sensors, parallelism, or the branching logic needed for Workflow Y. Option D swaps the correct assignments.

A Final Reflection

The sticky note on the monitor wasn’t a failure of technology. It was a failure of design — someone built a pipeline that needed a human in the loop by default, rather than designing automation as a first-class requirement from the start.

What I’ve come to appreciate about automation design is that it’s really about trust. A pipeline that runs itself earns trust through consistent, correct, repeatable execution. A pipeline that needs a human to press start every morning is a pipeline that the business can’t fully trust — because the human might forget, or be sick, or leave and put a sticky note on the monitor for the next person.

Cymbal’s Pipeline Automation Initiative didn’t just remove the sticky notes. It built a system where the pipelines are self-describing (tags, documentation), self-operating (cron schedules, event-driven triggers), self-healing (retries with exponential backoff, SLA alerts), and self-validating (row count checks, quality gates in the DAG).

That’s what automation really means.

Section 5.3 covers Organising Workloads — BigQuery Editions, slot reservations, and interactive vs batch workload management.