Towards a Unified Airflow: Toss Bank’s PoC for Cluster Consolidation — Part 2

Toss Bank and Apache Airflow

Recap: Last Time

In **Part 1, we explored the strategic reasons why Toss Bank decided to consolidate six clusters and how we separated the platform and business logic environments through ‘Dag Identification’ and ‘Task Env Isolation.’**

But logical isolation is insufficient. How can we prevent one team’s mistake from leading to the physical resource exhaustion of the entire cluster, and how can we build an environment where users can safely execute tasks without infrastructure setup? Part 2 introduces the specific PoC results.

Section 3: The Blueprint of the Unified Airflow Cluster (Part 2)

3.3 k8s Namespace & ResourceQuota: Physical Resource Firewall

Even if the execution environment is isolated by containers, all Pods ultimately run on a single large Kubernetes cluster playground. If a specific team accidentally pushes thousands of tasks simultaneously, a ‘mutual destruction’ scenario can occur, halting even the platform’s core components like the scheduler and API server.

3.3.1) Separation of Core and Workspace: “Separating Admin and User Spaces”

We created physical boundaries using Namespaces.

• airflow-Core Namespace: Only core components like the scheduler and web server are deployed here, protecting them from external interference.
• airflow-Workspace-{Team} Namespace: Each team (DA, DE, ML, etc.) is allocated a dedicated space. All user tasks (KPO Pods, Worker Pods) are created and destroyed only within their team’s Namespace.

Separation of Core and Workspace

To automate this physical isolation, we verified a logic using task_policy to automatically route each team’s task workload to its dedicated Namespace, as follows:

# Automated team-based namespace assignment logic verified in PoC
@hookimpl
def task_policy(task: BaseOperator) -> None:
    # 1. Identify the team (extract prefix) based on the Dag file path
    # Note: _get_team_prefix_fileloc is a custom logic defined previously
    prefix = _get_team_prefix_fileloc(task.dag.fileloc)

    # 2. Retrieve the dedicated namespace for the team from the mapping table
    target_namespace = TEAM_WORKLOAD_NAMESPACE_MAP.get(prefix, "default")

    # 3. Dynamically inject the appropriate namespace based on the execution environment
    if isinstance(task, KubernetesPodOperator):
        # For KubernetesPodOperator, assign the namespace attribute directly
        task.namespace = target_namespace
    else:
        # For KubernetesExecutor, route the execution to the target namespace via pod_override
        task.executor_config = {
            "pod_override": k8s.V1Pod(
                metadata=k8s.V1ObjectMeta(namespace=target_namespace)
            )
        }

This code is a powerful tool that allows the platform operator to forcefully change the destination of a task at execution time without modifying individual Dags. We confirmed that this perfectly routes the tasks launched by each team to execute only within their team’s fence (airflow-Workspace-{Team}), without encroaching upon the central airflow-Core.

3.3.2) ResourceQuota: “Setting the Upper Limit for Team Resource Usage”

We applied a physical limit by setting ResourceQuota on each team’s Namespace. We confirmed that even if a specific team requests excessive resources, it is blocked at the K8s level once the limit is exceeded, acting as a ‘resource firewall’ that does not harm the overall cluster availability.

3.4 Platform Guardrails: Safety Net Ensuring Autonomy

Encouraging users to “write code freely” grants autonomy but also comes with risks. We designed a ‘No-Ops’ experience where the system protects itself even if a user makes a mistake, and the platform automatically handles complex settings.

3.4.1) Pre-Deployment Prevention (GitHub PR CI Test)

The best failure response is preventing problematic code from reaching the scheduler. We can test scenarios like checking for a DagBag Import Error in the CI phase or pre-emptively blocking ‘scheduler killers (e.g., top-level external connections)’. For example, the following code can be run to check for DagBag Import Errors before deployment:

import sys
from airflow.models.dagbag import DagBag
...
dagbag = DagBag(dag_folder=REPO_ROOT, include_examples=False)
if dagbag.import_errors:
    sys.exit(1)
sys.exit(0)

3.4.2) Automated Provisioning & Centralized Infrastructure Governance

To ensure users do not need to know the complex properties of the infrastructure, essential settings are automatically injected (Mutated) based on the identified team information.

• Automatic Injection of Script & Config: Volumes containing execution scripts (PVC) and essential configuration files (ConfigMap/Secrets) are automatically mounted at execution time based on team ownership.
• Dynamic Identity Injection: For security, authentication Keytabs are not managed by the user directly; instead, the platform dynamically injects them into the execution environment based on team ownership. We confirmed that this prevents the exposure of sensitive information and allows strict central control of permissions.

3.4.3) Cluster-Level Safety Net (Cluster Policy — Validation & Skip)

Even after deployment, the Airflow scheduler verifies in real-time whether the Dag complies with internal standards and is suitable for the current cluster environment every time it parses the Dag.

• AirflowClusterPolicyViolation (Enforcing Mandatory Rules): If a Dag violates internal standards, an Import Error is intentionally raised to reject registration.
• Scenario (Email Domain Check): If the alarm recipient email domain is not an authorized domain, such as @tossbank.com, a policy violation error is thrown, fundamentally blocking the deployment of a misconfigured Dag.
• AirflowClusterPolicySkipDag (Selective Activation by Environment): Does not raise an error but silently skips the parsing itself based on specific conditions.
• Scenario (Environment Selection): Dags tagged with only_dev are only parsed normally in the Development (Dev) cluster, and the scheduler ignores them in the Production (Prod) cluster, preventing deployment mistakes between environments.

Section 4: Remaining Challenges — Bottlenecks in ‘Shared Resources’

Despite establishing the isolation strategy, the single-cluster model still faces a ‘shared resource interference’ problem. Although high fences have been built between teams, the core engine supporting the platform is ultimately shared by all teams.

4.1 Bottleneck in the Central Control System: Celery Worker Tracking

Bottleneck — Celery Workers

The business logic is isolated into team-specific Namespaces, but the central Celery Worker group, which commands and tracks their status, remains a company-wide shared resource.

• Instability Factor: If thousands of KPO jobs are launched simultaneously, even though the actual logic runs in each team’s Namespace, a bottleneck can occur in the central message queue that manages them. This results in one team’s job surge delaying the ‘command’ of other teams’ jobs.
• Direction for Solution: To address this, we are reviewing the ‘Multiple Celery Worker Sets’ feature recently introduced in the Airflow Helm Chart (1.19.0+). The goal is to achieve isolation even at the control plane level by assigning dedicated worker groups per team or purpose and allowing them to autoscale independently via KEDA.

4.2 Parsing Bottlenecks and Open Source Contribution: Limitations and Overcoming the Single Dag Processor

Bottleneck — Dag Processor

The most sensitive shared resource is the Dag Parsing area. In the current Airflow structure, a single central Dag Processor sequentially parses all registered teams’ code (Bundles).

• Nature of the Problem: If a specific team deploys code with high parsing load or an error occurs, the entire processor loop is delayed. This directly leads to ‘operational interference,’ specifically a company-wide delay in the Dag update cycle.
• Solving Together with the Open Source Ecosystem: I am directly participating in the open-source architectural improvements to fundamentally resolve this issue.
• Proposal (Issue #61037): Proposed an architectural improvement to physically isolate resources and failure propagation scope by allowing the deployment of independent Dag Processors per bundle.
• Pull Request (PR #61039): To realize this, I implemented and submitted code for the deployPerBundle feature in the Airflow Helm Chart, which allows the creation of independent processor Pods per bundle. It is currently under review.

Conclusion: Integration Preparing for the Future, A Journey Towards Trust

Our PoC journey is not just about solving past problems; it is deeply aligned with the core philosophy of Airflow 3, which has become the standard for modern data orchestration. We are proactively reviewing the standard architecture envisioned by the next generation of Airflow and integrating it into Toss Bank’s environment, realizing the following roadmap:

• AIP-67 (Multi-team deployment of Airflow components): The structural answer for securely providing a single cluster to multiple teams. Internalizing team-specific independent Dag Processor allocation and task routing has become the global standard for the ‘team-specific identification and isolation’ we have pursued.
• AIP-69 (Edge Executor): The foundation for multi-cluster orchestration, maximizing environmental separation beyond the physical and logical limits of a single cluster. It ensures scalability to flexibly execute tasks even in different network environments when combined with the Task SDK.
• AIP-72 (Task Execution Interface — Task SDK): The key technology for completely Decoupling the user’s execution environment from the Airflow Core. This allows the platform team to focus on Core updates, and users to concentrate only on their business logic, achieving ‘true dependency liberation.’

The path from six clusters to a single platform was not just a technical migration; it was a process of preparing in advance for the next generation of Airflow.

Through this PoC, we confirmed how the principles of ‘Identification, Isolation, and Protection’ align with the core roadmap of Airflow 3. Although not every process is perfect and challenges remain, we are finding the answers together, using the open-source ecosystem and the direction of AIPs as our compass.

The Toss Bank Data Platform Team will continue to build a more robust and transparent data ecosystem with Airflow, and we hope our journey serves as a practical reference for fellow engineers navigating the complexities of Airflow multi-tenancy in large-scale organizations.