Optimizing Apache Airflow with Centralized LDAP Authentication

This document serves as an essential resource for establishing a centralized login system for Airflow utilizing LDAP, which is critical for enhancing both security and user management. As organizations increasingly rely on data-driven decision-making, secure access to platforms such as Airflow becomes paramount. LDAP authentication facilitates a seamless login experience while maintaining robust security measures.

This guide describes the necessary configurations and procedural steps required to enable LDAP authentication on the Apache Airflow webserver. By doing so, it promotes efficient access and role management for your team. Additionally, it offers valuable insights for system administrators and developers aiming to optimize their Airflow environment.

Pre-requisites for LDAP Authentication on Apache Airflow Cluster

Before enabling LDAP authentication on your Apache Airflow cluster running on ECS containers, ensure that the following prerequisites are met:

1. Apache Airflow Setup: Ensure that your Apache Airflow cluster is properly set up on ECS containers or platform of your choice, it could be EC2, Azure VM or on-premise host. This includes having the Airflow webserver and scheduler containers running, as well as the Celery executor containers configured for task execution.
2. Python-LDAP Module: The python-ldap module must be installed in the Airflow virtual environment. This module is essential for enabling LDAP authentication. You can install it using pip: pip install python-ldap
3. Configuration of webserver_config.py: You need to update the webserver_config.py file located in your Airflow setup. This file contains the necessary configurations for LDAP authentication. Ensure that you replace the placeholder values with actual values specific to your LDAP setup.
4. Network Access: Ensure that your ECS containers have network access to the LDAP server. This may involve configuring security groups and network ACLs to allow traffic on the required ports.
5. Testing Environment: It is advisable to test the LDAP authentication in a non-production environment before deploying it to production. This helps in identifying any configuration issues without affecting the live system.

By ensuring these prerequisites are met, you can smoothly enable LDAP authentication for your Apache Airflow cluster on ECS containers.

Steps to enable LDAP authentication on LDAP AF main server

To enable LDAP authentication on Airlfow, LDAP auth method is used in webserver_config.py file of Airflow webserver. For more authentication methods and details, please follow Apache’s official documentation.

• Update the content of “webserver_config.py” file on Airflow webserver. This file can usually be found in airflow home (~/airflow).

Replace the parameters with the actual values in the file.

1. <password> : Use the password for an account you’re using for connecting to your ldap server.

2. <LDAP_SEARCH_FILETR> :

(|(memberOf=cn=group1,ou=Universal Groups,ou=Groups,dc=example,dc=com)
(memberOf=cn=group2,ou=Universal Groups,ou=Groups,dc=example,dc=com)
(memberOf=cn=group3,ou=Universal Groups,ou=Groups,dc=example,dc=com)
(memberOf=cn=group4,ou=Universal Groups,ou=Groups,dc=example,dc=com))

3. <Auth_roles_mapping>:

{
 "CN=group1,OU=Universal Groups,OU=Groups,DC=example,DC=com": ["Admin"],
 "CN=group2,OU=Universal Groups,OU=Groups,DC=example,DC=com": ["User"]
}

Sample webserver_config.py

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
"""Default configuration for the Airflow webserver"""
import os

#from flask_appbuilder.security.manager import AUTH_DB

from flask_appbuilder.security.manager import AUTH_LDAP
# from flask_appbuilder.security.manager import AUTH_OAUTH
# from flask_appbuilder.security.manager import AUTH_OID
# from flask_appbuilder.security.manager import AUTH_REMOTE_USER

basedir = os.path.abspath(os.path.dirname(__file__))

# Flask-WTF flag for CSRF
WTF_CSRF_ENABLED = True

# ----------------------------------------------------
# AUTHENTICATION CONFIG
# ----------------------------------------------------
# For details on how to set up each of the following authentication, see
# http://flask-appbuilder.readthedocs.io/en/latest/security.html# authentication-methods
# for details.

# The authentication type
# AUTH_OID : Is for OpenID
# AUTH_DB : Is for database
# AUTH_LDAP : Is for LDAP
# AUTH_REMOTE_USER : Is for using REMOTE_USER from web server
# AUTH_OAUTH : Is for OAuth

# Uncomment to setup Full admin role name
# AUTH_ROLE_ADMIN = 'Admin'

# Uncomment to setup Public role name, no authentication needed
# AUTH_ROLE_PUBLIC = 'Public'

# self registration
# AUTH_USER_REGISTRATION = True

# The recaptcha it's automatically enabled for user self registration is active and the keys are necessary
# RECAPTCHA_PRIVATE_KEY = PRIVATE_KEY
# RECAPTCHA_PUBLIC_KEY = PUBLIC_KEY

# Config for Flask-Mail necessary for user self registration
# MAIL_SERVER = 'smtp.gmail.com'
# MAIL_USE_TLS = True
# MAIL_USERNAME = 'yourappemail@gmail.com'
# MAIL_PASSWORD = 'passwordformail'
# MAIL_DEFAULT_SENDER = 'sender@gmail.com'

# The default user self
# AUTH_USER_REGISTRATION_ROLE = "Public"

# When using OAuth Auth, uncomment to setup provider(s) info
# Google OAuth example:
# OAUTH_PROVIDERS = [{
#   'name':'google',
#     'token_key':'access_token',
#     'icon':'fa-google',
#         'remote_app': {
#             'api_base_url':'https://www.googleapis.com/oauth2/v2/',
#             'client_kwargs':{
#                 'scope': 'email profile'
#             },
#             'access_token_url':'https://accounts.google.com/o/oauth2/token',
#             'authorize_url':'https://accounts.google.com/o/oauth2/auth',
#             'request_token_url': None,
#             'client_id': GOOGLE_KEY,
#             'client_secret': GOOGLE_SECRET_KEY,
#         }
# }]

# When using LDAP Auth, setup the ldap server
AUTH_TYPE = AUTH_LDAP
AUTH_LDAP_SERVER = "ldaps://xyz.example.com:3269"
AUTH_LDAP_USE_TLS = False

# registration configs
AUTH_USER_REGISTRATION = True  # allow users who are not already in the FAB DB
AUTH_USER_REGISTRATION_ROLE = "User"  # this role will be given in addition to any AUTH_ROLES_MAPPING
AUTH_LDAP_FIRSTNAME_FIELD = "givenName"
AUTH_LDAP_LASTNAME_FIELD = "sn"
AUTH_LDAP_EMAIL_FIELD = "mail"  # if null in LDAP, email is set to: "{username}@email.notfound"

AUTH_LDAP_SEARCH = "dc=example,dc=com"  # the LDAP search base
AUTH_LDAP_SEARCH_FILTER="(|(memberOf=cn=group1,ou=Universal Groups,ou=Groups,dc=example,dc=com)(memberOf=cn=group2,ou=Universal Groups,ou=Groups,dc=example,dc=com)(memberOf=cn=group3,ou=Universal Groups,ou=Groups,dc=example,dc=com)(memberOf=cn=group4,ou=Universal Groups,ou=Groups,dc=example,dc=com))"
AUTH_LDAP_UID_FIELD = "cn"  # the username ("common name") field
AUTH_LDAP_BIND_USER = "cn=BindUser,ou=Domain Accounts,dc=example,dc=com"
AUTH_LDAP_BIND_PASSWORD = "<password>"  # the special bind password for search

# a mapping from LDAP DN to a list of FAB roles
# NOTE: Values are case sensitive, use the exact DN string as it is in LDAP
AUTH_ROLES_MAPPING = {
 "CN=group1,OU=Universal Groups,OU=Groups,DC=example,DC=com": ["Admin"],
 "CN=group2,OU=Universal Groups,OU=Groups,DC=example,DC=com": ["User"]
}

# the LDAP user attribute which has their role DNs
AUTH_LDAP_GROUP_FIELD = "memberOf"

# if we should replace ALL the user's roles each login, or only on registration
AUTH_ROLES_SYNC_AT_LOGIN = True

# force users to re-auth after 30min of inactivity (to keep roles in sync)
PERMANENT_SESSION_LIFETIME = 1800

# When using OpenID Auth, uncomment to setup OpenID providers.
# example for OpenID authentication
# OPENID_PROVIDERS = [
#    { 'name': 'Yahoo', 'url': 'https://me.yahoo.com' },
#    { 'name': 'AOL', 'url': 'http://openid.aol.com/<username>' },
#    { 'name': 'Flickr', 'url': 'http://www.flickr.com/<username>' },
#    { 'name': 'MyOpenID', 'url': 'https://www.myopenid.com' }]

# ----------------------------------------------------
# Theme CONFIG
# ----------------------------------------------------
# Flask App Builder comes up with a number of predefined themes
# you can use for Apache Airflow.
# http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes
# Please make sure to remove "navbar_color" configuration from airflow.cfg
# in order to fully utilize the theme. (or use that property in conjunction with theme)
# APP_THEME = "bootstrap-theme.css"  # default bootstrap
# APP_THEME = "amelia.css"
# APP_THEME = "cerulean.css"
# APP_THEME = "cosmo.css"
# APP_THEME = "cyborg.css"
# APP_THEME = "darkly.css"
# APP_THEME = "flatly.css"
#.css"
# APP_THEME = "lumen.css"
# APP_THEME = "paper.css"
# APP_THEME = "readable.css"
# APP_THEME = "sandstone.css"
# APP_THEME = "simplex.css"
# APP_THEME = "slate.css"
# APP_THEME = "solar.css"
# APP_THEME = "spacelab.css"
# APP_THEME = "superhero.css"
# APP_THEME = "united.css"
# APP_THEME = "yeti.css"

• After updating the webserver_config.py file, restart your Airflow services to enable LDAP authentication for your users. If you're running Airflow in non-root mode on ECS, recreate your container to refresh the services.

By following the steps outlined in this document, you have successfully configured Airflow to enable LDAP authentication. This setup ensures secure and streamlined user management, leveraging your existing LDAP directory.