📩 MWAA and SES for Sending Email Notifications without IAM Access Keys: A Step-by-Step Guide

Airflow on AWS (MWAA)

Introduction

Airflow is a powerful tool for orchestrating data workflows, and sending notifications are crucial to any automated pipeline. In this tutorial, we’ll walk through the process of configuring MWAA Airflow to send emails using AWS Simple Email Service (SES). By the end of this guide, you’ll be able to trigger email notifications directly from your Airflow workflows without the need to configure credentials on MWAA, by directly using MWAA’s role. This can be especially useful for notifying teams of task completion, failures, or other important events.

We’ll cover how to:

• Set up the correct permissions in AWS.
• Configure Airflow to send emails via SES.
• Write an Airflow DAG to send email notifications.

Why AWS SES for Airflow Emails?

AWS Simple Email Service (SES) is a cost-effective, scalable service for sending transactional and marketing emails. Integrating SES with Airflow provides a seamless way to send email notifications without the need for external services or complex configurations. Plus, AWS’s security and monitoring features ensure reliable email delivery.

🔑 Step 1: Configure AWS Permissions

To allow Airflow to send emails via SES, you need to grant the correct permissions to the Managed Workflows for Apache Airflow (MWAA) role in AWS. This step involves creating an AWS Identity and Access Management (IAM) policy that gives SES permission to send emails.

Here’s how you can do that:

1. Navigate to the IAM service in your AWS Management Console.
2. Locate and select the role used by your MWAA environment.
3. Attach the following policy to grant the necessary SES permissions.

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Effect":"Allow",
      "Action":[
        "ses:SendEmail",
        "ses:SendRawEmail"
      ],
      "Resource":"arn:aws:ses:[region]:[account_id]:identity/[domain.com]"
    }
  ]
}

This policy allows SES to send both standard and raw emails. Adjust the region account ID and domain according to your environment. If not sure, just for the testing set *Resource: “”.**

⚙️ Step 2: Configure Airflow to Use SES as the Email Backend

Next, you must configure Airflow to use AWS SES as the email backend. In your Airflow configuration (such as in airflow.cfg or your environment variables), set the email backend to SES, and specify the correct "from" email address. This has been documented on https://airflow.apache.org/docs/apache-airflow/stable/howto/email-config.html#send-email-using-aws-ses

However, on MWAA we cannot modify airflow.cfg we have to use the configuration options.

• email.email_backend : airflow.providers.amazon.aws.utils.emailer.send_email
• email.from_email: no_reply@[domain.com]

✅ This will allow MWAA to use the IAM executor role to send emails directly to SES, by using the domain on from_email

📨 Step 3: Writing a Test DAG to Send Emails

Now, let’s create an Airflow DAG to test the email functionality. We’ll use the EmailOperator to send an email upon the successful completion of a simple task.

Here’s the code for a basic DAG:

from datetime import datetime, timedelta
import airflow
from airflow.operators.email import EmailOperator
from airflow.operators.python import PythonOperator

default_args = {
    "owner": "airflow",
    "start_date": datetime(2021, 1, 1),
}

dag_email = airflow.DAG(
    dag_id="send_mail_dag",
    default_args=default_args,
    schedule_interval="@once",
    dagrun_timeout=timedelta(minutes=60),
    description="use case of email operator in airflow",
)

def start_task_func():
    print("task started")

start_task = PythonOperator(
    task_id="executetask", python_callable=start_task_func, dag=dag_email
)

send_email = EmailOperator(
    task_id="send_email",
    to="your.email@domain.com",
    subject="Ingestion Complete",
    html_content="Success! Email notification from MWAA.",
    dag=dag_email,
)

start_task >> send_email

Breakdown of the DAG:

1. DAG Definition: The send_mail_dag is defined to run only once (@once schedule interval). This DAG will send an email after executing a simple Python task.
2. Python Task (start_task): This task prints a message to the logs ("task started") as a placeholder for any logic you might want to execute before sending the email.
3. Email Task (send_email): The EmailOperator sends an email to your.email@domain.com with a subject and a simple HTML body. Replace the recipient's email with your own for testing.

Running the DAG:

• Once you’ve added the DAG to your Airflow environment, trigger it manually from the Airflow UI.
• Monitor the DAG’s progress, and once the tasks are complete, check your inbox for the email notification!

👨‍💻 Step 4: MWAA Local Runner Configuration

When developing locally, you may want to test sending emails via SES. For this, you’ll need to configure the MWAA Local Runner with SMTP credentials. This setup allows Airflow to use the SMTP service to send emails when running locally.

I will not show how to configure Local Runner, follow the official guide from their repo. Link to MWAA Local Runner Startup Script: https://github.com/aws/aws-mwaa-local-runner/blob/v2.10.1/startup_script/startup.sh

Here’s a startup script (startup.sh) that configures the necessary environment variables for SMTP to be equivalent to the settings on MWAA. You can use secret manager, or hardcode the credentials (remember to add script to .gitignore).

PD: If you decide to query SecretManager, you will need to have your personal IAM Access Keys configured on MWAA Local Runner to be able to connect if using Secret Manager.

#!/bin/bash

echo $MWAA_AIRFLOW_COMPONENT

# Set the SMTP Environment variables with the SMTP host, port, and mail from
export AIRFLOW__SMTP__SMTP_HOST=email-smtp.eu-west-1.amazonaws.com
export AIRFLOW__SMTP__SMTP_STARTTLS=True
export AIRFLOW__SMTP__SMTP_SSL=False
export AIRFLOW__SMTP__SMTP_PORT=587
export AIRFLOW__SMTP__SMTP_MAIL_FROM=no_reply_localrunner@[domain.com]

# Get the SMTP username and password from secrets manager > https://repost.aws/knowledge-center/mwaa-ses-send-emails
username=$(aws secretsmanager get-secret-value --secret-id airflow/variables/smtp_user --region eu-west-1 --query SecretString --output text)
password=$(aws secretsmanager get-secret-value --secret-id airflow/variables/smtp_password --region eu-west-1 --query SecretString --output text)
#username=AWS_SMTP_USER
#password=AWS_SMTP_ACCES_KEYS

# Set the SMTP Environment variables with the username and password retrieved from Secrets Manager
export AIRFLOW__SMTP__SMTP_USER=$username
export AIRFLOW__SMTP__SMTP_PASSWORD=$password

# Print the SMTP user
echo "SMTP user is $AIRFLOW__SMTP__SMTP_USER"

Based on the structure of the DAG above, you will be able to receive emails. Play around with html_content or HTML templates to create a improved format.