Monitoring Step functions using middy and CloudWatch alarms

Ever been in that scenario where a user finds out about a critical issue before anyone on the team does? We didn’t want to wait until that happened to us. So we asked ourselves: how can we make sure it never does?

The answer is monitoring, but not the kind where you set up a dashboard and check it occasionally. The kind where the system tells you something is wrong before anyone else notices.

That’s what this post is about. We’ll walk through how to set up monitoring for both types of Step Function failures — the ones AWS surfaces automatically, and the silent business-level ones it doesn’t — using middy, AWS Lambda Powertools, and CloudWatch alarms defined in CDK.

The problem with Step Function observability

AWS Step Functions are excellent for orchestrating multi-step workflows. The visual execution graph in the console is genuinely useful for debugging. But out of the box, monitoring might be a bit basic, and it misses an entire category of failure.

The obvious failure: execution errors

If a Lambda throws an unhandled exception, the step fails, the execution stops, and the built-in ExecutionsFailed metric goes up. You can set an alarm on that. Done ✅

But here’s the thing: in a real pipeline, you often don’t want one bad record to stop the entire execution. You want the workflow to keep running for everything else while still capturing the fact that something went wrong with that specific record. So instead of letting the error propagate, you catch it in the Lambda and return something like:

return { …event, Status: “FAILED”, Message: errorMessage };

The step succeeds as far as Step Functions is concerned. The execution continues. But that failure is now invisible, there is no ExecutionsFailed metric, no alarm, nothing.

The hidden failure: business-level errors

This is the gap that actually worries us. A step can quietly return Status: “FAILED” on every single invocation and the Step Functions console will show all executions as green. Nobody will know unless they open the execution and read the output of each state.

This is exactly the kind of thing that leads to a client calling before the team notices.

What we needed

1. A way to surface soft failures: emit a signal when a step returns Status: “FAILED”, even though the execution itself didn’t error out.

2. Alarms on both kinds of failure: hard execution errors and these business-level soft failures.

3. Enough context to debug quickly: not just “something failed” but which execution, which record, and what the error was.

Our workflow

For context, our pipeline looks something like this:

Each box is a Lambda function. Any of them can fail. Before our monitoring changes, a hard failure went red in the console but told us nothing about which step caused it — and a soft failure didn’t even do that. Either way, we had no proactive signal.

The building blocks

Before getting into code, here is the stack we use:

• **middy:** middleware framework for Lambda handlers.
• **AWS Lambda Powertools for TypeScript: specifically the Metrics utility, which publishes metrics via EMF** (Embedded Metrics Format) through CloudWatch Logs.
• AWS CDK: for defining alarms and dashboards as code.
• Amazon DynamoDB: for storing error context so alarm notifications include actionable details

The key thing about Powertools Metrics / EMF: unlike calling PutMetricData directly, metrics are emitted as structured JSON in your Lambda’s log output. CloudWatch Logs picks them up and turns them into proper CloudWatch metrics automatically. No extra API call, no added latency, and they show up in the same log stream as your application logs: which is very useful when you’re debugging.

Emitting metrics from inside each step

Because we have many Lambdas across multiple Step Functions, we wrap middy and Powertools into a single Lambda Layer that every handler imports from. This means:

• The middy configuration is in one place
• Adding a new handler means one import, not copy-pasting boilerplate
• The timeoutEarlyInMillis: 0 setting is applied everywhere consistently. This is important because middy’s default early-timeout behaviour conflicts with Step Functions’ own timeout handling

// lib/lambda/middy-layer/index.ts

import middyCore from "@middy/core";
import { Metrics, MetricUnit } from "@aws-lambda-powertools/metrics";
import { logMetrics } from "@aws-lambda-powertools/metrics/middleware";
import type { Handler } from "aws-lambda";

/**
 * Creates a Metrics instance scoped to the current stack and step.
 * Reads POWERTOOLS_METRICS_NAMESPACE and POWERTOOLS_SERVICE_NAME from env
 * both are set in CDK on each Lambda function definition.
 */
export const pipelineMetrics = () =>
  new Metrics({
    namespace: process.env.POWERTOOLS_METRICS_NAMESPACE,
    serviceName: process.env.POWERTOOLS_SERVICE_NAME,
  });

/**
 * Logs a named error metric with structured metadata attached.
 *
 * The metadata (executionId, recordId, errorMessage) does not become a
 * CloudWatch dimension — it stays in the EMF log line and is queryable via
 * Logs Insights. This is how you go from "the alarm fired" to "here is exactly
 * which execution failed and why".
 */
export const logErrorWithContext = async (
  metrics: Metrics,
  errorMessage: string,
  options: { recordId?: string; executionId?: string; metricName?: string } = {},
) => {
  const { recordId, executionId, metricName = "Error" } = options;

  metrics.addMetadata("errorMessage", errorMessage);
  if (recordId) metrics.addMetadata("recordId", recordId);
  if (executionId) metrics.addMetadata("executionId", executionId);

  try {
    const stack = process.env.POWERTOOLS_METRICS_NAMESPACE!;
    const service = process.env.POWERTOOLS_SERVICE_NAME!;
    const tableName = process.env.METRIC_CONTEXT_TABLE!;

    if (stack && service && tableName) {
      const timestamp = new Date().toISOString();
      // TTL matches CloudWatch's 14-day metric retention window
      const ttl = Math.floor(Date.now() / 1000) + 14 * 24 * 60 * 60;

      await dynamo.send(
        new PutItemCommand({
          TableName: tableName,
          Item: {
            stack_service: { S: `${stack}_${service}` },
            timestamp: { S: timestamp },
            executionId: executionId ? { S: executionId } : { NULL: true },
            recordId: recordId ? { S: recordId } : { NULL: true },
            errorMessage: { S: errorMessage },
            ttl: { N: ttl.toString() },
          },
        }),
      );
    }
  } catch (err) {
    console.error("Failed to write metric context to DynamoDB:", err);
  }
  metrics.addMetric(metricName, MetricUnit.Count, 1);
};

export function middy<E = unknown, R = unknown>(handler: Handler<E, R>) {
  // timeoutEarlyInMillis: 0 — let Step Functions own timeout handling
  return middyCore(handler, { timeoutEarlyInMillis: 0 });
}

export { Metrics, MetricUnit, logMetrics };

Using the layer in a handler

A handler that participates in a Step Function workflow imports from middy-layer and wraps its logic with .use(logMetrics(metrics)). That single middleware call is what flushes all accumulated metrics at the end of the invocation:

// lib/lambda/dataProcessor.ts

import { middy, pipelineMetrics, logMetrics, logErrorWithContext } from "middy-layer";

interface DataProcessorEvent {
  input: number;
  executionId?: string;
}

const metrics = pipelineMetrics();

const handler = async (event: DataProcessorEvent) => {
  const { executionId } = event;

  try {
    const result = {
      ...event,
      processedData: {
        originalValue: event.input,
        calculatedValue: event.input * 2,
        status: "processed",
      },
    };
    return { statusCode: 200, body: JSON.stringify(result) };
  } catch (e) {
    const errorMessage = e instanceof Error ? e.message : JSON.stringify(e);

    // Soft-failure signal: the execution will continue, but this metric
    // is what triggers the alarm. Metadata is queryable in Logs Insights
    // to pinpoint which execution failed and why.
    logErrorWithContext(metrics, errorMessage, {
      executionId,
      metricName: "StatusFAILED",
    });

    return { ...event, Status: "FAILED", Message: errorMessage };
  }
};

export const lambdaHandler = middy(handler).use(logMetrics(metrics));

A few things to notice:

• The metric name StatusFAILED is intentional — it matches what we declare in the monitoring config (more on that below), which is what triggers the alarm.
• We pass executionId through the event. Step Functions doesn’t inject this automatically into the Lambda context; we extract it in the ASL definition and pass it as part of the event payload.
• The logMetrics middleware flushes metrics after the handler completes — on both the success and error path, and even if it throws. You don’t need to call anything at the end of the function body; on a successful invocation it simply flushes an empty set of metrics.
• logErrorWithContext is now async because of the DynamoDB write. Make sure to await it in the handler.

The metric registry

Instead of scattering metric names across handler files, we centralise them in one config file. This is the source of truth for what gets alarmed on:

Instead of scattering metric names across handler files, we centralise them in one config file. This is the source of truth for what gets alarmed on:

// lib/monitoring/stacks-and-metrics.ts

export const dataPipelineStepMetrics: Record<string, string[]> = {
  dataProcessor:       ["StatusFAILED"],
  validationProcessor: ["StatusFAILED"],
  finalProcessor:      ["StatusFAILED"],
  errorHandler:        ["StatusFAILED"],
  // add further steps here as the pipeline grows
};

Keys are the POWERTOOLS_SERVICE_NAME values set on each Lambda in CDK. Values are the metric names the handler emits. To get an alarm for a new step, you need to do two things: add it to this map, and make sure its Lambda has POWERTOOLS_METRICS_NAMESPACE, POWERTOOLS_SERVICE_NAME , and METRIC_CONTEXT_TABLE set in its environment (the first two matching the map key).

set in its environment (matching the map key). The CDK construct reads the map and creates the alarm automatically.

Alarming on those metrics in CDK

This is the CDK construct that brings everything together. You instantiate it once per state machine, pass it the metric registry, and it creates:

• Alarms on the three built-in Step Functions metrics (Failed, Throttled, TimedOut)
• One alarm per entry in the metric registry
• A CloudWatch dashboard with widgets for all of the above

// lib/monitoring/base-step-function-monitoring.ts

export class BaseStepFunctionMonitoring extends Construct {
  constructor(scope: Construct, id: string, props: BaseStepFunctionMonitoringProps) {
    super(scope, id);

    const stateMachine = sfn.StateMachine.fromStateMachineArn(
      this,
      `${props.dashboardName}StateMachine`,
      Fn.importValue(props.stepFunctionArnImportValue),
    );

    const alarmAction = new cwActions.SnsAction(props.alarmTopic);

    // Built-in Step Function metrics — no Lambda changes needed
    for (const [label, metric] of [
      ["Failed",    stateMachine.metricFailed()],
      ["Throttled", stateMachine.metricThrottled()],
      ["TimedOut",  stateMachine.metricTimedOut()],
    ] as [string, cw.IMetric][]) {
      const alarm = new cw.Alarm(this, `${props.dashboardName}-${label}`, {
        alarmName: `${props.dashboardName}-StepFunction${label}-${props.stage}`,
        metric,
        threshold: 1,
        evaluationPeriods: 1,
        comparisonOperator: cw.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
        treatMissingData: cw.TreatMissingData.NOT_BREACHING,
      });
      alarm.addAlarmAction(alarmAction);
    }

    // Custom per-step metrics from the registry
    const customMetrics = createMetricsFromMap(props.stackName, props.stepMetrics);

    for (const metric of customMetrics) {
      const service = metric.dimensionsMap?.["service"] ?? "unknown";
      const alarm = new cw.Alarm(this, `${props.dashboardName}-${service}-${metric.metricName}`, {
        alarmName: `${props.dashboardName}-${service}-${metric.metricName}-${props.stage}`,
        metric,
        threshold: 1,
        evaluationPeriods: 1,
        comparisonOperator: cw.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
        treatMissingData: cw.TreatMissingData.NOT_BREACHING,
      });
      alarm.addAlarmAction(alarmAction);
    }

    // Dashboard
    const dashboard = new cw.Dashboard(this, `${props.dashboardName}Dashboard`, {
      dashboardName: `${props.dashboardName}-${props.stage}`,
    });
    dashboard.addWidgets(...createWidgets(customMetrics));
  }
}

The createMetricsFromMap helper deserves a mention. It converts the { stepName: metricName[] } registry into cw.Metric objects using the stack name as the CloudWatch namespace and the step name as the service dimension. This is exactly how Powertools EMF publishes the metrics — so the alarm dimensions match what the Lambda emits automatically.

// lib/monitoring/create-metrics.ts

export function createMetricsFromMap(
  namespace: string,
  stepMetrics: Record<string, string[]>,
): cw.Metric[] {
  return Object.entries(stepMetrics).flatMap(([service, metricNames]) =>
    metricNames.map(
      (metricName) =>
        new cw.Metric({
          namespace,
          metricName,
          dimensionsMap: { service },
          statistic: "Sum",
          period: Duration.minutes(1),
        }),
    ),
  );
}

Wiring it into your stack

// lib/step-function-example-stack.ts (excerpt)

const stackName = "DataPipelineStack";

// Set on each Lambda function so Powertools knows the namespace and step name
const dataProcessorLambda = new lambda.Function(this, "DataProcessorLambda", {
  // ...
  environment: {
    POWERTOOLS_METRICS_NAMESPACE: stackName,
    POWERTOOLS_SERVICE_NAME: "dataProcessor", // must match stacks-and-metrics.ts key
    METRIC_CONTEXT_TABLE: metricContextTable.tableName,
},
});

// After deploying the state machine, wire up all monitoring in one call
new BaseStepFunctionMonitoring(this, "DataPipelineMonitoring", {
  dashboardName: "DataPipeline",
  stepFunctionArnImportValue: "StateMachineArn", // CloudFormation export from the SFN stack
  stepMetrics: dataPipelineStepMetrics,
  stackName,
  alarmTopic,
  stage: "dev",
});

That’s it. Every step that is listed in dataPipelineStepMetrics now has its own alarm. Every alarm routes to the SNS topic.

Making the notification actionable

The first version of this setup got us alerted, but the notification only included the service name and metric — enough to know something was wrong, but not enough to act on immediately. We know something is wrong but not which execution or what the error was.

That’s why logErrorWithContext also writes to DynamoDB. When the alarm Lambda fires, it queries that table using the namespace and service name from the CloudWatch alarm payload, finds the error context closest in time to when the alarm fired, and includes it in the notification.

The DynamoDB table

const metricContextTable = new dynamodb.Table(this, "MetricContextTable", {
  partitionKey: { name: "stack_service", type: dynamodb.AttributeType.STRING },
  sortKey: { name: "timestamp", type: dynamodb.AttributeType.STRING },
  timeToLiveAttribute: "ttl",
  billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
});

The partition key is stack_service — the namespace and service name joined with an underscore (e.g. DataPipelineStack_dataProcessor). This matches exactly what logErrorWithContext writes and what the alarm Lambda queries. The TTL attribute is set to 14 days, matching CloudWatch’s own metric retention window, so the table stays clean automatically.

The alarm notification Lambda

// functions/monitoring/alarm-notification/index.ts (simplified)

export const handler = async (event: SNSEvent): Promise<void> => {
  for (const record of event.Records) {
    const message = JSON.parse(record.Sns.Message);
    const namespace = message.Trigger.Namespace;
    const service =
      message.Trigger.Dimensions.find((d: { name: string }) => d.name === "service")
        ?.value ?? "";
    const alarmTimestamp = record.Sns.Timestamp;

    // Find the error context closest to when the alarm fired
    const context = await getClosestMetricContext(
      namespace,
      service,
      alarmTimestamp,
      process.env.METRIC_CONTEXT_TABLE!,
    );

    let text = `Alarm: ${message.AlarmName}\n`;
    if (context?.executionId) {
      text += `Execution ID: ${context.executionId.split(":").pop()}\n`;
    }
    if (context?.errorMessage) {
      text += `Error: ${context.errorMessage.substring(0, 150)}\n`;
    }
    text += `Time: ${context?.timestamp ?? alarmTimestamp}\n`;

    await axios.post(process.env.WEBHOOK_URL!, { text });
  }
};

getClosestMetricContext queries DynamoDB by stack_service, takes the 10 most recent items, filters to those within a 60-second window of the alarm timestamp, and returns the closest match. The time window matters because there is a small delay between a Lambda emitting a metric and CloudWatch triggering the alarm.

The result is a notification that goes from:

Alarm: DataPipeline-dataProcessor-StatusFAILED-prod service: dataProcessor

to

Alarm: DataPipeline-dataProcessor-StatusFAILED-prod service: dataProcessor Execution ID: abc123def Error: Cannot read properties of undefined (reading ‘id’) Time: 2026–05–04T19:30:00.000Z

That’s the difference between “something broke, go find it” and “here’s exactly what broke and where to look.”

How it fits together

The built-in Step Function metrics sit alongside the custom ones. Together they give you two levels of coverage: did the overall execution succeed or fail, and which specific step caused it?

What changed for us

Before this setup, our monitoring was reactive: someone opened the console and noticed something was wrong, usually after a user reported it.

After rolling this out, the flow is the opposite. The alarm fires within minutes of the first StatusFAILED metric appearing. The notification includes the state machine, the affected step, the execution ID, and the specific record that failed — before anyone outside the team knows there was a problem.