A successful 1,000 TPS checkout game day: pilot light to full scale with AWS ARC Region switch

Multi-Region architecture is usually introduced as protection from a cloud Region outage. That is important, but it is not the only reason to build a regional recovery path. The more practical question is often mean time to recovery (MTTR): when the signals are clear that the active Region cannot serve a tier-0 customer journey safely, how long should the team keep repairing the same failure domain before moving the workload?

The impairment does not have to be an AWS regional event. A bad application release, identity or authorization failure, configuration rollout, secret-distribution problem, service-mesh fault, dependency saturation, or another tier-0 platform defect can make one regional cell unusable while the recovery Region remains healthy. From the customer’s perspective, the cause matters less than the outcome: checkout, login, or another critical path is unavailable. If independent application and infrastructure signals agree that the source cell is impaired and local recovery time is uncertain, a controlled regional failover can be the fastest way to reduce MTTR.

Regional failover is not the first response to every alert. A zonal or service-level recovery is usually safer for a contained fault. In an active/passive design, moving Regions can change database-writer ownership, DNS state, and the location of every new request. For a continuously busy application, even a graceful transition can create a bounded period of retries or unavailability. The decision therefore needs explicit entry criteria: the affected tier is critical, the destination is healthy, the source is unlikely to recover inside the remaining RTO, data-safety requirements are understood, and the recovery plan has been exercised.

That creates a difficult cost trade-off. Keeping the passive Region permanently at peak capacity shortens the cutover, but it also pays for idle duplicate compute every hour. Keeping only a pilot light or warm standby reduces waste, but the destination must be expanded before traffic moves. Doing that manually during an incident adds decisions and serial handoffs directly to MTTR.

This is where Amazon Application Recovery Controller (ARC) Region switch becomes useful. The plan can connect CloudWatch evidence to an automatic trigger, pre-scale the destination, wait for the requested capacity, fence writes, move the database writer, verify it, redirect traffic, and reopen the application in a fixed order. Automation removes the human from routine initiation and sequencing while retaining the guardrails, cooldowns, observability, and abort path that operators need.

I built and ran that transition in AWS — not as a slide. The active cell in us-west-2 served a synthetic critical checkout experience with ten Amazon ECS/Fargate tasks. The passive cell in us-east-2 idled at one task. Gatling generated 1,000 new payment transactions per second. A CloudWatch payment error-rate signal automatically started an ARC Region switch plan, which pre-scaled the east cell, fenced writes with a sorry page, switched the Aurora Global Database writer, moved Route 53, and reopened checkout.

The game day succeeded. ARC automatically started the recovery, scaled east from 1 to 10 tasks before sending traffic, moved the Aurora writer exactly once, completed all six safety gates, redirected Route 53, and reopened checkout. The DAG took 4m06.787s; ARC reported actualRecoveryTime of 4m38.78s—comfortably inside the 15-minute RTO.

[Image: ARC plan evaluation and the first safety gates]

The client graph was not perfectly flat, and that is acceptable in a useful game day. The deliberately conservative sorry page returned retryable 503s while writer ownership changed. Those responses are evidence that the write fence worked: no payment was silently accepted by the wrong side, and idempotency made retries safe. The outcome to optimize next is the length of that controlled pause, not the correctness of the recovery sequence.

Why ARC is useful here

Without orchestration, a team must remember — in an incident — to scale compute, block writes, promote a database, verify the writer, change traffic, and reopen the application in exactly the right order. Each manual checkpoint extends MTTR and creates another opportunity to act on stale information. ARC turns that operational knowledge into a versioned, evaluated and observable plan.

That is especially valuable for cost-efficient active/passive designs. Running ten idle tasks in east all month would buy fast readiness by paying for full duplicate compute. Running no application task at all is cheaper but leaves more untested assumptions. This lab keeps one active task in the passive Region, continuously proving the image, secret, networking, load balancer and database connection. ARC then expands that cell to the observed ten-task peak before it touches the database or DNS.

The result is a practical middle path:

• The standby application path is alive and testable.
• Passive Fargate task count is 90% lower than a ten-task full-scale standby.
• Recovery ordering is encoded, evaluated and repeatable.
• Operators, the CLI and CloudWatch alarms all invoke the same plan.
• Every stage produces one execution timeline for review.

What was deployed

The repository creates two independent regional cells:

Active/passive checkout topology: Route 53 selects between two isolated regional VPC cells; Aurora Global Database maintains one writer and a cross-Region secondary.

The diagram uses the official AWS Architecture Icons package. The measured standby profile is warm standby: ten active tasks in us-west-2 and one continuously healthy task in us-east-2.

• Checkout compute: us-west-2 starts with ten healthy Fargate tasks; us-east-2 starts with one.
• Application ingress: each Region has its own Application Load Balancer.
• Database role: the west Aurora PostgreSQL cluster is the global writer; the east cluster is the secondary.
• Aurora Serverless v2 capacity: both regional clusters are configured for 0.5–16 ACU.
• Traffic: Route 53 failover records select the active regional ALB through ARC-vended health checks.

The tasks run in public subnets to avoid NAT hourly cost in this short-lived lab, but their security groups accept no public application ingress. Only the ALB can reach port 3000. The Aurora clusters are not public and accept PostgreSQL only from the regional task security group. Secrets Manager stores the database secret and replicates it to the other Region. Logs are encrypted, IAM roles are narrowly scoped, container images are pinned, and the API uses Helmet plus a small JSON body limit.

The demo name is app.arc-demo.example. .example is reserved and deliberately not delegated, so it is safe for documentation but not a real public production name. ARC can still operate its Route 53 records and health checks; Gatling uses a local resolver mapping to reach the currently active ALB. A real deployment should use a delegated domain and ACM TLS.

Although people often say “pods,” ECS calls these units tasks. There are no Kubernetes pods in this architecture.

One plan, six ordered safety gates

The ARC Region switch plan is versioned and evaluated. Version 4 passed evaluation and has this activation workflow:

ARC Region switch workflow: CloudWatch starts one evaluated plan whose six gates pre-scale compute, fence writes, switch and verify Aurora, move Route 53, and reopen checkout.

1. Pre-scale passive checkout to peak. The native ECSServiceScaling block targets 100% of the source service’s sampled maximum from the last 24 hours. East moved from 1 to 10 tasks and was healthy in 44.265s.
2. Show checkout sorry page. A custom Lambda changes the deactivating cell’s ALB listener to a fixed 503 response. New writes stop at the edge.
3. Promote Aurora global writer. Graceful mode performs a switchover; ungraceful mode permits a failover. The measured graceful switchover took 65.235s.
4. Verify the writer. Another custom Lambda calls DescribeGlobalClusters and refuses to proceed unless the activating regional cluster is IsWriter=true.
5. Redirect Route 53 traffic. ARC flips its vended health checks using the Route 53 data plane. This stage took 120.698s, including DNS safety timing.
6. Remove the sorry page. The old regional listener is restored after the writer and traffic transition are complete.

[Image: The automatic triggers attached to the plan]

This order is the design. Scaling after DNS would overload a one-task standby. Moving DNS before writer verification would send payment writes to a reader. Removing the fence before traffic is aligned could create split-brain behavior at the application edge.

How the pre-scale gate calculated capacity and knew when to continue

The compute-readiness gate is not a custom Lambda. It is ARC’s native ECSServiceScaling execution block. This is the actual configuration from the plan:

- Name: Pre-scale passive checkout to peak
  Description: ARC grows the target service to 100 percent of the source
    service's observed 24-hour peak before touching the database or DNS.
  ExecutionBlockType: ECSServiceScaling
  ExecutionBlockConfiguration:
    EcsCapacityIncreaseConfig:
      TimeoutMinutes: 10
      TargetPercent: 100
      CapacityMonitoringApproach: sampledMaxInLast24Hours
      Ungraceful:
        MinimumSuccessPercentage: 90
      Services:
        - ClusterArn: !Ref PrimaryClusterArn
          ServiceArn: !Ref PrimaryServiceArn
        - ClusterArn: !Ref SecondaryClusterArn
          ServiceArn: !Ref SecondaryServiceArn

Services gives ARC exactly one ECS service in each Region. With sampledMaxInLast24Hours, ARC uses the maximum running-task count it observed for the source service during the previous 24 hours. TargetPercent: 100 means the destination should match all of that observed capacity. The calculation is effectively ceil(1.00 × source capacity): the source peak was ten, so the destination target was ten.

Because the service uses Application Auto Scaling, ARC updates its minimum capacity and ECS desired count. It then waits until the requested ECS service capacity is fulfilled before it advances to the write fence or any stateful and traffic-changing step. TimeoutMinutes: 10 bounds that wait. The MinimumSuccessPercentage: 90 value is the degraded rule for an ungraceful execution; it was not used to redefine success in this measured graceful run.

Capacity fulfillment and application readiness are complementary checks. The ECS service is attached to an ALB target group that probes /readyz every 15 seconds and requires two successful checks, while ECS provides a 90-second health-check grace period and deployment rollback:

Service:
  Properties:
    HealthCheckGracePeriodSeconds: 90
    DeploymentConfiguration:
      DeploymentCircuitBreaker:
        Enable: true
        Rollback: true
    LoadBalancers:
      - ContainerName: app
        ContainerPort: 3000
        TargetGroupArn: !Ref TargetGroup

TargetGroup:
  Properties:
    HealthCheckPath: /readyz
    HealthCheckIntervalSeconds: 15
    HealthyThresholdCount: 2
    Matcher:
      HttpCode: '200'

In the measured execution, east moved from one to ten desired and running tasks in 44.265 seconds, and the dashboard showed ten healthy ALB targets before the later Route 53 stage moved checkout traffic. ARC supplied the capacity gate; ECS and the load balancer supplied the workload-health evidence. That separation is important: task count alone is not the same as a working application.

During plan evaluation, ARC also checks that both services exist, the destination’s configured maximum can satisfy the requested percentage, the plan role has the required permissions, and its resource monitor has collected running-task data. Those checks move several common recovery surprises from incident time to plan-validation time.

Pilot light, warm standby, and full scale are states — not separate plans

The same DAG handles all three topologies:

• Pilot light — 0 east Fargate tasks. The image, ECS cluster, ALB, VPC, replicated secret, and Aurora secondary already exist. This has the lowest standby compute cost and the longest application scale-up.
• Warm standby — 1 east Fargate task (the measured idle state). The complete application path, image, secret, networking, ALB health checks, and database read path are continuously provable. The trade-off is a small continuous compute cost.
• Full scale — 10 east Fargate tasks. The destination already matches the observed active peak. Compute takeover is fastest, but standby compute cost is highest.

At one standby task instead of ten, the passive Fargate task count is 90% lower. That is not a 90% reduction in the entire DR bill: the second ALB, Aurora secondary, storage, I/O, cross-Region replication, public IPv4, Route 53 health checks, logs, and ARC-related usage remain. Aurora Serverless v2 also keeps a 0.5 ACU floor in each Region and scales toward 16 ACU only when demand arrives.

ARC makes the transition topology-agnostic. If east is at one task, it scales. If it is already at ten, the same stage is effectively satisfied. It never scales the old cell down automatically; that is an intentional post-recovery decision.

The critical checkout transaction

Gatling’s transaction is literally named payments txn:

private final ChainBuilder paymentTransaction = tryMax(4).on(
  exec(http("payments txn")
    .post("/api/payments")
    .header("idempotency-key", "#{paymentId}")
    .body(StringBody("{\"amountCents\":#{amountCents}}"))
    .check(status().in(200, 201)))
    .pause(Duration.ofMillis(100))
);

setUp(
  scenario("Critical checkout experience")
    .feed(paymentData)
    .exec(paymentTransaction)
    .injectOpen(
      rampUsersPerSec(1).to(1000).during(Duration.ofSeconds(30)),
      constantUsersPerSec(1000).during(Duration.ofSeconds(480))
    )
).protocols(http.shareConnections());

Each virtual user gets one stable idempotency key. Retries reuse it. PostgreSQL enforces a unique constraint and the API’s single CTE returns either the newly created payment (201) or the already durable payment (200). That converts retries from a double-charge risk into a replay-safe lookup.

Connection sharing matters. Creating 1,000 fresh TCP connections per second tests connection churn more than payment throughput. Gatling’s shared connection pool better approximates browser, CDN, or API-gateway keep-alive behavior for this experiment.

Before: the idle and steady states

At 18:00:30 UTC both cells were deliberately set to one task. At 18:02:00 UTC the pre-test steady state was established: west had ten running tasks, east had one, the writer was west, and the CloudWatch payment alarm was OK.

Before the recovery fence, a Gatling sample contained 69,988 requests: 69,960 succeeded and 28 returned transient 502s — 99.96% success, not a fabricated 100%. The CloudWatch dashboard shows the capacity and payment-error transition around the event.

[Image: CloudWatch shows east pre-scaling from one to ten tasks before taking checkout traffic]

[Image: CloudWatch shows the payment alarm and the bounded safety-fence window]

Trigger: from bad payment signal to automatic execution

The automatic trigger deliberately watches the customer-facing payment signal, not an AWS service-status event. That means the same recovery path can react to a regional infrastructure problem or to a software and tier-0 platform failure that produces the same unsafe checkout outcome.

The west payment alarm combines load-balancer and target 5xx responses, divides them by request count, and turns red when the error rate exceeds 2%:

PaymentSloAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    ComparisonOperator: GreaterThanThreshold
    DatapointsToAlarm: 1
    EvaluationPeriods: 1
    Metrics:
      - Id: requests
        MetricStat:
          Metric:
            Namespace: AWS/ApplicationELB
            MetricName: RequestCount
          Period: 60
          Stat: Sum
        ReturnData: false
      - Id: elb5xx
        MetricStat:
          Metric:
            Namespace: AWS/ApplicationELB
            MetricName: HTTPCode_ELB_5XX_Count
          Period: 60
          Stat: Sum
        ReturnData: false
      - Id: target5xx
        MetricStat:
          Metric:
            Namespace: AWS/ApplicationELB
            MetricName: HTTPCode_Target_5XX_Count
          Period: 60
          Stat: Sum
        ReturnData: false
      - Id: payment_error_rate
        Expression: IF(requests>0,100*(elb5xx+target5xx)/requests,0)
        Label: Payment checkout error rate
        ReturnData: true
    Threshold: 2
    TreatMissingData: notBreaching

The plan first registers that alarm by name, then maps a transition to red to an east activation:

AssociatedAlarms:
  primaryPaymentHealth:
    AlarmType: applicationHealth
    ResourceIdentifier: !Ref PrimaryPaymentAlarmArn
  primaryPaymentTrigger:
    AlarmType: trigger
    ResourceIdentifier: !Ref PrimaryPaymentAlarmArn

Triggers:
  - Action: activate
    TargetRegion: !Ref SecondaryRegion
    Description: Automatically activate the east checkout when the west
      payment error-rate alarm turns red.
    MinDelayMinutesBetweenExecutions: 1
    Conditions:
      - AssociatedAlarmName: primaryPaymentTrigger
        Condition: red

The same alarm is associated twice for two different purposes: applicationHealth makes it part of plan health evaluation, while trigger makes it eligible to start an execution. ARC starts the plan when the alarm transitions into the configured condition; merely attaching a trigger to an alarm that is already red does not initiate a surprise failover.

A second trigger watches the Route 53 direct health check for the west /readyz endpoint. Route 53 global health-check metrics live in us-east-1, so that trigger alarm is intentionally in us-east-1 even though the application Regions are west and east-2. Together, the payment SLO and direct endpoint health show how recovery automation can be based on application evidence rather than waiting for a cloud provider to declare a regional incident.

For this timed demo, MinDelayMinutesBetweenExecutions was shortened to one minute and the payment alarm used a deliberately aggressive one-of-one evaluation. In production, use enough datapoints to resist flapping, combine independent conditions or composite alarms where appropriate, use a longer cooldown—often 60 minutes or more—and preserve an operator abort path. Automation should remove slow manual sequencing, not remove recovery governance.

The measured execution comment proves its origin:

Execution started by automated trigger [Automatically activate the east checkout when the west payment error-rate alarm turns red.]

[Image: ARC execution history highlights the measured automatic activation]

ARC execution APIs are regional. An east activation appears when list-plan-executions is called against us-east-2, not only against the plan’s primary us-west-2 endpoint. The first runner queried only the west endpoint, so it did not surface the already-successful automatic run and started a redundant operator fallback. The repository now polls the target execution Region. This is exactly the kind of operational detail a successful game day should uncover before an incident.

During: reject ambiguous writes, then reopen

The sorry page is a write fence, not a cosmetic banner. While the database writer changes, the deactivating ALB returns:

<h1>Checkout is temporarily paused</h1>
<p>We are switching to our recovery Region. No payment has been submitted.</p>

That produces a clear, retryable 503 and ensures the app does not pretend an ambiguous checkout succeeded. After east owns the Aurora writer and Route 53 points to it, ARC removes the fixed response and the old regional listener returns to normal health.

[Image: Automatic ARC execution completed every pre-scale, database and traffic gate]

The measured stage timing was:

Stage Duration Pre-scale east from 1 to 10 44.265s Enable sorry-page fence 5.055s Aurora graceful switchover 65.235s Verify east writer 5.039s Route 53 traffic stage 120.698s Remove sorry-page fence 5.041s Six-stage DAG wall time 246.787s (4m06.787s) ARC actualRecoveryTime 278.78s (4m38.78s)

Afterwards both Regions had ten healthy tasks, the global writer was us-east-2, ARC’s authoritative Route 53 state selected east, and a direct payment against east returned 201. Replaying the same idempotency key returned the existing durable payment.

Client outcome during the protected transition

The complete Gatling report contains 678,268 HTTP attempts because tryMax(4) adds retries:

Client result Measured value Successful HTTP attempts 434,042 Controlled safety-page 503 attempts, including retries 244,190 Transient 502s 36 p95 response time 141ms

The raw attempt count includes up to four retries for each logical payment, so it intentionally magnifies the safety-page window. The important game-day result is that rejection was explicit and bounded: checkout did not produce ambiguous writes while Aurora changed owners, the plan reopened the application, and replaying an idempotency key returned the already durable payment.

[Image: Gatling client-side outcomes during the safety fence]

For production, I would keep the same safety boundary and make the pause feel smoother: stop new checkout submissions before recovery, drain in-flight writes, expose Retry-After, let clients queue or retry with jitter, and graph logical payment outcomes separately from raw attempts. ARC gives us the stable, repeatable workflow on which those experience improvements can build.

Planned switchover versus emergency failover

These modes should remain explicit:

• Switchover (graceful) is for a planned exercise or controlled evacuation. Both Regions are reachable, Aurora synchronizes the secondary, and writer movement targets zero data loss. This is what the measured run used.
• Failover (ungraceful) is for an impaired source Region. ARC is allowed to promote the reachable secondary without waiting for the former writer. The recovery point is bounded by replication lag, so recent writes can be lost. Fencing and idempotency are even more important.

The plan config declares switchoverOnly for graceful runs and permits Aurora failover only in ungraceful mode. An operator cannot accidentally turn a routine game day into an emergency data-plane operation with a hidden default.

Start it from the console, CLI, or an alarm

In the console, open the plan, choose Execute plan, select us-east-2, choose Activate, select Graceful, add a comment, acknowledge the confirmation, and execute. Follow the execution in the target Region’s execution view.

[Image: Manual Execute plan dialog in ARC]

The equivalent CLI call is:

aws arc-region-switch start-plan-execution \
  --region us-east-2 \
  --plan-arn "$PLAN_ARN" \
  --target-region us-east-2 \
  --action activate \
  --mode graceful \
  --latest-version 4 \
  --comment "Planned checkout evacuation to us-east-2"

The API endpoint Region and --target-region must both be us-east-2 for this activation. A real CLI exercise completed earlier in 3m03.696s and is stored in the repository evidence.

For automatic recovery, the plan’s native trigger watches the associated payment SLO alarm. Route 53 direct health is a separate supported trigger path. In a production design, require enough datapoints to avoid flapping, use composite alarms where appropriate, set a conservative cooldown, and preserve a human abort path.

Operate and tear down deliberately

The repository supports the full lifecycle:

# local app + PostgreSQL
docker compose up --build -d

# deploy or restore the AWS lab
./infra/scripts/lab.sh up

# observe current state
./infra/scripts/lab.sh status

# run the measured 1,000 TPS scenario
./infra/scripts/run-experiment.sh

# destructive: remove the billable lab and database data
CONFIRM_DESTROY=YES ./infra/scripts/lab.sh down

# recreate it later
./infra/scripts/lab.sh up

The teardown requires an explicit confirmation because it deletes the lab database. For this run I left all resources deployed for inspection.

The lesson

ARC made the stateful recovery contract executable: destination capacity first, write fence second, writer movement and verification third, traffic last, then reopen. That same plan moved a one-task warm standby to a ten-task active cell without paying for ten idle standby tasks all month or maintaining a second recovery design.

This was a successful game day: automatic initiation worked, east reached peak capacity before traffic, Aurora preserved one writer, all six stages completed, Route 53 selected east, checkout reopened, and recovery stayed well inside the RTO. ARC was useful because the result did not depend on an operator improvising six interdependent changes under pressure. The remaining optimization — shortening and smoothing the controlled checkout pause — is now a measured engineering problem rather than a disaster-recovery unknown.

References

• AWS: Region switch plans
• AWS: ECS service scaling block
• AWS: Route 53 health-check block
• AWS: Region switch triggers
• AWS: Custom action Lambda block
• AWS CLI: start-plan-execution
• Gatling: open workload models