How to test WAF traffic filter rules on AEMCaaS

Sharing our WAF rules testing experience, hiccups faced and how we solved

Photo by Ed 259 on Unsplash

Prologue

On May 2025, we had a production outage at our AEMCaaS. Server crashed due to Surge in Traffic. The Traffic Surge was again caused by us; we were working on a migration that resulted in 1000s of graphql queries every second. We quickly stopped the leak. But lessons learnt, even AEMCaaS was vulnerable to DDoS.

Adobe helped us in the situation. They run a performance audit report. One production recommendation from audit was; our cdn.yaml was missing traffic filter rules as per

https://experienceleague.adobe.com/en/docs/experience-manager-learn/cloud-service/security/traffic-filter-and-waf-rules/using-waf-rule

https://experienceleague.adobe.com/en/docs/experience-manager-learn/cloud-service/security/traffic-filter-and-waf-rules/using-traffic-filter-rules#_self

We finally implemented the WAF config, deployed but got stuck when trying to test the WAF rules

Sharing our WAF rules testing experience, hiccups and how we solved.

The Rule

Per adobe documentation, we added below rule into the /repo/config/cdn.yaml and deployed through the CDN pipeline

trafficFilters:
    rules:
      - name: graphql-rate-limit
        when:
          allOf:
            - reqProperty: tier
              equals: publish
            - reqProperty: path
              matches: "^/graphql/execute.json/.*"
        rateLimit:
          limit: 500
          window: 10
          penalty: 300
          count: all
          groupBy:
            - reqProperty: clientIp
        action: block
      - name: waf-attack-protection
        when:
          reqProperty: tier
          equals: 'publish'
        action:
          type: block
          wafFlags:
            - ATTACK

The rule is self-explanatory. When the traffic exceeds 500 requests per second (averaged over 10 seconds), then CDN should block the request. Very simple helloworld rule. Next was the testing challenge.

StressTest Attempt 1 — Simple sequential traffic

We tried a simple bash script that will hit 550 requests to graphql from same IP. Here is the script

for i in {1..550}                                                                                                                    
  do                                                                                                                                   
  curl -s -o /dev/null \                                                                                                               
  -w "%{http_code}\n" \                                                                                                                
  "https://<stage-domain>/graphql/execute.json/<graphql-query-endpoint>"                                                               
  done | sort | uniq -c

The first 500 requests should pass. And rest 50 is expected to fail with 429.

This test FAILED. Meaning we received 200 for all 550 requests

Reason: The test used a sequential curl loop, where each request completes before the next request starts. Against a remote endpoint, 550 sequential requests are typically spread over more than 10 seconds, so the traffic didnt cross the threshold inside a single 10-second window.

We needed better script to place concurrent requests, with tools like Vegeta or hey.

StressTest Attempt 2— With cachbuster

Script was tweaked to include cache buster

for i in {1..50}; 
  do
  curl -s -o /dev/null -w "%{http_code}\n" \
    "https://<stage>/graphql/execute.json/<query>?bust=$i" &
  done
wait

We also tried with hey. The substitution for {{.i}} didnt work, so reverted to Bash script.

From the log-forwarding to splunk, CDN logs showed

Actual: match=set-graphql-surrogate-key,action=log

Expected: match=graphql-rate-limit,action=block

The actual logs confirms, requests are picking the cdn.yaml config. But the expectedgraphql-rate-limit rule was not firing. This confirmed the rule did not trip the test pattern. Rather the rule was skipped or dropped.

This could be due to, how rate limiting behaves at the CDN edge. Rate-limit counters are maintained at the edge, not as one single global counter. A short burst of 50 simultaneous requests can be distributed across multiple edge nodes, so no individual counter necessarily exceeds the configured threshold. Also, rate-limit blocking applies to traffic after the threshold is detected; it does not retroactively block the requests that established the breach. If the test sends one burst and then stops, there may be no later requests left for the penalty to affect.

To validate this more reliably, we needed a sustained test from the same source for at least 30 seconds, keeping traffic above the configured threshold long enough for the counter and penalty to engage.

For example: hey -z 30s -q 20 -c 5 “https://<stage>/graphql/execute.json/”

To keep every request in a CDN MISS, we needed a shell loop that continuously varies the query string during the test instead of using a literal placeholder with hey. With this sustained pattern, we expected initial 200 responses, followed by blocked or non-200 responses after the threshold is crossed.

Thereby expecting the, match=graphql-rate-limitto appear in the CDN log rules field.

Also we needed a cooldown between test runs, because penalty: 300, was blocking the client IP for 300 seconds once triggered.

StressTest Attempt 3— Sustained concurrent bash loop

The final working script

end=$((SECONDS+35))
i=0
while [ $SECONDS -lt $end ]; do
  curl -s -o /dev/null -w "%{http_code}\n" \
    "https://<stage>/graphql/execute.json<query>?bust=$i" &
  i=$((i+1))
done
wait

To maintain the traffic above the threshold for < 30 secs, we were able to run a sustained concurrent bash loop. The rate limit rule was triggered correctly. Initial requests passed followed by 429 rejections.

We could validate correct 429 status from splunk logs as well

cache: PASS status: 429 rules: match=set-graphql-surrogate-key,action=log url: /graphql/execute.json/...?bust=674 aem_tier: publish

Thereby, after limiting the test execution to under 30 seconds, and placing burst of query requests, we were able to test rate limit traffic rule at CDN.

Happy Coding!