Permission Sensitive Caching in AEM (2/7)

Setting up the Dispatcher and CDN

Part 2 of 7 on protecting content behind a login on AEM

• Part 1 — AEM Publish: Access Control and Closed User Groups
• Part 2 — Setting up the Dispatcher and CDN
• Part 3 — Implementing the permission check servlet
• Part 4 — More CDN considerations
• Part 5 — Referencing Private Content
• Part 6 — Using OIDC as Single Sign On (planned)
• Part 7 — Permission-sensitive caching in the CDN (planned)

In this article, I want to share my experience with permission-sensitive caching — specifically, how you can make effective use of caching even when some content is protected on Publish and requires a login.

Adobe’s official documentation [1] only scratches the surface. But permission-sensitive caching is more complex and not exactly an out-of-the-box experience. So this turned into a surprisingly deep learning exercise — one I don’t plan to repeat anytime soon. This article also serves as a future reference for my future self ;-)

This part mostly addresses the Dispatcher. The other parts will cover the implementation in the Publish backend, which has its own challenges.

Quick recap to set the context

You probably know that AEM only reaches peak performance when rendered pages can be cached.

Pages are typically cached in the Dispatcher and in a Content Delivery Network (CDN). If you are using AEM as a Cloud Service (AEMaaCS), a CDN is already included in the stack — this is important to keep in mind, as we will see later.

The cache-layering structure has a huge impact on performance. In my last project, the page delivery time across layers differed by almost an order of magnitude. Here are some typical load times (HTML only).

• Delivery from CDN: ~30ms
• Delivery from Dispatcher: ~ 150ms
• Re-rendering from Publish: ~ 1000ms

Problem statement

While the cache is fast, it is also a bit dumb: It does not know if the user has access to a particular resource. By default, it delivers pages to anyone who requests them. Access control is done by the Publish tier, which is not involved in the delivery from cache.

So, if you have content that requires both fast delivery and access control, you need to implement Permission Sensitive Caching (PSC), in which the Dispatcher first asks the Publish system for permission before delivering a resource from the cache.

Digression: Why two caching layers?

With a CDN sitting on top of the Dispatcher, you might wonder why we need to cache in the Dispatcher at all. There are two reasons:

1. The CDN usually caches on a TTL basis, with pages expiring after a fixed period. Let’s say we set the TTL to one hour. If a CDN-cached page is accessed during that period, it can be delivered directly from the CDN. After that, it has to be re-fetched from the origin. This means that pages which are not accessed frequently — say, only every two hours — are unlikely to remain cached in the CDN. In this case, the Dispatcher can serve as a stopgap: The Dispatcher caches pages until the next publication, so if nothing has been published in the meantime, chances are that it can still serve the page from its cache — even after it has expired from the CDN.
2. Pages that are protected by a login cannot be cached in the CDN built into AEMaaCS. In this scenario, the Dispatcher is the only¹ caching layer available.

By default, the Dispatcher treats all pages as public. In the following sections, we will explore how to configure the Dispatcher to also account for access control.

Understanding Permission Sensitive Caching

To benefit from both the performance a cache offers and the access control an AEM Publish server provides, we have to configure Permission Sensitive Caching (PSC) in both the Dispatcher and the Publish system. For that, we need to:

1. Configure the Dispatcher to cache, but always check with the Publish system whether the current user has access privileges before actually serving from cache. This adds some latency — but deciding whether to grant access or not is still faster than rendering from Publish per request.
2. Configure the Dispatcher to specify which areas require this additional check. You might have some areas that are clearly public, in which case the permission check is obsolete.
3. Implement code in Publish to determine whether the current user session has access to a resource.

Public and private subareas

In the simplest case, the website can be divided into public and private parts, where only the latter requires login and access control. The Dispatcher does not know which resources are public and which are private. It is advisable to take this into account when designing the content structure, so that it can be configured statically for URL patterns.

For example, we could define /content/mysite/home/ as public and everything below /content/mysite/home/private/* as private. This would mean that the whole site can be cached (even in the CDN), only pages under private* are Dispatcher-only cacheable and require permission checking.

In my last project, however, private pages could potentially be anywhere, not in an Area defined by a URL pattern. This made the setup more complex but not impossible. I’ll elaborate more on that in part two.

For now, let’s focus on the simpler case first. Also, I think this is the more common case.

How the Dispatcher decides where to deliver from

We have to deal with four dimensions:

• The resource is in a private/public area
• User is logged in / logged out
• User has access to the resource / does not have access to the resource
• Resource is cached / is not cached in the Dispatcher

The conditions are checked according to the flow chart below:

The conversation between the Dispatcher and Publish comes in two forms:

1. Check permissions to access a resource: The Dispatcher calls a servlet /bin/permissioncheck on the Publish system, which either returns HTTP 200 or HTTP 40x — not allowed
2. The Dispatcher can ask the Publish system to render the page. This is the usual /content/mysite/home/… request.

(A) When the resource is public, the page is either rendered or served from cache. Nothing new here.

(B) When the resource is private, the Dispatcher first checks if it is cached. If so, it asks the Publish for permission (via /bin/permissioncheck) to deliver it to the user. If the Publish system allows access, the page is served from the cache.

(C) If the permission check denies delivering from cache, the Dispatcher issues a new resource request to the Publish system — see details below.

(D) If the resource was not cached, the request is directly passed through to the Publish to re-render and deliver the page. In this case, the Dispatcher acts only as a proxy.

Let’s take a closer look at two noteworthy points highlighted in the diagram:

(1) The Dispatcher makes a HEAD request to /bin/permissioncheck?uri=/content/mysite/home/…²³ to check for permissions. The request contains a parameter “uri” denoting the requested page. On the Publish side, a servlet is implemented that responds to this request.

The “permissioncheck protocol” is simple:

• Listen for HEAD requests.
• Take the content-path from the uri parameter.
• Test if the request would have access to the resource defined by the content path.
• If so, return an HTTP 200 response code.
• If not, return some 40x response code.⁴

(2) Requests that can not be served from the Dispatcher cache are handled by the Publish system. This can be either because:

• The resource is not in the cache
• The request failed the permission check

It took me a while to understand that the permission check does not differentiate between the states not in cache, not logged in, and no access. It only says, “I can’t handle this request: Do not deliver from cache.” And if the Dispatcher cannot deliver from cache, the Publish system needs to handle the request as an actual — uncached — live request to the original resource (e.g. /content/mysite/home/…).

The Publish system then decides what to do with that request, e.g.

• User is not logged in -> Redirect to a login page (see Part 2)
• User does not have access to the resource -> Return HTTP 403
• User does have access: Return the resource with an HTTP 200 — The Dispatcher then caches that rendition for future request.

—

¹ There is an early prototype in the making to implement PSC also in the CDN. Stay tuned if you want to learn more.

² The URL usually is /bin/permissioncheck — though this path can be configured.

³ This servlet must be implemented by you. There is no built-in permission check. AEM only defines the protocol.

⁴ It doesn’t matter which 40x code the servlet returns. All cases, be it 401 — Not Authorized or 403 — Forbidden, indicate to the Dispatcher that it must not deliver from cache but that the Publish needs to handle the request directly. I still implement my own permission check, returning different response codes (i.e., 401 and 403) to ease debugging — not because Dispatcher requires such differentiation.

Side note

I highly recommend reading my 2023 article on dispatcher debugging. Especially when implementing security, it is important to understand what is happening behind the scenes and avoid making (false) assumptions.

[embed]Debugging Your AEM Dispatcher Configuration When it comes to debugging the Dispatcher configuration, your options are limited: Introspection, or experimentation…medium.com

Example conversation

Below is a conversation I captured using a web debugging proxy between the Dispatcher and Publish.

I am calling a page /content/dummycorp-de/de/home/login, which is the root node of the area that requires login.

1. First line: At the time of the request, the page “login” was in the cache. Here, the user is not logged in. The permission check returned HTTP 401.
2. Second line: The Dispatcher patches the request directly to Publish.
3. Third line: The Publish finds I am not logged in and forwards to a configured login service with an HTTP 302. (The login service in this example is an external OIDC service. We cannot see the actual login, but we can see the response with the token to the “j_security_check” URL that creates the local AEM session.

4. Fourth line: After a successful OIDC login, the Browser would request the page again. We can then see the subsequent permission check call returning an HTTP 200 (as said before, the page was in the cache from a previous request).
   Red line: I then added a CUG (Closed User Group) to the page and re-published. I CUG, my test user was not a member of. So I would not get access.
   :

5. Line five: Reload the page while the user is still logged in. The permission check servlet recognizes that I am not in the specific CUG and returns an HTTP 403 (again, anything != 200 would have yielded the same result).
6. Line six: The Dispatcher transfers the request to the Publish system to deliver the page. The Publish returns an HTTP 404.

Side Note: You might have expected an HTTP 403 in the last response. But 404 is typical. Sling does not differentiate between “nothing here (404)” and “nothing here for you (403)”. This was a deliberate choice by the Sling team for security considerations. Sling does not want to convey any information when the user has no access. This is closely related to Sling's use of SEO-friendly URLs: Imagine you requested a page /product-launches/october/smartphone-V-512GB and received an HTTP 403 — Forbidden. Even without access, you could still infer that a 512GB smartphone launch is planned for October by guessing at URLs.

Configuring the Dispatcher

So much for the theory. Let’s configure the Dispatcher.

You need to configure at least those files:

Rewrite rules and HTTP Headers

conf.d/available_vhosts/dummycorp.vhost and conf.d/rewrites/dummycorp.rules

Cache configuration and permission check

conf.dispatcher.d/available_farms/mysites.farmand cache/mysites-rules.any:

farm configuration

## conf.dispatcher.d/available_farms/mysites.farm

/mysitesfarm {

  ...

  /virtualhosts {
    "www.dummycorp.de"
    ...
  }

  /renders { ... }
  /filter { ... }

  /cache {

    /rules {
      $include "../cache/mysites-rules.any"
    }


    /invalidate {
      # by default, no auto-invalidate
      /all  { /type "deny"  /glob "*" }

      # auto-invalidate for html pages
      /html { /type "allow" /glob "*.html" }
      ...
    }

    # enable PSC for known regions

    /auth_checker {

      # where the permissioncheck is configured on Publish
      /url "/bin/permissioncheck"

      /filter {

        # by default, content is public, does not need permissioncheck
        /default { /glob "*" /type "deny" }

        # the area "/login" _is_ protected and does require a permissioncheck
        /dummycorp-login-area {
            /glob "/content/dummycorp/de/home/login*"
            /type "allow"
        }

        ...
    }
  }
}

rules configuration

## cache/mysites-rules.any     

# Allow caching  
/default {
 /type "allow" /glob "*" 
}

# Don't cache csrf login tokens
/csrf {
  /type "deny" /glob "/libs/granite/csrf/token.json" 
}

...

vhost configuration — Keep the CDN from caching

So far, only permission-sensitive caching is implemented in the Dispatcher. We need to ensure the CDN does not cache the content. Otherwise, it could deliver protected content to any user — including logged-out users or users lacking proper privileges. On AEMaaCS, the CDN is based on Fastly. We can easily tell Fastly in an HTTP response header not to cache the login-protected area [3]:

## conf.d/available_vhosts/dummycorp.vhost

<VirtualHost *:80>
 ServerName www.dummycorp.de
 ...

 <IfModule mod_headers.c>
   <LocationMatch "^/content/dummycorp/de/home/login.*">
     Header set Cache-Control "no-store, no-cache, max-age=0"     
     # Alternative: 
     #Header set Surrogate-Control "no-store, no-cache, max-age=0"
   </LocationMatch>
   ...

 </IfModule>

  ...

</VirtualHost>

Note: You can use either the Cache-Control header or the Surrogate-Control header. Cache-Control is evaluated by the CDN and all downstream caches, including the browser cache. This is the most secure setting. Surrogate-Control is evaluated only by the Fastly CDN, so the protected page can still be cached in the browser. It depends on your own security requirements which option you want to choose.

URL shortening and expanding

Usually, you do not expose the full path on the public side. Instead of [https://www.dummycorp.de/content/dummycorp/de/home/foobar.html](https://www.dummycorp.de/content/dummycorp/de/home/foobar)you would want a shorter path like [https://www.dummycorp.de/foobar](https://www.dummycorp.de/content/dummycorp/de/home/foobar). The corporate name and the region are already encoded in the domain name. ‘/home‘ can be assumed to be the docroot ‘/‘. And the .htmlsuffix seems to bother some SEO people…

So, when someone calls a shortened URL yo need to expand it to the full path. This is important because/bin/ermissioncheck?uri=/content/… expects full paths. You cannot rely on path mapping from Sling.

## conf.d/available_vhosts/dummycorp.vhost

<VirtualHost *:80>
 ServerName www.dummycorp.de
 ...
 <IfModule mod_rewrite.c>
   RewriteEngine on
   Include conf.d/rewrites/dummycorp.rules
 </IfModule>
...

## conf.d/rewrites/dummycorp.rules

...

# add base path /content/dummycorp/de/home to urls:

RewriteRule "^(/?)$" "/content/dummycorp/de/home.html" [PT,L]
RewriteRule "^/(.*?)$" "/content/dummycorp/de/home/de/home$1.html" [PT,L]

...

Now, when a browser requests a shortened URL

[https://www.dummycorp.de/foobar](https://www.dummycorp.de/content/dummycorp/de/home/foobar)

The rewrite rule expands the path to

[https://www.dummycorp.de/content/dummycorp/de/home/foobar.html](https://www.dummycorp.de/content/dummycorp/de/home/foobar)

And before delivering from cache, the Dispatcher asks the Publish system via permissioncheckfor permission:

[/bin/permissioncheck?uri=/content/dummycorp/de/home/foobar.html](https://www.dummycorp.de/content/dummycorp/de/home/foobar)

Let’s have a coffee

Wow. You really made it to the bottom of this article. Congratulations on your endurance. You have earned yourself a hot cup of coffee. If you would like to invite me, you can do so at https://buymeacoffee.com/achimkoch.

Stay tuned for the next articles in the series.

Cheers

-achim

References

[1] https://experienceleague.adobe.com/en/docs/experience-manager-dispatcher/using/configuring/permissions-cache

[2] https://medium.com/@achimkoch/debugging-your-aem-dispatcher-configuration-9ed9a86ab970

[3] https://www.fastly.com/documentation/reference/http/http-headers/Surrogate-Control/