Don’t use OSGi configs in AEMaaCS

AEM services are built on top of the OSGi Service Component Runtime (SCR). A feature of the SCR is that you can divide the code from its configuration.

Thus, one party can develop the code, another party can apply adapt the behavior via a limit set of configuration options.

It is necessary to separate code and configuration when two different parties perform development and installation. I.e. when a service is built by a product vendor (Adobe) and the software is installed and configured by their customer or agency (you).

However, there is less value in separation when the same party provides both artifacts. In this article I will point out a few reasons when not making your own code configurable and a few reasons when not to provide a configuration for built-in services.

TL;DR:

• In contrast to a product vendor, you deploy your software only in one environment. No need to make it configurable. Just because Adobe does it does not make it “best practice”. Adobe’s use case is different.
• OSGi configs are over-engineered for simple use cases, where only one variable changes.
• Not having OSGi configs makes your code more coherent, cleaner, and less error prone.
• Changing an OSGi config on AEM as a Cloud Service requires a deployment anyway — you can deploy a static variable in code in the same time.
• Adobe engineering reserves a few configuration files for their own purposes. Providing configs for those can led to conflicts. Better use environment variables in these cases.

Some History

Dividing responsibilities

The idea of separating code and configuration is as old as modern hard- and software. The same goes for dividing responsibilities. However we must put this into perspective. What is best practice for a piece of COTS software (commercial of the shelf) not necessarily hast to be applied to custom software that is deployed only once.

This can also have negative effects. E.g. around the year 2000, Sun specified the Java Enterprise Edition (JEE). They introduced a host of roles to separate concerns and specialities. Among them were Product Providers, Enterprise Bean Developers and Application Deployer. The former would develop, the latter provide the configuration adapted and tuned to the environment and infrastructure. Back then, the large number of specialized roles was often criticized as being over-engineered, off from reality and simply impractical — especially for smaller teams.

Dividing responsibilities also requires more documentation and communication — both disciplines that have few friends in the development community — and also add cost.

Enter: DevOps

Around 2008, DevOps was coming into vogue. The main idea was to re-unite the roles of developers and operators to reduce the communication and governance overhead and to encourage shared responsibilities for the running system (not only shipping software).

Being a merely project governance approach first, it soon took up speed when sometime later deployment automation tools and container technologies entered the market. Software and configuration could be bundled into containers, a DevOps team could ship whole software-defined systems and did no longer have to manually install releases on running systems.

Where OSGi fits in

OSGi was started at the same time as JEE and thus pre-dates DevOps. It shares a couple of JEE’s ideas. To name a few, you would deploy to an already running application server and you’d separate code and the configuration.

As you deploy to a running server, the promise was that deployments are seamless and would not require any downtime — ideal for a web platform, where downtime means not servicing customers — and not generating revenue.

AEM and OSGi

This made OSGi an excellent choice as a foundation for AEM: No downtimes and a division between Product Provider (Adobe) and Application Developer (you).

AEM also added some interesting features to the mix: E.g. Run-modes: An environment would run in a certain mode like dev/stage/production or author/publish. You then deploy all configurations for all modes and the environment picks the one it needs to apply.

OSGi Then and Now

OSGi’s concepts were working great … at the time. But reality and complexity caught up eventually. A few observations:

Run modes: The way configurations are handled by run modes has few side-effects.

• Pro: Configurations are created by developers and shipped / deployed with the software. This encourages a DevOps-like deployment process; shipping the proper configuration is a joint responsibility.
• Con: Multiple versions of the same configuration file. Let’s assume a service requires 10 configuration variables. But only 2 differ between the environments. Developers often copy the configurations for all environments and changing the diverging variables — initially. But updating the versions with additional variables required manual synchronisation and oft is neglected.

Configuration default fallback cascade: Separating the service configuration from code requires you to define fallback for undefined values. This is possible with OSGi configs — but not perfectly documented: A configuration value now can be either

• code — intrinsic ( if (value==null) )
• defined as explicit default in the OCD, or
• defined explicitly in the service configuration
• overloaded for a particular run mode
• manually overloaded in the OSGi console

I always find it a nightmare to code-review and debug not knowing at build-time what value AEM will pick.

Unicorn-ness: Always deploying on top of a running application server leads to creating unicorn environments.Systems become the sum of all past deployments and cannot easily be re-created from scratch.

Validity of pre-prod tests: Desynchronized configurations and only loosely coupled lifecycles compound to a bigger problem: Production and pre-production systems no longer behave the same. And the validity of tests decreases.

Downtime: As AEM grew more complex, restarting individual services upon configuration change took longer and became noticeable in the frontend — as ISE 500 errors during the deployment. Depending on the service and configuration this could be a couple of seconds. More central services can even take minutes to restart. Note, that a service always restarts when you provide a new configuration.

Staggered deployments: These points lead to a practice of “staggered deployments”. You would remove a couple of publish nodes from the load balancer, deploy and smoke test and if all looks good, you’ll deploy the other nodes. This makes deployments a more elaborate undertaking than simply uploading and publishing a content package. Yes — this can be automated, but it adds to cost & complexity of the system.

Separation of responsibilities? In AEM projects, I have rarely seen a separation of development and operations. It was always the same team developing and maintaining the platform. A nominal Ops team might provide 24/7 shifts but rarely can do more than monitor and restart according to run-books. Thus, the bulk of the responsibility lies in the developer’s field anyway.

Enter: AEM as a Cloud Service

AEM as a Cloud Service acknowledges and addresses most of the issues that come from AEM outgrowing its original paradigms:

• Environments are always built from scratch, which prevents platforms from evolving into unicorns.
• AEM is automatically blue/green-deployed: A pristine environment is created. When it is up and running (and caches are warmed up), the load balancer switches to the new environment and discards the old one.
• You can longer configure OSGi services on the running environment. You no longer “fix” things in CRXDE. This puts the responsibility in the hands of the Dev(Ops) team. It does no longer evoke the false perception that some Ops team should apply changes on the life system.
• OSGi configurations now support interpolation of environment variables. See below:

OSGi Configuration with Interpolation

Variable interpolation in AEMaaCS made handling of OSGi configurations much easier. [1]

If you need to configure dev, stage, and prod differently, you no longer deploy three separate config files per run-mode but one file that has a placeholder.

Example:

// .../config/org.apache.sling.security.impl.ReferrerFilterAmendmentImpl~datev.cfg.json

{
  "allow.empty": false,
  "allow.hosts": [
  ],
  "allow.hosts.regexp": [
    "$[env:HOSTS_REGEXP;default=https://localhost:450?]"
  ],
  "filter.methods": [
    "POST",
    "PUT",
    "DELETE",
    "COPY",
    "MOVE"
  ],
  "exclude.agents.regexp": [
    ""
  ]
}

dev, stage and prod require a different hostname. But instead of providing three different configurations, there is only one and the actual value that needs to changed is defined in Cloud Manager as an environment variable.

Local development environments either use the default or define the value in the. crx-quickstart/bin/start script:

#!/bin/bash
#
# This script configures the start information for this server.
#
# The following variables may be used to override the defaults.
# For one-time overrides the variable can be set as part of the command-line; e.g.,
#
#     % CQ_PORT=1234 ./start
#

export HOSTS_REGEXP="http://localhost:450?"
#      ^^^^^^^^^^^^

# TCP port used for stop and status scripts
if [ -z "$CQ_PORT" ]; then
 CQ_PORT=4502
fi

# hostname of the interface that this server should listen to

...

How to Configure Built-in Services

In the past you provided OSGi config files for built-in services if you need to adapt something. In AEMaaCS often this is not required anymore. You simply set a variable in Cloud Manager as explained above.

As Adobe now hosts the Environments, it is less and less likely that you want to configure environment-specific values.

In fact, there are cases where it must be avoided: E.g., when the configuration is “claimed” by Adobe engineering to be configured for different client environments.

I would loosely classify built-in services in AEMaaCS as follows:

• Keep your fingers off: Check the service in the developer console. If the values “smell” like Adobe infrastructure, don’t touch.
• Free to configure: If a service is not instantiated with a configuration file, you probably can provide a configuration.
• Free to adapt: The service is already instantiated with a set of default values and interpolation variables. In these cases, set the environment variables but DO NOT provide a config file. You might step on Adobe’s feet who already provided a configuration and might want to change it in the future… which they can’t if you overlaid the config. Always check the developer console first if the config contains interpolation variables. If it does, only override those. Only provide a configuration if this is documented (or you are told to do so by Adobe).
• Third party infrastructure: If a configuration is for a third party service, such as an SMTP server, you — of course — have to configure it.

Examples:

The ExternalizerImpl can be configured via environment variables. No need to provide a config:

- pid: "com.day.cq.commons.impl.ExternalizerImpl"
  properties:
    externalizer.domains:
    - "local $[env:AEM_EXTERNALIZER_LOCAL;default=http://localhost:4502]"
    - "author $[env:AEM_EXTERNALIZER_AUTHOR;default=http://localhost:4502]"
    - "publish $[env:AEM_EXTERNALIZER_PUBLISH;default=http://localhost:4503]"
    - "preview $[env:AEM_EXTERNALIZER_PREVIEW;default=http://localhost:4503]"

BearerAuthenticationHandler config: The config below is constantly changing and smells heavily like infrastructure. Overriding the config would probably have broken the system:

- pid: "com.adobe.granite.auth.oauth.impl.BearerAuthenticationHandler"
  properties:
    auth.bearer.sync.ims: true
    disable:
    - "$[env:bearerHandlerDisable;default=false;type=boolean]"
    oauth.bearer.configid: "configid"
    oauth.clientIds.allowed:
    - "$[secret:ims/clientId]"
    - "cc-europa-desktop_0_1"
    - "cc-europa-desktop_1_0"
    - "cc-europa-desktop_2_0"
    ...
    - "cc-europa-desktop_10_0"
    ...
    - "$[env:ADOBE_PROVIDED_CLIENT_ID;default=disabled]"
    - "$[env:ADOBE_PROVIDED_CLIENT_ID_02;default=disabled]"
    - "FrameioAEMPromiseTokenConsumer"
    oauth.jwt.support: true
    online.validation.with.post: true
    path: "/"
    service.ranking: -10
    use.ims.offline.token.validator: true

However, Adobe might tell you to override the interpolated variavle ADOBE_PROVIDED_CLIENT_ID.

In CRX there are areas that are marked as protected — you can’t override them. I don’t know if the same level of protection is applied to OSGi configs, too. If in doubt — keep your fingers of.

Configure Your Own Services

Let’s get back to if and how you want to make your own services configurable. Of course you can do that at will. But I would argue, that you not always require a full-blown OSGi configuration with interpolation variables.

Lightweight Configuration

My hypothesis is, you no longer need OSGi configs at all — at least not to configure your own services and rarely to configure AEM services.

You’ll create much cleaner code if you provide necessary configuration in-line in the class files as static variables.

Only because we can provide a configuration for an OSGi service, does not necessarily mean we must and it’s the best way. Oftentimes, this is over-engineering.

• Providing OCDs (Object Class Definitions) to the make a service configurable adds clutter and makes the code less readable.
• In code reviews, you need to check multiple locations to determine what the actual configured values of a service are — the config file, the run modes, and the defaults). This adds to the cognitive load.
• There is no point in providing means to configure a service at runtime because this no longer is supported in AEM as a Cloud Service anyway. For local development and tuning, building, and deploying via Maven a bundle (via autoInstallBundle) with static member, variables is fast enough. Plus: You save time not having to develop and testing the OCD.

Important: Changing environment variables requires only a restart of a Cloud Service environment. Deploying an OSGi config requires a full build and takes much longer.

Example:

Let’s say, we have a service that needs to know on which environment it is running. The “classic” approach with an OSGi config could look like so:

package com.mysite.core.services;

import org.osgi.service.component.annotations.Activate;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.ConfigurationPolicy;
import org.osgi.service.metatype.annotations.AttributeDefinition;
import org.osgi.service.metatype.annotations.Designate;
import org.osgi.service.metatype.annotations.ObjectClassDefinition;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@Component(
    service = MyService.class,
    immediate = true,
    configurationPolicy = ConfigurationPolicy.OPTIONAL
)
@Designate(ocd = MyService.Cfg.class)

public class MyService {

    @ObjectClassDefinition(name = "Basic OSGi Service Configuration")
    public @interface Cfg {
        @AttributeDefinition(
            name = "Environment",
            description = "The environment from which to greet from."
        )
        String environment() default "AEM";
    }

    private Cfg cfg;
    private static final Logger log = LoggerFactory.getLogger(MyService.class);

    @Activate
    protected final void activate(final Cfg cfg) throws Exception {
        this.cfg = cfg;
        log.info("Hello {}", cfg.environment());
    }
}

That’s already quite a mouthful. And honestly quite scary for fresh AEM developers. Not only do we have to define the service itself, but also provide and reference the ObjectClassDefinition (OCD) to enable configurability.

Not to forget, we need to provide the according run-mode configurations[2].

// .../config.dev/com.mysite.core.services.MyService.cfg.json

{
  "environment":"DEVELOPMENT"
}

// .../config.stage/com.mysite.core.services.MyService.cfg.json
{
  "environment":"STAGE"
}

// .../config.prod/com.mysite.core.services.MyService.cfg.json
{
  "environment":"STAGE"
}

If we have multiple services and each requires a configuration this can add up to quite a number of artifacts:

Ok — earlier I said, this can be achieved by interpolating from the environment, so one config file would be enough:

// .../config/com.mysite.core.services.MyService.cfg.json
{
  "environment":"$[env:ENVIRONMENT;LOCAL]"
}

This can already reduce the number of files:

Central Configuration

In [3] I read an interesting idea to move all configurations into a generic utility class that is then used by specific services. Can reduces the number of configuration files even more:

This reduces the number of config files even more. But it would make the architecture less modular. And the modules can not easily be re-used. But coming back to my earlier argument. Re-using modules is more a thing for product vendors and library maintainers, less for custom AEM projects.

Non-OSGi Configuration

But here is the thing: In Java, you can consume environment variables directly. You don’t require an OSGi configuration for that. This seems to have been forgotten by devs (like me) that were “indoctrinated” on OSGi for a longer time.

Variables you set in Cloud Manager are just that. Environment variables. Knowing that makes the code so much easier to read:

package com.mysite.core.services;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.apache.commons.lang3.StringUtils;

@Component(
    service = MyService.class,
    immediate = true)

public class MyService {
    private static final Logger log = LoggerFactory.getLogger(MyService.class);

    @Activate
    protected final void activate(final Cfg cfg) throws Exception {
        log.info("Hello {}", 
                 StringUtils.defaultIfEmpty(
                     System.getenv("ENVIRONMENT"), "LOCAL"));
    }
}

Note: This approach works well, but it is not the official protocol. Adobe might change the way AEM communicates with the cloud environment in the future. I find this very unlikely, though. Still, if that makes you feel uneasy, but still want to have cleaner code, choose one of the other options of this article.

Zero Configuration

But let’s face it: Often we don’t need any configuration at all. We sometimes introduce some to be able to fine-tune later. Or we defer decisions how to configure at the time of implementation because we lack details. In such cases, I'd argue, you don’t need any configuration at all.

Simple static variables can often be enough:

package com.mysite.core.services;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@Component(
    service = MyService.class,
    immediate = true)

public class MyService {

    private static final String ENVIRONMENT = "World";

    private static final Logger log = LoggerFactory.getLogger(MyService.class);

    @Activate
    protected final void activate() throws Exception {
        log.info("Hello {}", ENVIRONMENT);    }
}

Zero Service

If we reduce the “service” that much, you might want to consider not to implement the ”service” as an SCR component at all. A simple static helper class sometimes is enough. You will be deploying your code as a monolith anyway. Often there is no need to create a service as an OSGi service that could be plugged and removed at runtime:

package com.mysite.core.services;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public static class MyService {

    private static final String ENVIRONMENT = "World";
    private static final Logger log = LoggerFactory.getLogger(MyService.class);

    protected static final void greet(){
        log.info("Hello {}", ENVIRONMENT);
    }
}

Rules

As my — personal - rules of thumb:

• Configure externally, when there are differences between the environments (laterally).
• Configure externally, when the same service is instantiated multiple times in one environment (i.e. via a factory).
• If a service requires more than three configuration variables or when the total number of environment variables moves toward the mark 50, consider aggregating variables in classic OSGi config files.
• Use static variables for environment-independent configuration, for stuff might evolve over time (vertically).
• Implement a OSGi services when it needs to consume other services — your own or runtime services.
• Helper/Utility classes can be static Java classes — if you can run the methods in context of the calling service (which passes a ResourceResolver as method parameter).

Conclusion

Embrace clean coding.

YAGNI — You ain’t gonna need it (yet)

Do not try to anticipate, what someone else might want to do with the code in the future. If configuration is not required now — don’t provide a means for it. Software is mendable. You can add configurability if and when it is required.

KISS — Keep it simple and stupid

Do not over-engineer. If the same problem can be solved in a simple way, do. Avoid super elaborate techniques. Think of the people who are going to maintain the code. They might not be as smart as you are.

There is no One-size-fits-all

Know the pros and tradeoffs of each pattern. Don’t blindly follow some “best practice” but understand what you are doing and make informed decisions.

References

[1] https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/implementing/deploying/configuring-osgi

[2] https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/implementing/deploying/overview

[3] https://medium.com/@toimrank/aem-as-a-cloud-service-osgi-configuration-secret-and-environment-variables-667a0ddfb1d5