How Nelm compares to Helm 4: Current differences and future plans

The recent release of Helm 4 provides an excellent opportunity to compare it with the alternative we’ve been developing in werf, Nelm. This article examines the new features of both projects, details their differences, and outlines the future roadmap for Nelm.

Helm 4 evolution

Helm 4 introduced a pack of new features for the Cloud Native community. Perhaps the most significant user-facing changes were there adoption of Kubernetes Server-Side Apply (SSA) instead of the 3-Way Merge (to resolve issues with incorrect resource updates) and kstatus-based resource watching. The rest of the new features are mostly focused on reducing technical debt.

While implementing SSA is a noteworthy achievement deserving of its own release, the community seemed to expect more from Helm 4. Among the most popular feature requests were an alternative to Go templates and improved handling of Custom Resource Definition (CRD) deployments.

The pace of Helm’s development accelerated leading up to the new release. However, given the tremendous adoption of Helm in the industry and strict backward compatibility requirements, further significant architectural changes will likely be postponed until the next major Helm release.

How Nelm differs from Helm 4

Nelm is a modern alternative to Helm 4, focusing on introducing new major features while maintaining backward compatibility with Helm charts and releases.

Nelm was created in werf, following the tool’s users’ needs for improved and more powerful deployment. Later, it became a standalone project that can be used on its own (without werf) to deploy Helm charts to Kubernetes. Under the hood, Nelm uses parts of the Helm codebase, but its most troublesome parts, particularly the deployment engine, have been rewritten from scratch.

While Nelm has supported Server-Side Apply for a long time, it had more user-facing features to offer and continued to evolve. For instance, it recently introduced resource lifecycle management via the werf.io/delete-policy, werf.io/ownership, and werf.io/deploy-on annotations. Let’s examine the key differences between Nelm and Helm 4.

1. Deploying CRDs

Helm recommends placing Custom Resource Definitions (CRDs) in the chart’s crds directory. However, resources in this directory cannot be updated and are only deployed during the initial release installation. The crds directory is ignored during subsequent helm upgrade operations.

As a workaround, some users deploy CRDs as regular resources by putting them in the templates directory. However, such an approach makes it harder to maintain the deployment order. On top of that, since CRD manifests are so large, you are risking hitting the release Secret’s size limit.

To get around these issues, some well-known Open Source charts even create a separate subchart just for deploying CRDs.

With Nelm, you just move your CRDs in the crds directory. Nelm features the fully-fledged deployment mechanism for this directory, so CRDs get updated and deployed every time you run an upgrade.

2. Defining deployment order

In Helm, deployment order is typically defined using Helm hooks. This method is adequate for simple Jobs that need to run before or after a rollout.

But what if a Job requires a Deployment to run? Or what if a Job must be run halfway through the release? No standard solutions exist in Helm for these scenarios.

Before each rollout, Nelm builds a graph of operations with the Kubernetes cluster’s resources, which defines their deployment order:

It also provides a simple yet powerful way for setting this order: the [werf.io/deploy-dependency](https://github.com/werf/nelm/?tab=readme-ov-file#werfiodeploy-dependency-id-annotation) annotation. This annotation creates a dependency between operations in the graph, thus defining their rollout sequence. For example, the following configuration:

kind: Deployment
metadata:
  name: backend
  annotations:
    werf.io/deploy-dependency-db: state=ready,kind=StatefulSet,name=postgres

… means that the backend Deployment will only be created or updated after the postgres StatefulSet is created/updated and ready. The graph will look like this:

The werf.io/deploy-dependency annotation works for both regular resources and hooks. We plan to add support for specifying dependencies on entire charts in the future.

As an alternative, Nelm also features the [werf.io/weight](https://github.com/werf/nelm/?tab=readme-ov-file#werfioweight-annotation) annotation. It works similarly to helm.sh/hook-weight but applies to both hooks and regular resources.

There’s also the [external-dependency.werf.io/resource](https://github.com/werf/nelm/?tab=readme-ov-file#idexternal-dependencywerfioname-annotation) annotation, which lets you specify a dependency for resources outside of the Helm release, such as a Secret that an operator creates.

Of course, regular Helm hooks and their weights are also supported.

3. Resource lifecycle

Helm lets you prevent a resource from being deleted using helm.sh/resource-policy: keep and control when hooks are deleted using helm.sh/hook-delete-policy. But what if you need to deploy an immutable Job mid-release? Or clean up a regular resource after its deployment? Or manage the same resource across different releases?

We recently added to Nelm a whole new set of features for managing resource lifecycle:

1. The [werf.io/delete-policy](https://github.com/werf/nelm/?tab=readme-ov-file#werfiodelete-policy-annotation) annotation, which is similar to helm.sh/hook-delete-policy, allows a resource to be recreated instead of updated (before-creation), recreated only upon encountering a “field is immutable” error (before-creation-if-immutable), or deleted after a successful (succeeded) or failed (failed) deployment. This annotation, like all others in Nelm, applies to both hooks and regular resources.
2. The [werf.io/ownership](https://github.com/werf/nelm/?tab=readme-ov-file#werfioownership-annotation) annotation enables hook-like behavior for regular resources. Specifically, it prevents applying or validating release annotations for the resource, and it protects the resource from deletion if it has been removed from the chart or if the release itself is being deleted.
3. Another annotation, [werf.io/deploy-on](https://github.com/werf/nelm/?tab=readme-ov-file#werfiodeploy-on-annotation), allows rendering a resource only during a release install, upgrade, rollback, or uninstall, similar to what you can already do with Helm hooks. Still, using this annotation does not convert the resource into a hook.

With these annotations, it is possible to replicate the behavior of a Helm hook without formally declaring one. For example, this hook:

metadata:
  annotations:
    helm.sh/hook: pre-install
    helm.sh/hook-delete-policy: before-hook-creation

… is similar to the following non-hook resource:

metadata:
  annotations:
    werf.io/deploy-on: install
    werf.io/delete-policy: before-creation
    werf.io/ownership: anyone

In general, we recommend Nelm users avoid using Helm hooks when authoring charts. This simplifies charts, allows for more flexible resource behavior, and accelerates rollouts by eliminating the separate hook deployment phase. However, using hooks may still be justified if maintaining compatibility with vanilla Helm is a requirement.

4. Advanced resource tracking

Helm 3 included a basic mechanism for waiting for certain regular Kubernetes resources to become ready. Helm 4 replaced it with kstatus, which improved readiness detection accuracy, but did not introduce any fundamental changes.

Nelm features its own advanced resource tracking system. Compared to Helm 4, it:

• is more accurate than kstatus at detecting when a resource is ready;
• can track not just readiness, but also whether a resource exists or not, and can detect and react to errors like failing probes;
• supports readiness tracking for popular Custom Resources with manually defined rule sets;
• determines the readiness of other Custom Resources heuristically, which works for most resources (no false positives);
• displays real-time status, errors, logs, and events for resources in the terminal during deployment.

Detailed Kubernetes resources’ tracking while installing a release via Nelm

Tracking requires no initial configuration but can be fine-tuned or disabled via command-line flags and annotations.

5. Encrypting values.yaml and other files

Helm doesn’t have built-in support for encrypted files in a chart; this functionality is provided by the helm-secrets plugin.

Nelm, on the other hand, comes with out-of-the-box support for encrypted values files and any other encrypted files in the chart’s secrets directory. Working with secrets in Nelm is easier than using the helm-secrets plugin.

Generate a secret key and create an encrypted values file:

NELM_SECRET_KEY=$(nelm chart secret key create)
nelm chart secret values-file edit secret-values.yaml

After that, you can use the encrypted values just like any other values:

# templates/secret.yaml
kind: Secret
stringData:
  mySecret: {{ .Values.mySecretValue }}

nelm release install -n foo -r bar

On top of that, Nelm can encrypt arbitrary files within the chart’s secrets directory.

6. Release planning

Nelm natively implements an analog of the helm diff plugin for Helm. The nelm release plan install command accurately displays the changes that will be applied to the Kubernetes cluster’s resources during the next rollout.

The output is precise as it is based on the plan of operations with resources, which is devised before every deployment. On top of that, unlike helm diff, this plan is based on resource updates performed via Server-Side Apply, not a 3-Way Merge.

We’re also working on a way to create and save a plan with a single command (nelm release plan install --save-plan) and then pass it to another command (nelm release install --use-plan). This means you can approve a plan and be certain that Nelm will not perform any unplanned actions. This workflow cannot be implemented with Helm and helm diff.

What’s missing in Nelm

First of all, Nelm does not support Helm 3 CLI plugins. They depend on the Helm CLI, including its command structure, options, and even on the way the logs are rendered. Achieving compatibility would require rewriting a significant portion of the Helm codebase, which is time-consuming and seems to be a pointless task. Instead, we are implementing the functionality of the most popular plugins natively within Nelm (e.g., helm diff and helm secrets).

Secondly, Nelm lacks support for post-renderers. Instead, we’ll introduce a replacement for Go templates (more on that below) and provide out-of-the-box resource patching, eliminating the need to install external plugins or configure anything. The reasoning behind this approach is detailed in issues #54 and #115.

Currently, Nelm cannot be used with Argo CD or Flux. We will address this via a Nelm operator, with its Custom Resources being deployed via Argo CD, Flux, or any other GitOps tool.

Finally, tools like Helmfile and Helmwave are not compatible with Nelm. We will likely resolve this by implementing a native Nelmfile accessible right from the Nelm CLI. The Helmwave project considers switching to Nelm itself.

What’s next for Nelm after the Helm 4.0 release

Nelm serves as the deployment engine for werf, a tool currently used in over 20,000 projects. On top of that, Nelm is actively being integrated into other products, such as the Deckhouse Kubernetes Platform. Being such an essential building block secures a solid future for Nelm, thanks to our commitment to further developing it.

The Helm 4.0 release didn’t really change anything. Thanks to its initial focus on bringing new capabilities to those in need, Nelm is still far ahead in features and improvements, and we expect this lead to grow. Over the past year, we have stabilized Nelm v1, refactored the entire codebase, and added many new features. We are also excited to have two new full-time developers joining the Nelm team very soon and to see an increasing community engagement in the project development.

Future plans

Over the next six months, we intend to release Nelm v2, migrate to the Helm 4 codebase, and release the Nelm operator for Argo CD and Flux integration.

Plans for the next year include an alternative to Go templates (our current proposal involves using TypeScript for that), chart patching, and downloading charts directly from Git.

We will continue to actively develop Nelm, just as we have been developing and supporting werf for nine years. You can learn more about Nelm and try it out in our GitHub repository.