Migrating John Lewis Digital away from Ingress NGINX: Part 1

Overview

In this blog I talk about how we overcame the challenge of the Ingress NGINX project becoming end of life by migrating to Envoy Gateway and providing an abstraction to simplify configuration for our tenants and future proof against underlying changes to our internet ingress architecture.

In this part I talk about the background and the preparation we took; in the second part I talk about how we went about the implementation.

Background to JLDP

The John Lewis Digital Platform is a Kubernetes-based platform where our ~140 tenant services run microservice applications based on a paved road approach. This provides a simplified platform for teams to deploy their applications, so that they can concentrate on developing business functionality, while still being in control of their application operations.

Ingress Migration

In JLDP we’ve historically used the open source Ingress NGINX for our second tier of internet ingress, behind Google Cloud Load Balancers. Tenants created Ingress resources in their own namespaces, which would configure routes on their Ingress NGINX pods, exposing their microservices to the internet. We provided guidance on how to write these Ingress resources, but no restrictions, leading to an array of different configurations in the tenant namespaces.

The Ingress NGINX project announced that they were no longer going to be developing or supporting the software, leaving us with a matter of months to migrate all of our tenants to a different solution.

Analysis of existing configuration

Before we could choose a replacement, we needed to understand the scale of the complexity in our cluster and whether there were any blockers to migration, or particular requirements for our replacement.

We did this by pulling out an extract of all the Ingress resources in our clusters using kubectl and writing a script to summarise, providing counts of all nginx annotations and unique values of those we thought could be a problem (e.g. nginx.ingress.kubernetes.io/configuration-snippet, which allowed for arbitrary, non-standard configuration that was impossible to migrate automatically.)

We determined that the number of people using advanced features or with logic in their ingresses was at a level that was practical to migrate, and we also worked with teams to suggest workarounds and migration of logic into their applications. We have an engineering principle called Smarts in the Node not the Network which we used to support this.

Choosing a replacement

We knew that the more modern Gateway API had been produced to cover the Ingress functionality, so we were keen to move to an implementation of this.

We knew we were short on time, so completed a paper-based evaluation rather than testing all the available options ourselves. One of our team members had been to a talk about Envoy Gateway at Kubecon and was impressed by the array of configuration options available. We are also heavy users of other technologies based on Envoy Proxy, so have the benefit of familiarity with it. These factors made it our lead option.

We were impressed by the documentation and having spent time reviewing John Howard’s detailed evaluation of Gateway implementations, we decided that Envoy Gateway was going to be the option that had the fewest compromises for our needs.

Related changes

As we had decided to migrate from the Ingress to Gateway API within Kubernetes, and teams had custom nginx annotations on their Ingress resources, we knew that we would need all teams to make some changes. We were going to offer them as much support as we could, but wanted to make their migration process as simple as possible, and take the opportunity to make improvements that would make our lives easier if we needed to do a similar migration in the future.

Ingress Abstraction

In JLDP we provide our consumers with a custom Kubernetes resource we call the Microservice, which is used as the basis of automatically creating the Deployment and other associated resources based on our opinionated rules.

Most of our tenants have simple ingress requirements, and we took the opportunity to extend the Microservice spec, automatically generating HTTPRoute resources for teams. An example of the YAML that teams have to add is:

apiVersion: crd.jldp.com/v1
kind: Microservice
spec:
  routesFromInternet:
    edgeSecurityPolicy:
      type: IdentityAwareProxy
    hostnames:
    - my-very-cool-application-((ENV)).jldp.com
    pathPrefixes:
    - /my-application-path

This allows them to convert their Ingress to a simple spec that they add to their Microservice resources, while giving us the benefit that we can control future implementation changes through our Kubernetes controller code without requiring teams to update their own definitions.

For the few teams that have more complex requirements, they can define their own custom HTTPRoutes.

Identity-Aware Proxy

We previously offered teams a basic way to restrict their services so that they could only be accessed by a user with an appropriate Google account. This used to be based on a discontinued open source project and some complex custom Kubernetes wiring which was difficult to maintain.

As we were already making drastic changes, we decided to switch to using GCP’s native Identity-Aware Proxy to provide this functionality and offer it up in our abstraction (as you can see above).

Cloud Armor simplification

We had a number of services who were using Google Cloud Armor to apply additional protection to their ingress endpoints. As this had started as a custom configuration for a few services, we owned the configuration for each of our tenants, defining which resources were protected and which policy applied to them.

We were already giving our tenants more control over their load balancing to allow them to use Identity-Aware Proxy so we decided to follow a similar self-service model for Cloud Armor. This reduces the central complexity and allows this configuration to be defined in the same places as the routes themselves.

Design

Architecture

We deploy Envoy Gateway in Gateway Namespace Mode, meaning that each tenant has their own copy of the Envoy Gateway Proxy deployment. This provides data-plane isolation between tenants, while still requiring only a single control-plane deployment. The ingress routes that they define in their Microservices (or custom HTTPRoutes) target their Envoy Gateway, creating proxy configuration. We call these “Tier 2” routes. We have allowed tenants to choose which features of the Google Cloud Load Balancers they use as part of defining their HTTPRoute by adding an annotation, so we also need to configure the GCLB based upon this information. As we use GKE Gateway we just needed to create HTTPRoute resources targeting a different Gateway. We do this with a custom Kubernetes controller which watches for “Tier 2” HTTPRoutes and creates a corresponding “Tier 1” HTTPRoute.

We stop tenants from modifying resources in their namespace that are owned by Platform (such as the Tier 1 HTTPRoutes and all the GKE Gateway configuration) using a combination of Kubernetes RBAC and software to enforce security policies.

Migration process

We wanted to give our tenant teams full control over their migration process on a per-route basis to ensure that they could manage this at a suitable time for their services and allow them to roll back themselves if they had any issues.

We achieved this by adding code in our custom Kubernetes controller so that it added two backends to the new Tier 1 routes it created — the first was the new Envoy Gateway Proxy and the second was the old Ingress Controller. We also made sure that the new T1 routes took priority over the old Google Cloud Load Balancing setup by being more specific. In most cases this was straightforward, but we needed to identify our most complex tenants and design the new HTTPRoutes to ensure they always took priority.

We controlled the weights of the two backends by allowing tenants to set an annotation on their Microservice or custom HTTPRoutes to control the percentage of migrated traffic. Our custom Kubernetes controller read this and set the weights of the two respective services.

If the annotation was not present, or set to “100”, we would send all traffic to the Envoy Gateway Proxy. If it was “0” we would send all traffic to Ingress NGINX, and percentages between those values behave as you would expect.

We also created dashboards so teams could monitor their own migration.

Read on to Part 2 to discover how the migration went.