Giving AI Hero new Superpowers with Kubernetes

AI Hero has always been a fully-containerized application. However, it was deployed primarily on top of EC2 instances. With our migration to Kubernetes, AI Hero has become much more powerful — we’ve improved the speed of our services, our platform has become super-scalable, we have a faster release cycle, and it has become enterprise-ready as it is easily deployable on-prem.

Update: Declarative MLOps — Streamlining Model Serving on Kubernetes: Here’s a recording of a talk I gave that summarizes this article as well as includes other best practices to build your own containers and use GitOps to deploy them using CI/CD.

[embed]Declarative MLOps — Streamlining Model Serving on Kubernetes

If you’d like to read more premium content like this from me and other authors across Medium, please consider becoming a paid subscriber using this link — https://medium.com/@rparundekar/membership

From a User Experience perspective, this change has allowed us to introduce “Superpowers”. Consider data uploaded to AI Hero as “data under construction” — we stage your business-critical operational data like your CRM, Customer data, App data, etc. The user then gets a spreadsheet-like interface they can use to clean, enrich, and enforce data quality for their operational needs. Think of it as an Airtable/Google Sheets-like interface, but each column in the spreadsheet is powered by AI.

A screengrab of a CRM data cleanup in progress — errors are flagged y AI Hero with an orange alert and cells auto-corrected with AI Hero’s Superpowers are marked with a lightning bolt ⚡️ (Note: It’s fake data)

The (ever-growing) list of AI Hero’s Superpowers look like this:

• Reformat data and flag missing or incorrect values based on “expectations” on the schema and range of values expected of the data.
• Normalize logical data types like addresses, phone numbers, emails, URLs with automated checks.
• Compute derived attributes on-the-fly.
• Update outdated values automatically with 3rd party integrations (e.g. update CRM details using Apollo, etc.)
• Use state-of-the-art AI like Large Language Models (e.g. OpenAI’s GPT-3, etc.) to enrich data.

Photo by Growtika on Unsplash

The K8s-native Architecture

In a previous blog, we published about AI Hero’s technical architecture. In this article, we explain the changes we’ve made to it to become Kubernetes Native. You can see the architecture below.

Our Previous Cloud-native Architecture

Our new architecture looks like this:

The New Architecture — Built on Kubernetes

Before we jump into how AI Hero’s new Superpowers are implemented on top of this architecture — let me briefly describe the underlying changes that enable these superpowers. With our new architecture, we made a three important improvements over the previous one.

First, all our microservices are deployed as Pods. This gives us compute scalability. If the number of customers actively using the platform increases, our pods can autoscale as the workload increases using kubernetes’ Horizontal Pod Autoscaling (HPA).

Second, our data storage is partitioned for each project. Instead of having one schema-less database that stores all our customer data (sorted by projects), we create one database for each customer project. For uploads and larger files, we also This makes data access faster. Additional advantage from both of these changes is that it also allows us to “forget” customer data on demand — delete the project, and the project folder in the bucket and the project database is deleted.

Third, our processing has become more event-driven. While we still rely on a hybrid model of processing (RESTful / Event-driven), our non-instantaneous workloads have become eventful. Previously they were running as “jobs” on a queue. With event-driven paradigm, our ability to process data has parallelized — we can cascade updates to the customer data with each “Superpower” doing its own thing.

AI Hero’s New Superpowers

The Superpowers listed above and others can be grouped into three broad categories.

Schema Superpowers — AI Hero can enforce “data contracts” on data imported into, under construction, and exported out of AI Hero. When users define the logical data type (e.g. by using a data type like first_name instead of SQL-likevarchar2) AI Hero can automatically check that the attribute values for all the entity data match expectations for this data type. It can also take advantage of the nature of certain data type (e.g. AI Hero can apply its knowledge of nicknames to resolve duplicates in a CRM , or fix user entered data like URLs by resolving redirects). Additionally, since attributes are grouped by the entity schema, we can enforce some dependencies between these (e.g. we’d know that first_name and last_name are the name and we can use that to match entities.)

Entity Superpowers — When the user changes a cell value in the spreadsheet, AI Hero can compute derrived attributes for that row/entity on the fly. If one of the attributes is changed, all attributes that are derrived from it can change using a cascading set of rules. Downstream schema constraints can also be enforced for these flagging data that don’t match expectations in the data contracts.

Attribute Superpowers — Each attribute can be fixed or enriched using (A) formulas or integrations (B) fine-tuned state-of-the-art ML model or (C) Human-in-the-loop.

(A) Formulas and Integrations — The value of a derrived attribute can be computed using a formula (e.g. row['is_adult'] = True if row['age'] >18 else Falseor row['first_name'] = row['name'].split(‘ ‘)[0] ). It can also be updated using an integration with a 3rd-party API (e.g. phone_number for customers in a CRM can be updated with data vendors like Apollo/ZoomInfo/etc.)

(B) Machine Learning — The value of a derrived attribute can also be predicted using ML models. AI Hero can train models on data that’s partially filled and can fill in the missing or incorrect values using classification or regression models trained with AutoML. A more sophisticated way to use this would be to use Large Language Models (like OpenAI’s GPT-3), and AI Hero can help you define a prompt to fill the data.

(C) Human-in-the-loop — Often the best way to fix and enrich data is to rely on the knowledge of experts on your team. With AI Hero, you can easily assign a task for any entity to an expert on your team. Team members can collaborate on updating the data using moder forms with schema constraints enforced. For example, to update outdated information, you might need a human-in-the-loop confirm that the updated information matches expectations. The expert can either review or fix the data for that entity/row.

Technology Deep-Dive

Eventful

The ability to provide a seamless user experience for scheduling and executing non-instantaneous Superpowers is achieved with our in-house event management platform “Eventful”. Though still in its infancy, it is proving to be easy for us to launch new superpowers by building dedicated workers listening to events.

Eventful is a hybrid messaging + HTTP server that drives events and event response state-tracking in AI Hero. HTTP endpoints are used to subscribe to event types for each Superpower which can be triggered through the UI or from updates to the data. Once triggered, Eventful sends the event for the Superpower to the subscribers on the channel they were asked to listen to.

More specifically, Eventful works like this:

a. Subscribe Request — Any microservices interested in listening to an event (e.g. for a Superpower or a UI event) subscribe by providing the event type with an HTTP request. It also mentions how it is interested in listening to the events — using a webhook (if the microservcice is HTTP) or a queue (if the microservice is a message queue listener).

b. Subscription Details in Response — Eventful’s response for this HTTP request is confirmation of the subscription. It creates the topic if it doesn’t exist and returns the details for listening to the topic back to the subscriber.

c. Start listening — For webhooks, the subscriber listens for HTTP calls. For message-queue based listeners, the subscriber then opens a connection to the event queue. Our message queues are currently implemented with Redis for each consumer group for an event type, but can be Kafka in the future.

d. Triggers and Forwards — When an event is triggered, Eventful pulls up its list of listeners for each consumer group, and sends the event payload with headers (e.g. Authorization, event-chain identifier, previous event identifier, etc.) to the webhook or the message queue.

There’s a lot of scope for making Eventful more robust. For example, if there are no listeners, the topic for the event type can be deleted. Eventful could also set pod requirements to 0. We’ve open-sourced Eventful, and are looking for contributors. Reach out to me for more info.

Managing Kubernetes Cluster with Terraform

To become Kubernetes-native, we need an easily replicable and deterministic way of creating each and every environment. We have our staging and production environments managed almost entirely with Terraform. We also use terraform to set up a cluster as needed for sandbox/feature development. Other than these environments, we also use Terraform to spin up clusters for K8s for research and development.

We’re using AWS EKS for hosting our cluster. You can see below what our cluster config in Terraform looks like. (Note: the .tfbelow is partial. There’s a little more going on in there, but I’ve abstracted it for now)

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "19.0.4"

  cluster_name    = local.cluster_name
  cluster_version = "1.24"

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  cluster_endpoint_public_access = true

  eks_managed_node_group_defaults = {
    ami_type = "AL2_x86_64"
  }

  eks_managed_node_groups = {
    one = {
      name = "platform"

      instance_types = ["t3.xlarge"]

      min_size     = 1
      max_size     = 3
      desired_size = 2
    }
  }
}

Terraform runs have a state. To manage the state and enable team access we’ve used HashiCorp’s Terraform Cloud. We’ve found that maintaining a state there helps multiple people deploy and orchestrate the infrastructure as needed.

The only thing that’s not managed by Terraform is the connection to 3rd party applications we are using under the hood — AWS Elasticache and MongoDB Atlas.

What’s next: To enable continuous improvements to the platform, we develop with feature-branches. One thing we’d love to build towards is spinning up clusters for developing each feature branch automatically.

Using Ambassador Edge Stack for Ingress (North-South traffic)

As of now, this is purely a choice of convenience. Our previous architecture used Ambassador as an API gateway. We used Ambassador for defining the public and private routes, with the /extauth pattern for authenticating the private routes. Ambassador creates an ingress as a Classic AWS LoadBalancer for us automatically. In contrast with Emissary Ingress, Ambassador Edge-Stack has support for TLS termination and good integration with Cert-Manager.

Certificate Management with Cert-Manager and Let’s Encrypt

Configuring Cert-Manager for TLS termination as described here, allows us to add a verified https:// for each of our Ambassador endpoints in our clusters. For example, it gets a verified subdomain certificate for [https://app.aihero.studio](https://app.aihero.studio) automatically using LetsEncrypt.

What’s next: We’re actively investigating the pros and cons of different service mesh and will migrate to one in the near future. Istio, Traefik, and Consul are all options under consideration.

Platform Observability

Observability is essential to providing a robust service over Kubernetes. We’ve found NewRelic’s Kuberenetes integration very comprehensive to gain observablity into the cluster, the pod logs, telemetry, and even the browser. We’re still investigating the right tool for us. Another possible tooling could be what’s provided with service meshes. For example, Istio comes with it own set of tools for observability.

Future Work on the Platform

Currently working on:

• CI: We’re currently setting up our development process with automated image builds and tagging with Github Actions. After building the container and running tests, we want to push the image (after auto-tagging it) into AWS ECR.
• CD: While development is still local, our staging and production environments are currently deployed directly with kubectl . We plan to use GitOps practices (e.g. using Flux/ArgoCD) to deploy the containers to these environments.
• Managing Secrets with HashiCorp Vault: We plan to migrate our secrets and configs into HCP Vault, giving us the flexibility to provision, administer, and roll over secrets from a central secure platform.

Near-term:

• Columnar Operations as a DAG: To improve the compute of the attributes, we want to migrate to a fully distributed columnar load and processing of the data. This needs some intricate orchestration with cascading compute of attributes that depend on other derrived attributes with Eventful.
• Workload Parallelization: Compute should also be parallelized, both at the attribute level and micro-batches for large data. This would follow columnar operations, as it is an optimization over it.
• HPA and Cluster Autoscaling: To achieve the parallization scale that AI Hero aims for, we want to fully embrace autoscaling both at the pod as well as the cluster level. For this we are exploring kubernetes-native HPA, as well as tools like Karpenter that aim to provide autoscaling.

I’m super excited to build these features out into the platform. I’d love to hear what you think!

This article is written by Rahul Parundekar for AI Hero.

*AI Hero has Superpowers to keep your data Fresh. Using AI Hero, you can Clean, Enrich, and ensure Quality of your mission-critical operational data. AI Hero’s Hero Tables helps you fix incorrect, missing, and outdated data with the help of AI, Code, Integrations, and Human-in-the-loop.*

*Rahul is an AI expert with 14+ years of experience in architecting and building AI products, engineering, research, and leadership, and is passionate about improving the Human Experience through AI. He loves to learn about how ML is used in practice and helps fellow ML practitioners with his experience. Talk to him here!*