Delivering Data Platform Artifacts using Custom K8S Operators

Generated using https://aistudio.google.com/prompts/new_image

Large SaaS deployments often span multiple K8S clusters across different regions. Developing and delivering applications for such a large-scale deployment requires automation to ensure that applications are deployed efficiently, consistently and reliably to all the clusters with traceability. This blog discusses how the IFS engineering team automated artifact deployment using custom K8S operators in its brand-new data platform.

K8S Operators for Handling Deployments

Today, cloud environments heavily use K8S operators to automate application delivery. For example, the Flux operator works on FluxInstance CR (custom resource). Together they automatically deploy and upgrade Helm charts from Git repository to the K8S cluster. Likewise, ArgoCD also uses a Controller (similar to operator) that works on its ‘Application’ CRs.

Similar to how ArgoCD and other Kubernetes operators work; it is possible to write a custom K8S operator that manages custom deployments on any K8S based platform. In fact, as you may already know, operators were introduced in K8S to manage complex application deployments. A custom K8S operator works on a custom Kubernetes CRD (Custom Resource Definition) to handle complex deployments.

What are artifacts?

In the current context, an artifact is a deployable component from the new IFS data platform — such as a Spark script, a Trino View definition or a Superset dashboard. These artifacts need to be deployed dynamically to clusters because the platform enables developers to build and release dashboards independently of the platform’s release cycle. This offers a flexible approach to releasing features (such as new Superset dashboards) without a platform release.

• Superset Dashboards — New dashboards can be developed dynamically and shipped without depending on the data platform release cycles
• Trino Views — New Trino views may be required to support the new dashboards.
• Spark Scripts — Spark scripts create curated data, in the Medallion data lake architecture. This allows us to deliver value added data to customers dynamically without a formal data platform release cycles.

Architecture for the Automation Process

When designing the artifact deployment approach, the existing GitOps operational model was considered. The resulting architecture can be illustrated as below.

High-Level Architecture Diagram

A CRD should be defined for each artifact that needs to be deployed automatically. Along with the CRD a custom K8S operator is also written that can read the CR and deploy it to the cluster. For example, a CRD for the Trino view can be as follows.

spec:
 viewName: "{spec.viewName}"
 viewSQL: "SELECT …" 
 schemaName: "assets"

The operator detects the CR and calls the TrinoManager API which will create the Trino View definition in the metadata objects of the catalog.

These CRDs and operators are deployed along with the data platform releases. Now our data platform is ready to automatically deploy artifacts as soon as their CRs are pushed to the K8S cluster.

In this Architecture, when CRs are created or updated, ArgoCD, ensures the changes are automatically rolled out to all relevant clusters. After that, custom operators perform the deployments based on the CRs.

Artifact deployment Workflow

Now that the custom operators are in place, developers can create and release dashboards and curated data to customers on the fly.

1. IFS Developers create Dashboards, Trino Views and Spark Scripts using developer tools (we are developing a browser-based development tool for this purpose) and test it.
2. After the development is completed, they export the artifacts. The CRs are created during the export process. Developers commit the exported artifacts to the Git Repo.
3. ArgoCD detects the committed CRs and deploy them to the K8S cluster.
4. The relevant operators in the cluster detect the CRs and process them. The operator can call another API to deploy the artifact to the relevant products. For example, as a result of deploying the TrinoView CR, the relevant operator will call an API that creates the View in the data lake.
5. If the artifact carries entitlements, it can be more complicated logic handling. But the basic flow is the same.

Advantages of this Architecture

In addition to allowing dynamic deployments, this architecture has several other advantages.

• Scalability: Manually deploying artifacts to SaaS deployments spanning across multiple regions becomes error-prone. Operators automate this reliably.
• Declarative deployment: Define the desired state in CR, and the operator ensures the cluster matches it.
• Consistency: The same operator can apply artifacts to dev, staging, and prod clusters without changing the artifact definition.
• Integration with GitOps: This architecture seamlessly integrates with GitOps, leveraging its key benefits such as version control and automated deployments.

Conclusion

Using custom K8S operators for application/artifact delivery is becoming an increasingly common design in modern Kubernetes environments and GitOps workflows. In this blog we discussed how custom operators are designed to deploy artifacts in the new data platform.

Acknowledgements

Special thanks to Arnoud Roo, Principal Architect at IFS, and the other members of the Data Service Foundation team for their invaluable contributions to the design of this architecture.