Lenses SQL Kafka stream processors scale at Kubernetes

[embed]

What’s great about Lenses SQL engine for Apache Kafka is that not only is supported by an intuitive user interface, but also how it scales kafka stream processors.

Lenses is the first Kafka platform that supports out of the box deployment and scalability on Kubernetes. In the above video we showcase how to create and scale Lenses SQL processors via Lenses but also via Kubernetes. Processor’s scale can be managed via Lenses by simply updating the runners.

Lenses supports multiple Kubernetes clusters and namespaces to better organise your pods, but also pipelines which acts as an additional label to be attached to the metrics of the pods.

Create Lenses SQL Stream Processors via Lenses UI

Processors runners state. Scale up or down via the Lenses UI.

Creating a Lenses SQL kafka stream processor will also create the visual topology of each step described by the SQL query. This gives us a better understanding of the data flow, by getting the details of each interactive node. In a single view we can see the input topic data as well as the output topic data, so that we can verify how the query processes the data:

Creating a Kafka stream processing application becomes as easy as creating a SQL query and post it via the Lenses UI or API endpoints or even your Kubernetes charts.

But is Kubernetes the only way to scale processors? No. We understand that not all the organisations have adopted a containerised orchestration. So Lenses also supports scale via Kafka Connect, which is one of the native Kafka APIs. Lenses SQL Engine utilises the connect workers in order to execute the processors at scale.

Visit Lenses docs to find out more on Lenses Scaling processors modes.

Relevant Blogs

Kafka stream processing via Lenses SQL Kafka stream processing via Lenses SQL, scale with Kafka Connect Using Lenses to easily scale SQL stream processors in Kubernetes

Download Lenses at http://www.landoop.com/downloads/