Kafka Meets SQL : create your Kafka topic with CREATE STREAM in ksqlDB

Kafka + SQL = ksqlDB

ksqlDB is a real-time streaming database built on Apache Kafka. It enables users to process, transform, and analyze streaming data using SQL-like queries.

Install ksqlDB

Download and extract ksqlDB:

cd /opt
sudo wget https://packages.confluent.io/archive/7.5/confluent-community-7.5.0.tar.gz
sudo tar -xvf confluent-community-7.5.0.tar.gz
sudo mv confluent-7.5.0 ksqldb

Start ksqlDB Server:

nohup /opt/ksqldb/bin/ksql-server-start /opt/ksqldb/etc/ksqldb/ksql-server.properties &

Start ksqlDB CLI:

/opt/ksqldb/bin/ksql http://localhost:8088

You should now see a ksqlDB CLI prompt like

Create a Kafka Topic for ksqlDB

Inside the ksqlDB CLI, create a Kafka topic:

CREATE STREAM pageviews (
    user_id STRING,
    page STRING
) WITH (KAFKA_TOPIC='pageviews-topic', VALUE_FORMAT='JSON', PARTITIONS=2, REPLICAS=1);

This will create the topic automatically with 2 partitions and 1 replica.

Verify the topic with topic list command

bin/kafka-topics.sh --list --bootstrap-server MR.ANO.WER.143:9092

Or you can check the topic exists inside ksqlDB

SHOW topics;

 Kafka Topic                 | Partitions | Partition Replicas 
---------------------------------------------------------------
 default_ksql_processing_log | 1          | 1                  
 pageviews-topic             | 2          | 1                  
---------------------------------------------------------------

Congratulations we are done with creating kafka topic with SQL like query in ksqlDB.