Kafka Multinode Installation & KRaft Setup on Java 11

🧩 Kafka Multinode Installation & KRaft Setup on Java 11 (Step-by-Step Guide)

Apache Kafka has revolutionized real-time data streaming — but with the introduction of KRaft mode (Kafka Raft Metadata mode), Kafka has become simpler and more reliable by removing the dependency on ZooKeeper.

In this article, we’ll walk through how to install and configure a multi-node Kafka cluster using KRaft mode on Java 11 — from scratch.

🚀 Step 1: Install Java 11

Kafka requires Java to run. We’ll use OpenJDK 11, which is fully compatible with Kafka 3.x.

sudo apt update
sudo apt install openjdk-11-jdk -y
java -version

If you have multiple Java versions installed, make sure Java 11 is the active one:

sudo update-alternatives --config java

✅ You should see openjdk version "11.x.x" as output.

📦 Step 2: Download & Extract Kafka

Let’s download and set up Kafka 3.1.0 (you can use any compatible version):

wget https://archive.apache.org/dist/kafka/3.7.1/kafka_2.13-3.7.1.tgz
tar -xzf kafka_2.13-3.1.0.tgz
mv kafka_2.13-3.1.0 ~/kafka
cd ~/kafka

🌐 Step 3: Configure Hostnames on Both Machines

For a multi-node cluster, each machine must recognize the other’s hostname.

Open the /etc/hosts file:

sudo nano /etc/hosts

You’ll see:

127.0.0.1 localhost
127.0.1.1 hostname

Add your machine IPs like this:

172.20.xx.xx   iiita
172.29.xx.xx   hostname

Find your system’s IP address using:

ip addr show

Sample output:

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP>
    inet 192.168.1.10/24 brd 192.168.1.255 scope global dynamic enp0s3

👉 Here, 192.168.1.10 is your machine’s IP address.

⚙️ Step 4: Configure Kafka for KRaft Mode

Open Kafka’s KRaft configuration file:

nano config/kraft/server.properties

Now ensure the following essential properties are set (adjust according to your setup):

############################# Server Basics #############################
node.id=1                # Change to 2 on the second machine

########################## KRaft Controller######################
controller.quorum.voters=1@hostname1:9093,2@hostname2:9093

######################### Listeners #############################
listeners=PLAINTEXT://hostname:9092,CONTROLLER://hostname:9093
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
inter.broker.listener.name=PLAINTEXT
advertised.listeners=PLAINTEXT://hostname:9092

######################### Log Directories ########################
log.dirs=/home/<USER>/kafka/kraft-combined-logs
metadata.log.dir=/home/<USER>/kafka/kraft-metadata-logs

########################## KRaft Mode ############################
process.roles=broker,controller
controller.listener.names=CONTROLLER
num.partitions=6

######################## Internal Topics #########################
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=1

💡 Replace <USER> with your actual username. Each node should have a unique node.id, and all nodes share the same controller.quorum.voters setup.

Save and exit (Ctrl + X, then Y).

💾 Step 5: Format Log Directories & Initialize the Cluster

Before starting Kafka, clear and format the log directories:

rm -rf /home/user/kafka/kraft-combined-logs/*
rm -rf /home/user/kafka/kraft-metadata-logs/*

Generate a unique Cluster ID:

bin/kafka-storage.sh random-uuid

Example output:

c0a8e8e4-0f2e-4d33-9b2a-7c2e3cf2c1f5

Format the storage:

bin/kafka-storage.sh format -t <CLUSTER_ID> -c config/kraft/server.properties

⚠️ Use the same Cluster ID on all machines in your Kafka cluster.

🧩 Step 6: Start Kafka in KRaft Mode

Now it’s time to start Kafka!

bin/kafka-server-start.sh config/kraft/server.properties

If everything is configured properly, Kafka will start successfully without ZooKeeper, fully powered by KRaft.

⚡ Step 7: Verify Your Kafka Cluster Setup

Let’s test the setup with a simple producer-consumer example.

✅ Create a Topic

bin/kafka-topics.sh --create --topic test-topic --bootstrap-server hostname:9092 --partitions 1 --replication-factor 1

✅ List Topics

bin/kafka-topics.sh --list --bootstrap-server hostname:9092

✅ Start a Producer

bin/kafka-console-producer.sh --topic test-topic --bootstrap-server hostname:9092

✅ Start a Consumer (on another node)

bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server hostname:9092

If you send messages from the producer, they’ll instantly appear in the consumer terminal 🎉

To verify running processes:

jps

Example output:

8804 Kafka
10076 ConsoleProducer
10413 ConsoleConsumer
10745 Jps

Author: Neeraj Kumar Kannoujiya