Setup Kafka and Schema Registry with Confluent

Schema Registry can be used for schema management as a centralized schema repository. Further schema registry and schemas can be utilized for schema validation, serialization and deserialization of streaming messages of a topic.

For the local environment setup, it can use different approaches as described in the Confluent documentation.

Below use the docker compose to start Kafka broker and Schema Registry.

Confluent Platform All In One Project

• Clone the repository

*Confluent cp-all-in-one* provides Docker Compose to start the components.

git clone https://github.com/confluentinc/cp-all-in-one.git

• Navigate to the repository

cd cp-all-in-one

• Navigate to the required solution

cd cp-all-in-one-community

Start Schema Registry and Kafka Broker

• Docker Compose file

Simplified sample Docker Compose YAML file with confluentinc/cp-kafka:8.0.0 image and confluentinc/cp-schema-registry:8.0.0 image.

services:
  broker:
    image: confluentinc/cp-kafka:8.0.0
    hostname: broker
    container_name: broker
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
  schema-registry:
    image: confluentinc/cp-schema-registry:8.0.0
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - broker
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:29092'
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

Above *cp-all-in-one provided Docker Compose files *can be used if required other Confluent components.

• Start using Docker Compose

docker-compose up

Schema Registry API

• Validate using the Schema Registry API

Get Cluster Level Configuration

curl --location 'http://localhost:8081/config'

Get Global Mode

curl --location 'http://localhost:8081/mode'

List Subjects

curl --location 'http://localhost:8081/subjects'