Connecting Apache Kafka to Dapr on Huawei Cloud CCE

Publishing events to Huawei Cloud DMS Kafka using Dapr pub/sub in Kubernetes

This article demonstrates how to integrate Apache Kafka, provided by Huawei Cloud Distributed Message Service (DMS), with Dapr running on Cloud Container Engine (CCE). By using Dapr’s pub/sub building block, applications can publish events through a stable API while keeping Kafka-specific configuration externalized and environment-specific. This approach allows the messaging backend to be changed or reconfigured without modifying application code.

Prerequisites:

• Huawei Cloud CCE Cluster with External Access

CCE Cluster in the Huawei Cloud Console

You need a running Cloud Container Engine (CCE) cluster in Huawei Cloud. If you are unsure how to create a CCE cluster, refer to the official documentation.

In this example, a CCE Turbo cluster running containerd was used, but any modern CCE cluster version is suitable. Ensure the cluster has at least three worker nodes, as Dapr deploys multiple control-plane pods and requires sufficient resources.

To enable external access for the CCE cluster, create an Elastic IP and follow the official Huawei Cloud steps. This is required because Helm will be used to install Dapr into the cluster.

Elastic IP on the Huawei Cloud Console

• Huawei Cloud DMS Kafka Instance

DMS Kafka in the Huawei Cloud Console

Create a Kafka instance using Huawei Cloud Distributed Message Service (DMS). The official DMS Kafka creation documentation can be found here.

For easier access, enable the Public Access in the Access Mode during instance creation and buy an Elastic IP and assign it to your instance.

When assigning a security group to your DMS instance, make sure that the Apache Kafka ports are accessible through inbound traffic.

For testing purposes only, Plaintext Access was enabled. This configuration is not recommended for production environments.

Enabling Internet Access for the CCE Cluster

To allow the CCE cluster to access external resources, outbound internet connectivity must be configured using a NAT Gateway. This enables workloads running in private subnets to initiate connections to the internet.

The exact configuration steps depend on the network topology. Refer to the official Huawei Cloud documentation for instructions on creating and associating a NAT Gateway with the cluster.

Nat Gateway interface in the Huawei Cloud Console

Enabling Automatic Topic Creation (Optional)

If you prefer not to create Kafka topics manually, enable automatic topic creation in the DMS Kafka console. With this option enabled, publishing to a non-existent topic will automatically create it.

Enabling automatic topic creation before creation

Enabling automatic topic creation after creation

Deploying Dapr in CCE

Dapr can be installed on CCE using either the official Dapr CLI or the Helm chart. Both methods perform the same installation. This example uses Helm. The official tooling documentation can be found here.

Here are the full Helm deployment steps if you’re using a Linux VM:

# Debian/Ubuntu-based
sudo apt-get install curl gpg apt-transport-https --yes
curl -fsSL https://packages.buildkite.com/helm-linux/helm-debian/gpgkey | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/helm.gpg] https://packages.buildkite.com/helm-linux/helm-debian/any/ any main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm

# RHEL/CentOS-based
sudo dnf install helm

# Add the official Dapr Helm chart.
helm repo add dapr https://dapr.github.io/helm-charts/

# Or also add a private Dapr Helm chart.
helm repo add dapr http://helm.custom-domain.com/dapr/dapr/ \
   --username=xxx --password=xxx
helm repo update

# See which chart versions are available
helm search repo dapr --devel --versions

# Install the dapr chart
helm upgrade --install dapr dapr/dapr \
--version=1.16 \
--namespace <YOUR-NAMESPACE> \
--create-namespace \
--wait

# Replace <YOUR-NAMESPACE> with your specified namespace

Dapr pods being shown in the K9s Kubernetes terminal interface

After installation, the Dapr control-plane pods should be visible in the cluster.

Verifying Kafka Network Reachability (Optional)

Confirm that the Kafka external listener is reachable from inside the Kubernetes cluster (run this inside any pod that has internet access):

nc -vz <KAFKA-EXTERNAL-IP> 9094

A temporary kcat pod can be used to validate Kafka connectivity and inspect metadata directly from within the cluster.

Create a temporary pod and open a shell:

kubectl run kafka-test \
  --rm -it \
  --restart=Never \
  --image=edenhill/kcat:1.7.1 \
  --command -- sh

Test Kafka connectivity:

kcat -b 119.8.84.83:9094 -L

Creating the Dapr Kafka Pub/Sub Component

Create a Dapr pub/sub component that defines how Dapr connects to Kafka. This component is Kafka-specific, while applications interact only with Dapr’s pub/sub API.

Example kafka-pubsub.yaml,replace<EXTERNAL-IP>with your DMS Kafka’s IP:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: kafka-pubsub
  namespace: <YOUR-NAMESPACE>
spec:
  type: pubsub.kafka
  version: v1
  metadata:
  - name: brokers
    value: "<EXTERNAL-IP>:9094"

  - name: authType
    value: none

  - name: disableTls
    value: "true"

  - name: consumerGroup
    value: dapr-test-group

Apply the component:

kubectl apply -f kafka-pubsub.yaml

Confirm it exists:

kubectl get components -n <YOUR-NAMESPACE> 
kubectl describe component kafka-pubsub -n <YOUR-NAMESPACE>

Deploying a Dapr-Enabled Test Workload

The following deployment runs a minimal example container and injects the Dapr sidecar for testing purposes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dapr-test
  namespace: <YOUR-NAMESPACE>
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dapr-test
  template:
    metadata:
      labels:
        app: dapr-test
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "dapr-test"
    spec:
      containers:
      - name: app
        image: busybox
        command: ["sh", "-c", "sleep 3600"]

Apply the deployment and wait until the pod is running:

kubectl apply -f dapr-test.yaml
kubectl get pods -n default

Confirm the Dapr sidecar is running:

kubectl logs deploy/dapr-test -c daprd -n default

Publishing a Test Message to Kafka Using Dapr

BusyBox does not include curl, so wget is used to send the HTTP request, enter the application container:

kubectl exec -it deploy/dapr-test -c app -n default -- sh

Publish a message via Dapr:

wget -qO- \
  --header='Content-Type: application/json' \
  --post-data='{"msg":"hello from dapr"}' \
  http://localhost:3500/v1.0/publish/kafka-pubsub/<YOUR-TOPIC>

The published message can now be observed in the Huawei Cloud DMS Kafka console, where it appears as a CloudEvent produced by Dapr.

DMS Kafka’s Message query interface showing the published JSON message

Alternatively, message delivery can be confirmed using kcat:

kcat -b 119.8.84.83:9094 -C -t <YOUR-TOPIC> -o end

Conclusion

We successfully configured Dapr in Kubernetes to publish messages to Huawei Cloud DMS Kafka and verified end-to-end delivery by observing the events in the Kafka console.