Kafka ACLs Authorization: Usage & Best Practices

Overview

Kafka Access Control Lists (ACLs) provide a robust authorization framework that determines which authenticated users can perform specific operations on Kafka resources. This comprehensive guide explores Kafka ACL concepts, configuration approaches, management tools, and best practices for implementing effective authorization in your Kafka deployments.

Understanding Kafka Authorization and ACLs

Kafka authorization determines what an authenticated entity can do once its identity has been verified. Similar to how an ATM allows you access only to your accounts after PIN verification, Kafka enables specific actions for authenticated clients based on their permissions[17]. Authorization in Kafka is implemented through Access Control Lists (ACLs), which specify which users can perform which operations on specific resources.

Kafka ACLs [17]

Core Authorization Concepts

Kafka Core ACLs Concept [17]

The authorization framework in Kafka is pluggable and configured using the authorizer.class.name property. Two primary authorizer implementations are available:

• AclAuthorizer: For ZooKeeper-based clusters, storing ACLs in ZooKeeper
• StandardAuthorizer: For KRaft-based clusters, storing ACLs in the cluster metadata[18]

Each ACL consists of five core components that together define a permission:

Additionally, ACLs can specify a host parameter (IP address) to limit connections from specific locations, and pattern types (LITERAL, PREFIX, or WILDCARD) to match resources[17].

Kafka ACLs Workflow [109]

The Default Behavior

By default, if a resource has no associated ACLs, access is determined by the allow.everyone.if.no.acl.found property. Amazon MSK sets this to true by default, meaning resources without explicit ACLs are accessible to all principals[9]. However, once you add ACLs to a resource, only authorized principals can access it.

SASL Authentication with Kafka ACLs

Before authorization can occur, clients must be authenticated. Kafka commonly uses Simple Authentication and Security Layer (SASL) mechanisms, which provide the authenticated identities that ACLs reference.

SASL Mechanisms for Authentication

Kafka supports several SASL mechanisms, each with different security characteristics:

It’s important to distinguish between SASL/PLAIN (the authentication mechanism) and SASL_PLAINTEXT/SASL_SSL (the security protocol). The former refers to username/password credentials, while the latter indicates whether the connection is encrypted with TLS.

Configuring Kafka ACLs

Setting up ACLs involves both broker and client configuration steps.

Broker Configuration

To enable ACL authorization on Kafka brokers, add the following to server.properties:

# Enable ACL authorization
authorizer.class.name=kafka.security.authorizer.AclAuthorizer

# Default permission when no ACLs exist for a resource
allow.everyone.if.no.acl.found=true

# Enable SASL mechanisms
sasl.enabled.mechanisms=PLAIN,SCRAM-SHA-512

# Configure security protocol
listeners=SASL_SSL://hostname:9093
security.inter.broker.protocol=SASL_SSL

JAAS Configuration

Java Authentication and Authorization Service (JAAS) configuration is essential for SASL authentication. For brokers, JAAS configuration should be prefixed with the listener name and SASL mechanism[13]:

listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="admin" \
    password="admin-secret" \
    user_admin="admin-secret" \
    user_alice="alice-secret";

Client Configuration

Clients need corresponding configuration to authenticate to the broker:

bootstrap.servers=hostname:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="alice" \
    password="alice-secret";

Managing Kafka ACLs

Various tools are available for creating, listing, and deleting ACLs in Kafka.

Command-Line Tools

The primary tool for managing ACLs is the kafka-acls command-line interface:

kafka-acls --bootstrap-server localhost:9092 \
  --command-config admin-client.properties \
  --add \
  --allow-principal User:alice \
  --operation Read \
  --operation Write \
  --topic orders

Redpanda provides rpk acl commands for similar functionality:

rpk acl create \
  --allow-principal 'User:Charlie' \
  --operation all \
  --topic pings

Confluent Platform offers the confluent kafka acl command suite with additional capabilities[14].

GUI Tools

Conduktor provides a graphical interface for ACL management with views for visualizing relationships between principals and resources. It offers a wizard to simplify ACL creation based on common use cases and supports importing/exporting ACLs in CSV format[7].

Importing and Exporting ACLs

For managing ACLs at scale, tools like Kafka Security Manager (KSM) allow using external sources (like GitHub repositories) as the source of truth for ACLs. This provides auditability, rollback capabilities, and prevents unauthorized ACL modifications directly in Kafka[15].

Common Use Cases and ACL Patterns

Different scenarios require different ACL configurations:

Producer Access

To grant a user write access to a topic:

kafka-acls --bootstrap-server localhost:9092 \
  --add \
  --allow-principal User:producer \
  --operation Write \
  --operation Create \
  --operation Describe \
  --topic orders

Consumer Access

To grant a user read access to a topic and consumer group:

kafka-acls --bootstrap-server localhost:9092 \
  --add \
  --allow-principal User:consumer \
  --operation Read \
  --operation Describe \
  --topic orders \
  --group order-processors

Admin Access

For administrative users who need cluster-wide permissions:

kafka-acls --bootstrap-server localhost:9092 \
  --add \
  --allow-principal User:admin \
  --operation All \
  --cluster

KRaft Mode Considerations

When using KRaft mode (ZooKeeper-less Kafka), some special considerations apply:

• KRaft-backed clusters cannot use SCRAM for controller-to-controller authentication
• SASL credentials should be created before brokers start running
• For KRaft with SASL/PLAIN, you need the configuration property sasl.mechanism.controller.protocol=PLAIN[3]

Best Practices for Kafka ACLs

Implementing the following practices can enhance security and manageability:

Security Recommendations

• Always use TLS with SASL to encrypt credentials in transit
• Prefer SASL/SCRAM or SASL/GSSAPI over SASL/PLAIN in production environments
• Implement proper credential management and rotation procedures
• Configure ACLs with the principle of least privilege
• Regularly audit and review ACL assignments

Mechanism Selection

Choose your SASL mechanism based on your existing infrastructure:

Avoiding Common Issues

• Always use TLS with SASL/PLAIN to prevent credential exposure
• Ensure correct JAAS configuration for each listener and mechanism
• When using KRaft mode, set super.users correctly to allow broker-to-controller communication
• Verify that client configurations match broker configurations for the selected mechanism[13]

Troubleshooting Common ACL Issues

Common authorization and authentication issues include:

• SaslAuthenticationException: Verify correct credentials and SASL mechanism configuration
• SSL handshake failed: Check TLS certificates and truststore/keystore configuration
• Could not find KafkaServer entry in JAAS configuration: Ensure proper JAAS configuration for controllers in KRaft mode
• Unexpected Kafka request during SASL handshake: Verify client is properly configured for SASL authentication

Conclusion

Kafka ACLs provide a flexible and powerful mechanism for controlling access to your Kafka resources. By understanding the core concepts, implementing appropriate authentication mechanisms, and following best practices, you can create a secure and well-managed Kafka deployment that balances security needs with operational requirements.

Remember that ACLs are just one component of a comprehensive security strategy for Kafka. Combining ACLs with proper network security, TLS encryption, and secure credential management creates a defense-in-depth approach that effectively protects your Kafka infrastructure and data.

If you find this content helpful, you might also be interested in our product AutoMQ. AutoMQ is a cloud-native alternative to Kafka by decoupling durability to S3 and EBS. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. AutoMQ now is source code available on github. Big Companies Worldwide are Using AutoMQ. Check the following case studies to learn more:

• Grab: Driving Efficiency with AutoMQ in DataStreaming Platform
• Palmpay Uses AutoMQ to Replace Kafka, Optimizing Costs by 50%+
• AutoMQ help Geely Auto(Fortune Global 500) solve the pain points of Kafka elasticity in the V2X scenario
• How Asia’s Quora Zhihu uses AutoMQ to reduce Kafka cost and maintenance complexity
• XPENG Motors Reduces Costs by 50%+ by Replacing Kafka with AutoMQ
• Asia’s GOAT, Poizon uses AutoMQ Kafka to build observability platform for massive data(30 GB/s)
• AutoMQ Helps CaoCao Mobility Address Kafka Scalability During Holidays
• JD.com x AutoMQ x CubeFS: A Cost-Effective Journey at Trillion-Scale Kafka Messaging

AutoMQ Architecture

References:

1. Kafka-ACLs CLI Error with Confluent Cloud Instance
2. Kafka Security: Managing Consumer Groups Across
3. Migrating Security ACLs to New Cluster (KRaft)
4. Java KafkaAdminClient: Querying ACLs by Principal
5. KafkaTopical: The Kafka UI for Engineers and Admins
6. MirrorMaker Seems Too Complicated for What It Is
7. Kafka Access Control List (ACL)
8. Built-in Security with ACLs
9. Amazon MSK ACLs
10. Kafka Demo: ACL Authorization
11. Access Management Authorization ACL
12. Simple ACL Authorization
13. How to Manage Kafka ACLs for Enhanced Security
14. Confluent Kafka ACL List
15. Kafka Security Manager
16. Creating an ACL
17. Security Authorization
18. ACLs Overview
19. Kafka ACL Management
20. How Can I Start Kafka with User Anonymous
21. Kafka KRaft Authentication
22. AWS MSK ACL
23. Gathering Opinions on Kafka Management Tools
24. Company Decide to Use Kafka MSK
25. What’s the Most Complex Piece of Technology In
26. MSK Topic Level Security
27. How Do You Identify Producers Writing to Kafka
28. AWS MSK Kafka ACL Infrastructure as Code
29. I Made a New GUI for Apache Kafka
30. A List of GUI Tools for Working with Apache Kafka
31. Kafka Alternatives Most DE Job Descriptions Want
32. Manage Network Policies and Kafka ACLs in a
33. Looking for Resources on Kafka/Confluent CI/CD Best
34. ZooKeeper ACLs Kafka
35. Managing ACLs
36. Access Control ACL
37. ACLs Advanced Insights
38. Secure Kafka Deployment Best Practices
39. Configuring Kafka
40. Confluent CLI Kafka ACL Index
41. What are the Necessary Kafka Permissions Required by Conduktor Console
42. Authorization ACL
43. Essential Kafka Security Best Practices for 2024
44. For Some Reason My Mirror Maker 2 Runs But Does
45. Are Micro Services That Both Produce and Consume
46. Deep Dive into Apache Kafka Storage Internals
47. Why Kafka and Not FIFO Queues for Cloud Providers
48. Keeping Track of Users
49. MirrorMaker2 Not Replicating Any Consumer Groups
50. What Do You Think is the Biggest Differences
51. What Do You Do for Syslog
52. Configuring Security for Kafka is More
53. What Can Linux Do That Windows Server Can’t
54. Logging to Syslog Server
55. Kafka vs RabbitMQ
56. A Deep Dive into Apache Kafka Challenges and Solutions
57. Apache Kafka Concepts: Authorization and ACLs Explained
58. No Access Denied: Our Transition to Kafka ACLs
59. System Design Deep Dives: Kafka
60. Improve Kafka Security with ACLs
61. Get Started with Kafka Architecture
62. Confluent Platform Discussions
63. Kafka ACL Issues Using Java Code
64. Kafka Architecture Deep Dive
65. Kafka Tools: Kafka ACLs
66. Deep Dive into Apache Kafka’s Advanced Capabilities
67. Getting Started: Apache Kafka .NET
68. Kafka in K8s
69. Blog on Multinode KRaft-based Kafka Cluster
70. Jikkou: Declarative ACLs Configuration for Apache Kafka
71. Suggestions for UI for AWS Managed Kafka
72. How We Reset Kafka Offsets on Runtime
73. Most Frustrating Parts of Kafka
74. Databricks and Snowflake: Social Media Discussion
75. Integration Digest for February 2025
76. Actively Maintained Official Golang Resources
77. Kafka Authentication Issue
78. Building Data Warehouse Using Kafka
79. No Access to Topic in Heroku Kafka
80. Kafka ACL Tutorial
81. Kafka Encryption and Security Best Practices
82. Security for Humans and Applications in Apache Kafka
83. How to Set Up Redpanda Kafka
84. Kafka Security Best Practices Guide
85. Kafka Architecture and Tools Guide
86. Kafka Security Learning Guide
87. Redpanda Cloud Documentation
88. Comprehensive Guide on Kafka Authentication and Authorization
89. Apache Kafka SASL/SSL Configuration Issues
90. Software Architecture Hot Posts
91. GraphQL New Posts
92. EDI Rising Posts
93. How to Get Current ACLs Details from Kafka Cluster
94. Red Hat AMQ Streams: Configuring Kafka
95. Configuring ACL for Kafka Topic
96. Kafka UI Access Control Lists
97. Lost Trying to Understand the Current State of .NET
98. EC2 Most Basic Ubuntu Server Becomes Unresponsive
99. Web Development Has to be One of the Most
100. Issues Adding ACLs in KRaft Mode Kafka Cluster
101. Kafka ACLs Preventing Client Connection
102. Kafka Service Broken After Applying ACL
103. Kafka ACLs Unable to Describe Group or View Offsets
104. Is Anyone Exposing Kafka Publicly?
105. Looking for Suggestions on the Definitive Guide v2
106. Protect Sensitive Data and Prevent Bad Practices
107. I Built a Kafka GUI Client for Operating Kafka
108. I am Newly Joined in a Company and I was Given a
109. IBM Developer