Rabbit MQ

RabbitMQ is an open-source message broker that enables applications to communicate asynchronously by sending and receiving messages through queues. A queue is a data structure that stores items in a specific order until they are processed. RabbitMQ supports the Advanced Message Queuing Protocol(AMQP) protocol but it also supports other protocols like: STOMP, MQTT.

AMQP standardizes the behavior between producers and consumers. It’s platform independent and technology independent. Each message has headers(key-value pairs), properties(key-value pairs) and body/payload(byte array). It supports a maximum of 2G per message and the messages are sent in frames of 131KB, by default.

A queue in RabbitMQ is implemented as an Erlang process, and therefore it requires more memory on the server. Each Erland process has its own isolated heap(which ensures security and reliability) stored in a separate memory block. The headers and properties of a message are stored in the process heap, while the body of the message is stored in a separate memory area called binaries. This design avoids copying large message payloads between different processes, since binaries can be shared by reference.

Exchanges

In RabbitMQ, a producer never sends a message directly to a queue. It uses an exchange which is bound with a queue through binding keys. An exchange determines in which bounded queue a message should be placed.

Exchange Types: Nameless — this is the default type. It compares the routing key with queue names and allows the producer to send messages “directly” to the queues. Fanout — routes the received messages to all queues that are bound to the defined exchanger. Direct — determines the destination queue by comparing the binding key to the routing key. The routing key is sent together with a message to the exchanger. Topic — determines the queue by matching the binding key defined as regex pattern, with the received routing key. The routing key is defined as a list of words which are delimited by dots. (star) can substitute exactly one word and #(hash) can substitute zero or more words. The maximum allowed length of a topic rounting key is 255 characters. Header — determines the queue by comparing the binding keys with any or all headers. When defining a binding between an exchange and a queue, the headers and their values must be defined for each binding together with the x-match header which indicates if all or any* of headers should be taken into account. Headers that start with x- are not taken into account to evaluate matches.

Message Order From producer’s perspective, messages are always held in the queue in the order in which they were published.

From consumer’s perspective, the messages are delivered in the same order only if:

• messages are published in a single channel
• they pass through a single exchange
• the are stored in a single queue
• they are consumed by exactly one consumer(one outgoing channel) The order is not guaranteed for prioritized queues and when messages are rejected by the consumer and requeued (requeue=true).

Queue types

Work/Task queues (Producer -> Queue -> Many Consumers) By default, RabbitMQ distributes messages to consumers in a round-robin behavior. If a consumer asks for a batch of messages(prefetch/basic.qos > 1), each message must be acknowledge individually. If a consumer dies or its channel is closed before acknowledging a message, RabbitMQ requeues and delivers the unacknowledged messages to another consumer.

Dead Letter Exchange Queues (DLX) They are used to route the messages that are rejected by consumers, expire due to TTL or are discarded because a queue exceeds its length limit.

Example flow: Producer → Exchanger → Queue → Consumer Exchange.DLX → Queue.DLX To route messages from Queue to Queue.DLX, we must configure the following property on the original queue: x-dead-letter-exchange = Queue.DLX

Quorum queues They are a new standard of queues in RabbitMQ and they are highly available by default while classic queues need a mirroring policy to achieve high availability. When classic mirrored queues, the queue is replicated across multiple nodes. One node acts as the master, handling all reads and writes, while the others nodes host mirrors that replicate the master’s state. In contrast, quorum queues use a leader–follower model based on the Raft consensus algorithm, providing stronger consistency guarantees and safer failover behavior.

*Raft-based leader–follower model* - One node is elected as the leader.

• Other nodes act as followers
• All writes (publish, ack, delete) go through the leader
• The leader replicates operations to followers using a Raft log
• If the leader fails, a follower is automatically elected
• An operation is committed only after a majority (quorum) of nodes confirm it. If a majority is lost, the queue becomes unavailable (but data is safe).
• Operations are allowed only when a quorum (more than half of the nodes) is available.
• Provides Split-brain protection:only a majority of nodes can continue processing requests. Split brain is a failure scenario in a distributed system where the cluster becomes partitioned, and multiple parts of the system believe they are the leader and continue operating independently.

Priority queues Priority queues allow messages to be published with an associated priority, so that messages with higher priority are delivered before lower-priority messages.

A priority queue is created by setting the x-max-priority argument on the queue. This value defines the maximum supported priority level. Message priorities range from 0 to 255, but only values up to x-max-priority are respected.

Messages without an explicit priority default to priority 0.

Lazy queues Help optimize memory usage by storing most message data on disk instead of RAM, which can increase the latency when messages are consumed. They are useful when memory is more important than throughput, such as when consumers are slow and queues grow large, consuming more memory than necessary. Lazy queues are especially recommended for dead letter queues where messages are often accumulated and consumed infrequently. Lazy mode can be enabled at queue creation time or applied later using policies.

Data safety in RabbitMQ is achieved through a combination of durability, acknowledgments, replication, and failure handling.

Durable messages and queues Durable queues survive broker restarts. Durability is a property of queues and exchanges. They are automatically recreated when server starts up. This is possible only if the messages are sent as persistent. Persistent messages are written to disk. This way they are stored in special persistence log files, allowing them to restore once server gets up. Persistence has no effect on non-durable queues. Persistent messages are removed from a durable queue once they are consumed and acknowledged. Both are required for end-to-end durability.

Acknowledgments (ACKs) RabbitMQ sends a message to only a consumer using round-robin method. After delivery, the message becomes unacknowledged and it’s not available to other consumers. The consumer must acknowledge the message after processing. Once a message is acknowledged, the message is permanently removed from the queue. If the consumer crashes, the message is re-queued and only then it may be consumed by another consumer. ACKs prevent message loss during consumer failures.

High Availability Using quorum queues and publisher confirms, messages are acknowledged only after the messages are replicated to the majority of nodes.

Failure handling

• Split-Brain Protection - If a consumer crashes, messages are re-queued

Transactions in RabbitMQ RabbitMQ supports AMQP transactions on a channel using:tx.select, tx.commit and tx.rollback. They allow you to group publishes of messages and acknowledgments(after consuming messages) so that all operations succeed together or are rolled back. Transactions are not recommended because of their very poor performance. RabbitMQ was designed for asynchronous, high-throughput messages.

Publisher Confirms By default, when producer publishes messages to a queue, RabbitMQ doesn’t send any ACK to the producer. A producer publishes messages as a stream, sending them one by one as they are produced, providing low-latency. Publisher confirms is a RabbitMQ extension to implement reliable publishing to make sure that the published messages have safely reached the broker. The producer sends a message, RabbitMQ processes it and sends an ACK back to the producer. If anything fails, the producer receives a NACK. An ACK is sent only after the message is routed to at least one queue, is persisted to disk and replicated to a majority. The ACKs can be processed asynchronously by the publisher.

A RabbitMQ cluster is a group of RabbitMQ nodes that work together to provide scalability, high availability, and fault tolerance. All nodes in a cluster share: Exchanges, Bindings, Virtual hosts, Users and permissions. This information is called cluster metadata. Queues are node-local by default: A queue lives on the node where it is declared and it is referenced by its unique name. A queue name can be provided by the client or auto-generated by RabbitMQ. Both Producer and Consumer can declare a queue. When a node starts, up to 16384 messages per queue are loaded into RAM (this limit is configurable).

Using a cluster with quorum queues improves availability, but doesn’t increase throughput. Throughput scales by adding more queues and sharding workloads, not by simply adding nodes. Partitioning means splitting messages across multiple queues, each handled individually — producers route messages based on a key and consumers scale per partition.

Both Federation and Shovel are used to move messages between RabbitMQ brokers, but they serve different purposes. Federation: an upstream broker exposes exchanges or queues and a downstream broker pulls messages only when needed. Messages stay in the upstream until consumed downstream. Shovel: A shovel consumes messages from a source and immediately republishes them to a destination. Messages are removed from the source once acknowledged. Use Federation for selective, on-demand, WAN-friendly message sharing Use Shovel for bulk transfer, migration, or replication.