RabbitMQ Integration in Adobe Commerce (Magento 2): A Simpler Approach to Asynchronous Messaging

Figure 1

RabbitMQ is a powerful message broker that streamlines communication between different parts of a system, making it an excellent choice for Magento. In this blog, we’ll dive into how RabbitMQ can enhance your Magento installation, along with some configuration tips and best practices.

RabbitMQ Architecture

RabbitMQ is an open-source message broker that facilitates communication between different services and applications by sending messages between them. This decouples the components, allowing for enhanced scalability, reliability, and performance.

Figure 2

Why Use RabbitMQ in Adobe Commerce?

Asynchronous Processing: RabbitMQ allows Magento to handle tasks asynchronously, which can speed up processes like order fulfillment, email notifications, and other background tasks.

Load Distribution: When you have high traffic, RabbitMQ can help distribute the load across multiple workers, ensuring no single component becomes a bottleneck.

Fault Tolerance: With RabbitMQ, if one component fails, the messages are queued until the system can process them again, making your application more resilient.

Setting Up RabbitMQ with Adobe Commerce

To set up RabbitMQ in Magento, follow these steps:

1. Install RabbitMQ You can install RabbitMQ on your server using the following command:

sudo apt-get install rabbitmq-server

2. Configure RabbitMQ in Adobe Commerce You’ll need to update Magento’s configuration files to make it aware of RabbitMQ. Open env.php, located in app/etc/, and add the following configuration settings:

'queue' => [
 'amqp' => [
   'host' => 'localhost',
   'port' => '5672',
   'user' => 'guest',
   'password' => 'guest',
   'virtualhost' => '/'
 ]
],

3. Enable Magento Modules for Messaging

Enable the required modules that support messaging:

php bin/magento module:enable Magento_Amqp

php bin/magento setup:upgrade

Modules like Magento_WebapiAsync and Magento_InventoryAsyncApi can also benefit from RabbitMQ if you’re using asynchronous APIs or MSI.

4. Enable RabbitMQ Management Plugin This plugin provides a UI to monitor and manage RabbitMQ.

rabbitmq-plugins enable rabbitmq_management

Figure 3

Creating a Custom Module for RabbitMQ Integration

Create a module Vendor_RabbitMQ that publishes messages to a queue, which are then received by a consumer.

Step 1: Define the Module (etc/module.xml)

app/code/Vendor/RabbitMQ/etc/module.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:Module/etc/module.xsd">
    <module name="Vendor_RabbitMQ" setup_version="1.0.0"/>
</config>

Step 2: Register the Module (registration.php)

Create app/code/Vendor/RabbitMQ/registration.php:

<?php
\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component\ComponentRegistrar::MODULE,
    'Vendor_RabbitMQ',
    __DIR__
);

Step 3: Create communication.xml under app/code/Vendor/RabbitMQ/etc

This file contains a list of topics. These are intended to contain message queue information shared between implementations.

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Communication/etc/communication.xsd">
    <topic name="vendor_rabbitmq_digitalorder_create" request="string"/>
</config>

Step 4: Create queue_topology.xml under app/code/Vendor/RabbitMQ/etc

This file defines the message routing rules and declares queues and exchanges.

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/topology.xsd">
    <exchange name="vendor-rabbitmq-digitalorder-create" type="topic" connection="amqp">
        <binding id="vendor_rabbitmq_digitalorder_create" topic="vendor_rabbitmq_digitalorder_create" destinationType="queue" destination="vendor_rabbitmq_digitalorder_create"/>
    </exchange>
</config>

Step 5: Create queue_publisher.xml under app/code/Vendor/RabbitMQ/etc

This file defines which connection and exchange to use to publish messages for a specific topic.

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/publisher.xsd">
    <publisher topic="vendor_rabbitmq_digitalorder_create">
        <connection name="amqp" exchange="vendor-rabbitmq-digitalorder-create" />
    </publisher>
</config>

/ Magento\Framework\MessageQueue\PublisherInterface /

$this->publisher->publish(‘vendor_rabbitmq_digitalorder_create’, $order->getId());

Step 6: Create queue_consumer.xml under app/code/Vendor/RabbitMQ/etc

This file defines the relationship between an existing queue and its consumer.

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/consumer.xsd">
    <consumer name="vendor_rabbitmq_digitalorder_create_consumer" queue="vendor_rabbitmq_digitalorder_create" connection="amqp" handler="Vendor\RabbitMQ\Model\Consumer\DigitalOrderConsumer::process"/>
</config>

Step 7: Create DigitalOrderConsumer.php under app/code/Vendor/RabbitMQ/Model/Consumer

<?php

namespace Vendor\RabbitMQ\Model\Consumer;

use Magento\Sales\Api\OrderRepositoryInterface;

class DigitalOrderConsumer
{
    protected $orderRepository;

    public function __construct(
        OrderRepositoryInterface $orderRepository
    ) {
        $this->orderRepository = $orderRepository;
    }

    public function process(string $orderId)
    {
        try {
            $order = $this->orderRepository->get((int)$orderId);
        } catch (\Exception $e) {
            return;
        }
    }
}

Step 8: Run Magento Queue Consumers After configuring RabbitMQ, you can start your consumers, which will listen for messages and perform the required tasks.

php bin/magento queue:consumers:start your_consumer_name

Handling Failures

When the consumer function is triggered, the corresponding message in the queue is locked to prevent concurrent processing by other consumers. If the system encounters an issue during consumption — such as a failure or interruption — it automatically removes the lock by clearing the associated lock ID by using exception handling. The message status is then updated to “InProgress”, making it eligible for reprocessing. In a subsequent attempt, the system will detect this status and reprocess the message accordingly.

try {
    /* Logic for process message which you publish in the queue*/
} catch (\Exception $e) {
  $queue->reject($message, false, $e->getMessage());
  $queue->acknowledge($message);
  if ($lock) {
    $this->resource->getConnection()
        ->delete($this->resource->getTableName('queue_lock'), ['id = ?' => $lock->getId()]);
  }
}

Dead Letter Queue

A Dead Letter Queue (DLQ) in RabbitMQ is a queue that receives messages that can’t be processed successfully by consumers. If a message in RabbitMQ is rejected, not acknowledged, or expired, it can be routed to a DLQ for debugging or reprocessing purposes. Magento 2 doesn’t configure DLQs out of the box, but you can set them up manually through RabbitMQ configuration and queue definitions.

Setting Up DLQ for Magento 2 Queues

1. Configure RabbitMQ Queues with DLX

rabbitmqadmin declare queue \
  name=vendor_module_order_sync \
  durable=true \
  arguments='{"x-dead-letter-exchange":"dlx.exchange"}'

2. Define Dead Letter Exchange and Queue

# Create DLX
rabbitmqadmin declare exchange name=dlx.exchange type=direct durable=true
# Create DLQ
rabbitmqadmin declare queue name=dlq.vendor_module_order_sync durable=true
# Bind DLQ to DLX
rabbitmqadmin declare binding source=dlx.exchange destination=dlq.vendor_module_order_sync routing_key=vendor_module_order_sync

Challenges with RabbitMQ Integration

1. Complex Configuration: Integrating RabbitMQ adds architectural complexity to Magento setups. Misconfigurations can lead to performance issues or lost messages.
2. Learning Curve: Developers may require time to learn and understand RabbitMQ effectively.
3. Dependency Management: Additional dependencies can complicate system maintenance and updates.
4. Error Handling: Requires robust error-handling strategies to manage message failures.
5. Monitoring Needs: Necessitates ongoing monitoring and management to ensure reliability.
6. Resource Intensive: Requires additional system resources, which may impact performance.

Conclusion

Integrating RabbitMQ with Adobe Commerce significantly enhances its capability to handle high loads, improving overall user experience and operational efficiency. As your business grows, leveraging this messaging system can be a game-changer.

Overall, RabbitMQ is a more powerful and feature-rich message broker compared to MySQL MQ, making it a preferable choice for Adobe Commerce users who require advanced messaging capabilities and scalability. However, for users looking for a simpler and more lightweight solution, MySQL MQ may be a suitable option.

Reference Links

1. https://www.rabbitmq.com/docs
2. https://experienceleague.adobe.com/en/docs/commerce-operations/installation-guide/prerequisites/rabbitmq