MirrorMaker 2 in Production — How We Built a Real-Time Data Backbone Across Organizations

Imagine a customer visits an electronics subsidiary’s app to purchase a high-end gaming laptop, only to find the item is currently out of stock. In a traditional retail environment, that "Out-of-Stock" event is often treated as static data buried in a weekly inventory report. It might take days for that intent-to-purchase data to be analyzed and shared across the group’s other brands—if the information is shared at all. By the time a home office or furniture subsidiary identifies this customer as a prime lead for chairs or specialized peripherals for their new setup, the customer has already completed their entire purchase with a competing global retailer.

By implementing a Unified Real-Time Backbone, this inventory gap is transformed into a strategic pivot. The moment the laptop search fails, the event is replicated from the subsidiary to the group’s central backbone. The Home Office subsidiary instantly “listens” to this stream and triggers a real-time offer: “Setting up a new rig? Get 20% off our pro-gaming chairs while you wait for your laptop.” Simultaneously, this allows the electronics entity to notify the customer that the laptop is available for immediate pickup at a nearby location. This “Data Liquidity” ensures that an inventory “No” at one subsidiary becomes a successful “Yes” for the group.

This ‘Opportunity Gap’ isn’t unique to retail; it is a systemic challenge across the financial sector as well.

Another Example would be if a customer applies for a credit card at a bank and, due to specific internal criteria, receives a “Rejected” status. In a traditional banking environment, that data is immediately locked away in a silo. It might take days or even weeks for that information to trickle down to other subsidiaries — if it ever moves at all. By the time a subsidiary specializing in credit cards identifies this customer as a prime lead for a different product, they have likely already signed with a competitor.

I decided to bridge this “Opportunity Gap” by turning a static rejection into a real-time strategic pivot.

The Vision: Data Liquidity Across the Group

My goal was to build a Unified Real-Time Backbone that connected multiple distinct entities — each specializing in a related function — into a single, fluid ecosystem. I envisioned a system where:

• Instant Cross-Selling: The moment a customer is rejected at the group level, the relevant subsidiary receives a real-time event.
• Proactive Engagement: Within seconds, the subsidiary triggers an automated campaign to enroll the client in a more suitable product.
• Seamless Integration: Any entity within the group could “plug in” to the data stream to satisfy internal use cases without waiting for batch migrations or complex data pipeline.

II. Section II: The Engine Under the Hood — What is MirrorMaker 2?

To make this vision a reality, I had to solve a massive architectural puzzle. I was tasked with bridging two very different worlds: a central Confluent Kafka Cluster at the parent organization and a distributed Apache Kafka at the subsidiary level. Because I was operating in a highly regulated environment, I couldn’t just “copy and paste” data. I needed a solution that was:

1. Fully Secured: Using TLS authentication, complex ACLs, and certificate-based authorization.
2. Bidirectional: Ensuring data flowed seamlessly both ways between the entities.
3. Production-Resilient: Capable of handling massive throughput while avoiding common pitfalls like infinite replication loops.

This is where MirrorMaker 2 (MM2) became the cornerstone of my architecture.

What You’ll Learn in This Deep-Dive While basic Kafka tutorials are common, production-grade deep-dives into multi-org MirrorMaker 2 setups are rare. In this article, I will share the hard-won lessons from the “production trenches,” including:

• The Hybrid Architecture: How I bridged OpenShift-based Kafka with Confluent clusters.
• Security & Certificate Blueprints: My step-by-step process for creating JKS-format Truststores and managing Subject Alternative Names (SAN) for external listeners.
• The “Confluent-to-Apache” Secret: How to use config.properties.exclude to stop Confluent-specific metadata from breaking your open-source Kafka replication.
• Solving the Memory Wall: Why my brokers kept hitting OutOfMemoryException and how I tuned resource limits for stable startup.
• Host Alias Mastery: Using the Strimzi Operator to manually map IPs within pods to ensure connectivity across entity boundaries.

This next section is critical because it bridges the gap between the business “why” and the technical “how.” To make this architecture section stand out, I recommend a transition from a general explanation of the tool to a high-level view of your specific, production-hardened setup.

Section II: The Engine Under the Hood: What is MirrorMaker 2?

A Note on the Implementation: While the architectural concepts discussed here apply to any Kafka environment, I will be using Strimzi (the Kafka Operator for OpenShift) as the primary example. All YAML configurations and deployment strategies shown are production-tested within an OpenShift ecosystem.

Before diving into the complex multi-org setup, it is important to understand the engine under the hood. MirrorMaker 2 (MM2) is the industry-standard framework for replicating data between Kafka clusters. Unlike its predecessor, MM2 is built on the Kafka Connect framework, which makes it more resilient, scalable, and significantly easier to manage in production environments.

In an enterprise “Backbone” architecture, MM2 does more than just copy data; it manages the entire lifecycle of cross-cluster communication through three specialized connectors. Each plays a distinct role in ensuring data integrity and observability:

1. The Source Connector: The Data Mover

• What it does: This is the primary worker that identifies the topics matching your defined patterns and creates the topics with the same configuration and replicates the actual records from the source cluster to the target.

2. The Checkpoint Connector: The Offset Synchronizer

• What it does: This connector periodically emits “checkpoints” that track the offsets of consumer groups in the source cluster and maps them to the corresponding offsets in the target cluster.
• Note: This is the “failover hero.” it’s very important if you are building a DR cluster and the same consumer will read from source and target, It ensures that if a consumer needs to switch from source cluster to the target cluster, it can resume exactly where it left off. Without this, consumers would risk either missing events or being flooded with duplicate data they’ve already processed.

3. The Heartbeat Connector: The Health Monitor

• What it does: It emits a “heartbeat” signal between the clusters to verify that the replication path is active and healthy.
• Note: This is your go-to check to test if the replication is happening and your MM2 instance is running.

At its core, an MM2 deployment consists of three logical pillars:

1. Cluster Definitions:

This is where you define the “Source” and “Target” clusters, including their bootstrap servers and security protocols, Because I was bridging two organizations, I had to define Cluster Aliases, bootstrap servers, and the specific Security Certificates (TLS) for each.

clusters:
    - alias: Cluster-1
      authentication:
        certificateAndKey:
          certificate: X-Cert.cer
          key: X.key
          secretName: X-user
        type: tls
      bootstrapServers: >-
        xxx.xx.xxx.xxx:9091,xxx.xx.xxx.xxx:9091,...
      config:
        ssl.endpoint.identification.algorithm: ''
      tls:
        trustedCertificates:
          - certificate: X_CA.crt
            secretName: X-cluster
    - alias: Cluster-2
      ...

2. Mirrors:

This layer defines the data flow. It defines the source cluster and target cluster (which we configured before) & defines what topic pattern you want to replicate (in the below code snippet i will be replicating any topic in the source cluster which name starts with PRE_), also you can add any Kafka connect configuration for each connector if needed.

mirrors:
      sourceCluster: Cluster-1
      targetCluster: Cluster-2
      groupsPattern: .*
      topicsPattern: PRE_.*

Important takeaways:

• Topic/Group Patterns: You don’t have to replicate everything; you can use “White-lists” or patterns to ensure only the relevant financial events (like loan rejections) cross organizational boundaries.
• Without topic patterns any topic created in the source will be automatically created and replicated on the target which might cause disasters
• Having topic patterns is very important as in Production you will most likely create a user for Mirrormaker2 with some permissions on a prefix to avoid any unnecessary replication

3. Connector Customization:

Checkpoints, Heartbeats & Source connectors might need customization based on your setup and needs.

mirrors:
      ...
      - checkpointConnector:
        config:
          checkpoints.topic.replication.factor: 3
      heartbeatConnector:
        config:
          heartbeats.topic.replication.factor: 3
      sourceConnector:
        config:
          replication.factor: 1
          offset-syncs.topic.replication.factor: 3

While most people use MM2 for simple Disaster Recovery (DR), I used it as a strategic integration layer to create a bidirectional data highway between different corporate entities.

Section Ill: The Architecture: Multi Org Real-Time Data Replication

Building a real-time data layer for a major group was never going to be a “one-off” project. It was a strategic program designed as a standard for all subsidiaries to eliminate data silos across four distinct entities. The architecture decision I made was to move away from legacy batch processing and build a Unified Real-Time Backbone where the main organization acts as the central nervous system for its subsidiaries.

1. The Micro View: Connecting a Single Subsidiary

The core of this setup involved bridging two distinct Kafka ecosystems: a subsidiary running Apache Kafka on OpenShift (managed by Strimzi) and a main organization utilizing a Confluent Cluster. I strategically decided to deploy the MirrorMaker 2 (MM2) instance on the subsidiary side. This “Decentralized Bridge” approach ensures that the subsidiary maintains full control over its replication workloads and resource scaling without adding overhead to the central backbone.

Both clusters were SSL encrypted and Authorization was enabled.

Critical Implementation Advice: The “Production Trenches”

Building this link taught me that the “default” configurations found in most documentation are rarely sufficient for production-grade stability.

• Pro-Tip on Connectivity: A common pitfall in enterprise networking is whitelisting only the bootstrap server’s IP. Because MM2 pods are dynamic and can be scheduled on any worker node within the OpenShift cluster, you must ensure that your network request covers connectivity from all worker node IPs to the target cluster’s bootstrap servers
• The Memory Wall: By default, Kafka broker memory limits are often too low for the heavy lifting required during a multi-org sync. To avoid the dreaded OutOfMemoryException during pod startup, I recommend increasing memory limits to at least 4Gi
• The Confluent-to-Apache “Secret”: If you are replicating from a Confluent environment to standard Apache Kafka, you will inevitably encounter errors where the target cluster rejects Confluent-specific metadata. My solution was to use the config.properties.exclude parameter within the MM2 configuration to strip out these incompatible properties, ensuring a seamless data flow
• Security & Persistence: In a regulated environment, always set your Authorization and Authentication to TLS (SSL) for a fully secured cluster. . Furthermore, ensure you are using Persistent Volumes for each broker; this guarantees that even if a pod fails, your data remains safe and synchronized
• Storage Sizing: To size your storage needs, you would calculate the estimated Daily Data Size x Replication Factor x Retention in Days while adding buffer.
• Kafka Encrypted Listeners: Subject Alternative Names must be added to the TLS encrypted Kafka listener, or you would face SSL issues while connecting

listeners:
      - authentication:
          type: tls
        configuration:
          bootstrap:
            alternativeNames:
              - <Cluster Name>-kafka-tls-bootstrap
              - <Cluster Name>-kafka-tls-0
              - <Cluster Name>-kafka-tls-1
              - <Cluster Name>-kafka-tls-2
        name: tls
        port: 9093
        tls: true
        type: route

• Kafka’s Host Aliases: we will need to adjust the hosts file in our pod [/etc/hosts] but all pod configuration are handled by the Operator so we need to add the required host entries using Host Aliases, and this solves the issues with connecting to Confluent Kafka as the certificates for Confluent will have IPs which your Kafka cluster doesn’t know about.

template:
     pod:
       hostAliases:
         - hostnames:
             - p1sxxxxxx.main.entity
           ip: xxx.xx.xxx.xxx

2. The Macro View: Scaling to a 4-Entity Unified Backbone

Once the initial bridge between the first subsidiary and the main organization was stable and secured, I moved to the program’s ultimate goal: scaling this architecture across all four entities

The Hub-and-Spoke Model with a Twist

In this macro architecture, the main organization acts as the central Hub, while each subsidiary functions as a Spoke. However, unlike traditional one-way data pipelines, this backbone is a living, bidirectional ecosystem.

• Decentralized Deployment: To maintain performance and autonomy, I deployed a dedicated MirrorMaker 2 (MM2) instance within each subsidiary’s OpenShift cluster. This allowed each entity to manage its own replication workloads and scaling requirements without creating a single point of failure at the central hub.
• The Power of Repeatability: By using the Strimzi Operator, I was able to treat the “Micro View” as a blueprint. The same declarative YAML configurations — from TLS security to resource limits of 4Gi — were replicated across all entities, ensuring consistent governance and a faster rollout.

Operational Excellence: Cross-Entity Synergies

The true value of the Macro View isn’t just the connectivity; it’s the business synergy it enables. With all four entities plugged into the same backbone, we achieved what I call “Data Liquidity”

1. The Real-Time Lead Recovery System: In this 4-org setup, a “Loan Rejection” event at the main org doesn’t just sit in a database. It is instantly replicated to the backbone, where the Loans Subsidiary and Credit Card Subsidiary can both “listen” to that stream. Within seconds, a potential lost customer is identified and offered an alternative product tailored to their financial profile.
2. Isolated Governance: While the data flows through a central backbone, I utilized KafkaUser configs and strict ACLs to ensure that Subsidiary A could never see the private financial streams of Subsidiary B unless explicitly authorized.
3. Heterogeneous Stability: By consistently using config.properties.exclude across all four links, I ensured that the different versions and flavors of Kafka (Confluent vs. Apache) across the entire group communicated without a single metadata error.

Conclusion: From Data Silos to a Living Ecosystem

Building this Unified Real-Time Backbone was about more than just moving data; it was about shifting an entire financial group from a state of static, fragmented information to a living ecosystem. By dismantling legacy silos and replacing them with a production-hardened MirrorMaker 2 architecture, I successfully transformed customer engagement from a process that once took days to one that now happens in seconds.

The results speak for themselves. In the traditional setup, a loan rejection was a missed opportunity buried in a database, often taking days to reach a subsidiary. Today, that same rejection serves as a real-time lead. Because the data now flows across the backbone instantly, a subsidiary can reach out and enroll a client in an alternative product before they even have the chance to consider a competitor.

This transformation was not just a theoretical success but a technical one, achieved by:

• Securing the Core: Leveraging the Strimzi Operator on OpenShift to implement a fully secured cluster using TLS authentication and authorization.
• Production Hardening: Applying critical lessons from the “trenches,” such as increasing memory limits to 4Gi to prevent startup failures and using config.properties.exclude to bridge the gap between Confluent and Apache Kafka.
• Ensuring Data Liquidity: Creating a bidirectional highway that connects four distinct organizations into a single, synchronized entity.

For any data engineer working in a highly regulated industry, this project proves that you don’t have to choose between security and speed. With a strategic approach to MirrorMaker 2, you can build a system that is as resilient as it is fast, turning your data into your organization’s greatest competitive advantage.

I hope these lessons from my time building a unified backbone help you avoid the common pitfalls of multi-org Kafka. Have questions or need a bit of support with your own deployment? I’m just a message away on **LinkedIn. Let’s connect and talk all things Kafka, Lakehouses, and production-grade Data engineering.**