Why “Database as Queue” Breaks Down at Scale

Introduction

When I started my software engineering career, most of the applications I built followed a simple and familiar architecture: a relational database (usually MySQL or PostgreSQL), a Java or .NET application server, and a web or mobile frontend.

At that time, the systems I worked on handled only a few thousand transactions per minute, so this architecture was more than sufficient. When we faced performance issues, we simply scaled the database vertically. It was simple, reliable, and worked well for the scale we were operating at.

As our applications evolved, PostgreSQL became a popular choice because of its powerful JSON support and flexible querying capabilities. In many cases, it allowed us to combine relational data with semi-structured JSON data without introducing a separate NoSQL database.

As I progressed in my career and started working on large-scale distributed systems, I realized that this architecture was no longer enough. Building systems that serve millions of users while remaining highly available, scalable, resilient, and performant required a completely different approach.

Instead of relying on a single database and a monolithic application, modern architectures consist of multiple specialized components — SQL and NoSQL databases, distributed caches, search engines, load balancers, microservices, and messaging platforms such as Kafka, RabbitMQ, Amazon SQS, and Redis Streams.

Initially, I kept asking myself a simple question:

Why do we need so many different technologies?

The answer became clear as I spent time learning distributed systems, cloud-native architecture, design patterns, and real-world engineering case studies. Every component exists to solve a specific problem at scale, and every architectural decision comes with its own trade-offs.

One of the biggest lessons I learned is that distributed system design is not about choosing a single database or technology. It is about understanding the strengths, limitations, and trade-offs of each component, and using the right tool for the right problem.

This series is a collection of those lessons — exploring why modern distributed systems are designed the way they are and how to make better architectural decisions as systems grow in scale.

The Problem

As I started working on more large-scale systems, I noticed an interesting pattern. Many teams preferred solving problems using the technologies they already had instead of introducing new components into the architecture. In many cases, this is the right approach. Following the KISS (Keep It Simple, Stupid) principle helps keep systems easier to build, operate, and maintain.

However, over time I also realized that every architectural decision comes with trade-offs. Before adopting any solution, it is important to understand not only the problem it solves but also the challenges it introduces, especially when the system grows in scale.

One pattern I repeatedly came across was using a relational database as a queue. At first, I found this idea quite interesting. Instead of introducing a dedicated messaging system like Kafka, RabbitMQ, Amazon SQS, or Redis Streams, many teams simply stored events in a database table and used background workers to process them asynchronously.

The approach is straightforward. As part of a business transaction, the application writes both the business data and the event into the same database transaction. Background workers then continuously poll the event table, fetch pending records, process them, and mark them as completed. Databases such as PostgreSQL even provide features like FOR UPDATE SKIP LOCKED, allowing multiple workers to process events concurrently without picking the same records.

At first glance, this looks like an excellent solution. It keeps the architecture simple, avoids introducing another distributed component, and eliminates the dual-write problem because both the business data and the event are committed atomically within the same transaction.

For systems with moderate traffic, I completely agree that this can be a practical and effective solution.

But as I continued designing systems for much larger workloads, a new question came to my mind.

Does this approach still work when the system starts processing millions of events?

To answer that, we first need to understand how relational databases are designed internally, what they are optimized for, and how different data access patterns affect their performance. Only then can we properly evaluate the strengths and limitations of using a database as a queue.

The Data Access Pattern

This is one of the most important concepts in system design — and surprisingly, one of the most overlooked.

Many engineers start by asking questions like:

• “Should I use MySQL or Cassandra?”
• “Should I use PostgreSQL or DynamoDB?”

Experienced engineers usually start with a different question:

“What are my data access patterns?”

Once you understand the access patterns, the database choice often becomes much more obvious.

A Data Access Pattern describes how an application stores, retrieves, updates, and processes its data over time. It helps us understand the characteristics of the workload rather than just the structure of the data.

It answers questions such as:

• How is data written?
• How is data read?
• How frequently is it accessed?
• Which components access the data?
• Is access random or sequential?
• What are the latency requirements?
• Does the data change frequently?
• Is strong consistency required?
• Is the data accessed by primary key, secondary indexes, or multiple attributes?

A simple way to think about it is:

Data Model describes what the data looks like. Data Access Pattern describes how the application uses that data.

Before selecting a database, experienced architects first try to understand the application’s access patterns:

• Read pattern: How is data typically retrieved? By primary key, secondary index, time range, or full-text search?
• Write pattern: Is the workload append-only, update-heavy, delete-heavy, or a combination of these?
• Query pattern: Do you need joins, aggregations, full-text search, graph traversal, or analytical queries?
• Scale pattern: How many reads and writes per second will the system handle, and how will that grow over time?
• Consistency pattern: Are ACID transactions required, or is eventual consistency acceptable?
• Growth pattern: Will the workload remain evenly distributed, or will certain keys, partitions, or tenants become hotspots?

Once you understand these access patterns, the architectural decisions become much clearer. Instead of searching for a single database that solves every problem, you start selecting technologies that are optimized for specific workloads. In many cases, the right solution is not to replace your primary database, but to complement it with other specialized datastores. Each datastore excels at a particular access pattern, and understanding those trade-offs is the key to designing scalable, resilient, and efficient systems.

As we’ll see in the next section, a queue generates a very different access pattern from a traditional OLTP application. Understanding that difference is the key to evaluating when the Database as Queue pattern is a good choice and when it can become a scalability bottleneck.

Real Production Examples

After understanding the importance of data access patterns, I started looking at several production systems that were using the Database as Queue pattern.

Over the last few years, I found multiple teams using this approach successfully in production. Their motivation was simple — to keep the architecture simple by avoiding an additional messaging system. This aligns well with the KISS principle, and for many workloads, it can be a perfectly valid solution.

What caught my attention was the scale at which some of these systems were operating. A few of them were designed to process millions of records per minute while still using a relational database as their queue.

At this point, I became curious. Instead of asking whether this was the right or wrong approach, I started asking a different question:

“Does the data access pattern of these systems match what a relational database is optimized for?”

To answer that, I analyzed the workloads of a few real production systems.

Product 1 (large scale schedule generation service)

Data Access Pattern (per minute)

• Fewer than 100 inserts
• 1–2 million record reads
• 1–2 million record updates (re-queuing records)
• No deletes
• Strong read-after-write consistency (all reads served from the primary database)

Product 2 (large scale job execution platform)

Data Access Pattern (per minute)

• 1 million record inserts
• 1 million record reads
• Approximately 5 million record updates (multiple state transitions)
• Bulk deletion of approximately 5 million processed records every 5 minutes
• Strong read-after-write consistency (all reads served from the primary database)

At first glance, both systems appeared to work well. However, when I looked closely at their data access patterns, I noticed something interesting. These workloads were very different from a typical OLTP application and raised an important question:

Is a relational database really optimized for this kind of workload, or are we forcing it to solve a problem it wasn’t primarily designed for?

Let’s first understand how relational databases work internally before answering that question.

Why Database as Queue Becomes an Anti-pattern

The production examples raised an important question:

Why do these data access patterns become a problem for a relational database?

To answer that, we first need to understand how relational databases such as PostgreSQL are designed internally.

In recent years, I’ve worked extensively with PostgreSQL, so I’ll use it as the primary reference throughout this article.

Before diving into the internals, it’s important to distinguish between two concepts.

• Database as Queue is a design pattern where a relational database stores work items, and background workers continuously fetch, process, and update those records.
• An anti-pattern is different. A design becomes an anti-pattern when it is applied beyond the workload it was intended to handle. The problem is not that the approach is incorrect — it is that the underlying data access pattern no longer aligns with what the database is optimized for.

If we look back at the production examples, most of the database activity is no longer related to managing long-lived business data. Instead, the majority of operations are dedicated to managing queue state:

• Reading the next work item
• Marking it as In Progress
• Re-queuing it when processing fails
• Updating its state multiple times during processing
• Eventually deleting the processed records

In other words, the database spends a significant portion of its resources managing short-lived queue operations rather than serving transactional business data.

This is why many experienced architects consider Database as Queue an anti-pattern for certain large-scale workloads. Not because PostgreSQL or MySQL cannot handle it — they certainly can, and for many systems it is a perfectly valid starting point. The challenge begins when queue traffic dominates the workload.

At that point, the database starts spending a considerable amount of effort managing queue state instead of efficiently serving business transactions. This leads to increased write amplification, more WAL generation, heavier index maintenance, growing MVCC cleanup pressure, and higher operational overhead. As the workload continues to grow, these characteristics align much more closely with a dedicated messaging or log-based system than with a relational database.

How Does This Typically Show Up in Production?

The first signs are usually visible in production long before the system reaches its limits. Common symptoms include:

• Increasing database CPU utilization
• Higher disk I/O due to continuous writes and background cleanup
• Increased WAL generation and replication lag
• Growing table and index bloat
• Higher query latency and reduced throughput
• Network saturation between the application and the database
• Degradation of database SLOs and SLIs, eventually impacting the application’s SLA

The important point is that these symptoms are often consequences of the workload itself rather than PostgreSQL or MySQL being “slow.” Understanding why this happens requires looking at how a relational database manages writes, updates, indexes, and concurrency internally.

What PostgreSQL Is Optimized For

To understand why the Database as Queue pattern can become a scalability bottleneck, we first need to understand what PostgreSQL is designed to optimize.

PostgreSQL, like most relational databases, is built for Online Transaction Processing (OLTP) workloads, where data represents durable business entities such as customers, orders, payments, or products. In these systems, correctness, consistency, and durability are more important than maximizing write throughput.

To achieve this, PostgreSQL uses mechanisms such as B+ Tree indexes for efficient lookups, Write-Ahead Logging (WAL) for durability and crash recovery, and Multi-Version Concurrency Control (MVCC) for high concurrency and transactional consistency. These features intentionally perform additional work during writes to guarantee data integrity, efficient reads, and reliable recovery.

This works exceptionally well for traditional OLTP workloads. However, a queue generates a very different data access pattern — continuous inserts, reads, updates, retries, and deletes of short-lived records. As queue traffic begins to dominate the workload, PostgreSQL spends more time managing transactional metadata than processing business data.

Note: When using a relational database as a queue, you can often improve scalability by optimizing the schema and data access patterns.

For example, separate highly mutable fields (such as queue state, retry count, or last updated timestamp) from relatively immutable business data. This reduces MVCC overhead, minimizes row versioning, and makes frequent updates more efficient.

Likewise, avoid large-scale row-by-row deletes wherever possible. Techniques such as table partitioning (for example, PostgreSQL partitioning or pg_partman) allow old partitions to be dropped efficiently instead of deleting millions of rows.

The goal is not to replace the database at the first sign of scale, but to ensure the workload aligns with what a relational database is designed to handle. Optimize the design first, understand the trade-offs, and introduce a dedicated messaging system only when the workload fundamentally exceeds the strengths of a relational database.

Recommended Architecture

If your data access pattern analysis shows that a relational database is no longer a good fit for queueing, consider separating the queue from the transactional database.

A common approach is to use a dedicated messaging system such as Kafka, RabbitMQ, Redis Streams, Amazon SQS, or another queue that best fits your workload. However, writing directly to both the database and the messaging system introduces the well-known dual-write problem, where one operation may succeed while the other fails.

A widely adopted solution is the Transactional Outbox Pattern. As part of the same database transaction, the application writes both the business data and an event to an outbox table. A CDC (Change Data Capture) engine then continuously publishes these events from the outbox table to the messaging system, ensuring reliable event delivery while decoupling queue processing from the relational database.

This approach preserves transactional consistency, avoids the dual-write problem, and allows each component to focus on the workload it is best suited for.

For a detailed explanation of the dual-write problem and the Transactional Outbox Pattern, see the Confluent article:

[embed]Understanding the Dual-Write Problem and Its Solutions The post discusses the Dual-Write Problem in distributed systems, where atomic updates across multiple systems like…www.confluent.io

Final Thoughts

While discussing these systems with their respective engineering teams, I analyzed how the Database as Queue pattern behaved as the workload continued to grow. In one case, the system experienced increased xactsync latency in Amazon Aurora PostgreSQL under a write-intensive workload, highlighting how a design that works well at one scale can become a bottleneck as the workload evolves.

This doesn’t mean the original design was wrong. Every system is built based on the requirements, assumptions, and trade-offs that exist at a particular point in time. As the business grows and the workload changes, the architecture must evolve as well.

This is why understanding data access patterns is so important. Good architecture is not about choosing the most popular database or technology — it’s about selecting the right tool for the workload you’re trying to solve.

With good observability, teams can detect these bottlenecks early, understand how the workload is evolving, and optimize the architecture incrementally. However, making the right architectural decisions early is always beneficial, especially in large-scale systems where replacing core components later can be complex, expensive, and disruptive.

The goal of this article is not to discourage the Database as Queue pattern. It is to encourage engineers to understand its strengths, recognize its limitations, and make informed architectural decisions based on their application’s data access patterns, scalability requirements, and trade-offs.