Will 2025 Be the Year Real-Time Analytics Finally Goes Mainstream?

For years, streaming was too complex and costly. But serverless capabilities and new use cases are making real-time analytics a realistic option for most data teams—a look at how streaming is becoming more accessible in the data space.

Photo by Mathew Schwartz on Unsplash

Since the Hadoop era and the early days of Big Data, one prediction kept coming back year after year: “Next year will be the year of streaming.” There was a constant expectation that eventually, most (all?) data pipelines would evolve to real-time patterns instead of batch processing. Yet, despite the hype, real-time analytics has largely remained limited to tech giants and niche industries with highly specialized streaming needs.

By this time, the general consensus in the data space is “you don’t need streaming” / “batch is (mostly) always enough”. However, I believe 2025 might finally be the year streaming makes sense for common data use cases. This year, two key factors might allow real-time analytics to break out of its niche and finally hit the mainstream.

Streaming’s unfulfilled promise

Streaming is exciting. The ability to process and analyze data at scale in real time, unlocking insights and enabling immediate action, has been a core promise of data platforms ever since the Hadoop era. But then the realities of its technical challenges, coupled with the lack of concrete and valuable-enough use cases, often lead data teams to the catch-all “But do we really need it?” counter-argument.

Technologies like Apache Storm and Apache Flink solved many of the technical hurdles, but the fundamental constraints remained: Building and maintaining streaming pipelines required complex infrastructure, specialized skill sets, and significant financial investment. As streaming for pure data movement gained traction in the software engineering world, with patterns like event-based architectures and a meteoric rise for Apache Kafka, the data world remained stuck with batch processing, and architectures like the one below became the standard:

Sample (highly-abstracted) data movement and transformation architecture (image by author)

This architecture often resulted in data arriving at the data platform’s landing zone (typically an object store) in real-time but only being moved and transformed in batches, often on a daily schedule. Two arguments typically justified such architectures:

• Ingesting and transforming data in real-time within the warehouse is more complex and expensive than batch processing.
• The value generated by real-time use cases doesn’t justify the added cost and complexity.

However, these arguments are rapidly losing their validity. Let’s explore the two major shifts that could make 2025 a tipping point for real-time analytics.

The democratization of streaming technology

Over the past few years, major cloud data platforms like Snowflake and Databricks have steadily released features and capabilities designed to simplify real-time data ingestion and processing.

Snowflake, after initially releasing Snowpipe for automated ingestion from external files (which still required data to be staged in an object store), introduced Snowpipe Streaming in 2023. This allowed users to leverage a new SDK to consume real-time data directly from streaming systems like Kafka in a serverless manner, with potentially very reasonable costs when configured correctly. Additionally, in 2024, Snowflake introduced Dynamic Tables, enabling users to build declarative data pipelines with near real-time freshness guarantees (as low as minutes!) without the need for a separate orchestrator. While cost optimization still requires careful configuration and pipeline design, the reduced maintenance burden and streamlined architecture can make the investment worthwhile in many scenarios.

Importantly, both Snowpipe Streaming and Dynamic Tables are compatible with Snowflake’s Apache Iceberg tables, which means that you can maintain an independent storage layer while still benefiting from a streaming architecture powered by serverless features.

And this is not limited to Snowflake. Similar trends are evident across other major data platforms. Databricks launched Delta Live Tables in 2022 and made them available for serverless use cases last year, while BigQuery continues to enhance its streaming capabilities within the GCP ecosystem with features like BigQuery Subscriptions. These new capabilities, combined with a design that leverages their serverless nature for cost optimization, make streaming architectures a viable alternative to batch-based options. While some complexity and cost considerations remain, the landscape is drastically different from five years ago when streaming was often prohibitively complex, expensive, and simply a bad idea for most teams.

The same data architecture with the introduction of serverless real-time features (separate storage can still be maintained via Apache Iceberg) — image by author

These features enable the creation of end-to-end streaming pipelines with minimal effort, using SQL and Python — languages familiar to most data professionals —, and leveraging serverless components that can be heavily optimized for cost while minimizing the maintenance burden, even at scale. This is a far cry from the requirements for real-time data transformation five years ago.

However, it’s important to note that if your current batch pipelines are effectively meeting your needs, a migration to streaming might not be the best approach (yet) — the barriers are lower, but they haven’t disappeared entirely.

The expanding universe of real-time use cases

The second key factor driving streaming’s potential in the analytics space is that its concrete use cases now extend far beyond operational dashboards. As organizations reach higher levels of data maturity after years of relying on batch processing, they’re gaining more clarity on the yet-to-be-tackled challenges that demand real-time data at scale. To be more specific, we can mention the following use cases:

• Real-time data anomaly detection: In an era where data contracts are becoming the new norm, enforcing these contracts (by detecting contract violations) still happens in a delayed manner due to batch processes. For instance, if a semantic change alters data in a critical table, a daily batch process might mean a full day of corrupted data before a data quality check flags the issue. With a real-time pipeline, the problem could be detected in minutes, and the check itself would still be executed via a simple SQL query on the data platform.
• Real-time data activation: A common data activation use case involves using data from one SaaS system to trigger actions in another (e.g., using CRM data to trigger automated marketing processes). Historically, teams either directly connected these tools (and opened the door to ad-hoc and non-governed data transformations) or ingested data into the warehouse and then moved it to different destinations via a Reverse ETL process (so that all transformations remain governed). The latter approach, however, lacked real-time capabilities due to the data platform’s batch pipelines, blocking many valuable use cases such as triggering a marketing automation sequence a few moments after a user performs a specific action. Having real-time pipelines that flow through the data platform opens the door to such scenarios in a governed and scalable manner, and can be extended to other data activation use cases like personalization engines.
• Real-Time “advanced” metrics monitoring: Many SaaS tools offer real-time access to basic metrics that you can monitor (e.g., button clicks). But once you want to calculate more complex product analytics metrics that combine data from multiple systems (like correlating multiple UI and backend events), things become more tricky. With real-time analytics at the data platform level, even metrics that combine multiple sources (like product data mixed with the context coming from your CRM and your marketing tool) would be available in near-real-time, so that you’re immediately alerted if they deviate from expected ranges.

I’ve focused these examples on the analytics space because more generic streaming use cases (like operational monitoring, AI/ML fine-tuning, and fraud detection) were already achievable even when analytics are done in batch (Lambda architecture to the rescue), but moving analytics to a real-time pattern would drastically simplify the overall architecture and how data moves across systems. (So, yes, Kappa is cool again.)

Not for everyone (yet), but a big step forward

Although things are advancing (fast) in the right direction and the potential is exciting, batch processing is here to stay. Streaming won’t suddenly become the default for every data team or every use case, and batch will continue to be the better fit for tasks where latency isn’t a (major) concern. Furthermore, challenges like limited streaming support in current tooling and frameworks downstream of cloud data platforms and the still-inevitable (but more manageable) elevated costs still need to be addressed. Nevertheless, the convergence of more accessible technology and expanding use cases means that streaming will become far more commonplace this year. The cost-benefit analysis is shifting, making real-time a viable option even for smaller data teams with limited resources.

If you’re wondering what this looks like in practice, Canva’s data team recently shared a great blog post about their real-time analytics architecture, illustrating both the technical implementation and the associated costs.

We’ve been hearing about the “year of streaming” for more than a decade, and after many underwhelming years, skepticism is definitely understandable. But the landscape is genuinely changing. With powerful, low-maintenance tools and a growing need for real-time data, 2025 might finally be the year that streaming analytics moves beyond its niche status and becomes a fundamental part of the modern data stack. However, like any other area in the world of tech, everything everywhere is always moving, forever — so many possible developments could still delay the widespread adoption of streaming, even with the technical hurdles seemingly resolved. (Looking at you, AI).

For more data engineering content you can subscribe to my newsletter, Data Espresso, in which I discuss various topics related to data engineering and technology in general:

[embed]Data Espresso | Mahdi Karabiben | Substack Data engineering updates and commentary to accompany your afternoon espresso. Click to read Data Espresso, by Mahdi…dataespresso.substack.com