Behind the Scenes at Flink Forward 2025 | Interview with Erik Schmiegelow, Hivemind

**Hivemind specializes in developing comprehensive data processing solutions for businesses across various industries. Since 2014, the company has offered expert services in data engineering, data analytics, machine learning, and real-time streaming data processing. As one of the sponsors of [Flink Forward 2025](https://www.flink-forward.org/)**, they caught my attention, and I had the opportunity to speak with Erik Schmiegelow, director and co-founder.

Here’s my conversation with Erik about why companies are rethinking expensive data platforms, the shift from batch to streaming, and how real-time data unlocks AI’s true potential:

Jana: Hivemind has been around since 2014. What was the founding vision, and how has your approach to data engineering evolved over the past decade?

Erik: We actually didn’t plan to be a consultancy — we just became one over time. Initially, we started with a product. We wanted to make real-time assessments of social media streams. This was 2014, and the only tools available were Kafka and Hadoop. We had these crazy streams and signals from social media in amounts we hadn’t seen before, except maybe in banking with financial transactions. The product challenges were intense, and we learned so much.

Unfortunately, the platforms like Facebook and Twitter suddenly started charging for their streams, and the costs didn’t align with our client revenue, so we had to pivot. But we had a huge asset: the incredible amount of knowledge we’d gathered building that platform, plus the tooling and approaches. We realized we knew how to solve these problems, and there were a lot of clients out there with exactly the same issues.

From there, we moved into financial services, advertising, and commerce, helping clients build successful streaming-oriented platforms. Back then, ten years ago, everything was super batch-oriented, and it still is, but not as bad. Over time, we discovered our strength was in thinking differently — switching the paradigm. Rather than waiting until X hours have passed to look back and process data, we flip that. Everything is a stream, and batch is just a large window of a stream. It’s all about creating a resilient, fault-tolerant architecture with consistent snapshotting, resumption points, and checkpointing so that even if everything stops for days, you can resume and catch up.

Jana: Who are your typical clients? What industries do you work with?

Erik: We’re industry-agnostic in the sense that we help clients across all industries, but certain sectors really need real-time intelligence and decision-making more than others: energy, mobility, manufacturing, and finance. These are typically industries dealing with IoT and sensor data collection.

Electric mobility is a very strong sector. The difference between a combustion engine car and an electric one is huge. A combustion engine is mechanical, while an electric vehicle is basically a computer on wheels collecting data all the time. The game changer in fleet management is understanding that data stream and optimizing your fleet accordingly. You can save energy, optimize charging lifecycles, and optimize routes based on the data you collect. The visibility comes with data collection, and companies realize there’s a lot of untapped potential for optimization just by collecting and making sense of data in real time.

The other typical industries are the usual contenders for streaming: payment systems, banks, and advertising. Things like real-time profile prediction, buying intention predictions for better advertising, or cross-selling based on customer profiles — all real-time decision making and streaming.

We have two home markets: London and Berlin. There are real differences in the clients we have in Central Europe versus the UK. The UK is more service-oriented, so it’s more financial services, but there’s a completely different approach in how they apply the technology in both markets.

Jana: What are the most common misconceptions clients have about streaming complexity when they first approach you?

Erik: The number one thing clients say is, “Oh my God, it’s going to be so expensive!” There’s a fallacy driven by vendor marketing where people think streaming means huge investments. If they’re thinking about Confluent, Snowflake, or Databricks, there’s evidence those products can be very expensive. But that’s not the whole story.

If you have an MVP approach, looking at a specific use case and building something that exactly serves that purpose and nothing else, it’s very cheap. Often, switching to streaming is much cheaper than batch.

I’ll give you an example from five or six years ago. A company was running a huge post-calculation system on a Qlik platform on top of a cluster of eight servers with something like two terabytes of RAM. That batch process ran for eight hours every night, doing the same calculation over and over again. The operational costs were in the hundreds of thousands of dollars.

We switched them to streaming with incremental updates and calculations. It was a relatively simple aggregate report for commerce. We moved it to a simple cluster and slashed costs by 80%. All those machines weren’t necessary because we were just continuously updating and incrementing the calculations, with one scheduled sanity check run. The combination of stream processing, stream aggregation, and a validation framework running on a scheduled basis was sufficient. So the misconception that batch is easier and cheaper is just outright wrong.

Jana: I’ve checked your tech stack. I saw technologies like Flink, Kafka, Spark, etc. Is demand for streaming growing among clients? And how has this changed with AI technologies?

Erik: I think the answer is yes. There are two main drivers here. First, there’s a real change in streaming demand because people realize they can only answer specific questions in real time. Companies need to make close-enough-time decisions. For instance, in energy, especially renewables, you want to plan the utilization of your assets like wind farms or solar panels depending on the weather. You need to map weather data into your decision-making to plan energy delivery to the grid. And you want to plan maintenance when there’s no sun or no wind, because it would be foolish to stop them from producing energy when conditions are good.

The other interesting driver is data stack modernization. A lot of companies invested heavily in lake house or data lake infrastructures. Now, with economic pressures, people are more conscious of costs and questioning whether it was a good investment. These platforms are very expensive, and they’re batch — they’re not delivering real-time insights. That’s when people start asking about ROI. We had one client in manufacturing, a global corporation, where data produced in the US was validated in Europe and post-processed in India. Singapore sometimes had to wait 72 hours for the right aggregated and validated data. That’s a killer. So if you’re paying top dollar for your data platform, you want those insights as quickly as possible, not wait for 12 to 36 hours.

And that brings us to AI. An AI model is kind of useless for decision-making if you’re just asking the model as it is, because they’re trained on datasets with an end date. If they’re not connected to anything, they’ll give you answers from months ago. You need it to tap into your enterprise data, and to give accurate answers, it needs to tap into real-time data streams. That’s why AI and real-time are intricately connected — they need to be deployed together, especially for use cases like observability and alerting.

Jana: As someone from the Apache Beam community, I’m curious — do your clients use Beam? If so, for what kind of projects?

Erik: Very rarely, actually. We use Google Cloud a lot, especially in the AI space. It’s a really good development and deployment environment with great tools, and it’s very easy to build applications there. BigQuery is quite popular, especially in the marketing space for analytics, and that’s where we occasionally see Beam, but it’s fairly rare.

Most of our clients are on Microsoft or AWS stacks. We’re cloud-agnostic and our infrastructure engineers specialize in all three clouds, but Google is fairly small in terms of market presence. In the UK and Germany, it’s pretty much the same story — most deployments are on Azure or AWS. We do have clients running on Google, but it’s mostly focused on marketing intelligence.

Jana: I watched your webinar with Ververica on wind energy. Do you work closely with them?

Erik: There’s an enormous opportunity for the renewable energy industry. Regardless of what the US administration says, renewables are here to stay because they’re the most obvious answer to our future energy requirements.

The challenge is that renewables pose significant problems for grid operators who need to modernize how they manage decentralized energy collection and distribution. Because the energy market requires quick reactions and flexibility, we’re seeing a clear movement toward streaming technologies. Operators across the board are adopting this approach to increase revenues and better utilize their assets.

The market is quite fragmented — you have smaller operators alongside big utilities. Some of our clients use the Ververica Platform, but most operators still use standard scalable systems, collect data, and try to make sense of it in Excel. It’s quite horrible, and it doesn’t really work. And we are here to change it.

Jana: And for readers who might be interested — are you hiring?

Erik: We’re a remote company hiring across Europe and the UK. We’re looking to fill infrastructure engineering and data engineering positions, especially in the first quarter of 2026. I also want to highlight our commitment to diversity and inclusion. Our goal is to be a company that has less than 50% of my type: white male dude with a beard (laughs). We’re getting there, and we welcome people from all kinds of backgrounds to apply.

Jana: Thank you so much. It was a pleasure talking to you.

Erik: Thank you very much for having me!

🌐 To learn more about Hivemind, visit their **website or check out their [LinkedIn](https://www.linkedin.com/company/hvemind-technologies/). You can also connect with Erik directly on [LinkedIn](https://www.linkedin.com/in/erikschmiegelow/)**.

📷 All photos are from the **Flink Forward Barcelona 2025** album.