Smart Meter Platforms with Real-time Analytics on Microsoft Azure Fabric

By Robert Manke and Ashraf I.

1. Introduction

In the rapidly evolving landscape of energy management, the integration of IoT technology and advanced analytics is revolutionizing the way we can monitor, control, and optimize our energy consumption. Smart meter platforms stand as a critical component of this transformation, providing unparalleled insights into energy consumption patterns and enabling more efficient grid management while empowering consumers to save energy.

Beyond that, consumption data can be transmitted in real-time, providing consumers with immediate access to their usage patterns. This instantaneous feedback mechanism empowers everyone to actively monitor their consumption habits, enabling them to identify unusual spikes or irregularities that might indicate potential equipment malfunctions or system defects. Furthermore, this transparent approach to consumption tracking serves as a powerful motivational tool, encouraging customers to adopt more energy-conscious behaviors and make informed decisions that ultimately lead to significant cost savings. The ability to visualize their actual usage data creates a heightened awareness of consumption patterns, naturally incentivizing more efficient resource utilization and promoting sustainable practices that benefit both the individual consumer’s budget and broader environmental conservation efforts.

This article delves into the architecture and benefits of a cutting-edge smart meter platform harnessing real-time data generation and Microsoft Azure Fabric for comprehensive analytics.

2. The Evolution of Energy Metering

Traditional energy meters have served as basic measuring devices, providing cumulative consumption readings with limited granularity. With mostly quarterly (e.g. Germany) manual readings as the norm, the information gathered using these devices has generally sufficed for billing purposes alone. However, as energy grids face increasing complexity due to renewable energy integration, distributed generation, and escalating demand, traditional methods have become insufficient.

Smart meters herald a paradigm shift in this domain. Equipped with sensors that record consumption data at short intervals, these digital devices transmit the collected information to central systems.

As Wang et al. (2025)¹ demonstrate, smart meter data can be utilized in numerous ways:

1. Customer identification

2. Load forecasting

3. Evaluation of household physical infrastructure

4. Residential behavior analysis

5. Utility operation optimization

In this article, we want to focus on a mixture of the topics mentioned above. Specifically, we want to address the following aspects in our initial setup:

• Load forecasting

• Residential behavior analysis

Nevertheless, we are aiming to enhance our smart-meter-generator to be able to implement other dimensions as well.

One key aspect we are going to emphasize is the real-time component of smart-meter data. Smart-meter data is gathered and submitted in real-time or near-real time, which creates a lot of complexity in terms of analytical processing and real-time anomaly detection.

Thus, we are introducing the concept of a Smart Meter Cloud Data Platform. The objective of this platform is to establish a scalable enterprise solution for analyzing smart meter data. It is designed to deliver real-time insights, enabling organizations to monitor incoming data streams for potential errors or anomalies as they occur.

A key differentiator of our platform is the integration of a real-time data generator with advanced analytics capabilities. This combination ensures that businesses can not only capture data continuously but also act on it immediately, supporting operational efficiency and proactive decision-making. In the following sections, we outline the technical design of the data generator and explain how it supports the overall platform strategy.

3. Data Generation — Continuously Generation of Meter Data

Like Ezhilarasi et al. (2023)², we considered how we could generate smart meter data. The Authors focused on creating smart-meter data for a certain time peroid using Prophet, a machine learning model dedicated to time series forecasting which has been developed and made open source by Facebook. In contrast to the authors mentioned above, the focus of our approach was to establish continuous data generation and thus place greater emphasis on the concept of real-time smart-meter-data generation, as it occurs in the real-world, combined with real-time analytics.

Our Smart Meter Platform is built upon a sophisticated multi-layered architecture that seamlessly integrates data generation, streaming, storage, and orchestration components to create a comprehensive utility data simulation environment.

3.1 Data Generation Layer

At the heart of the platform are data generators designed for multiple utility types, each engineered to produce highly realistic consumption patterns. The Electricity Consumption Generator forms the core component, simulating authentic electricity usage patterns through complex algorithms that account for time-of-day variations, seasonal fluctuations, and day-of-week factors. This generator applies consumption patterns that accurately reflect typical daily usage variations observed in real-world scenarios.

Complementing this is the Heating Usage Generator, which models heating consumption with sensitivity to seasonal changes. This component produces significantly higher usage during winter months while maintaining lower baseline consumption during summer periods, closely mimicking actual heating behavior patterns.

The platform also features dedicated Water Consumption Generators with separate modules for cold and warm water simulation. These generators create realistic usage patterns throughout the day, accounting for typical household water consumption behaviors and peak usage times. In this article we are going to focus on the electricity-data-generator³.

Each generator incorporates a comprehensive set of variables including base consumption values that establish realistic usage baselines, time-of-day pattern adjustments that reflect natural consumption rhythms, seasonal variation factors that account for climate-related usage changes, and distinct weekday versus weekend differences that capture lifestyle variations. Additionally, random variation components simulate real-world unpredictability, while geospatial data integration incorporating latitude and longitude coordinates enables location-specific consumption modeling. Finally, we enriched our data-generators with an acceleration factor. This factor enables us to speed up the simulation, so we don’t necessarily have to wait a day to get one day of simulated data..The default setting simulates a day in 30 min of real-time. Concluding, the formula behind our usage patterns can be understood as:

Where:

• C = total Consumption
• B = base_value
• P_h = pattern at hour
• D_w = pattern at weekday
• S_s = pattern at Season
• R = random factor

Whereas the time acceleration formula is defined as:

With:

• D_sim= simulated time
• D_0 = the real calender datetime
• a = time acceleration factor
• t_now= current seconds till unix epoch
• t0=real time when acceleration was last set

To make the data plausible for testing the analytical purposes of our smart-meter platform, we created a backend based on a Postgres database. Inside the database we created a table for Households. This table includes information on simulated households, including all devices mentioned above, in a quantity that corresponds to the number of rooms existing in those simulated households. As German and European Laws define the quantity of mandatory devices, in relation to the number of existing rooms⁴. So, in Short: this component of our platform is used as backend to store comprehensive metadata about simulated households, including unique identifiers (for households and devices) and precise geolocation information necessary for realistic simulation scenarios. This will give us the option to simulate usage patterns on sophisticated household level.

3.2 Data Streaming Infrastructure

The platform leverages Apache Kafka as its core streaming technology, providing robust and scalable data distribution capabilities. Kafka Producers serve as custom-built components that efficiently send generated data to specific Kafka topics, such as “electricity-meter” and “heating-meter”, ensuring an organized data flow and topic-based consumption patterns.

A comprehensive Error Handling system provides robust error detection and reporting capabilities, capturing and logging delivery failures while maintaining dedicated error topics for systematic issue tracking and resolution. The platform implements configurable Message Delivery Guarantees through adjustable acknowledgment levels and sophisticated retry mechanisms, ensuring reliable data delivery even under challenging network conditions.

Performance Optimization features include advanced message batching, compression algorithms, and intelligent buffering systems that maximize throughput while minimizing resource usage, ensuring efficient operation even under high-volume data generation scenarios.

3.3 Containerization and Orchestration

The platform embraces full containerization to enable easy deployment and dynamic scaling capabilities. Docker Containers encapsulate each component, including generators, database systems, and Kafka infrastructure, running in isolated environments with clearly defined dependencies and resource allocation.

Kubernetes Orchestration leverages Helm charts to provide declarative configuration management for deploying the entire platform to Kubernetes clusters, enabling automated scaling and management.

3.4 Technical Summary

The Smart Meter Platform delivers several significant technical advantages that distinguish it as a comprehensive simulation solution. Scalability represents a core strength, with the system designed to handle thousands of simulated households while maintaining efficient resource utilization and consistent performance.

One core underlying strength of the generator platform lies in its modular design. We can easily spin up multiple Jobs on Kubernetes and increase the size of real-time data to test multiple technologies for its capabilities of handling different data sizes and throughput. In Addition, the platform only needs a Kafka client as endpoint, to push the ongoing generated data.

The presented diagram shows how the platform is constructed. The left plane shows the internal database, which is used to store the data of the artificially generated households, the middle plane describes the data generators mentioned above and the core producer used to deliver the data to the data platform. Each unit of generator, reads the uuids from the database and generates usage/consumption data based on the variables mentioned above. The generated data is pushed to Kafka. Data generation and pushing are carried out continuously.

Fault Tolerance is achieved through comprehensive error handling and recovery mechanisms distributed throughout the system, ensuring continuous operation even under adverse conditions. Realistic Simulation capabilities stem from sophisticated algorithms that produce data closely mimicking real-world consumption patterns, enabling accurate testing and analysis scenarios.

The platform’s Configurability provides extensive options for customizing simulations to different scenarios and requirements, while Containerization through Docker and Kubernetes supports consistent deployment across diverse environments. Finally, Cloud Readiness features seamless integration with cloud services, providing extended capabilities and enhanced scalability for enterprise deployments.

4. A modern Smart-Meter-Data-Platform: Azure Fabric

4.1 Microsoft Fabric: A Brief Introduction

Microsoft Fabric is a unified data platform that brings together multiple services — including Data Engineering, Data Factory, Data Science, Real-Time Intelligence, Data Warehouse, and Databases — into a single SaaS environment. At its core lies OneLake Storage, a centralized system that seamlessly integrates lakehouses, warehouses, and other workloads, ensuring that data remains accessible and consistent across the organization.

The platform is AI-powered, embedding advanced intelligence directly into workflows to simplify analytics and reduce the need for complex integrations. With its end-to-end workflow, Fabric covers every stage of the data lifecycle: ingestion, preparation, storage, analysis, and visualization.

The true business value of Microsoft Fabric lies in its ability to eliminate data silos, enhance scalability, and accelerate insights. By streamlining processes and unifying data, organizations can make faster, smarter decisions that drive innovation and growth.

4.2 Microsoft Fabric Eventhouse

A Microsoft Fabric Eventhouse is a specialized environment built for highly scalable, time-based events and telemetry data processing. It is designed to handle diverse workloads such as time series, log data, IoT measurements, and smart meter information. Powered by proven Kusto technology (the same foundation as Azure Data Explorer), Eventhouse is optimized for streaming scenarios that demand speed, precision, and scale — such as comprehensive smart meter platforms.

Eventhouse provides a robust solution for storing and analyzing massive volumes of event-driven data. Its architecture is tailored for append-only data streams with precise timestamp handling, making it ideal for:

• Near real-time analytics

• Anomaly detection

• Continuous monitoring

• Telemetry and time-series analysis

Eventhouse is particularly effective in smart meter implementations, where it can:

• Ingest measurement data via Kafka topics (consumption readings, voltage levels, device status).

• Store both raw and pre-aggregated data efficiently.

• Enable anomaly detection to identify unusual consumption patterns or system failures.

• Perform load analysis across multiple time windows (5-minute, 15-minute, hourly, daily).

• Power near real-time dashboards for grid operations and consumption transparency.

• Transfer aggregated data to Lakehouse or Warehouse systems for long-term analytics and reporting.

5. Using Fabric as Smart-Meter-Platform

5.1 Initiation and Fabric integration

Let’s bring our Data-Generator into production and start producing and consuming smart-meter-data. For our demonstration we used the Azure Tech-Stack, mainly consisting of Azure Container Registry, Azure Kubernetes Services and of course Azure Fabric.

At the beginning we dockerized our code and pushed it to the Azure Container Registry. These container images are then consumed by Kubernetes, which initiates the corresponding Pods and Jobs. As illustrated in the diagram below, the electricity‑generator is deployed and executed as a Kubernetes Job.

This job generates electricity data for all previously generated artificial households and their corresponding connected devices.⁵ The diagram below illustrates the incoming messages from this job within Azure Event Hub. Under the current configuration, a single job achieves an average throughput of approximately 30,000 messages per minute.

The incoming data is processed within an Eventstream in Azure Fabric and gets subsequently stored in both an Eventhouse and a Lakehouse. This architecture ensures real‑time handling of event data while also enabling long‑term storage and advanced analytics on historic data. Fabric allows you to integrate this in an easy manner by creating a pipeline and setting transformations, which are then applied to the incoming data. In the given example, we are casting data into other datatypes.

Now let’s dive into our real-time analytics using Fabrics real-time Dashoard.

5.2 Real-Time-Dashboard

The real‑time dashboard provides continuous insights derived from the preceding eventstream. Updated every 10 seconds, it delivers a rapid and reliable overview of ongoing developments.

Through this dashboard, we gain visibility into:

• Incoming data volumes and overall system throughput

• Daily fluctuations in consumption patterns

• Regional anomalies highlighting unusual usage or demand spikes

• Forecasts for the upcoming hours, enabling proactive decision‑making

• Device‑level alerts that identify irregular or noteworthy behavior (anomaly detection)

This capability ensures that operational teams can monitor performance in real time, respond quickly to emerging trends, and maintain transparency across the entire energy ecosystem. Let’s get into detail for each component mentioned above.

The gif below demonstrates how incoming data can be displayed on Fabric’s real-time-dashboard.

The following Screenshots show a preview of our Dashboard, which we generated to check the capabilities of Fabric’s Eventhouse for examining Smart-Meter-Data.

Let’s dive into the different elements we set up, to see what we can implement using Fabric.

5.3 Data Volumes

By leveraging a chart of this kind, we gain real‑time visibility into incoming data volumes. This continuous monitoring not only provides transparency into system activity but also enables proactive adjustments based on expected data flows.

Such visualization is particularly valuable for:

• Detecting potential issues early, such as bottlenecks or irregular behavior.
• Supporting operational decisions by aligning system capacity with actual demand.
• Cost Efficiency by using auto scale features that automatically adjust their resource useage to the demand

In essence, the chart transforms raw data into actionable insights, ensuring that teams can maintain stability, optimize performance, and respond swiftly to emerging challenges.

5.4 Seasonality

A time‑series graph has been integrated into the dashboard to visualize average electricity consumption by time of day. This representation allows stakeholders to clearly observe daily usage patterns generated by the simulation model.

The data reflects typical household behavior:

• Morning hours Reduced consumption during early activity.

• Midday: Moderately lower usage as households are less active.

• Afternoon: Noticeable increase in demand as residents return home.

• Evening: Peak usage followed by a gradual decline as activity slows.

By presenting consumption trends in this structured way, the dashboard provides actionable insights into load distribution, demand forecasting, and operational planning. This enables energy providers to anticipate peak periods, optimize grid performance, and ensure reliable service delivery.

When viewed over a broader time horizon (image below), the consumption pattern remains stable throughout the week, reflecting consistent daily usage behaviors. However, the data also reveals a notable increase in electricity demand during weekends, driven by higher household activity and extended occupancy.

If we would keep the simulation running for a longer period, we would see how the data would differ between different seasons of the year.

This insight is valuable for:

• Operational planning: Anticipating weekend peaks to ensure grid stability.

• Forecasting: Adjusting predictive models to account for recurring weekly fluctuations.

• Customer transparency: Providing households with clear visibility into their consumption habits.

By highlighting both stability and weekend variability, the dashboard supports more accurate demand management and resource allocation.

5.5 Forecast

To extend our insights beyond real‑time monitoring, we leverage built‑in time series forecasting capabilities. This allows us to estimate electricity consumption for the upcoming hours — or any defined horizon — based on historical usage patterns.

The forecast provides a forward‑looking view of demand, enabling:

• Proactive planning for grid operations and resource allocation.

- Early detection of deviations between predicted and actual consumption.

• Alerting mechanisms when anomalies or unexpected spikes occur.

In practice, our forecast line (blue) shows a slight deviation from the regular consumption curve and was unable to anticipate a simulated spike. This intentional error injection validates the alert functionality, ensuring the system can flag irregularities and support rapid response.

By combining real‑time monitoring with predictive analytics, the dashboard evolves into a decision‑support tool that not only tracks current performance but also anticipates future demand and highlights potential risks.

The alert-functionality is defined as an action-item, which triggers different actions, e.g. sending E-Mails, Teams-Messages or runs other jobs. You can inspect the frequency of an alert fired in the corresponding alert dashboard⁶.

5.6 Geographical analysis

Smart‑meter data typically includes longitude and latitude information, which can be leveraged within Microsoft Fabric to create geospatial visualizations. By plotting this data geographically, we gain immediate visibility into where specific consumption patterns or anomalies occur.

For example, the dashboard highlights increasing electricity usage in the Berlin/Potsdam region. Such deviations may indicate an unexpected event, operational issue, or unforeseen demand surge.

The ability to perform real‑time geographical analysis is highly valuable, as it provides:

• Rapid insight into affected regions

• Contextual understanding of anomalies within their geographic setting

• Actionable intelligence for targeted investigation and response

This capability transforms raw location data into strategic situational awareness, enabling faster decision‑making and more effective resource allocation.

6. Conclusion

We need to draw our conclusion in two ways. First, we must take a closer look at our data generator. We have noticed that it can be enhanced by introducing additional “noise”. By this, we mean incorporating more distortions and variability, which would increase the degree of realism in the generated data (e.g. we could implement simulating big events like world cups where the water useage is during the breaks could be as much as three times the normal amount). Also, we aim to add the other meters we developed, water and heating meter, into our framework.

Furthermore, we identified several opportunities to expand our dataset:

· Technical degradation of devices: One option is to simulate random technical decay. For example, certain devices could begin transmitting implausible consumption values, or we could enrich device data with parameters such as voltage levels, contamination indicators, etc. This would allow us to evaluate device conditions more effectively and broaden our analytical capabilities, including the calculation of predictive maintenance scenarios.

· Household data enrichment: Another option is to extend the underlying household data by adding further determining variables, such as the number of household members or the household size in square meters.

· Industrial data integration: It would also be possible to expand the generator with data from industrial enterprises. This, however, requires significantly more research into consumption values and a focus on specific industries to ensure the generation of plausible measurements.

From a technical perspective, we observed that our generator can produce very large volumes of data. In our tests, we achieved approximately 100,000 data points per minute (per meter, using a pod). The limiting factor in this process proved to be the Event Hub.

In addition to the generator, we explored Fabric as a platform for real-time analysis of smart meter data. The Smart‑Meter platform described here demonstrates how IoT‑based measurement devices, combined with Microsoft Azure Fabric, enable a new level of transparency and responsiveness in modern energy supply.

By tightly integrating:

• Real‑time data generation (Kafka producer jobs, time simulation, geographic adaptation)

• Robust streaming infrastructure (Eventhouse/Kusto, schema‑on‑write, high throughput rates)

• Persistent storage (OneLake/Lakehouse)

• Advanced analytics tools (KQL, Power BI)

The Fabric Eventhouse has proven highly effective, enabling us to analyze data both geographically and in terms of potential anomalies. Moreover, the use of the Kusto Query Language also allows for straightforward implementation of forecasting methods. The platform creates a comprehensive ecosystem that delivers immediate, precise insights into consumption patterns, anomalies, and optimization opportunities.

For future articles, we plan to test additional tools and develop a comparative framework to systematically evaluate and benchmark different solutions. For example, testing our data-generator regarding the shift-left premise and for example comparing Fabric with Apache Flink (confluent cloud) or databricks, could lead to some meaningful insights.

Want to learn more?

Feel free to visit our website.

About us

msg’s Data & Analytics unit empowers organizations to become data-driven leaders through comprehensive, technology-agnostic support across the entire data lifecycle. From crafting tailored data strategies and governance frameworks to implementing cutting-edge use cases with advanced analytics and AI, we focus on delivering tangible business value. Our expertise includes designing and building scalable cloud data platforms on AWS, Azure, GCP, and Kubernetes, utilizing leading technologies like Databricks, MS Fabric, Confluent, etc. ensuring our clients gain a decisive competitive edge through optimized processes, data-informed decisions, and innovative, data-powered business models.

[1] Wang, Yishi, Andrew S. Jin, und Kelly T. Sanders. „A systematic review of literature utilizing residential smart meter data“. Renewable and Sustainable Energy Reviews 225 (Januar 2026): 116130. https://doi.org/10.1016/j.rser.2025.116130.

[2] P., Ezhilarasi, Ramesh L., Xiufeng Liu, und Jens Bo Holm-Nielsen. „Smart Meter Synthetic Data Generator development in python using FBProphet“. Software Impacts 15 (März 2023): 100468. https://doi.org/10.1016/j.simpa.2023.100468.

[3] The additional Meters will be introduced in the following Articles.

[4] The are some exceptions defined, for example for small apartments.

[5] The other two running Pods visible on this screenshot, are the database-generator, the job which creates the artificial data and the corresponding cloud-database-generator, which uploads and the artificially created data to a cloud database. In our case the data was uploaded to a postgres hosted in Azure.

[6] Alerting and all corresponding options will be discussed in further articles.