The Journey of Data: Building Data Pipelines for Smart Cities

Overview

“Data is the new oil. It’s valuable, but if unrefined, it cannot really be used” — Clive Humby

Have you ever pondered the intricate journey of data, originating from thousands of IoT devices, as it navigates through complex networks before appearing on the screens we interact with daily? In my past year in, I have not only pondered, but have dug deep into the heart of this question. While my exploration continues, I have paused to share some fascinating insights that shed light on the journey of data.

What are data pipelines?

Data pipelines are systems designed to deliver data from point A to point B. Let’s use road networks as an analogy. Imagine data as vehicles, travelling on roads within a city. Vehicles have a starting point and a destination. They need to know which route to take, when to enter the highway, when to exit the highway, which lanes are eligible, when to stop in an intersection etc. Well-designed road networks will lead to smooth traffic flow and vehicles will be reaching their destinations promptly.

Similar to vehicles, data needs to be routed and managed. Data also comes in different shapes and sizes. Some are large, some are small, and many have different formats. A well-designed data pipeline should handle all this data as well as its traffic in a robust manner. In essence, data pipelines are responsible for transporting data from data sources such as IoT devices to frontend systems in an efficient and timely manner.

But…how do we build one?

FIWARE — Standardized Data Models

Before we can start building data pipelines, we need to ponder about the data and ask ourselves, can the data be standardized? Below are some crucial pointers as to why standardization is important:

FIWARE offers data modelling standards in the form of NGSI-LD (Next Generation Service Interface — Linked Data). Smart Data Models is a library of NGSI-LD data models that offers models for various entities. This library, amongst many other ontologies/vocabularies, were used to create a standard for our data. Below is an example of a ‘Building’ data model:

A non-exhaustive list of well-defined attributes of a ‘Building’ model from smartdatamodels.org

FIWARE also offers open-source software components that can be used to build NGSI-LD compliant data pipelines.

We have found some tools to standardize our data. What’s next?

Building on AWS

Next is our hosting infrastructure. We employ a hybrid (on-prem and on-cloud) strategy, but I will focus on the cloud deployment. The journey of our data starts from sensors, our first layer. The data is then transported through a total of 9 layers. Below is a visual representation of the 9 layers:

Data Pipeline Layers

The 9 layers of the data pipelines

The system contains 9 layers, and each layer plays a crucial role:

1. Sensors — Data source

2. Edge — Edge device that connects the data to the Cloud and publishes the incoming data to the IoT client

3. IoT Client — IoT client service that listens for published data and pushes it into appropriate message queues

4. Message Queues — Queue system that buffers incoming data into appropriate message queues

5. Data Exchange Service — Microservice that pulls data from the message queues and sends it to the data storage interface services

6. Database Service & Timeseries Service — Data storage interface services that ingests the data into the data storages

7. Entity Database & Timeseries Database — Data storage services that stores the data in appropriate formats

8. Data Fetching Service — Microservice that fetches the data from the data storage, processes it, and delivers it to the frontend apps

9. Frontend Applications — Receives the requested data and displays it in a readable, user-friendly manner

The data pipelines

The data pipelines support both bi-directional data flow:

1. Northbound: Data flow from sensor layer to frontend layer

2. Southbound: Data flow from frontend layer to sensor layer

Currently, 4 data pipelines are developed and implemented:

1. Data Points Pipeline: This is the most used pipeline in which sensor data is sent (northbound)
2. Command Pipeline: This pipeline is used to send commands (southbound) to actuators or digital switches and ACK messages are sent back in response (northbound)
3. Logging Pipeline: This pipeline is used to send system or program logs (northbound)
4. Device Ops (Operations) Pipeline: This pipeline is used to manage the sensors and other device information via the Edge (northbound and southbound)

Our data pipeline system is complete!

Integrating FIWARE onto AWS

Deploying and integrating FIWARE technologies onto the Cloud infrastructure posed a technical challenge. FIWARE technologies were not particularly designed to be built with Cloud services. One of the major issues was the availability of compatible cloud native services.

To note an example, one of the FIWARE technologies we experimented with was a TSDB, a timeseries database. The particular TSDB was not readily available as a native service in the Cloud. The available off-the-shelf options to deploy the TSDB did not meet our regulations/compliance. Hence, we had to research deeper on how we can manually deploy the TSDB and maintain the deployment.

For those who are familiar with AWS, we trialed with both ECS and EKS, with both Fargate and EC2 variants. We went with EKS EC2 in the end due to its configurability.

Nonetheless, it was an exciting challenge to take up and explore the various intricacies of deploying open-source technologies on the cloud.

The Framework of Open-Source technologies

Open-Source technologies such as FIWARE come with their own framework as they are designed in a manner that the users are subjected to. This can be a boon or a bane.

The Boon

The framework is well-defined and as users of this open source, we have a guide to follow. This also means that some features are readily available out-of-the-box. One example is the TSDB offered by FIWARE, auto generates schemas of the tables in which the data is ingested into. This feature is useful in abstracting the complexity of schema creation for each incoming data point.

The Bane

As users of a well-defined framework, we are not able to escape the methodology/standards of the technology. Let’s use the same feature, auto generated schemas, as an example. This feature is not configurable and hence we lack control over the schemas. If the generated schema poses incompatibility issues with other services, then we have to find workarounds. We end up going against the system, leading to deeper and complex challenges.

Finding the Balance

Regardless, this is the nature of open-source technologies. We have to find the balance of utilizing their out-of-the-box solutions while mitigating their inflexibilities.

Conclusion

After journeying through the phases of research, design, implementation and deployment, witnessing the data pipeline in action feels like magic. We have deployed this data pipeline in a few facilities with close to a few hundred IoT devices, with plans to scale up this system to even more facilities.

As we stand on the brink of a data-driven future, the ability to harness and streamline vast amounts of data will define the success of our smart cities. By building robust data pipelines, we not only transform raw data into actionable insights for better city planning and operations, but also pave the way for a smarter, more connected urban landscape.

The journey of data, much like the lifeblood of our cities, is what will drive innovation, efficiency, and sustainability for generations to come.