Recently, the Keepler team for Madrileña Red de Gas deployed a data platform on AWS for collecting data from gas smart meters, using, for the first time worldwide, the native integration of AWS IoT Core with public LoRaWAN networks in collaboration with the operator Everynet.
Our client, Madrileña Red de Gas, is a leading company in the distribution of natural gas. According to the new government policies aligned with the 2030 agenda, gas distributors will need to implement remote telemetry reading in their meters; this will require Madrileña Red de Gas to install approximately 900,000 additional devices.
Contents
The Challenge: Designing the Smart Metering of the Future
The goal of the project was to deploy an IoT architecture on the AWS public cloud, agnostic to different device manufacturers, centralizing the data from smart meters from multiple providers on a single scalable and dynamic data platform.
This innovative solution developed by Keepler on the AWS public cloud can receive, ingest, and process data sent by gas smart meters using LoRaWAN technology over public networks. It then analyzes and provides useful information to business and technical teams through AWS’s native data analytics and Business Intelligence tools.
The architecture has been designed to be modular and scalable, which allows and facilitates the incorporation of new Smart Meters in an agile and dynamic way, adapting only the necessary components to process, store and decode the specific information sent by each device.
A pioneering LoRaWAN project
AWS announced in 2023 the support for public LoRaWAN networks natively through the operator Everynet, making this three-way collaboration project between Keepler, Madrileña Red de Gas and AWS a pioneer in using this functionality in a real environment.
Using LoRaWAN through public networks directly integrated into AWS services allows the client to achieve significant savings in the maintenance of the LoRaWAN network infrastructure. This is because it eliminates the need to deploy, maintain, and manage private gateways, utilizing only those already installed by the current network provider (public gateways). Additionally, it enables cost unification in a single centralized panel through AWS’s own billing service.
The technical solution
The solution combines and integrates both worlds: the physical world of devices, or smart meters, with the public cloud world through the integration of gas smart meters with AWS IoT Core.
AWS IoT Core for LoRAWAN acts as the LNS (LoRAWAN Network Server), receiving and decrypting messages from various deployed devices. Once messages are received from the smart meters, they are routed through rules configured within the AWS IoT Core service, defining the routing actions associated with each message, for example sending the data for storage and/or processing to other AWS services such as S3 and Kinesis Firehose.
Each rule has been configured with two fundamental actions: the first is responsible for sending the data to Kinesis Firehose for real-time processing, and the second is responsible for storing the raw data directly in S3 for long-term storage, which enables options such as audits, security analysis, possible reprocessing, advanced analytics, anomaly detection, etc.
Multi-branch architecture with support for multiple physical device manufacturers (Smart Meters) from data ingestion to consumption by business units.
The solution is designed to process data in real-time or near real-time, making the data from each smart meter available for review and analysis with minimal latency between sending, receiving, and processing. This near real-time processing (nRT) is done using the native integration between Kinesis Firehose and AWS Lambda. When a message reaches the platform through the LNS (AWS IoT Core), it is routed via an IoT Core rule to Kinesis Firehose, which is configured to pre-process the data using a Lambda function that decodes the payload (binary or textual) and returns the data in JSON format back to Kinesis Firehose. Firehose then dumps the data to S3 in an optimal columnar format for analysis and processing, such as Parquet. Once the data is stored in S3, AWS Glue allows us to define the data model that will govern the stored data in different layers or buckets.
With the complete dataset stored in S3 and updated in nRT, along with the defined data model and the data catalog stored in AWS Glue, our standard base DataLake architecture is defined.
Exploiting the information of a DataLake in AWS is a task that is carried out in a very efficient way in terms of performance and cost using the AWS Athena service. This service will allow us to launch SQL queries on the different layers of our DataLake that contain the data from the different smart meters, and, in addition, it will serve to define the data sets that will be consumed later by each of the dashboards designed and defined in QuickSight (the native BI tool of AWS).
The great versatility and power that come from defining a Data Lake architecture for IoT solutions is that we can centralize various data sources on a single platform, allowing us to cross-reference, explore, and analyze diverse and heterogeneous data sets, generating new information that provides both technical and business value. For this gas smart meter solution, we incorporated an open data source, such as weather information from AEMET. Thus, in the QuickSight Dashboards, we can show information provided by the devices (such as accumulated gas consumption, consumption forecast for the coming dates, and device battery status), including climatic data from AEMET, enriching the forecast data according to temperatures.
The data platform is designed to be multi-manufacturer, allowing Madrileña Red de Gas to have different smart meter providers deployed under the same network, unifying all the required device variety into a single data platform. Since each manufacturer can use its own encoding and encryption mechanism for their device messages, IoT Core configures a dedicated rule for each manufacturer, invoking a specifically designed Lambda function to decode the corresponding payload. It is in IoT Core where the relationships between the rules, message topics, and associated smart meters are registered and established.
Technical Challenges faced during the Project
Security
To ensure the end-to-end security of the data and the integrity of the information, several encryption mechanisms have been implemented, both at rest and in transit, ensuring that the data remains secure throughout all phases of the life cycle.
For IoT devices connected via the LoRaWAN network, data is encrypted using private keys generated on the fly thanks to the OTAA configuration and the specifications of the communication protocol itself. AWS IoT Core acts as the LNS, decrypting the messages from each smart meter.
To maintain data security at rest, encryption has been enabled in the different architectural elements using KMS with AWS-managed keys.
Multiple Manufacturers
Initially, the architecture was designed to employ smart meters from a single manufacturer, dealing with a unique payload and defining the data schema throughout the architecture based on this payload’s fields.
However, the need to incorporate smart meters from new manufacturers was identified along the project, requiring the evolution of the solution to a multi-branch architecture capable of decoding various types of payloads. This required evolving the whole architecture diagram, data schemas, and dynamically creating the different components involved in receiving and decoding each message.
The architecture was adapted with an IoT Core rule for each manufacturer, used as the destination when registering smart meters. This creates a scalable branch-like architecture to support multiple manufacturers supporting multiple encoding and encryption protocols.
Infrastructure as Code (IaC)
The entire presented architecture is deployed following the best practices defined for IaC. Terraform is used to write all the necessary infrastructure code. The biggest challenge was preparing dynamic Terraform scripts to dynamically generate infrastructure components based on the number of smart meter providers at any given time and facilitating the agile and dynamic addition or removal of a particular provider when the infrastructure is deployed.
Data Enrichment
The solution not only aimed to show a forecast based on each client’s gas consumption and technical data from sensors or AWS services, but also opted to add data from an external source like AEMET.
In this way, a data extraction process was designed to obtain weather information and cross-reference it with consumption data to offer a more precise forecast, detecting consumption patterns based on weather conditions and predicting possible failures and anomalies based on climatic conditions.
Dashboards with Key KPIs
The dashboard aims to answer two main requirements: the status of each device and gas consumption. Additionally, it is enriched with open data from AEMET to add value, facilitating analysis and decision-making for business teams.
Three key visualizations stand out: accumulated gas consumption and forecast, last device upload, and temperature forecast (AEMET).
Accumulated Gas Consumption and Forecast
This visualization informs the daily accumulated consumption of all devices and shows future metrics based on machine learning using QuickSight’s integrated machine learning forecasting capabilities. It has been configured to show the consumption forecast for the next three days.
Last Device Upload
This visualization captures the status of each device, showing the last upload from each device, including: device ID, upload time, battery level, battery voltage, magnetic interference, valve status, time synchronization, and gas volume measured (m3) by each device.
Additionally, a series of color codes and icons have been added to show indications depending on the value of the different fields:
- A red cross in the Device field indicates more than 4 hours since the last signal upload (configurable detection window).
- A “normal” value in the Battery Level field shows a green thumbs-up. Values below “normal” are shown with a red circle.
- A “normal” value in the Magnetic field shows a green thumbs-up.
- A value other than “sync” in the Time sync field is shown in red font color.
Temperature Forecast (AEMET)
This visualization shows the maximum and minimum temperatures in Pozuelo de Alarcón for the next three days (configurable for any city/community in Spain), along with the sky condition, snow level, and precipitation probability. These data are obtained from AEMET’s open data catalog. In addition to maximum and minimum temperatures, we have relative humidity, precipitation, wind chill, snow, and a description of sky conditions.
This third-party open data helps complement the sensor information and helps detect and categorize possible anomalies in gas production and consumption caused by adverse weather.
Business Impact
With this development, Madrileña Red de Gas has a centralized data platform that is agnostic of any smart meter manufacturer, enabling it to receive and process device data regardless of origin, verifying that it is feasible to have reliable ingestion and real-time processing solutions for devices connected to low-power networks. This enables working with LoRaWan and NB-IoT networks in the future, facilitating the adoption of both types of networks for the client.
ℹ An NB-IoT network is part of the LPWA (Low Power Wide Area) network type, designed specifically for the communication of IoT devices with specific information sending capabilities in volume and time, operating in hard-to-reach areas with less stable connectivity.
Lessons Learned
- The solution is designed as a multi-branch architecture, allowing the addition of new manufacturers with different data types without modifying previous manufacturers, ensuring backward compatibility and scalability.
- It is not possible to automate the solution’s deployment 100% as ad-hoc development is required for each new manufacturer due to adapting the different schemas and encoding and encryption mechanisms according to each manufacturer’s specifications.
- IoT solutions develop a close relationship with device firmware; if they are updated, the message decryption scripts may need modification, so ongoing support is essential.
- Maintaining fluid communication with manufacturers is crucial to provide updated specifications, as we entirely depend on their documentation to decode each message (especially for devices sending messages in binary format).
IoT Platform Evolution
- Applying AI/ML to fill data gaps and generate artificial data based on variables like:
- Neighbors of the affected device
- Historical data of the affected device
- Incorporating advanced AI/ML processes:
- Pattern and/or anomaly detection
- Consumption prediction based on demand history
- Implementing a CICD pipeline to deploy architecture components or register new IoT devices automatically
- Remote supply cut on demand (for devices supporting this functionality)
- Adapting computing services to be more cost-efficient in high data processing demand scenarios and number of devices.
- Incorporating new external data to enrich KPIs and Dashboards: population density data, seismological data...
This project positions Madrileña Red de Gas as an innovative and highly competitive company in its sector, developing a pioneering project worldwide using IoT technologies with public LoRaWAN networks. Moreover, it guarantees regulatory compliance required by 2030, well in advance.
The importance of building future-proof data environments that can scale with business growth is highlighted, as well as the business value that the analytics layer brings to the vast IoT data available to energy, manufacturing, and industrial sectors.