A2A - Data Replication

Energy & Utility

Quantyca supports A2A in real-time offloading of data from SAP, Salesforce and other systems to GCP via Qlik Replicate.

Solutions:

Data Platform - Data Infrastructure

Technologies:

Qlik - Google Cloud Platform - Python

A2A Data Replication Energia generata da pale eoliche al tramonto

Business Summary

Established in 2008 from the fusion of a number of local utilities in Lombardy, A2A Energia is now present throughout Italy with electricity and gas supply services, public lighting, district heating and energy efficiency solutions. It supplies energy to over 2.5 million customers and is first in the sector in terms of satisfaction levels.

By 2030 A2A aims to double its customer base while maintaining high service levels, which is why it has embarked on a programme of progressive replatforming and migration to the cloud in recent years.

Quantyca is working with A2A in offloading old and new systems (SAP IS-U, SAP S/4HANA, Salesforce, etc.) to the new Cloud Data Platform (developed in Google Cloud Platform). The tool identified is Qlik Replicate and allows reliable, highly efficient and fully governed real-time, near-rea-time or batch replication.

Challenges

Before starting the replatforming project, A2A’s application map consisted mainly of on-premises solutions. These met existing needs, but considering the business objectives, one of which was to double the customer base in less than ten years, the current solution was no longer sufficient. A path of evolution of the application map was thus undertaken, aimed at exploiting the enormous benefits offered by cloud platforms.

A2A chose a Cloud Data Platform based on GCP, and it was clear that the data ingestion system from the operational systems to the new Data Platform also had to be redesigned to maximise the benefits. That is why it trusted Quantyca, with over a decade of experience in data management, and Qlik, through its Qlik Data Integration suite.

The key pillars of our solution proposal were:

Scalability

The solution had to take into account not only current data volumes but also future increases, closely linked to the planned business expansion.

Efficiency

With tens of TB of data and millions of CRUD events per day, the solution would have to be highly efficient to enable high responsiveness and low latency.

Update frequency

The previous updating of the data platform every 24 hours would have limited the possible use cases, it was essential to move to real-time or near real-time ingestion.

Reliability

It had to be ensured that the data transferred to the new data platform matched 100% with the content of the source systems because only then could decisions based on the content of the new platform be made with conviction.

Governance

All ingestion flows were to be easily governed, always having a clear view of the situation and being able to monitor the correct functioning of the replication in real time, being alerted promptly in case of anomalies.

Solution

A2A’s requirements were certainly challenging but with our experience in the field of data management we were able to propose a highly effective and efficient architecture capable of supporting the planned evolution.

In fact, the technological choice fell on the Qlik Data Integration suite and in particular on Qlik Replicate, capable of covering the entire life cycle of the data, guaranteeing low latency, high quality and extremely precise governance.

For some specific requirements, we also developed a customised batch data extraction system. We were able to maintain very high standards of efficiency by exploiting parallel data transfer mechanisms to the destination.

Architecture

With Qlik Replicate, it was possible to retain an extremely simple architecture, connecting directly to the source systems and equally directly to the final destination, Google BigQuery.

For only one use case, due to specific requirements for extraction from the Salesforce system, we added a custom application developed by us in Python for batch data extraction.

Integration

We have integrated several systems including:

Salesforce
With more than 100 replicated objects in real-time and the possibility of batch scheduling for particular objects
SAP IS-U
With almost 200 tables weighing almost 30TB replicated in real-time
SAP S4/HANA
With more than 250 objects extracted in near real-time or batch
Several RDBMS (Oracle Database, Microsoft SQL Server)
Almost 500 tables, belonging to different operational systems, replicated in real-time

Infrastructure

Considering the variety of source systems, and in particular their distribution across different on-premises and cloud networks, we created a multi-node infrastructure that would allow the concept of Data Locality to be exploited.

The Replicate nodes were in fact appropriately positioned to maximise the efficiency of data extraction and transfer, while maintaining centralised governance through the use of a Qlik Enterprise Manager instance capable of managing them all.

Results

With the support of Quantyca and Qlik, A2A was able to put into production several offloading projects from legacy and non-legacy operational systems.

Dozens of TBs of data are already being replicated, with millions of CRUD events managed every day in real-time or near real-time with a maximum latency of 5 minutes.

The solution implemented is allowing A2A to focus on new use cases and new integrations to support them, with low maintenance effort and high elasticity and responsiveness in the presence of evolutions. Data, previously scattered and structured in silos, are now centralised in a single Cloud Data Platform and therefore more accessible and available to multiple consumers, whether analytical, operational or other.