Data Pipelines Modernization

Data Platform

Data Strategy

Automating technological evolution.

Industries:

Retail & FMCG

Solutions:

Data Platform - Data Strategy

Overview

Many organizations today face significant challenges due to the use and adaptation of obsolete software, which often hinder agility and pose risks. The software lifecycle often sees the decline of initially promising and widely adopted solutions through a gradual reduction in market presence and support, superseded by other solutions, or the consolidation and reduction of advanced skills.

Despite the obvious disadvantages of maintaining outdated solutions, there is often a tendency to procrastinate or postpone the transition to new technology due to the considerable expenses involved and the typically high impacts on all involved systems. However, it is overlooked that the accumulated technological debt over time proliferates management and evolutionary costs. Additionally, it may happen suddenly (e.g., due to regulatory issues or end of life of software, etc.) that part of one’s platform still needs to be modernized, leading to much tighter timelines and higher risks.

Challenges

The issues and limitations caused by maintaining obsolete software are manifold:

Adopted software may restrict the ability to implement updates due to technical and/or functional limitations (e.g., inability to interface with modern technologies, inability to handle desired scalability/configurability of new workflows, etc.).
Software may reach end-of-maintenance, leading to bugs/issues that the vendor is no longer obligated to fix, significantly increasing maintenance costs and risks for the company.
Knowledge associated with workflows developed on obsolete software often dissipates over time, making interventions complex when modifications or updates are needed.

However, transitioning to a new tool also presents challenges that must be considered and addressed:

Identifying the new replacement is not straightforward, considering various factors such as:

• Features and capabilities offered
• Reliability of the producing company
• Costs, including licensing and necessary infrastructure
• Required skills

Switching to the new tool often requires a significant financial investment and a lengthy commitment, as automating the migration of existing workflows is rarely feasible, necessitating manual operation.

Solutions

When evaluating the transition to a new platform, it could be advantageous to consider adopting tools that facilitate migration to the identified new product. Automation or semi-automation of migration from older software can indeed offer numerous benefits, including a significant reduction in time and consequently costs, as well as increased precision and reliability of the resulting processes.

In our experience, we have utilized a migration tool provided by Semarchy to migrate hundreds of workflows previously created by the client using the Sunopsis product. Semarchy is a data management suite that, in addition to various capabilities, offers the possibility of semi-automatically translating flows from other software, including Sunopsis and Oracle ODI, or translating SQL/PLSQL scripts into flows.

It should be emphasized that the process is considered semi-automatic, as there are various scenarios that may require manual intervention:

Development of brand-new components

Specific components may have been developed on the original software to manage particular behaviors without an equivalent on Semarchy.

Changes to the behavior of the original software

The behavior of the original software might have been altered to meet specific requirements, which could make the mapping to Semarchy less accurate.

Differences in functionality between the two tools

It’s possible that a feature used in the original software doesn’t have a direct match or is implemented differently on Semarchy, thus hindering automatic mapping by the tool.

In these situations, manual intervention can bridge the necessary gap to achieve a correct migration:

Through a thorough analysis process to identify the best configuration parameters.
Leveraging direct communication with Semarchy developers, who are available to implement new features and provide support for tool customizations where needed.

It’s important to emphasize that the effort required to configure and potentially customize the tool to achieve correct migrations remains significantly lower than conducting entirely manual migrations without the assistance of such tools.

The complete route

The path towards modernizing the data platform consists of two phases:

Securement

All obsolete elements need to be secured. It’s advisable to perform this operation as efficiently as possible, ideally leveraging tools that support migration, at least semi-automatically.

Technological Debt Reduction

Once the securement phase is completed, focus can shift to reducing technological debt and streamlining the current state. This may involve integrating best practices, enhancing alignment with data governance, and optimizing processes.

Full Security phases

The security phase, in the case of migration to Semarchy xDI, consists of 3 steps:

1. Extraction

Data is extracted from source systems and stored in an internal structure of the migration tool.

2. Migration

After selecting and inputting the most suitable configuration parameters and converting the previously extracted data into Semarchy format, the new flows are generated.

3. Validation

The new flows are tested, ideally by conducting parallel checks with existing flows, before being replaced in production.

Technological Debt Reduction

There are various viable approaches to reduce technological debt, among which it is crucial to consider defining clear and robust guidelines, which may include:

Naming Convention

The definition and application of a naming convention are crucial for organizing and identifying projects, metadata, and deliverables clearly and effectively.

Log Management

Utilizing a standard for log management, which allows centralizing information on flow execution in one place, is important. This facilitates monitoring and issue resolution, enhancing visibility and traceability of development and execution activities.

Standardization for Data Flow Design

Fundamental for aligning deliverables as much as possible, simplifying developers’ work, and easing product consultation and maintenance.

Metadata Management

Defining standards and procedures for metadata management enables organizing and cataloging information on data, processes, and resources coherently. This promotes better understanding and traceability of involved elements, reducing the risk of duplication, errors, and outdated information.

Additionally, considering adopting a data governance tool can enable the implementation of automated policies and ensure compliance, thus enhancing process efficiency and security.

It’s important to note that these improvements should not only be integrated into all new developments but also retroactively applied to migrated data from previous tools. This ensures consistent alignment with best practices and contributes to overall technological debt reduction.

Benefits

Reduction of Migration Time and Costs

Increased precision and reliability of resulting processes

Reduction of operational risks