Data & AI Team Topologies

Data Infrastructure

Data Platform

Data Strategy

Data Products

Data Governance

Design team architecture to optimize data & AI product delivery

Industries:

Finance & Insurance - Retail & FMCG - Energy & Utility - Transportation - Life Science - Industrial

Solutions:

AI - Data Infrastructure - Data Platform - Data Strategy - Data Products - Data Governance

Background

In today’s highly dynamic and competitive market environment, it is critical for companies to optimize value generation streams based on data & AI products to keep up with the speed of change required, seize opportunities and continuously meet customer expectations.

The increasingly central and strategic role that data & AI are taking on requires organizations to be able to:
• manage the cognitive load of the data & AI landscape and maintain domain knowledge over time, to be efficient in the evolution of products and services
• distribute solution development responsibilities to different working groups ensuring common governance and quality standards, to increase production capacity
• reduce delivery times, to anticipate competitors and seize market opportunities
• introduce technological and methodological innovations in a sustainable way, to maintain a satisfactory user experience

As explained in the book “Managing Data as a Product”, an organization is a complex socio-technical system, made up of people and technologies. Social architecture (also called organizational architecture) defines how people are organized and interact to generate the expected outcomes. Technical architecture defines how technologies are composed and integrated to produce value. There is a strong correlation between people and technologies (and similarly between their respective architectures). Interactions between people and technologies take place within processes.

Any organization can be viewed as a complex sociotechnical system

At the operational level, people are typically organized into teams, each of which can implement one or more business capabilities. In the book “Team Topologies” [2], Matthew Skelton and Manuel Pais state: “Relying on individuals to comprehend and effectively deal with the volume and nature of information required to build and evolve modern software is not sustainable… We must, therefore, start with the team for effective software delivery”. Therefore, the team must be considered to all intents and purposes as the fundamental element of the social architecture that enables the generation of value.

The social component is the one that most influences the entire enterprise architecture, as described by Conway’s law: “Organizations which design systems… are constrained to produce designs which are copies of the communication structures of these organizations”. Therefore, in each organization, the way in which the responsibilities and perimeters of the teams are divided and in which they interact within the processes inevitably affects the choices that can be made in the design of data & AI architectures, influencing the effectiveness and efficiency of generating the business outcome, as well as the ability or not to meet stakeholder expectations.

Quantyca has seen on the field, working with customers of different sizes, corporate cultures and industries, that organizational models that are not suitable for the context often cause obstacles to the speed of delivery and scalability of data platforms. The problems that are often observed are:

Monolithic designs

In contexts where data platforms are developed and maintained by central teams, the design of the data model and architectural solutions tends to be monolithic, with no logical and physical boundaries between the different business subdomains. As a result, as volumes increase, the cognitive load increases as well and the model and platform become overly complex, with a high number of internal interconnections between the constituent components hindering their extensibility and maintainability.

Lack of oversight of the data lifecycle

The division of responsibilities by technical functions of data management and stage of data pipelines traditionally adopted (e.g.: offloading teams, teams of data modelers, teams of DWH engineers, BI teams, operations teams) prevents teams from having full awareness of the end-to-end lifecycle of the data they process, the operational processes that generate them, the users who consume them and the purposes of use, the data quality problems inherent in the data, their domain semantics. As a result, the overall service offered to the company by the data management function is not very effective and far from what the business expects, because local optimization is favored over global optimization.

Bottlenecks

Organizational models that involve excessive dependencies on specialized teams (DBAs, AMS, Infrastructures) make processes inefficient as these teams struggle to respond to the competing and increasingly pressing demand from multiple stakeholders (data engineers, AI engineers, business users, analysts, data scientists, external data providers). As a result, the lead time for implementing new data & AI products or small improvements on existing products is far higher than expected, and the company is likely to be constantly behind in releasing products and services to the market.

Interoperability and Quality Issues

The lack of common policies, methodologies and standards for the development and deployment of data & AI products represents a significant source of risk for the contexts in which different actors, whether they are internal development teams or external consultants, operate in the ecosystem, as each of them adopts its own practices. As a result, the products produced are poorly interoperable and the level of quality is not uniform between them, causing a major obstacle to the reuse of data assets and AI functionalities.

Tech debt

The lack of teams that are focused on investigating, evaluating, experimenting and disseminating the new practices offered by the global technology landscape prevents the company from evolving technologies and methodologies in a sustainable way, causing technical debt that further affects the cost of maintaining data & AI solutions.

The structure of work teams and the ways in which they interact need to be designed and adapted over time in a way that is consistent and synergistic with the evolution of technological architecture, so that the overall design of the entire enterprise socio-technical system enables efficient delivery. It is necessary to empower teams (more than individuals) as the main organizational elements on which delivery is based and put them in a position to perform at their best. At the same time, it is important to always keep in mind that the common goal must be the success of the entire organization, not of individual teams, departments or business functions.

Some of the questions to ask yourself during the design phase are the following:

• What are the main business value streams?
• What is the purpose and topology that each team must have to produce value?
• What business or technical capabilities do teams need to implement?
• What skills and behavioral characteristics do members of a particular team need to have to effectively contribute to the overall performance of the team?
• How should teams communicate, interact, and relate to each other to contribute to the company’s shared strategic goals and execute business as usual?

Quantyca helps companies optimize the topology of their teams according to the specific context and business goals by acting synergistically on the following key organizational elements:

🠖 Clear and explicit responsibilities
they allow to remove ambiguity, maximize each team’s focus on its purpose and improve the behavioral characteristics that each individual needs to contribute to their team’s purpose

🠖 Team autonomy and empowerment
they allow teams to minimize interdependencies and waiting times, as well as to reduce bottlenecks and unnecessary cross-team communications, eliminate bureaucratic procedures that slow down value generation flows

🠖 Computational Federated Governance
it maintains products’ governance and quality without impacting the scalability of organizational architecture, with an inclusive approach to policy definition and leveraging automation for enforcing.

Challenges

The adoption of an operating model based on the principles described above is to all intents and purposes a Change Management process, which requires changing some perspectives:

From an upstream/downstream perspective to a customer/supplier perspective

In traditional organizational models, teams are normally engaged within processes (through ticket opening or other means) in response to the on-demand needs of other teams or routinely according to pre-established procedures. To increase the efficiency and scalability of data management initiatives, it is necessary to consider that the main mission of some teams is to work “at the service” of other ones, with the aim of making the latter as performing as possible. This requires a change of perspective, such that the teams “served” are to be considered real customers, from the point of view of the supplier team; therefore, the relationship between them must be based on caring for the needs of the customer team, collecting feedback from them and seeking to improve their experience.

From a focus on technology functions to a focus on business outcome

Data assets and AI applications need to be managed with a product management approach, ensuring that there is a unique, persistent and continuous oversight of the entire life cycle of a data/AI product. An assembly line approach, in which teams are made up of specialists from a specific technical function but know little about the upstream and downstream phases of the data value chain, receive input from upstream teams, and “pass the ball” to downstream teams, is no longer suitable for ensuring the oversight, care, and knowledge of the data needed to deliver true value to the business. Instead, those working with data and AI need to have a deep understanding of the business domain and be aligned with the goals of the business function or line of business they support. In addition, to be able to manage the entire data & AI lifecycle autonomously, teams must be cross-functional, i.e. they must be made up of people with heterogeneous and complementary skills (design thinking, development, data-ops, product management…).

From centralized governance to federated and computational governance

The traditional methodology of applying governance was based on restricting individuals and teams that had permissions to perform certain high-privileged operations (for example: updating the data model, starting CICD pipelines for the release of workloads to the production environment, configuring orchestration workflows, provisioning infrastructure resources…). In this way, it was ensured that standards and good practices were respected, which were known by a small pool of people, preventing other people, who did not know about these practices, from circumventing them. This governance model is no longer suitable for scenarios that require organizational scalability and speed of delivery. Instead, governance can be applied in a more scalable and inclusive way, by involving representatives from each group working in the data & AI ecosystem in decisions and making them responsible for ensuring that common policies are enforced within their team. In cases where it is necessary to strengthen the verification of some policies and the compliance with guardrails, it is possible to make policies “computational” (policy as code) and enforce them automatically through the shared platform.

Solution

Quantyca proposes a framework that extends the one described in the book “Team Topologies”, making it more suitable to be applied for the data & AI management function. In the proposed model, the operational teams, which are responsible for implementing data & AI development capabilities, are distinguished from the management committees, which have decision-making power over various aspects that govern the activities of the operational teams.

Applying the framework in each specific context may not necessarily require the establishment of all of the operational teams and coordination committees described below. Choices, both in the initial phase of a transformation roadmap and in the target scenario, must be made following an assessment of the current state of an organization, evaluating all aspects related to corporate culture, structural factors of the company (e.g. its size), organizational constraints, maturity of skills and strategic goals that the company sets itself.

When defining organizational architecture, it is necessary to consider both the static nature of teams, called topology, and the dynamics between teams, called mode of interaction. It is a good practice to rationalize all possible topologies and modes of interaction that teams can adopt in a narrow set to reduce the complexity of the overall operating model.

Mapping common industry team types to the fundamental topologies sets up organizations for success, removing gray areas of ownership and overloaded/underloaded teams
Team Topologies

We recommend that you identify the topology for each team by selecting one of the four key topologies:

Stream-aligned

Stream-aligned teams aim to implement a single end-to-end value generation flow as efficiently and autonomously as possible to produce a business outcome (it can take the form of the development of a product line, a set of features or a specific user experience). In data & AI, these teams manage the entire lifecycle of data & AI products that are relevant to a business subdomain, from ideation through design, development, testing, release and ongoing maintenance. As a result, the selection of members must be such as to cover all the capabilities that the team needs to manage all phases of the product pipeline, as well as the basic product management skills. This does not mean that the team must have specialized and advanced skills on each of the capabilities, but a fundamental cross-functional coverage (data & AI architecture, data modelling, software development, workflow management, data-ops, infrastructure development, user experience, product management) is sufficient. To maximize the level of service offered, it is essential that these teams consolidate and retain the relevant domain knowledge over time, working side by side with business SMEs, Data Owners and Data Stewards.

Enabling

Enabling teams aim to support stream-aligned teams in acquiring skills on the most innovative methodologies and technologies offered by the data & AI landscape, as well as in the adoption of global policies and standards that are decided in a federated way to adopt, to which all stream-aligned teams must comply to ensure that the products created are interoperable. Some examples of methodological and technological areas on which enabling teams focus their efforts to spread knowledge to stream-aligned teams are metadata management and data-ops but, more generally, the work of enabling teams is central to the dissemination of all aspects of Data & AI Culture. The presence of enabling teams allows stream-aligned teams to focus on the core value stream for the business, while introducing innovations in a sustainable way and keeping accumulated technical debt low. Members selected for enabling teams need to possess continuous learning, research and development, training and change management skills.

Complicated Subsystem

Complicated Subsystem teams aim to develop, maintain and administer subsystems or parts of a system that have a high technical or domain complexity, such as to require advanced and specialized skills in a functional area. In the data & AI field, it is sometimes convenient to provide Complicated Subsystem teams for the management of legacy systems or for the development of highly specialized domain functionalities that require dedicated SMEs (e.g., a real-time inventory calculation system, a customer recommendation or customer segmentation engine). It is recommended to resort to the creation of teams of this topology only when strictly necessary and the choice must be motivated by a significant reduction in the cognitive load on stream-aligned teams. Members chosen to be part of the Complicated Subsystem teams need to have deep skills and significant experience in the area of interest (they must be SMEs in their own right).

Platform

Platform teams aim to reduce the cognitive load of stream-aligned teams, creating services that implement common technical functionalities, which are exposed through portals or APIs and can be used independently by stream-aligned teams. These services are free from domain logic, abstracting the complexity of the underlying technological infrastructure, implementing specific technical tasks in a standardized, optimized and governed way and simplifying the Developer Experience. In the field of data & AI, Platform teams are highly recommended to implement the foundation of the technological components that make up the architecture and to develop platform services to support the life cycle of data products and their use by data consumers. The availability of such platform services avoids the need for stream-aligned teams to possess advanced technical skills. On the contrary, the members who make up the Platform teams need to have deep skills in coding, data-ops and product development, as well as a predisposition for collaboration and customer support.

It is important to clearly define the topology of a team, because this helps both team members themselves develop the skills and behavioral expectations required to best contribute to the team’s purpose and other teams understand the value that a specific team is called to provide. In practice, especially in simple organizational settings, it may be reasonable to assign multiple responsibilities to a single team, although this is generally not an optimal choice. In any case, even in such scenarios, the team assigned multiple roles is required to have sufficient maturity to distinguish the various topologies it implements and to adapt its way of working in the continuous transition from one topology to another.

The key topologies describe provide patterns for designing the static nature of a team. However, since the company is a socio-technical system in which different actors and teams interact, the complexity to be addressed often lies in the dynamics of interactions. The book “Team Topologies” suggests three key modes of interaction on which it is recommended to map the dynamic between teams:

Collaboration

It involves two teams collaborating closely with each other, partially sharing responsibilities, objectives, the scope of action and complementing each other with their respective skills. This is suitable for exploration or prototyping, but not recommended for scalable delivery, as it increases the overall cognitive load of the teams involved and reduces efficiency due to the high degree of coordination required between them.

X-as-a-Service

It involves minimal interaction between the teams involved, in which one team plays the role of service provider and the other team of service consumer. The consumer team uses the services offered by the provider through the exposed APIs. The perimeters of responsibility are clearly distinct, and the interaction is regulated by agreements stipulated on APIs and on the expected level of service (it is a “low coupling” model). This is an optimal way to ensure scalable, secure, efficient, and consistent delivery, while it is less effective when experimenting or introducing innovation.

Facilitating

It involves interaction between two teams in which one of them has the task of assisting and supporting the other in their work, facilitating the acquisition of knowledge and mastery of the technological aspects necessary for the generation of value and for the continuous management of evolutionary change. The facilitator team does not develop the products themselves but works alongside the facilitated team to contribute to the latter’s success. It is an optimal way for contexts and phases of a roadmap in which it is necessary to preserve scalability and speed of delivery without giving up on continuously introducing, managing and supporting change, avoiding generating excessive technical debt.

The way two teams interact can change over time, based on the signals you sense in your day-to-day processes and the shape your overall architecture is taking.

In the field of data & AI, the most suitable model for most scenarios is the one shown in the following figure (although, in certain special cases, it is advisable to evaluate slight variations on a case-by-case basis).

The most common team architecture design for data & AI management

The most common situations are those that involve Platform Teams operating in X-as-a-Service mode towards product teams (stream-aligned), while in Collaboration mode with Enabling teams (especially in the ideation and first prototyping phases of platform services). On the contrary, the Enabling teams usually work in Collaboration mode towards the product teams in the initial stages of change, managing developments together, and then gradually transitioning to a Facilitating mode (the one recommended when the product team is fully operational), to make the product teams increasingly autonomous. The interaction between Complicated Subsystem Teams (if any) and product teams is usually Collaboration or X-as-a-Service.

In addition to the operational teams described so far, the framework proposed by Quantyca suggests considering the establishment of the management committees represented in the following figure.

The management committees proposed by Quantyca’s Team Topologies Framework

Each of the committees has very specific responsibilities:

Strategy Steering Committee

It has the authority to define and periodically monitor the Data & AI Strategy and execute Portfolio Management activities. It is the body with the highest decision-making power, which determines the priorities of investments and the allocation of the budget to the programs in the portfolio in line with the strategic goals. It is made up of Management and Advisory figures.

Federated Demand Commitee

It has the authority to prioritize tasks in the backlog which are requested by users and impact the operations of multiple teams, especially BAU tasks. For example, this committee defines how to allocate the capacity of a business subdomain to evolve a source-aligned data product in the face of a request made by a different business subdomain to meet its own business case. It is composed of a representation of Project Managers / Product Managers for each operational team.

Project Management Commitee

It has the authority to monitor the operational progress of business cases, resolve any roadblocks, and manage the backlog synchronization of the various teams required to complete running business cases as efficiently as possible. It consists of a representation of Project Manager / Product Manager / Team Leader for each operational team.

Federated Governance Committee

It has the authority to define global governance policies, standards, and shared architectural choices that all teams must adhere to. It is composed of a representation of Data Architect / Team Leader of all the operational teams involved and of specialized figures (Governance Advisor, Enterprise Architect, Data Governance SME).

Federated Modelling Committee

It has the authority to define and evolve the enterprise conceptual model (Core Domain Ontology) for the purpose of constructing the Knowledge Graph, which expresses the portion of semantics shared by all business subdomains, on which they can find an agreement during the modeling phase. It is composed of a representation of Data Owners / Data Stewards selected by each business subdomain and specialized figures (Governance Advisors, Knowledge Modelers).

The complete route

Quantyca, through the Organizational & Change Governance CoE, has refined the design and implementation of the framework based on feedback collected from on-field contexts and consolidated the customer support experience to gradually adopt the tools described.

The complete adoption journey involves the following steps:

1 Assessment AS-IS Team Topologies framework

Mapping of the current state of ownership distribution model, existing teams and its topology, how teams interact

2 Progettazione TO-BE Team Topologies framework

Design of the desired target state in alignment with the design of the target Platform Blueprint

3 Definizione Minimum Viable Team Topologies framework

Design of the initial state of the Team Topologies framework, to be adopted in the early stages of an incremental change roadmap, in alignment with the design of the minimum viable Platform Blueprint. In the early stages, it is usually recommended to adopt a more simplified version of the framework, introducing the minimum of the necessary teams and committees.

4 Implementazione Minimum Viable Team Topologies framework

Bootstrap of the teams envisaged in the initial phase of the roadmap, sharing of the purpose and expectations to each team, kick-off of the federated committees

5 Monitoraggio e raffinamento periodico

Periodic evaluation of the efficiency of the teams and the potential organizational signals that suggest a change, introduction of changes to the Team Topologies framework

Benefits

Adopting a Team Topologies framework specifically tailored for data & AI management such as the one described has three main advantages:

Focus on business outcome

Foster an operating model that focuses on value streams instead of individual stages of the pipeline

Improved team performance

Improve the performance and motivation of individual teams by clarifying their purpose, expected value, topology and how they are expected to interact with others

Enhanced operational efficiency

Drive the speed and agility of data & AI product development by minimizing handoffs, wait times, interdependencies, and unnecessary interactions between teams