The Modern Data Stack (MDS)

AI generated image showing a modern data stack

What is a Modern Data Stack? Well, it's a lot like an old data stack, that's been modernised, so it has indoor plumbing and central heating 😇 ... apparently I should be taking this more seriously, so lets start that again, a modern data stack is very similar to the original definition of a data stack, but it's more modular and most of it's in the cloud ☁️.

"A data stack is a collection of technology tools used to gather, store, process, and analyse raw data to derive actionable insights. It acts as a data pipeline, transforming data from various sources into business intelligence. Modern stacks are typically cloud-native and modular, focusing on scalability and ease of integration."

As the name implies a data stack is made up of layers. The number and function of each layer, and exactly what each layer contains is a matter of debate, and to a large extent depends on what you are doing with it. So I've somewhat arbitrarily decided on four main layers -

Let's have a very quick look at these layers now. We can go into them in more detail later.

The Data Ingestion layer

A picture showing a number of items connected by lines to a green funnel that leads to a representation of a database, with the words Data Ingestion underneath

The first layer we encounter is the Data Ingestion layer. This is the layer where data is pulled from the data sources into the Data Stack. It may be modified or transformed as it's pulled in, but these transformations will be basic.

Other representations of a Data Stack will treat the data sources as a separate layer, but I don't think they are. To me, a layer in the Data Stack is something where you do something to the data like ingesting it or storing it, it's not something static. Data sources are static, at least from the perspective of the Data Stack they are, they just exist.

The Data Storage Layer

A representation of a database by using a disk icon which is coloured blue

Once the data has been ingested from the data sources, it needs to be stored somewhere.This will normally this will be a database, like Snowflake or Databricks. This is the Data Storage layer, and this is the heart of the modern Data Stack. It is worth spending some time to make sure you make the right choice.

The Data Transformation Layer

A picture of a red cog with a square, a triangle and a circle moving around it connected by arrows indicating a clockwise rotation. Underneath are the words Data Transformation also in red.

Once you've got your data and you've stored it in your database, you're almost certainly going to need to manipulate it before you can use it. This is done in the data transformation layer. The data comes in and glean, structured, actionable data comes out.

The Data Utilisation Layer

This is where the data is used. This layer is often referred to the Analytics or Visualisation layer in other representations of the Data Stack, but I prefer to use a more generalised term that shows that it covers both of these and more. This is the layer where the business gets the value from the data, where insights are generated.

Other Components

There are other components involved in a modern Data Stack. Typically these will be Orchestration and Governance. To me, these are not layers, they are processes that happen to the entire Data Stack.

Orchestration

This is how data moves through the stack, and the processes and methodologies it uses to do so.

Governance

Data Governance is the boring but very necessary part that covers the overall management of your data to ensure that: there is accountability and responsibility for your data; that your data is of a high quality and trustworthy; and that your data is compliant and properly managed through out it's lifecycle.