The Greatest Reasons to use or not to use a De-centralized Data Management Architecture

-







Imagine having a dancing party for your data. Everyone in harmony waltzing, stepping on their partner's toes from time to time. 


Distributed data management is no less amusing. It's chaotic, occasionally hair-raising but with the right approach can even be perfect. 


A central huge data warehouse is a struggle to scale efficiently and hard to innovate. There is no clear ownership of the data domains and it is a single point of failure. During peak usage times, data access and processing can be slow. Even implementing updates or upgrades can be quite complex and time-consuming. Centralized databases are attractive targets for cyberattacks and successful breaches can compromise a large amount of sensitive data. 

As an alternative to a centralized Data Warehouse, data can be owned and managed by the domains, producing it. When considering a decentralized approach, we need to make sure there is a self-serve data infrastructure platform that allows different domains or teams to access and exchange data seamlessly. Each domain would take care of data quality, monitoring and observability to track data health. 

Benefits 

  • Improved scalability: each data domain is free to choose or change the data management system at any time and can scale up and down the resources easily. 
  • Fault Tolerance: different data domains can work independently, workload from one domain would not impact other domains' data availability. 
  • Reduced Latency: if there is no need to move the data around, data is available immediately. 

 Drawbacks

  • Complexity: managing decentralized systems is more complex than centralized ones. 
  • Ensuring data consistency across a decentralized network can be challenging, leading to data conflicts. 

There are multiple various decentralized data management options, including Data Mesh, Data Fabric and more. 

Data Mesh more focuses on decentralized ownership of domain-oriented data products and self-serve APIs and interfaces. Each domain has full control over its data and adheres to the standardized protocols. Each domain would have data experts, data engineers and data scientists. 

Data Fabric is a network that emphasizes unified data access, integration, and centralized control approach. Data Fabric allows organizations to blend diverse data sources, formats and structures into a single fabric. 

Data Mesh and Data Fabric approaches can be blended to achieve a flexible data management strategy, leveraging the strength of both approaches. Data can be managed within single domains with well-identified ownership, and interfaces and in addition, use Data Fabric unified and consistent access layer across the organization. 

Opening new doors and figuring things out keeps us moving forward.


Comments

Post a Comment

Popular posts from this blog

SQL Awesomeness: Finding a QUALIFY query clause in the depths of the database ocean

-
Image
After so many years of writing SQL queries, I cannot believe I never came across a QUALIFY query clause. It feels like discovering a treasure in the depths of the database ocean. Even ChatGPT was not aware of it! QUALIFY query clause was invented by Teradata and is not a part of SQL Standart but apparently, multiple database vendors are supporting it, for instance, Snowflake and Databricks. QUALIFY clause solves the challenge when applying a window function as a query filter. SQL Standart and most database vendors will not allow to run of a window function as a part of a WHERE clause because window functions get evaluated after the HAVING clause. We typically create a CTE or subquery adding a window function to be able to filter on a it at a later query stage. Here is an example of how you can use QUALIFY to piece-of-cake filter on the window function.  I will use the Snowflake database, as an example. Let us consider that we have a list of queries and we need to find the last query ex
Read more

Look back and realize how far you came

-
Image
It appeared out of nowhere while I was having a strong argument with the ton of dust that had taken over my computer room. My former favourite book, "Advanced Transact-SQL for SQL server 2000" brought an upset look to my face, the one I usually get from our NetApp admin. He usually gives me such a look when I ask him for more disk space. At the time of SQL 2000 there were far less good SQL books available than today and amazing SQL programming tricks were living on Itzik’s book pages; and, similar to the Rubeus Hagrid’s “The Monster Book of Monsters” it attacked anyone who attempted to open it, unleashing wild and exciting brilliant ideas. Nowadays there are a lot of SQL books but not all of them are worth reading. Some of them should be strongly avoided and if you touch them you should immediately wash your hands and have a drink or three. A few month ago I was asked to technically review some new SQL Server book. Reading their missleading claims of how the optim
Read more

The Greatest Reasons to use or not to use a Centralized Data Access Architecture

-
Image
When developing a modern Data Platform Layer, one of the main decisions is whether to opt for centralized or decentralized data access architecture.  There is no “one-fits-all” solution, both have advantages and disadvantages. A Centralized Data Access Architecture would usually mean duplicating data from the operational layer into the analytical layer and applying various transformations to data to support and speed up data analytics. Operational online transaction processing Layer , where all microservices and their operational databases are located. Analytical Data Layer , where we would have data lakes that support data Scientists' work and a data warehouse, that supports Business Intelligence. Transformations, ETL or ELT data pipelines , which are moving data from the operational layer into the analytical layer. I f we opt for a Centralized Data Access architecture, what would be the benefits and the drawbacks? CONSISTENCY : consolidating data into a central location can
Read more