The Greatest Reasons to use or not to use a Centralized Data Access Architecture

-

When developing a modern Data Platform Layer, one of the main decisions is whether to opt for centralized or decentralized data access architecture. 


There is no “one-fits-all” solution, both have advantages and disadvantages.



A Centralized Data Access Architecture would usually mean duplicating data from the operational layer into the analytical layer and applying various transformations to data to support and speed up data analytics.

  • Operational online transaction processing Layer, where all microservices and their operational databases are located.

  • Analytical Data Layer, where we would have data lakes that support data Scientists' work and a data warehouse, that supports Business Intelligence.

  • Transformations, ETL or ELT data pipelines, which are moving data from the operational layer into the analytical layer.

If we opt for a Centralized Data Access architecture, what would be the benefits and the drawbacks?

  • CONSISTENCY: consolidating data into a central location can help data accuracy and reliability by having a single source of truth and aligned data modelling standards.
  • PERFORMANCE: having all data in one place improves direct access query performance.
  • GOVERNANCE: easy monitoring and controlling data access
  • DEMOCRATIZED DATA ACCESS: users will access a singular data store and singular technology
  • COST: All domains’ data would be duplicated from operational to analytical layers.
  • DATA LATENCY: This is a fragile architecture that requires centralized responsibility for the Data Pipelines. Data pipeline might take some time to move the data, depending on how many transformations are applied on the way and also failures can lead to big delays.
  • MISALIGNMENT between data producers and ingestion team: Operational layer data changes when not communicated to the Pipeline team can lead to pipeline outages.
  • COMPLEXITY and TIGHT COUPLING: The more data grows, the Data pipelines eventually become a complex labyrinth that is difficult to maintain. Stages of pipelines are linearly dependent, new features might require changes in multiple pipeline components which might break down data quality or other pipelines or slow down deployments.
  • DATA QUALITY: since there is no central ownership of operational and analytical layers, data quality suffers, and data engineers lack domain expertise to understand the nuances of how data is produced and how data is consumed upstream.
  • HARD TO SCALE: scaling a huge system is hard (although cloud data warehouses make scaling a piece of cake)

Does this list sound familiar? Doesn't it sound similar to the monolith vs microservices arguments?

Splitting business responsibility into separate components that can be separately developed and managed and scaled makes perfect sense also in the Data Architecture world.

Subscribe to see upcoming blogs on decentralized data architectures.

Comments

Post a Comment

Popular posts from this blog

Unlocking Microsoft Fabric: A Simple Guide when you only have a personal account.

-
Image
" You cannot create a Microsoft capacity using a personal account. Use your organizational account instead." What a disappointment and frustration! I've been trying to set up Microsoft Fabric for a while now and figure out how to work around this error. I use my personal Gmail for an MVP subscription to learn and experiment with Azure services. When I visit app.fabric.microsoft.com and try to use my private Gmail address, I keep getting a similar message: It's frustrating because I'm eager to dive into Fabric and explore its capabilities. I would get the same error trying to set up Azure Data Catalog, PowerBI embedded, and more. What are the different account types and how do they differ? A work or school account is created through Active Directory or other cloud directory services, such as Microsoft 365. On the other hand, a personal account is one that's manually created for individual use, consisting simply of a username and password. After digging into ...
Read more

ETL to ELT journey: Break free your Transformations and discover Happiness and Data Zen

-
Every data integration pipeline consists of 3 stages: Data Extraction (E), Data Transformation (T) and Data Load (L) During the  Data Extraction stage, the source data is read from its origins: transactional databases, CRM or ERP systems or through data scraping from web pages. During the  Data Transformation stage, the necessary modifications are applied to the source data. This includes data filtering, enrichment or merging with existing or other source datasets, data obfuscation, dataset structure alignment or validation, fields renaming and data structuring, according to the canonical data warehouse model.  During the Data Loading stage, the data is stored in the pipeline destination, which could be a staging area, data lake or data warehouse. There are two principal methods for the data integration process: transferring it from where it originated to the destination, where the data will be used for analysis, ETL and ELT. The difference between ETL and ELT pipelines...
Read more

Coding is a rollercoaster of efficiency and eyebrow-raising discoveries.

-
Image
Data Engineers or Developers - many of us love to be gourmet chefs in the kitchen. When it comes to planning and design, we would rather throw all ingredients in the pot and see how it comes out.  Coding without a plan is like assembling a puzzle in a dark room. The result will most probably be unexpected and off the canvas. Whether you follow Waterfall or Agile development strategy, planning and design phases are non-negotiable and are essential to reduce development cycles and redo work. Once upon a time, one data engineer created an amazing piece-of-cake automation pipeline. This masterpiece had very complex logic, pulling data from multiple sources, and merging and persisting the data in a complex, incremental way. When the pipeline started to run successfully and automation flows worked, the data engineer got very excited and considered this development done. A few days later QA engineer found out that the result dataset was never created in the destination. Why has that ...
Read more