Data Operations Demystified: mastering 7 core principals

-


Last week I have attended an Enterprise Data World summit.

This is one of my favourite summits as they always have a broad choice of sessions on any level for data professionals. Unfortunately, due to work arrangements, I was only present on the second day of the summit.

The first session's original title was “Accelerating Data Management with DataOps” but the speaker didn’t show up. Summit organizers have asked another speaker, Doug Needham from DataOps.live, to fill up and he made up a replacement session just in a few minutes before the session was supposed to start.

That takes a lot of knowledge of the topic and courage to stand up for the disappointed audience that was expecting someone else.

Doug is a senior data solutions architect and was doing Data Ops long before the term was invented. This amazing hour was full of exciting aspects of data operations, examples from Doug’s own rich experience and an overview of the Data Ops product his company is working on.

I have very much enjoyed the session and summarised some of my takeaways as I always try to do.

Data Ops is a set of practices that aims to improve collaboration between data engineers, data analysts and other people who are managing and utilizing the data based on 7 core principles:

  • Build ELT data pipelines rather than ETL pipelines.

When raw data is loaded into data warehouse without any major transformations, you can easily script changes and transformations in the later pipeline stages and apply them as data-as-a-code format.

  • Agility, CD/CD and Data-as-A-Code principles

CI/CD and Agility contribute to efficiency, reliability and adaptability of DataOps processes and speed up development, deployment as well as data products consistency and quality.

Data-as-a-code is a concept that extends software development principles to the data managemnt and data pipelines development. It emphasizes treating data infrastructure, pipelines, and operations in a manner similar to how software code is developed, versioned, tested, and deployed.

  • Components design

Design pipelines with modularity in mind and using DAGs ( Directed Acyclic Graphs) to break down complex pipelines into manageable tasks.

  • Environment management

Examples can be column definitions reused in multiple tables, utilizing 0 time data copy features for setting up testing and development environments

  • Governance

Automated change controls and utilizing security principals, maintaining metadata repository/ data catalog to track information about data sources, transformations and lineage.

  • Automation.

The more you can automate, the better. Data Environment Configurations, Data Testing (having as many as possible data validations on columns and on relationships between columns) and Data Flows deployment automation.

  • Collaboration & Self-Service

Rely more on Git on all data objects development rather than using 3rd party tools to compare environments and figure out the changes.

Data Ops is not about the technology, it's more set of principles and if you incorporate the above principles into your Data Operations, you can proudly say that you produce agile data that your customers can trust.


Comments

Post a Comment

Popular posts from this blog

Unlocking Microsoft Fabric: A Simple Guide when you only have a personal account.

-
Image
" You cannot create a Microsoft capacity using a personal account. Use your organizational account instead." What a disappointment and frustration! I've been trying to set up Microsoft Fabric for a while now and figure out how to work around this error. I use my personal Gmail for an MVP subscription to learn and experiment with Azure services. When I visit app.fabric.microsoft.com and try to use my private Gmail address, I keep getting a similar message: It's frustrating because I'm eager to dive into Fabric and explore its capabilities. I would get the same error trying to set up Azure Data Catalog, PowerBI embedded, and more. What are the different account types and how do they differ? A work or school account is created through Active Directory or other cloud directory services, such as Microsoft 365. On the other hand, a personal account is one that's manually created for individual use, consisting simply of a username and password. After digging into ...
Read more

ETL to ELT journey: Break free your Transformations and discover Happiness and Data Zen

-
Every data integration pipeline consists of 3 stages: Data Extraction (E), Data Transformation (T) and Data Load (L) During the  Data Extraction stage, the source data is read from its origins: transactional databases, CRM or ERP systems or through data scraping from web pages. During the  Data Transformation stage, the necessary modifications are applied to the source data. This includes data filtering, enrichment or merging with existing or other source datasets, data obfuscation, dataset structure alignment or validation, fields renaming and data structuring, according to the canonical data warehouse model.  During the Data Loading stage, the data is stored in the pipeline destination, which could be a staging area, data lake or data warehouse. There are two principal methods for the data integration process: transferring it from where it originated to the destination, where the data will be used for analysis, ETL and ELT. The difference between ETL and ELT pipelines...
Read more

Coding is a rollercoaster of efficiency and eyebrow-raising discoveries.

-
Image
Data Engineers or Developers - many of us love to be gourmet chefs in the kitchen. When it comes to planning and design, we would rather throw all ingredients in the pot and see how it comes out.  Coding without a plan is like assembling a puzzle in a dark room. The result will most probably be unexpected and off the canvas. Whether you follow Waterfall or Agile development strategy, planning and design phases are non-negotiable and are essential to reduce development cycles and redo work. Once upon a time, one data engineer created an amazing piece-of-cake automation pipeline. This masterpiece had very complex logic, pulling data from multiple sources, and merging and persisting the data in a complex, incremental way. When the pipeline started to run successfully and automation flows worked, the data engineer got very excited and considered this development done. A few days later QA engineer found out that the result dataset was never created in the destination. Why has that ...
Read more