Having fun isn't hard when you have a modern data catalog

-
Data Catalog and Data Fabric are any data architecture enablers.

You can use centralized architecture or decentralized, Data Catalogs will enable effective management and help interact with the data.

Taking a closer look we figure out that Data Catalog is one of the main technology pillars of Data Fabric which has a much wider approach, including also data semantic enrichment, data preparation as well as data recommendation engines and various data orchestrators.

Data Fabric empowered by Data Catalog, is an abstraction layer that helps applications to connect to data, regardless of database technology and data server location, using built-in APIs.

However, a traditionally manually managed data catalog does not qualify as a Data Fabric unit.

Modern Data Catalog is actively driven by the meta-data and scans data sources regularly with no need for manual maintenance. Modern Data Catalogs usually would have built-in fully-automated end-to-end data lineage and enforce governance procedures as well as data access audits.

There are multiple products in the Data Catalog space in the Azure public cloud.

Azure Data Catalog: helps automatic data assets discovery and control who can discover and use which data assets. Supported data sources: Gen1 Data Lake containers and Blob storage, HDFS files and Hive metastore, MySQL, Postgres, Cassandra and MongoDB tables and views, Oracle database server and views, Azure Synapse, Sql Server,  Teradata, SAP HANA, DB2, HTTP endpoints and more. Since the Microsoft Purview launch, no new data catalog accounts can be created.

Microsoft Purview: Next Generation of Azure Data Catalog, a unified Data Governance solution over on-prem, multi-cloud and SaaS data services that helps to build the entire company data landscape, classify and tag sensitive information and establish security and governance procedures. The supported data sources list is very wide and can be found here.


Please note that there is no upgrade path from Azure Data Catalog to Microsoft Purview and the pricing model is different.

Adding to this salad, a Microsoft Fabric, this Azure service title name can be a little misleading. This is an analytic solution on top of Azure Data Lake Storage, a data lakehouse foundation, that helps to manage and process delta parquet files. In the future, this product will most probably grow into the full power Data Fabric abstraction layer but first, it will need to support a much broader number of data sources.

Microsoft Data Fabric items can be managed inside Microsoft Purview. Purview Data Catalog will show metadata of Microsoft Fabric data assets and help classify and protect them. All Data Fabric user activities are logged and available in the Purview audit log.

Comments

Post a Comment

Popular posts from this blog

Unlocking Microsoft Fabric: A Simple Guide when you only have a personal account.

-
Image
" You cannot create a Microsoft capacity using a personal account. Use your organizational account instead." What a disappointment and frustration! I've been trying to set up Microsoft Fabric for a while now and figure out how to work around this error. I use my personal Gmail for an MVP subscription to learn and experiment with Azure services. When I visit app.fabric.microsoft.com and try to use my private Gmail address, I keep getting a similar message: It's frustrating because I'm eager to dive into Fabric and explore its capabilities. I would get the same error trying to set up Azure Data Catalog, PowerBI embedded, and more. What are the different account types and how do they differ? A work or school account is created through Active Directory or other cloud directory services, such as Microsoft 365. On the other hand, a personal account is one that's manually created for individual use, consisting simply of a username and password. After digging into ...
Read more

ETL to ELT journey: Break free your Transformations and discover Happiness and Data Zen

-
Every data integration pipeline consists of 3 stages: Data Extraction (E), Data Transformation (T) and Data Load (L) During the  Data Extraction stage, the source data is read from its origins: transactional databases, CRM or ERP systems or through data scraping from web pages. During the  Data Transformation stage, the necessary modifications are applied to the source data. This includes data filtering, enrichment or merging with existing or other source datasets, data obfuscation, dataset structure alignment or validation, fields renaming and data structuring, according to the canonical data warehouse model.  During the Data Loading stage, the data is stored in the pipeline destination, which could be a staging area, data lake or data warehouse. There are two principal methods for the data integration process: transferring it from where it originated to the destination, where the data will be used for analysis, ETL and ELT. The difference between ETL and ELT pipelines...
Read more

Coding is a rollercoaster of efficiency and eyebrow-raising discoveries.

-
Image
Data Engineers or Developers - many of us love to be gourmet chefs in the kitchen. When it comes to planning and design, we would rather throw all ingredients in the pot and see how it comes out.  Coding without a plan is like assembling a puzzle in a dark room. The result will most probably be unexpected and off the canvas. Whether you follow Waterfall or Agile development strategy, planning and design phases are non-negotiable and are essential to reduce development cycles and redo work. Once upon a time, one data engineer created an amazing piece-of-cake automation pipeline. This masterpiece had very complex logic, pulling data from multiple sources, and merging and persisting the data in a complex, incremental way. When the pipeline started to run successfully and automation flows worked, the data engineer got very excited and considered this development done. A few days later QA engineer found out that the result dataset was never created in the destination. Why has that ...
Read more