Having fun isn't hard when you have a modern data catalog

-
Data Catalog and Data Fabric are any data architecture enablers.

You can use centralized architecture or decentralized, Data Catalogs will enable effective management and help interact with the data.

Taking a closer look we figure out that Data Catalog is one of the main technology pillars of Data Fabric which has a much wider approach, including also data semantic enrichment, data preparation as well as data recommendation engines and various data orchestrators.

Data Fabric empowered by Data Catalog, is an abstraction layer that helps applications to connect to data, regardless of database technology and data server location, using built-in APIs.

However, a traditionally manually managed data catalog does not qualify as a Data Fabric unit.

Modern Data Catalog is actively driven by the meta-data and scans data sources regularly with no need for manual maintenance. Modern Data Catalogs usually would have built-in fully-automated end-to-end data lineage and enforce governance procedures as well as data access audits.

There are multiple products in the Data Catalog space in the Azure public cloud.

Azure Data Catalog: helps automatic data assets discovery and control who can discover and use which data assets. Supported data sources: Gen1 Data Lake containers and Blob storage, HDFS files and Hive metastore, MySQL, Postgres, Cassandra and MongoDB tables and views, Oracle database server and views, Azure Synapse, Sql Server,  Teradata, SAP HANA, DB2, HTTP endpoints and more. Since the Microsoft Purview launch, no new data catalog accounts can be created.

Microsoft Purview: Next Generation of Azure Data Catalog, a unified Data Governance solution over on-prem, multi-cloud and SaaS data services that helps to build the entire company data landscape, classify and tag sensitive information and establish security and governance procedures. The supported data sources list is very wide and can be found here.


Please note that there is no upgrade path from Azure Data Catalog to Microsoft Purview and the pricing model is different.

Adding to this salad, a Microsoft Fabric, this Azure service title name can be a little misleading. This is an analytic solution on top of Azure Data Lake Storage, a data lakehouse foundation, that helps to manage and process delta parquet files. In the future, this product will most probably grow into the full power Data Fabric abstraction layer but first, it will need to support a much broader number of data sources.

Microsoft Data Fabric items can be managed inside Microsoft Purview. Purview Data Catalog will show metadata of Microsoft Fabric data assets and help classify and protect them. All Data Fabric user activities are logged and available in the Purview audit log.

Comments

Post a Comment

Popular posts from this blog

SQL Awesomeness: Finding a QUALIFY query clause in the depths of the database ocean

-
Image
After so many years of writing SQL queries, I cannot believe I never came across a QUALIFY query clause. It feels like discovering a treasure in the depths of the database ocean. Even ChatGPT was not aware of it! QUALIFY query clause was invented by Teradata and is not a part of SQL Standart but apparently, multiple database vendors are supporting it, for instance, Snowflake and Databricks. QUALIFY clause solves the challenge when applying a window function as a query filter. SQL Standart and most database vendors will not allow to run of a window function as a part of a WHERE clause because window functions get evaluated after the HAVING clause. We typically create a CTE or subquery adding a window function to be able to filter on a it at a later query stage. Here is an example of how you can use QUALIFY to piece-of-cake filter on the window function.  I will use the Snowflake database, as an example. Let us consider that we have a list of queries and we need to find the last query ex
Read more

Look back and realize how far you came

-
Image
It appeared out of nowhere while I was having a strong argument with the ton of dust that had taken over my computer room. My former favourite book, "Advanced Transact-SQL for SQL server 2000" brought an upset look to my face, the one I usually get from our NetApp admin. He usually gives me such a look when I ask him for more disk space. At the time of SQL 2000 there were far less good SQL books available than today and amazing SQL programming tricks were living on Itzik’s book pages; and, similar to the Rubeus Hagrid’s “The Monster Book of Monsters” it attacked anyone who attempted to open it, unleashing wild and exciting brilliant ideas. Nowadays there are a lot of SQL books but not all of them are worth reading. Some of them should be strongly avoided and if you touch them you should immediately wash your hands and have a drink or three. A few month ago I was asked to technically review some new SQL Server book. Reading their missleading claims of how the optim
Read more

The Greatest Reasons to use or not to use a Centralized Data Access Architecture

-
Image
When developing a modern Data Platform Layer, one of the main decisions is whether to opt for centralized or decentralized data access architecture.  There is no “one-fits-all” solution, both have advantages and disadvantages. A Centralized Data Access Architecture would usually mean duplicating data from the operational layer into the analytical layer and applying various transformations to data to support and speed up data analytics. Operational online transaction processing Layer , where all microservices and their operational databases are located. Analytical Data Layer , where we would have data lakes that support data Scientists' work and a data warehouse, that supports Business Intelligence. Transformations, ETL or ELT data pipelines , which are moving data from the operational layer into the analytical layer. I f we opt for a Centralized Data Access architecture, what would be the benefits and the drawbacks? CONSISTENCY : consolidating data into a central location can
Read more