tag:blogger.com,1999:blog-83542140793585332722024-03-25T00:15:57.363-07:00Maria's journey in the data fieldsAbout me:
My name is Maria Zakourdaev. I am a Cloud Data Architect and Microsoft Data Platform MVP
My twitter feed: @Maria_SQL
LinkedIN: https://www.linkedin.com/in/redheaddba/Unknownnoreply@blogger.comBlogger67125tag:blogger.com,1999:blog-8354214079358533272.post-82868102392312789082024-03-24T13:11:00.000-07:002024-03-25T00:15:25.882-07:00ETL to ELT journey: Break free your Transformations and discover Happiness and Data Zen<p>Every data integration pipeline consists of 3 stages: Data Extraction (E), Data Transformation (T) and Data Load (L)</p><p>During the <b>Data Extraction</b> stage, the source data is read from its origins: transactional databases, CRM or ERP systems or through data scraping from web pages.</p><p>During the <b>Data Transformation </b>stage, the necessary modifications are applied to the source data. This includes data filtering, enrichment or merging with existing or other source datasets, data obfuscation, dataset structure alignment or validation, fields renaming and data structuring, according to the canonical data warehouse model. </p><p>During the <b>Data Loading</b> stage, the data is stored in the pipeline destination, which could be a staging area, data lake or data warehouse.</p><p>There are two principal methods for the data integration process: transferring it from where it originated to the destination, where the data will be used for analysis, ETL and ELT.</p><p>The difference between ETL and ELT pipelines lies in whether the Data Transformation stage occurs before or after the Data Loading stage. </p><p>Let's look closer, at why this difference is significant.</p><h3 style="text-align: left;">ETL pipeline: data is transformed within a data integration tool and then loaded into the data storage or management solution.</h3><p><b>Transformations coupling & data latency: </b>During the ETL pipeline, multiple data transformations would usually be tightly linked and executed sequentially. Failure of one of them would disrupt other transformations, whether by falling the whole data flow or making monitoring the stage's success harder. Downstream data transformation availability will depend on upstream transformation execution times and delays.</p><p><b>Maintainability</b>: ETL pipelines will usually have complex logic that contains multiple transformation stages making them difficult to maintain, understand or apply changes.</p><div><b>Ability to scale:</b> The ETL pipeline can be scaled as a whole regardless of the different needs of each transformation operation.</div><p><b>Data processing efficiency: </b>The data transformations will be performed using the data interaction tool's ability to process the data, which would usually be iterative and not performed as a set operation, using the limited compute power of the data integration tool.</p><p><b>Raw data availability:</b> If the data transformation stage is applied before data is loaded from the source system and transformation operations take time, it will take time till the raw data is available for analysis.</p><p><br /></p><h3 style="text-align: left;">ELT pipeline: data is first loaded into the data storage solution and then transformed within a data store.</h3><p><b>Transformations decoupling & data latency: </b>When the data is being loaded into the data warehouse independently and as a first step, this operation can have an independent schedule from the downstream transformation operations, making the raw data available for analysis with minimal latency. Subsequent data transformation stages can operate as a separate pipeline, removing data pipeline complexity and enhancing flexibility. Each transformation can run individually with its own schedule and be scaled independently. Some of them will run very frequently allowing data analysts and data scientists fast access to aggregated data without dependency on additional heavy transformations that require more time and resources to be completed.</p><p><b>Maintainability</b>: ELT pipelines are typically less complex than ETL pipelines, easier to maintain and adapt to frequently changing requirements.</p><p><b>Ability to scale:</b> Each transformation stage can use as many resources as it needs, balancing the pipeline cost, execution time and performance. Each pipeline can be perfectly aligned to its unique requirements and goals.</p><p><b>Data processing efficiency: </b> Transformation data processing will be performed inside the data warehouse, it will use the warehouse resources as well as various data warehouse features that enable fast data processing, such as data clustering, indexes or intelligent query optimizers.</p><p><b>Raw data availability:</b> ELT pipelines load data in its raw form and make it available for analysis with very low latency, removing any impact from heavy data transformations.</p><p>Regardless, of whether we use cloud data integration platforms, like Azure Data Factory, or on-premice data integration tools, requiring dedicated hardware or virtual machine, optimizing for lower latency and faster data processing, it makes sense to separate the Extract and Load stages from Data Transformation stage, allowing different data transformation scenarios. </p><p></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-69846190767802589162024-03-03T10:13:00.000-08:002024-03-24T11:23:15.970-07:00Beyond Schedules and Speakers: Data TLV in a nutshell<p>Many of us have hobbies. Some hobbies are quite common, such as travelling, painting, or playing computer games.</p><p>I, however, have a rather unique hobby: organizing conferences. The more complicated the logistics and the more people who sign up, the more enjoyable it becomes. Yet, this hobby is quite time-consuming, time that could be dedicated to family or sleep.</p><p>As the event day approaches, the tension mounts. There are too many details to manage, too many things to take care of. The feeling of being overwhelmed and terrified at the same time, creeps in, as mishaps can occur at any moment. This is especially true in our small, brave country where the sound of rocket alarms can disrupt seemingly peaceful moments, with potentially dire consequences.</p><p>But eventually, the day arrives, and the energy is overwhelming. Rooms are filled with eager delegates ready to learn. There are excited speakers, delighted sponsors, and an abundance of delicious food, beer, and networking opportunities.</p><p>I want to extend my gratitude to the Data TLV team. They are my kindred spirits, sharing the same passion for organizing conferences. You guys are amazing, you are the soul of Data TLV. I am thrilled to collaborate with you. Here's to many more successful events!</p><div><span color="rgba(0, 0, 0, 0.9)" face="-apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif" style="background-color: white; font-size: 14px;"><br /></span></div><div><span color="rgba(0, 0, 0, 0.9)" face="-apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif" style="background-color: white; font-size: 14px;"><a href="https://www.linkedin.com/in/itaibinyamin/" target="_blank">Itai Binyamin</a></span></div><div><span color="rgba(0, 0, 0, 0.9)" face="-apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif" style="background-color: white; font-size: 14px;"><a href="https://www.linkedin.com/in/liorking/" target="_blank">Lior King</a></span></div><div><span color="rgba(0, 0, 0, 0.9)" face="-apple-system, system-ui, BlinkMacSystemFont, Segoe UI, Roboto, Helvetica Neue, Fira Sans, Ubuntu, Oxygen, Oxygen Sans, Cantarell, Droid Sans, Apple Color Emoji, Segoe UI Emoji, Segoe UI Emoji, Segoe UI Symbol, Lucida Grande, Helvetica, Arial, sans-serif"><span style="background-color: white; font-size: 14px;"><a href="https://www.linkedin.com/in/xeniata/" target="_blank">Xenia Tarnavsky</a></span></span></div><div><span color="rgba(0, 0, 0, 0.9)" face="-apple-system, system-ui, BlinkMacSystemFont, Segoe UI, Roboto, Helvetica Neue, Fira Sans, Ubuntu, Oxygen, Oxygen Sans, Cantarell, Droid Sans, Apple Color Emoji, Segoe UI Emoji, Segoe UI Emoji, Segoe UI Symbol, Lucida Grande, Helvetica, Arial, sans-serif"><span style="background-color: white; font-size: 14px;"><a href="https://www.linkedin.com/in/yuval-berkovitch/" target="_blank">Yuval Berkovitch</a></span></span></div><div><span color="rgba(0, 0, 0, 0.9)" face="-apple-system, system-ui, BlinkMacSystemFont, Segoe UI, Roboto, Helvetica Neue, Fira Sans, Ubuntu, Oxygen, Oxygen Sans, Cantarell, Droid Sans, Apple Color Emoji, Segoe UI Emoji, Segoe UI Emoji, Segoe UI Symbol, Lucida Grande, Helvetica, Arial, sans-serif"><span style="font-size: 14px;"><br /></span></span><div><span color="rgba(0, 0, 0, 0.9)" face="-apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif" style="background-color: white; font-size: 14px;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg7xXkIixdvG98J0V8Hiv67Q9RWVXl2bG6iB0KrygGVlbGcv4_6i6T-KyDhr4OHPZoCgDf9ebfX9KX1mW2Tr1tacL7BxoQ7fGafM_eTOy8fCz1ltVFywT4myZ_BUxR0qAfz3WurGhto2tVKMtP61raJcP7KNjOQQiHsZa4wPUbXvDdDBWokZ63Jiub843UP" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="533" data-original-width="800" height="426" src="https://blogger.googleusercontent.com/img/a/AVvXsEg7xXkIixdvG98J0V8Hiv67Q9RWVXl2bG6iB0KrygGVlbGcv4_6i6T-KyDhr4OHPZoCgDf9ebfX9KX1mW2Tr1tacL7BxoQ7fGafM_eTOy8fCz1ltVFywT4myZ_BUxR0qAfz3WurGhto2tVKMtP61raJcP7KNjOQQiHsZa4wPUbXvDdDBWokZ63Jiub843UP=w640-h426" width="640" /></a></div><br /><br /></span></div></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-5167209500058292222024-02-17T04:04:00.000-08:002024-02-23T07:25:18.457-08:00Having fun isn't hard when you have a modern data catalog<table>
<tbody><tr><td>Data Catalog and Data Fabric are any data architecture enablers. <br /><br />You can use centralized architecture or decentralized, Data Catalogs will enable effective management and help interact with the data.<br /><br />Taking a closer look we figure out that Data Catalog is one of the main technology pillars of Data Fabric which has a much wider approach, including also data semantic enrichment, data preparation as well as data recommendation engines and various data orchestrators.<br /><br />Data Fabric empowered by Data Catalog, is an abstraction layer that helps applications to connect to data, regardless of database technology and data server location, using built-in APIs.</td>
<td><img border="0" data-original-height="1024" data-original-width="1024" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPCbIOlST4ybbxhyOBP89uWDgcO8CR_Nt4mQF_mShOoJeNREHz2cmkg4WVOfgu7UmvTViHuBEMoW9aCXRJxpuleW8tjQVHQZ4eBTq4RI5cH7E-sXoGHuTqA0PWenPL9677F-Io93e6ot6VqImxQYOrXztakdHslQ2EV9_hLFhgX4O56U6OUb7wnxBmDKIr/s320/DALL%C2%B7E%202024-02-17%2013.10.20%20-%20Create%20an%20illustration%20that%20highlights%20the%20difference%20between%20an%20old-style%20data%20catalog%20and%20a%20modern%20augmented%20data%20catalog.%20The%20left%20side%20of%20the%20imag.webp" width="320" />
</td></tr></tbody></table>
<p>However, a traditionally manually managed data catalog does not qualify as a Data Fabric unit.</p><p>Modern Data Catalog is actively driven by the meta-data and scans data sources regularly with no need for manual maintenance. Modern Data Catalogs usually would have built-in fully-automated end-to-end data lineage and enforce governance procedures as well as data access audits.</p><p>There are multiple products in the Data Catalog space in the Azure public cloud.</p><p>Azure <a href="https://azure.microsoft.com/en-us/products/data-catalog" target="_blank">Data Catalog</a>: helps automatic data assets discovery and control who can discover and use which data assets. Supported data sources: Gen1 Data Lake containers and Blob storage, HDFS files and Hive metastore, MySQL, Postgres, Cassandra and MongoDB tables and views, Oracle database server and views, Azure Synapse, Sql Server, Teradata, SAP HANA, DB2, HTTP endpoints and more. Since the Microsoft Purview launch, no new data catalog accounts can be created.</p><p><a href="https://learn.microsoft.com/en-gb/purview/governance-solutions-overview" target="_blank">Microsoft Purview</a>: Next Generation of Azure Data Catalog, a unified Data Governance solution over on-prem, multi-cloud and SaaS data services that helps to build the entire company data landscape, classify and tag sensitive information and establish security and governance procedures. The supported data sources list is very wide and can be found <a href="https://learn.microsoft.com/en-us/purview/microsoft-purview-connector-overview" target="_blank">here</a>.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjW1Vl6dZ5r-wYH3Ms85DzaFdLjFQpe97CB8Y0ZL6hLdUO-GhrQGvfe_8D9KxuC3ZepTRj8JUwW81WkMa1JilyKexeWMElGQ6dR6NAu-dSiuN65-58sYgUg7aK5XFbbWpfShLFb4i94KHSF4nchQVr1yQumq5TAfjshcnemPqZJwXhpY8orNY_CjNnGF65g" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="509" data-original-width="1200" height="271" src="https://blogger.googleusercontent.com/img/a/AVvXsEjW1Vl6dZ5r-wYH3Ms85DzaFdLjFQpe97CB8Y0ZL6hLdUO-GhrQGvfe_8D9KxuC3ZepTRj8JUwW81WkMa1JilyKexeWMElGQ6dR6NAu-dSiuN65-58sYgUg7aK5XFbbWpfShLFb4i94KHSF4nchQVr1yQumq5TAfjshcnemPqZJwXhpY8orNY_CjNnGF65g=w640-h271" width="640" /></a></div><br />Please note that there is <a href="https://learn.microsoft.com/en-gb/purview/governance-solutions-overview" target="_blank">no upgrade path</a> from Azure Data Catalog to Microsoft Purview and the pricing model is different.<p></p><p>Adding to this salad, a <a href="https://learn.microsoft.com/en-us/fabric/get-started/microsoft-fabric-overview" target="_blank">Microsoft Fabric</a>, this Azure service title name can be a little misleading. This is an analytic solution on top of Azure Data Lake Storage, a data lakehouse foundation, that helps to manage and process delta parquet files. In the future, this product will most probably grow into the full power Data Fabric abstraction layer but first, it will need to support a much broader number of data sources.</p><p>Microsoft Data Fabric items can be managed inside Microsoft Purview. Purview Data Catalog will show metadata of Microsoft Fabric data assets and help classify and protect them. All Data Fabric user activities are logged and available in the Purview audit log.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-74716575685524436842024-01-13T03:17:00.000-08:002024-01-13T04:09:37.606-08:00Coding is a rollercoaster of efficiency and eyebrow-raising discoveries.<br /><p><br /></p><p>Data Engineers or Developers - many of us love to be gourmet chefs in the kitchen. When it comes to planning and design, we would rather throw all ingredients in the pot and see how it comes out. </p><p>Coding without a plan is like assembling a puzzle in a dark room. The result will most probably be unexpected and off the canvas.</p><p>Whether you follow Waterfall or Agile development strategy, planning and design phases are non-negotiable and are essential to reduce development cycles and redo work.</p><p>Once upon a time, one data engineer created an amazing piece-of-cake automation pipeline. This masterpiece had very complex logic, pulling data from multiple sources, and merging and persisting the data in a complex, incremental way. When the pipeline started to run successfully and automation flows worked, the data engineer got very excited and considered this development done. A few days later QA engineer found out that the result dataset was never created in the destination. Why has that happened? The excitement of the flow features and deployment success were strong. Target table existence was not listed in the success metrics? Or maybe there were no success metrics planned and listed at all?</p><p>When we consider some task done, neurons in our head are doing a happy dance stimulated by a surge of the hormone dopamine. Therefore, the 80/20 rule makes us happier. We dash to a minimal viable success with 80% of work done, get dopamine "high-fives" from our brain; and rush to the next task and the next dose of happiness. Working on the leftover 20% is boring and can take ten times more than the first 80%.</p><p>When you clean your room, pick up 80% of the mess and spend the remaining part of the day wondering how on earth a sock ended up in the fridge. Code-wise, it's a rollercoaster of efficiency and eyebrow-raising discoveries.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTx4Aygq6jcVZo8DajCsa_wZBthzQvn9y3P572SGapXKy09KAeO3HMSeZ6jT9v9sbaq5-Hi9pfqc8D9Ud9fTAeLfD1ZUMbW4tE0NhRug2jX71fGergbbQveMbUavnhjSFi7c9TTVnUDJP6-d5HcGc6rWdt1Jj3KTwuc155Xjmp1wa3mjq5dz_HKiDIIZhA/s538/pT5oLML8c.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTx4Aygq6jcVZo8DajCsa_wZBthzQvn9y3P572SGapXKy09KAeO3HMSeZ6jT9v9sbaq5-Hi9pfqc8D9Ud9fTAeLfD1ZUMbW4tE0NhRug2jX71fGergbbQveMbUavnhjSFi7c9TTVnUDJP6-d5HcGc6rWdt1Jj3KTwuc155Xjmp1wa3mjq5dz_HKiDIIZhA/s320/pT5oLML8c.jpg" /></a></div><p></p><p><br /></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-71911304106882478432023-11-21T03:31:00.000-08:002023-11-21T20:42:49.466-08:00The Greatest Reasons to use or not to use a De-centralized Data Management Architecture<div style="text-align: left;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjzAfW52eJ1BUrKNsVZeXbQhy1t2jB5gAQGVkC3y_A_ecFua83PzxPxoa9KtQRNL5wniyeJjXaqOG2Pv_RSI02S98xwwtQX7DtRdyMIRmLgw0QjcvYkZhEzqRuOdhgo7cfcYGp7wVTtNhtdNvKpgETvdVvei5h7SYHvKys8WI3RJ32KvyCC22Ba05N6t_Ih" style="clear: left; float: left; margin-bottom: 1em; margin-left: 1em;"><span style="clear: left; float: left; font-size: medium; margin-bottom: 1em; margin-right: 1em;"><img alt="" data-original-height="467" data-original-width="700" height="152" src="https://blogger.googleusercontent.com/img/a/AVvXsEjzAfW52eJ1BUrKNsVZeXbQhy1t2jB5gAQGVkC3y_A_ecFua83PzxPxoa9KtQRNL5wniyeJjXaqOG2Pv_RSI02S98xwwtQX7DtRdyMIRmLgw0QjcvYkZhEzqRuOdhgo7cfcYGp7wVTtNhtdNvKpgETvdVvei5h7SYHvKys8WI3RJ32KvyCC22Ba05N6t_Ih=w229-h152" width="229" /></span></a></div><p style="text-align: left;"><span style="font-size: medium;"><br /></span></p><p style="text-align: left;"><span style="font-size: medium;"><br /></span></p><p style="text-align: left;"><span style="font-size: medium;"><br /></span></p><p style="text-align: left;"><span style="font-size: medium;"><br /></span></p><p style="text-align: left;"><span style="font-size: medium;"><br /></span></p><p style="text-align: left;"><span style="font-size: medium;"><br /></span></p><p style="text-align: left;"><span style="font-size: medium;">Imagine having a dancing party for your data. Everyone in harmony waltzing, stepping on their partner's toes from time to time. </span></p><p style="text-align: left;"><span style="font-size: medium;"><br />Distributed data management is no less amusing. It's chaotic, occasionally hair-raising but with the right approach can even be perfect. </span></p><p style="text-align: left;"><span style="font-size: medium;"><br />A central huge data warehouse is a struggle to scale efficiently and hard to innovate. There is no clear ownership of the data domains and it is a single point of failure. During peak usage times, data access and processing can be slow. Even implementing updates or upgrades can be quite complex and time-consuming. Centralized databases are attractive targets for cyberattacks and successful breaches can compromise a large amount of sensitive data. <br /><br /></span></p><p style="text-align: left;"><span style="font-size: medium;">As an alternative to a centralized Data Warehouse, data can be owned and managed by the domains, producing it. When considering a decentralized approach, we need to make sure there is a self-serve data infrastructure platform that allows different domains or teams to access and exchange data seamlessly. Each domain would take care of data quality, monitoring and observability to track data health. <br /><br /></span></p><p style="text-align: left;"></p><h3 style="text-align: left;"><span style="font-size: medium;">Benefits </span></h3><ul style="text-align: left;"><li><span style="font-size: medium;">Improved scalability: each data domain is free to choose or change the data management system at any time and can scale up and down the resources easily. </span></li></ul><ul style="text-align: left;"><li><span style="font-size: medium;">Fault Tolerance: different data domains can work independently, workload from one domain would not impact other domains' data availability. </span></li></ul><ul style="text-align: left;"><li><span style="font-size: medium;">Reduced Latency: if there is no need to move the data around, data is available immediately. </span></li></ul><span style="font-size: medium;"><br /></span><h3 style="text-align: left;"><span style="font-size: medium;"> Drawbacks</span></h3><ul style="text-align: left;"><li><span style="font-size: medium;">Complexity: managing decentralized systems is more complex than centralized ones. </span></li></ul><ul style="text-align: left;"><li><span style="font-size: medium;">Ensuring data consistency across a decentralized network can be challenging, leading to data conflicts. </span></li></ul><span style="font-size: medium;"><br />There are multiple various decentralized data management options, including Data Mesh, Data Fabric and more. <br /><b><u><span><br /></span></u></b></span><p></p><p style="text-align: left;"><span style="font-size: medium;"><span style="font-weight: bold;"><span>Data Mesh</span></span> more focuses on decentralized ownership of domain-oriented data products and self-serve APIs and interfaces. Each domain has full control over its data and adheres to the standardized protocols. Each domain would have data experts, data engineers and data scientists. <br /><b><u><span><br /></span></u></b></span></p><p style="text-align: left;"><span style="font-size: medium;"><span style="font-weight: bold;"><span>Data Fabric</span></span> is a network that emphasizes unified data access, integration, and centralized control approach. Data Fabric allows organizations to blend diverse data sources, formats and structures into a single fabric. <br /><br /></span></p><p style="text-align: left;"><span style="font-size: medium;">Data Mesh and Data Fabric approaches can be blended to achieve a flexible data management strategy, leveraging the strength of both approaches. Data can be managed within single domains with well-identified ownership, and interfaces and in addition, use Data Fabric unified and consistent access layer across the organization. <br /><i><br /></i></span></p><p style="text-align: left;"><span style="font-size: medium;"><i>Opening new doors and figuring things out keeps us moving forward.</i></span></p></div><div style="text-align: left;"><br /></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-80269940027794442912023-10-26T04:33:00.007-07:002023-10-26T04:40:24.717-07:00SQL Awesomeness: Finding a QUALIFY query clause in the depths of the database ocean<p></p><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg3UeiHat4KkeblU0VRXgLzECzNC1NGVPqpm4AqQV-71_T1J9xSieLkXY6PLIgAfsryBpFtGRQpduF1bYbc8RbEu36zHT7jTXnjKQ5Glg3wepSZqx0VP0LO7t5KqJH6k2JBkeb7vomGxfqheEh0cuNxUV8oLg5PMyYIOU4k-jfNq1xt7Aij9mD0fGzyRQ84" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="727" data-original-width="970" height="150" src="https://blogger.googleusercontent.com/img/a/AVvXsEg3UeiHat4KkeblU0VRXgLzECzNC1NGVPqpm4AqQV-71_T1J9xSieLkXY6PLIgAfsryBpFtGRQpduF1bYbc8RbEu36zHT7jTXnjKQ5Glg3wepSZqx0VP0LO7t5KqJH6k2JBkeb7vomGxfqheEh0cuNxUV8oLg5PMyYIOU4k-jfNq1xt7Aij9mD0fGzyRQ84=w200-h150" width="200" hidden/></a></div><br /><p>After so many years of writing SQL queries, I cannot believe I never came across a QUALIFY query clause. It feels like discovering a treasure in the depths of the database ocean.</p><p>Even ChatGPT was not aware of it!</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhUEX2laH95N6zEDIUbkIJx_065yY53t69ny-91vABVJCTtB4sJEPDPAGDVRpmfFpMq_SFlmKOO0IN-AMWcp0QYmFdyi3aKTHpUPrmorA4uiUZLkg4dqRJ9G9SunWXN4IE_KkFNCK2ort7fIoK9_lAeGRJQ5BS_N8v9iS7BwenuA-oxXStI7gZ7b0x-B1dE" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="800" data-original-width="1694" height="302" src="https://blogger.googleusercontent.com/img/a/AVvXsEhUEX2laH95N6zEDIUbkIJx_065yY53t69ny-91vABVJCTtB4sJEPDPAGDVRpmfFpMq_SFlmKOO0IN-AMWcp0QYmFdyi3aKTHpUPrmorA4uiUZLkg4dqRJ9G9SunWXN4IE_KkFNCK2ort7fIoK9_lAeGRJQ5BS_N8v9iS7BwenuA-oxXStI7gZ7b0x-B1dE=w640-h302" width="640" /></a></div><br /><br /><p></p><p>QUALIFY query clause was invented by Teradata and is not a part of SQL Standart but apparently, multiple database vendors are supporting it, for instance, Snowflake and Databricks.</p><p>QUALIFY clause solves the challenge when applying a window function as a query filter. SQL Standart and most database vendors will not allow to run of a window function as a part of a WHERE clause because window functions get evaluated after the HAVING clause. We typically create a CTE or subquery adding a window function to be able to filter on a it at a later query stage.</p><p>Here is an example of how you can use QUALIFY to piece-of-cake filter on the window function. </p><p>I will use the Snowflake database, as an example. Let us consider that we have a list of queries and we need to find the last query executed by each user. Here is an example dataset:</p><p><span style="background-color: #2b2b2b; color: #cc7832; font-family: monospace; font-size: 13.0667px; white-space: pre;"><br class="Apple-interchange-newline" /></span></p><div style="background-color: #2b2b2b; color: #a9b7c6; font-family: "JetBrains Mono", monospace;"><pre><span style="font-size: xx-small;"><span style="color: #cc7832;">WITH </span>queries <span style="color: #cc7832;">as </span>(<span style="color: #cc7832;">SELECT </span>*<br /> <span style="color: #cc7832;">FROM </span>(<span style="color: #cc7832;">VALUES</span>(<span style="color: #6a8759;">'Maria'</span>,<span style="color: #6897bb;">1</span>,<span style="color: #6a8759;">'select * from testtable;'</span>,<span style="color: #6a8759;">'2023-10-26 04:00:30.642 +0000'</span>::<span style="color: #cc7832;">timestamp</span>),<br /> (<span style="color: #6a8759;">'Maria'</span>,<span style="color: #6897bb;">20</span>,<span style="color: #6a8759;">'select * from customers where name = ''Nike''; '</span>,<span style="color: #6a8759;">'2023-10-26 05:40:30.642 +0000'</span>),<br /> (<span style="color: #6a8759;">'Nika'</span>,<span style="color: #6897bb;">33</span>,<span style="color: #6a8759;">'select distinct col1 from tableB'</span>,<span style="color: #6a8759;">'2023-10-26 01:00:30.642 +0000'</span>),<br /> (<span style="color: #6a8759;">'Nika'</span>,<span style="color: #6897bb;">56</span>,<span style="color: #6a8759;">'select distinct col1 from tableB where cola != 1'</span>,<span style="color: #6a8759;">'2023-10-27 04:00:30.642 +0000'</span>),<br /> (<span style="color: #6a8759;">'Nika'</span>,<span style="color: #6897bb;">85</span>,<span style="color: #6a8759;">'describe testtable'</span>,<span style="color: #6a8759;">'2023-10-28 04:00:30.642 +0000'</span>)<br /> ) t(user_name,query_id, query_text,query_time)<br /> )<br /><span style="color: #cc7832;">SELECT </span>*<br /><span style="color: #cc7832;">FROM </span>queries<br /><span style="color: #cc7832;">WHERE </span>user_name <span style="color: #cc7832;">IN </span>(<span style="color: #6a8759;">'Maria'</span>,<span style="color: #6a8759;">'Nika'</span>)</span></pre></div><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEj2TZ7tJuTAIT-VG2pw9lKqdcOCr4xtm6THabUjd0eVNj3e841iNxiKYqo2_3ejg67GDfYzq1CVoQcvhwfjSDh-LESQ9-I42-YjfACwC5eBwijzi1OfVmO8jVXS-B2UCfhckQcReQnu3w8RCreIkByHx0JBe7TM2jT_mxzxLjjes84HYd_5umO0e8KFeg9j" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="235" data-original-width="1138" height="132" src="https://blogger.googleusercontent.com/img/a/AVvXsEj2TZ7tJuTAIT-VG2pw9lKqdcOCr4xtm6THabUjd0eVNj3e841iNxiKYqo2_3ejg67GDfYzq1CVoQcvhwfjSDh-LESQ9-I42-YjfACwC5eBwijzi1OfVmO8jVXS-B2UCfhckQcReQnu3w8RCreIkByHx0JBe7TM2jT_mxzxLjjes84HYd_5umO0e8KFeg9j=w640-h132" width="640" /></a></div><br /><p></p><span face="Arial, Helvetica, sans-serif" style="background-color: white; color: #222222; font-size: small;">Here is how elegantly QUALIFY helps to get the last query by each user!!!</span><div><span face="Arial, Helvetica, sans-serif" style="background-color: white; color: #222222; font-size: small;"><br /></span></div><div><div style="background-color: #2b2b2b; color: #a9b7c6; font-family: "JetBrains Mono", monospace;"><pre><span style="font-size: xx-small;"><span style="color: #cc7832;">WITH </span>queries <span style="color: #cc7832;">as </span>(<span style="color: #cc7832;">SELECT </span>*<br /> <span style="color: #cc7832;">FROM </span>(<span style="color: #cc7832;">VALUES</span>(<span style="color: #6a8759;">'Maria'</span>,<span style="color: #6897bb;">1</span>,<span style="color: #6a8759;">'select * from testtable;'</span>,<span style="color: #6a8759;">'2023-10-26 04:00:30.642 +0000'</span>::<span style="color: #cc7832;">timestamp</span>),<br /> (<span style="color: #6a8759;">'Maria'</span>,<span style="color: #6897bb;">20</span>,<span style="color: #6a8759;">'select * from customers where name = ''Nike''; '</span>,<span style="color: #6a8759;">'2023-10-26 05:40:30.642 +0000'</span>),<br /> (<span style="color: #6a8759;">'Nika'</span>,<span style="color: #6897bb;">33</span>,<span style="color: #6a8759;">'select distinct col1 from tableB'</span>,<span style="color: #6a8759;">'2023-10-26 01:00:30.642 +0000'</span>),<br /> (<span style="color: #6a8759;">'Nika'</span>,<span style="color: #6897bb;">56</span>,<span style="color: #6a8759;">'select distinct col1 from tableB where cola != 1'</span>,<span style="color: #6a8759;">'2023-10-27 04:00:30.642 +0000'</span>),<br /> (<span style="color: #6a8759;">'Nika'</span>,<span style="color: #6897bb;">85</span>,<span style="color: #6a8759;">'describe testtable'</span>,<span style="color: #6a8759;">'2023-10-28 04:00:30.642 +0000'</span>)<br /> ) t(user_name,query_id, query_text,query_time)<br /> )<br /><span style="color: #cc7832;">SELECT </span>*<br /><span style="color: #cc7832;">FROM </span>queries<br /><span style="color: #cc7832;">WHERE </span>user_name <span style="color: #cc7832;">IN </span>(<span style="color: #6a8759;">'Maria'</span>,<span style="color: #6a8759;">'Nika'</span>)<br />QUALIFY ROW_NUMBER() OVER (PARTITION <span style="color: #cc7832;">BY </span>user_name <span style="color: #cc7832;">ORDER BY </span>query_time <span style="color: #cc7832;">DESC </span>NULLS <span style="color: #cc7832;">LAST </span>) = <span style="color: #6897bb;">1<br /></span></span></pre></div></div><div><span face="Arial, Helvetica, sans-serif" style="background-color: white; color: #222222; font-size: small;"><br /></span></div><div><span face="Arial, Helvetica, sans-serif" style="background-color: white; color: #222222; font-size: small;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhbglL9v4oDOGThzEN1cYiHkpN6h0OYNZsrSBgRakXnxPyA2T1N1R0oiP6YsGsEtP-eQj_pVydpAv9hyscJxxq6kn2ED0JoTqIUj-NjcagHc9O0PEntxrstPNYaHf3fgCMRfyFu9uv60T4Gk76ywYVYXRFzYqsMfDBSK_LRExQ-fc4MLRS_TClE8kz3cr40" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="125" data-original-width="1137" height="70" src="https://blogger.googleusercontent.com/img/a/AVvXsEhbglL9v4oDOGThzEN1cYiHkpN6h0OYNZsrSBgRakXnxPyA2T1N1R0oiP6YsGsEtP-eQj_pVydpAv9hyscJxxq6kn2ED0JoTqIUj-NjcagHc9O0PEntxrstPNYaHf3fgCMRfyFu9uv60T4Gk76ywYVYXRFzYqsMfDBSK_LRExQ-fc4MLRS_TClE8kz3cr40=w640-h70" width="640" /></a></div><br /></span><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;">I find the QUALIFY clause very convenient and elegant and I wish all database vendors have supported it.<u></u><u></u></p><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;"><u></u> <u></u></p><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;">Here is an equivalent query using subquery:</p><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;"><br /></p><div style="background-color: #2b2b2b; color: #a9b7c6; font-family: "JetBrains Mono", monospace;"><pre><span style="color: #cc7832;"><span style="font-size: 9.8pt;">W</span><span style="font-size: xx-small;">ITH </span></span><span style="font-size: xx-small;">queries <span style="color: #cc7832;">as </span>(<span style="color: #cc7832;">SELECT </span>*<br /> <span style="color: #cc7832;">FROM </span>(<span style="color: #cc7832;">VALUES</span>(<span style="color: #6a8759;">'Maria'</span>,<span style="color: #6897bb;">1</span>,<span style="color: #6a8759;">'select * from testtable;'</span>,<span style="color: #6a8759;">'2023-10-26 04:00:30.642 +0000'</span>::<span style="color: #cc7832;">timestamp</span>),<br /> (<span style="color: #6a8759;">'Maria'</span>,<span style="color: #6897bb;">20</span>,<span style="color: #6a8759;">'select * from customers where name = ''Nike''; '</span>,<span style="color: #6a8759;">'2023-10-26 05:40:30.642 +0000'</span>),<br /> (<span style="color: #6a8759;">'Nika'</span>,<span style="color: #6897bb;">33</span>,<span style="color: #6a8759;">'select distinct col1 from tableB'</span>,<span style="color: #6a8759;">'2023-10-26 01:00:30.642 +0000'</span>),<br /> (<span style="color: #6a8759;">'Nika'</span>,<span style="color: #6897bb;">56</span>,<span style="color: #6a8759;">'select distinct col1 from tableB where cola != 1'</span>,<span style="color: #6a8759;">'2023-10-27 04:00:30.642 +0000'</span>),<br /> (<span style="color: #6a8759;">'Nika'</span>,<span style="color: #6897bb;">85</span>,<span style="color: #6a8759;">'describe testtable'</span>,<span style="color: #6a8759;">'2023-10-28 04:00:30.642 +0000'</span>)<br /> ) t(user_name,query_id, query_text,query_time)<br /> )<br /><span style="color: #cc7832;">SELECT </span>*<br /><span style="color: #cc7832;">FROM </span>( <span style="color: #cc7832;">SELECT </span>*,ROW_NUMBER() OVER (PARTITION <span style="color: #cc7832;">BY </span>user_name <span style="color: #cc7832;">ORDER BY </span>query_time <span style="color: #cc7832;">DESC </span>NULLS <span style="color: #cc7832;">LAST </span>) <span style="color: #cc7832;">as </span>rn<br /> <span style="color: #cc7832;">FROM </span>queries ) tbl<br /><span style="color: #cc7832;">WHERE </span>user_name <span style="color: #cc7832;">IN </span>(<span style="color: #6a8759;">'Maria'</span>,<span style="color: #6a8759;">'Nika'</span>)<br /> <span style="color: #cc7832;">AND </span>rn = <span style="color: #6897bb;">1</span></span></pre></div><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;"><br /></p><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;">The result is the same</p><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;"><br /></p><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhszWQcL25VWUuR1-4nDJFFY5gqF6fBcFGoOu1GJ9RGv0bGJ6AY8oqlt9Uqi-ZozApJJ9067edrBFbg32e2odq3VF7HwRyUCV9Ks5zlB0bUcZIg5Z9-UbER3mqJd1zXYHi2ciFflReh67YxGDemu2OHONDbL4NTbYtPqcBi0BCEP_c7-P8lqxOS03a8WJWK" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="134" data-original-width="1143" height="76" src="https://blogger.googleusercontent.com/img/a/AVvXsEhszWQcL25VWUuR1-4nDJFFY5gqF6fBcFGoOu1GJ9RGv0bGJ6AY8oqlt9Uqi-ZozApJJ9067edrBFbg32e2odq3VF7HwRyUCV9Ks5zlB0bUcZIg5Z9-UbER3mqJd1zXYHi2ciFflReh67YxGDemu2OHONDbL4NTbYtPqcBi0BCEP_c7-P8lqxOS03a8WJWK=w640-h76" width="640" /></a></div><br /><br /><p></p><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;">May all your queries sprint like caffeinated cheetahs</p><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;"><br /></p><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;">Yours</p><p class="MsoNormal" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;">Maria</p></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-45621655310240889262023-09-25T02:27:00.008-07:002023-09-25T02:42:53.756-07:00Data Operations Demystified: mastering 7 core principals<p></p><div class="separator" style="clear: both; text-align: left;"><span style="font-size: medium;"><img border="0" data-original-height="307" data-original-width="800" height="123" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjr7edGHMXVEzLhyoaIPAOvlyd5qHU9L3Dc58U5CLl-AFMMUABHFJqCBVZQ4KJ5zywjM_k9A8m4skHhlex8f_Nq5K7PXTIDcTubJmjJul0ekcjYN2EP_ZWuNVLKV6Y0wEwk7dI3nMuFioHQOnKve20xEN6m3iHEndO6CM8YdXuWakl7SUXN-LVqzAauTMdS/s320/operations-management-information-technology-operations-organization-team-management-business-removebg-preview.png" width="320" /></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-size: medium;"><br /></span></div><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">Last week I have attended an Enterprise Data World summit.</span></p><p style="text-align: left;"></p><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">This is one of my favourite summits as they always have a broad choice of sessions on any level for data professionals. Unfortunately, due to work arrangements, I was only present on the second day of the summit.</span></p><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">The first session's original title was “Accelerating Data Management with DataOps” but the speaker didn’t show up. Summit organizers have asked another speaker, <a href="https://www.linkedin.com/in/dougneedham/" target="_blank">Doug Needham</a> from DataOps.live, to fill up and he made up a replacement session just in a few minutes before the session was supposed to start.</span></p><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">That takes a lot of knowledge of the topic and courage to stand up for the disappointed audience that was expecting someone else.</span></p><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">Doug is a senior data solutions architect and was doing Data Ops long before the term was invented. This amazing hour was full of exciting aspects of data operations, examples from Doug’s own rich experience and an overview of the Data Ops product his company is working on.</span></p><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">I have very much enjoyed the session and summarised some of my takeaways as I always try to do.</span></p><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">Data Ops is a set of practices that aims to improve collaboration between data engineers, data analysts and other people who are managing and utilizing the data based on 7 core principles:</span></p><p></p><ul style="text-align: left;"><li><span style="font-family: verdana; font-size: medium;"><b>Build <a href="https://sqlofthenorth.blog/2021/03/29/elt-etl-design-patterns-with-azure-data-services/" target="_blank">ELT</a> data pipelines rather than <a href="https://learn.microsoft.com/en-us/azure/architecture/data-guide/relational-data/etl" target="_blank">ETL</a> pipelines.</b></span></li></ul><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">When raw data is loaded into data warehouse without any major transformations, you can easily script changes and transformations in the later pipeline stages and apply them as data-as-a-code format.</span></p></blockquote><p></p><ul style="text-align: left;"><li><span style="font-family: verdana; font-size: medium;"><b>Agility, <a href="https://learn.microsoft.com/en-us/azure/devops/pipelines/apps/cd/azure/cicd-data-overview?view=azure-devops" target="_blank">CD/CD</a> and Data-as-A-Code principles</b></span></li></ul><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">CI/CD and Agility contribute to efficiency, reliability and adaptability of DataOps processes and speed up development, deployment as well as data products consistency and quality.</span></p></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">Data-as-a-code is a concept that extends software development principles to the data managemnt and data pipelines development. It emphasizes treating data infrastructure, pipelines, and operations in a manner similar to how software code is developed, versioned, tested, and deployed.</span></p></blockquote><p></p><ul style="text-align: left;"><li><span style="font-family: verdana; font-size: medium;"><b>Components design</b></span></li></ul><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">Design pipelines with modularity in mind and using DAGs ( Directed Acyclic Graphs) to break down complex pipelines into manageable tasks.</span></p></blockquote><p></p><ul style="text-align: left;"><li><span style="font-family: verdana; font-size: medium;"><b>Environment management</b></span></li></ul><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">Examples can be column definitions reused in multiple tables, utilizing 0 time data copy features for setting up testing and development environments</span></p></blockquote><p></p><ul style="text-align: left;"><li><span style="font-family: verdana; font-size: medium;"><b>Governance</b></span></li></ul><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">Automated change controls and utilizing security principals, maintaining metadata repository/ data catalog to track information about data sources, transformations and lineage.</span></p></blockquote><p></p><ul style="text-align: left;"><li><span style="font-family: verdana; font-size: medium;"><b>Automation</b>.</span></li></ul><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">The more you can automate, the better. Data Environment Configurations, Data Testing (having as many as possible data validations on columns and on relationships between columns) and Data Flows deployment automation.</span></p></blockquote><p></p><ul style="text-align: left;"><li><span style="font-family: verdana; font-size: medium;"><b>Collaboration & Self-Service</b></span></li></ul><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: verdana; font-size: medium;">Rely more on Git on all data objects development rather than using 3rd party tools to compare environments and figure out the changes.</span></p></blockquote><p><span style="font-family: verdana; font-size: medium;">Data Ops is not about the technology, it's more set of principles and if you incorporate the above principles into your Data Operations, you can proudly say that you produce agile data that your customers can trust.</span></p><p><br /></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-14389470636282143892023-08-08T03:16:00.012-07:002023-08-13T04:15:45.631-07:00The Greatest Reasons to use or not to use a Centralized Data Access Architecture<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzXBatbNZK-tIyoVuhDzOXufsZXjoA9ZOCS4jgwTeiLfZNqJ9B7hQHfR1ObrK2yHL29-5-xEiKUt3kbKBi9buaaq5LnZzVSEI7XEftEL2jkiylG2FROLY7rF3qley-xXMWfthSTXV_ig9YmXnv7LTAr3Oc4J-ddA54Dcfsrao-NeN04_AJIzt452Rv6XAV/s640/diagram-removebg-preview.png" style="display: none; margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="287" data-original-width="640" height="287" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzXBatbNZK-tIyoVuhDzOXufsZXjoA9ZOCS4jgwTeiLfZNqJ9B7hQHfR1ObrK2yHL29-5-xEiKUt3kbKBi9buaaq5LnZzVSEI7XEftEL2jkiylG2FROLY7rF3qley-xXMWfthSTXV_ig9YmXnv7LTAr3Oc4J-ddA54Dcfsrao-NeN04_AJIzt452Rv6XAV/w640-h287/diagram-removebg-preview.png" width="640" /></a></p><p><span style="font-family: verdana; text-align: justify;">When developing a modern Data Platform Layer, one of the main decisions is whether to opt for </span><b style="font-family: verdana; text-align: justify;"><u>centralized</u></b><span style="font-family: verdana; text-align: justify;"> or </span><b style="font-family: verdana; text-align: justify;"><u>decentralized</u></b><span style="font-family: verdana; text-align: justify;"> data access architecture. </span></p><div class="separator" style="clear: both; text-align: justify;"><span style="text-align: start;"><span style="font-family: verdana;"><br /></span></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="text-align: start;"><span style="font-family: verdana;">There is no “one-fits-all” solution, both have advantages and disadvantages.</span></span></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGFXZwXNKGm3UZB9liGNVU3P8W97E-gT2nf8lA1HJ32fQqhiCiyaGS28W6zaKhadPK9dF7PTWH44-LjhfJj2KCws1GhG9LdzfBtbheo-iitOqKmQY1xXDY6oAR5vOfEyO0f8h9yaAk5RVRbV0tj0wt_Z0jNXBXsD14iRJF8PjlpNulgdW4HwWtpNcJcnGE/s1111/what_isit-removebg-preview%20(1).png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="224" data-original-width="1111" height="130" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGFXZwXNKGm3UZB9liGNVU3P8W97E-gT2nf8lA1HJ32fQqhiCiyaGS28W6zaKhadPK9dF7PTWH44-LjhfJj2KCws1GhG9LdzfBtbheo-iitOqKmQY1xXDY6oAR5vOfEyO0f8h9yaAk5RVRbV0tj0wt_Z0jNXBXsD14iRJF8PjlpNulgdW4HwWtpNcJcnGE/w640-h130/what_isit-removebg-preview%20(1).png" width="640" /></a></div><br /><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: verdana;">A Centralized Data Access Architecture would usually mean duplicating data from the <b>operational layer </b>into the <b>analytical layer</b> and applying various <b>transformations</b> to data to support and speed up data analytics.</span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: verdana;"><br /></span></div><ul style="margin-bottom: 0in; margin-top: 0in;" type="disc"><li class="m_696990076680314829MsoListParagraphCxSpFirst" style="line-height: 15.5467px; margin: 0in;"><span style="font-family: verdana;"><b><span style="color: #38761d;">Operational online transaction processing Layer</span></b><span style="color: #6aa84f;">,</span><span style="color: #222222;"> where all microservices and their operational databases are located.</span></span></li></ul><div><span style="color: #222222; font-family: verdana;"><br /></span></div><ul style="margin-bottom: 0in; margin-top: 0in;" type="disc"><li class="m_696990076680314829MsoListParagraphCxSpMiddle" style="line-height: 15.5467px; margin: 0in;"><span style="font-family: verdana;"><b><span style="color: #38761d;">Analytical Data Layer</span></b><span style="color: #222222;">, where we would have data lakes that support data Scientists' work and a data warehouse, that supports Business Intelligence.<u></u><u></u></span></span></li></ul><div><span style="color: #222222; font-family: verdana;"><br /></span></div><ul style="margin-bottom: 0in; margin-top: 0in;" type="disc"><li class="m_696990076680314829MsoListParagraphCxSpLast" style="line-height: 15.5467px; margin: 0in 0in 8pt;"><span style="font-family: verdana;"><b><span style="color: #38761d;">Transformations, ETL or ELT data pipelines</span><span style="color: #222222;">,</span></b><span style="color: #222222;"> which are moving data from the operational layer into the analytical layer.</span></span></li></ul><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzXBatbNZK-tIyoVuhDzOXufsZXjoA9ZOCS4jgwTeiLfZNqJ9B7hQHfR1ObrK2yHL29-5-xEiKUt3kbKBi9buaaq5LnZzVSEI7XEftEL2jkiylG2FROLY7rF3qley-xXMWfthSTXV_ig9YmXnv7LTAr3Oc4J-ddA54Dcfsrao-NeN04_AJIzt452Rv6XAV/s640/diagram-removebg-preview.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="287" data-original-width="640" height="287" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzXBatbNZK-tIyoVuhDzOXufsZXjoA9ZOCS4jgwTeiLfZNqJ9B7hQHfR1ObrK2yHL29-5-xEiKUt3kbKBi9buaaq5LnZzVSEI7XEftEL2jkiylG2FROLY7rF3qley-xXMWfthSTXV_ig9YmXnv7LTAr3Oc4J-ddA54Dcfsrao-NeN04_AJIzt452Rv6XAV/w640-h287/diagram-removebg-preview.png" width="640" /></a></div><span style="font-family: verdana;"><div><span style="font-family: verdana;"><br /></span></div>I</span><span style="font-family: verdana;">f we opt for a Centralized Data Access architecture, what would be the benefits and the drawbacks?</span><div><span style="font-family: verdana;"><br /></span><div><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTlBfrBsNUndYyEA5CK2RODB2vxoFQzYtZ_mqlC-e9nXVNaKJ_r9or_GV9BrvfQ0oleFxK9To4kBmXbG-MBTMDZfZEaKZ8IAxT92BhBg4KcUpV5Q4KkPHsc9RiBCa4HFMRfJQh3ls5mnpNQ2NLvQIE8JBfTleDqOPEd0dMWTGXYIN7Dfs3iD5Pm8CgM-yj/s359/benefits%20new1.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="87" data-original-width="359" height="78" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTlBfrBsNUndYyEA5CK2RODB2vxoFQzYtZ_mqlC-e9nXVNaKJ_r9or_GV9BrvfQ0oleFxK9To4kBmXbG-MBTMDZfZEaKZ8IAxT92BhBg4KcUpV5Q4KkPHsc9RiBCa4HFMRfJQh3ls5mnpNQ2NLvQIE8JBfTleDqOPEd0dMWTGXYIN7Dfs3iD5Pm8CgM-yj/s320/benefits%20new1.png" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><ul style="color: #222222; margin-bottom: 0in; margin-top: 0in; text-align: start;" type="disc"><li class="m_696990076680314829MsoListParagraphCxSpFirst" style="line-height: 15.5467px; margin: 0in;"><span style="font-family: verdana;"><b>CONSISTENCY</b>: consolidating data into a central location can help data accuracy and reliability by having a single source of truth and aligned data modelling standards.<u></u><u></u></span></li><li class="m_696990076680314829MsoListParagraphCxSpMiddle" style="line-height: 15.5467px; margin: 0in;"><span style="font-family: verdana;"><b>PERFORMANCE</b>: having all data in one place improves direct access query performance.<u></u><u></u></span></li><li class="m_696990076680314829MsoListParagraphCxSpMiddle" style="line-height: 15.5467px; margin: 0in;"><span style="font-family: verdana;"><b>GOVERNANCE</b>: easy monitoring and controlling data access<u></u><u></u></span></li><li class="m_696990076680314829MsoListParagraphCxSpLast" style="line-height: 15.5467px; margin: 0in 0in 8pt;"><span style="font-family: verdana;"><b>DEMOCRATIZED DATA ACCESS:</b> users will access a singular data store and singular technology</span></li></ul></div><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6kHkx3VBZaEX_6GwScFbCDzjln96gFEsLSNxPT-6CslgoueSUcqbc6GY7Rc9vX0lCKuGw2ndXlMO97Pa6rZr19Pu89_4HnLYvHEHN-AerzwD9zJm7Sp1zzP2BGj_Feogf1d0sgLf2xjaxDgFWtO6GmH4kFlZ6RR4qpglzLupwLLTIKBlnLx92riHh4-Ig/s438/drawbacks.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="103" data-original-width="438" height="75" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6kHkx3VBZaEX_6GwScFbCDzjln96gFEsLSNxPT-6CslgoueSUcqbc6GY7Rc9vX0lCKuGw2ndXlMO97Pa6rZr19Pu89_4HnLYvHEHN-AerzwD9zJm7Sp1zzP2BGj_Feogf1d0sgLf2xjaxDgFWtO6GmH4kFlZ6RR4qpglzLupwLLTIKBlnLx92riHh4-Ig/s320/drawbacks.png" width="320" /></a></div><ul style="color: #222222; margin-bottom: 0in; margin-top: 0in;" type="disc"><li class="m_696990076680314829MsoListParagraphCxSpFirst" style="line-height: 15.5467px; margin: 0in;"><span style="font-family: verdana;"><b>COST</b>: All domains’ data would be duplicated from operational to analytical layers.<u></u></span></li><li class="m_696990076680314829MsoListParagraphCxSpMiddle" style="line-height: 15.5467px; margin: 0in;"><span style="font-family: verdana;"><b>DATA LATENCY:</b> This is a fragile architecture that requires centralized responsibility for the Data Pipelines. Data pipeline might take some time to move the data, depending on how many transformations are applied on the way and also failures can lead to big delays.<u></u><u></u></span></li><li class="m_696990076680314829MsoListParagraphCxSpMiddle" style="line-height: 15.5467px; margin: 0in;"><span style="font-family: verdana;"><b>MISALIGNMENT</b> between data producers and ingestion team: Operational layer data changes when not communicated to the Pipeline team can lead to pipeline outages.<u></u><u></u></span></li><li class="m_696990076680314829MsoListParagraphCxSpMiddle" style="line-height: 15.5467px; margin: 0in;"><span style="font-family: verdana;"><b>COMPLEXITY</b> and <b>TIGHT COUPLING</b>: The more data grows, the Data pipelines eventually become a complex labyrinth that is difficult to maintain. Stages of pipelines are linearly dependent, new features might require changes in multiple pipeline components which might break down data quality or other pipelines or slow down deployments.<u></u><u></u></span></li><li class="m_696990076680314829MsoListParagraphCxSpMiddle" style="line-height: 15.5467px; margin: 0in;"><span style="font-family: verdana;"><b>DATA QUALITY:</b> since there is no central ownership of operational and analytical layers, data quality suffers, and data engineers lack domain expertise to understand the nuances of how data is produced and how data is consumed upstream.<u></u><u></u></span></li><li class="m_696990076680314829MsoListParagraphCxSpLast" style="line-height: 15.5467px; margin: 0in 0in 8pt;"><span style="font-family: verdana;"><b>HARD TO SCALE</b>: scaling a huge system is hard (although cloud data warehouses make scaling a piece of cake)</span></li></ul><div><span style="color: #222222; font-family: verdana;"><span style="font-size: 14.6667px;"><br /></span></span></div><div><div style="margin: 0px; text-align: left;"><span style="font-family: verdana;">Does this list sound familiar? Doesn't it sound similar to the </span><span style="font-family: verdana;">monolith vs microservices arguments?</span></div><div style="margin: 0px; text-align: left;"><span style="font-family: verdana;"><br /></span></div><div style="margin: 0px; text-align: left;"><span style="font-family: verdana;">Splitting business responsibility into separate components that can be separately developed and managed and scaled <b>makes perfect sense also in the Data Architecture world.</b></span></div><div style="margin: 0px; text-align: left;"><span style="font-family: verdana;"><br />Subscribe to see upcoming blogs on <b>decentralized data architectures</b>.</span></div><p class="MsoNormal" style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;"><u></u></p><p class="MsoNormal" style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin: 0px;"><u></u></p></div></div></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-75849152422367488302023-06-04T09:41:00.004-07:002023-10-10T13:09:30.346-07:00Truth is multidimensional and depends on the perspective you observe it from<p><span style="font-size: medium;">Just as your Power BI visualizations.</span></p><p><span style="font-size: medium;">I often want to build a dashboard with some sleek elegant visualizations and at the same time show as much data as possible. </span></p><p><span style="font-size: medium;">Tooltip is a great feature when you want to show additional data without making visualizations huge and cumbersome. Charts in Power BI have out-of-the-box boring text tooltips. </span></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg2xoDf-OkaWEEkwMvtly-7NHHyXyL2V4Po3p3IUwzCfbkKMaciAfDsXYyJLvZaYIfucC66bYy59_OxCtjYwzrRBqrSvkXlelqeA_XCkpgq5PpkqPzEqU87kSO1u0wBPW14cBlJG39Uar3vfl0YfGQfgadMsT16uGDCGJhdbvwz046Mo8gXOK91B7LwXt7y" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="527" data-original-width="720" height="469" src="https://blogger.googleusercontent.com/img/a/AVvXsEg2xoDf-OkaWEEkwMvtly-7NHHyXyL2V4Po3p3IUwzCfbkKMaciAfDsXYyJLvZaYIfucC66bYy59_OxCtjYwzrRBqrSvkXlelqeA_XCkpgq5PpkqPzEqU87kSO1u0wBPW14cBlJG39Uar3vfl0YfGQfgadMsT16uGDCGJhdbvwz046Mo8gXOK91B7LwXt7y=w640-h469" width="640" /></a></div><p style="clear: both; text-align: left;"><span style="font-size: medium;"><br />But if we want to add a tooltip to Tables and Matrix visuals, the process is not straightforward.</span></p><p style="clear: both; text-align: left;"><span style="font-size: medium;"><br />I have recently figured out how to build a custom tooltip in a Power BI and got so excited about it, that now I show it to just everyone willing to see it. </span></p><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgDDKbtFSfSqA83wHh1z1OZOvDRf-swQADJ633zWUe21qKifKZtHfF76CS074ddiVcvwL3D7sBA_dGN9nyWQExsiioIp8rCP5nw3b0nbIwiQo-OyW01nvCX7zQ8Va8wbAAvX87l243Ju_dXet3Ve1-mqMcocl2QroLtooS8j1MbEYb6Ek7gc44zdfx6a0et" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="638" data-original-width="599" height="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEgDDKbtFSfSqA83wHh1z1OZOvDRf-swQADJ633zWUe21qKifKZtHfF76CS074ddiVcvwL3D7sBA_dGN9nyWQExsiioIp8rCP5nw3b0nbIwiQo-OyW01nvCX7zQ8Va8wbAAvX87l243Ju_dXet3Ve1-mqMcocl2QroLtooS8j1MbEYb6Ek7gc44zdfx6a0et=w600-h640" width="600" /></a></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both;"><br /></div><p style="clear: both; text-align: left;"><span style="font-size: medium;">Today I will share it with you.</span></p><p style="clear: both; text-align: left;"><span style="font-size: medium;"><br /></span></p><h2 style="clear: both; text-align: left;">My favorite dataset</h2><div class="separator" style="clear: both; text-align: left;"><span style="font-size: medium;">I will use a <a href="https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page" target="_blank">yellow taxi dataset</a>. This free dataset contains billions of trips in New York City for the last 10 years and is provided by the NYC Taxi and Limousine Commission (TLC). It contains information on the pickup and dropoff locations, number of passengers, trip distance, fare amount, tip amount, etc.</span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-size: medium;">Using Power BI, I am loading the January 2022 data from the URL:</span></div><div class="separator" style="clear: both; text-align: left;"><a href="https://s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2022-01.csv"><span style="font-size: medium;">https://s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2022-01.csv</span></a></div></div><span style="font-size: medium;"><br /></span></div><span style="font-size: medium;">I am using this PowerBI query t<b><u>o load the file from a web locat</u></b><b><u>ion</u></b> and adjust datatypes:</span><div><br /></div><div><span class="biFF1" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #ac9739; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">let</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
Source </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">=</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> Csv</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">.</span><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">Document</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">(</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">Web</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">.</span><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">Contents</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">(</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"https://s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2022-01.csv"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">)</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">Delimiter</span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">=</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">","</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> Columns</span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">=</span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">18</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> Encoding</span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">=</span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">1252</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> QuoteStyle</span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">=</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">QuoteStyle</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">.</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">None</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">)</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
#</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"Promoted Headers"</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">=</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> Table</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">.</span><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">PromoteHeaders</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">(</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">Source</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">PromoteAllScalars</span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">=</span><span class="J0kGp" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">true</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">)</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
#</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"Changed Type"</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">=</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> Table</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">.</span><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">TransformColumnTypes</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">(</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">#</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"Promoted Headers"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"VendorID"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> Int64</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">.</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">Type</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"tpep_pickup_datetime"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type datetime</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"tpep_dropoff_datetime"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type datetime</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"passenger_count"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> Int64</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">.</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">Type</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"trip_distance"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type number</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"RatecodeID"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> Int64</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">.</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">Type</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"store_and_fwd_flag"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type text</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"PULocationID"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> Int64</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">.</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">Type</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"DOLocationID"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> Int64</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">.</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">Type</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"payment_type"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> Int64</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">.</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">Type</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"fare_amount"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type number</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"extra"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type number</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"mta_tax"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type number</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"tip_amount"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type number</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"tolls_amount"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type number</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"improvement_surcharge"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type number</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"total_amount"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type number</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">{</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"congestion_surcharge"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> type number</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">}</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">)</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
</span><span class="biFF1" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #ac9739; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">in</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
#</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"Changed Type"</span></div><div><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;"><br /></span></div><div><br /></div><div><p style="text-align: left;"><span style="font-size: medium;">I will also load the pickup locations description table from <a href="https://s3.amazonaws.com/nyc-tlc/misc/taxi+_zone_lookup.csv">https://s3.amazonaws.com/nyc-tlc/misc/taxi+_zone_lookup.csv</a></span></p><div><br /></div><h2 style="text-align: left;">Table Example. </h2><p style="text-align: left;"><span style="font-size: medium;">I have put here a simple table showing tips and the total trip amount, depending on the trip distance. ( I have added a “tripDistanceBucket” column using the below IF construct )</span></p></div><div><br /></div><div><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">TripDistanceBucket </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">=</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
</span><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">IF</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">(</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">'yellow_tripdata_2022-01'</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">trip_distance</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;"><</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">10</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"0m-10m"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
</span><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">IF</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">(</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">'yellow_tripdata_2022-01'</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">trip_distance</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">>=</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">10</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">&&</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">'yellow_tripdata_2022-01'</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">trip_distance</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;"><</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">20</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"10m-20m"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
</span><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">IF</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">(</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">'yellow_tripdata_2022-01'</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">trip_distance</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">>=</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">20</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">&&</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">'yellow_tripdata_2022-01'</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">trip_distance</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;"><</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">30</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"20m-30m"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
</span><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">IF</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">(</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">'yellow_tripdata_2022-01'</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">trip_distance</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">>=</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">30</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">&&</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">'yellow_tripdata_2022-01'</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">trip_distance</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;"><</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">40</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"30m-40m"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
</span><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">IF</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">(</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">'yellow_tripdata_2022-01'</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">trip_distance</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">>=</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">40</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">&&</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">'yellow_tripdata_2022-01'</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">trip_distance</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;"><</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">50</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"40m-50m"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
</span><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">IF</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">(</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">'yellow_tripdata_2022-01'</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">trip_distance</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">>=</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">50</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">&&</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">'yellow_tripdata_2022-01'</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">[</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">trip_distance</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">]</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="iYKHF" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;"><</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="_3p6CM" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #c76b29; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">60</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;"> </span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"50m-60m"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">,</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
</span><span class="qTWn2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #22a2c9; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">"60m and more"</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">)</span><span style="background-color: rgba(0, 0, 0, 0.06); font-family: Inconsolata, Menlo, Consolas, monospace; font-size: 16px; white-space-collapse: preserve;">
</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">)</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">)</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">)</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">)</span><span class="KviDO" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgba(0, 0, 0, 0.06); border: 0px; box-sizing: inherit; color: #5e6687; font-family: Inconsolata, Menlo, Consolas, monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 16px; font-stretch: inherit; font-variant-alternates: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; font-variant-position: inherit; font-variation-settings: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space-collapse: preserve;">)</span></div><div><br /><p></p><div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEioIcAX-ryz1mouyrvmAvRVuOFUDMlvV6PtS73UlLNNK9WW0IuNtNnRiKtbWCqEgLV8izM4_buzscj6006yjhMlldd2IDLXmApe1QUCMTdnf12TIX-qccKSLP5taLi_BJP7uC11yDm66OQs4B5HDJfBg-mdvwsonna9mhlZauHqvP7WB_Tri7Egyauv0x-4" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="406" data-original-width="584" height="445" src="https://blogger.googleusercontent.com/img/a/AVvXsEioIcAX-ryz1mouyrvmAvRVuOFUDMlvV6PtS73UlLNNK9WW0IuNtNnRiKtbWCqEgLV8izM4_buzscj6006yjhMlldd2IDLXmApe1QUCMTdnf12TIX-qccKSLP5taLi_BJP7uC11yDm66OQs4B5HDJfBg-mdvwsonna9mhlZauHqvP7WB_Tri7Egyauv0x-4=w640-h445" width="640" /></a></div><br /><br /></div><p style="text-align: left;"><span style="font-size: medium;">Now, I want to add more information in a tooltip</span></p><h2 style="text-align: left;">Here is how to build a tooltip in 3 simple steps:</h2><p style="text-align: left;"><b><span style="font-size: x-large;">First</span></b><span style="font-size: medium;">, I will add a new page and will change the page information to “Allow use as a tooltip”. I will call this page "Cell tooltip"</span></p></div></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjbkyXlwBHridGhFWsNL7x6Z2P3azf6is7NY0CG9U2ua8ZeyYJ6a8VXCXGi9jFwVDocjSVtXrYTpAVFNqBAgCeQim42SPpYmQdhIXwOoaaEmg_6uzOFV1wCQ1HoTSmR-4bV-av-f6SjbMngZ9d1nF_d-MrhFFVm1KWUcOTt3BA_dq3bRTGlFO_r-gAXvj2k" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="443" data-original-width="253" height="400" src="https://blogger.googleusercontent.com/img/a/AVvXsEjbkyXlwBHridGhFWsNL7x6Z2P3azf6is7NY0CG9U2ua8ZeyYJ6a8VXCXGi9jFwVDocjSVtXrYTpAVFNqBAgCeQim42SPpYmQdhIXwOoaaEmg_6uzOFV1wCQ1HoTSmR-4bV-av-f6SjbMngZ9d1nF_d-MrhFFVm1KWUcOTt3BA_dq3bRTGlFO_r-gAXvj2k=w228-h400" width="228" /></a></div><br /></div><p style="text-align: left;"><b><span style="font-size: x-large;">Next</span></b><span style="font-size: medium;">, to this new “tooltip” page, I will add 3 additional visualizations: the number of passengers by pickup location, the total number of passengers, and the total trips count.</span></p><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgbusfODtBb_CJU1bkFodhPjuixIhpVY_z-0HBJvQPiTN-KYfza_iUnr3b1B8SxiE67Cfe-VORpgvg6s4j7cgE7wPGvO3MwSbx1U99QxbJfkP4dc9L2oRQx7-c1p4xF8wVAs00e8QGcUkshDy8FoqQmURFqY8rBKKc-kjOOyTjtj30ZGIVACNA_VTUmKfM0" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="391" data-original-width="372" height="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEgbusfODtBb_CJU1bkFodhPjuixIhpVY_z-0HBJvQPiTN-KYfza_iUnr3b1B8SxiE67Cfe-VORpgvg6s4j7cgE7wPGvO3MwSbx1U99QxbJfkP4dc9L2oRQx7-c1p4xF8wVAs00e8QGcUkshDy8FoqQmURFqY8rBKKc-kjOOyTjtj30ZGIVACNA_VTUmKfM0=w608-h640" width="608" /></a></div><br /><br /></div><div><p style="text-align: left;"><b><span style="font-size: x-large;">Finally</span></b><span style="font-size: medium;">, I will return to the main page and choose “General” -> “tooltips”. </span></p><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEh9P6rcvpmuj25PvsGtJsZTTsI5yZ3DE0Jv64OjzW309HD1jPuDFGEx90iKjkCgxzhIZ4t0Vzba4bA0PbSi9e9jQzgHtSIC4_AHjsG4tuDeNKnav156xXkDJ8sOXmQDEyUBTIqMIRJJhWvEZ7PBy154eKINrysu4s7Biio4dO7W3hH2mM9qMzFdYZrNuq0Z" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="790" data-original-width="358" height="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEh9P6rcvpmuj25PvsGtJsZTTsI5yZ3DE0Jv64OjzW309HD1jPuDFGEx90iKjkCgxzhIZ4t0Vzba4bA0PbSi9e9jQzgHtSIC4_AHjsG4tuDeNKnav156xXkDJ8sOXmQDEyUBTIqMIRJJhWvEZ7PBy154eKINrysu4s7Biio4dO7W3hH2mM9qMzFdYZrNuq0Z=w291-h640" width="291" /></a></div><br /><br /></div><div><br /></div><div><span style="font-size: medium;">The tooltip tab is supposed to show in the "General" -> "Page" dropdown.</span></div></div><div><span style="font-size: medium;"><br /></span></div><div><b><span style="font-size: x-large;">All done </span></b><span style="font-size: medium;">At this point, when my mouse stops on the table cell, I will get additional information in a tooltip based on the cell where I am now. You can use the same tooltip page in any visualization, chart or table.</span></div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEinvogtqPJG5nN3CKtbXIrCTi1a8-hNQGJtdUrx4eJkH38YaAwu4v2m4LvQfN-9UtA6TKwy4-q3fGlElk-_Gz0e6eVJ_Yvt-UEAxfDhBsF55Nb9wJssLvaqYR23H2JW4HE7JJb_gUo2RICd7fW5N1OEUot9unpyuqGYsReglRwS_MKjAV3tPc1zcM21eZLe" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="729" data-original-width="691" height="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEinvogtqPJG5nN3CKtbXIrCTi1a8-hNQGJtdUrx4eJkH38YaAwu4v2m4LvQfN-9UtA6TKwy4-q3fGlElk-_Gz0e6eVJ_Yvt-UEAxfDhBsF55Nb9wJssLvaqYR23H2JW4HE7JJb_gUo2RICd7fW5N1OEUot9unpyuqGYsReglRwS_MKjAV3tPc1zcM21eZLe=w605-h640" width="605" /></a></div><br /><br /></div><div><br /></div><div><p style="text-align: left;"><span style="font-size: medium;">This feature allows us to create multi-dimensional tables and charts.<br /><br />May all your visualisations make sense</span></p><p style="text-align: left;"><span style="font-size: medium;"><br />Yours,<br />Maria</span></p></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-48621180329912602622023-05-14T04:27:00.009-07:002023-05-14T08:32:29.007-07:00How Data Mesh architecture and Data Catalogs help decentralized data teams.<p>Not too long ago, Data Administrators had to change their long habit of having a monolith database.</p><p>They were forced to accept and agree to the <b>Polyglot persistence</b> - the developer's teams have started to choose different data storage and technologies that would support each application team's data model requirements.</p><p>The time has arrived to <b>break down</b> also the <b>Data Lake monolith paradigm</b>. </p><p><b>Refactoring monolith Data Lake</b> makes a lot of sense. </p><p>The central data lake as well as the central data team is often <b>a huge bottleneck</b>. The central data team is usually busy with fixing broken data pipes and taking care of constant data changes made by the domain owners/development teams. </p><p><b>Data Mesh architecture</b> is coming to the rescue here. Instead of a centralized data team, there would be <b>multiple decentralised domain data teams,</b> producing data sets or consuming other teams' data sets. Domain data team usually knows their domain data very well and are aware of any changes that are happening. </p><div><span style="color: #2b00fe;">Data Mesh architecture helps organizations <b>to grow and scale data infrastructures</b> fast without any inter-dependencies. </span></div><div><br /></div><div>One of the keys to <b>organized</b> data lake decentralization is <b>clear Data Visibility through Data Catalog</b>.</div><div><br /></div><div>Interdependent domain teams, that consume each other's data, as well as Data Analysts, Data Scientists need to have a clear understanding of each Data Mesh Data Asset characteristics:</div><div><ol style="text-align: left;"><li><b>Data Metadata and Lineage, </b></li><li><b>Data Quality, </b></li><li><b>Data Freshness </b></li><li><b>Data Availability.</b></li></ol></div><p><b><span style="color: #2b00fe;">Data Catalog is an organized centralized data assets inventory in the organization that would discover Data Assets and their metadata across data domains.</span></b></p><p><b>Data Catalogs</b> will answer questions, similar to "Where is this data living?", "Who is the data set owner?", "How often is this data set updated?", "Where is this data coming from?", "Which datasets is this dashboard taking data from?", "How the data is being used"</p><p>Data Catalogs would enable the domain teams and data analysts to find relevant data assets, enforce common business vocabulary, and ensure data freshness and correctness.</p><p>Which Data Catalog products are you using in your organization?</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIRKbqXOu2xIGXtCkCVJ-NPrR9i6wxgSgihw5__O14mtk3p834OTdQ5_yGmr4BRifabbEa6BOtydMbfq7xjMAQrggBRhR1xa0nUCoT9W0cheV_0euR7D7QvyNq_m11QOvLYfgJ7OcB_WakSU9sSgT2GGwdjC-G5GdE7eucxCeT82VXyC_V_O578-Krjg/s1600/WhatsApp%20Image%202023-05-14%20at%2018.06.00.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1115" data-original-width="1600" height="430" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIRKbqXOu2xIGXtCkCVJ-NPrR9i6wxgSgihw5__O14mtk3p834OTdQ5_yGmr4BRifabbEa6BOtydMbfq7xjMAQrggBRhR1xa0nUCoT9W0cheV_0euR7D7QvyNq_m11QOvLYfgJ7OcB_WakSU9sSgT2GGwdjC-G5GdE7eucxCeT82VXyC_V_O578-Krjg/w617-h430/WhatsApp%20Image%202023-05-14%20at%2018.06.00.jpeg" width="617" /></a></div><br /><p><br /></p><p><br /></p><p><br /></p><p><br /></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-8980811377366312782023-04-08T08:38:00.006-07:002023-04-10T10:44:43.090-07:00Are you familiar with DATAIKU?<div style="text-align: left;"><span style="background-color: rgba(0, 0, 0, 0.03);"><span style="font-size: medium;">If you want to make DATA a part of EVERYDAY decision-making, then you must try this amazing Data Analysis Platform.</span></span></div><div style="text-align: left;"><span style="background-color: rgba(0, 0, 0, 0.03);"><span style="font-size: medium;"><br /></span></span></div><div style="text-align: left;"><span style="background-color: rgba(0, 0, 0, 0.03);"><span style="font-size: medium;">Dataiku is a tool for everyone, it has Notebooks and Python for Coders, Visual data flows for Clickers, relationships, statistics and visual data forecasting for Decision Makers.</span></span></div><div style="text-align: left;"><span style="background-color: rgba(0, 0, 0, 0.03);"><span style="font-size: medium;"><br /></span></span></div><div style="text-align: left;"><span style="background-color: rgba(0, 0, 0, 0.03);"><span style="font-size: medium;">It's technology agnostic, you can install it on a public cloud, use it as SaaS service or install on-premises. You also can choose ANY DATA PROCESSING ENGINE that will process your workload, use Azure Synapse, Spark or Sql Server and analyze the data WITHOUT ANY DATA MOVEMENT, in "a spreadsheet" like manner. </span></span></div><div style="text-align: left;"><span style="background-color: rgba(0, 0, 0, 0.03);"><span style="font-size: medium;"><br /></span></span></div><div style="text-align: left;"><span style="background-color: rgba(0, 0, 0, 0.03);"><span style="font-size: medium;">Dataiku has many enterprise-scale features, like build-in flow audit, Data Quality features, easy deployments between Dataiku environments and much more.</span></span></div><div style="text-align: left;"><span style="background-color: rgba(0, 0, 0, 0.03);"><span style="font-size: medium;"><br /></span></span></div><div style="text-align: left;"><span style="background-color: rgba(0, 0, 0, 0.03);"><span style="font-size: medium;">https://www.dataiku.com/</span></span></div><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;"><span style="background-color: rgba(0, 0, 0, 0.03);"><span style="font-size: medium;"><br /></span></span></div><div style="text-align: left;"><span style="background-color: rgba(0, 0, 0, 0.03);"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPvlJFu_ZRyV1lPindoWC-e7Xz1oRbgMjsQkHnjSe4UYIdgTat8_1RcXhvcIxNBicTYhPkADWC50ybfjQq4imHghaT97N9SuuSP1If9ET-e8I4vBcHyLl1AnXPw__lXRkwGgR1WAQfN5Yjqa6Oq3teE26RVcke9ms55m5BmL6wCA0zsAY7uUq5nAWdbA/s1600/WhatsApp%20Image%202023-04-10%20at%2020.30.21.jpeg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPvlJFu_ZRyV1lPindoWC-e7Xz1oRbgMjsQkHnjSe4UYIdgTat8_1RcXhvcIxNBicTYhPkADWC50ybfjQq4imHghaT97N9SuuSP1If9ET-e8I4vBcHyLl1AnXPw__lXRkwGgR1WAQfN5Yjqa6Oq3teE26RVcke9ms55m5BmL6wCA0zsAY7uUq5nAWdbA/s16000/WhatsApp%20Image%202023-04-10%20at%2020.30.21.jpeg" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><span style="font-size: medium;"><br /></span></span></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-57228823040024802502023-04-08T08:18:00.015-07:002023-04-10T10:04:30.304-07:00Everything you need to consider when choosing COSMOSDB API<p style="text-align: left;"><span style="font-size: medium;"> <span>Azure CosmosDB is a modern distributed data store that can handle any data volume, any data velocity ( data arrival speed) and any data variety (different types of data).</span></span></p><p style="text-align: left;"><span style="font-size: medium;">CosmosDb requires minimal setup and management efforts.</span></p><p style="text-align: left;"><span style="font-size: medium;">It is very easy to integrate CosmosDB into your existing data infrastructure using various APIs that can either mimic your existing data management systems, like MongoDB, PostgreSQL or Cassandra and provide you with under 10s latency from anywhere, 99.999% availability and instant scalability.</span></p><p style="text-align: left;"><span style="font-size: medium;">From the cost perspective, storage costs and utilization costs are almost the same regardless of which API you are planning to use.</span></p><p style="text-align: left;"><span style="font-size: medium;">There is neither an autoscale nor serverless option for PostgreSQL API.</span></p><p style="text-align: left;"><span style="font-size: medium;"><span>Serverless NoSql API, </span><span>Serverless </span><span>Gremlin API, </span><span>Serverless </span><span>MongoDB API, </span><span>Serverless</span><span> Cassandra API and Serverless Table API are available as only as Single Region write architecture. If you are interested in Multi-region write cluster, you will need to use "Provisioned throughput", either manual or automated.</span></span></p><p style="text-align: left;"><span style="font-size: medium;">Below you can find major considerations that can lead you towards which API would be the best choice for your data infrastructure.</span></p><p></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1mrg7HuFS6LjQJdHxI9ezGsCWIos4sCCDwbt6-DiPvaiDdknTnNxKQQM5Ee_58d5iItIA3cq7ABK6EEFX8TWZN8d-YDIxW6i6iU1TVaNYypcpnfv6n6RsuJZuxIi5An7XV_bXqtwW1OgePirMzqrMHeVoqP4T_Pb-JR_ReeBuGccM7bnlctZdce-OCg/s1600/WhatsApp%20Image%202023-04-10%20at%2019.50.58%20(1).jpeg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1mrg7HuFS6LjQJdHxI9ezGsCWIos4sCCDwbt6-DiPvaiDdknTnNxKQQM5Ee_58d5iItIA3cq7ABK6EEFX8TWZN8d-YDIxW6i6iU1TVaNYypcpnfv6n6RsuJZuxIi5An7XV_bXqtwW1OgePirMzqrMHeVoqP4T_Pb-JR_ReeBuGccM7bnlctZdce-OCg/w640-h480/WhatsApp%20Image%202023-04-10%20at%2019.50.58%20(1).jpeg" width="640" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><br /></div><br /><br /></div><br /><br /><p></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-269834614567025572023-03-25T07:03:00.004-07:002023-03-25T07:13:39.668-07:00SQLBITS session summary: How to start BLOGGING<p>Attending conferences is a lot of fun. </p><p>Networking with smart people during the conferences is a lot of fun and learning.</p><p>Attending conference sessions is a lot of learning and fun for the first 10 min until the mischievous phone flashes and pulls your attention. Twitter notifications, work emails and kids in WhatsApp behave like they have agreed to pull your attention from the session. The moment that I open the sneaky device - I completely lose focus and attention. </p><p>I have found a way that keeps me actively engaged during the session, I write a colourful summary of things being said. It's a very intensive and exhausting process but I like the result that I can share with you and I succeed in keeping my focus sharp.</p><p>Here is my summary from <a href="https://voiceofthedba.com/2023/03/18/blogging-for-the-tech-professional-powerpoint/" target="_blank">Steve Jones session</a> at <a href="https://sqlbits.com/" target="_blank">SqlBits</a> on How to start blogging.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgNgNdtt7QzTmF2EGRje38o1NnC4TrLTtkPa6MAnNSRublV6GEhqUc0AMLueSQHjC6C3giLBJIpq_we5aqrs71Y1HK-g6wCVUmSLESoO4BTLvXR48yHxwHluPU0jy7kdiKA3Fp2ZQAQEtGE2rsqHZlGOS2x2naa2_IM-rCu1WAeATZL-Fm3CYcULmCjEA" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1684" data-original-width="2230" src="https://blogger.googleusercontent.com/img/a/AVvXsEgNgNdtt7QzTmF2EGRje38o1NnC4TrLTtkPa6MAnNSRublV6GEhqUc0AMLueSQHjC6C3giLBJIpq_we5aqrs71Y1HK-g6wCVUmSLESoO4BTLvXR48yHxwHluPU0jy7kdiKA3Fp2ZQAQEtGE2rsqHZlGOS2x2naa2_IM-rCu1WAeATZL-Fm3CYcULmCjEA=s16000" /></a></div><br /><br /><p></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-88989783163617131212023-02-21T10:54:00.003-08:002023-02-21T11:05:08.652-08:00Doing things right or do right things? How to find row count of every table in database efficiently.<br /><p>One Data Engineer had to replicate data from one well-known database vendor to a less-known database vendor.</p><p>He used select count(*) to validate that tables row counts were equal on the source and target.</p><p>It worked so slowly, that he got fired without ever knowing whether the table's content was equal or not.</p><p>Often laziness is a first step towards efficiency. Rather than doing count(*) on each table, the unfortunate DBA could have used internal statistics that every decent database vendor is maintaining, stored in the system views.</p><p>Need to take into consideration that it will never be 100% accurate and will depend on a few things</p><p></p><ul style="text-align: left;"><li>How often do database objects change</li><li>What is the internal or manual schedule for statistics refresh.</li><li>For a lot of database vendors, statistics will get refreshed automatically only when the changed data is more than 10% of the total table rows but this is usually configurable per table.</li><li>The percentage of rows used to calculate the statistics. The most accurate statistics are gathered when all rows in the table are processed (i.e. a 100% sample)</li><li>How often DBA have scheduled custom statistics to refresh.</li><li>How much approximation each database vendor allows themselves to have.</li><li>Keep in mind that there are usually no statistics on external tables</li></ul><p></p><p>I will list here system objects/views that can be utilized to view table sizes for some database vendors.</p><div><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmNCXo4OxxZOMoCkXAmLhFBbt8sZFU-S9cpzUb7ffpdAZgMrmUwOqWuB7jaQ8HmvWJIGi3z3W75fZ5JPQmvqCF-rSBGH3_lbtMwv36MWusd75dIEyotrGqUs1ijf94HdpqzSqp3oFm__CfNlc6ujnZiVLEm6sqp06KQgpWYzQgDRcygiXPlyf5JBYNfw/s1322/Screen%20Shot%202023-02-21%20at%2020.46.58.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1122" data-original-width="1322" height="543" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmNCXo4OxxZOMoCkXAmLhFBbt8sZFU-S9cpzUb7ffpdAZgMrmUwOqWuB7jaQ8HmvWJIGi3z3W75fZ5JPQmvqCF-rSBGH3_lbtMwv36MWusd75dIEyotrGqUs1ijf94HdpqzSqp3oFm__CfNlc6ujnZiVLEm6sqp06KQgpWYzQgDRcygiXPlyf5JBYNfw/w640-h543/Screen%20Shot%202023-02-21%20at%2020.46.58.png" width="640" /></a></div><div><br /></div><span id="docs-internal-guid-2212a1da-7fff-5ef2-699c-2a03db75aa3c"><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Will be happy to hear in comments if I have touched anything that is alive in this cosmos.</span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Yours,</span></p><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Maria</span></p><br /><p dir="ltr" style="line-height: 1.295; margin-bottom: 8pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">"Do what you do so well that they will want to see it again and bring their friends " / Walt Disney.</span></p><div><span face="Calibri, sans-serif" style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div></span><div class="separator" style="clear: both; text-align: center;"><br /></div></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-8403544728190027592022-12-11T22:31:00.213-08:002023-04-11T02:03:20.337-07:00What is Data Governance?<p></p><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCeSPO8-7Tg1FXZRcn2dOS0M0tMTpZx3ya44SKn7dBYhElc80PiKdLza6npwihlkPtVSpl6d05W4GdXrilpsZw2xkNHbWf3sBXIENvL3TkVhzniyEKNMjijwDDkB1t1AR286dXQAcGdW5GHvqiWf32WvYshpgs7KflWOleLb-WVqJ0JZd2jXGHi9uoZg/s403/governance.jpeg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="125" data-original-width="403" height="198" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCeSPO8-7Tg1FXZRcn2dOS0M0tMTpZx3ya44SKn7dBYhElc80PiKdLza6npwihlkPtVSpl6d05W4GdXrilpsZw2xkNHbWf3sBXIENvL3TkVhzniyEKNMjijwDDkB1t1AR286dXQAcGdW5GHvqiWf32WvYshpgs7KflWOleLb-WVqJ0JZd2jXGHi9uoZg/w640-h198/governance.jpeg" width="640" /></a><br /><br /><p><span style="font-size: medium;">"What is Data Governance?", a curious kid asks, peeking above my shoulder into the laptop screen. He is 14 and frequently asks questions with no interest in knowing the answer. Just like many other people around me.</span></p><p><span style="font-size: medium;">"That's a great question", - the first thing you would say when you have no good strategy for how to approach the question. The second part is to think aloud.</span></p><p><span style="font-size: medium;">After a few minutes of gathering my thoughts: "consider the term "data" as a synonym of "useful information". We use the information to support decision-making and choosing strategy. </span></p><p><span style="font-size: medium;">Regardless of whether we are talking about a household or business, having a proper strategy ensures efficient business management and somewhat helps to forecast the future.</span></p><p><span style="font-size: medium;">Data Governance is a system that controls every aspect of the data lifecycle - the series of stages the data goes through, from being captured, stored and used, to data asset destruction. This system helps to ensure and sustain:</span></p><p></p><ul style="text-align: left;"><li><span style="font-size: medium;"><b>Data Quality:</b> is the ability to measure the accuracy of the data, its completeness, reliability and relevance and the ability to track errors and count data issues found. <span style="color: #38761d;">For example, how many times I saw bread crumbs in the bed linens regardless of being assured that you haven't eaten a sandwich in bed. The difference between expectations of your grades to actual grades. </span></span></li><li><span style="font-size: medium;"><b>Data Catalog and Lineage:</b> clear data categorization, organization and relationship between data assets. Ability to understand data asset owner and purpose, attributes' data types and purpose and visualize data changes and ability to view the context in which data assets existed at any given point in time. <span style="color: #38761d;">For example, looking at the pancakes in the garbage bin and visualising eggs linage that starts from the raw eggs to being beaten with the yoghurt, cooked on a pan, transferred to the lunch box and figuring out how they ended up in the garbage bin the next morning - that would be an Eggs data lineage. </span></span></li><li><span style="font-size: medium;"><b>Data Accessibility, Compliance and Security:</b> processes that we establish to support data access, that help to manage and secure data resources. <span style="color: #38761d;">In order to change the time allowed for you to play games on the phone should be secured with a fingerprint to avoid security breaches or comply with the rule of light off at 10pm.</span></span></li><li><span style="font-size: medium;"><b>Data Observability:</b> monitoring the health of important systems in your organization, processes, data pipelines, APIs etc., gathering logs, and defining KPIs to measure strategy success. We can have metrics that help us to reduce costs and manage risks. <span style="color: #38761d;">I have a clear maximum threshold for the number of hoodies that were lost this year in school. I also calculate how many packed uneaten sandwiches I have found under your bed and the average time that took to grow such a fluffy spectacular green mold on them. </span> </span></li></ul><p></p><p><span style="font-size: medium;">Naturally, when I was done with my speech, I figured out that the kid had disappeared out of my sight. The only audience that I had was my dog. Animals are the best listeners and would appreciate anything you have to say. </span></p><p><span style="font-size: medium;">Dear audience, if you haven't yet tuned off, will be happy to hear your take on Data Governance and if there is an additional category that is missing from my list.</span></p><p><br /></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-89832157339573337442022-12-05T05:58:00.001-08:002023-02-24T06:03:11.962-08:00Cloud Scalability vs Cloud Elasticity<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhr-Ovjt05_CsEfwN0WEneP6MrhDOMCFMZqWTjtlDm6NhPfZ01hBmZ4WIuv03Po89WDOM8639psPEW3CHjf_kVIwxYcgkhXeoNz85IqYMuDKndSVOb8jp592hBT-HlaizKcAhkG1Bo2ZuQ9NdgCXf5NaZSCPpNMeJyKWedy1bxtUmTC5TmgskfIMV0EUA/s2186/Screen%20Shot%202023-02-24%20at%2016.01.33.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1196" data-original-width="2186" height="350" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhr-Ovjt05_CsEfwN0WEneP6MrhDOMCFMZqWTjtlDm6NhPfZ01hBmZ4WIuv03Po89WDOM8639psPEW3CHjf_kVIwxYcgkhXeoNz85IqYMuDKndSVOb8jp592hBT-HlaizKcAhkG1Bo2ZuQ9NdgCXf5NaZSCPpNMeJyKWedy1bxtUmTC5TmgskfIMV0EUA/w640-h350/Screen%20Shot%202023-02-24%20at%2016.01.33.png" width="640" /></a></div><br /><p><br /></p><p><br /></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-51869634054737246452022-10-25T09:01:00.006-07:002023-04-11T01:53:57.088-07:00Modern Modular Data Pipelines Example<p> <span style="background-color: white; color: #222222; font-family: "Helvetica Neue"; font-size: large;">I would like to share with you my favourite example of the modern data pipeline.</span></p><span style="background-color: white; color: #222222; font-family: "Helvetica Neue"; font-size: large;">It's amazing.</span><div><span style="background-color: white; color: #222222; font-family: "Helvetica Neue"; font-size: large;"><br /></span><span style="background-color: white; color: #222222; font-family: "Helvetica Neue"; font-size: large;">The <b>first cool thing </b>that we see is that this great pipeline is utilizing a full range of cloud services built for diverse use cases. Choosing the correct tool for each use case can be one of the key factors for the success of your idea, allowing you to get things running as fast as possible without reinventing the wheel.<br /></span><span style="background-color: white; color: #222222; font-family: "Helvetica Neue"; font-size: large;"><b>Another cool thing </b>is if we want to pull data from any non-trivial data source, like Twitter or Jira or GitHub, Azure Databricks is our first friend.<br /></span><span style="background-color: white; color: #222222; font-family: "Helvetica Neue"; font-size: large;">However, t<b>he most noticeable advantage </b>of this pipeline, is that instead of having a monolithic data flow, this pipeline is actually multiple pipelines that are running in parallel. Short, simple, and independent pipelines. Multiple independent pipelines can work in parallel and on different frequencies. One pipeline failure would not impact others. This is an easy way to scale each pipeline separately to speed up only specific tasks and save money on keeping other operations on lower compute.</span></div><div><span style="background-color: white; color: #222222; font-family: "Helvetica Neue"; font-size: large;"><br /></span></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJmN0TvqUpXBd-BQBzBpbyoPs6cbSlYVXrhI7F4Fjg2kfqpdTcvzq9c846AcT6J1BTuAyOyqRHRcR33xznsmLbzKlfhqHUH_MUI08XZ7iOpTcRoCIQFIzCmHk9MYXdWvD9cdPtMSPLRmTk1kur5kwThNkQoOPzPNr3aHAEUBJKfkRGr9aZuMO3U9me0Q/s1069/ci-cd-pipeline-cloud-architecture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="584" data-original-width="1069" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJmN0TvqUpXBd-BQBzBpbyoPs6cbSlYVXrhI7F4Fjg2kfqpdTcvzq9c846AcT6J1BTuAyOyqRHRcR33xznsmLbzKlfhqHUH_MUI08XZ7iOpTcRoCIQFIzCmHk9MYXdWvD9cdPtMSPLRmTk1kur5kwThNkQoOPzPNr3aHAEUBJKfkRGr9aZuMO3U9me0Q/s16000/ci-cd-pipeline-cloud-architecture.png" /></a></div><br /><span style="color: #222222; font-family: Helvetica Neue; font-size: medium;"><br /></span><div><span style="background-color: white; color: #222222; font-family: "Helvetica Neue"; font-size: large;"><br /></span></div><div><span style="background-color: white; color: #222222; font-family: "Helvetica Neue"; font-size: large;">Pipeline Image Source: <a href="https://devblogs.microsoft.com/cse/2018/12/12/databricks-ci-cd-pipeline-using-travis/">https://devblogs.microsoft.com/cse/2018/12/12/databricks-ci-cd-pipeline-using-travis/</a></span></div><div><span style="background-color: white; color: #222222; font-family: "Helvetica Neue"; font-size: large;"><br /></span></div></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-45134252605302563462022-10-13T09:16:00.189-07:002023-02-24T06:55:35.573-08:00When should you use Azure Databricks?<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3EgFW-AIVUmFSr02yRmprHO_KzBgoJYk62GxmgunqdC0VHOy8htjZ-OdUGBlwbz1casAg6SxEzAPbEW1RFXQA7VNgQSywbE5s8DtE-Xyne9gYijzsZrEqgCrpSi2rhQCAfzlFkRdVV6LO64W_ikyPlM7Azdxu00qnY1ZA4MvOYdahK7BNU1Aw6zuy9g/s1024/databricks.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="511" data-original-width="1024" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3EgFW-AIVUmFSr02yRmprHO_KzBgoJYk62GxmgunqdC0VHOy8htjZ-OdUGBlwbz1casAg6SxEzAPbEW1RFXQA7VNgQSywbE5s8DtE-Xyne9gYijzsZrEqgCrpSi2rhQCAfzlFkRdVV6LO64W_ikyPlM7Azdxu00qnY1ZA4MvOYdahK7BNU1Aw6zuy9g/w640-h320/databricks.png" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">Once upon a time, Sql Server was our central tool for data management, for both OLTP (online transactions processing) or OLAP(online analytical processing) database systems.</span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">We have used Sql Server Agent Jobs to pull the data from FTP or any other source. We have used Sql Server stored procedures to pull the data into the Staging database. We have used Sql Server stored procedures to enrich and aggregate the data. And we have used Sql Server as a data serving layer. </span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><span>These days we need to consider utilizing various cloud services. A</span><span>ttempts to lift and shift existing systems into the cloud often end up being quite expensive if we tend to keep Sql Server taking care of all data pipeline stages. </span></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">There are multiple great services in the Azure cloud and Microsoft tends to build each product with multiple features allowing it to take care of multiple pipeline stages. This does not mean that we need to go back to the monolith architecture, let's find out where each service fits.</span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><span>Data Factory can </span><span>pull the data from any source, </span><span>orchestrate the flow, and enrich and aggregate the data.</span></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><span>DataBricks can </span><span>pull the data from any source, </span><span>orchestrate the flow, enrich and aggregate the data, serve the data from data lakehouse and visualize it using dashboards.</span></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><span>Synapse Analytics with integrated pipelines can </span><span>pull the data from any source, </span><span>orchestrate the flow, enrich and aggregate the data and serve the data.</span><span> </span></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">PowerBI can pull the data from any source, enrich and aggregate the data and serve and visualize the data. </span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">Sql Managed instance can pull the data from any source, enrich and aggregate the data and serve the data. </span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><span>In the above image, we can see where Microsoft positions each of the abovementioned cloud services and</span><span> how Azure services can be used as a lego piece to build the data warehouse pipeline architecture.</span></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">Azure Data Factory should be used as an ingestion and orchestration service. Load data from external or internal sources and store it in the Azure Data Lake Storage.</span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">Azure Data Lake should be housing data of all types, except data used for online transaction processing.</span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">Azure Databricks's purpose is to prepare, transform, enrich and train datasets.</span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">Azure Synapse's purpose is to serve clean aggregated data from Azure Data Lake Storage.</span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">Power BI will be a visualization layer, reading data off the Azure Data Lake storage or Azure Synapse Analytics and generating analytical reports and dashboards.</span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">Sql Managed Instance is not in the above image because its too expensive to be used in data warehousing architecture. Sql Server's purpose is to take care of online transaction processing, store and serve relational data and support high amounts of inserts, updates and deletes while making sure the data is consistent. This is a very difficult zone, therefore, this service price is quite high.</span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both; text-align: justify;"><span style="font-size: medium;">Can be great to hear your opinion or observations in the comments...</span></div><div style="text-align: justify;"><br /></div></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-42862082628366912942022-09-24T11:25:00.010-07:002022-09-28T04:04:44.698-07:00Over time you learn more things and do better (DATA TLV puzzles and excitement)<p></p><div class="separator" style="clear: both; text-align: center;"><img border="0" data-original-height="827" data-original-width="1055" height="157" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTihzEG-IfI8rJsw-vpCObV2m3Qh-1U0NppbyaOgCI0DFXLc6OSweCTHBVtmyQT-Q8QkHNS8UM77GcB1gLqqy1K9aYoj0cP_IjXaIU14M2NDyGWxz9J_OvTmCC4JorktMveT-R9gdGheZf68x5EhJLChDjOjx-vAkz_HxG06XnE2oQXdZPGbUCDDmezQ/w200-h157/logo_final_with_people.jpg" width="200" /></div><div class="separator" style="clear: both; text-align: left;"><div class="separator" style="clear: both; text-align: center;"><img border="0" data-original-height="3265" data-original-width="4898" height="185" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj12aN3IVoL40i-wo9ZobzEx6qvWqLYr8peZBW2m6qRfMkcofUkBMsa1MpkHWQxIw9C33BCCx94BitL769mzgrJN1L9Ddai1Ff22qZMDmVaNfydBx6pECVHsW8zebhZPEeImvGDr4QjCL8HU3pEEFje6yDmfsI0S7hY_4v_E0wUAbPfSon97aFBdovjYw/w281-h185/IMG_2146.JPG" width="281" /> <img border="0" data-original-height="2252" data-original-width="4000" height="187" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSoDZiEFG7eFyn4LsnHn8yYHDxEmmcCsyDFuJR413xvK8o66iJFy6wvn1ERNPkfbWmRg3fI-AaEfKoUpR9bSmwn3PD7AXIuuZoYsDH1M8UUyB21FZf7bSOP941nspbfwDZQXO_ooubfYzCdneDSInTEknEPjy04CuoD9UKlOqbhq3_E5MFb52JrvOFNg/w334-h187/20220915_111457.jpg" width="334" /></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: verdana; font-size: large;"><br /></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: verdana; font-size: large;">I took the first sip from my morning coffee when the last summit session had started, somewhere around 15:00. I heavily sank to a sofa and realized that this is the first time I had a chance to sit down today. Another organizer was sitting nearby with the exact "stick a fork in me, I am done" look, staring nowhere. "Oh, I am so tired" - that was my 18-year-old nephew Sonya anchored on the other side of the sofa. She had helped us with registrations and timekeeping in classes and has absorbed all speakers' disappointment with outdated presentation equipment. "Most speakers finish before their time", she said with an exhausted look. "Very smart and intelligent people. I didn't understand anything they talked about."</span></div></div><p></p><p><span style="font-family: verdana; font-size: medium;">I wish we knew how to make sure the summit meets everyone's expectations.</span></p><p><span style="font-family: verdana; font-size: medium;">Every sponsor wants to deliver a session. The more sponsored sessions mean fewer slots for community speakers sessions. The toughest part of summit organization is selecting sessions that will enter the agenda when there are too many submissions. How do you say to someone you work with, your friend, that his session was not accepted? How to deal with sadness in their eyes? To allow more community speakers to speak, organizers open more parallel rooms. Accepted speakers will be pleased but attendees start to complain that it's hard to decide which class to attend. They feel that no matter which session they choose, they are missing something more interesting. At the same time, speakers complain that their rooms are not full. Another tremendous challenge is to figure out, how many people will show up. How can we calculate the approximate attendance when most people don't RSVP. How had Corona affected in-person events? Are people hungry for frontal meetups or have got used to learning from home? </span></p><p><span style="font-family: verdana; font-size: medium;">The more attendees you assume, the more budget you need. This triggers more sponsors and fewer community sessions. Now we have got the deadlock.</span></p><p><span style="font-family: verdana; font-size: medium;">Why do we do this heavy work every year? What do we get out of it? Is that about grateful, tired and overloaded with new knowledge attendees with their hands full of free swag? </span></p><p><span style="font-family: verdana; font-size: medium;">You can't make everybody happy. Unless you are chocolate. </span></p><p><span style="font-family: verdana; font-size: medium;">Me and my cold coffee have left the venue like the captain that is the last one off the ship. People who haven't shown up were lying on a floor in a pile of tags.</span></p><p><span style="font-family: verdana; font-size: medium;"><span>It is hard to believe that more than four hundred people a few hours ago crowded the place, finally happy to meet old friends, after two years of lockdown, masks, </span>and online meetups. Fabulous to see them talking to each other and eating ice cream. Smiling, exchanging ideas, and solving each other challenges. Drinking alcohol cocktails between sessions. Hugging each other on magnetic pictures that will find their way to the home refrigerators to remind us about the Data TLV summit, the annual Data People gathering.</span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeWj3WtyI72NrtxkDNptlcW5Ks4y8EUsPDY_jTo40KONfDw_i0nQ6VVoJu0LECCe6g2E3ZAKNMTutrgRlmkmc62xBSauRMHA4EPx6HhX1VfIrpjo4H1kecVZ0UXKUO1poRGH6WbXGl2DrUdvyKf1lyPZqxap9z-03--Kjqt0baHnO4h5_SFv8qNHI37Q/s1145/new_one.jpg" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="1052" data-original-width="1145" height="294" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeWj3WtyI72NrtxkDNptlcW5Ks4y8EUsPDY_jTo40KONfDw_i0nQ6VVoJu0LECCe6g2E3ZAKNMTutrgRlmkmc62xBSauRMHA4EPx6HhX1VfIrpjo4H1kecVZ0UXKUO1poRGH6WbXGl2DrUdvyKf1lyPZqxap9z-03--Kjqt0baHnO4h5_SFv8qNHI37Q/s320/new_one.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><img border="0" data-original-height="3016" data-original-width="4528" height="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5Jx7lSqzA6jmXt-twojf_NDTcoVRpQgX34K9Jze9NLvt8QFv1eHoSF1MQzHNLxkIr5QiUIKfBU-rIWiJQr4rb_G7DtjmfUGM904lWRzvENrOvN7h7odHcG1iwr-xjbjn4C3YVitUws38aeaLZ1PIuD_x-LlbkZ_TV_zGEExYggoivrNNd_TUm2LdP3Q/w225-h150/358.JPG" width="225" /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi92Q9IkBrn3Yzebxa_kj0VhmYTpioOsQyPzWx-6f1CB7mYZ2vZoD9HrXtO__Yin5ScFYy860D6wUWp0bRvFpM0MF1-xkfozrUejdVDfgjbc4YVSmBR6qO_PDkmYBocbdT9hQWFLktqN1pxrRfWBodIalsb20l_nVMuwI67B1UXnOd6wSnaeXM8YaygPQ/s4528/543.JPG" style="margin-left: 1em; margin-right: 1em; text-align: right;"><img border="0" data-original-height="3016" data-original-width="4528" height="146" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi92Q9IkBrn3Yzebxa_kj0VhmYTpioOsQyPzWx-6f1CB7mYZ2vZoD9HrXtO__Yin5ScFYy860D6wUWp0bRvFpM0MF1-xkfozrUejdVDfgjbc4YVSmBR6qO_PDkmYBocbdT9hQWFLktqN1pxrRfWBodIalsb20l_nVMuwI67B1UXnOd6wSnaeXM8YaygPQ/w220-h146/543.JPG" width="220" /></a><img border="0" data-original-height="3016" data-original-width="4528" height="152" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibbROWQddOLu_-uv8AQ1VXtgMzi2gq6B5ZY7nONN2nk3xORmF68tQ8TbSE0PIErAafPTze83TVTxX0ZFY2H9UFE4dULcTQkQ_9YzFfVWoAFyIoenX2UcfkPoJBLiEoT_6odw0NiHlY6pWyQ3Ru6RBI0EemrAyiElsT-9XXsiPZ4LRVwDK_pKgqwap-NQ/w229-h152/619.JPG" width="229" /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgERwHkKQU9yaknNGFevRNSZ5RDUhECQrRHmb0Az07sW0rrMHO5kA7r7crAiBDPoqYisxwIi3I9jgUkBwGJimn2fZ63fht5ytV68rTGSwNP1k9eLf0Ymt4VbblAX7BefdiQkknczI414MB1mx8qz1HiPI9x5aG6c4P27GmDN3CzsJgoITs2v0md49ffaA/s4528/672.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3016" data-original-width="4528" height="152" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgERwHkKQU9yaknNGFevRNSZ5RDUhECQrRHmb0Az07sW0rrMHO5kA7r7crAiBDPoqYisxwIi3I9jgUkBwGJimn2fZ63fht5ytV68rTGSwNP1k9eLf0Ymt4VbblAX7BefdiQkknczI414MB1mx8qz1HiPI9x5aG6c4P27GmDN3CzsJgoITs2v0md49ffaA/w228-h152/672.JPG" width="228" /></a><img border="0" data-original-height="3016" data-original-width="4528" height="145" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggvY87FutzA1dDyHzOYYsd5NVEKJEEKFl7J0uug7dxsmwLLv76BUMOFZio1RCuxaWGN67Wnfv0HLIy-k3Z8WGRpwV-WlVSj_K_5oScxqhvkfUKyKMyxvWCG3LQzznElXS5-8n4BHbbyy_VjgZWwjcpWE6KGrqBQ3antZBPkvlX4f8SkFa4zT3XoUcaVA/w218-h145/757.JPG" width="218" /><img border="0" data-original-height="3016" data-original-width="4528" height="158" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7GeD2iVlxV5U4aMRUomZRG3vP9cfqmOj8GFwpCVhGjdsBuSXXO8B3EVfmIfRikHZPkST4P2TT_JzgTdSjuo4xSENDhQHkdCi0zNpx2_2P6PZgxxVKsTMDkRMG-OT-BdwfP0ZZmjjTJOAZ0leIelu-ge-R9P3L_oEd2E__HoglqxI_qxKo4Ic4Do-m5A/w238-h158/237.JPG" width="238" /><img border="0" data-original-height="3265" data-original-width="4898" height="154" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7RyImYjSOQiPdlgIRUx9tn1mnw_b7ntDWQ1cntUjs4INbsdSfG0PNUqIdma8lX-f1I4CDNeAlbry1nz1h07URUH_N2nIA6hQoLwj4B9yuc396HgvLjkQdOSb9w5wTMkO2qnhqyjh3ex0KTH2ZvRw_JVNZbPjt7k6-EYJGOAPtbt__Dd1YqOoKdiWEkA/w231-h154/IMG_2162%20(1).JPG" width="231" /><img border="0" data-original-height="3265" data-original-width="4898" height="167" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJ3qFYuqFq7XIMUS_X4k_8gwf568GnxTPss5zDVvQHp431nVayGDY9cV0mnSxb9v1edpG1JkE9Cyo6taK-iA5QXkYWi3dVixXdUycVj9b2OKk-sjQr_876cjHc4FzzFhQqkEyrPxl3L6ztEKqmqPG3Wo2RpI60sJpZWTUaA0GOYiLYFXX3ah8LvV_9JQ/w250-h167/IMG_2149%20(1).JPG" width="250" /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfntnj4Xqdk984of-XYO-QrPjrjktjNx26bGWPTKsZYGXGe434gv1z-C7zD0PKMIjKUB4LMpCKRyR5Zur-4CX-zdBBi_i-apkTWQ86HEX_mBjjdmD-LUztKumMUiKI3-wpeE4iAMbumDJrKCdt3m2CmMf2kmwWpqBvWIElMYJWw2Gd2hIXcPD9Mla-Xg/s4032/20220915_131722.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2268" data-original-width="4032" height="149" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfntnj4Xqdk984of-XYO-QrPjrjktjNx26bGWPTKsZYGXGe434gv1z-C7zD0PKMIjKUB4LMpCKRyR5Zur-4CX-zdBBi_i-apkTWQ86HEX_mBjjdmD-LUztKumMUiKI3-wpeE4iAMbumDJrKCdt3m2CmMf2kmwWpqBvWIElMYJWw2Gd2hIXcPD9Mla-Xg/w266-h149/20220915_131722.jpg" width="266" /></a><div class="separator" style="clear: both; text-align: center;"><img border="0" data-original-height="3016" data-original-width="4528" height="170" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1kLGox9xY-7GbGI3b8Bb-QvjM3C2z9IFQBregW-KmtHWreP5GM49em0lnRMkdE6LQDxXbrZfNcmvSFotKPucQPv-xQY8L2Tg-EiEoQZZqScRQws2IDEjEfqzpGocP5BJMgwNNbTgUJvA8FH8wVTc3kojRQrD2hhfFcqyG7NUmL-BAGFT81nDgYvB7XQ/w255-h170/669.JPG" width="255" /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhluDwFjK_LKVoAwIqi4VmW-qqsiE18MV1sWgBMe61QLtGaCAKnvex_hX3gulhgMO2QIW4koadwvCwRZNS_Wa4gyEva9MmF_e69D1GSTW57Rgm66ZmaVJfVPwMiG4igI-Ce6W80ByF-TevIbkNOxzXQXqyVcIj6YYwgvvcM8ey8adk_0QLNpyRI1tWuzg/s4528/428.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3016" data-original-width="4528" height="170" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhluDwFjK_LKVoAwIqi4VmW-qqsiE18MV1sWgBMe61QLtGaCAKnvex_hX3gulhgMO2QIW4koadwvCwRZNS_Wa4gyEva9MmF_e69D1GSTW57Rgm66ZmaVJfVPwMiG4igI-Ce6W80ByF-TevIbkNOxzXQXqyVcIj6YYwgvvcM8ey8adk_0QLNpyRI1tWuzg/w255-h170/428.JPG" width="255" /></a></div><br /><br /><br /></div><p></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-83720940729958770002022-09-24T10:49:00.006-07:002022-09-24T10:49:46.512-07:00Inverted Index for full-text searches or common words detection<p><span style="font-size: medium;"> </span><span style="font-size: large;">Sometimes there are properties in the document with unstructured text, like newspaper articles, blog posts, or book abstracts. The inverted index is easy to build and is similar to the data structures search engines use. </span></p><div><div class="separator" style="clear: both; text-align: left;"><div class="separator" style="clear: both;"><span style="font-size: large;">Such document structures can help in various complex search patterns, like common word detection, full-text searches, or document similarity searches, using humming distance or l2distance algorithms. Inverted indexes are useful when the number of keywords is not too large and when the existing data is either totally immutable or rarely changed, but frequently searched.</span></div><div class="separator" style="clear: both;"><span style="font-size: medium;"><br /></span></div><div class="separator" style="clear: both;"><span style="font-size: large;">Usually, the documents are "parents," and the words inside the document are "children." To build an inverted index, we invert this relation to make the words "parents" and documents "children":</span></div><div class="separator" style="clear: both;"><ul style="text-align: left;"><li><span style="font-size: medium;">Take all or a subset of keywords from the document and pair it with the document ID</span></li><ul><li><span style="font-size: medium;">DocId1: keyword1</span></li><li><span style="font-size: medium;">DocId1: keyword2</span></li><li><span style="font-size: medium;">DocId1: keyword3</span></li><li><span style="font-size: medium;">DocId2: keyword</span></li><li><span style="font-size: medium;">DocId2: keyword1</span></li></ul><li><span style="font-size: medium;">Revert the order by taking all unique keywords and making a list of documents where those keywords appear.</span></li></ul></div><div class="separator" style="clear: both;"><br /></div></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPIDRoeOoajk-INypNGzdJ0fHcO3CVgapI4RWjH65jfim0eXAOzqW_Tj7fmmC_mBgApdS2T5mhY8i1ZW660wtfiVw8LsDAYtboAHlmh0DjL6--uLGPzIQss3laIkWIcfWZEawnN8CHT7jMoVrZGP7qYV1mZ35UJq0mJiwtlAUiSdFWUcidgj6KcJ5jNg/s788/Capture.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="541" data-original-width="788" height="440" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPIDRoeOoajk-INypNGzdJ0fHcO3CVgapI4RWjH65jfim0eXAOzqW_Tj7fmmC_mBgApdS2T5mhY8i1ZW660wtfiVw8LsDAYtboAHlmh0DjL6--uLGPzIQss3laIkWIcfWZEawnN8CHT7jMoVrZGP7qYV1mZ35UJq0mJiwtlAUiSdFWUcidgj6KcJ5jNg/w640-h440/Capture.jpg" width="640" /></a></div><br /><span style="font-size: medium;"><div>Read more on the Inverted index and other data modeling structures in my blog <a href="https://www.mssqltips.com/sqlservertip/7357/nosql-data-models-embedded-composite-key-inverted-index-single-table/" target="_blank">here</a>.</div><div><br /></div><div>Yours,</div><div>Maria</div></span><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><br /></span></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-5909113401996098872022-09-16T10:39:00.001-07:002022-09-24T10:41:44.786-07:00Data Orchestration, Ingestion, and Data preparation in Azure cloud – which tool should you choose?<p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-9eh8o" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"></p><div style="text-align: left;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="font-size: medium;"><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><br /></span><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHo1IXfs6ZGH4_s0IwmJ0kTkRFAPwLSJkZ9B3_itRHxIsj9myfzlu0CXsvCCz8Snj0YQrR-bwj_ksa8kWz7bpbTJO-NXd1-e8AsUhmvioS7yDR-nPs2nG6wqe5525r2js5FWzkPrMzbyesWuv1Tz5ESb75rCUKiv9B34F4i-gB0rIeLWNR-6olV0o_jQ/s1000/2068860.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1000" data-original-width="1000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHo1IXfs6ZGH4_s0IwmJ0kTkRFAPwLSJkZ9B3_itRHxIsj9myfzlu0CXsvCCz8Snj0YQrR-bwj_ksa8kWz7bpbTJO-NXd1-e8AsUhmvioS7yDR-nPs2nG6wqe5525r2js5FWzkPrMzbyesWuv1Tz5ESb75rCUKiv9B34F4i-gB0rIeLWNR-6olV0o_jQ/s320/2068860.jpg" width="320" /></a></div><br /></span></span><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="font-size: medium;">There is a lot of tooling around data enrichment and data orchestration in the Azure cloud and many services with similar features. Azure Data Factory, Azure Databricks, Azure Synapse Pipelines, and SSIS services can move data from one data store to another, clean the data, enrich the data by merging entities, and perform data aggregations. They can transform the data from one format into another and partition it according to the business logic. </span></span></span><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="font-size: medium;"><br role="presentation" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></span></span><span style="font-size: medium;"><span style="font-family: inherit; font-style: inherit; font-variant-caps: inherit; font-variant-ligatures: inherit; font-weight: inherit; white-space: pre-wrap;">Why would we choose one service over another? Read my new blog post to find out: </span><a class="_3Bkfb _1lsz7" data-hook="linkViewer" href="https://www.mssqltips.com/sqlservertip/7380/azure-cloud-data-processing-strategy-and-tools-selection/" rel="noopener noreferrer" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; cursor: pointer; font-family: var(--ricos-custom-link-font-family,unset); font-style: var(--ricos-custom-link-font-style,unset); font-variant-caps: inherit; font-variant-ligatures: inherit; font-weight: var(--ricos-custom-link-font-weight,unset); line-height: var(--ricos-custom-link-line-height,unset); margin: 0px; min-height: var(--ricos-custom-link-min-height,unset); outline: 0px; padding: 0px; text-decoration-line: none; vertical-align: baseline; white-space: pre-wrap;" target="_blank"><u class="_3zM-5" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">https://www.mssqltips.com/sqlservertip/7380/azure-cloud-data-processing-strategy-and-tools-selection/</u></a><span style="font-family: inherit; font-style: inherit; font-variant-caps: inherit; font-variant-ligatures: inherit; font-weight: inherit; white-space: pre-wrap;"> </span> </span></div><p></p><div data-hook="rcv-block5" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div data-hook="rcv-block6" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-15678373544990702622022-05-27T10:50:00.001-07:002022-09-24T11:01:02.130-07:00If you want to torture data - store it in CSV format<p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-83mri" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"></span></span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXrnvP8DcFtjffVnKrtcHe8o48tZWe6D-HEIg2uYniIGVGUuHwajeKleEUT7EUIB74SUshDWSQx70eh9fgWw3Sh2t2oHkVny5STOaMVwHyaBkdvP7NoZxsC8XLhEnIYmho4dVO38-sPHkEmNiHeJpuxmhRYumDj0E9cSasSqcmRxjLjUB-9WT7aN-Ong/s598/%D1%8C%D1%83.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="575" data-original-width="598" height="308" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXrnvP8DcFtjffVnKrtcHe8o48tZWe6D-HEIg2uYniIGVGUuHwajeKleEUT7EUIB74SUshDWSQx70eh9fgWw3Sh2t2oHkVny5STOaMVwHyaBkdvP7NoZxsC8XLhEnIYmho4dVO38-sPHkEmNiHeJpuxmhRYumDj0E9cSasSqcmRxjLjUB-9WT7aN-Ong/s320/%D1%8C%D1%83.jpg" width="320" /></a></div><br />Are you using CSV files as a primary file format for your data? <p></p><div data-hook="rcv-block3" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-a1ffp" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">CSV is a very useful file format if you want to open the file in Excel and analyze it right away.</span></span></p><div data-hook="rcv-block4" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-2ppsj" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">CSV format stores tabular data in plain text, it is old and was wildly used in the early days of business computing. </span></span></p><div data-hook="rcv-block5" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-6cf7j" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">However, if you plan to keep raw data in the data lake, you should reconsider using CSV. There are many modern file formats that were designed for data analysis.</span></span></p><div data-hook="rcv-block6" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-gvd4" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">In the cloud world of data lakes and schema-on-read querying systems, like AWS Glue or Databricks, CSV files will slow you down. </span></span></p><div data-hook="rcv-block7" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-5ahvm" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><br role="presentation" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></span></div><div data-hook="rcv-block8" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-cs95b" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Today I want to talk about Parquet, a modern file format, invented for fast analytical querying.</span></span></p><div data-hook="rcv-block9" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-dccoj" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><br role="presentation" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></span></div><div data-hook="rcv-block10" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-ake3g" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Parquet files organize data in columns, while CSV files organize data in rows. </span></span></p><div data-hook="rcv-block11" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-2lve8" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Columnar storage allows much better compression so Parquet data files need less storage, 1 TB of CSV files can be converted into 100GB of parquet files – which can be a huge money saver when cloud storage is used. This also means that scanning parquet file is much faster than scanning CSV files – fewer data would be scanned and there is no need to load unneeded columns into memory and aggregations will run faster. Parquet files contain both data and metadata, information about data schema and structure. When you load the file, having metadata helps the querying tool define proper data types.</span></span></p><div data-hook="rcv-block12" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-bevug" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><br role="presentation" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></span></div><div data-hook="rcv-block13" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-5rp5" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">To test both formats' performance, I will use January 2022 New York Yellow taxi dataset available in both CSV and Parquet online:</span></span></p><div data-hook="rcv-block14" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-8q205" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><a class="_3Bkfb _1lsz7" data-hook="linkViewer" href="https://nyc-tlc.s3.amazonaws.com/trip+data/yellow_tripdata_2022-01.parquet" rel="noopener noreferrer" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; cursor: pointer; font-family: var(--ricos-custom-link-font-family,unset); font-size: var(--ricos-custom-link-font-size,unset); font-style: var(--ricos-custom-link-font-style,unset); font-weight: var(--ricos-custom-link-font-weight,unset); line-height: var(--ricos-custom-link-line-height,unset); margin: 0px; min-height: var(--ricos-custom-link-min-height,unset); outline: 0px; padding: 0px; text-decoration-line: none; vertical-align: baseline;" target="_blank"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; color: #1155cc; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><u class="_3zM-5" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">https://nyc-tlc.s3.amazonaws.com/trip+data/yellow_tripdata_2022-01.parquet</u></span></a></span></span></p><div data-hook="rcv-block15" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-65fn1" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><a class="_3Bkfb _1lsz7" data-hook="linkViewer" href="https://s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2022-01.csv" rel="noopener noreferrer" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; cursor: pointer; font-family: var(--ricos-custom-link-font-family,unset); font-size: var(--ricos-custom-link-font-size,unset); font-style: var(--ricos-custom-link-font-style,unset); font-weight: var(--ricos-custom-link-font-weight,unset); line-height: var(--ricos-custom-link-line-height,unset); margin: 0px; min-height: var(--ricos-custom-link-min-height,unset); outline: 0px; padding: 0px; text-decoration-line: none; vertical-align: baseline;" target="_blank"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; color: #1155cc; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><u class="_3zM-5" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">https://s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2022-01.csv</u></span></a></span></span></p><div data-hook="rcv-block16" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-7h4l1" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Both files have 2463931 rows. However, the CSV file is 239MB, and the Parquet file is only 36MB.</span></span></p><div data-hook="rcv-block17" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-31pj8" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><br role="presentation" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></span></div><div data-hook="rcv-block18" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-7a5ki" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">I will be using the Azure Databricks Spark cluster to analyze the dataset. I want to see the total number of passengers that took a yellow taxi between the 20th and 26th of January. I will run the script twice, first using the parquet file and then using the CSV file.</span></span><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"></span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-5YbY9K4VdvtzlNYQxc-P4HXqb5U8bLQVzgNniWO2T-69HymtM0fVrUzBl-k_K6TEKmGZ_zialSPAN56o5oVUWlt350PMmp56TqbYUcL6todhAE_9O5-x9h6j8AZznmmju6Hq4tpnbZ9TcLY_0-uFnG3DMmsFBdOXBIu1am6LKlN6Yiikyvct5PaQtA/s736/Capture.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="458" data-original-width="736" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-5YbY9K4VdvtzlNYQxc-P4HXqb5U8bLQVzgNniWO2T-69HymtM0fVrUzBl-k_K6TEKmGZ_zialSPAN56o5oVUWlt350PMmp56TqbYUcL6todhAE_9O5-x9h6j8AZznmmju6Hq4tpnbZ9TcLY_0-uFnG3DMmsFBdOXBIu1am6LKlN6Yiikyvct5PaQtA/s16000/Capture.jpg" /></a></div><br /><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><br /></span><p></p><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-4i029" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Querying parquet took 5.19 sec</span></span></p><div data-hook="rcv-block26" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-8cf1i" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Querying CSV took 2.28 min</span></span></p><div data-hook="rcv-block27" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-bbap9" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">As you see, querying CSV files is slower, requires more IO, and will take much more space on your cloud storage. </span></span></p><div data-hook="rcv-block28" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-dc42n" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><br role="presentation" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></span></div><div data-hook="rcv-block29" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-15sbd" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Yours,</span></span></p><div data-hook="rcv-block30" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-9hbui" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Maria</span></span></p><div data-hook="rcv-block31" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-apj2s" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><br role="presentation" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></span></div><div data-hook="rcv-block32" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: "open sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"></span></span></p><p class="mm8Nw _1j-51 roLFQS _1FoOD _3M0Fe Z63qyL roLFQS public-DraftStyleDefault-block-depth0 fixed-tab-size public-DraftStyleDefault-text-ltr" id="viewer-79v28" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: "open sans", sans-serif; font-size: var(--ricos-custom-p-font-size,unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.38; margin: 0px; min-height: var(--ricos-custom-p-min-height,unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="_2PHJq public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background-attachment: initial; background-clip: initial; background-color: var(--ricos-custom-p-background-color,unset); background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: 0px; box-sizing: inherit; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">"If you torture the data long enough, it will confess."</span></span></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-80220671327372789882021-07-12T07:28:00.009-07:002021-07-12T07:28:49.047-07:00Embrace Delta Lakes and reduce the SQL Server compute resources contention<p><span style="font-family: inherit; font-size: inherit; font-style: inherit; font-variant-caps: inherit; font-variant-ligatures: inherit; font-weight: inherit; white-space: pre-wrap;"></span></p><div class="separator" style="clear: both; text-align: left;"><a href="https://lh3.googleusercontent.com/-dxk_Kzresqo/YOxRSUR6A4I/AAAAAAABvvA/HYYvxq5w40s4mBwuLhBTlh5WNEQNNwWjgCLcBGAsYHQ/image.png" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="332" data-original-width="531" height="250" src="https://lh3.googleusercontent.com/-dxk_Kzresqo/YOxRSUR6A4I/AAAAAAABvvA/HYYvxq5w40s4mBwuLhBTlh5WNEQNNwWjgCLcBGAsYHQ/w400-h250/image.png" width="400" /></a></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: inherit; font-size: inherit; font-style: inherit; font-variant-caps: inherit; font-variant-ligatures: inherit; font-weight: inherit;"><br /></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: inherit; font-size: inherit; font-style: inherit; font-variant-caps: inherit; font-variant-ligatures: inherit; font-weight: inherit;">Data management tools are evolving at a great speed, and this can be overwhelming.</span></div><p></p><div data-hook="rcv-block3" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-bo6n4" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;"><span style="background-color: transparent; font-family: inherit; font-size: inherit; font-style: inherit; font-variant-caps: inherit; font-variant-ligatures: inherit; font-weight: inherit;">Data volumes and variety evolve and grow as well.</span></span></p><div data-hook="rcv-block5" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-4cnnl" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;"><br style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></div><div data-hook="rcv-block6" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-f9be2" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">Data Engineers are required to transform those waterfalls of data into business insights.</span></p><div data-hook="rcv-block7" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-2cuvp" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;"> </span></p><div data-hook="rcv-block8" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-6854p" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">The data is arriving from a vast range of sources, like social-media networks, 3rd party partners or internal micro services. If you are experienced SQL Server DBA, you know how versatile the product is. It is very tempting and feels correct to use the tool that you know the best.</span></p><div data-hook="rcv-block9" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-4ea7v" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">We can use SQL Server for almost any data management task. We can use SQL Server to watch over the storage for a new unprocessed files. </span></p><div data-hook="rcv-block10" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-9i36" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">We can load the raw data into SQL Server Staging Area SQL Server database. </span></p><div data-hook="rcv-block11" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-fj3md" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">We can efficiently clean, enrich and aggregate the data using highly expensive relational database resources (even if you are not using SQL Database in the cloud, every Enterprise edition core still cost about $7K)</span></p><div data-hook="rcv-block12" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-241i1" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">SQL Server Relational engine high cost echoes the product complexity and extraordinary features that help to manage data consistency and integrity under high transactional load. Those expensive resources should be mainly dedicated to OLTP workloads as well as to condensed, well aggregated and focused Data Marts.</span></p><div data-hook="rcv-block13" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-2gji9" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;"><br style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></div><div data-hook="rcv-block14" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-b0m75" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">I want to suggest adding to your DBA toolbelt an emerging class of data management technologies called <span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 700; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Data Lakes</span>. </span></p><div data-hook="rcv-block15" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-7qoah" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">Data Lakes are designed to hold, process and store data and they are very cost effective. Data Lakes usually use affordable hardware and data sets do not need to be indexed. Data can be stored either in raw format (Data scientists in many cases will prefer to analyze an original data) as well as in cleaned, enriched, and aggregated formats. </span></p><div data-hook="rcv-block16" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-dsmbe" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">Aggregated data can be loaded into your Data Warehouse, which can be our beloved SQL Server, serving Data Analysts queries and reports. Aggregated data sets are smaller and you will not need to dramatically increase the size of data warehouse every few month.</span></p><div data-hook="rcv-block17" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-1hdja" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;"><br style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></div><div data-hook="rcv-block18" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-23cgv" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">Delta Lake <a class="_2qJYG _2E8wo" href="https://delta.io/" rel="noopener noreferrer" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; cursor: pointer; font-family: var(--ricos-custom-link-font-family, unset); font-size: var(--ricos-custom-link-font-size, unset); font-stretch: inherit; font-style: var(--ricos-custom-link-font-style, unset); font-variant: inherit; font-weight: var(--ricos-custom-link-font-weight, unset); line-height: var(--ricos-custom-link-line-height, unset); margin: 0px; min-height: var(--ricos-custom-link-min-height, unset); outline: 0px; padding: 0px; text-decoration-line: none; vertical-align: baseline;" target="_blank">https://delta.io/</a> is an open-source project that enables building data warehouses on a top of storage systems, <span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 700; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">it's a</span> <span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 700; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">storage</span> <span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 700; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">layer </span>from Spark which runs on top of an <span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 700; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Azure Data Lake, </span>ensuring data integrity with ACID transactions. Delta lake technology leverages efficient Apache Parquet files compression, allows to specify and enforce schema to make sure datatypes are correct and all required columns are present. Delta Lake enables table schema changes, data versioning and full audit trail of the changes.</span></p><div data-hook="rcv-block19" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-co8h8" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;"><br style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></div><div data-hook="rcv-block20" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-nudn" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">In Azure cloud, we have the following services that integrate with Delta Lake </span></p><div data-hook="rcv-block21" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-4nmko" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;"><br style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></div><div data-hook="rcv-block22" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-cj8hh" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">1. <span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 700; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Azure Databricks</span> – has integrated Delta Lake into their managed services, it allows you to use SQL, Python, R or Scala to query the Delta Lake</span></p><div data-hook="rcv-block23" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-dckoo" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">2. <span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 700; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Azure Synapse Analytics</span> – serverless data pools utilize Spark to read and write data from your Delta Lake in PySpark, .NET and Scala</span></p><div data-hook="rcv-block24" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-5sqvu" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">3. <span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 700; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Azure Data Factory</span> – supports Delta Lake connector in Copy activity, in Data Flow and Databricks Notebook activities </span></p><div data-hook="rcv-block25" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-f2hgi" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">4. <span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 700; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Azure HDInsight</span>: in Spark and Hive you can download and install Delta Lake.</span></p><div data-hook="rcv-block26" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-7phh7" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">5. <span style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 700; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Power BI</span>: you can query Delta Lake tables directly.</span></p><div data-hook="rcv-block27" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-63rhp" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;"> </span></p><div data-hook="rcv-block28" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-8291t" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">Do not fall back on known methods, take advantage of the low storage and low processing costs.</span></p><div data-hook="rcv-block29" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-1nb9t" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">Make your company data agile and allow your users to access original data immediately as soon as it arrives to Data Lake, way before your aggregations are ready.</span></p><div data-hook="rcv-block30" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-5pj5d" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">Keep your Sql Server compute resources dedicated to end user queries.</span></p><div data-hook="rcv-block31" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-6qe7o" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;"><br style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></div><div data-hook="rcv-block32" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-cdbvn" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;"><em style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; font-family: inherit; font-size: inherit; font-stretch: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">As Jeff Bezos says, what is dangerous is not to evolve.</em></span></p><div data-hook="rcv-block33" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><div class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-5kps5" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;"><br style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); box-sizing: inherit;" /></span></div><div data-hook="rcv-block34" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="empty-line"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-bqfhm" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">Yours</span></p><div data-hook="rcv-block35" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; font-family: HelveticaNeue, Helvetica, Arial; font-size: 16px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;" type="paragraph"></div><p class="XzvDs _208Ie _1atvN _2QAo- _25MYV _2WrB- _1atvN public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr" id="viewer-2i7or" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; direction: ltr; font-family: var(--ricos-custom-p-font-family, unset); font-size: var(--ricos-custom-p-font-size, unset); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.5; margin: 0px; min-height: var(--ricos-custom-p-min-height, unset); outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap;"><span class="vkIF2 public-DraftStyleDefault-ltr" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0); background: transparent; border: 0px; box-sizing: inherit; direction: ltr; display: block; font: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; z-index: 10 !important;">Maria</span></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-69593178567699085372019-08-15T01:20:00.001-07:002022-09-24T06:54:37.432-07:00SQL Server Insert Parent and Child Records with One Statement<br /><br /><br />
<br />
<span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwEw2sEiAbvA3EBkeN7DFIcA1m5lclFZBj4QUzZvisDlKuQuvjtybKHDDFfDM_JqeHuYWjL8GK0UE0l8kx_IVHj6sFbSHptkyR6n_F1OcuzvWYiQ8W78kuU0p7YRzHyaHyIG1mso2f7MKdBRC0dNeRS3z6RCeXspCCoEjJAKoa76fT1ZJR4sJHNYm17A/s720/child-holding-hands-silhouette-24.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="720" data-original-width="427" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwEw2sEiAbvA3EBkeN7DFIcA1m5lclFZBj4QUzZvisDlKuQuvjtybKHDDFfDM_JqeHuYWjL8GK0UE0l8kx_IVHj6sFbSHptkyR6n_F1OcuzvWYiQ8W78kuU0p7YRzHyaHyIG1mso2f7MKdBRC0dNeRS3z6RCeXspCCoEjJAKoa76fT1ZJR4sJHNYm17A/s320/child-holding-hands-silhouette-24.png" width="190" /></a></div>A few days ago, one of the developers asked me if that was possible to generate test data by performing multiple nested </span><a href="https://www.mssqltips.com/sqlservertutorial/2514/sql-server-insert-command-tutorial/" style="background-color: white; box-sizing: inherit; color: #008cba; font-family: "helvetica neue", Helvetica, Roboto, Arial, sans-serif; font-size: 16px; line-height: inherit; text-decoration-line: none;">INSERT statements</a><span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;">, each of them involving inserting new rows into several parent tables and</span><span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; box-sizing: inherit; color: #222222; font-size: 16px; font-weight: 700; line-height: inherit;"> in the same statement</span><span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;"> reusing the autogenerated primary keys for the foreign key columns in the child table. </span><br />
<span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;"><br /></span>
<span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;">The developer was working with PostgreSql and so I tried to find a solution for both PostgreSql and SQL Server to learn the differences in the ANSI features implementation in these different database systems.</span><br />
<span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;"><br /></span>
<span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;">Read the solution on my blogpost here: </span><span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="color: #222222;"><a href="https://www.mssqltips.com/sqlservertip/6142/sql-server-insert-parent-and-child-records-with-one-statement/">https://www.mssqltips.com/sqlservertip/6142/sql-server-insert-parent-and-child-records-with-one-statement/</a></span><br />
<br />
<span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;"><br /></span>
<span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;">Yours,</span><br />
<span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;">Maria</span><br />
<span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;">P.S. </span><span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;">(Picture is taken from Kendra Little website)</span><span face=""helvetica neue" , "helvetica" , "roboto" , "arial" , sans-serif" style="background-color: white; color: #222222; font-size: 16px;"> </span><br />
<br /><br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8354214079358533272.post-49288016670298975452019-08-05T04:19:00.000-07:002019-08-05T06:54:25.981-07:00RIP SQLblog.com<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://rlv.zcache.co.nz/loud_crying_boy_funny_cartoon_drawing_tears_mouse_pad-radcc3ef8418f44f4863a1504d9b55870_x74vi_8byvr_307.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" height="200" src="https://rlv.zcache.co.nz/loud_crying_boy_funny_cartoon_drawing_tears_mouse_pad-radcc3ef8418f44f4863a1504d9b55870_x74vi_8byvr_307.jpg" title="" width="200" /></a></div>
DBbest did their best but failed to maintain great knowledge sharing place where brightest SQL used to exchange new ideas and shared their wisdom.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-3SSI_gYM6iE/XUgwJsNuZjI/AAAAAAABVlI/L404m7vxOccBXjEhIWk-0WOXpx55aU-fgCLcBGAs/s1600/Screen%2BShot%2B2019-08-05%2Bat%2B16.32.58.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="195" data-original-width="806" height="96" src="https://1.bp.blogspot.com/-3SSI_gYM6iE/XUgwJsNuZjI/AAAAAAABVlI/L404m7vxOccBXjEhIWk-0WOXpx55aU-fgCLcBGAs/s400/Screen%2BShot%2B2019-08-05%2Bat%2B16.32.58.png" width="400" /></a></div>
<br />
I want to thank Adam Machanic for giving me the opportunity to host my blog over there and be visible to the huge number of SQLblog readers.<br />
<br />
I will move my favourite articles over here if I succeed to find them.<br />
<br />
Yours,<br />
MariaUnknownnoreply@blogger.com0