Serverless ETL: Read, Enrich and Transform Data with AWS Glue Service

-


More and more companies are aiming to move away from managing their own servers and moving towards a cloud platform. Going server-less, offers a lot of benefits like lower administrative overhead and server costs. In the server-less architecture, developers work with event driven functions which are being managed by cloud services. Such architecture is highly scalable and boosts developer productivity.
AWS Glue service is an ETL service that utilizes a fully managed Apache Spark environment. Glue ETL that can clean, enrich your data and load it to common database engines inside AWS cloud (EC2 instances or Relational Database Service) or put the file to S3 storage in a great variety of formats, including PARQUET.
I have recently published 3 blogposts on how to use AWS Glue service when you want to load data into SQL Server hosted on AWS cloud platform.

2. Join and Import JSON files from s3 to SQL Server RDS instance Part 1 and Part 2

Are you using AWS Glue? What do you think about serverless ETL ? 
Yours
Maria

Comments

Post a Comment

Popular posts from this blog

Unlocking Microsoft Fabric: A Simple Guide when you only have a personal account.

-
Image
" You cannot create a Microsoft capacity using a personal account. Use your organizational account instead." What a disappointment and frustration! I've been trying to set up Microsoft Fabric for a while now and figure out how to work around this error. I use my personal Gmail for an MVP subscription to learn and experiment with Azure services. When I visit app.fabric.microsoft.com and try to use my private Gmail address, I keep getting a similar message: It's frustrating because I'm eager to dive into Fabric and explore its capabilities. I would get the same error trying to set up Azure Data Catalog, PowerBI embedded, and more. What are the different account types and how do they differ? A work or school account is created through Active Directory or other cloud directory services, such as Microsoft 365. On the other hand, a personal account is one that's manually created for individual use, consisting simply of a username and password. After digging into ...
Read more

ETL to ELT journey: Break free your Transformations and discover Happiness and Data Zen

-
Every data integration pipeline consists of 3 stages: Data Extraction (E), Data Transformation (T) and Data Load (L) During the  Data Extraction stage, the source data is read from its origins: transactional databases, CRM or ERP systems or through data scraping from web pages. During the  Data Transformation stage, the necessary modifications are applied to the source data. This includes data filtering, enrichment or merging with existing or other source datasets, data obfuscation, dataset structure alignment or validation, fields renaming and data structuring, according to the canonical data warehouse model.  During the Data Loading stage, the data is stored in the pipeline destination, which could be a staging area, data lake or data warehouse. There are two principal methods for the data integration process: transferring it from where it originated to the destination, where the data will be used for analysis, ETL and ELT. The difference between ETL and ELT pipelines...
Read more

Replication fun: setting NFR on all objects

-
If you ever tried to set up replication on the large environment with numerous servers, hundreds of databases then you, for sure, spend some time searching/writing scripts to set flag NOT FOR REPLICATION on thousands of identities, constraints and triggers. I want to share here my scripts and I hope it will save some readers hours of headache. Feel free to use them, just add comments below - there is nothing better than feedback. Here we go. ******************************************************************************** 1. Identities. The resutset is a list of identity columns on all tables in all databases where NOT FOR REPLICATION option is not set and fixing script: DROP TABLE #IdentityNFR CREATE TABLE #IdentityNFR (ServerName varchar(250), Dbname varchar(250),TableName varchar(250), IdentityColName varchar(250), IsNFR varchar(10),FixingScript varchar(4000)) exec sp_MSforeachdb ' IF ''?'' NOT IN (''tempdb'',''master'',''mo...
Read more