Serverless ETL: Read, Enrich and Transform Data with AWS Glue Service

-


More and more companies are aiming to move away from managing their own servers and moving towards a cloud platform. Going server-less, offers a lot of benefits like lower administrative overhead and server costs. In the server-less architecture, developers work with event driven functions which are being managed by cloud services. Such architecture is highly scalable and boosts developer productivity.
AWS Glue service is an ETL service that utilizes a fully managed Apache Spark environment. Glue ETL that can clean, enrich your data and load it to common database engines inside AWS cloud (EC2 instances or Relational Database Service) or put the file to S3 storage in a great variety of formats, including PARQUET.
I have recently published 3 blogposts on how to use AWS Glue service when you want to load data into SQL Server hosted on AWS cloud platform.

2. Join and Import JSON files from s3 to SQL Server RDS instance Part 1 and Part 2

Are you using AWS Glue? What do you think about serverless ETL ? 
Yours
Maria

Comments

Post a Comment

Popular posts from this blog

Unlocking Microsoft Fabric: A Simple Guide when you only have a personal account.

-
Image
" You cannot create a Microsoft capacity using a personal account. Use your organizational account instead." What a disappointment and frustration! I've been trying to set up Microsoft Fabric for a while now and figure out how to work around this error. I use my personal Gmail for an MVP subscription to learn and experiment with Azure services. When I visit app.fabric.microsoft.com and try to use my private Gmail address, I keep getting a similar message: It's frustrating because I'm eager to dive into Fabric and explore its capabilities. I would get the same error trying to set up Azure Data Catalog, PowerBI embedded, and more. What are the different account types and how do they differ? A work or school account is created through Active Directory or other cloud directory services, such as Microsoft 365. On the other hand, a personal account is one that's manually created for individual use, consisting simply of a username and password. After digging into ...
Read more

ETL to ELT journey: Break free your Transformations and discover Happiness and Data Zen

-
Every data integration pipeline consists of 3 stages: Data Extraction (E), Data Transformation (T) and Data Load (L) During the  Data Extraction stage, the source data is read from its origins: transactional databases, CRM or ERP systems or through data scraping from web pages. During the  Data Transformation stage, the necessary modifications are applied to the source data. This includes data filtering, enrichment or merging with existing or other source datasets, data obfuscation, dataset structure alignment or validation, fields renaming and data structuring, according to the canonical data warehouse model.  During the Data Loading stage, the data is stored in the pipeline destination, which could be a staging area, data lake or data warehouse. There are two principal methods for the data integration process: transferring it from where it originated to the destination, where the data will be used for analysis, ETL and ELT. The difference between ETL and ELT pipelines...
Read more

The backbone your data pipelines have been waiting for.

-
Image
Kafka isn’t just a buzzword—it’s the backbone your data pipelines have been waiting for. Who new that messaging system, like Apache Kafka should hold a central place in a Data Engineer toolbelt.  Apache Kafka is a low-latency distributed data streaming platform for real-time data processing. Kafka can handle large volumes of data and is very helpful for distributed data integration projects. Top 2 reasons why you might need Kafka in your Data Integration architecture 1. Support multiple destinations by decoupling data producers and data consumers. Data in source will be processed only once, which lowers an overall cost in consumption-based data producer databases and we can add new/change existing destinations without changing the extraction components. 2. Ability to deal with massive amounts of data , supports high throughput and scalability. Decoupling pipeline extract and load stages is an important Data Integration principle and can improve pipeline flexibility, extract and loa...
Read more