Azure Data Factory +Synapse Analytics End to End ETL project

Complete end to end Project on OTT platform using Azure Data Factory and Azure Synapse Analytics from Scratch with Labs

4.36 (154 reviews)

Udemy

platform

English

language

Other

What you will learn

Create incremental pipeline based on last modified date

Calling Azure Synapse notebook from Azure Data Factory

Configure CICD from Scratch in Azure Data Factory

Build end to end project using Azure Data Engineering Services

Create a Self Hosted Integration Runtime and ingest data from On-premise Environment

How to transform data with Azure Synapse Analytics

Create and configure an Azure Data Lake

Project Oriented Learning of Azure Data Engineering services

Create an Azure Key vault service

Using Secrets of Key vault in Linked services

Create and configure an Azure SQL Database

Writing PySpark transformation logic in Synapse Notebook

Use of Triggers in Azure Data Factory

Create an incremental data load pipeline to ingest file in daily basis

Build an Automated pipeline in Azure Data Factory

Setup CICD environment from Scratch

Create incremental pipeline based on file name and pick today's file

Perform Data Analysis Using PySpark Code using Synapse Notebook

Create and configure an Azure Data Factory

Create and configure an Azure Synapse Analytics

Orchestrate all pipelines in Azure Data Factory

Loading Data into Azure SQL Database

Ingest Data from On-premise to Azure Datalake in Azure Cloud

Send an Automatic Alert email when a pipeline is failed

Transform only Today's file in Azure Synapse Analytics

Report transformed data in Power BI

Complete hands-on project using Azure Data Engineering Services

Provides insights on services that needed to clear DP-203

Understand how GetMetaData, ForEach, If Condition Activities works

Why take this course?

Are you looking to build an end-to-end ETL project using Azure data engineering services? Look no further than this comprehensive course on Azure Data Factory and Azure Synapse Analytics. This course is designed to guide you through the process of creating all the necessary services from scratch, before building an ETL project from start to finish.

Throughout this course, you'll gain practical, hands-on experience with Azure Data Factory and Azure Synapse Analytics, learning how to use these powerful data engineering tools to create a highly effective ETL solution. You'll explore the many features and capabilities of these platforms, as well as their integration with other Azure services like

1. Azure SQL Database

2. Azure Synapse Analytics

3. Azure Key Vault

4. Azure Data Factory for Orchestration,

5. Azure Storage solutions (Azure Datalake Gen2)

6. Microsoft Power BI

7. Azure Logic Apps

In addition to the core ETL project, this course also includes an additional section on CICD (Continuous Integration and Continuous Deployment) on Azure Data Factory, helping you to fully automate your data engineering workflow.

Whether you're a beginner or an experienced data engineer, this course is designed to help you gain a comprehensive understanding of Azure Data Factory and Azure Synapse Analytics. By the end of the course, you'll be able to confidently create and manage your own ETL projects on the Azure cloud.

Enrol now and take your data engineering skills to the next level!

Screenshots

Azure Data Factory +Synapse Analytics End to End ETL project - Screenshot_01

Azure Data Factory +Synapse Analytics End to End ETL project - Screenshot_02

Azure Data Factory +Synapse Analytics End to End ETL project - Screenshot_03

Azure Data Factory +Synapse Analytics End to End ETL project - Screenshot_04

Reviews

Jorge

September 6, 2023

Es buena elección, buen contenido, sí es útil para iniciar rápido con necesidades básicas entre los componentes involucrados en el curso

August 8, 2023

One of the top three courses on this subject on Udemy - hands down! Learned a lot in my journey to become a data engineer. 5 stars PLUS!

Shankar

June 27, 2023

One of the best practical course I've seen. Learnt many new things, please consider making a course on Python like you have in transformation section . It will be really helpful for learners. Highly recommended course to gain practical learning ! Keep up the good work!

Hassan

June 18, 2023

It was good experience of learning but there are some things that needs to be improved example if he haven't used the access keys and virtual machines to do the work for demo. Because it was giving a lot of errors to me. Its better if he had worked with local machine and also about the dataset was simple csv file from on-prem, there should be a video of using dataset from api calling or .bak files.

Surya

June 8, 2023

The video in which Azure virtual network is explained, You have created Az virtual network ,bu at the datafactory Netwrking section , you selected public end point though you created a private end point . You need to select that , public end point will anyway be successful.

Natnael

May 30, 2023

The codes and instruction you provide are full of errors. Its really hard to follow along because of all the errors gets thrown. You also need to show some of the settings you might have changed, because when I try to follow along these videos for the most part I dont have permissions to access or add anything. I struggled to learn from this course because of all the extra research I have to do for every button i press.

PRASAD

May 29, 2023

Good course. Covering E2E steps of Azure integration. Can you please share all the Pyspark scripts used in the course as a separate file?

Mike

May 16, 2023

The file share method to expose files did not work for me despite working on my own personal machine. This method did work: https://learn.microsoft.com/en-us/azure/storage/files/storage-how-to-use-files-windows#prerequisites

Henry

May 15, 2023

Great course however there was a section that I followed exactly but was getting an error. I had to find an alternative way because I didn't get my question answered this is why I can't give it a 5 star. Otherwise great course very informative.

Jesús

May 10, 2023

Really good course! Like most of students i had to deal with vnet & IP problems for ingesting data from on premise enviroment but overall is a really good course and the CI/CD section is the cherry on TOP. Totally recommended if you wanna see how a real Data Engineer END TO END project.

Chris

May 8, 2023

Most was very good but I could not connect to my on prem directories and files. There were insufficient instruction on how to handle the varied security or other networking issues that could vary. I am evaluating this course for a larger group to take (individually), and this will become a blocker.