Master Data Engineering using Azure Data Analytics

Learn Azure Storage for Data Lake, ADF for ETL, BigQuery for Data Warehouse, Databricks for Big Data Pipeline, etcs

4.58 (176 reviews)

Udemy

platform

English

language

Other

What you will learn

Data Engineering leveraging Services under Azure Data Analytics such as Azure Storage, Data Factory, Azure SQL, Synapse, Databricks, etc.

Setup Development Environment using Visual Studio Code on Windows

Building Data Lake using Azure Storage (Blob and ADLS)

Build Data Warehouse using Azure Synapse

Implement ETL Logic using ADF Data Flow with Azure Storage as Source and Target

In Depth Coverage of Orchestration using ADF Pipeline

Overview of Azure SQL and Azure Synapse Serverless and Dedicated Pool Features

Implement ETL Logic using ADF Data Flow with Azure SQL as Source and Azure Synapse as Target

Using Data Copy to copy data between different sources and targets

Performance Tuning Scenarios of ADF Data Flow and Pipelines

Build Big Data Solutions using Azure Databricks

Overview of Spark SQL and Pyspark Data Frame APIs

Build ELT Pipelines using Databricks Jobs and Workflows

Orchestrate Databricks Notebooks using ADF Pipelines

Why take this course?

Data Engineering is all about building Data Pipelines to get data from multiple sources into Data Lakes or Data Warehouses and then from Data Lakes or Data Warehouses to downstream systems. As part of this course, I will walk you through how to build Data Engineering Pipelines using Azure Data Analytics Stack. It includes services such as Azure Storage (both Blob and ADLS), ADF Data Flow, ADF Pipeline, Azure SQL, Azure Synapse, Azure Databricks, and many more.

As part of this course, first, you will go ahead and set up the environment to learn using VS Code on Windows and Mac.
Once the environment is ready, you need to sign up for Azure Portal. We will provide all the instructions to sign up for Azure Portal Account including reviewing billing as well as getting USD 200 Credit valid for up to a month.
We typically use Azure Storage as Data Lake. As part of this course, you will learn how to use Azure Storage as Data Lake along with how to manage the files in Azure Storage using tools such as Azure Storage Explorer.
ADF (Azure Data Factory) is used for both ETL as well as Orchestration. First, you will understand how to perform ETL using ADF Data Flow. The source and target will be Files in Azure Storage Account. As part of this process, you will also learn how to set up Linked Services and Data Sets in ADF (Azure Data Factory).
Once ADF Data Flow is ready, you will go ahead and build Pipeline for Orchestration using ADF Pipeline. You will also learn how to parameterize and also how to take care of baseline load.
You will also understand key performance tuning techniques using ADF Pipeline such as controlling the number of partitions, custom integration runtimes (IR), etc.
Azure provides RDBMS as different services for Postgres, SQL Server, etc. You will learn how to set up Azure SQL Once the Azure SQL is set up, you will also understand how to create required tables and run queries against them.
ADF provides ADF Data Copy to copy data from different sources and different targets. Once the Database tables are ready you will use ADF Data Copy to copy data into the tables.
Azure provides Synapse Analytics for Data Warehouse. You will get an overview of both serverless as well as dedicated pools. You will end up setting up Dedicated Pool for ETL using ADF.
Once Azure SQL and Azure Synapse are ready, you will build ETL Pipeline using ADF Data Flow and Orchestrate using ADF Pipeline.
Azure Databricks is the service for Big Data Processing using Spark Engine. You will learn how to set up Azure Databricks, integrate with ADLS, and also managing secrets.
You will also get an overview of Spark SQL and Pyspark Data Frame APIs using Azure Databricks.
You will also build ELT Pipeline using Databricks Jobs and Workflows where tasks are defined based on Pyspark as well as Spark SQL.
You will also understand how to build ADF Pipelines to orchestrate Databricks Notebooks.

Reviews

Shrey

July 8, 2023

Overall Helpful course was some good information. ADF part covered in detail, synapse is briefly touched and Databricks part felt was a lil repetitive. No mention of how to use Azure function in ADF or individually for ETL

Prashant

May 8, 2023

Teaches each and everything as per the guideline and i am happy with the course hoping to get the job soon.

Ronak

April 18, 2023

Durga is the best tutor in field of Data Engineering. He has explained each and every concept with hands-on and with simple language that even a school children can understand the concepts. I would like to recommend this course who wants to learn data engineering using Azure cloud can go with this course.

Dipankar

April 15, 2023

While the course covered a lot of useful information on Azure Data Analytics, there were a few issues that detracted from the overall experience. One of the main issues I encountered was the presence of repetitive videos. For instance, videos number 207 and 181 were the same, which felt like a waste of time. Similarly, sections 21 and 22 contained mostly the same videos, except for one covering Google Cloud. Additionally, I would have liked to see information on Databricks Delta Live Tables and Instance Pools. These are important topics that were not covered. I hope the course author takes this feedback into account and makes improvements in the future.

Ravi

March 7, 2023

This is the one of few courses in Udemy, which covered Azure Databricks, Synapse and ADF services in one course. Core content/concept explained with an example and in a understandable manner. Few topics yet to be added in the respective services.

Charts

Price

Rating

Enrollment distribution

Master Data Engineering using Azure Data Analytics - Distribution chart