Apache Airflow: The Hands-On Guide

Master Apache Airflow from A to Z. Hands-on videos on Airflow with AWS, Kubernetes, Docker and more

4.52 (4109 reviews)

Udemy

platform

English

language

Software Engineering

What you will learn

Coding Production Grade Data pipelines by Mastering Airflow through Hands-on Examples

How to Follow Best Practices with Apache Airflow

How to Scale Airflow with the Local, Celery and Kubernetes Wxecutors

How to Set Up Monitoring with Elasticsearch and Grafana

How to Secure Airflow with authentication, crypto and the RBAC UI

Core and Advanced Concepts with Pros and Limitations

Mastering DAGs with timezones, unit testing, backfill and catchup

Organising the DAG folder and keep things clean

Why take this course?

Apache Airflow is a platform created by the community to programmatically author, schedule and monitor workflows.

It is scalable, dynamic, extensible, and modulable.

Without any doubt, mastering Airflow is becoming a must-have and an attractive skill for anyone working with data.

What you will learn in the course:

Fundamentals of Airflow are explained such as what Airflow is, how the scheduler and the web server work
The Forex Data Pipeline project is an incredible way to discover many operators in Airflow and deal with Slack, Spark, Hadoop, and more
Mastering your DAGs is a top priority, and you can play with timezones, unit test your DAGs, structure your DAG folder, and much more.
Scaling Airflow through different executors such as the Local Executor, the Celery Executor, and the Kubernetes Executor will be explained in detail. You will discover how to specialize your workers, add new workers, and what happens when a node crashes.
A Kubernetes cluster of 3 nodes will be set up with Rancher, Airflow, and the Kubernetes Executor local to run your data pipelines.
Advanced concepts will be shown through practical examples such as templating your DAGs, how to make your DAG dependent on another, what are Subdags and deadlocks, and more.
You will set up a Kubernetes cluster in the cloud with AWS EKS and Rancher to use Airflow and the Kubernetes Executor.
Monitoring Airflow is extremely important! That's why you will know how to do it with Elasticsearch and Grafana.
Security will also be addressed to make your Airflow instance compliant with your company. Specifying roles and permissions for your users with RBAC, preventing them from accessing the Airflow UI with authentication and password, data encryption, and more.

In addition:

Many practical exercises are given along the course so that you will have occasions to apply what you learn.
Best practices are stated when needed to give you the best ways of using Airflow.
Quiz are available to assess your comprehension at the end of each section.
Answering your questions fast is my top priority, and I will do my best for you.

I put a lot of effort into giving you the best content, and I hope you will enjoy it as much as I wanted to do it.

At the end of the course, you will be more confident than ever in using Airflow.

I wish you a great success!

Marc Lamberti

Reviews

November 19, 2023

It was a fantastic course to let me learn the airflow from 0 to hero. But we need the updated version airflow , smiling

Guilherme

August 20, 2023

The material used has the entire solution already in it. You need to keep deleting files to start doing the hands on lessons. Poor attention to detail and outdated concepts and explanations. Although the course is about Airflow 2.0 he doesn't even mention the TaskFlow API.

Artur

August 18, 2023

Great material I've been working with Airflow for couple of years and still found a few interesting features Highly recommend

Taufiq

August 1, 2023

The Explaination was very good, very smoot and the speaker clearly understandable for me because i dont speak english. that really helpful.

Ricardo

July 19, 2023

It has been a little bit challenging and sometimes content is outdated as not using @task decorator would be nice if there is something about that but maybe is on the rest of the course

Aleksandra

July 13, 2023

Started out as a great course and the topics planned to be covered seemed interesting. The Sections 2 and 3 were great as introduction, but after that the course quality gradually decreases with each section. Some parts were not explained but only meant to be mirrored from the screen, Additionally, troubles with docker images start from Section 3. Unfortunately, each section needed some debugging of docker image building, so I would definetely not recommend this for someone who does not understand docker concepts. It takes time and definetely decreases the motivation to continue with the course if you constantly need to debug/fix some errors not directly related to the topic you are trying to learn.

Murat

July 7, 2023

I actually do not know where to start. The course is quite extensive and it covers many topics which is great. However, even as an experienced data engineer I spent hours and hours debugging the course material. The content is outdated. So many different errors. So many questions without answers. No response from the instructor. Teaching in udemy is not an hobby, it is a full time work especially if you are teaching a course with over 20h of lessons. They should be always updated, students should be answered each day and helped with any problems. I definitely do not suggest this course to those who are not experinced on the topic and not willing to spend hours on debugging the code.

Amyth

July 3, 2023

Marc! you have truly delivered value with this course, It's simple and to the point! If you are here to learn Airflow, look no further! buy this course!

Muhammad

June 30, 2023

This course provides a decent understanding of the core concepts of airflow modules. However, there are a few drawbacks to consider. Firstly, the course heavily relies on Docker, and I understand for the sake of simplicity but most of the Docker compose is configured with outdated versions of airflow. This can be problematic, as newer versions have improved. Moreover, if you're interested in learning how to configure airflow, the course lacks comprehensive documentation in this area. Trying to replicate the code structure on your own machine using the latest airflow version will be frustrating and quite challenging, the instability is noticeable and contains breaking changes across different versions, making it very difficult to upgrade. Even if you make some changes in the configuration files, it often involves guesswork due to the lack of clear instructions in the course materials. Overall, while the course provides an introduction to airflow modules, it falls short in terms of documentation and adapting to newer versions and also lacks writing simple dags, Forexample in the first section the instructor helped to setup airflow, and suddenly jumped to develop a full real-life pipeline which contains irrelevant of airflow such as Hadoop, spark etc, these should be added in the course but if I am learning the course from zero level, I first want to master airflow and how it works and then move toward more advanced concepts, These limitations can frustrate learners who desire a more seamless learning experience.

Ioannis

June 7, 2023

Once again Marc has complexed a tutorial more than it was needed. Instead of setting up airflow in basic form, as he did in previous tutorial , and step by step explain the hive containers he installs and why, he does this with a docker file, and out of the blue he considers that everybody needs to understand hive containers and spark. He should have start with a simple set up, start a new simple project, explain what is the project's goal and step by step increase the complexity. Airflow is a tool for multiple software. Not everybody knows spark and hive. there are thousands of projects that can start by just copy files from one folder to another, then projects like the one he mentioned in the previous tutorial setting up a database with random users (and it would be a good idea to show people how to connect to containerized postreges db with third party IDE not only with linux cli )

Satya

May 21, 2023

Sebagai seseorang yang baru pertama kali belajar docker dan apache airflow. materi di kursus ini mudah dimengerti

Hernán

May 10, 2023

Amazing and super complete course on Apache Airflow. It is long and from time to time I have needed to do an extra effort to keep watching the videos but it worthed every minute.

Alin-Cristian

May 3, 2023

Takes time to explain a lot of context which can normally get you confused :) Course could do with more updates, but for more advanced users, it's by all means still doable. No major mismatches with the course content. There are many, many hurdles going through this course in 2023. You will spend a lot of time debugging different issues. Some of them were answered in the discussion forum/questions but many still remain unanswered. Marc promises often to have patience and wait for his explanation which I many times don't ever see. And we're talking issues from years ago that make it hard to follow what's happening. On top of that, I would add that towards the end of this course, Marc just gives you steps to follow but stops sharing any wisdom. It's a very monkey see monkey do type of learning, where you're not grasping new ideas but rather trying to keep up with a very outdated piece of tech. All the while, you might not be familiar with stuff like Redis, rabbitMQ, Kubernetes, Elastic Stack, Grafana, etc, etc etc, but you will have to dedicate quite a lot of extra time on these if you at least want to understand what the ping they are. These techs are barely touched upon as an explanation as to why they are used, which is quite a shame. I know the course is Airflow not other stacks, but I think a man with obvious good understanding of them should at least try to share a few words.

Kamran

May 1, 2023

The content itself is good, but the intonation makes it super difficult to watch a lesson without stopping. This might be super subjective opinion, and it should not be viewed as the rating of the quality of the content itself, but rather the audio/intonation.

Cheng-Hsien

April 5, 2023

Marc is a great instructor who explains every topic in a very clear manner. The "practice" video following each topic is well designed and helps build solid understanding. Throughout the videos, Marc also repeatedly demonstrates the same docker commands and that helps me build my muscle memory for docker (I am a newbie). To follow this course, I did run into multiple issues with the docker containers like other learners did (I am behind a corporate firewall) but I guess that is something everyone in IT should be able to learn to solve.

Charts

Price

Rating

Enrollment distribution

Apache Airflow: The Hands-On Guide - Distribution chart