Data Engineering with Google Dataflow and Apache Beam on GCP

First steps to Extract, Transform and Load data using Apache Beam and Deploy Pipelines on Google Dataflow

4.36 (550 reviews)

Udemy

platform

English

language

Data Science

What you will learn

Apache Beam

ETL

Python

Google Cloud

DataFlow

Google Cloud Storage

Big Query

Why take this course?

This course wants to introduce you to the Apache Foundation's newest data pipeline development framework: The Apache Beam, and how this feature is becoming popular in partnership with Google Dataflow. In a summary, we want to cover the following topics:

1. Understand your inner workings

2. What are your benefits

3. Explain how to use on your local machine without installation via Google Colab for development

4. Its main functions

5. Configure Apache Beam python SDK locallyvice

6. How to deploy this resource on Google Dataflow to a Batch pipeline

This course is dynamic, you will be receiving updates whenever possible.

It is important to remember that this course does not teach Python, but uses it. So, get comfortable with knowing Python basics, defining a function, creating objects and data types.

Also, if you are interested in learning section 4, which consists of deploying a pipeline on Google Dataflow, you will need to have a free counter in GCP. It's a simple process, but it requires a credit card!

I kindly ask you you to consider all the efforts to put this course together and give a nice rate at the end of the course, even tough the course is simple, it was made with all good intent to share knowledge for cheap price. Thanks and hope you enjoy!

___________________________________________________________________________________________________________

Requirements:

· Basic knowledge of Python

· Have Python 3.7 or greater installed locally (from section 4)

· Free account at GCP (from section 4)

Schedule:

· Section 2 – Concepts

· Section 3 – Main Functions

· Section 4 – Apache Beam on Google Dataflow

Reviews

Jonathan

September 15, 2023

This course was very straightforward to get started with Dataflow. It removes the abstraction that comes with GCP and so many connected tools. Anyone considering using Apache Beam with a Dataflow engine, or local... should take this course. Thank you!

Ashwin

September 15, 2023

Excellent explanation of all concepts along with hands on . I would definitely recommend this course.

Ravikumar

September 6, 2023

Brilliant course. Just in little more than 2 hours, it gives overall idea of how Apache beam works with Google Dataflow.

Prashant

August 26, 2023

Last chapter was really very conceptual. hope you will share streaming chapter with real time scenario. thanks

Viswa

June 6, 2023

It's going amazing so far. Only feedback I have is because the lecture is in English which I assume is with the intent to educate as many people as possible so it would be nice if the language chosen for GCP Console is also English while showing things in the console.

Aparajita

May 17, 2023

Did not expect this course to be in any other language than English. I was really looking forward to the streaming data part but found it to be in Portuguese.

Nicholas

May 15, 2023

Blank black screens, pretty much going through the motions at this point and doesn't give any sort input on why the things they are saying are important.

John

May 9, 2023

I appreciate this course. I have had trouble learning Python Apache Beam, and this course definitely was a good resource. It is definitely beginner/foundation knowledge, but that's all you need when jumping into a software. Cassio is a good instructor. English isn't his first language, but I think he does pretty good in conveying lessons, at least better than the Apache Beam docs. I do wish that the streaming part wasn't just English Caption, but I won't dock the rating for that as I am only focused on batch in my current project, and am rating only the stuff I used.

Fernando

May 4, 2023

easy to follow, even tought the streaming part is just with subtitles (recorded in another language). Helps a lot!

Setory

May 2, 2023

The teacher have good knowledge about the topic. Clear and simple project to get started and deploy first models.

Lian

April 27, 2023

Very good course to get started with apache beam using python and deploy running pipelines in google dataflow

Ahmad

April 12, 2023

The batch processing in this course is very useful, especially creating the template. But unfortunately the streaming section is not in English so I cannot following along

Prashant

February 14, 2023

Amazing Explanation. After watching this Course, Now I can create my own custom template and execute the jobs. Thanks !! for the content.

Naresh

November 9, 2022

just Okay. need more explanation about use or writing of python files. when run py file, how dataflow job works and step by step mapping steps from apache beam pipeline to dataflow job.

Heber

November 4, 2022

A very practical course to learn about the use of apache beam with python and gcp. Thank you so much!

Charts

Price

Rating

Enrollment distribution

Data Engineering with Google Dataflow and Apache Beam on GCP - Distribution chart