Master AWS Lambda Functions for Data Engineers using Python

Build Lambda Functions using Python, Lambda Triggers, Deploy using layers and Docker, Validate using Glue and Athena

4.63 (164 reviews)

Udemy

platform

English

language

Other

What you will learn

Setup required tools on Windows to develop the code for ETL Data Pipelines using Python and AWS Services

Setup Project or Development Environment to develop applications using Python and AWS Services

Getting Started with AWS by creating account in AWS and also configure AWS CLI as well as Review Data Sets used for the project

Develop Core Logic to Ingest Data from source to AWS s3 using Python boto3

Getting Started with AWS Lambda Functions using Python 3 Run-time Environment

Refactor the application, build zip file to deploy as AWS Lambda Function

Create AWS Lambda Function using Zip file and Validate

Troubleshoot issues related to AWS Lambda Functions using AWS Cloudwatch

Build custom docker image for the application and push to AWS ECR

Create AWS Lambda Function using the custom docker image in AWS ECR

Develop Applications using AWS Lambda Functions by adding Python Modules as Layers

Why take this course?

Do you want to learn AWS Lambda Functions by building an end-to-end data pipeline using Python as Programming Language and other key AWS Services such as Boto3, S3, Dynamodb, ECR, Cloudwatch, Glue Catalog, Athena, etc? Here is one course using which you will learn AWS Lambda Functions by implementing an end-to-end pipeline by using all the services mentioned.

As part of this course, you will learn how to develop and deploy lambda functions using the zip files, custom docker images as well as layers. Also, you will understand how to trigger lambda functions from Eventsbridge as well as Step Functions.

Set up required tools on Windows to develop the code for ETL Data Pipelines using Python and AWS Services. You will take care of setting up Ubuntu using wsl, Docker Desktop, and Visual Studio Code along with Remote Development Extension Kit so that you can develop Python-based applications using AWS Services.
Setup Project or Development Environment to develop applications using Python and AWS Services on Windows and Mac.
Getting Started with AWS by creating an account in AWS and also configuring AWS CLI as well as Review Data Sets used for the project
Develop Core Logic to Ingest Data from source to AWS s3 using Python boto3. The application will be built using Boto3 to interact with AWS Services, Pandas for date arithmetic, and requests to get the files from the source via REST API.
Getting Started with AWS Lambda Functions using Python 3.9 Run-time Environment
Refactor the application, and build a zip file to deploy as AWS Lambda Function. The application logic includes capturing bookmarks as well as Job Run details in Dynamodb. You will also get an overview of Dynamodb and how to interact with Dynamodb to manage Bookmark as well as Job Run details.
Create AWS Lambda Function using a Zip file, deploy using AWS Console and Validate.
Troubleshoot issues related to AWS Lambda Functions using AWS Cloudwatch
Build a custom docker image for the application and push it to AWS ECR
Create AWS Lambda Function using the custom docker image in AWS ECR and then validate.
Get an understanding of AWS s3 Event Notifications or s3-based triggers on Lambda Function.
Develop another Python application to transform the data and also write the data in the form of Parquet to s3. The application will be built using Pandas by converting 10,000 records at a time to Parquet.
Build orchestrated pipeline using AWS s3 Event Notifications between the two Lambda Functions.
Schedule the first lambda function using AWS EventsBridge and then validate.
Finally, create an AWS Glue Catalog table on the s3 location which has parquet files, and validate by running SQL Queries using AWS Athena.
After going through the complete life cycle of Deploying and Scheduling Lambda Function and also validating the data by using Glue Catalog and AWS Athena, you will also understand how to use Layers for Lambda Function.

Here are the key takeaways from this training:

Develop Python Applications and Deploy as Lambda Functions by using a Zip-based bundle as well as a custom docker image.
Monitor and troubleshoot the issues by going through Cloudwatch logs.
The entire application code used for the demo along with the notebook used to come up with core logic.
Ability to build solutions using multiple AWS Services such as Boto3, S3, Dynamodb, ECR, Cloudwatch, Glue Catalog, Athena, etc

Reviews

RABINDRA

September 16, 2023

too long and no explanation . while installing python and docker not sure which stage in project we are going to need these. too much explanation for simple installation and things making us tired already. does not mentioned why we are doing certain things. speaking too fast.

MooseData

January 13, 2023

Review: TLDR - compared to top notch Udemy courses this one falls in 3 star ranking due to high level of instructions repetition, lack of 'why' explanations and missing details on Windows environment setup. On the other hand, instructions are so detailed (idiot-proof) that you are able to finish all tasks and course is informative on various way of interacting with AWS Lambda, including various deployment features, also relative to Udemy AWS courses this is great, hence 4 stars. Pros: Course is informative, more 'hands on' than theory (apart from application layout you will see very few presentation slides or animations, but pure coding) and gives a zero to hero approach, including environment setup, which was great for me coming from DB background. I think the layout is ok- first insight on tech stack we are going to use and then how to connect the dots on your local env, finally applying this to AWS Lambda. Cons: At the beginning of the course you are given details on setup of local environment, but one thing is missing - how to use Ubuntu WSL in Visual Studio Code (with AWS credentials setup), which may cause plenty of issues later on as you attempt to execute linux syntax commands in a Visual Studio default Powershell terminal. Also if you want to use AWS Cli in ubuntu, it has to be separately installed there (apart from Windows installation) Moreover Docker doesn't work in WSL v1 so you either have to containerize application in Windows Environment or move to WSL v2 (make sure check your default WSL version to be 2 before starting ubuntu). Fortunately most of the issues is quickly answered. I think it would be also beneficial to give 1minute description on relationships between virtual env / WSL / VSC. Also with any linux command used for the first time, it would be good to have explanation of options used (e.g why 'ls -ltr'?) This course could have been shortened by 20-30% if: Sometimes instructor is troubleshooting his own issues which takes time (e.g. some commands not running etc) Or repeating steps from previous lessons - e.g. Checking whether configuration is ok, which as a learning exercise may be ok, but takes time. Or repeating for 1 minute what we did in an exercise which took 3 minutes Exercices/lessons shouldn't be taking 3 minutes, rather ~10min. Or repeating topics already covered -e.g. sections on S3: lessons 36-38 vs 94-95. Or giving trivial statements - e.g. if you won't tick acknowledge you want I think the author favorite statement is 'this is nothing but…' which is frustrating to hear for 100th time. On some topics lecturer just repeats himself without any further meaningful insight. On the other topics an overview would be helpful -e.g. lecture 77 would greatly benefit from just providing a schema on how deployment process would be arranged (a picture is worth 1000 words). Accent is not that strong and well understandable but I think the instructor could be 10-15% slower - it would help to understand instructor better.

Smita

November 22, 2022

The instructor is doing the setup without giving details about 1. what is ubuntu 2. why we need virtual env for this project.

Jose

September 22, 2022

The content is really good, I highly recommend it. There are some concepts that are repeated several times (unnecessary). Some good additions could be versions/aliases, enabling lambda in VPC.

Deepak

September 16, 2022

I was looking for this course from long time. Thanks for creating this wonderful course. Covered each topic in detail and lots of learning is there.

Elmer

August 26, 2022

I some chapters your explanation goes too fast and in other chapters you have an entire module only to explain how to configure your environment in windows.

Charts

Price

Rating

Enrollment distribution

Master AWS Lambda Functions for Data Engineers using Python - Distribution chart