Apache Kafka Series - Kafka Connect Hands-on Learning

Kafka Connect - Learn How to Source Twitter Data, Store in Apache Kafka Topics & Sink in ElasticSearch and PostgreSQL

4.51 (5103 reviews)
Udemy
platform
English
language
Software Engineering
category
40,816
students
4.5 hours
content
Mar 2024
last update
$99.99
regular price

What you will learn

Configure and run Apache Kafka Source and Sink Connectors

Learn concepts behind Kafka Connect & the Kafka Connect architecture

Launch a Kafka Connect Cluster using Docker Compose

Deploy Kafka Connectors in Standalone and Distributed Mode

Write your own Kafka Connector

Description

A Comprehensive and Brand New Course for Learning Apache Kafka Connect Framework with Hands-on Training – (Launched in April 2017)

Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems. Apache Kafka Connect is a common framework for Apache Kafka producers and consumers.  

Apache Kafka Connect offers an API, Runtime, and REST Service to enable developers to define connectors that move large data sets into and out of Apache Kafka in real time. It inherits strong concepts such as fault-tolerance and elasticity thanks to being an extension of Apache Kafka. Kafka Connect can ingest entire databases, collect metrics, gather logs from all your application servers into Apache Kafka topics, making the data available for stream processing with low latency.  

Kafka Connect standardises integration of other data systems with Apache Kafka, simplifying connector development, deployment, and management.  

In this course, we are going to learn the Kafka Connector deployment, configuration and management with hands-on exercises. We are also going to see the distributed and standalone modes to scale up to a large, centrally managed service supporting an entire organisation or scale down to development, testing, and small production deployments. The REST interface to submit and manage connectors to your Kafka Connect cluster via an easy to use REST API’s.  

Students Loved this course – Students have rated the course with 5 stars...!!

Tin Pavlinic says “This course is an excellent resource for those getting started with Apache Kafka-Connect. Stephane explains concepts very clearly and provides plenty of practical examples. The setup is easy. You can follow the examples along on your own machine, and then experiment further. I highly recommend this Apache Kafka course.

Alan V says "This course is GREAT: 1. The teacher provides clear explanations of the role to Kafka Connector in Kafka Streaming Solution architecture. 2. The course uses docker-compose.yml to help learners quick build a practice environment. 3. The course is well organized and easy to understand the idea, behind Kafka Connector. The teacher guides you how to use the tool step by step. 4. The provided shell commands are all well commented and steps by steps to guide users. If you read the comments in the shell script files after the course, it will remind you what the teacher just taught in the videos. 5. The teacher is very responsible to learners' questions and very helpful. If you are a professional developer, I highly recommend you to attend this course.

Sagar Joshi says "It was very engaging and interesting course. I recommend taking this Apache Kafka course. Stephane has provided all the required details and its really a hands-on training course. Thank you..!!" 

===============================

Instructor

My name is Stephane Maarek, and I'll be your instructor in this course. I teach about Apache Kafka, the Kafka ecosystem and Kafka Certifications with my focus always on helping my students improve their professional proficiencies. I am also the co-founder of Conduktor: an enterprise Apache Kafka platform & UI to help everyone use Kafka.

Throughout my career in designing and delivering these certifications and courses, I have already taught 1,000,000+ students and gotten 350,000+ reviews!

With Apache Kafka becoming much more than a buzzword out there, I've decided it's time for students to properly learn how to be a Kafka professional. So, let’s kick start the course! You are in good hands!

===============================

This Course Also Comes With:

  • Lifetime Access to All Future Updates

  • A responsive instructor in the Q&A Section

  • Links to interesting articles, and lots of good code to base your next applications onto

  • Udemy Certificate of Completion Ready for Download

This is the course that could improve your career!

Apache Kafka is a skill in high demand and there are not enough people to fulfill all the open positions. You can boost your income, take on new roles and fun challenges. Many of my students are now the Kafka experts of their companies! You can be the next!

I hope to see you inside the course!

=======================

Note: Looking for more advanced Kafka concepts? There are many volumes in the Apache Kafka Series:

  • Learn Kafka for Beginners v2 (great to start)

  • Kafka Connect Hands On Learning

  • Kafka Streams for Data Processing

  • KSQL on ksqlDB - Hands On!

  • Kafka Cluster Setup & Administration

  • Confluent Schema Registry & Kafka REST Proxy

  • Kafka Security (SSL SASL ACL)

  • Kafka Monitoring and Operations

Happy learning!


 

Content

Course Introduction

Important Pre-Requisites
Course Objectives
Course Structure
About Your Instructor

Code Download

Code Download

Kafka Connect Concepts

What is Kafka Connect?
Kafka Connect Architecture Design
Connectors, Configuration, Tasks, Workers
Standalone vs Distributed Mode
Distributed Architecture in Details

Setup and Launch Kafka Connect Cluster

Important information about installation
Docker on Mac (recent versions)
Docker Toolbox on Mac (older versions)
Docker on Linux (Ubuntu as an example)
Docker on Windows 10 64bit
Docker Toolbox on Windows (older versions)
Important note for Docker Toolbox users
Starting Kafka Connect Cluster using Docker Compose

Troubleshooting Kafka Connect

It's not working! What to do?
Where to view logs?
Asking a question in the Q&A

Kafka Connect Source - Hands On

Kafka Connect Source Architecture Design
FileStream Source Connector - Standalone Mode - Part 1
FileStream Source Connector - Standalone Mode - Part 2
FileStream Source Connector - Distributed Mode
List of Available Connectors
Twitter Source Connector - Distributed Mode - Part 1
Twitter Source Connector - Distributed Mode - Part 2
Section Summary

Kafka Connect Sink - Hands On

Kafka Connect Sink Architecture Design
ElasticSearch Sink Connector - Distributed Mode - Part 1
ElasticSearch Sink Connector - Distributed Mode - Part 2
Kafka Connect REST API
JDBC Sink Connector - Distributed Mode

Writing your own Kafka Connector

Goal of the section: GitHubSourceConnector
Finding the code and installing required software
Description of the GitHub Issues API
Using the Maven Archetype to get started
Config Definitions
Connector Class
Writing a schema
Data Model for our Objects
Writing our GitHub API HTTP Client
Source Partition & Source Offsets
Source Task
Building and running a Connector in Standalone Mode
Command for next lecture
Deploying our Connector on the Landoop cluster
More Resources for Developers

Advanced Concepts

Setting up Kafka Connect in Production (1/2)
Setting up Kafka Connect in Production (2/2)
What's next?
Congratulations and Thank you!
THANK YOU!
Bonus Lecture: Student Special Coupons for my Other courses

Screenshots

Apache Kafka Series - Kafka Connect Hands-on Learning - Screenshot_01Apache Kafka Series - Kafka Connect Hands-on Learning - Screenshot_02Apache Kafka Series - Kafka Connect Hands-on Learning - Screenshot_03Apache Kafka Series - Kafka Connect Hands-on Learning - Screenshot_04

Reviews

Roberto
November 15, 2023
It was a great course, Stephane is quite fluent in Apache Kafka Connect, just as a comment, the topic related to creating our own connector was somewhat tedious, but perhaps necessary, but in general I recommend this course.
Anonymized
October 24, 2023
Overall good course. Sometimes I felt it's a bit rushed, where there's more "do like I show" instead of do it so because so and so
Naeem
October 10, 2023
Excellent Kafka Connect course by Stephane, clears to concepts to me, will help me progress my career in data integration pipeline development.
Gabriela
October 4, 2023
The course provides solid theoretical content. However, practical materials, like code and packages, are outdated and no longer usefu
Parwand
October 2, 2023
Amazing course, I learned so much and I love the blend of the Theory and the hands-on .. thanks ! 4,5 Stars to motivate you to make better and better course later .. good Luck ..
Osamah
September 27, 2023
The course provides solid theoretical content. However, practical materials, like code and packages, are outdated and no longer useful. Keeping these resources current would significantly enhance the learning experience.
Howard
September 11, 2023
This was an important course that covered topics related to work I am doing. My main issue with this course was that a hands-on portion requires the student to integrate with the Twitter (now X) API to gather Tweet data and add it to a Kafka topic. At the time the course was written, I'm sure that Twitter API access could be obtained quickly, but at the time I took the course, it would take a prohibitively longer time to get that access. I would have liked a hands-on portion of the course for creating a custom Kafka Sink Connector. I was able to use the skills I learned in this course, as well as Confluent documentation, to create one, but I think I could have done so much faster if there had been directed learning.
Aditya
September 10, 2023
Twitter App streaming is outdated. Please make a new video on how to stream and sink data via twitter (after 2022)
Adriana
September 9, 2023
Another excellent Kafka course by Stephane, unfortunately with some outdated parts that end up making progress through the course difficult. With some research, it was possible to complete the course, but it was not possible to create Twitter Source Connector, due to the Twitter API V2 update.
George
September 4, 2023
If I could time-warp back to 2017, I'd say this was a great course. Unfortunately, in 2023, it's mostly broken. Twitter Source API's no longer work, ElasticSearch Sink API's require more complex version information than the elasticsearch docker image in use, and partial solutions to Source Connector issues (replacing Twitter with Wikimedia) are incompatible with current Sink Connectors (ElasticSearch and JDBC). If this course were to be updated in 2023 or later, I would look forward to going through it again. However, I suspect a lot of work and changes would be needed to get things running
Raúl
August 29, 2023
The explanation of the course is fantastic. It would be great to have the content updated because some of the parameters or commands used are deprecated.
Nicolae
August 27, 2023
This is a very good course on Kafka Connect! The main reason I haven't rated it 5 stars is that the section on the Twitter Source Connector no longer works. That example of a connector should be swapped for a different connector. I also have a bit of a preference towards code-along tutorials, so I expected the "Writing your own Kafka Connector" section to be more in the style of a code-along, but that was not why I rated the course 4.5 stars
Meisam
August 23, 2023
The course, though looks like the only reference for developing Kafka connectors, but outdated and not updated after around 5 years. Also, the questions in Q&A section are not responded back, some more than a couple of years. But overall, acceptable
Manoj
August 22, 2023
Course is outdated and needs lots of catch up work. I have taken multiple courses from Stephane and this one disappoint me :(
Aliaksandr
August 16, 2023
The theory is good, but practical materials are outdated and unusable. If I'd bought this course with full price I'd be very unhappy. Fortunatelly I've bought it with discount coupon.

Charts

Price

Apache Kafka Series - Kafka Connect Hands-on Learning - Price chart

Rating

Apache Kafka Series - Kafka Connect Hands-on Learning - Ratings chart

Enrollment distribution

Apache Kafka Series - Kafka Connect Hands-on Learning - Distribution chart

Related Topics

1141696
udemy ID
3/11/2017
course created date
8/8/2019
course indexed date
Bot
course submited by