Apache Kafka for Beginners - Learn Kafka by Hands-On

Deep dive in to Apache Kafka concepts and learn to build Kafka producers/consumers using Java.

4.50 (728 reviews)

Apache Kafka for Beginners - Learn Kafka by Hands-On


5 hours


Jun 2021

Last Update
Regular Price

Exclusive  SkillShare  Offer
Unlimited access to 30 000 Premium SkillShare courses

What you will learn

Understand all the Kafka concepts and Kafka core internals.

Learn Kafka internals through practice

Build Kafka Producer applications using Java

Learn the Kafka Producer Techniques for reliable data delivery

Build Kafka Consumer applications using Java

Advance Kafka Consumer Techniques to handle duplicate processing of records

Consumer Rebalance and different techniques to handle them

Consumer techniques to seek to a specific offset

Learn to build Custom Serializer and Deserializer in Kafka

By the end of the course you will have a complete understanding of Apache Kafka and build Kafka consumer applications using Java


[MAJOR UPDATE : This course was completely redone on March 2020]


Learn the fundamentals and advanced concepts of Apache Kafka in this course. This course will give you a good understanding of all the concepts through Hands-On

  • This course will walk you through the Kafka concepts step by step

  • All the Kafka concepts will be taught through Hands-On

  • This course will  increase your technical expertise in Kafka

  • This course will give you good exposure to build Kafka Client applications using Java

  • This course will be an eyeopener for people who think Kafka is a complex system and you will find yourself in a better place after you complete this course

  • This course will transfer you from no Kafka experience to an experienced Kafka user.

Course Overview:

Getting Started with Kafka

  • In this section, I will give you all a quick introduction to Apache Kafka, terminologies and different client APIs that are part of Kafka

Download and Install Kafka

  • In this section, we will download the Kafka distribution from the Kafka Website.

Understanding Kafka Components and its Internals - (Theory + Hands-On)

In this section, we will explore the Kafka internals from a theoretical perspective followed by hands-on exploring the Kafka internals.

  • Learn about the Topics and Partitions in Kafka

  • Setup a Local Kafka Cluster with Multiple Brokers

  • Producer/Consumer messages in the Kafka Cluster

  • Learn about Consumer Offsets and Consumer Groups

  • Commit Log and Retention Policy

  • Learn about Kafka Load Distribution and Fault Tolerance and Robustness

Kafka Producer API - Hands-On

In this section, we will learn about building the Kafka Producer to producer messages into Kafka using the producer API

  • Setting up the base-project for Producer

  • Understand how the producer API works

  • Produce messages to the Kafka topic using the Synchronous/Asynchronous approach

  • Understand how Kafka producer API works behind the scenes

Kafka Producer API - Guaranteed Message Delivery Configurations

In this lecture, we will learn about the different configurations that guarantee message delivery into Kafka.

  • Explore some of the producer API Configurations

  • Configuring the acks and min.insync.replicas for guaranteed message delivery

  • Override the retries and retry-back-off-ms value\

Kafka Consumer API - Hands On

In this section, we will learn about building Kafka consumers to read messages from Kafka using the Consumer API.

  • Setting up the base-project for Consumers

  • Consuming Messages using Consumer API - poll() loop

  • Build Kafka Consumer using the Consumer API

  • auto.offset.reset - earliest vs latest

Consumer Groups & Consumer Rebalance

In this section, we will learn about Consumer Groups and Consumer rebalance.

  • Scaling the Consumer using Consumer Groups

  • Consumer Rebalance

  • Consumer Rebalance & max-poll-interval-ms

Consumer Offsets - Default and Manual Offset Management - Hands-On

  • Committing Offsets - Default Offset Commit Strategy

  • Manually Committing Offsets

  • Committing Offsets Synchronously - using commitSync()

  • Committing Offsets Asynchronously - using commitAsync()

  • Committing Specific Offsets - using CommitSync()

By the end of this, you will have a complete understanding of how Apache Kafka works and how to build Kafka client applications using Producer and Consumer API.


Getting Started

About the Course

Getting Started with Kafka

Introduction to Kafka

Kafka Terminologies and Client APIs

Download and Install Kafka

Download and Install Kafka

Understanding Kafka Components and its Internals - (Theory + Hands On)

Kafka Topics and Partitions - Theory

Set Up a ZooKeeper/Kafka Broker in Local

Create Topic, Produce and Consume Messages using the Command Line Interface(CLI)

Produce and Consume Messages with Key

Consumer Offsets

Consumer Groups

Commit Log and Retention Policy

Kafka as a Distributed Streaming System

Setting up a Kafka Cluster in Local with 3 Kafka Brokers

How Kafka Cluster distributes the Client Requests ? - Leader/Follower

How Kafka handles Data Loss ? - Replication and In-Sync-Replica(ISR)

Fault Tolerance and Robustness in Kafka

Kafka Producers

Download and Install Eclipse

Hands on - Kafka Producer coding using Java

Hands on - How to alter the configuration of a topic ?

Hands on - Producer Partitioning Mechanism using Java

Kafka Consumers

Hands on - Kafka Consumer using Java

Hands On - Offset Management in Kafka consumer using Java

Consumer Group in Kafka Consumer - Insight and Benefits

Hands on - Consumer Group using Java

Hands on - Group Id and its Impacts

Hands on - Reset to a specific Offset value in Kafka Consumer using Java

Kafka Consumer - Additional Configurations

Kafka Client - GUI Tool

Kafka - GUI Client

Apache Camel + Kafka Integration

Camel - Quick Introduction

Hands on - Simple Camel Example

Camel Architecture

Hands on - Kafka Consumer using Apache Camel

Hands on - Kafka Producer using Apache Camel

Apache Spark + Kafka Integration

Spark - Spark Streaming - Quick Introduction

Hands on - Kafka Consumer using Spark Streaming

Apache Kafka - Additional Configurations

Hands on - How to delete a topic ?

Apache Kafka + Spring Boot Integration

About this Section

Spring - Quick Introduction

Spring Boot - Quick Introduction

Simple Spring Boot App - Hands on Part 1

Simple Spring Boot App - Hands on Part 2

Spring Boot - Kafka Consumer - Hands On

Spring Boot - Kafka Producer Hands On

Spring Boot - Kafka Consumer Manual Offset Commit - Hands on

Spring Boot - Kafka Consumer Group - Hands on

Docker - Dockerize Kafka, Zookeeper, Producer and Consumer

Docker - Quick Introduction

How to Install Docker Tool Box on Mac ?

How to Install Docker Tool Box on Windows 10?

Hands on - How to create a docker image for Kafka Producer and Consumer?

How to create an account in Docker Hub and push the images to Docker Hub ?

Hands on - Running basic docker commands

Hands on - How to run a zookeeper/Kafka in Docker ?

Demo - Connect to Dockerized kafka/zookeeper using Dockerized Spring boot app

Apache Kafka - Security and Authentication

Why Kafka security ?

Hands on - How to enable SSL in Kafka Broker?

Hands on - How to access the SSL secured topic using Console Producer/Consumer ?

Hands on - How to configure SSL in Kafka Producer using Java?

Hands on - How to configure SSL in Kafka Consumer using Java?

Bonus Section

Coupon for my Other Courses


Congratulations !


Nouroudine1 May 2020

Great Dilip you explain very well. I'm very happy to by your course you again. See you in apache camel and SpringBoot.

Jayakeerthi13 April 2020



At End of session few videos explanation is so first , not able to understand and no clear explanation. it could be better if instructor spend some more time for explain videos. please dont remove any video you can update those if it became depricated. kindly request

Estrella1 April 2020

Very clear and concise information. Easy to follow and easy to understand. Just enough background information to provide a basis for understanding the subject.

Objectsolve30 December 2019

Great course. Needs to be updated. Kafka Streaming and more examples on Streaming in general would be nice. AckMode moved. Use try with resources, Constructor instead of field injection.

Franz23 December 2019

I feel that the course lacked a lot of real information (and best practices). Since this is entitled "deep dive hands on..." I understood that it could have been a really complete course, which was not so for my taste. On many occasions it was only copy & paste lessons, which greatly detracted from the course in general. It could have been a good course taking into account more real scenarios, however it was not so. I am not happy with this course, I think that much more complete examples are missing.

Mohsen18 December 2019

There are many concepts that are not explained until after. The installation tutorial was confusing. It did not show how the last piece is done. For instance, I have a mac and what the instructor said was that if you have a mac, you should use the files in blah blah folder. Though how should I " use" them? it is not clear...

Avinash11 December 2019

The concepts are very clearly explained along with hands on coding exercises. The pace of the course is also optimal to understand the concepts. I would recommend this course for anyone who wants to get hands on with Apache Kafka.

Aloisio10 December 2019

I guess he speaks so fast some moments.. i mean..for me a not english native speaker its litle hard to understand sometimes...

Gurmeet27 July 2019

it is a pretty good course, I wish he also showed some connector examples. Like some very well known databases and social websites

Gaurav30 May 2019

So far I am almost 50% done with the course. It started well but couple of sections, I feel, have been covered very fast like Camel and Spark. Though the instructor copies everything (code) from the GitHub but does not spend time to explain the keywords/functions of the code. If you are copying the code, it will definitely run. So you can make the videos longer, but should explain each word of it in detail. I am yet to cover the remaining sections and will leave my feedback. Update:- Well, the same goes with Spring Boot etc.. Well I think the instructor has the knowledge but tried to cover a lot of things inside this course and that made me confused. No real time scenarios/case studies were covered. The focus always remained just to make the code run and print hello, how are you, are you alive, etc. messages.

Bryan23 May 2019

Para hispano hablantes poco entendible, habla muy rapido. Las diapositivas tienen textos muy pequeños y no realiza ampliación de pantalla al momento de la demostración codificada por ello es difícil seguirlo

Andrey22 May 2019

Sometimes it is difficult to understand, because you speak very fast and the pronunciation is not so clear

Bahadir21 May 2019

This course is very helpful to understand Apache-Kafka mentality. I suggest taking this course who is starting to learn kafka. There is one thing to say to improve quality of the course is the imports of the codes is wasting time. Thank You

Lennarth11 March 2019

Excellent, thank you. I would just suggest edit some videos to trim mistakes while coding, but other than that the content was really useful.


Udemy ID


Course created date


Course Indexed date
Course Submitted by