Course Introduction
Introduction
Lesson 1: Getting Started with Big Data and Apache Kafka
Learning Objective
Data Evolution Over Time
Big Data Overview
Big Data Analytics
Messaging System
Introduction and Features of Kafka
Kafka Usecases
Kafka Terminologies
Kafka Components
Kafka Architecture
Kafka Clusters
Kafka Industry Usecases
Demo - Install Zookeeper
Demo - Install Kafka
Demo - Setup a Single Node Single Broker Cluster
Demo - Setup a Multi Node Single Broker Cluster
Key Takeaway
Knowledge Check
Lesson 2: Kafka Producer
Learning Objective
Overview of Producer and It's Architecture
Kafka Producer Configuration
Kafka Producer Optional Configuration
Kafka Producer Configuration Objects
Demo - Create a Kafka Producer
Sending Messages
Serializers and Custom Serializers
Demo - Creating a Custom Serializer
Serializers Challenges and Serializing using Apache Avro
Demo - Serializing Using Apache Avro
Partitions
Demo - Setup Custom Partition
Key Takeaway
Knowledge Check
Lesson 3: Kafka Consumer
Learning Objective
Overview of Kafka Consumer
Consumer Groups
Partition Rebalance and Creating a Consumer
Poll loop and Its Functioning
Kafka Configuring Consumer - Part 1
Kafka Consumer Configuration - Part 2
Demo - Create Kafka Consumer
Commit and Offset
Ways of Commiting Offset - Automatic Offset
Ways of Commiting Offset - Commit Current Offset
Ways of Commiting Offset - Asynchronous Commit
Ways of Commiting Offset - Combining Synchronous and Asynchronous Commits
Ways of Commiting Offset - Commit Specified Offset
Rebalance Listeners
Consuming Records with Specific Offset
Deserializers
Demo - Create and Use Custom Deserializer
Key Takeaway
Knowledge Check
Lesson 4: Kafka Operations and Performance Tuning
Learning Objective
Kafka Internals Overview
Replication and Replica Types
Preffered Ladder, Request and Request Processing
Types of Requests
Partition Allocation, File Management and Segments
File Format, Index and Compaction
Kafka Reliability and Reliablity Methods
Broker Configuration for Replication
Producer in Reliable System
Consumer in Reliable System
Key Takeaway
Knowledge Check
Lesson 5: Kafka Cluster Architectures and Administering Kafka
Learning Objective
Cluster Mirroring
Multi Cluster Architecture - Hub, Spokes, and Active Active Architecture
Multi Cluster Architecture - Active-Standby Architecture and Strech Clusters
MirrorMaker Configuration
MirrorMaker Deployment and Tuning
Demo - Setting up MirrorMaker
Administering Kafka - Topic Operations
Administering Kafka - Consumer Group Operations
Dynamic Configuration Changes
Partition Management
Console Producer Tool
Console Consumer Tool
Key Takeaway
Knowledge Check
Lesson 6: Kafka Monitoring and Schema Registry
Learning Objective
Monitoring and Its Importance
Server or Infrastructure Monitoring and Application Monitoring
Kafka Monitoring
Kafka Broker Metrics - Under Replicated Partitions
Kafka Broker Metrics - Others
Topic and Partition Specific Metrics
Logging and Client Monitoring
Producer and Consumer Metrics
Quotas and Lag Monitoring
Monitoring Dashboard
Demo - Setting up Open Source Health Monitor
Kafka Schema Registry
Kafka Component and Architecture
Kafka Schema Registry - Internal working and Use-cases
Kafka Schema Registry Working
Demo - Using Kafka Schema Registry With Kafka
Key Takeaway
Knowledge Check
Lesson 7: Kafka Streams and Kafka Connectors
Learning Objective
Kafka Stream Overview
Kafka Stream
Kafka Stream Architecture and Working
Kafka Stream Components
Kafka Stream Architecture Tasks, Threading Model and Local State Store
Kafka Stream Architecture - Record Buffer
Memory Management and Streaming Data Pipeline
Kafka Stream DSL
K-Stream Operations
K-Table
K-Table Operations
Aggregation and Windowing
Processor Topology and Stream Processor
Stream and Processor APIs
Processor APIs - Create Topology
Kafka Connectors
Standalone and Sink Connector Configuration
Running Kafka Connect
Kafka Connector Distributed Mode
HTTP Rest Interface
Demo - Kafka Connector
Demo - Create an Application using Kafka Streams
Key Takeaway
Knowledge Check
Lesson 8: Integration of Kafka with Storm
Learning Objective
Apache Storm
Real-time Analytics
Apache Storm Architecture
Apache Storm Components
Apache Storm Topology Part - 1
Apache Storm Topology Part - 2
Kafka Spout
Integration of Apache Storm and Kafka
Demo - Simple Standalone Application using Kafka and Storm
Key Takeaway
Knowledge Check
Lesson 9: Kafka Integration with Spark and Flume
Learning Objective
Introduction to Spark
Spark Components
Resilient Distributed Dataset
RDD Operations - Transformation - Map, FlatMap and Filter
RDD Operations - Transformation - Join, Distinct, First and Take
RDD Operations - Actions
Data Sets and Spark Session
Data Sets and Spark Session Operations
Spark Stream
Spark Integration with Kafka
Demo - Running Small Standalone Application in Spark with Kafka
Flume
Flume Connectors
Flume Kafka to HDFS Configuration
Demo - Creating Flume agent Sending data from Kafka to HDFS
Key Takeaway
Knowledge Check
Lesson 10: Admin Client and Securing Kafka
Learning Objective
Admin Client
Demo - Perform Various Admin Tasks using Admin Client
Kafka Security
Kafka Security Component
SASL
Configure SSL in Kafka
Secure using ACLs
Key Takeway
Knowledge Check
Course End Project