Stream processing frameworks for big data: the internals

A deep dive into the internals of Flink, Spark Streaming, Structured Streaming, and Kafka Streams

4.08 (6 reviews)

Udemy

platform

English

language

Development Tools

What you will learn

The features and internals of Flink, Spark Streaming, Structured Streaming and Kafka Streams.

How to select the right stream processing framework for a use case.

The current state-of-the-art of distributed stream processing.

References to equivalent implementations in all frameworks.

This is not a programming course! This is a course on understanding how these systems work.

Why take this course?

Do you need to use stream processing for your next project but have no idea where to begin? Or do you want to grow into a data engineering role and want to start building up knowledge on stream processing?

In this course, we give a detailed explanation and comparison of several popular stream processing frameworks. At the finish line, you will be able to make a well-grounded selection of the right framework for your use case or to start your learning process. We will cover Flink, Kafka Streams, Spark Streaming and Structured Streaming. These are the four frameworks that are currently the state-of-the-art in the industry.

You will understand their features, characteristics and differences. This course gives you the perfect primer to start learning and better understand the APIs and programming languages behind these frameworks.

This course covers all relevant aspects:

- their general characteristics

- APIs

- latency and throughput performance

- scalability

- elasticity

- fault tolerance

- state management

- deployment

- ...

We will dive deeply into the workings and the advantages and disadvantages of the different mechanisms and approaches.

!!! This course is not a programming course but focuses on more theoretical aspects.

At the end, you will be provided with a concise overview on what was covered.

The content of this course is based on the results of Giselle's PhD work in which she benchmarked and analyzed these frameworks on all these characteristics.