Description

Learn core Apache Kafka features along with creating Java, Node.js and Python producers and consumers

Explore an interactive and easy way to learn Apache Kafka with this practical guide. Featuring interesting activities to help you build projects using APIs for programming languages such as Node.js and Python, this course will ensure you get hands-on experience. You will learn how Apache Kafka works and also understand its core features. After delving into the architecture of Kafka, you will focus on how to build custom Apache Kafka producers and consumers using native Java API. The course will gradually assist you in working through a variety of activities related to Apache Kafka, from creating a cluster with multiple brokers and developing topics with replication factors, through to launching the performance monitor for testing the performance of consumers and producers. The course will also demonstrate how to use Apache Kafka API to create your own consumers and producers using Python and Node.js. By the end of this course, you'll be well-versed with Apache Kafka and have the skills and confidence you need to build projects using it. All codes and supporting files are available at- https://github.com/PacktPublishing/Apache-Kafka-Complete-Developer-s-Guide

What You Will Learn

Get to grips with the Apache Kafka architecture
Create producers and consumers using a built-in console
Use Python and Node.js to produce and consume messages
Design fault-tolerant clusters with topic replication across multiple brokers
Build a Kafka cluster with multiple brokers
Delve into the components of the Apache Kafka cluster

Audience

This course is designed for beginners who want to learn Apache Kafka and its architecture. Anyone who wants to build producers and consumers in other languages such as Node.js and Python, or anyone who wants to build projects using Kafka and its APIs will find this practice-oriented course helpful.

Approach

This course uses a complete practice-based approach, which ensures that students get hands-on experience in building projects using different APIs.

Key Features

Understand how Apache Kafka works and its core features * Build custom Apache Kafka producers and consumers using native Java API * Learn and practice how to run multiple brokers on the same computer

Github Repo

https://github.com/packtpublishing/apache-kafka-complete-developer-s-guide

About the Author

Bogdan Stashchuk

Bogdan Stashchuk has over 20 years of experience as a software engineering instructor. He excels at breaking down complex topics into easy-to-follow steps. His courses are designed with hands-on exercises, ensuring that learners can actively participate and apply what they learn. From start to finish, students can follow along and complete tasks just as Bogdan demonstrates in his lectures. He also includes challenging assignments with detailed solutions. This approach helps learners understand and remember the material long after they've completed the course. Through his dedication and expertise, Bogdan ensures a valuable and effective learning experience for everyone.

Course Outline

1. Apache Kafka Installation Overview

Apache Kafka Installation Overview: Apache Kafka Installation Overview

2. Installing Apache Kafka on the Mac and Unix-like systems

Apache Kafka Installation Overview: Installing Apache Kafka on the Mac and Unix-like systems

3. Installing Apache Kafka on the Mac

Apache Kafka Installation Overview: Installing Apache Kafka on the Mac

4. Installing Ubuntu on MacOS using VirtualBox

Apache Kafka Installation Overview: Installing Ubuntu on MacOS using VirtualBox

2. Installing Apache Kafka on the remote Ubuntu server

1. SECTION 2 Introduction

Installing Apache Kafka on the remote Ubuntu server: SECTION 2 Introduction

2. Creating remote Ubuntu Virtual Private Server

Installing Apache Kafka on the remote Ubuntu server: Creating remote Ubuntu Virtual Private Server

3. Installing Apache Kafka on Virtual Private Server

Installing Apache Kafka on the remote Ubuntu server: Installing Apache Kafka on Virtual Private Server

3. Installing Apache Kafka on Windows

1. SECTION 3 Introduction

Installing Apache Kafka on Windows: SECTION 3 Introduction

2. Installing Apache Kafka on Windows

Installing Apache Kafka on Windows: Installing Apache Kafka on Windows

3. Starting Zookeeper and Kafka server on Windows

Installing Apache Kafka on Windows: Starting Zookeeper and Kafka server on Windows

4. Installing Ubuntu on Windows using VirtualBox

Installing Apache Kafka on Windows: Installing Ubuntu on Windows using VirtualBox

5. Installing Apache Kafka on Ubuntu using GUI

Installing Apache Kafka on Windows: Installing Apache Kafka on Ubuntu using GUI

4. Starting Apache Zookeeper and Kafka Broker

1. SECTION 4 Introduction

Starting Apache Zookeeper and Kafka Broker: SECTION 4 Introduction

2. Observing contents of the Kafka folder

Starting Apache Zookeeper and Kafka Broker: Observing contents of the Kafka folder

3. Reading and editing Kafka files using VisualStudio Code

Starting Apache Zookeeper and Kafka Broker: Reading and editing Kafka files using VisualStudio Code

4. Trying to start Kafka Server

Starting Apache Zookeeper and Kafka Broker: Trying to start Kafka Server

5. Observing Kafka Server logs

Starting Apache Zookeeper and Kafka Broker: Observing Kafka Server logs

6. Starting Zookeeper

Starting Apache Zookeeper and Kafka Broker: Starting Zookeeper

7. Starting Kafka Server while Zookeeper is up and running

Starting Apache Zookeeper and Kafka Broker: Starting Kafka Server while Zookeeper is up and running

8. Observing logs folder and current kafka server setup

Starting Apache Zookeeper and Kafka Broker: Observing logs folder and current kafka server setup

5. Creating and exploring Kafka Topic

1. SECTION 5 Introduction

Creating and exploring Kafka Topic: SECTION 5 Introduction

2. How to connect to Kafka cluster

Creating and exploring Kafka Topic: How to connect to Kafka cluster

3. Create new Kafka topic

Creating and exploring Kafka Topic: Create new Kafka topic

4. What happened after creation of the new topic

Creating and exploring Kafka Topic: What happened after creation of the new topic

5. Read details about topic

Creating and exploring Kafka Topic: Read details about topic

6. Producing and consuming Messages

1. SECTION 6 Introduction

Producing and consuming Messages: SECTION 6 Introduction

2. Send some messages using Kafka Console Producer

Producing and consuming Messages: Send some messages using Kafka Console Producer

3. Consuming messages using Kafka Console Consumer

Producing and consuming Messages: Consuming messages using Kafka Console Consumer

4. Consuming messages from the beginning

Producing and consuming Messages: Consuming messages from the beginning

5. Running multiple consumers

Producing and consuming Messages: Running multiple consumers

6. Running multiple producers

Producing and consuming Messages: Running multiple producers

7. What was changed in the Kafka logs

Producing and consuming Messages: What was changed in the Kafka logs

7. What is Apache Kafka and how it works

1. SECTION 7 Introduction

What is Apache Kafka and how it works: SECTION 7 Introduction:

2. What is Apache Kafka

What is Apache Kafka and how it works: What is Apache Kafka

3. Broker

What is Apache Kafka and how it works: Broker

4. Broker cluster

What is Apache Kafka and how it works: Broker cluster

5. Zookeeper

What is Apache Kafka and how it works: Zookeeper

6. Zookeeper ensemble

What is Apache Kafka and how it works: Zookeeper ensemble

7. Multiple Kafka clusters

What is Apache Kafka and how it works: Multiple Kafka clusters

8. Default ports of Zookeeper and Broker

What is Apache Kafka and how it works: Default ports of Zookeeper and Broker

9. Kafka Topic

What is Apache Kafka and how it works: Kafka Topic

10. Message structure

What is Apache Kafka and how it works: Message structure

11. Topics and Partitions

What is Apache Kafka and how it works: Topic and Partitions

12. Spreading messages across partitions

What is Apache Kafka and how it works: Spreading messages across partitions

13. Partition Leader and Followers

What is Apache Kafka and how it works: Partition Leader and Followers

14. Controller and it's responsibilities

What is Apache Kafka and how it works: Controller and it's responsibilities

15. How Producers write messages to the topic

What is Apache Kafka and how it works: How Producers write messages to the topic

16. How Consumers read messages from the topic

What is Apache Kafka and how it works: How Consumers read messages from the topic

8. GitHub Repository and Diagrams for the course

1. SECTION 8 Introduction

GitHub Repository and Diagrams for the course: SECTION 8 Introduction

2. GitHub repository and list of basic Kafka commands

GitHub Repository and Diagrams for the course: GitHub repository and list of basic Kafka commands

3. Diagrams for the course

GitHub Repository and Diagrams for the course: Diagrams for the course

9. EXAMPLE 1 - Topic with Multiple Partitions

1. SECTION 9 Introduction

EXAMPLE 1 - Topic with Multiple Partitions: SECTION 9 Introduction

2. Cleaning up existing kafka installation

EXAMPLE 1 - Topic with Multiple Partitions: Cleaning up existing kafka installation

3. Creating topic with multiple partitions

EXAMPLE 1 - Topic with Multiple Partitions: Creating topic with multiple partitions

4. How messages were spread across different partitions

EXAMPLE 1 - Topic with Multiple Partitions: How messages were spread across different partitions

5. Reading messages from specific partition

EXAMPLE 1 - Topic with Multiple Partitions: Reading messages from specific partition

6. Reading messages from specific offset in specific partition

EXAMPLE 1 - Topic with Multiple Partitions: Reading messages from specific offset in specific partition

7. Reading details about topic and __consumer_offsets topic

EXAMPLE 1 - Topic with Multiple Partitions: Reading details about topic and __consumer_offsets topic

8. Summary for multiple partitions example

EXAMPLE 1 - Topic with Multiple Partitions: Summary for multiple partitions example

10. EXAMPLE 2 - Kafka Cluster with Multiple Brokers

1. SECTION 10 Introduction

EXAMPLE 2 - Kafka Cluster with Multiple Brokers: SECTION 10 Introduction

2. Example overview - run multiple brokers

EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Example overview - run multiple brokers

3. Creating separate configuration files for brokers

EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Creating separate configuration files for brokers

4. Launching three brokers

EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Launching three brokers

5. Getting cluster information and broker details from Zookeeper

EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Getting cluster information and broker details from Zookeeper

6. Creating multiple-partition topic in the Kafka cluster

EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Creating multiple-partition topic in the Kafka cluster

7. Looking at logs folders of every broker

EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Looking at logs folders of every broker

8. Producing and consuming messages in the cluster

EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Producing and consuming messages in the cluster

9. Details about topic in the cluster

EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Details about topic in the cluster

10. Simulating broker failure in the cluster

EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Simulating broker failure in the cluster

11. Summary for broker cluster and topic without replication

EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Summary for broker cluster and topic without replication

11. EXAMPLE 3 - Multiple Brokers and Topic with Replication

1. SECTION 11 Introduction

EXAMPLE 3 - Multiple Brokers and Topic with Replication: SECTION 11 Introduction

2. Preparing for the next example with replication

EXAMPLE 3 - Multiple Brokers and Topic with Replication: Preparing for the next example with replication

3. Launching brokers and creating topic with replication

EXAMPLE 3 - Multiple Brokers and Topic with Replication: Launching brokers and creating topic with replication

4. Observing logs folder and details of the topic

EXAMPLE 3 - Multiple Brokers and Topic with Replication: Observing logs folder and details of the topic

5. Producing and consuming messages in the topic with replication

EXAMPLE 3 - Multiple Brokers and Topic with Replication: Producing and consuming messages in the topic with replication

6. Observing how messages were stored in the partitions on different brokers

EXAMPLE 3 - Multiple Brokers and Topic with Replication: Observing how messages were stored in the partitions on different brokers

7. Bringing down one of three brokers and observing changes

EXAMPLE 3 - Multiple Brokers and Topic with Replication: Bringing down one of three brokers and observing changes

8. Bringing down another broker in the cluster

EXAMPLE 3 - Multiple Brokers and Topic with Replication: Bringing down another broker in the cluster

9. Bringing back both brokers

EXAMPLE 3 - Multiple Brokers and Topic with Replication: Bringing back both brokers

10. Summary for replication

EXAMPLE 3 - Multiple Brokers and Topic with Replication: Summary for replication

12. EXAMPLE 4 - Kafka Consumer Groups

1. SECTION 12 Introduction

EXAMPLE 4 - Kafka Consumer Groups: SECTION 12 Introduction

2. Example with consumer groups overview

EXAMPLE 4 - Kafka Consumer Groups: Example with consumer groups overview

3. Exploring default consumer groups

EXAMPLE 4 - Kafka Consumer Groups: Exploring default consumer groups

4. Starting consumer in the custom consumer group

EXAMPLE 4 - Kafka Consumer Groups: Starting consumer in the custom consumer group

5. Starting second consumer in the same consumer group

EXAMPLE 4 - Kafka Consumer Groups: Starting second consumer in the same consumer group

6. Launching one more consumer in the same group

EXAMPLE 4 - Kafka Consumer Groups: Launching one more consumer in the same group

7. Idle consumers in the group

EXAMPLE 4 - Kafka Consumer Groups: Idle consumers in the group

8. Summary for consumer groups

EXAMPLE 4 - Kafka Consumer Groups: Summary for consumer groups

13. EXAMPLE 5 - Performance Testing

1. SECTION 13 Introduction

EXAMPLE 5 - Performance Testing: SECTION 13 Introduction

2. Overview of the performance testing example

EXAMPLE 5 - Performance Testing: Overview of the performance testing example

3. Starting cluster and launching basic performance test

EXAMPLE 5 - Performance Testing: Starting cluster and launching basic performance test

4. Increasing performance test parameters

EXAMPLE 5 - Performance Testing: Increasing performance test parameters

5. Testing consumer performance

EXAMPLE 5 - Performance Testing: Testing consumer performance

6. Getting non-zero LAG values for consumers

EXAMPLE 5 - Performance Testing: Getting non-zero LAG values for consumers

7. Performance test example summary

EXAMPLE 5 - Performance Testing: Performance test example summary

14. PROJECT 1 - Java

1. SECTION 14 Introduction

PROJECT 1 - Java: SECTION 14 Introduction

2. Project Files for the Java section

PROJECT 1 - Java: Project Files for the Java section

3. Installing IntelliJ editor

PROJECT 1 - Java: Installing IntelliJ editor

4. Creating and configuring Maven project

PROJECT 1 - Java: Creating and configuring Maven project

5. Starting Kafka Cluster

PROJECT 1 - Java: Starting Kafka Cluster

6. Creating Java Producer

PROJECT 1 - Java: Creating Java Producer

7. Continue Creating Java Producer

PROJECT 1 - Java: Continue Creating Java Producer

8. Launching Java Producer

PROJECT 1 - Java: Launching Java Producer

9. Explaining most common Producer parameters

PROJECT 1 - Java: Explaining most common Producer parameters

10. Modifying Serializer type

PROJECT 1 - Java: Modifying Serializer type

11. Producing meaningful messages with delay

PROJECT 1 - Java: Producing meaningful messages with delay

12. Refactoring Producer by adding previous example

PROJECT 1 - Java: Refactoring Producer by adding previous example

13. Creating consumer with autocommitting - PART 1

PROJECT 1 - Java: Creating consumer with autocommitting - PART 1

14. Creating consumer with autocommitting - PART 2

PROJECT 1 - Java: Creating consumer with autocommitting - PART 2

15. Consumer parameters overview

PROJECT 1 - Java: Consumer parameters overview

16. Consumer with Manual Committing

PROJECT 1 - Java: Consumer with Manual Committing

17. Consumer with Partitions Assignment

PROJECT 1 - Java: Consumer with Partitions Assignment

18. Launching multiple consumers in the same consumer group

PROJECT 1 - Java: Launching multiple consumers in the same consumer group

19. CHALLENGE - Subscribe vs Assign with consumer groups

PROJECT 1 - Java: CHALLENGE - Subscribe vs Assign with consumer groups

15. PROJECT 2 - Node.js

1. SECTION 15 Introduction

PROJECT 2 - Node.js: SECTION 15 Introduction

2. Installing Node.js with NPM

PROJECT 2 - Node.js: Installing Node.js with NPM

3. Starting up Kafka cluster with 3 brokers

PROJECT 2 - Node.js: Starting up Kafka cluster with 3 brokers

4. Initializing Node.js project

PROJECT 2 - Node.js: Initializing Node.js project

5. Final Node.js project filesv

PROJECT 2 - Node.js: Final Node.js project files

6. Creating basic Node.js producer

PROJECT 2 - Node.js: Creating basic Node.js producer

7. Producing random animal names

PROJECT 2 - Node.js: Producing random animal names

8. Creating Node.js consumer

PROJECT 2 - Node.js: Creating Node.js consumer

16. PROJECT 3 - Python

1. SECTION 16 Introduction

PROJECT 3 - Python: SECTION 16 Introduction

2. Installing Python

PROJECT 3 - Python: Installing Python

3. Final Python project files

PROJECT 3 - Python: Final Python project files

4. Launching basic Python producer

PROJECT 3 - Python: Launching basic Python producer

5. Launching consumer and receiving messages

PROJECT 3 - Python: Launching consumer and receiving messages

6. Generating fake names in the messages by producer

PROJECT 3 - Python: Generating fake names in the messages by producer

17. Course Summary

1. Course Summary

Course Summary: Course Summary

Course Images

Apache Kafka Complete Developer's Guide

By Packt

Booking options

Highlights