Apache Kafka A-Z with Hands-On Learning

1. Introduction

This video presents an overview of the course.

1. Learning Objectives

This video highlights the learning objectives of this section.

2. Data Evolution over Time

This video explains how data has evolved over time.

3. Big Data Overview

This video presents an overview of big data.

4. Big Data Analytics

This video introduces you to big data analytics.

5. Messaging System

This video explains the basic messaging system and how it works. It also highlights the issues and how Kafka resolves those issues.

6. Introduction and Features of Kafka

This video introduces you to Kafka and its features.

7. Kafka Use Cases

This video explains Kafka use cases, such as messaging, metrics and logging, stream processing, and more.

8. Kafka Terminologies

This video introduces you to Kafka terminologies, such as message, batch, and more.

9. Kafka Components

This video explains the different types of Kafka components.

10. Kafka Architecture

This video presents an overview of Kafka's architecture.

11. Kafka Clusters

This video explains the concept of Kafka clusters.

12. Kafka Industry Use Cases

This video highlights the concept of Kafka industry use cases.

13. Demo - Installing Zookeeper

This video demonstrates how to install Apache Zookeeper.

14. Demo - Installing Kafka

This video explains how to install Apache Kafka.

15. Demo - Setting up a Single Node and Single Broker Cluster

This video demonstrates how to set up a single node and a single broker cluster.

16. Demo - Setting up a Multi-node Single Broker Cluster

This video explains how to set up a multi-node single broker cluster.

17. Key Takeaways

This video summarizes what you have learnt in this section.

1. Learning Objective

This video explains the learning objectives of this section.

2. Overview of a Kafka Producer and Its Architecture

This video presents an overview of a Kafka producer and its architecture.

3. Kafka Producer Configuration

This video explains the Kafka producer configuration.

4. Kafka Producer Optional Configuration

This video explains the optional configuration of a Kafka producer.

5. Kafka Producer Configuration Objects

This video introduces you to Kafka producer configuration objects.

6. Demo - Creating a Kafka Producer

This video demonstrates how to create a Kafka producer.

7. Sending Messages

This video focuses on how a Kafka producer sends messages to broker and explains the different ways of sending messages.

8. Serializers and Custom Serializers

This video introduces you to serializers and custom serializers.

9. Demo - Creating a Custom Serializer

This video demonstrates how to create a custom serializer.

10. Serializers Challenges and Serializing Using Apache Avro

This video focuses on serializers challenges and explains how to serialize using Apache Avro.

11. Demo - Serializing Using Apache Avro

This video demonstrates how to serialize using Apache Avro.

12. Partitions

This video highlights the concept of partitions in Kafka.

13. Demo - Setting up a Custom Partition

This video explains how to set up a custom partition.

14. Key Takeaway

This video highlights what you have learnt from this section.

1. Learning Objective

This video explains the learning objective of this section.

2. Overview of Kafka Consumers

This video presents an overview of Kafka consumers.

3. Consumer Groups

This video highlights the concept of consumer groups.

4. Partition Rebalance and Creating a Kafka Consumer

This video introduces you to partition rebalance and explains how to create a consumer.

5. Poll Loop and Its Functioning

This video focuses on the concept of a poll loop and explains its functioning.

6. Configuring a Kafka Consumer - Part 1

This is the first part of the two-part video that explains properties, such as fetch.min.bytes, fetch.max.wait.ms, and so on. It also explains how to configure a Kafka consumer.

7. Configuring a Kafka Consumer - Part 2

This is the second part of the two-part video that explains properties, such as auto.offset.reset, enable.auto.commit, and so on. It also explains how to configure a Kafka consumer.

8. Demo - Creating a Kafka Consumer

This video demonstrates how to create a Kafka consumer.

9. Commit and Offset

This video explains the concepts of commit and offset in Kafka.

10. Ways of Committing Offset - Automatic Offset

This video shows how to commit offset using automatic offset.

11. Ways of Committing Offset - Commit Current Offset

This video demonstrates how to commit offset using commit offset.

12. Ways of Committing Offset - Asynchronous Commit

This video explains how to commit offset using asynchronous offset.

13. Ways of Committing Offset - Combining Synchronous and Asynchronous Commits

This video shows how to commit offset using synchronous and asynchronous commits.

14. Ways of Committing Offset - Commit Specified Offset

This video demonstrates how to commit offset using commit specified offset.

15. Rebalance Listeners

This video explains the concept of rebalance listeners in Apache Kafka.

16. Consuming Records with Specific Offset

This video demonstrates how to consume records with specific offset.

17. Deserializers

This video introduces you to deserializers in Kafka.

18. Demo - Creating and Using Custom Deserializer

This video demonstrates how to create and use custom deserializer.

19. Key Takeaways

This video highlights what you have learnt in this section.

1. Learning Objective

This video explains the learning objectives of this section.

2. Kafka Internals Overview

This video presents an overview of Kafka Internals.

3. Replication and Replica Types

This video focuses on replication and explains the different types of replica.

4. Preferred Ladder, Request, and Request Processing

This video presents an overview of Preferred Ladder, request, and request Processing.

5. Types of Requests

This video presents an overview of the different types of requests.

6. Partition Allocation, File Management, and Segments

This video introduces you to partition allocation, file management, and segments in Apache Kafka.

7. File Format, Index, and Compaction

This video highlights the concepts of file format, index, and compaction.

8. Kafka Reliability and Reliability Methods

This video presents an overview of Kafka reliability and reliability methods.

9. Broker Configuration for Replication

This video explains the broker configuration for replication.

10. Producer in a Reliable System

This video explains the concept of a producer in a reliable system.

11. Consumer in a Reliable System

This video focuses on the concept of a consumer in a reliable system.

12. Key Takeaways

This video highlights what you have learnt in this section.

1. Learning Objective

This video explains the learning objectives of this section.

2. Cluster Mirroring

This video presents an overview of cluster mirroring.

3. Multi-cluster Architecture - Hub, Spokes, and Active Architecture - Part 1

This is the first part of the two-part video that focuses on multi-cluster architecture, such as hub, spokes, and active architecture.

4. Multi-cluster Architecture - Active/Stand-by Architecture and Stretch Clusters Part - 2

This is the second part of the two-part video that provides an overview of the multi-cluster architecture, such as active/stand-by architecture and stretch clusters.

5. Configuring MirrorMaker - Part 1

This video is the first part of the two-part video that demonstrates how to configure MirrorMaker in Apache Kafka.

6. Configuring MirrorMaker - Part 2

This video is the second part of the two-part video that explains how to configure MirrorMaker in Apache Kafka.

7. Demo - Setting up MirrorMaker

This video demonstrates how to set up MirrorMaker in Kafka.

8. Administering Kafka - Topic Operations

This video presents Administering Kafka and using different CLI options.

9. Administering Kafka - Consumer Group Operations

This video provides an overview of consumer group operations

10. Dynamic Configuration Changes

This video focuses on dynamic configuration changes.

11. Partition Management

This video provides an overview of partition management.

12. Console Producer Tool

This video introduces you to the console producer tool.

13. Console Consumer Tool

This video provides an introduction to the console consumer tool.

14. Key Takeaways

This video focuses on what you have learnt in the section.

1. Learning Objective

This video presents the learning objectives of this section.

2. Monitoring and Its Importance

This video presents an overview of monitoring and its importance.

3. Server or Infrastructure Monitoring and Application Monitoring

This video provides details on server or infrastructure monitoring and application monitoring.

4. Monitoring Kafka

This video demonstrates how to monitor Kafka.

5. Kafka Broker Metrics - Under-replicated Partitions - Part 1

This is the first part of the two-part video that focuses on Kafka broker metrics, such as under-replicated partitions.

6. Kafka Broker Metrics - Others- Part 2

This video provides an overview of Kafka broker metrics - Active Controller Count.

7. Topic and Partition Specific Metrics

This video focuses on topic and partition specific metrics.

8. Logging and Client Monitoring

This video focuses on logging and client monitoring.

9. Producer and Consumer Metrics

This video focuses on producer and consumer metrics.

10. Quotas and Lag Monitoring

This video focuses on quotas and lag monitoring.

11. Monitoring Dashboard

This video presents an overview of the monitoring dashboard.

12. Demo - Setting up an Open Source Health Monitor

This video demonstrates how to set up an open-source health monitor.

13. Kafka Schema Registry

This video introduces you to the Kafka Schema Registry.

14. Kafka Components and Its Architecture

This video presents an overview of Kafka components and its architecture.

15. Kafka Schema Registry - Internal working and Use cases

This video provides an overview of the Kafka Schema Registry, focusing on internal working and use cases.

16. Working of the Kafka Schema Registry

This video explains the working of the Kafka Schema Registry.

17. Demo - Using the Kafka Schema Registry with Kafka

This video demonstrates how to use the Kafka Schema Registry with Kafka.

18. Key Takeaway

This video highlights what you have learnt from this section.

1. Learning Objective

This video presents the learning objectives of this section.

2. Kafka Streams Overview

This video presents an overview of Kafka Streams.

3. Kafka Streams

This video explains more about Kafka Streams.

4. Kafka Streams Architecture and Working

This video explains the Kafka Streams architecture and its working.

5. Kafka Streams Components

This video focuses on the Kafka Streams components.

6. Kafka Streams Architecture Tasks, Threading Model and Local State Store

This video focusses on Kafka Streams architecture tasks, the threading model, and the local state store.

7. Kafka Streams Architecture - Record Buffer

This video provides an overview of Kafka Streams architecture, focuses on record buffer.

8. Memory Management and Streaming Data Pipeline

This video provides an overview of memory management and streaming data pipeline.

9. Kafka Streams Domain Specific Language (DSL)

This video introduces you to Kafka Streams DSL.

10. KStream Operations

This video presents an overview of KStream operations.

11. KTable Operations

This video focuses on KTable operations.

12. KTable Operations

This video explains more about KTable operations.

13. Aggregation and Windowing

This video explains the concept of aggregation and windowing.

14. Processor Topology and Stream Processor

This video introduces you to processor topology and stream processor.

15. Stream and Processor Application Programming Interfaces (APIs)

This video focuses on Stream and Processor APIs.

16. Processor Application Programming Interfaces (APIs) and Creating Topology

This video provides an overview of Processor APIs and explains how to create topology.

17. Kafka Connectors

This video explains the concept of Kafka Connectors.

18. Stand-alone and Sink Connector Configuration

This video explains how to configure stand-alone and sink connectors.

19. Running Kafka Connect

This video demonstrates how to run Kafka Connect.

20. Kafka Connector Distributed Mode

This video focuses on the Kafka Connector distributed mode.

21. Hypertext Transfer Protocol (HTTP) Representational State Transfer (REST) Interface

This video explains the HTTP REST interface.

22. Demo - Kafka Connector

This video demonstrates the working of a Kafka Connector.

23. Demo - Creating an Application Using Kafka Streams

This video demonstrates how to create an application using Kafka Streams.

24. Key Takeaway

This video highlights what you have learnt from this section.

1. Learning Objective

This video explains the learning objectives of this section.

2. Apache Storm

This video provides an overview of Apache Storm.

3. Real-time Analytics

This video provides an overview of real-time analytics.

4. Apache Storm Architecture

This video explains the architecture of Apache Storm.

5. Apache Storm Components

This video introduces you to the components of Apache Storm.

6. Apache Storm Topology Part - 1

This is the first part of the two-part video that presents an overview of the topology of Apache Storm.

7. Apache Storm Topology Part - 2

This is the second part of the two-part video that explains more about the topology of Apache Storm.

8. Kafka Spout

This video presents an overview of Kafka Spout.

9. Integration of Apache Storm and Kafka

This video explains the integration of Apache Storm and Kafka.

10. Demo - Simple Stand-alone Application Using Kafka and Storm

This video provides a demonstration of a simple stand-alone application using Kafka and Storm.

11. Key Takeaway

This video highlights what you have learnt from this section.

1. Learning Objective

This video presents the learning objectives of this section.

2. Introduction to Spark

This video provides an introduction to Spark.

3. Spark Components

This video explains the components of Spark.

4. Resilient Distributed Dataset

This video focuses on a resilient distributed dataset.

5. Resilient Distributed Dataset (RDD) Operations - Transformation - Map, FlatMap, and Filter

This video provides an overview of RDD operations, focusing on transformations such as map, FlatMap, and filter.

6. RDD Operations - Transformation - Join, Distinct, First, and Take

This video explains RDD operations, focusing on transformations such as join, distinct, first, and take.

7. Resilient Distributed Dataset (RDD) Operations - Actions

This video explains RDD operations, focusing on actions.

8. Data Sets and SparkSession

This video provides an introduction to data sets and SparkSession.

9. Data Sets and Spark Session Operations

This video explains more about data sets and SparkSession operations.

10. Spark Stream

This video provides an overview of Spark Stream.

11. Integrating Spark with Kafka

This video demonstrates how to integrate Spark with Kafka.

12. Demo - Running a Small Stand-alone Application in Spark with Kafka

This video explains how to run a small stand-alone application in Spark with Kafka.

13. Flume

This video introduces you to the stream capturing tool called Flume.

14. Flume Connectors

This video provides an overview of Flume Connectors.

15. Configuring Flume Kafka to Hadoop Distributed File System (HDFS)

This video explains how to configure Flume Kafka to HDFS.

16. Demo - Creating a Flume Agent and Sending Data from Kafka to Hadoop Distributed File System (HDFS)

This video demonstrates how to create a flume agent and send data from Kafka to HDFS.

17. Key Takeaways

This video highlights what you have learnt in this section.

1. Learning Objective

This video explains the learning objective of this course.

2. AdminClient

This video introduces you to AdminClient.

3. Demo - Performing Various Admin Tasks Using AdminClient

This video demonstrates how to perform various admin tasks using AdminClient.

4. Kafka Security

This video presents an overview of Kafka Security.

5. Kafka Security Components

This video introduces you to the components of Kafka Security.

6. Simple Authentication and Security Layer (SASL)

This video highlights the concept of SASL in Kafka.

7. Configuring Simple Authentication and Security Layer (SASL) in Kafka

This video explains how to configure SASL in Kafka.

8. Securing Using Access Control Lists (ACLs)

This video demonstrates how to secure Kafka using ACLs.

9. Key Takeaway

This video highlights what you have learnt from this course.