Kafka Streams API for Developers Using Java/Spring Boot 3.X

1. Course Introduction

In this video, the author introduces you to the course. He also takes you through the contents of the course along with who can best benefit from this course.

2. Prerequisites

In this video, the author will outline the prerequisites required for this course. These prerequisites include having Java 17 or a higher version installed on your system. It is essential to have prior experience working with Java and Docker. Additionally, a solid understanding of Kafka and Spring Boot is necessary to fully benefit from this course.

1. Introduction to Kafka Streams

In this video, the author will give an introduction to Kafka Streams and the different apps that we can build using Kafka Streams API.

2. Kafka Streams Terminologies - Topology and Processor

In this video, the author will explain the different terminologies in Kafka Streams API and how the Kafka streams applications are modelled.

3. Introduction to KStreams API

Gain a deeper understanding of the KStreams API as the author provides a concise overview, exploring its key features and functionalities.

1. Overview of the Greetings App

In this video, the author will give an overview of the app we are going to build.

2. Set Up the Greetings App

In this video, we will set up the base project for the greetings app in the local system by cloning the repository.

3. Topology of the Greetings App

In this video, we will code the Greetings app Kafka streams app, which will hold the topology of converting the message from lowercase to uppercase.

4. Build the Kafka Streams Launcher Application

In this video, we will create a Launcher class that will help execute the topology for our Greetings app.

5. Setting Up Kafka Environment and Testing Our Greeting App

In this video, we will set up the local Kafka environment and test our Greetings Kafka Streams application.

1. Filter and FilterNot

In this video, you will learn, code and explore the filter and filterNot operator using KStreams API.

2. Map/MapValues

In this video, we will code and explore the map and MapValues operator that's part of the Kafka streams using KStreams API.

3. FlatMapValues/FlatMap

In this video, we will code and explore flatMap and flatMapValues operators. The flatMap operator can be used when a single event is going to create a multiple possible event downstream.

4. peek

In this video, we will code and explore the peek operator using which we can access the events between the operators. This operator is handy when you must debug the processing logic.

5. merge

In this video, we will code and explore the merge operator to combine two Kafka Streams. This operator is used to combine two independent Kafka Streams into a single Kafka Stream.

1. How Key/Value Serialization and Deserialization Works in Kafka Streams

In this video, we will code and learn about serdes and how Kafka serialization and deserialization works.

2. Providing Default Serializer/Deserializer Using Application Configuration

In this video, we will code and learn how to provide a default serializer and deserializer to the Kafka Streams app.

3. Build Custom Serdes for Enhanced Greeting Messages

In this video, we will build custom Serdes for the new greeting message.

4. Usage of Custom Serde in the Greeting App

In this video, we will integrate Custom Serde that we created in the previous lecture for the new greeting message.

1. Build a Generic Serializer/Deserializer

In this video, we will build a generic JSON serializer/deserializer for the new greeting message.

2. Integrate Generic Serializer/Deserializer into the Greeting App

In this video, we will integrate the generic custom serdes for the new greeting message.

1. Overview of the Retail App

In this video, the author will give you an overview of the app that we are going to build in this section.

2. Build the Topology for the Orders Management App

In this video, we will build and implement the orders topology to start reading from the Orders Kafka topic.

3. Split the Restaurant/Retail Shopping Orders - Using Split and Branch Operator

In this video, we will explore the split and branch operator to branch the orders into two different topics.

4. Transform the Order Domain to a Revenue Domain Type

In this video, we will explore a new business requirement in our Orders kafka stream app. For this requirement, we don't need to the whole order but just the transaction amount that's been made at each store for each order.

1. Internals of Topology, Stream, and Tasks

In this video, the author will explain the internals of the topology, stream, tasks and thread.

2. Explore the Behavior of Streams by Modifying the Stream Threads

In this video, we will modify the num.stream.threads property and observe the behavior of a Kafka Streams application.

1. Failures in Kafka Streams

In this video, the author will introduce you to different failure points in a Kafka Streams application.

2. Default Deserialization Error Behavior

In this video, we will explore the default error behavior and the approaches to modify the custom error behavior in a Kafka Streams app.

3. Custom Deserialization Error Handler

In this video, we will code and implement the custom error handler for handling deserialization errors in a Kafka streams app.

4. Default and Custom Processor Error Handler

In this video, we will code and implement the custom error handler for handling processor errors in a Kafka Streams app.

5. Custom Production Error Handler

In this video, we will code and implement the custom error handler for handling production/serialization errors in a Kafka Streams app.

6. Error Handling When Kafka Cluster Is Down

In this video, we will explore the behavior of the Kafka Streams application if the Kafka cluster is down.

1. Introduction to KTable API

In this video, the author will give an introduction to KTable and how it works.

2. Build a Topology for KTable

In this video, we will code and explore KTable through a hands-on exercise.

3. KTable - Under the Hood

In this video, we will dive into the details of how KTable works under the hood.

4. GlobalKTable

In this video, we will code and explore the GlobalKTable and how it works internally.

1. StateFul Operations in Kafka Streams

In this video, the author will cover the different stateful operations and operators in Kafka Streams.

2. What Is Aggregation and How It Works?

In this video, the author will cover the concept of aggregation in Kafka Streams and how it works internally.

3. Aggregation Using "count" Operator

In this video, we will code and explore the count operator by implementing a simple use-case.

4. Aggregation Using "reduce" Operator

In this video, we will code and explore the reduce operator by implementing a simple use-case.

5. Aggregation Using "aggregate" Operator

In this video, we will code and explore the aggregate operator by implementing a simple use-case.

6. Using Materialized Store for count and reduce Operators

In this video, we will code and learn about using materialized views in the count and reduce operators.

1. How to Access the Results of Aggregation

In this video, the author will explain the different options of making the results of the aggregation available to the outside world.

1. Total Number of Orders by Each Store Using "count" Operator

In this video, we will code and implement the total number of orders made by the restaurant and the other departments by each store using the count operator.

2. Total Revenue by Each Store Using "aggregate" Operator

In this video, we will code and implement the total revenue made by the restaurant and the other department by each store using the count operator.

1. Effect of null Key in Stateful Operations and Repartition of Kafka Records

In this video, the author will show the effect of null keys or values in stateful operations and re-partition that happens behind the scenes during rekeying of the records.

2. Rekeying Using the "selectKey" Operator

In this video, we will code and explore rekeying records using the selectKey and Map operator.

1. Introduction to Joins and Types of Joins in Kafka Streams

In this video, the author will give an introduction to joins and different types of joins in Kafka Streams.

2. Explore innerJoin Using "join" Operator - Joining KStream and KTable

In this video, you will learn how to perform an inner join operation between a KStream and a KTable as we dive into coding and exploring this join scenario.

3. Explore innerJoin Using "join" Operator - Joining KStream and GlobalKTable

Explore the inner join operation between a KStream and a GlobalKTable in this video. Witness the coding process and gain insights into this type of join.

4. Explore innerJoin Using "join" Operator - Joining KTable and KTable

Code and explore the inner join operation between two KTables in this informative video. Understand the process and its implications in Kafka Streams.

5. Explore innerJoin Using "join" Operator - Joining KStream and KStream

In this session, we will join two KStreams together using the inner join operation in this video. Follow along with the coding demonstration and grasp the concepts involved.

6. Joining Kafka Streams Using "leftJoin" Operator

Use the "leftJoin" operator code and explore the left join operation in Kafka Streams with the "leftJoin" operator. Gain a deeper understanding of this join type through practical examples.

7. Joining Kafka Streams Using "outerJoin" Operator

Discover the outer join operation in Kafka Streams as we delve into the "outerJoin" operator. Witness the coding process and explore the possibilities of this join.

8. Join - Under the Hood

In this video, the author will show you some of the internals of the working of the Join operator.

9. Co-Partitioning Requirements in Joins

In this video, the author will explain about the co-partitioning requirements for performing the Join operation.

1. Join Aggregate Revenue with StoreDetails KTable

In this video, we will implement a join that joins the aggregated revenue and store details.

1. Introduction to Windowing and Time Concepts

In this video, the author will introduce windowing and time concepts in Kafka Streams. Windowing is one of the most frequently used processing methods for streams of data and it allows you to bucket stateful operations by time, without which your aggregations would endlessly accumulate.

2. Windowing in Kafka Streams - Tumbling Windows

In this video, we will code and explore tumbling windows in Kafka Streams. A tumbling window is a special subtype of hopping window where windowSize and advanceSize are the same.

3. Control Emission of Windowed Results Using "suppress" Operator

In this video, you will learn to suppress the results by buffering the records with the defined window using the suppress operator.

4. Windowing in Kafka Streams - Hopping Windows

In this video, we will code and explore hopping windows in Kafka Streams. A hopping window is bound by time and defined by a window size and the size of the time block at which it advances.

5. Windowing in Kafka Streams - Sliding Windows

In this video, we will code and explore the session windows in Kafka Streams. A sliding window is time-based; for a sliding window to be triggered, some user events must happen within a defined window of time.

1. New Requirements for the Order Management Application

In this video, we will code and learn about the new requirements for the order management application.

2. Implementing a CustomTimeStamp Extractor

In this video, we will code and implement our custom timestamp extractor. To implement this, we will be working on the orders-kafka-streams-app module and then implement the timestamp extractor.

3. Aggregate "Number of Orders" by Windows

In this video, we will go ahead and implement the aggregateOrdersByCount of orders by windows using the docker-compose-down.

4. Aggregate Revenue by Windows

In this video, we will code and implement the aggregation of revenue by windows.

5. Joins on the Windowed Data

In this video, you will learn how to perform joins on the windowed aggregated data.

1. Records with Timestamps before and after the CurrentTimestamp

In this video, we will code and learn about the behavior of Kafka records that have the timestamps that are before and after the current timestamp.

1. Introduction to Spring Boot and Kafka Streams

In this video, the author gives you an introduction to Spring Boot and the benefit of using Kafka Streams app using Spring Boot.

2. Set Up the Project - Greeting Streams App Using Spring Kafka Streams

In this video, we will set up base Kafka Streams app using Spring Boot.

3. Configuring the Kafka Stream Using application.yml

In this video, we will configure the Kafka Streams app to interact with the local cluster.

4. Build the Greeting Topology

In this video, we will code and implement the simple greetings topology using Spring Boot.

5. Test Greeting App in Local

In this video, we will test the topology end to end in our local environment.

1. Internals of Autoconfiguring Kafka Streams in Spring Boot

In this video, the author will walk you through the internals of how Spring Boot auto-configures Kafka Streams applications.

1. JSON Serialization/Deserialization Using JsonSerde

In this video, we will code and implement the JSON serialization and deserialization in Kafka Streams using Spring Boot.

2. JsonSerde Using Custom ObjectMapper

In this video, we will code and implement the custom ObjectMapper and pass it to the JsonSerde.

1. Handle Deserialization Error - Approach 1

In this video, we will code and explore handling deserialization errors.

2. Handle Deserialization Error Using Custom Error Handler - Approach 2

In this video, we will code and explore handling deserialization errors using a custom error handler.

3. Handle Deserialization Errors Using Spring Specific Approach- Approach 3

In this video, we will code and explore handling deserialization errors using a Spring approach.

4. Handle Uncaught Exceptions in the Topology

In this video, we will code and explore handling Uncaught exceptions in a Kafka Streams application using Spring Boot.

5. Handle Production Errors

In this video, we will code and explore handling exceptions when producing messages to the output topic.

1. Set Up the Base Project for Orders Kafka Streams App

In this video, we will set up the base project for the orders Kafka Streams application using Spring Boot.

2. Create the OrdersTopology

In this video, we will set up the topology for the orders-management-streams application.

1. Build a GET Endpoint to Retrieve the OrderCount by OrderType - Part 1

In this video, we will code and build a GET endpoint to retrieve the order count by order type.

2. Build a GET Endpoint to Retrieve the OrderCount by OrderType - Part 2

In this video, we will code and build a GET endpoint to retrieve the order count by order type.

3. Retrieve OrderCount by OrderType and LocationId

In this video, we will update the endpoint to retrieve order count by locationId and OrderType.

4. Build a GET Endpoint to Retrieve the OrderCount for All OrderTypes

In this video, we will build a new endpoint that is not going to take any location ID or the OrderType but is going to give a complete picture of all the OrderTypes.

5. Build a GET Endpoint to Retrieve the Revenue by OrderType

In this video, we will build a GET endpoint to retrieve the aggregated revenue by order type.

6. Global Error Handling for Useful Client Error Messages

In this video, we will implement the global error handler, using which we can provide a custom error message to the client.

1. Build a GET Endpoint to Retrieve OrderCount by OrderType

In this video, we will code and learn about the approach to query the windowed state store to retrieve the orders count by orderType.

2. Build a GET Endpoint to Retrieve the Windowed OrderCount for All OrderTypes

In this video, we will code and learn about querying windowed state store and retrieve the order count for all of the order types.

3. Build a GET Endpoint to Retrieve the Windowed OrderCount within a Time Range

In this video, we will code and learn about querying windowed state store within a specific range.

4. Build a GET Endpoint to Retrieve the Revenue by OrderType

In this video, we will dive into coding and explore the approach to query the windowed state store for retrieving the count of orders.

1. Testing Kafka Streams Using TopologyTestDriver

In this video, the author will give you an introduction to testing Kafka Streams.

2. Unit Testing Greetings App - Writing Data to a Output Topic

In this video, we will code and learn about writing a JUnit to test the greetings app.

3. Unit Testing Greetings App - Testing Multiple Messages

In this video, we will code and learn to write a test case to publish multiple messages and assert on the output topic with multiple messages.

4. Unit Testing Greetings App - Error Scenario

In this video, we will code and learn to write a test case to publish an error message that's going to throw an exception in the topology.

5. Unit Testing OrdersCount - Writing Data to a State Store

In this video, we will code and learn to write a test case to test orders count in our orders Kafka Streams app.

6. Unit Testing OrdersRevenue - Writing Data to a State Store

In this video, we will code and learn to write a test case to test orders revenue in our orders Kafka Streams app.

7. Unit Testing OrdersRevenue by Windows - Writing Data to a State Store

In this video, we will code and learn to write a test case to test orders revenue by windows in our orders Kafka Streams app.

8. Limitations of TopologyTestDriver

In this video, you will learn about the limitations of the TopologyTestDriver and explore it using a simple example.

1. Unit Test Using TopologyTestDriver in Spring Boot

In this video, we will code and learn about how to use TopologyTestDriver to test the topology in a Spring Boot application.

1. Introduction and Set Up Integration Test

In this video, the author will give an introduction to the integration test and set the integration test for our Kafka Streams app.

2. Integration Test for OrdersCount

In this video, we will write the integration test for the Orders Count functionality in our topology.

3. Integration Test for OrdersRevenue

In this video, we will write the integration test for the Orders Revenue functionality in our topology.

4. Integration Test for OrdersRevenue By Windows

In this video, we will write the integration test for the Orders Revenue by windows functionality in our topology.

1. Grace Period in Windowing

In this video, the author will introduce you to the concept of named grace periods in Kafka Streams and the use cases where this may be needed.

1. Package the Spring Boot App and Execute It as a Jar File

In this video, we will package the Spring Boot app as an executable jar and launch the app in our local.

1. What Is Exactly Once Processing and Why Is It Needed?

In this video, you will learn the importance of exactly once processing in Kafka Streams and understand the key concepts that are needed to implement them.

2. Set Up Exactly Once Processing in Kafka Streams

In this video, we will code and learn how to implement "exactly once processing" in a Kafka Streams app.

3. Transactions and Idempotent Producer - Under the Hood

In this video, you will learn about the internals of how Kafka Streams achieves exactly once processing.

4. Limitations and Performance Impacts of Kafka Transactions

In this video, you will learn and explore the limitations and performance impacts of Kafka transactions.

1. Running Kafka Streams Applications as Multiple Instances

In this video, the author will give some insights on the behavior of running multiple instances of the Kafka Streams app and the different challenges that are linked with it.

2. Set Up to Run the Kafka Streams as Multiple Instances

In this video, we will run multiple instances of the Kafka Streams app in our local.

3. Kafka Streams Metadata

In this video, we will code and learn about the Kafka Streams Metadata and how to access it.

4. Aggregate Data from Multiple Instances - Overview

In this video, the author will give an overview of the approach to aggregate data from multiple instances.

5. Aggregate Data from Multiple Instances - Fetching Metadata - Part 1

In this video, we are going to implement the logic for the OrderCount by OrderType.

6. Aggregate Data from Multiple Instances - Building RestClients - Part 2

In this video, we will build RestClients and make the restcall to get the data from another instance.

7. Aggregate Data from Multiple Instances - Testing End to End - Part 3

In this video, we will test the functionality to see if it is working as expected.

8. Key-Based Queries with Multiple Instances - Overview

In this video, the author will give an overview of key-based queries in a Kafka Streams app, which has multiple instances of the app.

9. Key-Based Queries Multiple Instances - Fetching Metadata- Part 1

In this video, we will code and learn about the technique to fetch the metadata about where the key is located.

10. Key-Based Queries Multiple Instances - Building RestClient and Testing- Part 2

In this video, we will build the RESTClient and then integrate it into our key-based query flow.

11. Refactor the Code for OrderCount for All OrderTypes Endpoint - /v1/orders/count

In this video, we will fix the get All OrdersCount endpoint to respond with the expected data.

12. Fix the Test Cases

In this video, we will fix the text cases that's broken with the changes that we made with distributed querying.

13. What about the Other Endpoints?

In this video, the author will explain the state of the other endpoints and how to deal with them if we run multiple instances of the Kafka Streams application.