Name: Master Big Data Ingestion and Analytics with Flume, Sqoop, Hive and Spark
Price: 70.99 GBP
Availability: InStock

<strong>A carefully structured advanced-level course on Apache Spark 3 to help you clear your job interviews. This course covers advanced topics and concepts that are part of the Databricks Spark certification exam. Boost your skills in Spark 3 architecture and memory management.</strong>
        <p>Apache Spark is a lightning-fast unified analytics engine for big data and machine learning. Since its release, Apache Spark has seen rapid adoption by enterprises across a wide range of industries. Internet powerhouses such as Netflix, Yahoo, and eBay have deployed Spark at a massive scale. It has quickly become the largest open-source community in big data. So, mastering Apache Spark opens a wide range of professional opportunities.

This course covers some advanced topics and concepts such as Spark 3 architecture and memory management, AQE, DPP, broadcast, accumulators, and multithreading in Spark 3 along with common job interview questions and answers. The objective of this course is to prepare you for advanced certification topics.

By the end of this course, you will have learned some advanced topics and concepts that are asked for in the Databricks Spark Certification or Spark job interviews. This will not only help you develop advanced skills in Apache Spark but also crack your job interviews.</p>
        <h2>What You Will Learn</h2><p>Learn AQE, DPP, Broadcast in Spark 3<br>Explore Spark 3 architecture<br>Understand memory management in Spark 3<br>Learn Spark AQE Dynamic Join Optimization<br>Explore accumulators and multithreading in Spark<br>Discover Spark Dynamic Partition Pruning</p>
        <h2>Audience</h2><p>This course is for anyone who wants to learn and develop advanced skills in Apache Spark or those who are preparing for job interviews and want to learn advanced skills.<br><br>Before proceeding with the course, you will need basic knowledge of Spark programming in Python - PySpark.</p>
        <h2>Approach</h2><p>This is a short and crisp yet comprehensive course on cracking your job interviews on Spark programming and around the topic of Apache Spark 3 architecture and memory management. Leverage your knowledge with some tips/tricks profoundly used in the corporate environment and reinforce your learning with four sets of quizzes.</p>
        <h2>Key Features</h2><p>Practice common job interview questions and answers * Deep dive into Spark 3 architecture and memory management * Prepare for Databricks Spark certification in a structured way</p>
        
        <h2>About the Author</h2><h3>ScholarNest </h3><p>ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills.

Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects.

It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.</p>
        <h2>Course Outline</h2><h4>1. Before You Begin</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Course Introduction</p><p style='margin: 0'>In this video, the author gives an overview of the course. He talks about the objective of the course and the prerequisites needed to continue with the course.</p></td></tr></tbody></table><br><h4>2. Spark Architecture</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Spark Cluster and Runtime Architecture</p><p style='margin: 0'>This video shows the concept of Spark cluster and runtime architecture.</p></td></tr><tr><td style='padding: 15px;'><p>2. Spark Submit and Important Options</p><p style='margin: 0'>In this video, the author describes Spark submit and important options.</p></td></tr><tr><td style='padding: 15px;'><p>3. Deploy Modes - Client and Cluster Mode</p><p style='margin: 0'>In this video, the author describes types of deploy modes in Spark.</p></td></tr><tr><td style='padding: 15px;'><p>4. Spark Jobs - Stage, Shuffle, Task, Slots</p><p style='margin: 0'>In this video, the author gives a detailed overview of the types of Spark jobs.</p></td></tr><tr><td style='padding: 15px;'><p>5. Spark SQL Engine and Query Planning</p><p style='margin: 0'>In this video, the author describes SQL engine and query planning in Spark.</p></td></tr><tr><td style='padding: 15px;'><p>6. Let's Practice - Quiz 1 Solution Video</p><p style='margin: 0'>In this video, the author discusses some frequently asked questions and answers in certification exams.</p></td></tr><tr><td style='padding: 15px;'><p>7. Let's Practice - Quiz 2 Solution Video</p><p style='margin: 0'>In this video, the author discusses more questions and answers that are commonly asked in Spark certification and other related exams.</p></td></tr></tbody></table><br><h4>3. Performance and Applied Understanding</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Spark Memory Allocation</p><p style='margin: 0'>In this video, the author describes the concept of Spark memory allocation in detail.</p></td></tr><tr><td style='padding: 15px;'><p>2. Spark Memory Management</p><p style='margin: 0'>In this video, the author shows how memory management works in Spark.</p></td></tr><tr><td style='padding: 15px;'><p>3. Spark Adaptive Query Execution</p><p style='margin: 0'>In this video, the author describes the details of Spark adaptive query execution and its capabilities and features.</p></td></tr><tr><td style='padding: 15px;'><p>4. Spark AQE Dynamic Join Optimization</p><p style='margin: 0'>In this video, the author describes AQE Dynamic Join Optimization and its importance.</p></td></tr><tr><td style='padding: 15px;'><p>5. Handling Data Skew in Spark Joins</p><p style='margin: 0'>In this video, the author discusses the concept of data skew in Spark joins and why it is used.</p></td></tr><tr><td style='padding: 15px;'><p>6. Spark Dynamic Partition Pruning</p><p style='margin: 0'>In this video, the author discusses dynamic partition pruning and its importance.</p></td></tr><tr><td style='padding: 15px;'><p>7. Data Caching in Spark</p><p style='margin: 0'>In this video, the author gives a detailed discussion of data caching and its importance.</p></td></tr><tr><td style='padding: 15px;'><p>8. Repartition and Coalesce</p><p style='margin: 0'>In this video, the author talks about repartition and coalesce in Spark.</p></td></tr><tr><td style='padding: 15px;'><p>9. Dataframe Hints</p><p style='margin: 0'>In this video, the author gives an overview of dataframe hints in Apache Spark.</p></td></tr><tr><td style='padding: 15px;'><p>10. Broadcast Variables</p><p style='margin: 0'>In this video, the author discusses in detail about Spark broadcast variables.</p></td></tr><tr><td style='padding: 15px;'><p>11. Accumulators</p><p style='margin: 0'>In this video, the author covers accumulators and its features.</p></td></tr><tr><td style='padding: 15px;'><p>12. Speculative Execution</p><p style='margin: 0'>In this video, the author gives a detailed description of Spark speculative execution.</p></td></tr><tr><td style='padding: 15px;'><p>13. Dynamic Resource Allocation</p><p style='margin: 0'>In this video, the author gives a detailed description of resource allocation and its types.</p></td></tr><tr><td style='padding: 15px;'><p>14. Spark Schedulers</p><p style='margin: 0'>In this video, the author describes Spark schedulers within an application.</p></td></tr><tr><td style='padding: 15px;'><p>15. Let's Practice - Quiz 3 Solution Video</p><p style='margin: 0'>In this video, the author discusses a few questions and answers that frequently appear in certification exams.</p></td></tr><tr><td style='padding: 15px;'><p>16. Let's Practice - Quiz 4 Solution Video</p><p style='margin: 0'>In this video, the author discusses a few questions and answers that frequently appear in certification exams.</p></td></tr></tbody></table><br><h4>4. Open-Ended Topics</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Help Me Add More Content - Demand for More</p><p style='margin: 0'>In this video, the author talks about the course and how to continue it further.</p></td></tr></tbody></table>

A carefully structured advanced-level course on Apache Spark 3 to help you clear your job interviews. This course covers advanced topics and concepts that are part of the Databricks Spark certification exam. Boost your skills in Spark 3 architecture and memory management.
Apache Spark is a lightning-fast unified analytics engine for big data and machine learning. Since its release, Apache Spark has seen rapid adoption by enterprises across a wide range of industries. Internet powerhouses such as Netflix, Yahoo, and eBay have deployed Spark at a massive scale. It has quickly become the largest open-source community in big data. So, mastering Apache Spark opens a wide range of professional opportunities. This course covers some advanced topics and concepts such as Spark 3 architecture and memory management, AQE, DPP, broadcast, accumulators, and multithreading in Spark 3 along with common job interview questions and answers. The objective of this course is to prepare you for advanced certification topics. By the end of this course, you will have learned some advanced topics and concepts that are asked for in the Databricks Spark Certification or Spark job interviews. This will not only help you develop advanced skills in Apache Spark but also crack your job interviews.
What You Will Learn
Learn AQE, DPP, Broadcast in Spark 3
Explore Spark 3 architecture
Understand memory management in Spark 3
Learn Spark AQE Dynamic Join Optimization
Explore accumulators and multithreading in Spark
Discover Spark Dynamic Partition Pruning
Audience
This course is for anyone who wants to learn and develop advanced skills in Apache Spark or those who are preparing for job interviews and want to learn advanced skills.

Before proceeding with the course, you will need basic knowledge of Spark programming in Python - PySpark.
Approach
This is a short and crisp yet comprehensive course on cracking your job interviews on Spark programming and around the topic of Apache Spark 3 architecture and memory management. Leverage your knowledge with some tips/tricks profoundly used in the corporate environment and reinforce your learning with four sets of quizzes.
Key Features
Practice common job interview questions and answers * Deep dive into Spark 3 architecture and memory management * Prepare for Databricks Spark certification in a structured way
About the Author
ScholarNest 
ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects. It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.
Course Outline
1. Before You Begin

1. Course Introduction
In this video, the author gives an overview of the course. He talks about the objective of the course and the prerequisites needed to continue with the course.
2. Spark Architecture

1. Spark Cluster and Runtime Architecture
This video shows the concept of Spark cluster and runtime architecture.
2. Spark Submit and Important Options
In this video, the author describes Spark submit and important options.
3. Deploy Modes - Client and Cluster Mode
In this video, the author describes types of deploy modes in Spark.
4. Spark Jobs - Stage, Shuffle, Task, Slots
In this video, the author gives a detailed overview of the types of Spark jobs.
5. Spark SQL Engine and Query Planning
In this video, the author describes SQL engine and query planning in Spark.
6. Let's Practice - Quiz 1 Solution Video
In this video, the author discusses some frequently asked questions and answers in certification exams.
7. Let's Practice - Quiz 2 Solution Video
In this video, the author discusses more questions and answers that are commonly asked in Spark certification and other related exams.
3. Performance and Applied Understanding

1. Spark Memory Allocation
In this video, the author describes the concept of Spark memory allocation in detail.
2. Spark Memory Management
In this video, the author shows how memory management works in Spark.
3. Spark Adaptive Query Execution
In this video, the author describes the details of Spark adaptive query execution and its capabilities and features.
4. Spark AQE Dynamic Join Optimization
In this video, the author describes AQE Dynamic Join Optimization and its importance.
5. Handling Data Skew in Spark Joins
In this video, the author discusses the concept of data skew in Spark joins and why it is used.
6. Spark Dynamic Partition Pruning
In this video, the author discusses dynamic partition pruning and its importance.
7. Data Caching in Spark
In this video, the author gives a detailed discussion of data caching and its importance.
8. Repartition and Coalesce
In this video, the author talks about repartition and coalesce in Spark.
9. Dataframe Hints
In this video, the author gives an overview of dataframe hints in Apache Spark.
10. Broadcast Variables
In this video, the author discusses in detail about Spark broadcast variables.
11. Accumulators
In this video, the author covers accumulators and its features.
12. Speculative Execution
In this video, the author gives a detailed description of Spark speculative execution.
13. Dynamic Resource Allocation
In this video, the author gives a detailed description of resource allocation and its types.
14. Spark Schedulers
In this video, the author describes Spark schedulers within an application.
15. Let's Practice - Quiz 3 Solution Video
In this video, the author discusses a few questions and answers that frequently appear in certification exams.
16. Let's Practice - Quiz 4 Solution Video
In this video, the author discusses a few questions and answers that frequently appear in certification exams.
4. Open-Ended Topics

1. Help Me Add More Content - Demand for More
In this video, the author talks about the course and how to continue it further.

Packt

Scala

Apache Spark

Big Data

A carefully structured advanced-level course on Apache Spark 3 to help you clear your job interviews. This course covers advanced topics and concepts that are part of the Databricks Spark certification exam. Boost your skills in Spark 3 architecture and memory management.

Apache Spark 3 Advance Skills for Cracking Job Interviews

General Admission

<strong>Get hands-on with Kafka monitoring setup with Prometheus and Grafana, Kafka operations and Kafka cluster upgrades Setup in AWS.</strong>
        <p>This course helps you learn Kafka administration, Kafka monitoring, Kafka operations, and Kafka upgrades. It covers the following topics:
Administration for Apache Kafka
o Setup UI tools such as Kafka Manager, Zoo Navigator, and Kafka Monitor to get a full view of your cluster
o Understand basic operations you can perform with these tools
Monitoring for Apache Kafka
o A Kafka that is not monitored is a ticking time-bomb. 
o Set up proper monitoring for Kafka and Zookeeper
o Store your metrics in Prometheus and visualize them in Grafana
Operations for Apache Kafka
o Perform the most common and difficult operations in your Apache Kafka cluster
o Roll-restart your brokers in only one command
o Update Kafka configurations safely with no downtime
o Rebalance partitions and change the replication factor of your topics
o Add, replace, and remove brokers.
Upgrades for Apache Kafka
o Understand the subtlety of upgrading a Kafka cluster
o Step-by-step explanations on upgrading a cluster with no downtime

All the code files are placed at https://github.com/PacktPublishing/Apache-Kafka-Series---Kafka-Monitoring-and-Operations</p>
        <h2>What You Will Learn</h2><p>&bull; Learn how to perform a safe and automated roll-restart of Kafka brokers<br>&bull; Update broker configurations in a safe way<br>&bull; Rebalance partitions in a Kafka cluster<br>&bull; Increase and decrease the replication factor of topics<br>&bull; Add a broker to a Kafka cluster<br>&bull; Service and replace a broker in a Kafka cluster<br>&bull; Remove a broker in a Kafka cluster<br>&bull; Install Command Line Interface (CLI) tools to automate workflows<br>&bull; Upgrade a Kafka cluster with no downtime</p>
        <h2>Audience</h2><p>Kafka administrators who want to learn how to set up monitoring, perform Kafka operations, and upgrade a Kafka cluster.</p>
        <h2>Approach</h2><p>This course will teach you Apache Kafka in depth, via practical examples.</p>
        <h2>Key Features</h2><p>&bull; Set up a multi-broker Kafka cluster in no time in AWS (using CloudFormation) * &bull; Set up administration tools such as Kafka Manager, ZooNavigator, and LinkedIn's Kafka Monitor * &bull; Set up monitoring using Grafana and Prometheus</p>
        <h2>Github Repo</h2><p>https://github.com/packtpublishing/apache-kafka-series---kafka-monitoring-and-operations</p>
        <h2>About the Author</h2><h3>Stéphane Maarek</h3><p>Stéphane Maarek is a solutions architect, consultant, and software developer who has a particular interest in all things related to big data and analytics. He is also a bestseller instructor on Udemy for his courses on Apache Kafka, Apache NiFi, and AWS Lambda. He loves Apache Kafka and regularly contributes to the Apache Kafka project.

Stéphane has also written a guest blog post that was featured on the Confluent website, the company behind Apache Kafka. He is also an AWS Certified Solutions Architect and has many years of experience with technologies such as Apache Kafka, Apache NiFi, Apache Spark, Hadoop, PostgreSQL, Tableau, Spotfire, Docker, Ansible, and more.</p>
        <h2>Course Outline</h2><h4>1. Course Overview and Introduction</h4><p></p><br><h4>2. Kafka Quick Setup in AWS</h4><p></p><br><h4>3. Kafka Administration Setup</h4><p></p><br><h4>4. Kafka Monitoring Setup: Grafana + Prometheus</h4><p></p><br><h4>5. Kafka Operations</h4><p></p><br><h4>6. Kafka Cluster Upgrade</h4><p></p><br><h4>7. Next Steps</h4><p></p>

Get hands-on with Kafka monitoring setup with Prometheus and Grafana, Kafka operations and Kafka cluster upgrades Setup in AWS.
This course helps you learn Kafka administration, Kafka monitoring, Kafka operations, and Kafka upgrades. It covers the following topics:
Administration for Apache Kafka
o Setup UI tools such as Kafka Manager, Zoo Navigator, and Kafka Monitor to get a full view of your cluster
o Understand basic operations you can perform with these tools
Monitoring for Apache Kafka
o A Kafka that is not monitored is a ticking time-bomb. o Set up proper monitoring for Kafka and Zookeeper
o Store your metrics in Prometheus and visualize them in Grafana
Operations for Apache Kafka
o Perform the most common and difficult operations in your Apache Kafka cluster
o Roll-restart your brokers in only one command
o Update Kafka configurations safely with no downtime
o Rebalance partitions and change the replication factor of your topics
o Add, replace, and remove brokers.
Upgrades for Apache Kafka
o Understand the subtlety of upgrading a Kafka cluster
o Step-by-step explanations on upgrading a cluster with no downtime All the code files are placed at https://github.com/PacktPublishing/Apache-Kafka-Series---Kafka-Monitoring-and-Operations
What You Will Learn
• Learn how to perform a safe and automated roll-restart of Kafka brokers
• Update broker configurations in a safe way
• Rebalance partitions in a Kafka cluster
• Increase and decrease the replication factor of topics
• Add a broker to a Kafka cluster
• Service and replace a broker in a Kafka cluster
• Remove a broker in a Kafka cluster
• Install Command Line Interface (CLI) tools to automate workflows
• Upgrade a Kafka cluster with no downtime
Audience
Kafka administrators who want to learn how to set up monitoring, perform Kafka operations, and upgrade a Kafka cluster.
Approach
This course will teach you Apache Kafka in depth, via practical examples.
Key Features
• Set up a multi-broker Kafka cluster in no time in AWS (using CloudFormation) * • Set up administration tools such as Kafka Manager, ZooNavigator, and LinkedIn's Kafka Monitor * • Set up monitoring using Grafana and Prometheus
Github Repo
https://github.com/packtpublishing/apache-kafka-series---kafka-monitoring-and-operations
About the Author
Stéphane Maarek
Stéphane Maarek is a solutions architect, consultant, and software developer who has a particular interest in all things related to big data and analytics. He is also a bestseller instructor on Udemy for his courses on Apache Kafka, Apache NiFi, and AWS Lambda. He loves Apache Kafka and regularly contributes to the Apache Kafka project. Stéphane has also written a guest blog post that was featured on the Confluent website, the company behind Apache Kafka. He is also an AWS Certified Solutions Architect and has many years of experience with technologies such as Apache Kafka, Apache NiFi, Apache Spark, Hadoop, PostgreSQL, Tableau, Spotfire, Docker, Ansible, and more.
Course Outline
1. Course Overview and Introduction

2. Kafka Quick Setup in AWS

3. Kafka Administration Setup

4. Kafka Monitoring Setup: Grafana + Prometheus

5. Kafka Operations

6. Kafka Cluster Upgrade

7. Next Steps


Programming

Kafka

Get hands-on with Kafka monitoring setup with Prometheus and Grafana, Kafka operations and Kafka cluster upgrades Setup in AWS.

Apache Kafka Series - Kafka Monitoring?? and Operations

<strong>A complete guide to the Cassandra architecture, the Cassandra query language, cluster management, and Java/Spark integration.</strong>
        <p>This Apache Cassandra training course teaches you to work with Cassandra. This course is intended for complete Cassandra novices.
This course covers the Cassandra architecture in detail, including replication, consistency, gossip protocol, write paths, read paths, Cassandra storage. and compaction. Once you have completed this video-based training course, you will have a solid understanding of Cassandra.
All the codes and supporting files are available at: https://github.com/PacktPublishing/Learn-Apache-Cassandra-in-Just-2-Hours</p>
        <h2>What You Will Learn</h2><p>What Cassandra is<br>How to install Cassandra<br>Understand the Cassandra data model with some hands-on exercise that will teach you how to create a keyspace, create a table, and insert and read data.<br>Different data types in Cassandra (with exercises)<br>Master partition and clustering keys and understand how data is distributed across nodes in a cluster. (We cover the Cassandra Architecture in detail, including replication, consistency, the gossip protocol, write paths, read paths, Cassandra storage, and compaction)<br>Understand Cassandra configuration files<br>Working with node tools to manage clusters<br>Integrate with the Cassandra Java driver to write and run Cassandra from Java programs.<br>Integrate Spark with Cassandra to perform analytics.</p>
        <h2>Audience</h2><p>This course is designed for beginners who want to master the NoSQL database: Apache Cassandra.</p>
        <h2>Approach</h2><p>This course is a seamless blend of videos and hands-on exercises that will help you learn efficiently as you progress through the course.</p>
        <h2>Key Features</h2><p>Master Apache Cassandra basics and its architecture * Integrate with the Cassandra Java driver to write and run Cassandra from Java programs</p>
        <h2>Github Repo</h2><p>https://github.com/packtpublishing/learn-apache-cassandra-in-just-2-hours</p>
        <h2>About the Author</h2><h3>Navdeep Kaur</h3><p>Navdeep Kaur - Technical Trainer
Navdeep Kaur is a big data professionals with 11 years of industry experience in different technologies and domains. She has a keen interest in providing training in new technologies. She has received CCA175 Hadoop and Spark developer certification and AWS solution architect certification. She loves guiding people and helping them achieves new goals.</p>
        <h2>Course Outline</h2><h4>1. Apache Cassandra Overview</h4><p></p><br><h4>2. Cassandra Architecture</h4><p></p><br><h4>3. Cassandra Configuration & Managing Cluster</h4><p></p><br><h4>4. Cassandra with Java</h4><p></p><br><h4>5. Cassandra With Spark</h4><p></p>

A complete guide to the Cassandra architecture, the Cassandra query language, cluster management, and Java/Spark integration.
This Apache Cassandra training course teaches you to work with Cassandra. This course is intended for complete Cassandra novices.
This course covers the Cassandra architecture in detail, including replication, consistency, gossip protocol, write paths, read paths, Cassandra storage. and compaction. Once you have completed this video-based training course, you will have a solid understanding of Cassandra.
All the codes and supporting files are available at: https://github.com/PacktPublishing/Learn-Apache-Cassandra-in-Just-2-Hours
What You Will Learn
What Cassandra is
How to install Cassandra
Understand the Cassandra data model with some hands-on exercise that will teach you how to create a keyspace, create a table, and insert and read data.
Different data types in Cassandra (with exercises)
Master partition and clustering keys and understand how data is distributed across nodes in a cluster. (We cover the Cassandra Architecture in detail, including replication, consistency, the gossip protocol, write paths, read paths, Cassandra storage, and compaction)
Understand Cassandra configuration files
Working with node tools to manage clusters
Integrate with the Cassandra Java driver to write and run Cassandra from Java programs.
Integrate Spark with Cassandra to perform analytics.
Audience
This course is designed for beginners who want to master the NoSQL database: Apache Cassandra.
Approach
This course is a seamless blend of videos and hands-on exercises that will help you learn efficiently as you progress through the course.
Key Features
Master Apache Cassandra basics and its architecture * Integrate with the Cassandra Java driver to write and run Cassandra from Java programs
Github Repo
https://github.com/packtpublishing/learn-apache-cassandra-in-just-2-hours
About the Author
Navdeep Kaur
Navdeep Kaur - Technical Trainer
Navdeep Kaur is a big data professionals with 11 years of industry experience in different technologies and domains. She has a keen interest in providing training in new technologies. She has received CCA175 Hadoop and Spark developer certification and AWS solution architect certification. She loves guiding people and helping them achieves new goals.
Course Outline
1. Apache Cassandra Overview

2. Cassandra Architecture

3. Cassandra Configuration & Managing Cluster

4. Cassandra with Java

5. Cassandra With Spark


Java

Database Administration

A complete guide to the Cassandra architecture, the Cassandra query language, cluster management, and Java/Spark integration.

Learn Apache Cassandra in Just 2 Hours

<strong>Get to grips with real-time stream processing using PySpark as well as Spark structured streaming and apply that knowledge to build stream processing solutions. This course is example-driven and follows a working session-like approach.</strong>
        <p>Take your first steps towards discovering, learning, and using Apache Spark 3.0. We will be taking a live coding approach in this carefully structured course and explaining all the core concepts needed along the way.

In this course, we will understand the real-time stream processing concepts, Spark structured streaming APIs, and architecture.

We will work with file streams, Kafka source, and integrating Spark with Kafka. Next, we will learn about state-less and state-full streaming transformations. Then cover windowing aggregates using Spark stream. Next, we will cover watermarking and state cleanup. After that, we will cover streaming joins and aggregation, handling memory problems with streaming joins. Finally, learn to create arbitrary streaming sinks.

By the end of this course, you will be able to create real-time stream processing applications using Apache Spark.

All the resources for the course are available at https://github.com/PacktPublishing/Real-time-stream-processing-using-Apache-Spark-3-for-Python-developers</p>
        <h2>What You Will Learn</h2><p>Explore state-less and state-full streaming transformations<br>Windowing aggregates using Spark stream<br>Learn Watermarking and state cleanup<br>Implement streaming joins and aggregations<br>Handling memory problems with streaming joins<br>Learn to create arbitrary streaming sinks</p>
        <h2>Audience</h2><p>This course is designed for software engineers and architects who are willing to design and develop big data engineering projects using Apache Spark. It is also designed for programmers and developers who are aspiring to grow and learn data engineering using Apache Spark.<br><br>For this course, you need to know Spark fundamentals and should be exposed to Spark Dataframe APIs. Also, you should know Kafka fundamentals and have a working knowledge of Apache Kafka. One should also have programming knowledge of Python programming.</p>
        <h2>Approach</h2><p>This course is example-driven and follows a working session-like approach. The course delivers live coding sessions and explains the concepts along the way.</p>
        <h2>Key Features</h2><p>Learn real-time stream processing concepts * Understand Spark structured streaming APIs and architecture * Work with file streams, Kafka source, and integrating Spark with Kafka</p>
        <h2>Github Repo</h2><p>https://github.com/PacktPublishing/Real-time-stream-processing-using-Apache-Spark-3-for-Python-developers</p>
        <h2>About the Author</h2><h3>Scholar Nest</h3><p>ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills.

Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects.

It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.</p>
        <h2>Course Outline</h2><h4>1. Before you Start</h4><p></p><br><h4>2. Setup your Environment</h4><p></p><br><h4>3. Getting started with Spark Structured Streaming</h4><p></p><br><h4>4. Spark Streaming with Kafka</h4><p></p><br><h4>5. Windowing and Aggregates</h4><p></p><br><h4>6. Stream Processing and Joins</h4><p></p><br><h4>7. Keep Learning</h4><p></p>

Get to grips with real-time stream processing using PySpark as well as Spark structured streaming and apply that knowledge to build stream processing solutions. This course is example-driven and follows a working session-like approach.
Take your first steps towards discovering, learning, and using Apache Spark 3.0. We will be taking a live coding approach in this carefully structured course and explaining all the core concepts needed along the way. In this course, we will understand the real-time stream processing concepts, Spark structured streaming APIs, and architecture. We will work with file streams, Kafka source, and integrating Spark with Kafka. Next, we will learn about state-less and state-full streaming transformations. Then cover windowing aggregates using Spark stream. Next, we will cover watermarking and state cleanup. After that, we will cover streaming joins and aggregation, handling memory problems with streaming joins. Finally, learn to create arbitrary streaming sinks. By the end of this course, you will be able to create real-time stream processing applications using Apache Spark. All the resources for the course are available at https://github.com/PacktPublishing/Real-time-stream-processing-using-Apache-Spark-3-for-Python-developers
What You Will Learn
Explore state-less and state-full streaming transformations
Windowing aggregates using Spark stream
Learn Watermarking and state cleanup
Implement streaming joins and aggregations
Handling memory problems with streaming joins
Learn to create arbitrary streaming sinks
Audience
This course is designed for software engineers and architects who are willing to design and develop big data engineering projects using Apache Spark. It is also designed for programmers and developers who are aspiring to grow and learn data engineering using Apache Spark.

For this course, you need to know Spark fundamentals and should be exposed to Spark Dataframe APIs. Also, you should know Kafka fundamentals and have a working knowledge of Apache Kafka. One should also have programming knowledge of Python programming.
Approach
This course is example-driven and follows a working session-like approach. The course delivers live coding sessions and explains the concepts along the way.
Key Features
Learn real-time stream processing concepts * Understand Spark structured streaming APIs and architecture * Work with file streams, Kafka source, and integrating Spark with Kafka
Github Repo
https://github.com/PacktPublishing/Real-time-stream-processing-using-Apache-Spark-3-for-Python-developers
About the Author
Scholar Nest
ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects. It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.
Course Outline
1. Before you Start

2. Setup your Environment

3. Getting started with Spark Structured Streaming

4. Spark Streaming with Kafka

5. Windowing and Aggregates

6. Stream Processing and Joins

7. Keep Learning


Data Engineering

Hadoop

Data Science

Python

Data Processing

Get to grips with real-time stream processing using PySpark as well as Spark structured streaming and apply that knowledge to build stream processing solutions. This course is example-driven and follows a working session-like approach.

Real-Time Stream Processing Using Apache Spark 3 for Python Developers

<strong>Learn the fundamentals and advanced concepts of Apache Kafka in this course. This course will give you a good understanding of all the concepts through hands-on practice.</strong>
        <p>This course will walk you through the Kafka concepts step by step. We will explore the Kafka internals from a theoretical perspective followed by hands-on exploring the Kafka internals. Then we will learn about building the Kafka Producer to produce messages into Kafka using the producer API. This is followed by learning the different configurations that guarantee message delivery into Kafka and building Kafka Consumer to read messages from Kafka using the consumer API.

In the next section, we will learn about consumer groups and consumer rebalance, consumer offsets, and the different strategies that are available for the consumer to commit the offsets.

Next, we will learn about consumer rebalance listeners to perform some clean-up before partition revocation or some initialization tasks during partition assignment. Then learn about the different techniques from the consumer to seek a different offset.

And finally, we will code and learn how to build custom Serializer/Deserializer and integrate it into our Kafka Producer/Consumer.

By the end of the course, you will have a complete understanding of Apache Kafka and will be able to build Kafka Consumer applications using Java.

The complete code bundle for this course is available at https://github.com/PacktPublishing/Apache-Kafka-for-Beginners---Learn-Kafka-by-Hands-On</p>
        <h2>What You Will Learn</h2><p>Understand all the Kafka concepts and Kafka core internals<br>Learn Kafka internals through practice<br>Build Kafka Producer applications using Java<br>Learn the Kafka Producer techniques for reliable data delivery<br>Build Kafka Consumer applications using Java<br>Learn to build custom Serializer and Deserializer in Kafka</p>
        <h2>Audience</h2><p>This course is designed for Kafka beginners and advanced level. Anyone interested in building Java applications using producer and consumer API or interested in learning advanced Kafka Producer and Consumer operations, then this course is for you.</p>
        <h2>Approach</h2><p>This course will walk you through the Kafka concepts step by step and all the Kafka concepts will be taught through hands-on practice.</p>
        <h2>Key Features</h2><p>Learn to build Kafka Producers/Consumers using Java * Covers advanced Kafka Producer and Consumer concepts * Hands-on oriented course</p>
        <h2>Github Repo</h2><p>https://github.com/PacktPublishing/Apache-Kafka-for-Beginners---Learn-Kafka-by-Hands-On</p>
        <h2>About the Author</h2><h3>Dilip Sundarraj</h3><p>Dilip Sundarraj is a software engineer who has experience with building software since 2008. He is passionate about learning modern technologies and staying up to date with all the modern technologies, tools, frameworks, and more. He loves to share his knowledge with the world and that is one of the key reasons for him to be in the online teaching industry.
He loves interacting with other software developers and believes that this helps him share knowledge and learn from them. During his leisure time, he loves to play cricket, watch movies, and work out in the gym to have a balance between physical and mental strength.
Dilip has a YouTube channel named Code with Dilip, where he has been sharing a lot of technical content related to languages, frameworks, best practices, and more.</p>
        <h2>Course Outline</h2><h4>1. Getting Started with Course</h4><p></p><br><h4>2. Getting Started with Kafka</h4><p></p><br><h4>3. Download and Install Kafka</h4><p></p><br><h4>4. Understanding Kafka Components and its Internals - (Theory + Hands On)</h4><p></p><br><h4>5. Kafka Producer API - Hands On</h4><p></p><br><h4>6. Kafka Producer API - Guaranteed Message Delivery Configurations</h4><p></p><br><h4>7. Kafka Consumer API - Hands On</h4><p></p><br><h4>8. Consumer Groups & Consumer Rebalance - Hands On</h4><p></p><br><h4>9. Consumer Offsets - Default and Manual Offset Management - Hands On</h4><p></p><br><h4>10. Consumer Rebalance Listeners - Hands On</h4><p></p><br><h4>11. Kafka Consumer - seekToBeginning(), seekToEnd() & seek() - Hands On</h4><p></p><br><h4>12. Custom Serializer and Deserializers in Kafka - Hands On</h4><p></p>

Learn the fundamentals and advanced concepts of Apache Kafka in this course. This course will give you a good understanding of all the concepts through hands-on practice.
This course will walk you through the Kafka concepts step by step. We will explore the Kafka internals from a theoretical perspective followed by hands-on exploring the Kafka internals. Then we will learn about building the Kafka Producer to produce messages into Kafka using the producer API. This is followed by learning the different configurations that guarantee message delivery into Kafka and building Kafka Consumer to read messages from Kafka using the consumer API. In the next section, we will learn about consumer groups and consumer rebalance, consumer offsets, and the different strategies that are available for the consumer to commit the offsets. Next, we will learn about consumer rebalance listeners to perform some clean-up before partition revocation or some initialization tasks during partition assignment. Then learn about the different techniques from the consumer to seek a different offset. And finally, we will code and learn how to build custom Serializer/Deserializer and integrate it into our Kafka Producer/Consumer. By the end of the course, you will have a complete understanding of Apache Kafka and will be able to build Kafka Consumer applications using Java. The complete code bundle for this course is available at https://github.com/PacktPublishing/Apache-Kafka-for-Beginners---Learn-Kafka-by-Hands-On
What You Will Learn
Understand all the Kafka concepts and Kafka core internals
Learn Kafka internals through practice
Build Kafka Producer applications using Java
Learn the Kafka Producer techniques for reliable data delivery
Build Kafka Consumer applications using Java
Learn to build custom Serializer and Deserializer in Kafka
Audience
This course is designed for Kafka beginners and advanced level. Anyone interested in building Java applications using producer and consumer API or interested in learning advanced Kafka Producer and Consumer operations, then this course is for you.
Approach
This course will walk you through the Kafka concepts step by step and all the Kafka concepts will be taught through hands-on practice.
Key Features
Learn to build Kafka Producers/Consumers using Java * Covers advanced Kafka Producer and Consumer concepts * Hands-on oriented course
Github Repo
https://github.com/PacktPublishing/Apache-Kafka-for-Beginners---Learn-Kafka-by-Hands-On
About the Author
Dilip Sundarraj
Dilip Sundarraj is a software engineer who has experience with building software since 2008. He is passionate about learning modern technologies and staying up to date with all the modern technologies, tools, frameworks, and more. He loves to share his knowledge with the world and that is one of the key reasons for him to be in the online teaching industry.
He loves interacting with other software developers and believes that this helps him share knowledge and learn from them. During his leisure time, he loves to play cricket, watch movies, and work out in the gym to have a balance between physical and mental strength.
Dilip has a YouTube channel named Code with Dilip, where he has been sharing a lot of technical content related to languages, frameworks, best practices, and more.
Course Outline
1. Getting Started with Course

2. Getting Started with Kafka

3. Download and Install Kafka

4. Understanding Kafka Components and its Internals - (Theory + Hands On)

5. Kafka Producer API - Hands On

6. Kafka Producer API - Guaranteed Message Delivery Configurations

7. Kafka Consumer API - Hands On

8. Consumer Groups & Consumer Rebalance - Hands On

9. Consumer Offsets - Default and Manual Offset Management - Hands On

10. Consumer Rebalance Listeners - Hands On

11. Kafka Consumer - seekToBeginning(), seekToEnd() & seek() - Hands On

12. Custom Serializer and Deserializers in Kafka - Hands On


Learn the fundamentals and advanced concepts of Apache Kafka in this course. This course will give you a good understanding of all the concepts through hands-on practice.

Apache Kafka for Beginners - Learn Kafka by Hands-On

Advance your data skills by mastering Spark programming in Python. This beginner's level course will help you understand the core concepts related to Apache Spark 3 and provide you with knowledge of applying those concepts to build data engineering solutions.
If you are looking to expand your knowledge in data engineering or want to level up your portfolio by adding Spark programming to your skillset, then you are in the right place. This course will help you understand Spark programming and apply that knowledge to build data engineering solutions. This course is example-driven and follows a working session-like approach. We will be taking a live coding approach and explaining all the concepts needed along the way. In this course, we will start with a quick introduction to Apache Spark, then set up our environment by installing and using Apache Spark. Next, we will learn about Spark execution model and architecture, and about Spark programming model and developer experience. Next, we will cover Spark structured API foundation and then move towards Spark data sources and sinks. Then we will cover Spark Dataframe and dataset transformations. We will also cover aggregations in Apache Spark and finally, we will cover Spark Dataframe joins. By the end of this course, you will be able to build data engineering solutions using Spark structured API in Python. All the resources for the course are available at https://github.com/PacktPublishing/Spark-Programming-in-Python-for-Beginners-with-Apache-Spark-3
What You Will Learn
Learn Apache Spark Foundation and Spark architecture
Learn data engineering and data processing in Spark
Work with data sources and sinks
Work with data frames and Spark SQL
Use PyCharm IDE for Spark development and debugging
Learn unit testing, managing application logs, and cluster deployment
Audience
This course is designed for software engineers willing to develop a data engineering pipeline and application using Apache Spark; for data architects and data engineers who are responsible for designing and building the organization's data-centric infrastructure, for managers and architects who do not directly work with Spark implementation but work with the people who implement Apache Spark at the ground level.

This course does not require any prior knowledge of Apache Spark or Hadoop; only programming knowledge using Python programming language is required.
Approach
This course is example-driven and follows a working session-like approach. The course delivers live coding sessions and explains the concepts along the way.
Key Features
Build your own data engineering solutions using Spark structured API in Python * Gain an in-depth understanding of the Apache Hadoop architecture, ecosystem, and practices * Learn to apply Spark programming basics
Github Repo
https://github.com/PacktPublishing/Spark-Programming-in-Python-for-Beginners-with-Apache-Spark-3
About the Author
ScholarNest 
ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects. It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.
Course Outline
1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">


In this section, we will cover an introduction to Apache Spark. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover big data history and primer. 
2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will understand the data lake landscape. 
3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will understand what Apache Spark is and get a quick introduction and overview of Apache Spark. 
2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">


In this section, you will learn how to install and use Apache Spark. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Spark development environments. 
2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Mac users to set-up Apache Spark in Local Mode Command Line REPL. 
3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Windows users to set-up Apache Spark in local mode command line REPL. 
4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Mac users to set-up Apache Spark in the PyCharm IDE. 
5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Windows users to set-up Apache Spark in the PyCharm IDE. 
6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Apache Spark in cloud and Databricks community and notebooks. 
7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will explore Apache Spark in Anaconda and Jupyter notebook. 
3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">


In this section, you will learn about Spark execution model and architecture. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover execution methods and learn how to run Spark programs. 
2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will understand Spark distributed processing model and how your program runs? 
3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Spark execution modes and cluster managers. 
4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we willsummarize Spark execution models and understand when to use what? 
5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this demo video, you will learn how to work with PySpark shell. 
6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this demo video, you will learn how to install multi-node Spark cluster. 
7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this demo video, you will learn how to work with notebooks in the cluster. 
8. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this demo video, you will learn how to work with Spark submit. 
9. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will summarize our learning from this section. 
4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">


In this section, you will learn about Spark programming model and developer experience. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, youwill learn how tocreate Spark project build configuration. 
2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, youwill learn how toconfigure Spark project application logs. 
3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, youwill learn how tocreate Spark session. 
4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, youwill learn how toconfigure Spark session. 
5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover anintroductionto data frame. 
6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover data frame partitions and executors. 
7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Spark transformations and actions. 
8. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Spark jobs stages and task. 
9. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, youwill learn how tounderstand your execution plan. 
10. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover unit testing Spark application. 
11. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will summarize our learning from this section. 
5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">


In this section, you will learn about Spark structured API foundation. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover an introduction to Spark APIs. 
2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover an introduction to Spark RDD API. 
3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this demo video, you will learn how to work with Spark SQL. 
4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Spark SQL engine and catalyst optimizer. 
5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will summarize our learning from this section. 
6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">


In this section, you will learn about Spark data sources and sinks. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Spark data sources and sinks. 
2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Spark DataFrameReader API. 
3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, youwill learn how toread CSV, JSON and Parquet files. 
4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, youwill learn how tocreate Spark DataFrame schema. 
5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Spark DataFrameWriter API. 
6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, youwill learn how towrite your data and manage layout. 
7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Spark databases and tables. 
8. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this demo video, you will learn how to work with Spark SQL tables. 
7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">


In this section, you will learn about Spark Dataframe and Dataset transformations. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover an introduction to data transformation. 
2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this demo video, you will learn how to work with Dataframe rows. 
3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover DataFrame rows and unit testing. 
4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Dataframe rows and unstructured data. 
5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this demo video, you will learn how to work with Dataframe columns. 
6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, youwill learn how tocreate and use UDF. 
7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Misc transformations. 
8. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">


In this section, you will learn about aggregations in Apache Spark. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Aggregating Dataframes. 
2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Grouping aggregations. 
3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Windowing aggregations. 
9. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">


In this section, you will learn about Spark Dataframe joins. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover Dataframe joins and column name ambiguity. 
2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover outer joins in Dataframe. 
3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, we will cover the internals of Spark join and shuffle. 
4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, you will learn how to optimize your joins. 
5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



In this video, you will learn how to implement Bucket joins. 
10. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">


Welcome to the concluding section of the course. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">



Thank you for joining. Happy learning! 

Advance your data skills by mastering Spark programming in Python. This beginner's level course will help you understand the core concepts related to Apache Spark 3 and provide you with knowledge of applying those concepts to build data engineering solutions.

Spark Programming in Python for Beginners with Apache Spark 3

This is a comprehensive and practical Apache Spark course. In this course, you will learn and master the art of framing data analysis problems as Spark problems through 20+ hands-on examples, and then scale them up to run on cloud computing services. Explore Spark 3, IntelliJ, Structured Streaming, and a stronger focus on the DataSet API.
'Big data' analysis is a hot and highly valuable skill-and this course will teach you the hottest technology in big data: Apache Spark. Employers including Amazon, eBay, NASA JPL, and Yahoo all use Spark to quickly extract meaning from massive datasets across a fault-tolerant Hadoop cluster. You will learn those same techniques using your own Windows system right at home. It is easier than you think, and you will learn from an ex-engineer and senior manager from Amazon and IMDb. In this course, you will learn the concepts of Spark's Resilient Distributed Datasets, DataFrames, and datasets. We will also cover a crash course in the Scala programming language that will help you with the course. You will learn how to develop and run Spark jobs quickly using Scala, IntelliJ, and SBT. You will learn how to translate complex analysis problems into iterative or multi-stage Spark scripts. You will learn how to scale up to larger datasets using Amazon's Elastic MapReduce service and understand how Hadoop YARN distributes Spark across computing clusters. We will also be practicing using other Spark technologies, such as Spark SQL, DataFrames, DataSets, Spark Streaming, Machine Learning, and GraphX. By the end of this course, you will be running code that analyzes gigabytes worth of information-in the cloud-in a matter of minutes. All the codes and supporting files for this course are available at https://github.com/PacktPublishing/Apache-Spark-with-Scala---Hands-On-with-Big-Data-
What You Will Learn
Learn the concepts of Spark's RDD, DataFrames, and Datasets
Get a crash course in the Scala programming language
Develop and run Spark jobs quickly using Scala, IntelliJ, and SBT
Translate complex analysis problems into iterative or multi-stage Spark scripts
Scale up to larger datasets using Amazon's Elastic MapReduce service
Understand how Hadoop YARN distributes Spark across computing clusters
Audience
This course is designed for software engineers who want to expand their skills into the world of big data processing on a cluster. It is necessary to have some prior programming or scripting knowledge.
Approach
This course is very hands-on; you will spend most of your time following along with the instructor as we write, analyze, and run real code together-both on your own system and in the cloud using Amazon's Elastic MapReduce service. Over eight hours of video content is included, with over 20 real examples of increasing complexity that you can build, run, and study yourself. Move through them at your own pace, on your own schedule.
Key Features
Understand the fundamentals of Scala and the Apache Spark ecosystem * Develop distributed code using the Scala programming language * Include practical examples to help you develop real-world Big Data applications with Spark with Scala
Github Repo
https://github.com/PacktPublishing/Apache-Spark-with-Scala---Hands-On-with-Big-Data-
About the Author
Frank Kane
Frank Kane has spent nine years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers all the time. He holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology and teaches others about big data analysis.
Course Outline
1. Getting Started
This section introduces the course, and you will get the development environment for Spark and Scala and create a histogram of real movie ratings with Spark.
1. Introduction and Installing the Course Materials, IntelliJ, and Scala
A brief introduction to the course, and then we will get your development environment for Spark and Scala all set up on your desktop, using IntelliJ and SBT. A quick test application will confirm that Spark is working on your system!
2. Introduction to Apache Spark
This video is a brief introduction to Apache Spark.
2. Scala Crash Course (Optional)
This section is about Scala. You will learn the basics, flow control, functions, and data structures in Scala.
1. (Activity) Scala Basics
We will go over the basic syntax and structure of Scala code with lots of examples. It's backward from most other languages, but you quickly get used to it.
2. (Exercise) Flow Control in Scala
We will go over the basic syntax and structure of Scala code with lots of examples. It's backward from most other languages, but you quickly get used to it. In this video, you will get some hands-on practice at the end.
3. (Exercise) Functions in Scala
Scala is a functional programming language and so, functions are central to the language. We will go over the many ways functions can be declared and used in Scala, and practice what you have learned.
4. (Exercise) Data Structures in Scala
We will cover the common data structures in Scala such as Map and List and put them into practice.
3. Using Resilient Distributed Datasets (RDDs)
In this section, you will learn how to use Spark RDD.
1. The Resilient Distributed Dataset
The core object of Spark programming is the Resilient Distributed Dataset, or RDD. Once you know how to use RDDs, you know how to use Spark. We will go over what they are and what you can do with them.
2. Ratings Histogram Example
Now that we understand Scala and have the theory of Spark behind us, let's start with a simple example of using RDDs to count up how many of each rating exists in the MovieLens dataset.
3. Spark Internals
How does Spark convert your script into a Directed Acyclic Graph and figure out how to distribute it on a cluster? Understanding how this process works under the hood can be important in writing optimal Spark driver scripts.
4. Key / Value RDDs, and the Average Friends by Age Example
RDDs that contain a tuple of two values are key/value RDDs, and you can use them much like you might use a NoSQL data store. We will use key/value RDDs to figure out the average number of friends by age in some fake social network data.
5. (Activity) Running the Average Friends by Age Example
We will run the average friends by age example on your desktop and give you some ideas for further extending this script on your own.
6. Filtering RDDs, and the Minimum Temperature by Location Example
We will cover how to filter data out of an RDD efficiently and illustrate this with a new example that finds the minimum temperature by location using real weather data.
7. (Activity) Running the Minimum Temperature Example, and Modifying It for Maximum
We will run our minimum temperature by location example and modify it to find maximum temperatures as well. Plus, some ideas for extending this script on your own.
8. (Activity) Counting Word Occurrences Using Flatmap()
flatmap() on an RDD can return variable amounts of new entries into the resulting RDD. We will use this as part of a hands-on example that finds how often each word is used inside a real book's text.
9. (Activity) Improving the Word Count Script with Regular Expressions
We extend the previous lecture's example using regular expressions to better extract words from our book.
10. (Activity) Sorting the Word Count Results
Finally, we sort the final results to see what the most common words in this book really are! And some ideas to extend this script on your own.
11. (Exercise) Find the Total Amount Spent by Customer
Your assignment is to write a script that finds the total amount spent per customer using some fabricated ecommerce data, using what you have learned so far.
12. (Exercise) Check Your Results and Sort Them by Total Amount Spent
We will review my solution to the previous lecture's assignment and challenge you further to sort your results to find the biggest spenders.
13. Check Your Results and Implementation Against Mine
Check your results for finding the biggest spenders in our ecommerce data against my own solution.
4. SparkSQL, DataFrames, and DataSets
This section is about SparkSQL, DataFrames, and DataSets.
1. Introduction to SparkSQL
Understand SparkSQL and the DataFrame and DataSet APIs used for querying structured data in an efficient, scalable manner.
2. (Activity) Using SparkSQL
We will revisit our fabricated social network data but load it into a DataFrame and analyze it with actual SQL queries!
3. (Activity) Using DataSets
We will analyze our social network data another way - this time using SQL-like functions on a DataSet, instead of actual SQL query strings.
4. (Exercise) Implement the Friends by Age Example Using DataSets
Earlier, we broke down the average number of friends by age using RDDs; see if you can do it using DataSets instead!
5. Exercise Solution: Friends by Age, with DataSets
Let's check the solution for the Friends by Age exercise using DataSets.
6. (Activity) Word Count Example Using DataSets
You will learn how to implement the word count activity with DataSets using SQL functions.
7. (Activity) Revisiting the Minimum Temperature Example, with DataSets
You will learn how to implement the weather data activity with DataSets using withColumn() functions.
8. (Exercise) Implement the Total Spent by Customer Problem with DataSets
We will have a look at the total spent by customer problem activity with DataSets.
9. Exercise Solution: Total Spent by Customer with DataSets
We will have a look at the possible solution for the total spent by customer problem activity with DataSets.
5. Advanced Examples of Spark Programs
In this section, we will be working on some advanced examples of Spark programming.
1. (Activity) Find the Most Popular Movie
We will revisit our movie ratings dataset and start off with a simple example to find the most-rated movie.
2. (Activity) Use Broadcast Variables to Display Movie Names
Broadcast variables can be used to share small amounts of data to all of the machines on your cluster. We will use them to share a lookup table of movie IDs to movie names and use that to get movie names in our final results.
3. (Activity) Find the Most Popular Superhero in a Social Graph
We introduce the Marvel superhero social network dataset and write a script to find the most-connected superhero in it. It's not who you might think!
4. (Exercise) Find the Most Obscure Superheroes
This exercise is all about finding the most obscure superheroes from the MostPopularSuperheroDataset.
5. Exercise Solution: Find the Most Obscure Superheroes
We will work on the possible solution to find the most obscure superheroes.
6. Superhero Degrees of Separation: Introducing Breadth-First Search
As a more complex example, we will apply a breadth-first-search (BFS) algorithm to the Marvel dataset to compute the degrees of separation between any two superheroes. In this lecture, we go over how BFS works.
7. Superhero Degrees of Separation: Accumulators, and Implementing BFS in Spark
We will go over our strategy for implementing BFS within a Spark script that can be distributed and introduce the use of Accumulators to maintain running totals that are synced across a cluster.
8. (Activity) Superhero Degrees of Separation: Review the Code and Run It!
Finally, we will review the code for finding the degrees of separation using breadth-first search, run it, and see the results!
9. Item-Based Collaborative Filtering in Spark, cache(), and persist()
Back to our movie ratings data-we will discover movies that are similar to each other just based on user ratings. We will cover the algorithm and how to implement it as a Spark script.
10. (Activity) Running the Similar Movies Script Using Spark's Cluster Manager
We will run our movie similarities script and see the results. In doing so, we will introduce the process of exporting your Spark script as a JAR file that can be run from the command line using the spark-submit script (instead of running from within the Scala IDE).
11. (Exercise) Improve the Quality of Similar Movies
Your challenge is to make the movie similarity results even better! Here are some ideas for you to try out.
6. Running Spark on a Cluster
This section is about running Spark on a cluster.
1. (Activity) Using spark-submit to Run Spark Driver Scripts
In a production environment, you will use spark-submit to start your driver scripts from a command line, cron job, or the like. We will cover the details on what you need to do differently in this case.
2. (Activity) Packaging Driver Scripts with SBT
Spark / Scala scripts that have external dependencies can be bundled up into self-contained packages using the SBT tool. We will use SBT to package up our movie similarities script as an exercise.
3. (Exercise) Package a Script with SBT and Run It Locally with spark-submit
We will work on an exercise where we will package a script with SBT and run it locally with spark-submit.
4. Exercise Solution: Using SBT and spark-submit
We will have a look at the possible solution for this exercise.
5. Introducing Amazon Elastic MapReduce
Amazon Web Services (AWS) offers the Elastic MapReduce service (EMR), which gives us a way to rent time on a Hadoop cluster of our choosing-with Spark pre-installed on it. We will use EMR to illustrate running a Spark script on a real cluster, so let's go over what EMR is and how it works first.
6. Creating Similar Movies from One Million Ratings on EMR
Let's compute movie similarities on a real cluster in the cloud, using one million user ratings!
7. Partitioning
Explicitly partitioning your datasets and RDDs can be an important optimization; we will go over when and how to do this.
8. Best Practices for Running on a Cluster
Other tips and tricks for taking your script to a real cluster and getting it to run as you expect.
9. Troubleshooting and Managing Dependencies
This video is on how to troubleshoot Spark jobs on a cluster using the Spark UI and logs, and more on managing dependencies of your script and data.
7. Machine Learning with Spark ML
This section is about machine learning with MLLib.
1. Introducing MLLib
MLLib offers several distributed machine learning algorithms that you can run on a Spark cluster. We will cover what MLLib can do and how it fits in.
2. (Activity) Using MLLib to Produce Movie Recommendations
We will use MLLib's Alternating Least Squares recommender algorithm to produce movie recommendations using our MovieLens ratings data. The results areunexpected!
3. Linear Regression with MLLib
A brief overview of what linear regression is and how it works, followed by a hands-on example of finding a regression and applying it to fabricated page speed versus revenue data.
4. (Activity) Running a Linear Regression with Spark
We will run our Spark ML example of linear regression using DataFrames.
5. (Exercise) Predict Real Estate Values with Decision Trees in Spark
This exercise is all about predicting real estate values with decision trees in Spark.
6. Exercise Solution: Predicting Real Estate with Decision Trees in Spark
We will have a look at the possible solution for this exercise.
8. Introduction to Spark Streaming
This section is about Spark Streaming.
1. The DStream API for Spark Streaming
Spark Streaming allows you to create Spark driver scripts that run indefinitely, continually processing data as it streams in! We will cover how it works and what it can do, using the original DStream micro-batch API.
2. (Activity) Real-Time Monitoring of the Most Popular Hashtags on Twitter
You will learn how to do real-time monitoring of the most popular hashtags on Twitter.
3. Structured Streaming
Structured Streaming is a newer DataFrame-based API in Spark for writing continuous applications.
4. (Activity) Using Structured Streaming for Real-Time Log Analysis
You will learn how to use Structured Streaming for real-time log analysis.
5. (Exercise) Windowed Operations with Structured Streaming
We will work on an exercise to search the top URLs that are streamed in the past 30 seconds.
6. Exercise Solution: Top URLs in a 30-Second Window
We will look at the possible solution for this exercise.
9. Introduction to GraphX
This section is about GraphX.
1. GraphX, Pregel, and Breadth-First Search with Pregel
We will cover Spark's GraphX library and how it works.
2. Using the Pregel API with Spark GraphX
We will revisit our "superhero degrees of separation" example and see how its breadth-first search algorithm could be implemented using Pregel and GraphX.
3. (Activity) Superhero Degrees of Separation Using GraphX
We will use GraphX and Pregel to recreate our earlier results analyzing the superhero social network data-but with a lot less code!
10. You Made It! Where to Go from Here
This section is about learning more and career tips.
1. Learning More, and Career Tips
You made it to the end! Here are some book recommendations if you want to learn more, as well as some career advice on landing a job in "big data".

This is a comprehensive and practical Apache Spark course. In this course, you will learn and master the art of framing data analysis problems as Spark problems through 20+ hands-on examples, and then scale them up to run on cloud computing services. Explore Spark 3, IntelliJ, Structured Streaming, and a stronger focus on the DataSet API.

Apache Spark with Scala - Hands-On with Big Data!

<strong>In this course, we will process massive streams of real-time data using Spark Streaming and create Spark applications using the Scala programming language (v2.12). We will also get our hands-on with some real live Twitter data, simulated streams of Apache access logs, and even data used to train machine learning models.</strong>
        <p>In this course, you will learn the basics of the Scala programming language; learn how Apache Spark operates on a cluster; set up discretized streams with Spark Streaming and transform them as data is received; analyze streaming data over sliding windows of time; maintain stateful information across streams of data; connect Spark Streaming with highly scalable sources of data, including Kafka, Flume, and Kinesis; dump streams of data in real-time to NoSQL databases such as Cassandra; run SQL queries on streamed data in real-time; train machine learning models in real-time with streaming data, and use them to make predictions that keep getting better over time; and also, package, deploy, and run self-contained Spark Streaming code to a real Hadoop cluster using Amazon Elastic MapReduce.

This course is very hands-on, filled with achievable activities and exercises to reinforce your learning. By the end of this course, you will be confidently creating Spark Streaming scripts in Scala and be prepared to tackle massive streams of data in a whole new way. You will be surprised at how easy Spark Streaming makes it!

All the codes and supporting files for this course are available at https://github.com/packtpublishing/streaming-big-data-with-spark-streaming-scala-and-spark-3-</p>
        <h2>What You Will Learn</h2><p>Process large amounts of real-time data using the Spark Streaming module<br>Create efficient Spark applications using the Scala programming language<br>Integrate Spark Streaming with various data sources<br>Integrate Spark Streaming with Spark SQL to query your data in real time<br>Train machine learning models with streaming data, and use for real-time predictions<br>Maintain stateful data across a continuous stream of input data</p>
        <h2>Audience</h2><p>If you are a student who wants to learn how to use Apache Spark or a big data professional who wants to process large amounts of data on a real-time basis, this course is for you. Some basic programming and scripting experience is required to get the most out of the course.</p>
        <h2>Approach</h2><p>This course is very hands-on, filled with achievable activities and exercises to reinforce your learning. In this course, we will cover some real live Twitter data, simulated streams of Apache access logs, and even data used to train machine learning models! You will write and run real Spark Streaming jobs right at home on your own PC.</p>
        <h2>Key Features</h2><p>Process streams of real-time data from various sources with Spark Streaming * Query your streaming data in real-time using Spark SQL * A comprehensive tutorial with practical examples to help you develop real-time Spark applications</p>
        <h2>Github Repo</h2><p>https://github.com/packtpublishing/streaming-big-data-with-spark-streaming-scala-and-spark-3-</p>
        <h2>About the Author</h2><h3>Frank Kane</h3><p>Frank Kane has spent nine years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers all the time. He holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology and teaches others about big data analysis.</p>
        <h2>Course Outline</h2><h4>1. Getting Started</h4><p>In this first section, we will introduce the course and set up the development environment for Spark and Scala. We will then create our first Spark Streaming application to display live Twitter feeds.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Introduction, and Getting Set Up</p><p style='margin: 0'>A brief introduction to the course, and then we will get your development environment for Spark and Scala all set up on your desktop. A quick test application will confirm Spark is working on your system!</p></td></tr><tr><td style='padding: 15px;'><p>2. [Activity] Stream Live Tweets with Spark Streaming!</p><p style='margin: 0'>Get set up with a Twitter developer account and run your first Spark Streaming application to listen to and print out live Tweets as they happen!</p></td></tr></tbody></table><br><h4>2. A Crash Course in Scala</h4><p>In this section, we will take a deep dive into Scala. Even if you've never used Scala before, we will cover all of the syntax and structure that you will need to know in order to begin mining your data. We will run through flow control, functions, and data structures with numerous examples throughout so that you can see everything in action and make sure you're understanding it.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. [Activity] Scala Basics</p><p style='margin: 0'>We start our crash course in the Scala programming language by covering some basics of the language: types and variables, printing, and Boolean comparisons.</p></td></tr><tr><td style='padding: 15px;'><p>2. [Exercise] Flow Control in Scala</p><p style='margin: 0'>Our Scala crash course continues, illustrating various means of flow control in Scala. For loops, do/while loops, while loops and so on.</p></td></tr><tr><td style='padding: 15px;'><p>3. [Exercise] Functions in Scala</p><p style='margin: 0'>Scala is a functional programming language, and so understanding how functions work and are treated in Scala is hugely important! This lecture covers the fundamentals, and lets you put it into practice.</p></td></tr><tr><td style='padding: 15px;'><p>4. [Exercise] Data Structures in Scala</p><p style='margin: 0'>We wrap up our Scala crash course with commonly used data structures using in Spark with Scala. Tuples, lists, and maps.</p></td></tr></tbody></table><br><h4>3. Spark Streaming Concepts</h4><p>In this section, we will introduce the core concepts of Spark and Spark Streaming. We will define and work with RDDs (Resilient Distributed Datasets) and show you examples of what they can do. RDDs are the key to understanding Spark, and therefore also Spark Streaming. This is also where we will dive deeper into Spark Streaming itself, understanding its inner workings and how the concept of windowing will enable you to aggregate and analyze data over time. Finally, we will take a look at reliability and fault tolerance with regards to Spark Streaming and a few ways in which you can maximize reliability.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Introduction to Spark</p><p style='margin: 0'>Before you can learn about Spark Streaming, you need to understand how Spark itself works at a high level! This video covers the why and how of Apache Spark, of which Spark Streaming is a component.</p></td></tr><tr><td style='padding: 15px;'><p>2. The Resilient Distributed Dataset (RDD)</p><p style='margin: 0'>The fundamental object of Spark programming is the Resilient Distributed Dataset (RDD), and this is used not just in Spark but also within Spark Streaming scripts. This lecture explains what they are and what you can do with them.</p></td></tr><tr><td style='padding: 15px;'><p>3. [Activity] RDD's in Action: Simple Word Count Application</p><p style='margin: 0'>Let's walk through and actually run a simple Spark script that counts the number of occurrences of each word in a book.</p></td></tr><tr><td style='padding: 15px;'><p>4. Introduction to Spark Streaming</p><p style='margin: 0'>We finally have all the pr-requisite knowledge to start talking about Spark Streaming itself in more detail! We will cover how it works, what it's for, and its architecture.</p></td></tr><tr><td style='padding: 15px;'><p>5. [Activity] Revisiting the PrintTweets application</p><p style='margin: 0'>Now that we know more, let's go revisit that first Spark Streaming application we ran in lecture two, and dive into how it really works.</p></td></tr><tr><td style='padding: 15px;'><p>6. Windowing: Aggregating data over longer time spans</p><p style='margin: 0'>Windowing allows you to analyze streaming data over a sliding window of time, which lets you do much more than just transform streaming data and store it someplace else. We will cover the concepts of the batch, window, and slide intervals, and how they work together to let you aggregate streaming data over some period of time.</p></td></tr><tr><td style='padding: 15px;'><p>7. Fault Tolerance in Spark Streaming</p><p style='margin: 0'>How can Spark Streaming do so much work continuously in a reliable manner? We will uncover some of its tricks for reliability, as well as tips for configuring Spark Streaming to be as reliable as possible.</p></td></tr></tbody></table><br><h4>4. Spark Streaming Examples with Twitter</h4><p>Now that we have seen the theory behind Spark Streaming, this section is dedicated to putting all of that into action. We will look at three practical examples of taking Twitter data and applying our new learning to it with file output, windowing, and live tracking, and analysis.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. [Exercise] Saving Tweets to Disk</p><p style='margin: 0'>We will build on our "print tweets" example to actually store the incoming tweets to disk and illustrate how Spark Streaming can handle file output.</p></td></tr><tr><td style='padding: 15px;'><p>2. [Exercise] Tracking the Average Tweet Length</p><p style='margin: 0'>Compute the average length of a tweet, using windowing in Spark Streaming.</p></td></tr><tr><td style='padding: 15px;'><p>3. [Exercise] Tracking the Most Popular Hashtags</p><p style='margin: 0'>This is a fun one! We will track the most popular hashtags in Twitter over time and watch how they change in real time!</p></td></tr></tbody></table><br><h4>5. Spark Streaming Examples with Clickstream / Apache Access Log Data</h4><p>This section will continue to look at some of the more advanced aspects of Spark Streaming, this time using Apache access logs to monitor website traffic in real time, showing the most accessed pages and flagging up and log errors as they occur. We will also use SparkSQL to query our data live. Finally, we will look at Spark 2's new structured streaming approach, and how we can use DataSets to streamline our analysis.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. [Exercise] Tracking the Top URL's Requested</p><p style='margin: 0'>We will simulate an incoming stream of Apache access logs and use Spark Streaming to keep track of the most-requested web pages in real time!</p></td></tr><tr><td style='padding: 15px;'><p>2. [Exercise] Alarming on Log Errors</p><p style='margin: 0'>This example will listen to an Apache access log stream and raise an alarm if too many errors are returned by the server in real time.</p></td></tr><tr><td style='padding: 15px;'><p>3. [Exercise] Integrating Spark Streaming with Spark SQL</p><p style='margin: 0'>We will integrate Spark Streaming with Spark SQL, allowing us to run SQL queries on data as it is streamed in! Again, we will use Apache logs as an example.</p></td></tr><tr><td style='padding: 15px;'><p>4. Introduction to Structured Streaming</p><p style='margin: 0'>Spark 2.0 introduced experimental support for Structured Streaming, a new DataSet-based API for Spark Streaming that is bound to become increasingly important. Learn how it works.</p></td></tr><tr><td style='padding: 15px;'><p>5. [Activity] Analyzing Apache Log files with Structured Streaming</p><p style='margin: 0'>As an example, we will stream Apache access logs in from a directory and use Structured Streaming to count up status codes over a one-hour moving window.</p></td></tr></tbody></table><br><h4>6. Integrating with Other Systems</h4><p>In this section, we will examine how Spark Streaming can be integrated with other popular data science systems. We will see how it can work with Apache Kafka to listen to and process messages in real time; how Apache Flume can work with Spark Streaming to process huge amounts of log data; using Amazon Kinesis with Spark Streaming to process live data on an AWS/EC2 cluster; and using Cassandra to store massive datasets and analyze them at speed. We will also cover how to write your own Data Receiver class in order to connect Spark Streaming to other proprietary systems.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Integrating with Apache Kafka</p><p style='margin: 0'>Apache Kafka is a popular and robust technology for publishing messages across a cluster on a large scale. We will show how to get Spark Streaming to listen to Kafka topics and process them in real time.</p></td></tr><tr><td style='padding: 15px;'><p>2. Integrating with Apache Flume</p><p style='margin: 0'>Flume is a popular technology for publishing log information at large scale, especially on a Hadoop cluster. We will illustrate how to set up both push-based and pull-based Flume configurations with Spark Streaming and discuss the tradeoffs of each.</p></td></tr><tr><td style='padding: 15px;'><p>3. Integrating with Amazon Kinesis</p><p style='margin: 0'>Amazon's Kinesis Streaming service is basically Kafka on AWS. If you're working with an AWS/EC2 cluster, you will want to know how to integrate Spark Streaming with Kinesis - and that's what this lecture covers.</p></td></tr><tr><td style='padding: 15px;'><p>4. [Activity] Writing Custom Data Receivers</p><p style='margin: 0'>What if you need to integrate Spark Streaming with some proprietary system that does not have an existing connection library? Well, you can always write your own Receiver class. This example shows you how and actually lets you build and run one.</p></td></tr><tr><td style='padding: 15px;'><p>5. Integrating with Cassandra</p><p style='margin: 0'>Cassandra is a popular "NoSQL" database that can be used to provide fast access to massive data sets to real-time applications. Dumping data transformed by Spark Streaming into a Cassandra database can expose that data to your larger, real-time services. We will show you how, and actually run a simple example.</p></td></tr></tbody></table><br><h4>7. Advanced Spark Streaming Examples</h4><p>In this section, we will run through some more advanced real-world examples of Spark Streaming. We will examine tracking stateful information, machine learning with K-Means clustering, and feeding data into linear regression models.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. [Exercise] Stateful Information in Spark Streams</p><p style='margin: 0'>Spark has the ability to track arbitrary state across streams of data as they come in, such as web sessions, running totals and so on. This example shows you how it all works, and challenges you to track your own state using our example as a baseline.</p></td></tr><tr><td style='padding: 15px;'><p>2. [Activity] Streaming K-Means Clustering</p><p style='margin: 0'>Spark Streaming integrates with some of Spark's MLLib (Machine Learning Library) capabilities. This example creates a real-time K-Means clustering example; it does unsupervised machine learning that continually gets better as more training data feeds into it.</p></td></tr><tr><td style='padding: 15px;'><p>3. [Activity] Streaming Linear Regression</p><p style='margin: 0'>Spark Streaming can also feed data in real-time to linear regression models, that get better over time as more data is fed into them. This example shows linear regression in action with Spark Streaming.</p></td></tr></tbody></table><br><h4>8. Spark Streaming in Production</h4><p>This section covers how to move your applications into a production environment, using the spark-submit command to process scripts from the command line. We will also look at packaging scripts and dependencies with SBT and running code on a real cluster with Amazon's Elastic MapReduce (EMR). Finally, we will troubleshoot some code. We will see what can go wrong, and how you can identify and debug issues and tune your Spark applications for peak performance.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. [Activity] Packaging and Running Spark Code in Production</p><p style='margin: 0'>Your production applications won't be run from within the Scala IDE; you will need to run them from a command line, and potentially on a cluster. The spark-submit command is used for this. We will show you how to package up your application and run it using spark-submit from a command prompt.</p></td></tr><tr><td style='padding: 15px;'><p>2. [Activity] Packaging Your Code with SBT</p><p style='margin: 0'>If your Spark Streaming application has external library dependencies that might not be already present on every machine in your cluster, the SBT tool can manage those dependencies for you, and package them into the JAR file you run with spark-submit. We will show you how it works with a real example.</p></td></tr><tr><td style='padding: 15px;'><p>3. Running on a Real Hadoop Cluster with EMR</p><p style='margin: 0'>We will run our simple word count example on a real cluster, using Amazon's Elastic MapReduce service! This just shows you what's involved in running a Spark Streaming job on a real cluster as opposed to your desktop; there are a few parameters to Spark-submit you need to worry about and getting your scripts and data in the right place is also something you need to deal with.</p></td></tr><tr><td style='padding: 15px;'><p>4. Troubleshooting and Tuning Spark Jobs</p><p style='margin: 0'>Spark jobs rarely run perfectly, if at all, on the first try - some tuning and debugging is usually required and arriving at the right scale of your cluster is also necessary. We will cover some performance tips, and how to troubleshoot what's going on with a Spark Streaming job running on a cluster.</p></td></tr></tbody></table><br><h4>9. You Made It!</h4><p>This final section runs through some recommendations on where to learn more about Spark Streaming.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Learning More</p><p style='margin: 0'>Want to learn more about Spark Streaming? Here are a few books and other resources I've found valuable.</p></td></tr></tbody></table>

In this course, we will process massive streams of real-time data using Spark Streaming and create Spark applications using the Scala programming language (v2.12). We will also get our hands-on with some real live Twitter data, simulated streams of Apache access logs, and even data used to train machine learning models.
In this course, you will learn the basics of the Scala programming language; learn how Apache Spark operates on a cluster; set up discretized streams with Spark Streaming and transform them as data is received; analyze streaming data over sliding windows of time; maintain stateful information across streams of data; connect Spark Streaming with highly scalable sources of data, including Kafka, Flume, and Kinesis; dump streams of data in real-time to NoSQL databases such as Cassandra; run SQL queries on streamed data in real-time; train machine learning models in real-time with streaming data, and use them to make predictions that keep getting better over time; and also, package, deploy, and run self-contained Spark Streaming code to a real Hadoop cluster using Amazon Elastic MapReduce. This course is very hands-on, filled with achievable activities and exercises to reinforce your learning. By the end of this course, you will be confidently creating Spark Streaming scripts in Scala and be prepared to tackle massive streams of data in a whole new way. You will be surprised at how easy Spark Streaming makes it! All the codes and supporting files for this course are available at https://github.com/packtpublishing/streaming-big-data-with-spark-streaming-scala-and-spark-3-
What You Will Learn
Process large amounts of real-time data using the Spark Streaming module
Create efficient Spark applications using the Scala programming language
Integrate Spark Streaming with various data sources
Integrate Spark Streaming with Spark SQL to query your data in real time
Train machine learning models with streaming data, and use for real-time predictions
Maintain stateful data across a continuous stream of input data
Audience
If you are a student who wants to learn how to use Apache Spark or a big data professional who wants to process large amounts of data on a real-time basis, this course is for you. Some basic programming and scripting experience is required to get the most out of the course.
Approach
This course is very hands-on, filled with achievable activities and exercises to reinforce your learning. In this course, we will cover some real live Twitter data, simulated streams of Apache access logs, and even data used to train machine learning models! You will write and run real Spark Streaming jobs right at home on your own PC.
Key Features
Process streams of real-time data from various sources with Spark Streaming * Query your streaming data in real-time using Spark SQL * A comprehensive tutorial with practical examples to help you develop real-time Spark applications
Github Repo
https://github.com/packtpublishing/streaming-big-data-with-spark-streaming-scala-and-spark-3-
About the Author
Frank Kane
Frank Kane has spent nine years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers all the time. He holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology and teaches others about big data analysis.
Course Outline
1. Getting Started
In this first section, we will introduce the course and set up the development environment for Spark and Scala. We will then create our first Spark Streaming application to display live Twitter feeds.
1. Introduction, and Getting Set Up
A brief introduction to the course, and then we will get your development environment for Spark and Scala all set up on your desktop. A quick test application will confirm Spark is working on your system!
2. [Activity] Stream Live Tweets with Spark Streaming!
Get set up with a Twitter developer account and run your first Spark Streaming application to listen to and print out live Tweets as they happen!
2. A Crash Course in Scala
In this section, we will take a deep dive into Scala. Even if you've never used Scala before, we will cover all of the syntax and structure that you will need to know in order to begin mining your data. We will run through flow control, functions, and data structures with numerous examples throughout so that you can see everything in action and make sure you're understanding it.
1. [Activity] Scala Basics
We start our crash course in the Scala programming language by covering some basics of the language: types and variables, printing, and Boolean comparisons.
2. [Exercise] Flow Control in Scala
Our Scala crash course continues, illustrating various means of flow control in Scala. For loops, do/while loops, while loops and so on.
3. [Exercise] Functions in Scala
Scala is a functional programming language, and so understanding how functions work and are treated in Scala is hugely important! This lecture covers the fundamentals, and lets you put it into practice.
4. [Exercise] Data Structures in Scala
We wrap up our Scala crash course with commonly used data structures using in Spark with Scala. Tuples, lists, and maps.
3. Spark Streaming Concepts
In this section, we will introduce the core concepts of Spark and Spark Streaming. We will define and work with RDDs (Resilient Distributed Datasets) and show you examples of what they can do. RDDs are the key to understanding Spark, and therefore also Spark Streaming. This is also where we will dive deeper into Spark Streaming itself, understanding its inner workings and how the concept of windowing will enable you to aggregate and analyze data over time. Finally, we will take a look at reliability and fault tolerance with regards to Spark Streaming and a few ways in which you can maximize reliability.
1. Introduction to Spark
Before you can learn about Spark Streaming, you need to understand how Spark itself works at a high level! This video covers the why and how of Apache Spark, of which Spark Streaming is a component.
2. The Resilient Distributed Dataset (RDD)
The fundamental object of Spark programming is the Resilient Distributed Dataset (RDD), and this is used not just in Spark but also within Spark Streaming scripts. This lecture explains what they are and what you can do with them.
3. [Activity] RDD's in Action: Simple Word Count Application
Let's walk through and actually run a simple Spark script that counts the number of occurrences of each word in a book.
4. Introduction to Spark Streaming
We finally have all the pr-requisite knowledge to start talking about Spark Streaming itself in more detail! We will cover how it works, what it's for, and its architecture.
5. [Activity] Revisiting the PrintTweets application
Now that we know more, let's go revisit that first Spark Streaming application we ran in lecture two, and dive into how it really works.
6. Windowing: Aggregating data over longer time spans
Windowing allows you to analyze streaming data over a sliding window of time, which lets you do much more than just transform streaming data and store it someplace else. We will cover the concepts of the batch, window, and slide intervals, and how they work together to let you aggregate streaming data over some period of time.
7. Fault Tolerance in Spark Streaming
How can Spark Streaming do so much work continuously in a reliable manner? We will uncover some of its tricks for reliability, as well as tips for configuring Spark Streaming to be as reliable as possible.
4. Spark Streaming Examples with Twitter
Now that we have seen the theory behind Spark Streaming, this section is dedicated to putting all of that into action. We will look at three practical examples of taking Twitter data and applying our new learning to it with file output, windowing, and live tracking, and analysis.
1. [Exercise] Saving Tweets to Disk
We will build on our "print tweets" example to actually store the incoming tweets to disk and illustrate how Spark Streaming can handle file output.
2. [Exercise] Tracking the Average Tweet Length
Compute the average length of a tweet, using windowing in Spark Streaming.
3. [Exercise] Tracking the Most Popular Hashtags
This is a fun one! We will track the most popular hashtags in Twitter over time and watch how they change in real time!
5. Spark Streaming Examples with Clickstream / Apache Access Log Data
This section will continue to look at some of the more advanced aspects of Spark Streaming, this time using Apache access logs to monitor website traffic in real time, showing the most accessed pages and flagging up and log errors as they occur. We will also use SparkSQL to query our data live. Finally, we will look at Spark 2's new structured streaming approach, and how we can use DataSets to streamline our analysis.
1. [Exercise] Tracking the Top URL's Requested
We will simulate an incoming stream of Apache access logs and use Spark Streaming to keep track of the most-requested web pages in real time!
2. [Exercise] Alarming on Log Errors
This example will listen to an Apache access log stream and raise an alarm if too many errors are returned by the server in real time.
3. [Exercise] Integrating Spark Streaming with Spark SQL
We will integrate Spark Streaming with Spark SQL, allowing us to run SQL queries on data as it is streamed in! Again, we will use Apache logs as an example.
4. Introduction to Structured Streaming
Spark 2.0 introduced experimental support for Structured Streaming, a new DataSet-based API for Spark Streaming that is bound to become increasingly important. Learn how it works.
5. [Activity] Analyzing Apache Log files with Structured Streaming
As an example, we will stream Apache access logs in from a directory and use Structured Streaming to count up status codes over a one-hour moving window.
6. Integrating with Other Systems
In this section, we will examine how Spark Streaming can be integrated with other popular data science systems. We will see how it can work with Apache Kafka to listen to and process messages in real time; how Apache Flume can work with Spark Streaming to process huge amounts of log data; using Amazon Kinesis with Spark Streaming to process live data on an AWS/EC2 cluster; and using Cassandra to store massive datasets and analyze them at speed. We will also cover how to write your own Data Receiver class in order to connect Spark Streaming to other proprietary systems.
1. Integrating with Apache Kafka
Apache Kafka is a popular and robust technology for publishing messages across a cluster on a large scale. We will show how to get Spark Streaming to listen to Kafka topics and process them in real time.
2. Integrating with Apache Flume
Flume is a popular technology for publishing log information at large scale, especially on a Hadoop cluster. We will illustrate how to set up both push-based and pull-based Flume configurations with Spark Streaming and discuss the tradeoffs of each.
3. Integrating with Amazon Kinesis
Amazon's Kinesis Streaming service is basically Kafka on AWS. If you're working with an AWS/EC2 cluster, you will want to know how to integrate Spark Streaming with Kinesis - and that's what this lecture covers.
4. [Activity] Writing Custom Data Receivers
What if you need to integrate Spark Streaming with some proprietary system that does not have an existing connection library? Well, you can always write your own Receiver class. This example shows you how and actually lets you build and run one.
5. Integrating with Cassandra
Cassandra is a popular "NoSQL" database that can be used to provide fast access to massive data sets to real-time applications. Dumping data transformed by Spark Streaming into a Cassandra database can expose that data to your larger, real-time services. We will show you how, and actually run a simple example.
7. Advanced Spark Streaming Examples
In this section, we will run through some more advanced real-world examples of Spark Streaming. We will examine tracking stateful information, machine learning with K-Means clustering, and feeding data into linear regression models.
1. [Exercise] Stateful Information in Spark Streams
Spark has the ability to track arbitrary state across streams of data as they come in, such as web sessions, running totals and so on. This example shows you how it all works, and challenges you to track your own state using our example as a baseline.
2. [Activity] Streaming K-Means Clustering
Spark Streaming integrates with some of Spark's MLLib (Machine Learning Library) capabilities. This example creates a real-time K-Means clustering example; it does unsupervised machine learning that continually gets better as more training data feeds into it.
3. [Activity] Streaming Linear Regression
Spark Streaming can also feed data in real-time to linear regression models, that get better over time as more data is fed into them. This example shows linear regression in action with Spark Streaming.
8. Spark Streaming in Production
This section covers how to move your applications into a production environment, using the spark-submit command to process scripts from the command line. We will also look at packaging scripts and dependencies with SBT and running code on a real cluster with Amazon's Elastic MapReduce (EMR). Finally, we will troubleshoot some code. We will see what can go wrong, and how you can identify and debug issues and tune your Spark applications for peak performance.
1. [Activity] Packaging and Running Spark Code in Production
Your production applications won't be run from within the Scala IDE; you will need to run them from a command line, and potentially on a cluster. The spark-submit command is used for this. We will show you how to package up your application and run it using spark-submit from a command prompt.
2. [Activity] Packaging Your Code with SBT
If your Spark Streaming application has external library dependencies that might not be already present on every machine in your cluster, the SBT tool can manage those dependencies for you, and package them into the JAR file you run with spark-submit. We will show you how it works with a real example.
3. Running on a Real Hadoop Cluster with EMR
We will run our simple word count example on a real cluster, using Amazon's Elastic MapReduce service! This just shows you what's involved in running a Spark Streaming job on a real cluster as opposed to your desktop; there are a few parameters to Spark-submit you need to worry about and getting your scripts and data in the right place is also something you need to deal with.
4. Troubleshooting and Tuning Spark Jobs
Spark jobs rarely run perfectly, if at all, on the first try - some tuning and debugging is usually required and arriving at the right scale of your cluster is also necessary. We will cover some performance tips, and how to troubleshoot what's going on with a Spark Streaming job running on a cluster.
9. You Made It!
This final section runs through some recommendations on where to learn more about Spark Streaming.
1. Learning More
Want to learn more about Spark Streaming? Here are a few books and other resources I've found valuable.

In this course, we will process massive streams of real-time data using Spark Streaming and create Spark applications using the Scala programming language (v2.12). We will also get our hands-on with some real live Twitter data, simulated streams of Apache access logs, and even data used to train machine learning models.

Streaming Big Data with Spark Streaming, Scala, and Spark 3!

Learn how to program using Apache Groovy.

The Complete Apache Groovy Developer Course

<strong>A complete course on Sqoop, Flume, and Hive: Ideal for achieving CCA175 and Hortonworks Spark Certification</strong>
        <p>In this course, you will start by learning about the Hadoop Distributed File System (HDFS) and the most common Hadoop commands required to work with HDFS. Next, you'll be introduced to Sqoop Import, which will help you gain insights into the lifecycle of the Sqoop command and how to use the import command to migrate data from MySQL to HDFS, and from MySQL to Hive. 

In addition to this, you will get up to speed with Sqoop Export for migrating data effectively, along with using Apache Flume to ingest data. As you progress, you will delve into Apache Hive, external and managed tables, working with different files, and Parquet and Avro. Toward the concluding section, you will focus on Spark DataFrames and Spark SQL. 

By the end of this course, you will have gained comprehensive insights into big data ingestion and analytics with Flume, Sqoop, Hive, and Spark. 

All code and supporting files are available at - https://github.com/PacktPublishing/Master-Big-Data-Ingestion-and-Analytics-with-Flume-Sqoop-Hive-and-Spark</p>
        <h2>What You Will Learn</h2><p>Explore the Hadoop Distributed File System (HDFS) and commands <br>Get to grips with the lifecycle of the Sqoop command <br>Use the Sqoop Import command to migrate data from MySQL to HDFS and Hive <br>Understand split-by and boundary queries <br>Use the incremental mode to migrate data from MySQL to HDFS <br>Employ Sqoop Export to migrate data from HDFS to MySQL <br>Discover Spark DataFrames  and gain insights into  working with different file formats and compression</p>
        <h2>Audience</h2><p>This course is for anyone who wants to learn Sqoop and Flume or those looking to achieve CCA and HDP certification.</p>
        <h2>Approach</h2><p>A complete course packed with step-by-step instructions, working examples, and helpful advice. This course is systematically divided into small sections that will help you understand each part individually and learn at your own pace.</p>
        <h2>Key Features</h2><p>Learn Sqoop, Flume, and Hive and successfully achieve CCA175 and Hortonworks Spark Certification  * Understand the Hadoop Distributed File System (HDFS), along with exploring Hadoop commands to work effectively with HDFS</p>
        <h2>Github Repo</h2><p>https://github.com/packtpublishing/master-big-data-ingestion-and-analytics-with-flume-sqoop-hive-and-spark</p>
        <h2>About the Author</h2><h3>Navdeep Kaur</h3><p>Navdeep Kaur - Technical Trainer
Navdeep Kaur is a big data professionals with 11 years of industry experience in different technologies and domains. She has a keen interest in providing training in new technologies. She has received CCA175 Hadoop and Spark developer certification and AWS solution architect certification. She loves guiding people and helping them achieves new goals.</p>
        <h2>Course Outline</h2><h4>1. Hadoop Introduction</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. HDFS and Hadoop Commands</p><p style='margin: 0'>Hadoop Introduction: HDFS and Hadoop Commands</p></td></tr></tbody></table><br><h4>2. Sqoop Import</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Sqoop Introduction</p><p style='margin: 0'>Sqoop Import: Sqoop Introduction</p></td></tr><tr><td style='padding: 15px;'><p>2. Managing Target Directories</p><p style='margin: 0'>Sqoop Import: Managing Target Directories</p></td></tr><tr><td style='padding: 15px;'><p>3. Working with Different File Formats</p><p style='margin: 0'>Sqoop Import: Working with Different File Formats</p></td></tr><tr><td style='padding: 15px;'><p>4. Working with Different Compressions</p><p style='margin: 0'>Sqoop Import: Working with Different Compressions</p></td></tr><tr><td style='padding: 15px;'><p>5. Conditional Imports</p><p style='margin: 0'>Sqoop Import: Conditional Imports</p></td></tr><tr><td style='padding: 15px;'><p>6. Split-by and Boundary Queries</p><p style='margin: 0'>Sqoop Import: Split-by and Boundary Queries</p></td></tr><tr><td style='padding: 15px;'><p>7. Field delimeters</p><p style='margin: 0'>Sqoop Import: Field delimeters</p></td></tr><tr><td style='padding: 15px;'><p>8. Incremental Appends</p><p style='margin: 0'>Sqoop Import: Incremental Appends</p></td></tr><tr><td style='padding: 15px;'><p>9. Sqoop Hive Import</p><p style='margin: 0'>Sqoop Import: Sqoop Hive Import</p></td></tr><tr><td style='padding: 15px;'><p>10. Sqoop List Tables/Database</p><p style='margin: 0'>Sqoop Import: Sqoop List Tables/Database</p></td></tr><tr><td style='padding: 15px;'><p>11. Sqoop Import Practice1</p><p style='margin: 0'>Sqoop Import: Sqoop Import Practice1</p></td></tr><tr><td style='padding: 15px;'><p>12. Sqoop Import Practice2</p><p style='margin: 0'>Sqoop Import: Sqoop Import Practice2</p></td></tr><tr><td style='padding: 15px;'><p>13. Sqoop Import Practice3</p><p style='margin: 0'>Sqoop Import: Sqoop Import Practice3</p></td></tr></tbody></table><br><h4>3. Sqoop Export</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Export from Hdfs to Mysql</p><p style='margin: 0'>Sqoop Export: Export from Hdfs to Mysql</p></td></tr><tr><td style='padding: 15px;'><p>2. Export from Hive to Mysql</p><p style='margin: 0'>Sqoop Export: Export from Hive to Mysql</p></td></tr></tbody></table><br><h4>4. Apache Flume</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Flume Introduction & Architecture</p><p style='margin: 0'>Apache Flume: Flume Introduction & Architecture</p></td></tr><tr><td style='padding: 15px;'><p>2. Exec Source and Logger Sink</p><p style='margin: 0'>Apache Flume: Exec Source and Logger Sink</p></td></tr><tr><td style='padding: 15px;'><p>3. Moving data from Twitter to HDFS</p><p style='margin: 0'>Apache Flume: Moving data from Twitter to HDFS</p></td></tr><tr><td style='padding: 15px;'><p>4. Moving data from NetCat to HDFS</p><p style='margin: 0'>Apache Flume: Moving data from NetCat to HDFS</p></td></tr><tr><td style='padding: 15px;'><p>5. Flume Interceptors</p><p style='margin: 0'>Apache Flume: Flume Interceptors</p></td></tr><tr><td style='padding: 15px;'><p>6. Flume Interceptor Example</p><p style='margin: 0'>Apache Flume: Flume Interceptor Example</p></td></tr><tr><td style='padding: 15px;'><p>7. Flume Multi-Agent Flow</p><p style='margin: 0'>Apache Flume: Flume Multi-Agent Flow</p></td></tr><tr><td style='padding: 15px;'><p>8. Flume Consolidation</p><p style='margin: 0'>Apache Flume: Flume Consolidation</p></td></tr></tbody></table><br><h4>5. Apache Hive</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Hive Introduction</p><p style='margin: 0'>Apache Hive: Hive Introduction</p></td></tr><tr><td style='padding: 15px;'><p>2. Hive Database</p><p style='margin: 0'>Apache Hive: Hive Database</p></td></tr><tr><td style='padding: 15px;'><p>3. Hive Managed Tables</p><p style='margin: 0'>Apache Hive: Hive Managed Tables</p></td></tr><tr><td style='padding: 15px;'><p>4. Hive External Tables</p><p style='margin: 0'>Apache Hive: Hive External Tables</p></td></tr><tr><td style='padding: 15px;'><p>5. Hive Inserts</p><p style='margin: 0'>Apache Hive: Hive Inserts</p></td></tr><tr><td style='padding: 15px;'><p>6. Hive Analytics</p><p style='margin: 0'>Apache Hive: Hive Analytics</p></td></tr><tr><td style='padding: 15px;'><p>7. Working with Parquet</p><p style='margin: 0'>Apache Hive: Working with Parquet</p></td></tr><tr><td style='padding: 15px;'><p>8. Compressing Parquet</p><p style='margin: 0'>Apache Hive: Compressing Parquet</p></td></tr><tr><td style='padding: 15px;'><p>9. Working with Fixed File Format</p><p style='margin: 0'>Apache Hive: Working with Fixed File Format</p></td></tr><tr><td style='padding: 15px;'><p>10. Alter Command</p><p style='margin: 0'>Apache Hive: Alter Command</p></td></tr><tr><td style='padding: 15px;'><p>11. Hive String Functions</p><p style='margin: 0'>Apache Hive: Hive String Functions</p></td></tr><tr><td style='padding: 15px;'><p>12. Hive Date Functions</p><p style='margin: 0'>Apache Hive: Hive Date Functions</p></td></tr><tr><td style='padding: 15px;'><p>13. Hive Partitioning</p><p style='margin: 0'>Apache Hive: Hive Partitioning</p></td></tr><tr><td style='padding: 15px;'><p>14. Hive Bucketing</p><p style='margin: 0'>Apache Hive: Hive Bucketing</p></td></tr></tbody></table><br><h4>6. Spark Introduction</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Spark Introduction</p><p style='margin: 0'>Spark Introduction: Spark Introduction</p></td></tr><tr><td style='padding: 15px;'><p>2. Resilient Distributed Datasets</p><p style='margin: 0'>Spark Introduction: Resilient Distributed Datasets</p></td></tr><tr><td style='padding: 15px;'><p>3. Cluster Overview</p><p style='margin: 0'>Spark Introduction: Cluster Overview</p></td></tr><tr><td style='padding: 15px;'><p>4. Directed Acyclic Graph (DAG) & Stages</p><p style='margin: 0'>Spark Introduction: Directed Acyclic Graph (DAG) & Stages</p></td></tr></tbody></table><br><h4>7. Spark Transformations & Actions</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Map/FlatMap Transformation</p><p style='margin: 0'>Spark Transformations & Actions: Map/FlatMap Transformation</p></td></tr><tr><td style='padding: 15px;'><p>2. Filter/Intersection</p><p style='margin: 0'>Spark Transformations & Actions: Filter/Intersection</p></td></tr><tr><td style='padding: 15px;'><p>3. Union/Distinct Transformation</p><p style='margin: 0'>Spark Transformations & Actions: Union/Distinct Transformation</p></td></tr><tr><td style='padding: 15px;'><p>4. GroupByKey/ Group people based on Birthday months</p><p style='margin: 0'>Spark Transformations & Actions: GroupByKey/ Group people based on Birthday months</p></td></tr><tr><td style='padding: 15px;'><p>5. ReduceByKey / Total Number of students in each Subject</p><p style='margin: 0'>Spark Transformations & Actions: ReduceByKey / Total Number of students in each Subject</p></td></tr><tr><td style='padding: 15px;'><p>6. SortByKey / Sort students based on their rollno</p><p style='margin: 0'>Spark Transformations & Actions: SortByKey / Sort students based on their rollno</p></td></tr><tr><td style='padding: 15px;'><p>7. MapPartition / MapPartitionWithIndex</p><p style='margin: 0'>Spark Transformations & Actions: MapPartition / MapPartitionWithIndex</p></td></tr><tr><td style='padding: 15px;'><p>8. Change number of Partitions</p><p style='margin: 0'>Spark Transformations & Actions: Change number of Partitions</p></td></tr><tr><td style='padding: 15px;'><p>9. Join / Join email address based on customer name</p><p style='margin: 0'>Spark Transformations & Actions: Join / Join email address based on customer name</p></td></tr><tr><td style='padding: 15px;'><p>10. Spark Actions</p><p style='margin: 0'>Spark Transformations & Actions: Spark Actions</p></td></tr></tbody></table><br><h4>8. Spark RDD Practice</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Scala Tuples</p><p style='margin: 0'>Spark RDD Practice: Scala Tuples</p></td></tr><tr><td style='padding: 15px;'><p>2. Extract Error Logs from log files</p><p style='margin: 0'>Spark RDD Practice: Extract Error Logs from log files</p></td></tr><tr><td style='padding: 15px;'><p>3. Frequency of word in Text File</p><p style='margin: 0'>Spark RDD Practice: Frequency of word in Text File</p></td></tr><tr><td style='padding: 15px;'><p>4. Population of each City</p><p style='margin: 0'>Spark RDD Practice: Population of each City</p></td></tr><tr><td style='padding: 15px;'><p>5. Orders placed by Customers</p><p style='margin: 0'>Spark RDD Practice: Orders placed by Customers</p></td></tr><tr><td style='padding: 15px;'><p>6. Movie Average Rating greater than 3</p><p style='margin: 0'>Spark RDD Practice: Movie Average Rating greater than 3</p></td></tr></tbody></table><br><h4>9. Spark Dataframes & Spark SQL</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Dataframe Intro</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Dataframe Intro</p></td></tr><tr><td style='padding: 15px;'><p>2. Dafaframe from Json Files</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Dafaframe from Json Files</p></td></tr><tr><td style='padding: 15px;'><p>3. Dataframe from Parquet Files</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Dataframe from Parquet Files</p></td></tr><tr><td style='padding: 15px;'><p>4. Dataframe from CSV Files</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Dataframe from CSV Files</p></td></tr><tr><td style='padding: 15px;'><p>5. Dataframe from Avro/XML Files</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Dataframe from Avro/XML Files</p></td></tr><tr><td style='padding: 15px;'><p>6. Working with Different Compressions</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Working with Different Compressions</p></td></tr><tr><td style='padding: 15px;'><p>7. DataFrame API Part1</p><p style='margin: 0'>Spark Dataframes & Spark SQL: DataFrame API Part1</p></td></tr><tr><td style='padding: 15px;'><p>8. DataFrame API Part2</p><p style='margin: 0'>Spark Dataframes & Spark SQL: DataFrame API Part2</p></td></tr><tr><td style='padding: 15px;'><p>9. Spark SQL</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Spark SQL</p></td></tr><tr><td style='padding: 15px;'><p>10. Working with Hive Tables in Spark</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Working with Hive Tables in Spark</p></td></tr></tbody></table>

A complete course on Sqoop, Flume, and Hive: Ideal for achieving CCA175 and Hortonworks Spark Certification
In this course, you will start by learning about the Hadoop Distributed File System (HDFS) and the most common Hadoop commands required to work with HDFS. Next, you'll be introduced to Sqoop Import, which will help you gain insights into the lifecycle of the Sqoop command and how to use the import command to migrate data from MySQL to HDFS, and from MySQL to Hive. In addition to this, you will get up to speed with Sqoop Export for migrating data effectively, along with using Apache Flume to ingest data. As you progress, you will delve into Apache Hive, external and managed tables, working with different files, and Parquet and Avro. Toward the concluding section, you will focus on Spark DataFrames and Spark SQL. By the end of this course, you will have gained comprehensive insights into big data ingestion and analytics with Flume, Sqoop, Hive, and Spark. All code and supporting files are available at - https://github.com/PacktPublishing/Master-Big-Data-Ingestion-and-Analytics-with-Flume-Sqoop-Hive-and-Spark
What You Will Learn
Explore the Hadoop Distributed File System (HDFS) and commands 
Get to grips with the lifecycle of the Sqoop command 
Use the Sqoop Import command to migrate data from MySQL to HDFS and Hive 
Understand split-by and boundary queries 
Use the incremental mode to migrate data from MySQL to HDFS 
Employ Sqoop Export to migrate data from HDFS to MySQL 
Discover Spark DataFrames and gain insights into working with different file formats and compression
Audience
This course is for anyone who wants to learn Sqoop and Flume or those looking to achieve CCA and HDP certification.
Approach
A complete course packed with step-by-step instructions, working examples, and helpful advice. This course is systematically divided into small sections that will help you understand each part individually and learn at your own pace.
Key Features
Learn Sqoop, Flume, and Hive and successfully achieve CCA175 and Hortonworks Spark Certification * Understand the Hadoop Distributed File System (HDFS), along with exploring Hadoop commands to work effectively with HDFS
Github Repo
https://github.com/packtpublishing/master-big-data-ingestion-and-analytics-with-flume-sqoop-hive-and-spark
About the Author
Navdeep Kaur
Navdeep Kaur - Technical Trainer
Navdeep Kaur is a big data professionals with 11 years of industry experience in different technologies and domains. She has a keen interest in providing training in new technologies. She has received CCA175 Hadoop and Spark developer certification and AWS solution architect certification. She loves guiding people and helping them achieves new goals.
Course Outline
1. Hadoop Introduction

1. HDFS and Hadoop Commands
Hadoop Introduction: HDFS and Hadoop Commands
2. Sqoop Import

1. Sqoop Introduction
Sqoop Import: Sqoop Introduction
2. Managing Target Directories
Sqoop Import: Managing Target Directories
3. Working with Different File Formats
Sqoop Import: Working with Different File Formats
4. Working with Different Compressions
Sqoop Import: Working with Different Compressions
5. Conditional Imports
Sqoop Import: Conditional Imports
6. Split-by and Boundary Queries
Sqoop Import: Split-by and Boundary Queries
7. Field delimeters
Sqoop Import: Field delimeters
8. Incremental Appends
Sqoop Import: Incremental Appends
9. Sqoop Hive Import
Sqoop Import: Sqoop Hive Import
10. Sqoop List Tables/Database
Sqoop Import: Sqoop List Tables/Database
11. Sqoop Import Practice1
Sqoop Import: Sqoop Import Practice1
12. Sqoop Import Practice2
Sqoop Import: Sqoop Import Practice2
13. Sqoop Import Practice3
Sqoop Import: Sqoop Import Practice3
3. Sqoop Export

1. Export from Hdfs to Mysql
Sqoop Export: Export from Hdfs to Mysql
2. Export from Hive to Mysql
Sqoop Export: Export from Hive to Mysql
4. Apache Flume

1. Flume Introduction & Architecture
Apache Flume: Flume Introduction & Architecture
2. Exec Source and Logger Sink
Apache Flume: Exec Source and Logger Sink
3. Moving data from Twitter to HDFS
Apache Flume: Moving data from Twitter to HDFS
4. Moving data from NetCat to HDFS
Apache Flume: Moving data from NetCat to HDFS
5. Flume Interceptors
Apache Flume: Flume Interceptors
6. Flume Interceptor Example
Apache Flume: Flume Interceptor Example
7. Flume Multi-Agent Flow
Apache Flume: Flume Multi-Agent Flow
8. Flume Consolidation
Apache Flume: Flume Consolidation
5. Apache Hive

1. Hive Introduction
Apache Hive: Hive Introduction
2. Hive Database
Apache Hive: Hive Database
3. Hive Managed Tables
Apache Hive: Hive Managed Tables
4. Hive External Tables
Apache Hive: Hive External Tables
5. Hive Inserts
Apache Hive: Hive Inserts
6. Hive Analytics
Apache Hive: Hive Analytics
7. Working with Parquet
Apache Hive: Working with Parquet
8. Compressing Parquet
Apache Hive: Compressing Parquet
9. Working with Fixed File Format
Apache Hive: Working with Fixed File Format
10. Alter Command
Apache Hive: Alter Command
11. Hive String Functions
Apache Hive: Hive String Functions
12. Hive Date Functions
Apache Hive: Hive Date Functions
13. Hive Partitioning
Apache Hive: Hive Partitioning
14. Hive Bucketing
Apache Hive: Hive Bucketing
6. Spark Introduction

1. Spark Introduction
Spark Introduction: Spark Introduction
2. Resilient Distributed Datasets
Spark Introduction: Resilient Distributed Datasets
3. Cluster Overview
Spark Introduction: Cluster Overview
4. Directed Acyclic Graph (DAG) & Stages
Spark Introduction: Directed Acyclic Graph (DAG) & Stages
7. Spark Transformations & Actions

1. Map/FlatMap Transformation
Spark Transformations & Actions: Map/FlatMap Transformation
2. Filter/Intersection
Spark Transformations & Actions: Filter/Intersection
3. Union/Distinct Transformation
Spark Transformations & Actions: Union/Distinct Transformation
4. GroupByKey/ Group people based on Birthday months
Spark Transformations & Actions: GroupByKey/ Group people based on Birthday months
5. ReduceByKey / Total Number of students in each Subject
Spark Transformations & Actions: ReduceByKey / Total Number of students in each Subject
6. SortByKey / Sort students based on their rollno
Spark Transformations & Actions: SortByKey / Sort students based on their rollno
7. MapPartition / MapPartitionWithIndex
Spark Transformations & Actions: MapPartition / MapPartitionWithIndex
8. Change number of Partitions
Spark Transformations & Actions: Change number of Partitions
9. Join / Join email address based on customer name
Spark Transformations & Actions: Join / Join email address based on customer name
10. Spark Actions
Spark Transformations & Actions: Spark Actions
8. Spark RDD Practice

1. Scala Tuples
Spark RDD Practice: Scala Tuples
2. Extract Error Logs from log files
Spark RDD Practice: Extract Error Logs from log files
3. Frequency of word in Text File
Spark RDD Practice: Frequency of word in Text File
4. Population of each City
Spark RDD Practice: Population of each City
5. Orders placed by Customers
Spark RDD Practice: Orders placed by Customers
6. Movie Average Rating greater than 3
Spark RDD Practice: Movie Average Rating greater than 3
9. Spark Dataframes & Spark SQL

1. Dataframe Intro
Spark Dataframes & Spark SQL: Dataframe Intro
2. Dafaframe from Json Files
Spark Dataframes & Spark SQL: Dafaframe from Json Files
3. Dataframe from Parquet Files
Spark Dataframes & Spark SQL: Dataframe from Parquet Files
4. Dataframe from CSV Files
Spark Dataframes & Spark SQL: Dataframe from CSV Files
5. Dataframe from Avro/XML Files
Spark Dataframes & Spark SQL: Dataframe from Avro/XML Files
6. Working with Different Compressions
Spark Dataframes & Spark SQL: Working with Different Compressions
7. DataFrame API Part1
Spark Dataframes & Spark SQL: DataFrame API Part1
8. DataFrame API Part2
Spark Dataframes & Spark SQL: DataFrame API Part2
9. Spark SQL
Spark Dataframes & Spark SQL: Spark SQL
10. Working with Hive Tables in Spark
Spark Dataframes & Spark SQL: Working with Hive Tables in Spark

A complete course on Sqoop, Flume, and Hive: Ideal for achieving CCA175 and Hortonworks Spark Certification

Master Big Data Ingestion and Analytics with Flume, Sqoop, Hive and Spark

Learn core Apache Kafka features along with creating Java, Node.js and Python producers and consumers
Explore an interactive and easy way to learn Apache Kafka with this practical guide. Featuring interesting activities to help you build projects using APIs for programming languages such as Node.js and Python, this course will ensure you get hands-on experience.
You will learn how Apache Kafka works and also understand its core features. After delving into the architecture of Kafka, you will focus on how to build custom Apache Kafka producers and consumers using native Java API. The course will gradually assist you in working through a variety of activities related to Apache Kafka, from creating a cluster with multiple brokers and developing topics with replication factors, through to launching the performance monitor for testing the performance of consumers and producers. The course will also demonstrate how to use Apache Kafka API to create your own consumers and producers using Python and Node.js.
By the end of this course, you'll be well-versed with Apache Kafka and have the skills and confidence you need to build projects using it. All codes and supporting files are available at-
https://github.com/PacktPublishing/Apache-Kafka-Complete-Developer-s-Guide

What You Will Learn
Get to grips with the Apache Kafka architecture
Create producers and consumers using a built-in console
Use Python and Node.js to produce and consume messages
Design fault-tolerant clusters with topic replication across multiple brokers
Build a Kafka cluster with multiple brokers
Delve into the components of the Apache Kafka cluster
Audience
This course is designed for beginners who want to learn Apache Kafka and its architecture. Anyone who wants to build producers and consumers in other languages such as Node.js and Python, or anyone who wants to build projects using Kafka and its APIs will find this practice-oriented course helpful.
Approach
This course uses a complete practice-based approach, which ensures that students get hands-on experience in building projects using different APIs.
Key Features
Understand how Apache Kafka works and its core features * Build custom Apache Kafka producers and consumers using native Java API * Learn and practice how to run multiple brokers on the same computer
Github Repo
https://github.com/packtpublishing/apache-kafka-complete-developer-s-guide
About the Author
Bogdan Stashchuk
Bogdan Stashchuk has over 20 years of experience as a software engineering instructor. He excels at breaking down complex topics into easy-to-follow steps. His courses are designed with hands-on exercises, ensuring that learners can actively participate and apply what they learn. From start to finish, students can follow along and complete tasks just as Bogdan demonstrates in his lectures. He also includes challenging assignments with detailed solutions. This approach helps learners understand and remember the material long after they've completed the course. Through his dedication and expertise, Bogdan ensures a valuable and effective learning experience for everyone.
Course Outline
1. Apache Kafka Installation Overview

1. Apache Kafka Installation Overview
Apache Kafka Installation Overview: Apache Kafka Installation Overview
2. Installing Apache Kafka on the Mac and Unix-like systems
Apache Kafka Installation Overview: Installing Apache Kafka on the Mac and Unix-like systems
3. Installing Apache Kafka on the Mac
Apache Kafka Installation Overview: Installing Apache Kafka on the Mac
4. Installing Ubuntu on MacOS using VirtualBox
Apache Kafka Installation Overview: Installing Ubuntu on MacOS using VirtualBox
2. Installing Apache Kafka on the remote Ubuntu server

1. SECTION 2 Introduction
Installing Apache Kafka on the remote Ubuntu server: SECTION 2 Introduction
2. Creating remote Ubuntu Virtual Private Server
Installing Apache Kafka on the remote Ubuntu server: Creating remote Ubuntu Virtual Private Server
3. Installing Apache Kafka on Virtual Private Server
Installing Apache Kafka on the remote Ubuntu server: Installing Apache Kafka on Virtual Private Server
3. Installing Apache Kafka on Windows

1. SECTION 3 Introduction
Installing Apache Kafka on Windows: SECTION 3 Introduction
2. Installing Apache Kafka on Windows
Installing Apache Kafka on Windows: Installing Apache Kafka on Windows
3. Starting Zookeeper and Kafka server on Windows
Installing Apache Kafka on Windows: Starting Zookeeper and Kafka server on Windows
4. Installing Ubuntu on Windows using VirtualBox
Installing Apache Kafka on Windows: Installing Ubuntu on Windows using VirtualBox
5. Installing Apache Kafka on Ubuntu using GUI
Installing Apache Kafka on Windows: Installing Apache Kafka on Ubuntu using GUI
4. Starting Apache Zookeeper and Kafka Broker

1. SECTION 4 Introduction
Starting Apache Zookeeper and Kafka Broker: SECTION 4 Introduction
2. Observing contents of the Kafka folder
Starting Apache Zookeeper and Kafka Broker: Observing contents of the Kafka folder
3. Reading and editing Kafka files using VisualStudio Code
Starting Apache Zookeeper and Kafka Broker: Reading and editing Kafka files using VisualStudio Code
4. Trying to start Kafka Server
Starting Apache Zookeeper and Kafka Broker: Trying to start Kafka Server
5. Observing Kafka Server logs
Starting Apache Zookeeper and Kafka Broker: Observing Kafka Server logs
6. Starting Zookeeper
Starting Apache Zookeeper and Kafka Broker: Starting Zookeeper
7. Starting Kafka Server while Zookeeper is up and running
Starting Apache Zookeeper and Kafka Broker: Starting Kafka Server while Zookeeper is up and running
8. Observing logs folder and current kafka server setup
Starting Apache Zookeeper and Kafka Broker: Observing logs folder and current kafka server setup
5. Creating and exploring Kafka Topic

1. SECTION 5 Introduction
Creating and exploring Kafka Topic: SECTION 5 Introduction
2. How to connect to Kafka cluster
Creating and exploring Kafka Topic: How to connect to Kafka cluster
3. Create new Kafka topic
Creating and exploring Kafka Topic: Create new Kafka topic
4. What happened after creation of the new topic
Creating and exploring Kafka Topic: What happened after creation of the new topic
5. Read details about topic
Creating and exploring Kafka Topic: Read details about topic
6. Producing and consuming Messages

1. SECTION 6 Introduction
Producing and consuming Messages: SECTION 6 Introduction
2. Send some messages using Kafka Console Producer
Producing and consuming Messages: Send some messages using Kafka Console Producer
3. Consuming messages using Kafka Console Consumer
Producing and consuming Messages: Consuming messages using Kafka Console Consumer
4. Consuming messages from the beginning
Producing and consuming Messages: Consuming messages from the beginning
5. Running multiple consumers
Producing and consuming Messages: Running multiple consumers
6. Running multiple producers
Producing and consuming Messages: Running multiple producers
7. What was changed in the Kafka logs
Producing and consuming Messages: What was changed in the Kafka logs
7. What is Apache Kafka and how it works

1. SECTION 7 Introduction
What is Apache Kafka and how it works: SECTION 7 Introduction:
2. What is Apache Kafka
What is Apache Kafka and how it works: What is Apache Kafka
3. Broker
What is Apache Kafka and how it works: Broker
4. Broker cluster
What is Apache Kafka and how it works: Broker cluster
5. Zookeeper
What is Apache Kafka and how it works: Zookeeper
6. Zookeeper ensemble
What is Apache Kafka and how it works: Zookeeper ensemble
7. Multiple Kafka clusters
What is Apache Kafka and how it works: Multiple Kafka clusters
8. Default ports of Zookeeper and Broker
What is Apache Kafka and how it works: Default ports of Zookeeper and Broker
9. Kafka Topic
What is Apache Kafka and how it works: Kafka Topic
10. Message structure
What is Apache Kafka and how it works: Message structure
11. Topics and Partitions
What is Apache Kafka and how it works: Topic and Partitions
12. Spreading messages across partitions
What is Apache Kafka and how it works: Spreading messages across partitions
13. Partition Leader and Followers
What is Apache Kafka and how it works: Partition Leader and Followers
14. Controller and it's responsibilities
What is Apache Kafka and how it works: Controller and it's responsibilities
15. How Producers write messages to the topic
What is Apache Kafka and how it works: How Producers write messages to the topic
16. How Consumers read messages from the topic
What is Apache Kafka and how it works: How Consumers read messages from the topic
8. GitHub Repository and Diagrams for the course

1. SECTION 8 Introduction
GitHub Repository and Diagrams for the course: SECTION 8 Introduction
2. GitHub repository and list of basic Kafka commands
GitHub Repository and Diagrams for the course: GitHub repository and list of basic Kafka commands
3. Diagrams for the course
GitHub Repository and Diagrams for the course: Diagrams for the course
9. EXAMPLE 1 - Topic with Multiple Partitions

1. SECTION 9 Introduction
EXAMPLE 1 - Topic with Multiple Partitions: SECTION 9 Introduction
2. Cleaning up existing kafka installation
EXAMPLE 1 - Topic with Multiple Partitions: Cleaning up existing kafka installation
3. Creating topic with multiple partitions
EXAMPLE 1 - Topic with Multiple Partitions: Creating topic with multiple partitions
4. How messages were spread across different partitions
EXAMPLE 1 - Topic with Multiple Partitions: How messages were spread across different partitions
5. Reading messages from specific partition
EXAMPLE 1 - Topic with Multiple Partitions: Reading messages from specific partition
6. Reading messages from specific offset in specific partition
EXAMPLE 1 - Topic with Multiple Partitions: Reading messages from specific offset in specific partition
7. Reading details about topic and __consumer_offsets topic
EXAMPLE 1 - Topic with Multiple Partitions: Reading details about topic and __consumer_offsets topic
8. Summary for multiple partitions example
EXAMPLE 1 - Topic with Multiple Partitions: Summary for multiple partitions example
10. EXAMPLE 2 - Kafka Cluster with Multiple Brokers

1. SECTION 10 Introduction
EXAMPLE 2 - Kafka Cluster with Multiple Brokers: SECTION 10 Introduction
2. Example overview - run multiple brokers
EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Example overview - run multiple brokers
3. Creating separate configuration files for brokers
EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Creating separate configuration files for brokers
4. Launching three brokers
EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Launching three brokers
5. Getting cluster information and broker details from Zookeeper
EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Getting cluster information and broker details from Zookeeper
6. Creating multiple-partition topic in the Kafka cluster
EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Creating multiple-partition topic in the Kafka cluster
7. Looking at logs folders of every broker
EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Looking at logs folders of every broker
8. Producing and consuming messages in the cluster
EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Producing and consuming messages in the cluster
9. Details about topic in the cluster
EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Details about topic in the cluster
10. Simulating broker failure in the cluster
EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Simulating broker failure in the cluster
11. Summary for broker cluster and topic without replication
EXAMPLE 2 - Kafka Cluster with Multiple Brokers: Summary for broker cluster and topic without replication
11. EXAMPLE 3 - Multiple Brokers and Topic with Replication

1. SECTION 11 Introduction
EXAMPLE 3 - Multiple Brokers and Topic with Replication: SECTION 11 Introduction
2. Preparing for the next example with replication
EXAMPLE 3 - Multiple Brokers and Topic with Replication: Preparing for the next example with replication
3. Launching brokers and creating topic with replication
EXAMPLE 3 - Multiple Brokers and Topic with Replication: Launching brokers and creating topic with replication
4. Observing logs folder and details of the topic
EXAMPLE 3 - Multiple Brokers and Topic with Replication: Observing logs folder and details of the topic
5. Producing and consuming messages in the topic with replication
EXAMPLE 3 - Multiple Brokers and Topic with Replication: Producing and consuming messages in the topic with replication
6. Observing how messages were stored in the partitions on different brokers
EXAMPLE 3 - Multiple Brokers and Topic with Replication: Observing how messages were stored in the partitions on different brokers
7. Bringing down one of three brokers and observing changes
EXAMPLE 3 - Multiple Brokers and Topic with Replication: Bringing down one of three brokers and observing changes
8. Bringing down another broker in the cluster
EXAMPLE 3 - Multiple Brokers and Topic with Replication: Bringing down another broker in the cluster
9. Bringing back both brokers
EXAMPLE 3 - Multiple Brokers and Topic with Replication: Bringing back both brokers
10. Summary for replication
EXAMPLE 3 - Multiple Brokers and Topic with Replication: Summary for replication
12. EXAMPLE 4 - Kafka Consumer Groups

1. SECTION 12 Introduction
EXAMPLE 4 - Kafka Consumer Groups: SECTION 12 Introduction
2. Example with consumer groups overview
EXAMPLE 4 - Kafka Consumer Groups: Example with consumer groups overview
3. Exploring default consumer groups
EXAMPLE 4 - Kafka Consumer Groups: Exploring default consumer groups
4. Starting consumer in the custom consumer group
EXAMPLE 4 - Kafka Consumer Groups: Starting consumer in the custom consumer group
5. Starting second consumer in the same consumer group
EXAMPLE 4 - Kafka Consumer Groups: Starting second consumer in the same consumer group
6. Launching one more consumer in the same group
EXAMPLE 4 - Kafka Consumer Groups: Launching one more consumer in the same group
7. Idle consumers in the group
EXAMPLE 4 - Kafka Consumer Groups: Idle consumers in the group
8. Summary for consumer groups
EXAMPLE 4 - Kafka Consumer Groups: Summary for consumer groups
13. EXAMPLE 5 - Performance Testing

1. SECTION 13 Introduction
EXAMPLE 5 - Performance Testing: SECTION 13 Introduction
2. Overview of the performance testing example
EXAMPLE 5 - Performance Testing: Overview of the performance testing example
3. Starting cluster and launching basic performance test
EXAMPLE 5 - Performance Testing: Starting cluster and launching basic performance test
4. Increasing performance test parameters
EXAMPLE 5 - Performance Testing: Increasing performance test parameters
5. Testing consumer performance
EXAMPLE 5 - Performance Testing: Testing consumer performance
6. Getting non-zero LAG values for consumers
EXAMPLE 5 - Performance Testing: Getting non-zero LAG values for consumers
7. Performance test example summary
EXAMPLE 5 - Performance Testing: Performance test example summary
14. PROJECT 1 - Java

1. SECTION 14 Introduction
PROJECT 1 - Java: SECTION 14 Introduction
2. Project Files for the Java section
PROJECT 1 - Java: Project Files for the Java section
3. Installing IntelliJ editor
PROJECT 1 - Java: Installing IntelliJ editor
4. Creating and configuring Maven project
PROJECT 1 - Java: Creating and configuring Maven project
5. Starting Kafka Cluster
PROJECT 1 - Java: Starting Kafka Cluster
6. Creating Java Producer
PROJECT 1 - Java: Creating Java Producer
7. Continue Creating Java Producer
PROJECT 1 - Java: Continue Creating Java Producer
8. Launching Java Producer
PROJECT 1 - Java: Launching Java Producer
9. Explaining most common Producer parameters
PROJECT 1 - Java: Explaining most common Producer parameters
10. Modifying Serializer type
PROJECT 1 - Java: Modifying Serializer type
11. Producing meaningful messages with delay
PROJECT 1 - Java: Producing meaningful messages with delay
12. Refactoring Producer by adding previous example
PROJECT 1 - Java: Refactoring Producer by adding previous example
13. Creating consumer with autocommitting - PART 1
PROJECT 1 - Java: Creating consumer with autocommitting - PART 1
14. Creating consumer with autocommitting - PART 2
PROJECT 1 - Java: Creating consumer with autocommitting - PART 2
15. Consumer parameters overview
PROJECT 1 - Java: Consumer parameters overview
16. Consumer with Manual Committing
PROJECT 1 - Java: Consumer with Manual Committing
17. Consumer with Partitions Assignment
PROJECT 1 - Java: Consumer with Partitions Assignment
18. Launching multiple consumers in the same consumer group
PROJECT 1 - Java: Launching multiple consumers in the same consumer group
19. CHALLENGE - Subscribe vs Assign with consumer groups
PROJECT 1 - Java: CHALLENGE - Subscribe vs Assign with consumer groups
15. PROJECT 2 - Node.js

1. SECTION 15 Introduction
PROJECT 2 - Node.js: SECTION 15 Introduction
2. Installing Node.js with NPM
PROJECT 2 - Node.js: Installing Node.js with NPM
3. Starting up Kafka cluster with 3 brokers
PROJECT 2 - Node.js: Starting up Kafka cluster with 3 brokers
4. Initializing Node.js project
PROJECT 2 - Node.js: Initializing Node.js project
5. Final Node.js project filesv
PROJECT 2 - Node.js: Final Node.js project files
6. Creating basic Node.js producer
PROJECT 2 - Node.js: Creating basic Node.js producer
7. Producing random animal names
PROJECT 2 - Node.js: Producing random animal names
8. Creating Node.js consumer
PROJECT 2 - Node.js: Creating Node.js consumer
16. PROJECT 3 - Python

1. SECTION 16 Introduction
PROJECT 3 - Python: SECTION 16 Introduction
2. Installing Python
PROJECT 3 - Python: Installing Python
3. Final Python project files
PROJECT 3 - Python: Final Python project files
4. Launching basic Python producer
PROJECT 3 - Python: Launching basic Python producer
5. Launching consumer and receiving messages
PROJECT 3 - Python: Launching consumer and receiving messages
6. Generating fake names in the messages by producer
PROJECT 3 - Python: Generating fake names in the messages by producer
17. Course Summary

1. Course Summary
Course Summary: Course Summary

Node.js

Apache Kafka

Learn core Apache Kafka features along with creating Java, Node.js and Python producers and consumers

Apache Kafka Complete Developer's Guide

<strong>Explore the Apache Kafka ecosystem and architecture, and learn client API programming in Java</strong>
        <p>This course is designed to get you up and running with the fundamentals and the working of Apache Kafka with the help of practical examples. You will also delve into client API programming in Java.  As you progress, you will not only cover the Apache Kafka stack, but also the architecture of Kafka components and Kafka client APIs (Producers and Consumers). The course will even guide you on how to apply the knowledge you've gained through the course to create efficient Kafka programs in Java.
This course is based on Apache Kafka 2.x. All the source code and examples used in this course on Apache Kafka 2.3 open-source distribution have been tested. Several examples in this course also make use of the Confluent Community Version of Kafka. The course will focus on the Confluent Community Version to explain and demonstrate functionalities that are only available in the Confluent Platform, such as Schema Registry and Avro Serdes.
This course will be making extensive use of IntelliJ IDEA as the preferred development IDE. However, based on your prior experience, you should be able to work with any other IDE designed for Java application development.
Although the course will be using Apache Maven as the preferred build tool, based on your prior experience, you should be able to use any other build tool designed for Java applications.
In addition to this, the course will use Log4j2 to teach you industry-standard log implementation in your applications.
This course is fully example-driven and several examples will be created in the class.
By the end of this course, you will have gained the skills you need to confidently build Kafka programs in Java.

All the codes and supporting files are available at-
https://github.com/PacktPublishing/Apache-Kafka-for-absolute-beginners</p>
        <h2>What You Will Learn</h2><p>Delve into the multi-node Confluent Community of Kafka<br>Discover the Kafka Storage architecture<br>Gain insights into the Kafka Cluster architecture<br>Understand Kafka Consumer API programming in Java<br>Get up to speed with JSON and AVRO Serialization</p>
        <h2>Audience</h2><p>This course is designed for software engineers, solution architects, and managers looking to implement Kafka and solve real-time stream processing problems. Architects who are not directly involved in Kafka implementation or anyone who is looking to implement Kafka Streams will also find this course helpful. This course is designed for both beginner and intermediate-level Apache Kafka users.</p>
        <h2>Approach</h2><p>This course features several working examples that will help you effectively apply the knowledge you gain through the course.</p>
        <h2>Key Features</h2><p>Get to grips with the Apache Kafka ecosystem and components * Delve into Kafka Producer API programming in Java</p>
        <h2>Github Repo</h2><p>https://github.com/packtpublishing/apache-kafka-for-absolute-beginners</p>
        <h2>About the Author</h2><h3>Prashant Kumar Pandey</h3><p>Prashant Kumar Pandey is passionate about helping people learn and grow in their careers by bridging the gap between their existing and required skills. In his journey to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. He is also the founder, lead author, and chief editor of the Learning Journal portal that has been providing various skill development courses, training sessions, and technical articles since 2018.</p>
        <h2>Course Outline</h2><h4>1. Kafka Ecosystem - The Big Picture</h4><p></p><br><h4>2. Apache Kafka - Getting Started</h4><p></p><br><h4>3. Apache Kafka - Storage Architecture</h4><p></p><br><h4>4. Apache Kafka - Cluster Architecture</h4><p></p><br><h4>5. Kafka Producer Internals</h4><p></p><br><h4>6. Advanced Kafka Producers</h4><p></p><br><h4>7. Types and Serialization</h4><p></p><br><h4>8. Micro Project - Summing Up the Producer</h4><p></p><br><h4>9. Kafka Consumers</h4><p></p><br><h4>10. Where to go Next</h4><p></p>

Explore the Apache Kafka ecosystem and architecture, and learn client API programming in Java
This course is designed to get you up and running with the fundamentals and the working of Apache Kafka with the help of practical examples. You will also delve into client API programming in Java. As you progress, you will not only cover the Apache Kafka stack, but also the architecture of Kafka components and Kafka client APIs (Producers and Consumers). The course will even guide you on how to apply the knowledge you've gained through the course to create efficient Kafka programs in Java.
This course is based on Apache Kafka 2.x. All the source code and examples used in this course on Apache Kafka 2.3 open-source distribution have been tested. Several examples in this course also make use of the Confluent Community Version of Kafka. The course will focus on the Confluent Community Version to explain and demonstrate functionalities that are only available in the Confluent Platform, such as Schema Registry and Avro Serdes.
This course will be making extensive use of IntelliJ IDEA as the preferred development IDE. However, based on your prior experience, you should be able to work with any other IDE designed for Java application development.
Although the course will be using Apache Maven as the preferred build tool, based on your prior experience, you should be able to use any other build tool designed for Java applications.
In addition to this, the course will use Log4j2 to teach you industry-standard log implementation in your applications.
This course is fully example-driven and several examples will be created in the class.
By the end of this course, you will have gained the skills you need to confidently build Kafka programs in Java. All the codes and supporting files are available at-
https://github.com/PacktPublishing/Apache-Kafka-for-absolute-beginners
What You Will Learn
Delve into the multi-node Confluent Community of Kafka
Discover the Kafka Storage architecture
Gain insights into the Kafka Cluster architecture
Understand Kafka Consumer API programming in Java
Get up to speed with JSON and AVRO Serialization
Audience
This course is designed for software engineers, solution architects, and managers looking to implement Kafka and solve real-time stream processing problems. Architects who are not directly involved in Kafka implementation or anyone who is looking to implement Kafka Streams will also find this course helpful. This course is designed for both beginner and intermediate-level Apache Kafka users.
Approach
This course features several working examples that will help you effectively apply the knowledge you gain through the course.
Key Features
Get to grips with the Apache Kafka ecosystem and components * Delve into Kafka Producer API programming in Java
Github Repo
https://github.com/packtpublishing/apache-kafka-for-absolute-beginners
About the Author
Prashant Kumar Pandey
Prashant Kumar Pandey is passionate about helping people learn and grow in their careers by bridging the gap between their existing and required skills. In his journey to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. He is also the founder, lead author, and chief editor of the Learning Journal portal that has been providing various skill development courses, training sessions, and technical articles since 2018.
Course Outline
1. Kafka Ecosystem - The Big Picture

2. Apache Kafka - Getting Started

3. Apache Kafka - Storage Architecture

4. Apache Kafka - Cluster Architecture

5. Kafka Producer Internals

6. Advanced Kafka Producers

7. Types and Serialization

8. Micro Project - Summing Up the Producer

9. Kafka Consumers

10. Where to go Next


Explore the Apache Kafka ecosystem and architecture, and learn client API programming in Java

Apache Kafka for Absolute Beginners

<strong>A beginner's guide to mastering real-time stream processing using Apache Kafka and Kafka Streams API</strong>
        <p>If you want to understand the concept of stream processing, this course is for you. Using Kafka, the course will help you get to grips with real-time stream processing and enable you to apply that knowledge to learn Kafka programming techniques.
This course uses the Kafka Streams library available in Apache Kafka 2.x. All the source code and examples on Apache Kafka 2.3 open-source distribution have been tested. You'll understand and explore Confluent Platform functionalities such as Schema Registry and Avro Serdes using the Confluent Community Version. This course makes extensive use of IntelliJ IDEA and Apache Maven as the preferred development IDE. You'll leverage Log4J2 and JUnit5 for industry-standard log implementation in your application and implementing unit test cases, respectively.

The code bundle is available at https://github.com/PacktPublishing/Apache-Kafka---Real-time-Stream-Processing-Master-Class
</p>
        <h2>What You Will Learn</h2><p>Learn Apache Kafka fundamentals and Kafka Architecture<br>Understand how to create streams using the Kafka Producer APIs<br>Design, develop, and test real-time stream processing applications using the Kafka Streams library<br>Get well-versed with the Kafka Streams architecture, Streams DSL, and the Processor API<br>Support microservice architecture and implement interactive queries in Kafka Streams<br>Explore serializing and deserializing and work on JSON and AVRO messages</p>
        <h2>Audience</h2><p>The Kafka Streams - Real-time Stream Processing course is for software engineers looking to develop a stream processing application using the Kafka Streams library. This course will also help Java programmers who want to learn everything necessary to start implementing real-time streaming applications using Apache Kafka. Data architects and data engineers responsible for designing and building the organization's data-centric infrastructure will benefit from this course.</p>
        <h2>Approach</h2><p>The course uses a progressive approach to teach a complex subject. It contains some fully example-driven sections to help you grasp key concepts easily.</p>
        <h2>Key Features</h2><p>Design, develop, and test real-time stream processing applications using Kafka Streams * Explore the Kafka Streams architecture, Streams DSL, and exactly-once processing in Apache Kafka * Autogenerate Java Objects from JSON and AVRO Schema definition</p>
        <h2>Github Repo</h2><p>https://github.com/PacktPublishing/Apache-Kafka---Real-time-Stream-Processing-Master-Class</p>
        <h2>About the Author</h2><h3>Prashant Kumar Pandey</h3><p>Prashant Kumar Pandey is passionate about helping people learn and grow in their careers by bridging the gap between their existing and required skills. In his journey to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. He is also the founder, lead author, and chief editor of the Learning Journal portal that has been providing various skill development courses, training sessions, and technical articles since 2018.</p>
        <h2>Course Outline</h2><h4>1. Before you Start</h4><p></p><br><h4>2. Introduction to Real-time Streams</h4><p></p><br><h4>3. Enter the world of Apache Kafka</h4><p></p><br><h4>4. Creating Real-time Streams</h4><p></p><br><h4>5. Enter the Stream Processing</h4><p></p><br><h4>6. Foundation for Real Life Implementations</h4><p></p><br><h4>7. States and Stores</h4><p></p><br><h4>8. KTable - An Update Stream</h4><p></p><br><h4>9. Real-time Aggregates</h4><p></p><br><h4>10. Timestamps and Windows</h4><p></p><br><h4>11. Joining Streams and Tables</h4><p></p><br><h4>12. Testing Streams Application</h4><p></p><br><h4>13. Interactive Query and Micro-Service Responses</h4><p></p><br><h4>14. Appendix</h4><p></p>

A beginner's guide to mastering real-time stream processing using Apache Kafka and Kafka Streams API
If you want to understand the concept of stream processing, this course is for you. Using Kafka, the course will help you get to grips with real-time stream processing and enable you to apply that knowledge to learn Kafka programming techniques.
This course uses the Kafka Streams library available in Apache Kafka 2.x. All the source code and examples on Apache Kafka 2.3 open-source distribution have been tested. You'll understand and explore Confluent Platform functionalities such as Schema Registry and Avro Serdes using the Confluent Community Version. This course makes extensive use of IntelliJ IDEA and Apache Maven as the preferred development IDE. You'll leverage Log4J2 and JUnit5 for industry-standard log implementation in your application and implementing unit test cases, respectively. The code bundle is available at https://github.com/PacktPublishing/Apache-Kafka---Real-time-Stream-Processing-Master-Class

What You Will Learn
Learn Apache Kafka fundamentals and Kafka Architecture
Understand how to create streams using the Kafka Producer APIs
Design, develop, and test real-time stream processing applications using the Kafka Streams library
Get well-versed with the Kafka Streams architecture, Streams DSL, and the Processor API
Support microservice architecture and implement interactive queries in Kafka Streams
Explore serializing and deserializing and work on JSON and AVRO messages
Audience
The Kafka Streams - Real-time Stream Processing course is for software engineers looking to develop a stream processing application using the Kafka Streams library. This course will also help Java programmers who want to learn everything necessary to start implementing real-time streaming applications using Apache Kafka. Data architects and data engineers responsible for designing and building the organization's data-centric infrastructure will benefit from this course.
Approach
The course uses a progressive approach to teach a complex subject. It contains some fully example-driven sections to help you grasp key concepts easily.
Key Features
Design, develop, and test real-time stream processing applications using Kafka Streams * Explore the Kafka Streams architecture, Streams DSL, and exactly-once processing in Apache Kafka * Autogenerate Java Objects from JSON and AVRO Schema definition
Github Repo
https://github.com/PacktPublishing/Apache-Kafka---Real-time-Stream-Processing-Master-Class
About the Author
Prashant Kumar Pandey
Prashant Kumar Pandey is passionate about helping people learn and grow in their careers by bridging the gap between their existing and required skills. In his journey to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. He is also the founder, lead author, and chief editor of the Learning Journal portal that has been providing various skill development courses, training sessions, and technical articles since 2018.
Course Outline
1. Before you Start

2. Introduction to Real-time Streams

3. Enter the world of Apache Kafka

4. Creating Real-time Streams

5. Enter the Stream Processing

6. Foundation for Real Life Implementations

7. States and Stores

8. KTable - An Update Stream

9. Real-time Aggregates

10. Timestamps and Windows

11. Joining Streams and Tables

12. Testing Streams Application

13. Interactive Query and Micro-Service Responses

14. Appendix


A beginner's guide to mastering real-time stream processing using Apache Kafka and Kafka Streams API

Apache Kafka - Real-time Stream Processing (Master Class)

Through this course, you will learn how to arrange Kafka a producer and consumer and Kafka Streams and Connectors accurately. You will also gain the skills needed to coordinate Kafka with existing application stages and to pass the Apache Kafka certification exam.
Do you want to learn Apache Kafka to store and process multiple, nonstop streams of information faster? Or do you want to prepare for the Apache Kafka certification exam? This course is an ideal resource to learn Apache Kafka and to reach your goals. The course starts with an introduction to big data and Apache Kafka. Next, you will get into the details of a Kafka producer and consumer and grasp the techniques to tune the performance of Kafka operations. Moving along, you will explore the Kafka architecture and learn how to administer Kafka effectively. Next, you will learn to monitor server, infrastructure, and application and understand how the Schema Registry works in Kafka. Later, you will get to grips with Kafka Streams and Connectors and learn the process of integrating Kafka with Apache Storm, Spark, and Flume. Towards the end, you will learn how to perform various admin tasks using the admin clients and uncover useful tips to secure Kafka. By the end of this course, you will have developed the Kafka skills needed to handle real-time data feed and to pass the Apache Kafka certification exam.
What You Will Learn
Create a Kafka producer and a Kafka consumer
Discover Kafka reliability methods
Become familiar with MirrorMaker deployment and tuning
Build a simple stand-alone application using Kafka and Storm
Create a Flume agent for sending data from Kafka to HDFS
Configure SSL certificate in Kafka
Audience
If you are into analytics or research or are a software developer, test engineer, or a project manager, who is looking to learn Apache Kafka and achieve an Apache Kafka certification, this course is for you. A basic understanding of big data is needed to get started with this course.
Approach
With the help of demos, examples, and activities, this course helps you to get started with Apache Kafka and prepares you to pass the Apache Kafka certification exam.
Key Features
Become familiar with the design of Kafka and its use cases * Design and introduce the Kafka group in single-hub and multi-hub arrangements * Execute a Kafka consumer and Kafka producer using real-time streaming information
About the Author
Learnkart Technology Private Limited 
Learnkart understands the importance of upskilling and its impact on you. They pride themselves in creating specialized e-learning courses that will not only prepare you for certification exams but also help you gain hands-on knowledge for real-world applications. Thousands of students have developed their skills with Learnkart through a variety of courses, such as PgMP, Risk Management Professional (RMP), speech recognition, Python, Amazon Web Services (AWS), and more. Learnkart regularly adds courses to its portfolio to lend more value to your learning experience.
Course Outline
1. Course Introduction

1. Introduction
This video presents an overview of the course.
2. Getting Started with Big data and Apache Kafka

1. Learning Objectives
This video highlights the learning objectives of this section.
2. Data Evolution over Time
This video explains how data has evolved over time.
3. Big Data Overview
This video presents an overview of big data.
4. Big Data Analytics
This video introduces you to big data analytics.
5. Messaging System
This video explains the basic messaging system and how it works. It also highlights the issues and how Kafka resolves those issues.
6. Introduction and Features of Kafka
This video introduces you to Kafka and its features.
7. Kafka Use Cases
This video explains Kafka use cases, such as messaging, metrics and logging, stream processing, and more.
8. Kafka Terminologies
This video introduces you to Kafka terminologies, such as message, batch, and more.
9. Kafka Components
This video explains the different types of Kafka components.
10. Kafka Architecture
This video presents an overview of Kafka's architecture.
11. Kafka Clusters
This video explains the concept of Kafka clusters.
12. Kafka Industry Use Cases
This video highlights the concept of Kafka industry use cases.
13. Demo - Installing Zookeeper
This video demonstrates how to install Apache Zookeeper.
14. Demo - Installing Kafka
This video explains how to install Apache Kafka.
15. Demo - Setting up a Single Node and Single Broker Cluster
This video demonstrates how to set up a single node and a single broker cluster.
16. Demo - Setting up a Multi-node Single Broker Cluster
This video explains how to set up a multi-node single broker cluster.
17. Key Takeaways
This video summarizes what you have learnt in this section.
3. Kafka Producers

1. Learning Objective
This video explains the learning objectives of this section.
2. Overview of a Kafka Producer and Its Architecture
This video presents an overview of a Kafka producer and its architecture.
3. Kafka Producer Configuration
This video explains the Kafka producer configuration.
4. Kafka Producer Optional Configuration
This video explains the optional configuration of a Kafka producer.
5. Kafka Producer Configuration Objects
This video introduces you to Kafka producer configuration objects.
6. Demo - Creating a Kafka Producer
This video demonstrates how to create a Kafka producer.
7. Sending Messages
This video focuses on how a Kafka producer sends messages to broker and explains the different ways of sending messages.
8. Serializers and Custom Serializers
This video introduces you to serializers and custom serializers.
9. Demo - Creating a Custom Serializer
This video demonstrates how to create a custom serializer.
10. Serializers Challenges and Serializing Using Apache Avro
This video focuses on serializers challenges and explains how to serialize using Apache Avro.
11. Demo - Serializing Using Apache Avro
This video demonstrates how to serialize using Apache Avro.
12. Partitions
This video highlights the concept of partitions in Kafka.
13. Demo - Setting up a Custom Partition
This video explains how to set up a custom partition.
14. Key Takeaway
This video highlights what you have learnt from this section.
4. Kafka Consumers

1. Learning Objective
This video explains the learning objective of this section.
2. Overview of Kafka Consumers
This video presents an overview of Kafka consumers.
3. Consumer Groups
This video highlights the concept of consumer groups.
4. Partition Rebalance and Creating a Kafka Consumer
This video introduces you to partition rebalance and explains how to create a consumer.
5. Poll Loop and Its Functioning
This video focuses on the concept of a poll loop and explains its functioning.
6. Configuring a Kafka Consumer - Part 1
This is the first part of the two-part video that explains properties, such as fetch.min.bytes, fetch.max.wait.ms, and so on. It also explains how to configure a Kafka consumer.
7. Configuring a Kafka Consumer - Part 2
This is the second part of the two-part video that explains properties, such as auto.offset.reset, enable.auto.commit, and so on. It also explains how to configure a Kafka consumer.
8. Demo - Creating a Kafka Consumer
This video demonstrates how to create a Kafka consumer.
9. Commit and Offset
This video explains the concepts of commit and offset in Kafka.
10. Ways of Committing Offset - Automatic Offset
This video shows how to commit offset using automatic offset.
11. Ways of Committing Offset - Commit Current Offset
This video demonstrates how to commit offset using commit offset.
12. Ways of Committing Offset - Asynchronous Commit
This video explains how to commit offset using asynchronous offset.
13. Ways of Committing Offset - Combining Synchronous and Asynchronous Commits
This video shows how to commit offset using synchronous and asynchronous commits.
14. Ways of Committing Offset - Commit Specified Offset
This video demonstrates how to commit offset using commit specified offset.
15. Rebalance Listeners
This video explains the concept of rebalance listeners in Apache Kafka.
16. Consuming Records with Specific Offset
This video demonstrates how to consume records with specific offset.
17. Deserializers
This video introduces you to deserializers in Kafka.
18. Demo - Creating and Using Custom Deserializer
This video demonstrates how to create and use custom deserializer.
19. Key Takeaways
This video highlights what you have learnt in this section.
5. Kafka Operations and Performance Tuning

1. Learning Objective
This video explains the learning objectives of this section.
2. Kafka Internals Overview
This video presents an overview of Kafka Internals.
3. Replication and Replica Types
This video focuses on replication and explains the different types of replica.
4. Preferred Ladder, Request, and Request Processing
This video presents an overview of Preferred Ladder, request, and request Processing.
5. Types of Requests
This video presents an overview of the different types of requests.
6. Partition Allocation, File Management, and Segments
This video introduces you to partition allocation, file management, and segments in Apache Kafka.
7. File Format, Index, and Compaction
This video highlights the concepts of file format, index, and compaction.
8. Kafka Reliability and Reliability Methods
This video presents an overview of Kafka reliability and reliability methods.
9. Broker Configuration for Replication
This video explains the broker configuration for replication.
10. Producer in a Reliable System
This video explains the concept of a producer in a reliable system.
11. Consumer in a Reliable System
This video focuses on the concept of a consumer in a reliable system.
12. Key Takeaways
This video highlights what you have learnt in this section.
6. Kafka Cluster Architectures and Administering Kafka

1. Learning Objective
This video explains the learning objectives of this section.
2. Cluster Mirroring
This video presents an overview of cluster mirroring.
3. Multi-cluster Architecture - Hub, Spokes, and Active Architecture - Part 1
This is the first part of the two-part video that focuses on multi-cluster architecture, such as hub, spokes, and active architecture.
4. Multi-cluster Architecture - Active/Stand-by Architecture and Stretch Clusters Part - 2
This is the second part of the two-part video that provides an overview of the multi-cluster architecture, such as active/stand-by architecture and stretch clusters.
5. Configuring MirrorMaker - Part 1
This video is the first part of the two-part video that demonstrates how to configure MirrorMaker in Apache Kafka.
6. Configuring MirrorMaker - Part 2
This video is the second part of the two-part video that explains how to configure MirrorMaker in Apache Kafka.
7. Demo - Setting up MirrorMaker
This video demonstrates how to set up MirrorMaker in Kafka.
8. Administering Kafka - Topic Operations
This video presents Administering Kafka and using different CLI options.
9. Administering Kafka - Consumer Group Operations
This video provides an overview of consumer group operations
10. Dynamic Configuration Changes
This video focuses on dynamic configuration changes.
11. Partition Management
This video provides an overview of partition management.
12. Console Producer Tool
This video introduces you to the console producer tool.
13. Console Consumer Tool
This video provides an introduction to the console consumer tool.
14. Key Takeaways
This video focuses on what you have learnt in the section.
7. Monitoring Kafka and Schema Registry

1. Learning Objective
This video presents the learning objectives of this section.
2. Monitoring and Its Importance
This video presents an overview of monitoring and its importance.
3. Server or Infrastructure Monitoring and Application Monitoring
This video provides details on server or infrastructure monitoring and application monitoring.
4. Monitoring Kafka
This video demonstrates how to monitor Kafka.
5. Kafka Broker Metrics - Under-replicated Partitions - Part 1
This is the first part of the two-part video that focuses on Kafka broker metrics, such as under-replicated partitions.
6. Kafka Broker Metrics - Others- Part 2
This video provides an overview of Kafka broker metrics - Active Controller Count.
7. Topic and Partition Specific Metrics
This video focuses on topic and partition specific metrics.
8. Logging and Client Monitoring
This video focuses on logging and client monitoring.
9. Producer and Consumer Metrics
This video focuses on producer and consumer metrics.
10. Quotas and Lag Monitoring
This video focuses on quotas and lag monitoring.
11. Monitoring Dashboard
This video presents an overview of the monitoring dashboard.
12. Demo - Setting up an Open Source Health Monitor
This video demonstrates how to set up an open-source health monitor.
13. Kafka Schema Registry
This video introduces you to the Kafka Schema Registry.
14. Kafka Components and Its Architecture
This video presents an overview of Kafka components and its architecture.
15. Kafka Schema Registry - Internal working and Use cases
This video provides an overview of the Kafka Schema Registry, focusing on internal working and use cases.
16. Working of the Kafka Schema Registry
This video explains the working of the Kafka Schema Registry.
17. Demo - Using the Kafka Schema Registry with Kafka
This video demonstrates how to use the Kafka Schema Registry with Kafka.
18. Key Takeaway
This video highlights what you have learnt from this section.
8. Kafka Streams and Kafka Connectors

1. Learning Objective
This video presents the learning objectives of this section.
2. Kafka Streams Overview
This video presents an overview of Kafka Streams.
3. Kafka Streams
This video explains more about Kafka Streams.
4. Kafka Streams Architecture and Working
This video explains the Kafka Streams architecture and its working.
5. Kafka Streams Components
This video focuses on the Kafka Streams components.
6. Kafka Streams Architecture Tasks, Threading Model and Local State Store
This video focusses on Kafka Streams architecture tasks, the threading model, and the local state store.
7. Kafka Streams Architecture - Record Buffer
This video provides an overview of Kafka Streams architecture, focuses on record buffer.
8. Memory Management and Streaming Data Pipeline
This video provides an overview of memory management and streaming data pipeline.
9. Kafka Streams Domain Specific Language (DSL)
This video introduces you to Kafka Streams DSL.
10. KStream Operations
This video presents an overview of KStream operations.
11. KTable Operations
This video focuses on KTable operations.
12. KTable Operations
This video explains more about KTable operations.
13. Aggregation and Windowing
This video explains the concept of aggregation and windowing.
14. Processor Topology and Stream Processor
This video introduces you to processor topology and stream processor.
15. Stream and Processor Application Programming Interfaces (APIs)
This video focuses on Stream and Processor APIs.
16. Processor Application Programming Interfaces (APIs) and Creating Topology
This video provides an overview of Processor APIs and explains how to create topology.
17. Kafka Connectors
This video explains the concept of Kafka Connectors.
18. Stand-alone and Sink Connector Configuration
This video explains how to configure stand-alone and sink connectors.
19. Running Kafka Connect
This video demonstrates how to run Kafka Connect.
20. Kafka Connector Distributed Mode
This video focuses on the Kafka Connector distributed mode.
21. Hypertext Transfer Protocol (HTTP) Representational State Transfer (REST) Interface
This video explains the HTTP REST interface.
22. Demo - Kafka Connector
This video demonstrates the working of a Kafka Connector.
23. Demo - Creating an Application Using Kafka Streams
This video demonstrates how to create an application using Kafka Streams.
24. Key Takeaway
This video highlights what you have learnt from this section.
9. Integration of Kafka with Storm

1. Learning Objective
This video explains the learning objectives of this section.
2. Apache Storm
This video provides an overview of Apache Storm.
3. Real-time Analytics
This video provides an overview of real-time analytics.
4. Apache Storm Architecture
This video explains the architecture of Apache Storm.
5. Apache Storm Components
This video introduces you to the components of Apache Storm.
6. Apache Storm Topology Part - 1
This is the first part of the two-part video that presents an overview of the topology of Apache Storm.
7. Apache Storm Topology Part - 2
This is the second part of the two-part video that explains more about the topology of Apache Storm.
8. Kafka Spout
This video presents an overview of Kafka Spout.
9. Integration of Apache Storm and Kafka
This video explains the integration of Apache Storm and Kafka.
10. Demo - Simple Stand-alone Application Using Kafka and Storm
This video provides a demonstration of a simple stand-alone application using Kafka and Storm.
11. Key Takeaway
This video highlights what you have learnt from this section.
10. Kafka Integration with Spark and Flume

1. Learning Objective
This video presents the learning objectives of this section.
2. Introduction to Spark
This video provides an introduction to Spark.
3. Spark Components
This video explains the components of Spark.
4. Resilient Distributed Dataset
This video focuses on a resilient distributed dataset.
5. Resilient Distributed Dataset (RDD) Operations - Transformation - Map, FlatMap, and Filter
This video provides an overview of RDD operations, focusing on transformations such as map, FlatMap, and filter.
6. RDD Operations - Transformation - Join, Distinct, First, and Take
This video explains RDD operations, focusing on transformations such as join, distinct, first, and take.
7. Resilient Distributed Dataset (RDD) Operations - Actions
This video explains RDD operations, focusing on actions.
8. Data Sets and SparkSession
This video provides an introduction to data sets and SparkSession.
9. Data Sets and Spark Session Operations
This video explains more about data sets and SparkSession operations.
10. Spark Stream
This video provides an overview of Spark Stream.
11. Integrating Spark with Kafka
This video demonstrates how to integrate Spark with Kafka.
12. Demo - Running a Small Stand-alone Application in Spark with Kafka
This video explains how to run a small stand-alone application in Spark with Kafka.
13. Flume
This video introduces you to the stream capturing tool called Flume.
14. Flume Connectors
This video provides an overview of Flume Connectors.
15. Configuring Flume Kafka to Hadoop Distributed File System (HDFS)
This video explains how to configure Flume Kafka to HDFS.
16. Demo - Creating a Flume Agent and Sending Data from Kafka to Hadoop Distributed File System (HDFS)
This video demonstrates how to create a flume agent and send data from Kafka to HDFS.
17. Key Takeaways
This video highlights what you have learnt in this section.
11. Admin Client and Securing Kafka

1. Learning Objective
This video explains the learning objective of this course.
2. AdminClient
This video introduces you to AdminClient.
3. Demo - Performing Various Admin Tasks Using AdminClient
This video demonstrates how to perform various admin tasks using AdminClient.
4. Kafka Security
This video presents an overview of Kafka Security.
5. Kafka Security Components
This video introduces you to the components of Kafka Security.
6. Simple Authentication and Security Layer (SASL)
This video highlights the concept of SASL in Kafka.
7. Configuring Simple Authentication and Security Layer (SASL) in Kafka
This video explains how to configure SASL in Kafka.
8. Securing Using Access Control Lists (ACLs)
This video demonstrates how to secure Kafka using ACLs.
9. Key Takeaway
This video highlights what you have learnt from this course.

Through this course, you will learn how to arrange Kafka a producer and consumer and Kafka Streams and Connectors accurately. You will also gain the skills needed to coordinate Kafka with existing application stages and to pass the Apache Kafka certification exam.

Apache Kafka A-Z with Hands-On Learning

Redhill

<h3><b>Apache training course description</b></h3>
      <p> A hands on training course covering installation, configuration and management of the Apache web server. </p>
        
				<h5><b>What will you learn</b></h5>
				<ul>
				 
      <li>Install Apache.</li>
					<li>Configure Apache.</li>
					<li>Manage Apache.</li>
					<li>Build static and dynamic web sites with Apache.</li>
					<li>Secure Apache.</li>
                  </ul>
                  
       
        <h5>
        <b>Apache training course details</b> </h5>
				<div>
					<div>
						<ul>
							<li><b> Who will benefit:</b> </li>
							
              <p>Technical staff working with Apache.</p>
              
							<li><b>Prerequisites:</b></li>
							
              <p> TCP/IP foundation for engineers.<br />
							UNIX fundamentals</p>
              
							<li><b>Duration</b></li>
							
              <p>3 days
						</p>
              
							
              
              
						</ul>
					</div>
				</div>
      <h3><b>Apache training course contents</b></h3>
      <ul>
				<li>Installing Apache</li><p>
What is Apache? Apache versions, history, downloading Apache, source distribution, compilation, binary distribution, installation, platform considerations.
<br /><span>Hands on</span> Downloading and installing Apache.

</p><li>Controlling the Apache server</li><p>
Running Apache, automatic Apache start, starting, stopping, restarting Apache. Checking Apache status.
<br /><span>Hands on</span> Server control.

</p><li>Configuration</li><p>
Serving webpages, setting the document root, applying configuration changes, Configuration files, httpd.conf, syntax, directives, modules, utilities, turning features on/off.
<br /><span>Hands on</span> basic Apache configuration.

</p><li>More configuration</li><p>
MIME, URL mapping, content negotiation, indexing, performance tuning.

</p><li>Logging</li><p>
log file content, configuration, log file locations, error logging, browser errors, error page configuration, forbidden index pages.
<br /><span>Hands on</span> Log files.

</p><li>Security</li><p>
File permissions, .htaccess, protecting files with passwords, password files, authentication, restricting access by IP address.

</p><li>Secure HTTP</li><p>
HTTPS, installing mod_ssl, certificates, configuring mod_ssl, http and https coexistence

</p><li>Virtual hosts</li><p>
Multiple sites on one server, separate configuration files, IP based, name based, port based, virtual host names, enabling, defining, configuring, aliases, testing, https virtual hosts.
<br /><span>Hands on</span> Virtual hosts.

</p><li>Dynamic sites</li><p>
Dynamic sites, CGI, PHP, PERL, CGI programs, example CGI scripts, Apache and CGI, CGI parameters, CGI issues, PHP, mod_php, Perl and Apache, mod_perl, installing mod_perl.
<br /><span>Hands on</span> CGI, PHP and Perl with Apache.

</p><li>Modules</li><p>
What are modules, standard modules, loading modules, mod_speling, mod_rewrite, other special purpose modules, URL rewriting, redirection, URL transformation, browser dependent pages.
<br /><span>Hands on</span> Working with modules.
</p>
      </ul>

Apache training course description
 A hands on training course covering installation, configuration and management of the Apache web server. 
What will you learn
Install Apache.
Configure Apache.
Manage Apache.
Build static and dynamic web sites with Apache.
Secure Apache.
Apache training course details
Who will benefit:
Technical staff working with Apache.
Prerequisites:
 TCP/IP foundation for engineers.
 UNIX fundamentals
Duration
3 days 
Apache training course contents
Installing Apache

What is Apache? Apache versions, history, downloading Apache, source distribution, compilation, binary distribution, installation, platform considerations.

Hands on Downloading and installing Apache. 
Controlling the Apache server

Running Apache, automatic Apache start, starting, stopping, restarting Apache. Checking Apache status.

Hands on Server control. 
Configuration

Serving webpages, setting the document root, applying configuration changes, Configuration files, httpd.conf, syntax, directives, modules, utilities, turning features on/off.

Hands on basic Apache configuration. 
More configuration

MIME, URL mapping, content negotiation, indexing, performance tuning. 
Logging

log file content, configuration, log file locations, error logging, browser errors, error page configuration, forbidden index pages.

Hands on Log files. 
Security

File permissions, .htaccess, protecting files with passwords, password files, authentication, restricting access by IP address. 
Secure HTTP

HTTPS, installing mod_ssl, certificates, configuring mod_ssl, http and https coexistence 
Virtual hosts

Multiple sites on one server, separate configuration files, IP based, name based, port based, virtual host names, enabling, defining, configuring, aliases, testing, https virtual hosts.

Hands on Virtual hosts. 
Dynamic sites

Dynamic sites, CGI, PHP, PERL, CGI programs, example CGI scripts, Apache and CGI, CGI parameters, CGI issues, PHP, mod_php, Perl and Apache, mod_perl, installing mod_perl.

Hands on CGI, PHP and Perl with Apache. 
Modules

What are modules, standard modules, loading modules, mod_speling, mod_rewrite, other special purpose modules, URL rewriting, redirection, URL transformation, browser dependent pages.

Hands on Working with modules.


Systems & Network Training

Network & Security

Network Engineer

Definitive Apache for Engineers

A beginner-level course that follows a step-by-step approach to learning the fundamentals and core concepts of Apache Kafka 3.0. You will work through interesting activities such as programming a Twitter producer and Elasticsearch consumer to understand the various concepts.
The high throughput and low latency of Apache Kafka have made it one of the leading distributed data-streaming enterprise technologies. It is used by many Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart, and LinkedIn. If you want to develop Apache Kafka skills to stream data easily, then this course is for you. The course starts by explaining the architecture of the Apache Kafka ecosystem, before going on to cover the core concepts of Kafka such as topics, partitions, brokers, replicas, producers, consumers, and more. Next, you will use native Kafka binaries to launch your own Kafka cluster on Windows, Mac OS X, and Linux. As you advance, you will get hands-on experience with the Kafka command-line interface (CLI), along with an understanding of how to create producers and consumers in Java to interact with Kafka. Next, a real-world project using Wikimedia as a source of data for a producer and OpenSearch as a sink for our consumers. Moving on, you will get to grips with advanced APIs such as Kafka Connect and Kafka Streams and perform a case study on the real-world applications of Kafka. Finally, you will get an overview of advanced Kafka for administrators and understand advanced topic configurations. By the end of this course, you will be well-versed with how Apache Kafka 3.0 plays an important role in data-streaming applications.
What You Will Learn
Understand the Apache Kafka ecosystem, architecture, core concepts, and operations
Start a personal Kafka development environment
Create producers and consumers in Java to interact with Kafka
Master concepts such as topics, partitions, brokers, producers, consumers
Discover how to change a topic configuration
Practice and understand log compaction
Audience
This course is for developers who want to learn the fundamentals of Apache Kafka, software architects who want to understand how Kafka fits into their solution architecture or anyone who is looking to understand how Apache Kafka works as a distributed system. Basic knowledge of Java and the Linux command line will be beneficial to effectively understand the concepts covered in the course.
Approach
With this course, you will get hands-on experience with Apache Kafka. Through interesting examples and activities, you will learn how to use Apache Kafka to build robust distributed data-streaming applications.
Key Features
Find out why Apache Kafka is an excellent choice of distributed data-streaming platform * Learn major CLIs: kafka-topics, kafka-console-producer, kafka-console-consumer, kafka-consumer-groups, kafka-configs * Get a solid grip on Apache Kafka by understanding various real-world examples
Github Repo
https://github.com/PacktPublishing/Apache-Kafka-Series---Learn-Apache-Kafka-for-Beginners-v3
About the Author
Stéphane Maarek
Stéphane Maarek is a solutions architect, consultant, and software developer who has a particular interest in all things related to big data and analytics. He is also a bestseller instructor on Udemy for his courses on Apache Kafka, Apache NiFi, and AWS Lambda. He loves Apache Kafka and regularly contributes to the Apache Kafka project. Stéphane has also written a guest blog post that was featured on the Confluent website, the company behind Apache Kafka. He is also an AWS Certified Solutions Architect and has many years of experience with technologies such as Apache Kafka, Apache NiFi, Apache Spark, Hadoop, PostgreSQL, Tableau, Spotfire, Docker, Ansible, and more.
Course Outline
1. Kafka Introduction

1. Course Introduction
In this video, the author introduces you to the course and his experience in this field.
2. Apache Kafka in Five Minutes
This video introduces Apache Kafka.
3. Course Objectives
In this section, you will learn about topics and partitions, how to replicate topics, and much more.
2. Kafka Theory

1. Topics, Partitions, and Offsets
This video explains topics, partitions, and offsets.
2. Producers and Message Keys
This video explains the concept of producers and message keys.
3. Consumers and Deserialization
This video explains the concept of consumers and deserialization.
4. Consumer Groups and Consumer Offsets
This video explains the consumer groups and consumer offsets.
5. Brokers and Topics
This video explains the concept of Kafka brokers and topics.
6. Topic Replication
This video explains the topic replication process in Kafka.
7. Producer Acknowledgments and Topic Durability
This video explains producer acknowledgments and topic durability.
8. Zookeeper
This video explains the concept of a Zookeeper in Kafka.
9. Kafka KRaft - Removing Zookeeper
This video explains Kafka KRaft. Kafka Raft (KRaft) is the consensus protocol that was introduced to remove Apache Kafka's dependency on Zookeeper for metadata management.
10. Theory Roundup
This video provides a roundup of the theory.
3. Starting Kafka

1. Important: Starting Kafka
This video demonstrates how to start Kafka in various operating systems and how we can overcome the challenge of starting Kafka.
2. Starting Kafka with Conduktor - Multi-Platform
This video demonstrates how to start Kafka with Conduktor.
3. Mac OS X - Download Kafka and Set Up PATH
This video demonstrates how to download Kafka and set up PATH on Mac OS X.
4. Mac OS X - Start Zookeeper and Kafka
This video demonstrates how to start a Zookeeper server and install Kafka on Mac OS X.
5. Mac OS X - Using Brew
This video demonstrates how to install Kafka on Mac OS X using Brew.
6. Linux - Download and Set Up Kafka in PATH
This video demonstrates how to download Kafka and set up PATH on a Linux system.
7. Linux - Start Zookeeper and Kafka
This video demonstrates how to start a Zookeeper server and install Kafka on a Linux server.
8. Windows WSL2 - Download Kafka and PATH Setup
This video demonstrates how to download Kafka and set up PATH on Windows.
9. Windows WSL2 - Start Zookeeper and Kafka
This video demonstrates how to start a Zookeeper server and install Kafka on Windows.
10. Windows WSL2 - How to Fix Problems
This video demonstrates the bug on WSL2 and how to fix the bug.
11. Windows Non-WSL2 - Start Zookeeper and Kafka
This video demonstrates how to start a Zookeeper server and install Kafka non-WSL2.
4. Starting Kafka without Zookeeper

1. Mac OS X - Start Kafka in KRaft Mode
This video explains how to start one Kafka Broker in KRaft mode without a Zookeeper on Mac OS.
2. Linux - Start Kafka in KRaft Mode
This video explains how to start one Kafka Broker using KRaft mode on Linux without a Zookeeper in this instance.
3. Windows WSL2 - Start Kafka in KRaft Mode
This video explains how to start one Kafka Broker with Zookeeper in KRaft Mode
5. Command Line Interface (CLI) 101

1. Command Line Interface (CLI) Introduction
This video introduces you to Kafka command line interface and how it comes bundled with Kafka binaries.
2. Kafka Topics Command Line Interface (CLI)
This video explains the CLI commands related to Kafka topics.
3. Kafka Console Producer Command Line Interface (CLI)
This video explains the kafka-console-producer CLI command to start producing data into our Kafka topics.
4. Kafka Console Consumer Command Line Interface (CLI)
This video explains the Kafka-console-consumer.sh CLI.
5. Kafka Consumers in a Group
This video explains the concept of Kafka consumers in a group and the -group parameter.
6. Kafka Consumer Groups Command Line Interface (CLI)
This video explains the consumer groups CLI, in which we can reset, delete, and manage the consumer groups.
7. Resetting Offsets
This video focuses on resetting offsets using the Kafka Offsets groups command.
6. Kafka UI

1. Conduktor - Demo
This video demonstrates Kafka UI Conduktor to get the full power of the UI.
7. Kafka Java Programming 101

1. Kafka SDK List
This video explains Kafka SDK list.
2. Creating Kafka Project
This video demonstrates how to create a Kafka project.
3. Java Producer
This video explains how to write a basic producer to send data to Kafka.
4. Java Producer Callback
This video explains how to create Kafka producer in Java using the callback function.
5. Java Producer with Keys
This video explains how to send messages in Kafka using keys.
6. Java Consumer
This video explains how to write a basic consumer to receive data from Kafka.
7. Java Consumer - Graceful Shutdown
This video explains a code to respond to termination signals and how to improve our Java code.
8. Java Consumer Inside a Consumer Group
This video explains the behavior of our consumers as part of our consumer groups.
9. Java Consumer Incremental Cooperative Rebalance and Static Group Membership
This video explains Java consumer incremental cooperative rebalance and static group membership.
10. Java Consumer Incremental Cooperative Rebalance - Practice
This video explains Java consumer incremental cooperative rebalance-practice.
11. Java Consumer Auto Offset Commit Behavior
This video explains the concept of Java consumer auto offset commit behavior.
12. Programming - Advanced Tutorials
This video explains programming-advanced tutorials.
8. Kafka Real-World Project

1. Real-World Project Overview
This video provides an overview of the real-world project.
9. Kafka Wikimedia Producer and Advanced Configurations

1. IMPORTANT: Start Local Kafka with Conduktor Using Docker
This video provides an overview of how to start Kafka with Conduktor using Docker Compose.
2. Wikimedia Producer Project Setup
This video provides an overview of the Wikimedia producer project setup.
3. Wikimedia Producer Implementation
This video explains Wikimedia producer implementation.
4. Wikimedia Producer Run
This video explains Wikimedia producer run using a demo.
5. Wikimedia Producer - Producer Config Intros
This video explains Wikimedia producer-producer Config Intros.
6. Producer Acknowledgements Deep Dive
This video explains Kafka producer acknowledgements.
7. Producer Retries
This video explains producer retries.
8. Idempotent Producer
This video explains idempotent producer.
9. Safe Kafka Producer Settings
This video explains the safe Kafka producer settings.
10. Wikimedia Producer Safe Producer Implementation
This video explains Wikimedia producer safe producer implementation.
11. Kafka Message Compression
This video explains Kafka message compression.
12. linger.ms and batch.size Producer Settings
This video explains how to implement a high throughput producer in Kafka.
13. Wikimedia Producer High Throughput Implementation
This video explains Wikimedia producer high throughput implementation.
14. Producer Default Partitioner and Sticky Partitioner
This video explains producer default partitioner and sticky partitioner.
15. [Advanced] max.block.ms and buffer.memory
This video explains max.block.ms and buffer.memory.
10. OpenSearch Consumer and Advanced Consumer Configurations

1. OpenSearch Consumer - Project Overview
This video provides an OpenSearch consumer-project overview.
2. OpenSearch Consumer - Project Setup
This video explains OpenSearch consumer-project setup.
3. Setting Up OpenSearch on Docker
This video focuses on setting up OpenSearch on Docker.
4. Setting Up OpenSearch on the Cloud
This video explains setting up OpenSearch on the Cloud.
5. OpenSearch 101
This video explains OpenSearch 101.
6. OpenSearch Consumer Implementation - Part 1
This video explains OpenSearch consumer implementation - part 1.
7. OpenSearch Consumer Implementation - Part 2
This video explains the setup of Kafka consumer and improving our code.
8. Consumer Delivery Semantics
This video explains consumer delivery semantics.
9. OpenSearch Consumer Implementation Part 3 - Idempotence
This video explains how to make our consumer idempotence.
10. Consumer Offsets Commit Strategies
This video explains consumer offsets commit strategies.
11. OpenSearch Consumer Implementation Part 4 - Delivery Semantics
This video explains OpenSearch consumer implementation delivery semantics.
12. OpenSearch Consumer Implementation Part 5 - Batching Data
This video explains OpenSearch consumer implementation part 5-batching data.
13. Consumer Offset Reset Behavior
This video explains consumer offset reset behavior.
14. OpenSearch Consumer Implementation Part 6 - Replaying Data
This video explains how the Kafka consumer is used to replay data.
15. Consumer Internal Threads
This video explains consumer internal threads.
16. Consumer Replica Fetching - Rack Awareness
This video explains consumer replica fetching-rack awareness.
11. Kafka Extended APIs for Developers

1. Kafka Extended APIs - Overview
This video explains the real-world applications in Kafka.
2. Kafka Connect Introduction
This video introduces you to Kafka Connect.
3. Kafka Connect Wikimedia and ElasticSearch Hands-On
This video explains Kafka Connect Wikimedia and ElasticSearch.
4. Kafka Streams Introduction
This video introduces you to Kafka Streams.
5. Kafka Streams Hands-On
This video demonstrates running a Kafka streams application.
6. Kafka Schema Registry Introduction
This video is an introduction to Kafka Schema Registry.
7. Kafka Schema Registry Hands-On
This video explains how to start Kafka with Schema Registry and Conduktor using Docker.
8. Which Kafka API Should I Use?
This video explains which Kafka API to use.
12. Real-World Insights and Case Studies (Big Data / Fast Data)

1. Choosing Partition Count and Replication Factor
This video explains how to determine the partition count and replication factor.
2. Kafka Topics Naming Convention
This video explains Kafka Topics naming convention.
3. Case Study - MovieFlix
This video explains the MovieFlix case study in Kafka.
4. Case Study - GetTaxi
This video explains the GetTaxi case study in Kafka.
5. Case Study - MySocialMedia
This video explains the MySocialMedia case study in Kafka.
6. Case Study - MyBank
This video explains the MyBank case study in Kafka.
7. Case Study - Big Data Ingestion
This video explains the big data ingestion case study in Kafka.
8. Case Study - Logging and Metrics Aggregation
This video explains the logging and metrics aggregation case study in Kafka.
13. Kafka in the Enterprise for Admins

1. Kafka Cluster Setup High-Level Architecture Overview
This video provides an overview of the Kafka cluster setup and high-level architecture.
2. Kafka Monitoring and Operations
This video explains the Kafka monitoring setup and operations.
3. Kafka Security
This video explains the concept of Kafka security.
4. Kafka Multi-Cluster and MirrorMaker
This video explains the concept of Kafka multi-cluster and MirrorMaker tool.
5. Advertised Listeners: Kafka Client and Server Communication Protocol
This video explains advertised listeners: Kafka client and server communication protocol.
14. Advanced Topics Configurations

1. Changing a Topic Configuration
This video explains how to change the topic configuration settings.
2. Segment and Indexes
This video explains the concept of segment and indexes files in Kafka.
3. Log Cleanup Policies
This video explains the Kafka log cleanup policies.
4. Log Cleanup Delete
This video explains how to delete the log cleanup policy.
5. Log Compaction Theory
This video explains the Kafka log compaction theory.
6. Log Compaction Practice
This video explains how to allow log compaction into practice.
7. Unclean Leader Election
This video explains how to enable an unclean leader election in Kafka.
8. Large Messages in Kafka
This video explains large messages in Kafka.
15. Next Steps

1. What's Next?
This video highlights the next concepts you can learn in the world of Apache Kafka.

A beginner-level course that follows a step-by-step approach to learning the fundamentals and core concepts of Apache Kafka 3.0. You will work through interesting activities such as programming a Twitter producer and Elasticsearch consumer to understand the various concepts.

Apache Kafka Series - Learn Apache Kafka for Beginners v3

<strong>Apache NiFi, a robust, open-source data ingestion/distribution framework, is the core of Hortonworks DataFlow (HDF)</strong>
        <p>Apache NiFi is a robust, open-source data ingestion and distribution framework-and more. It can propagate any data content from any source to any destination. NiFi is based on a different programming paradigm called Flow-Based Programming (FBP). We define Flow-Based Programming, but don't explain it. Instead, we tell you how NiFi works, and then you can connect it with our definition of Flow-Based Programming.

The field of big data is quite vast, and anyone who starts learning big data and its related technologies may find the experience very daunting. There are numerous big data technologies, and deciding where to begin can be overwhelming. However, there is a lot of scope for big data and its related technologies such as Apache NiFi, Spark, Flink, and more. So, start your big data journey today by learning about Apache NiFi.

All the necessary lab files are at
https://github.com/learnwithmanoj/apache-nifi-templates</p>
        <h2>What You Will Learn</h2><p>The core concepts of Apache NiFi<br>Key NiFi features such as backpressure, prioritization, expiration, monitoring, reporting tasks, data provenance, and more.<br>NiFi registry for version control<br>Install Apache NiFi in standalone and cluster modes</p>
        <h2>Audience</h2><p>This course targets all skill levels. People who entirely new to the data flow or data pipeline world; those already using another commercial tool and keen to move to an open-source solution; software Engineers; data Engineers; software architects; and data scientists will all benefit from this course.</p>
        <h2>Approach</h2><p>This course shows you how to Install Apache NiFi in its standalone and cluster modes. The author teaches complex concepts in a simple way; thus, even if you have no previous coding experience, you'll learn everything step-by-step in the perfect order.</p>
        <h2>Key Features</h2><p>Get hands-on with Apache NiFi * Implement custom processors in NiFi * Monitor NiFi * Master Hortonworks DataFlow (HDF)</p>
        <h2>Github Repo</h2><p>https://github.com/packtpublishing/apache-nifi---a-complete-guide-hortonworks-dataflow---hdf</p>
        <h2>About the Author</h2><h3>Manoj Ganesan Thangaraj</h3><p>Manoj G T:
Manoj is a Big Data Evangelist, a JavaScript Lover, a Web Developer and a Performance Freak. He is passionate about programming since he was 14 and he is always willing to learn new things and continue to seek to improve himself. His main objective is to provide high-quality course content to all his students in an easy and understanding way.</p>
        <h2>Course Outline</h2><h4>1. Apache NiFi Introduction</h4><p></p><br><h4>2. Apache NiFi Installation</h4><p></p><br><h4>3. Apache NiFi Basics</h4><p></p><br><h4>4. Hands-on with Apache NiFi</h4><p></p><br><h4>5. Apache NiFi Advanced Concepts</h4><p></p><br><h4>6. Monitoring in NiFi</h4><p></p><br><h4>7. NiFi Registry for Version Control</h4><p></p><br><h4>8. NiFi Cluster for Heavy Lifting</h4><p></p><br><h4>9. NiFi Custom Processor - Do the Impossible</h4><p></p>

Apache NiFi, a robust, open-source data ingestion/distribution framework, is the core of Hortonworks DataFlow (HDF)
Apache NiFi is a robust, open-source data ingestion and distribution framework-and more. It can propagate any data content from any source to any destination. NiFi is based on a different programming paradigm called Flow-Based Programming (FBP). We define Flow-Based Programming, but don't explain it. Instead, we tell you how NiFi works, and then you can connect it with our definition of Flow-Based Programming. The field of big data is quite vast, and anyone who starts learning big data and its related technologies may find the experience very daunting. There are numerous big data technologies, and deciding where to begin can be overwhelming. However, there is a lot of scope for big data and its related technologies such as Apache NiFi, Spark, Flink, and more. So, start your big data journey today by learning about Apache NiFi. All the necessary lab files are at
https://github.com/learnwithmanoj/apache-nifi-templates
What You Will Learn
The core concepts of Apache NiFi
Key NiFi features such as backpressure, prioritization, expiration, monitoring, reporting tasks, data provenance, and more.
NiFi registry for version control
Install Apache NiFi in standalone and cluster modes
Audience
This course targets all skill levels. People who entirely new to the data flow or data pipeline world; those already using another commercial tool and keen to move to an open-source solution; software Engineers; data Engineers; software architects; and data scientists will all benefit from this course.
Approach
This course shows you how to Install Apache NiFi in its standalone and cluster modes. The author teaches complex concepts in a simple way; thus, even if you have no previous coding experience, you'll learn everything step-by-step in the perfect order.
Key Features
Get hands-on with Apache NiFi * Implement custom processors in NiFi * Monitor NiFi * Master Hortonworks DataFlow (HDF)
Github Repo
https://github.com/packtpublishing/apache-nifi---a-complete-guide-hortonworks-dataflow---hdf
About the Author
Manoj Ganesan Thangaraj
Manoj G T:
Manoj is a Big Data Evangelist, a JavaScript Lover, a Web Developer and a Performance Freak. He is passionate about programming since he was 14 and he is always willing to learn new things and continue to seek to improve himself. His main objective is to provide high-quality course content to all his students in an easy and understanding way.
Course Outline
1. Apache NiFi Introduction

2. Apache NiFi Installation

3. Apache NiFi Basics

4. Hands-on with Apache NiFi

5. Apache NiFi Advanced Concepts

6. Monitoring in NiFi

7. NiFi Registry for Version Control

8. NiFi Cluster for Heavy Lifting

9. NiFi Custom Processor - Do the Impossible


Computer Programming

Apache NiFi, a robust, open-source data ingestion/distribution framework, is the core of Hortonworks DataFlow (HDF)

Apache NiFi - A Complete Guide (Hortonworks DataFlow - HDF)

<strong>This course does not require any prior knowledge of Apache Spark or Hadoop. The author explains Spark architecture and fundamental concepts to help you come up to speed and grasp the content of this course.

The course will help you understand Spark programming and apply that knowledge to build data engineering solutions.</strong>
        <p>Apache Spark is a lightning-fast unified analytics engine for big data and machine learning. Since its release, Apache Spark has seen rapid adoption by enterprises across a wide range of industries. Internet powerhouses such as Netflix, Yahoo, and eBay have deployed Spark at a massive scale. It has quickly become the largest open-source community in big data. So, mastering Apache Spark opens a wide range of professional opportunities.

This course starts with an introduction to Apache Spark where you see what Apache Spark is in brief. Then, you will be installing and using Apache Spark. After that, you will look at the Spark execution model and architecture in detail. Next, you will learn the Spark programming model and developer experience. Following that, you will look at the Spark Structured API foundation, and Spark data sources and sinks. Then, you will explore Spark Data frame and dataset transformations along with aggregations in Apache Spark. Finally, you will look at the Spark Data frame joins in detail.

By the end of this course, you will understand Spark programming and apply that knowledge to build data engineering solutions.

All the resource files are uploaded on the GitHub repository at https://github.com/PacktPublishing/Spark-Programming-in-Scala-for-Beginners-with-Apache-Spark-3</p>
        <h2>What You Will Learn</h2><p>Learn Apache Spark and Spark architecture<br>Look at data engineering and data processing in Spark<br>Work with data sources and sinks<br>Work with data frames, datasets, and Spark SQL<br>Use IntelliJ IDEA for Spark development and debugging<br>Understand unit testing, managing application logs, and cluster deployment</p>
        <h2>Audience</h2><p>This course is designed for software engineers willing to develop a data engineering pipeline and application using Apache Spark. It is also for data architects and data engineers who are responsible for designing and building the organization's data-centric infrastructure. It will also be beneficial for the managers and architects who do not directly work with Spark implementation, and still, they work with the people who implement Apache Spark at the ground level.<br><br>Before proceeding with the course, you will need basic knowledge of the Scala programming language.</p>
        <h2>Approach</h2><p>This is a hands-on and well-balanced course with theoretical and practical content. The course is example-driven and follows a working session-like approach. The author will be taking a live coding approach and explaining all the concepts needed along the way. The section will have a summary video at the end to revise the learned concepts.</p>
        <h2>Key Features</h2><p>A comprehensive course designed for the beginner-level for Spark programming in Scala * Deep dive into Spark 3 architecture and data engineering * Complete tested source code and examples used on Apache Spark 3.0.0 open-source distribution from the author's end</p>
        <h2>Github Repo</h2><p>https://github.com/PacktPublishing/Spark-Programming-in-Scala-for-Beginners-with-Apache-Spark-3</p>
        <h2>About the Author</h2><h3>Scholar Nest</h3><p>ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills.

Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects.

It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.</p>
        <h2>Course Outline</h2><h4>1. Apache Spark Introduction</h4><p></p><br><h4>2. Installing and Using Apache Spark</h4><p></p><br><h4>3. Spark Execution Model and Architecture</h4><p></p><br><h4>4. Spark Programming Model and Developer Experience</h4><p></p><br><h4>5. Spark Structured API Foundation</h4><p></p><br><h4>6. Spark Data Sources and Sinks</h4><p></p><br><h4>7. Spark Dataframe and Dataset Transformations</h4><p></p><br><h4>8. Aggregations in Apache Spark</h4><p></p><br><h4>9. Spark Dataframe Joins</h4><p></p><br><h4>10. Keep Learning</h4><p></p>

This course does not require any prior knowledge of Apache Spark or Hadoop. The author explains Spark architecture and fundamental concepts to help you come up to speed and grasp the content of this course. The course will help you understand Spark programming and apply that knowledge to build data engineering solutions.
Apache Spark is a lightning-fast unified analytics engine for big data and machine learning. Since its release, Apache Spark has seen rapid adoption by enterprises across a wide range of industries. Internet powerhouses such as Netflix, Yahoo, and eBay have deployed Spark at a massive scale. It has quickly become the largest open-source community in big data. So, mastering Apache Spark opens a wide range of professional opportunities. This course starts with an introduction to Apache Spark where you see what Apache Spark is in brief. Then, you will be installing and using Apache Spark. After that, you will look at the Spark execution model and architecture in detail. Next, you will learn the Spark programming model and developer experience. Following that, you will look at the Spark Structured API foundation, and Spark data sources and sinks. Then, you will explore Spark Data frame and dataset transformations along with aggregations in Apache Spark. Finally, you will look at the Spark Data frame joins in detail. By the end of this course, you will understand Spark programming and apply that knowledge to build data engineering solutions. All the resource files are uploaded on the GitHub repository at https://github.com/PacktPublishing/Spark-Programming-in-Scala-for-Beginners-with-Apache-Spark-3
What You Will Learn
Learn Apache Spark and Spark architecture
Look at data engineering and data processing in Spark
Work with data sources and sinks
Work with data frames, datasets, and Spark SQL
Use IntelliJ IDEA for Spark development and debugging
Understand unit testing, managing application logs, and cluster deployment
Audience
This course is designed for software engineers willing to develop a data engineering pipeline and application using Apache Spark. It is also for data architects and data engineers who are responsible for designing and building the organization's data-centric infrastructure. It will also be beneficial for the managers and architects who do not directly work with Spark implementation, and still, they work with the people who implement Apache Spark at the ground level.

Before proceeding with the course, you will need basic knowledge of the Scala programming language.
Approach
This is a hands-on and well-balanced course with theoretical and practical content. The course is example-driven and follows a working session-like approach. The author will be taking a live coding approach and explaining all the concepts needed along the way. The section will have a summary video at the end to revise the learned concepts.
Key Features
A comprehensive course designed for the beginner-level for Spark programming in Scala * Deep dive into Spark 3 architecture and data engineering * Complete tested source code and examples used on Apache Spark 3.0.0 open-source distribution from the author's end
Github Repo
https://github.com/PacktPublishing/Spark-Programming-in-Scala-for-Beginners-with-Apache-Spark-3
About the Author
Scholar Nest
ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects. It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.
Course Outline
1. Apache Spark Introduction

2. Installing and Using Apache Spark

3. Spark Execution Model and Architecture

4. Spark Programming Model and Developer Experience

5. Spark Structured API Foundation

6. Spark Data Sources and Sinks

7. Spark Dataframe and Dataset Transformations

8. Aggregations in Apache Spark

9. Spark Dataframe Joins

10. Keep Learning


This course does not require any prior knowledge of Apache Spark or Hadoop. The author explains Spark architecture and fundamental concepts to help you come up to speed and grasp the content of this course.

The course will help you understand Spark programming and apply that knowledge to build data engineering solutions.

Spark Programming in Scala for Beginners with Apache Spark 3

<div id="course-pills-1">Duration<p>2 Days</p><p>12 CPD hours</p>This course is intended for<p>This in an Introductory and beyond level course is geared for experienced Java developers seeking to be proficient in Apache Kafka.  Attendees should be experienced developers who are comfortable with Java, and have reasonable experience working with databases.</p>Overview<p>Working in a hands-on learning environment, students will explore<br />Overview of Streaming technologies<br />Kafka concepts and architecture<br />Programming using Kafka API<br />Kafka Streams<br />Monitoring Kafka<br />Tuning / Troubleshooting Kafka</p><p>Apache Kafka is a real-time data pipeline processor. It high-scalability, fault tolerance, execution speed, and fluid integrations are some of the key hallmarks that make it an integral part of many Enterprise Data architectures.  In this lab intensive two day course, students will learn how to use Kafka to build streaming solutions.</p></div><div id="course-pills-2"><h4>Introduction to Streaming Systems</h4><ul><li>Fast data</li><li>Streaming architecture</li><li>Lambda architecture</li><li>Message queues</li><li>Streaming processors</li></ul><h4>Introduction to Kafka</h4><ul><li>Architecture</li><li>Comparing Kafka with other queue systems (JMS / MQ)</li><li>Kaka concepts : Messages, Topics, Partitions, Brokers, Producers, commit logs</li><li>Kafka &amp; Zookeeper</li><li>Producing messages</li><li>Consuming messages (Consumers, Consumer Groups)</li><li>Message retention</li><li>Scaling Kafka</li></ul><h4>Programming With Kafka</h4><ul><li>Configuration parameters</li><li>Producer API (Sending messages to Kafka)</li><li>Consumer API (consuming messages from Kafka)</li><li>Commits , Offsets, Seeking</li><li>Schema with Avro</li></ul><h4>Kafka Streams</h4><ul><li>Streams overview and architecture</li><li>Streams use cases and comparison with other platforms</li><li>Learning Kafka Streaming concepts (KStream, KTable, KStore)</li><li>KStreaming operations (transformations, filters, joins, aggregations)</li></ul><h4>Administering Kafka</h4><ul><li>Hardware / Software requirements</li><li>Deploying Kafka</li><li>Configuration of brokers / topics / partitions / producers / consumers</li><li>Security: How secure Kafka cluster, and secure client communications (SASL, Kerberos)</li><li>Monitoring : monitoring tools</li><li>Capacity Planning : estimating usage and demand</li><li>Trouble shooting : failure scenarios and recovery</li></ul><h4>Monitoring and Instrumenting Kafka</h4><ul><li>Monitoring Kafka</li><li>Instrumenting with Metrics library</li><li>Instrument Kafka applications and monitor their performance</li></ul></div>

Duration
2 Days
12 CPD hours
This course is intended for
This in an Introductory and beyond level course is geared for experienced Java developers seeking to be proficient in Apache Kafka. Attendees should be experienced developers who are comfortable with Java, and have reasonable experience working with databases.
Overview
Working in a hands-on learning environment, students will explore
Overview of Streaming technologies
Kafka concepts and architecture
Programming using Kafka API
Kafka Streams
Monitoring Kafka
Tuning / Troubleshooting Kafka
Apache Kafka is a real-time data pipeline processor. It high-scalability, fault tolerance, execution speed, and fluid integrations are some of the key hallmarks that make it an integral part of many Enterprise Data architectures. In this lab intensive two day course, students will learn how to use Kafka to build streaming solutions.
Introduction to Streaming Systems
Fast data
Streaming architecture
Lambda architecture
Message queues
Streaming processors
Introduction to Kafka
Architecture
Comparing Kafka with other queue systems (JMS / MQ)
Kaka concepts : Messages, Topics, Partitions, Brokers, Producers, commit logs
Kafka & Zookeeper
Producing messages
Consuming messages (Consumers, Consumer Groups)
Message retention
Scaling Kafka
Programming With Kafka
Configuration parameters
Producer API (Sending messages to Kafka)
Consumer API (consuming messages from Kafka)
Commits , Offsets, Seeking
Schema with Avro
Kafka Streams
Streams overview and architecture
Streams use cases and comparison with other platforms
Learning Kafka Streaming concepts (KStream, KTable, KStore)
KStreaming operations (transformations, filters, joins, aggregations)
Administering Kafka
Hardware / Software requirements
Deploying Kafka
Configuration of brokers / topics / partitions / producers / consumers
Security: How secure Kafka cluster, and secure client communications (SASL, Kerberos)
Monitoring : monitoring tools
Capacity Planning : estimating usage and demand
Trouble shooting : failure scenarios and recovery
Monitoring and Instrumenting Kafka
Monitoring Kafka
Instrumenting with Metrics library
Instrument Kafka applications and monitor their performance

Nexus Human

Working with Apache Kafka (for Developers) (TTDS6760)

This video covers the essential topics necessary for working with Apache Maven. You will understand the techniques and methods to create multi-module Apache Maven projects from scratch, along with delving into topics needed for testing and deploying Java applications.

83 Apache courses in Cardiff delivered On Demand

Apache Spark 3 Advance Skills for Cracking Job Interviews

By Packt

Apache Kafka Series - Kafka Monitoring?? and Operations

By Packt

Learn Apache Cassandra in Just 2 Hours

By Packt

Real-Time Stream Processing Using Apache Spark 3 for Python Developers

By Packt

Apache Kafka for Beginners - Learn Kafka by Hands-On

By Packt

Spark Programming in Python for Beginners with Apache Spark 3

By Packt

Apache Spark with Scala - Hands-On with Big Data!

By Packt

Streaming Big Data with Spark Streaming, Scala, and Spark 3!

By Packt

The Complete Apache Groovy Developer Course

By Packt

Master Big Data Ingestion and Analytics with Flume, Sqoop, Hive and Spark

By Packt