Name: The Ultimate Hands-On Hadoop
Price: 134.99 GBP
Availability: InStock

Scala is doubtless one of the most in-demand skills for data scientists and data engineers. This competitive course will teach you the essential concepts and methodologies of Scala with a lot of practical implementations.
The course Scala from Beginner to Pro is refreshingly different. The well-thought-out quizzes and mini projects that cover all the important aspects will make your Scala learning journey that much easier. This course includes an overview of Hadoop and Spark with a hands-on project with Scala Spark. Right through the course, every theoretical explanation is followed by practical implementation. This course is designed to reflect the most in-demand Scala skills that you will start using right away at the workplace. The course is simple to follow and understand. Expressive, exhaustive, and practical with live coding sessions and replete with quizzes. Rich with state-of-the-art and up-to-date knowledge of this field. The 6 mini-projects and one Scala Spark project included in this course are a vital component of this course. These projects present you with a hands-on opportunity to experiment for yourself with trial and error. You get a chance to learn from the mistakes you make. Importantly, it's easy to understand the potential gaps that might exist between theory and practice. Code bundles are available here:
https://github.com/PacktPublishing/Scala-Spark-Master-Big-Data-with-Scala-and-Spark
What You Will Learn
Explore variables in data types in Scala
Explore functions and their usage in Scala
Learn data structures; namely, lists, ListBuffer, maps, sets, and stack
Understand the flow controls in Scala and different ways to control the flow
Explore the difference between Spark RDDs and Spark DFs
ETL pipeline from AWS S3 to AWS RDS using Spark
Audience
This course is for absolute beginners who want to learn Spark and Scala along with its implementation in realistic projects; people who want to develop smart solutions; data scientists; and machine learning experts.
Approach
This course follows a hand-on learning approach with exercises at the end of each section that will enable you to grasp the concepts by thinking about the right solution on your own and implementing it. It provides an overall interactive learning experience.
Key Features
Relate the practical concepts of Scala and Spark with real-world problems * Implement any project from scratch that requires Scala and Spark knowledge * Know the theoretical and practical aspects of Scala and Spark
Github Repo
https://github.com/PacktPublishing/Scala-Spark-Master-Big-Data-with-Scala-and-Spark/tree/main/03-Codes/Codes/06-Project
About the Author
AI Sciences
AI Sciences are experts, PhDs, and artificial intelligence practitioners, including computer science, machine learning, and Statistics. Some work in big companies such as Amazon, Google, Facebook, Microsoft, KPMG, BCG, and IBM. AI sciences produce a series of courses dedicated to beginners and newcomers on techniques and methods of machine learning, statistics, artificial intelligence, and data science. They aim to help those who wish to understand techniques more easily and start with less theory and less extended reading. Today, they publish more comprehensive courses on specific topics for wider audiences. Their courses have successfully helped more than 100,000 students master AI and data science.
Course Outline
1. Introduction

1. Why Scala
This video provides an overview of Scala and its scope.
2. Scala Applications
Explore the applications of Scala in this session.
3. About the Instructor
Let's get introduced to the instructor.
4. Introduction to Course
This video provides an overview of the entire course.
5. Projects Overview
This video provides an overview of all the projects.
2. Overview

1. What is Scala
Learn all about Scala in this video session.
2. Scala Setup (Local Machine)
Learn the steps to install and set up Scala in your local machine.
3. Hello World
Let's look at a sample Hello World code.
4. Scala Setup (Online)
Learn how to set up and run Scala code online.
5. Variables in Scala
In this session, you will learn all about variables in Scala.
6. Arithmetic Operations on Variables-1
In this lesson, we will explore arithmetic operations on variables.
7. Arithmetic Operations on Variables-2
Let's look at a few basic operations on strings.
8. Quiz (Arithmetic Operations)
Let's solve a short quiz for quick revision.
9. Solution (Arithmetic Operations)
In this lesson, we're going to discuss the solution to the problem we saw in our previous session.
10. Quiz (Strings)
Let's solve a short quiz for quick revision.
11. Solution (Strings)
In this lesson, we're going to discuss the solution to the problem we saw in our previous session.
12. Type Casting
This session covers the concept of type casting in details.
13. Taking Input from User
In this lesson, we will learn how to take input from the user.
14. Quiz (User Input and Type Casting)
Let's look at a quiz question in this video.
15. Solution (User Input and Type Casting)
Let's discuss the solution to the quiz question from our previous session.
3. Flow Control

1. Overview of Control Statements
This video provides an overview of the entire section.
2. If else statements
In this lesson, you will learn about If-else statements.
3. Conditions in If
Let's dive deeper into If-else statements and look at a few examples in this session.
4. Quiz (if statement)
It's knowledge check time! Let's look at the short quiz.
5. Solution (if statement)
Let's discuss the solution to the quiz question we saw in our previous lesson.
6. Nested If Else
In this lesson, we will learn about nested if else statements.
7. Quiz (Nested If Else)
It's knowledge check time! Let's look at a short quiz on nested if else.
8. Solution (nested if else)
In this video session, we will discuss the solution for the quiz we saw in our previous session.
9. Logical Operators
In this video session, we will learn about the different types of logical operators.
10. Quiz (Logical Operators)
Let's look at a problem statement in this short quiz.
11. Solution (Logical Operators)
Let's discuss the solution to the quiz we saw in the previous lesson.
12. If Else If
In this lesson, we will learn about another variation of the if-else statements, which is if-else-if statements in Scala.
13. Quiz (If Else If)
It's time for a short quiz on if-else-if statements.
14. Solution (If Else If)
Let's discuss the solution to the quiz in this session.
15. Overview of Loops
This video provides an overview of loops. Learn what loops are and the types.
16. Overview of While Loop
Learn what are while loops in this session.
17. While Loop
Let's explore while loops in detail in this video.
18. Quiz (While Loop)
It's time for a short quiz on while loops.
19. Solution 1 (While Loop)
Let's discuss the solution to the quiz on while loop (part 1).
20. Solution 2 (While Loop)
In this session, we will continue discussing the solution to the quiz (part 2).
21. Do While Loop
In this lesson, you will learn about do while loops.
22. For Loop
In this lesson, you will learn about for loops.
23. Quiz (For Loop)
It's time for a quiz on for loop.
24. Solution (For Loop)
Let's discuss the solution to the quiz on for loop in this session.
25. Quiz(For Loop)
Here is another problem question on for loop.
26. Solution(For Loop)
Let's look at the solution of the problem statement we discussed in the previous session.
27. Break
In this lesson, we will explore the Break statement.
28. Break Fix
In this video we will discuss Break fix.
29. Project Overview
This video provides an overview of the mini project that we will be building in the next few sessions.
30. Project Solution Design
In this session, we will design a solution for our game mini project.
31. Project Solution Code 1
Let's start coding the first part of the project.
32. Project Solution Code 2
Let's start coding for the second part of the mini-project.
33. Project Solution Code 3
Let's continue coding for our project. We will go ahead and code the third part in this session.
34. Project Solution Code 4
Let's complete the coding for the game-project and wrap up this section.
4. Functions

1. Overview of Functions
This video introduces you to functions.
2. Writing Addition Function
In this session, you will learn how to write the addition function.
3. Quiz (Basic Functions)
It's time for a quiz on basic functions.
4. Solution (Basic Functions)
Let's discuss the solution of the quiz on basic functions in this session.
5. Functions Common Issues
Learn the basic errors and common issues that you might encounter when using functions.
6. Named Arguments
This session covers functions with named arguments.
7. Quiz (String Concatenation Function)
It's time for a quiz on string concatenation functions.
8. Solution (String Concatenation Function)
Let's discuss the solution to the quiz on string concatenation.
9. Quiz (Dividing Code in Functions)
Let's look at another quiz on dividing code in functions.
10. Solution (Dividing Code in Functions)
Let's discuss the solution to the quiz on dividing code in functions.
11. Default Arguments
In this session, you will learn about default arguments in Scala.
12. Quiz (Default Arguments)
It's time for a quiz on default arguments.
13. Solution (Default Arguments)
Let's discuss the solution to the quiz on default arguments.
14. Anonymous Functions
In this lesson, you will learn all about the anonymous functions of Scala.
15. Quiz (Anonymous Functions)
It's time for a short quiz on anonymous functions.
16. Solution (Anonymous Functions)
Let's discuss the solution to the short quiz on anonymous functions in this session.
17. Scopes
In this lesson, you will learn all about Scopes.
18. Project Overview
In this video, you will get an overview of the mini-project. For this project, we will be writing a program for an ATM machine.
19. Checking Credentials
Let's start working on the mini-project. In this session, we will code for checking the credentials.
20. Prompting the Menu
In this session, we will work on prompting the menu for the mini-project.
21. Basic Functions
In this session, we will work on the global variables and the basic functions required for the project.
22. Breaking Code in More Functions
In this lesson, we will incorporate the Break statement.
23. Final Run
Let's wrap up our project in this session.
5. Classes

1. Introduction to Classes
This video introduces you to classes and various concepts related to classes.
2. Creating Class
In this lesson, you will learn how to create a class in Scala.
3. Class Constructor
In this lesson, you will learn about constructors in Scala.
4. Functions and Classes
Let's look at functions and classes in this session.
5. Project Overview
This session provides an overview of the next mini-project. Let's start working on our project and write a program for basic operations.
6. Basic Structure
In this session, we will work on creating the basic structure for our project.
7. Final Run
Let's wrap up our project in this session.
6. Data Structures

1. Introduction of Data Structures
This video introduces you to data structures in Scala.
2. Lists introduction
In this session, we will explore list data structures.
3. Lists Create and Delete Elements
Learn how to create, append and delete elements in a list in this session. Lists are immutable in Scala.
4. Lists Take
In this lesson, you will learn how to take from lists in Scala.
5. ListBuffer Introduction
In this lesson, we will discuss ListBuffer. A ListBuffer is a collection that contains mutable data.
6. Add Data in ListBuffer
In this session, you will learn how to add data in ListBuffer.
7. Remove Data from ListBuffer
In this lesson, you will learn how to remove data in ListBuffer.
8. Take Data from ListBuffer
Learn how to take elements from the ListBuffer in this session.
9. Project Overview
It's time to apply whatever we have learned so far and build a mini-project.
10. Project Architecture Discussion
Before we move on to the coding part, let's first discuss the project architecture in this session.
11. Project Architecture Implementation
We have already discussed the project architecture in our previous session. We will now go ahead and implement the project architecture.
12. User Input for Objects
In this session, you will learn how to get user input for objects.
13. Implementing the Control Flow
In this lesson, we will implement the control flow in the project.
14. Creating Required Functions Inside Class
In this session, we will create the required function inside the class and wrap up the project.
15. Overview of Maps
In this session, you will learn about maps in detail.
16. Creating Maps
In this lesson, you will learn how to create maps in Scala.
17. Check Key in Map
In this lesson, we will discuss how to check whether a key is present inside a map or not.
18. Update Value in Map
Learn how to update the key value pair in this lesson.
19. Add and Remove Items from Maps
Learn how to add and remove/delete items from maps.
20. Iterating in Maps
In this session, you will learn how to iterate in a map.
21. Project Overview
It's time for a mini-project. We are going to write a word count program.
22. Project Architecture
Let's discuss the project architecture in the session before we start the implementation process.
23. Project Structure Code
In this session, we will start creating the basic project structure.
24. Using Maps for Word Count
In this lesson, we will work on the code for getting the word count using maps.
25. Final Run
In this lesson, we will do a test run before wrapping up the project.
26. Sets Overview
Learn all about sets in this session.
27. Add and Remove Item from the Set
In this session, we will learn how to add items and remove items from sets.
28. Set Operations
In this session, we will discuss set operations, Union(), Intersection(), and Difference().
29. Overview of Stack
Learn about Stack's data structure in this pleasant.
30. Push and Pop in Stack
Let's discuss about two important functions in Stack, which are Push() and Pop().
31. Stack Attributes
In this lesson, we will explore some more functions related to Stack.
32. Project Overview
In this lesson, you will get an overview of the next mini-project we will be working on.
33. Project Architecture
Let's discuss the basic structure of the project.
34. Extra Closing Bracket Use Case
In this lesson, we will start working on the code.
35. Extra Starting Bracket Use Case
Let's wrap up the mini-project in this session.
7. Project

1. Project Introduction
In this session, you will get introduced to Scala Spark and get an overview of the entire project work.
2. Why Spark
In this session, let's explore a little bit more about Spark and why we use Spark.
3. Hadoop Ecosystem
In this lesson, we will explore the Hadoop ecosystem.
4. Spark Architecture
Learn about the Spark architecture in details in this session.
5. Spark Ecosystem
In this lesson, we will learn about the Spark core ecosystem.
6. DataBricks Account
In this lesson, let's explore Databricks and how to run our code in it.
7. Setting Up DataBricks Cluster
In this video lesson, you will learn how to set up Databricks cluster.
8. Spark Local Setup
Learn how to set up Spark in your local system in this session.
9. Spark Hadoop Setup
In this session, we will continue setting up Spark.
10. Spark RDDs
In this lesson, we will explore Spark RDDs in detail.
11. Spark RDDs (textFile, collect)
In this lesson, we will work with Spark RDDS.
12. Spark Local Run
In this lesson, we will replicate the same process we did in Databricks in our local machines.
13. Understanding Map
In this lesson, you will learn about Map functions in details.
14. Understanding Flat Map
In this session, we will discuss flat maps.
15. Understanding Reduce by Key
In this session, you will learn about reduce by key.
16. Word Count Example
In this lesson, we will look at an example on word count.
17. Spark DFs
In this lesson, we will learn about Spark Dataframes.
18. Spark DF Read Data
In this lesson, we will read the data in a dataframe.
19. Spark Print Schema, Select
In this video session we will learn about Spark Print Schema.
20. Spark Group By
Learn the grouping mechanism in Spark in this session.
21. Spark DF Write
In this session, you will learn how to write in a Spark Dataframe.
22. Creating S3 Bucket
In this session, you will learn how to create S3 buckets in AWS.
23. Creating Database in RDS
In this lesson, you will learn how to create a database in RDS.
24. Performing ETL
In this session, you will write a Scala Spark job inside AWS glue that will migrate the data from S3 to RDS.

Packt

Scala

Big Data

Scala is doubtless one of the most in-demand skills for data scientists and data engineers. This competitive course will teach you the essential concepts and methodologies of Scala with a lot of practical implementations.

Scala & Spark-Master Big Data with Scala and Spark

General Admission

<strong>This is an amazing course for anyone who wants to get started with functional programming with libraries such as cats-effect/ZIO and understand IO Monad in depth.

Learn about functional programming and effects from this most structured and well-thought-out course on Scala IOs.</strong>
        <p>Modern applications often need to describe complex workflows involving multiple external systems. These are notoriously difficult to get right as it requires handling errors, various retry strategies, and non-determinism because of concurrent executions. In this course, you will learn how to solve this problem using both imperative and functional programming.

This course is divided into three parts.

The first part covers building a CLI tool imperatively. Here, you will learn how to test code with external dependencies, handle errors, and retry if something goes wrong. One of the goals of the section is to discover how functional programmers came up with the notion of IO. It's the commonest way to action the functional programming world.

In the next part, you will create and implement an IO class from scratch; this will solve two different problems, of which one is searching for a flight aggregator. In the final part, you will extend IO to execute tasks concurrently. Here, you will be looking to add concurrency methods to IO implementation so that you can execute some tasks in the background.

By the end of the course, you will be able to read and write code using popular Scala libraries such as cats-effect, Monix, and ZIO. This skill is in high demand.

All the resource files are added to the GitHub repository at: https://github.com/PacktPublishing/Supercharge-Scala-Future---FP-Tower</p>
        <h2>What You Will Learn</h2><p>Install SBT, Scala, and IntelliJ IDEA<br>Understand the difference between IO (cats-effect, Monix, ZIO) and Future<br>Learn to use lazy evaluation to repeat actions and execute code concurrently<br>Write property-based tests for real-world examples<br>Understand the dangers of blocking code in a concurrent environment<br>Execute a project on flight search aggregator</p>
        <h2>Audience</h2><p>This course is designed for developers interested in concurrency, functional programming, and backend developers. Scala developers more familiar with Akka, Play, and Future will also gain a lot from this course. Individuals who want to gain the skills to join one of the many companies using functional Scala and learn to think functionally will be best suited to this course.<br><br>You will need to have moderate programming experience with Scala. The author will take you through downloading and installing SBT, Scala, and an IDE.</p>
        <h2>Approach</h2><p>This course is designed with a practical and hands-on approach. In this course, you will write lots of Scala code, test it using both example-based and property-based tests, experience issues caused by blocking logic and concurrent execution, and work your way to efficient solutions in small and incremental steps. The author guides you through all the exercises and provides a complete set of solutions.</p>
        <h2>Key Features</h2><p>Learn by demystifying complex concepts using explanations that are free of confusing jargon * Gain the skills to join one of the many companies using functional Scala * Learn to think functionally with the help of all requisite resources and learn from scratch</p>
        <h2>Github Repo</h2><p>https://github.com/PacktPublishing/Supercharge-Scala-Future---FP-Tower</p>
        <h2>About the Author</h2><h3>Julien Truffaut</h3><p>Fp-Tower by Julien Truffaut offers online Scala courses that help in learning the practical side of functional programming, so you can improve your skills and your career. He is a programmer who specializes in functional programming and the Scala programming language. He has been conducting functional programming seminars and online courses with remarkable success for over seven years. He focuses on demystifying complex subjects through straightforward explanations free of jargon.

He is the author of Monocle, an optics library for functional Scala developers, which allows them to quickly transform and access deeply nested data structures. This is comparable to XPath or jq, except that it applies to all immutable data structures.</p>
        <h2>Course Outline</h2><h4>1. Actions with Imperative Programming</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Introduction</p><p style='margin: 0'>This video provides an overview of the course.</p></td></tr><tr><td style='padding: 15px;'><p>2. Console Exercise - Testing</p><p style='margin: 0'>This video demonstrates how to take users' input through the command line.</p></td></tr><tr><td style='padding: 15px;'><p>3. Console Exercises - Error Handling</p><p style='margin: 0'>This video demonstrates how to take users' input through the command line, making it more user-friendly using the error handling feature.</p></td></tr><tr><td style='padding: 15px;'><p>4. Evaluation</p><p style='margin: 0'>This video demonstrates how code is executed in Scala and how to delay an execution to an appropriate time. You will start by reviewing the techniques to define a code expression in Scala.</p></td></tr><tr><td style='padding: 15px;'><p>5. Console Exercises - retry and onError</p><p style='margin: 0'>This video helps you learn the concept of retrying an operation with a single higher order function and reuse to refactor the code of readDataOfBirthRetry and readSubscribeToMailingListRetry.</p></td></tr><tr><td style='padding: 15px;'><p>6. Console Exercises - Code Organization</p><p style='margin: 0'>This video demonstrates and explains code organization.</p></td></tr></tbody></table><br><h4>2. Build Your Own IO</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. What Is an IO?</p><p style='margin: 0'>This video provides an overview about IO (Input/Output).</p></td></tr><tr><td style='padding: 15px;'><p>2. IO Composition</p><p style='margin: 0'>This video explains about IO composition.</p></td></tr><tr><td style='padding: 15px;'><p>3. For Comprehension</p><p style='margin: 0'>This video demonstrates the comprehension part.</p></td></tr><tr><td style='padding: 15px;'><p>4. Error Handling with IO</p><p style='margin: 0'>This video talks about error handling with IO.</p></td></tr><tr><td style='padding: 15px;'><p>5. Refactoring</p><p style='margin: 0'>This video explains the concept of refactoring.</p></td></tr></tbody></table><br><h4>3. Project: Search Flight Aggregator</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. First Implementation</p><p style='margin: 0'>This video provides an introduction and the project's implementation from scratch.</p></td></tr><tr><td style='padding: 15px;'><p>2. Traverse and Sequence</p><p style='margin: 0'>This video demonstrates the concepts of traverse and sequence in this project.</p></td></tr><tr><td style='padding: 15px;'><p>3. Concurrent IO</p><p style='margin: 0'>This video explains one of the toughest concepts, which is Concurrent IO.</p></td></tr><tr><td style='padding: 15px;'><p>4. Non-Blocking IO</p><p style='margin: 0'>This video explains about Non-Blocking IO.</p></td></tr></tbody></table><br><h4>4. Conclusion</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Thank You and Conclusion</p><p style='margin: 0'>This video talks about the course conclusion and helps you with the next steps.</p></td></tr></tbody></table>

This is an amazing course for anyone who wants to get started with functional programming with libraries such as cats-effect/ZIO and understand IO Monad in depth. Learn about functional programming and effects from this most structured and well-thought-out course on Scala IOs.
Modern applications often need to describe complex workflows involving multiple external systems. These are notoriously difficult to get right as it requires handling errors, various retry strategies, and non-determinism because of concurrent executions. In this course, you will learn how to solve this problem using both imperative and functional programming. This course is divided into three parts. The first part covers building a CLI tool imperatively. Here, you will learn how to test code with external dependencies, handle errors, and retry if something goes wrong. One of the goals of the section is to discover how functional programmers came up with the notion of IO. It's the commonest way to action the functional programming world. In the next part, you will create and implement an IO class from scratch; this will solve two different problems, of which one is searching for a flight aggregator. In the final part, you will extend IO to execute tasks concurrently. Here, you will be looking to add concurrency methods to IO implementation so that you can execute some tasks in the background. By the end of the course, you will be able to read and write code using popular Scala libraries such as cats-effect, Monix, and ZIO. This skill is in high demand. All the resource files are added to the GitHub repository at: https://github.com/PacktPublishing/Supercharge-Scala-Future---FP-Tower
What You Will Learn
Install SBT, Scala, and IntelliJ IDEA
Understand the difference between IO (cats-effect, Monix, ZIO) and Future
Learn to use lazy evaluation to repeat actions and execute code concurrently
Write property-based tests for real-world examples
Understand the dangers of blocking code in a concurrent environment
Execute a project on flight search aggregator
Audience
This course is designed for developers interested in concurrency, functional programming, and backend developers. Scala developers more familiar with Akka, Play, and Future will also gain a lot from this course. Individuals who want to gain the skills to join one of the many companies using functional Scala and learn to think functionally will be best suited to this course.

You will need to have moderate programming experience with Scala. The author will take you through downloading and installing SBT, Scala, and an IDE.
Approach
This course is designed with a practical and hands-on approach. In this course, you will write lots of Scala code, test it using both example-based and property-based tests, experience issues caused by blocking logic and concurrent execution, and work your way to efficient solutions in small and incremental steps. The author guides you through all the exercises and provides a complete set of solutions.
Key Features
Learn by demystifying complex concepts using explanations that are free of confusing jargon * Gain the skills to join one of the many companies using functional Scala * Learn to think functionally with the help of all requisite resources and learn from scratch
Github Repo
https://github.com/PacktPublishing/Supercharge-Scala-Future---FP-Tower
About the Author
Julien Truffaut
Fp-Tower by Julien Truffaut offers online Scala courses that help in learning the practical side of functional programming, so you can improve your skills and your career. He is a programmer who specializes in functional programming and the Scala programming language. He has been conducting functional programming seminars and online courses with remarkable success for over seven years. He focuses on demystifying complex subjects through straightforward explanations free of jargon. He is the author of Monocle, an optics library for functional Scala developers, which allows them to quickly transform and access deeply nested data structures. This is comparable to XPath or jq, except that it applies to all immutable data structures.
Course Outline
1. Actions with Imperative Programming

1. Introduction
This video provides an overview of the course.
2. Console Exercise - Testing
This video demonstrates how to take users' input through the command line.
3. Console Exercises - Error Handling
This video demonstrates how to take users' input through the command line, making it more user-friendly using the error handling feature.
4. Evaluation
This video demonstrates how code is executed in Scala and how to delay an execution to an appropriate time. You will start by reviewing the techniques to define a code expression in Scala.
5. Console Exercises - retry and onError
This video helps you learn the concept of retrying an operation with a single higher order function and reuse to refactor the code of readDataOfBirthRetry and readSubscribeToMailingListRetry.
6. Console Exercises - Code Organization
This video demonstrates and explains code organization.
2. Build Your Own IO

1. What Is an IO?
This video provides an overview about IO (Input/Output).
2. IO Composition
This video explains about IO composition.
3. For Comprehension
This video demonstrates the comprehension part.
4. Error Handling with IO
This video talks about error handling with IO.
5. Refactoring
This video explains the concept of refactoring.
3. Project: Search Flight Aggregator

1. First Implementation
This video provides an introduction and the project's implementation from scratch.
2. Traverse and Sequence
This video demonstrates the concepts of traverse and sequence in this project.
3. Concurrent IO
This video explains one of the toughest concepts, which is Concurrent IO.
4. Non-Blocking IO
This video explains about Non-Blocking IO.
4. Conclusion

1. Thank You and Conclusion
This video talks about the course conclusion and helps you with the next steps.

Java

Programming

This is an amazing course for anyone who wants to get started with functional programming with libraries such as cats-effect/ZIO and understand IO Monad in depth.

Learn about functional programming and effects from this most structured and well-thought-out course on Scala IOs.

Supercharge Scala Future | FP-Tower

<strong>In this course, we will process massive streams of real-time data using Spark Streaming and create Spark applications using the Scala programming language (v2.12). We will also get our hands-on with some real live Twitter data, simulated streams of Apache access logs, and even data used to train machine learning models.</strong>
        <p>In this course, you will learn the basics of the Scala programming language; learn how Apache Spark operates on a cluster; set up discretized streams with Spark Streaming and transform them as data is received; analyze streaming data over sliding windows of time; maintain stateful information across streams of data; connect Spark Streaming with highly scalable sources of data, including Kafka, Flume, and Kinesis; dump streams of data in real-time to NoSQL databases such as Cassandra; run SQL queries on streamed data in real-time; train machine learning models in real-time with streaming data, and use them to make predictions that keep getting better over time; and also, package, deploy, and run self-contained Spark Streaming code to a real Hadoop cluster using Amazon Elastic MapReduce.

This course is very hands-on, filled with achievable activities and exercises to reinforce your learning. By the end of this course, you will be confidently creating Spark Streaming scripts in Scala and be prepared to tackle massive streams of data in a whole new way. You will be surprised at how easy Spark Streaming makes it!

All the codes and supporting files for this course are available at https://github.com/packtpublishing/streaming-big-data-with-spark-streaming-scala-and-spark-3-</p>
        <h2>What You Will Learn</h2><p>Process large amounts of real-time data using the Spark Streaming module<br>Create efficient Spark applications using the Scala programming language<br>Integrate Spark Streaming with various data sources<br>Integrate Spark Streaming with Spark SQL to query your data in real time<br>Train machine learning models with streaming data, and use for real-time predictions<br>Maintain stateful data across a continuous stream of input data</p>
        <h2>Audience</h2><p>If you are a student who wants to learn how to use Apache Spark or a big data professional who wants to process large amounts of data on a real-time basis, this course is for you. Some basic programming and scripting experience is required to get the most out of the course.</p>
        <h2>Approach</h2><p>This course is very hands-on, filled with achievable activities and exercises to reinforce your learning. In this course, we will cover some real live Twitter data, simulated streams of Apache access logs, and even data used to train machine learning models! You will write and run real Spark Streaming jobs right at home on your own PC.</p>
        <h2>Key Features</h2><p>Process streams of real-time data from various sources with Spark Streaming * Query your streaming data in real-time using Spark SQL * A comprehensive tutorial with practical examples to help you develop real-time Spark applications</p>
        <h2>Github Repo</h2><p>https://github.com/packtpublishing/streaming-big-data-with-spark-streaming-scala-and-spark-3-</p>
        <h2>About the Author</h2><h3>Frank Kane</h3><p>Frank Kane has spent nine years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers all the time. He holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology and teaches others about big data analysis.</p>
        <h2>Course Outline</h2><h4>1. Getting Started</h4><p>In this first section, we will introduce the course and set up the development environment for Spark and Scala. We will then create our first Spark Streaming application to display live Twitter feeds.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Introduction, and Getting Set Up</p><p style='margin: 0'>A brief introduction to the course, and then we will get your development environment for Spark and Scala all set up on your desktop. A quick test application will confirm Spark is working on your system!</p></td></tr><tr><td style='padding: 15px;'><p>2. [Activity] Stream Live Tweets with Spark Streaming!</p><p style='margin: 0'>Get set up with a Twitter developer account and run your first Spark Streaming application to listen to and print out live Tweets as they happen!</p></td></tr></tbody></table><br><h4>2. A Crash Course in Scala</h4><p>In this section, we will take a deep dive into Scala. Even if you've never used Scala before, we will cover all of the syntax and structure that you will need to know in order to begin mining your data. We will run through flow control, functions, and data structures with numerous examples throughout so that you can see everything in action and make sure you're understanding it.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. [Activity] Scala Basics</p><p style='margin: 0'>We start our crash course in the Scala programming language by covering some basics of the language: types and variables, printing, and Boolean comparisons.</p></td></tr><tr><td style='padding: 15px;'><p>2. [Exercise] Flow Control in Scala</p><p style='margin: 0'>Our Scala crash course continues, illustrating various means of flow control in Scala. For loops, do/while loops, while loops and so on.</p></td></tr><tr><td style='padding: 15px;'><p>3. [Exercise] Functions in Scala</p><p style='margin: 0'>Scala is a functional programming language, and so understanding how functions work and are treated in Scala is hugely important! This lecture covers the fundamentals, and lets you put it into practice.</p></td></tr><tr><td style='padding: 15px;'><p>4. [Exercise] Data Structures in Scala</p><p style='margin: 0'>We wrap up our Scala crash course with commonly used data structures using in Spark with Scala. Tuples, lists, and maps.</p></td></tr></tbody></table><br><h4>3. Spark Streaming Concepts</h4><p>In this section, we will introduce the core concepts of Spark and Spark Streaming. We will define and work with RDDs (Resilient Distributed Datasets) and show you examples of what they can do. RDDs are the key to understanding Spark, and therefore also Spark Streaming. This is also where we will dive deeper into Spark Streaming itself, understanding its inner workings and how the concept of windowing will enable you to aggregate and analyze data over time. Finally, we will take a look at reliability and fault tolerance with regards to Spark Streaming and a few ways in which you can maximize reliability.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Introduction to Spark</p><p style='margin: 0'>Before you can learn about Spark Streaming, you need to understand how Spark itself works at a high level! This video covers the why and how of Apache Spark, of which Spark Streaming is a component.</p></td></tr><tr><td style='padding: 15px;'><p>2. The Resilient Distributed Dataset (RDD)</p><p style='margin: 0'>The fundamental object of Spark programming is the Resilient Distributed Dataset (RDD), and this is used not just in Spark but also within Spark Streaming scripts. This lecture explains what they are and what you can do with them.</p></td></tr><tr><td style='padding: 15px;'><p>3. [Activity] RDD's in Action: Simple Word Count Application</p><p style='margin: 0'>Let's walk through and actually run a simple Spark script that counts the number of occurrences of each word in a book.</p></td></tr><tr><td style='padding: 15px;'><p>4. Introduction to Spark Streaming</p><p style='margin: 0'>We finally have all the pr-requisite knowledge to start talking about Spark Streaming itself in more detail! We will cover how it works, what it's for, and its architecture.</p></td></tr><tr><td style='padding: 15px;'><p>5. [Activity] Revisiting the PrintTweets application</p><p style='margin: 0'>Now that we know more, let's go revisit that first Spark Streaming application we ran in lecture two, and dive into how it really works.</p></td></tr><tr><td style='padding: 15px;'><p>6. Windowing: Aggregating data over longer time spans</p><p style='margin: 0'>Windowing allows you to analyze streaming data over a sliding window of time, which lets you do much more than just transform streaming data and store it someplace else. We will cover the concepts of the batch, window, and slide intervals, and how they work together to let you aggregate streaming data over some period of time.</p></td></tr><tr><td style='padding: 15px;'><p>7. Fault Tolerance in Spark Streaming</p><p style='margin: 0'>How can Spark Streaming do so much work continuously in a reliable manner? We will uncover some of its tricks for reliability, as well as tips for configuring Spark Streaming to be as reliable as possible.</p></td></tr></tbody></table><br><h4>4. Spark Streaming Examples with Twitter</h4><p>Now that we have seen the theory behind Spark Streaming, this section is dedicated to putting all of that into action. We will look at three practical examples of taking Twitter data and applying our new learning to it with file output, windowing, and live tracking, and analysis.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. [Exercise] Saving Tweets to Disk</p><p style='margin: 0'>We will build on our "print tweets" example to actually store the incoming tweets to disk and illustrate how Spark Streaming can handle file output.</p></td></tr><tr><td style='padding: 15px;'><p>2. [Exercise] Tracking the Average Tweet Length</p><p style='margin: 0'>Compute the average length of a tweet, using windowing in Spark Streaming.</p></td></tr><tr><td style='padding: 15px;'><p>3. [Exercise] Tracking the Most Popular Hashtags</p><p style='margin: 0'>This is a fun one! We will track the most popular hashtags in Twitter over time and watch how they change in real time!</p></td></tr></tbody></table><br><h4>5. Spark Streaming Examples with Clickstream / Apache Access Log Data</h4><p>This section will continue to look at some of the more advanced aspects of Spark Streaming, this time using Apache access logs to monitor website traffic in real time, showing the most accessed pages and flagging up and log errors as they occur. We will also use SparkSQL to query our data live. Finally, we will look at Spark 2's new structured streaming approach, and how we can use DataSets to streamline our analysis.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. [Exercise] Tracking the Top URL's Requested</p><p style='margin: 0'>We will simulate an incoming stream of Apache access logs and use Spark Streaming to keep track of the most-requested web pages in real time!</p></td></tr><tr><td style='padding: 15px;'><p>2. [Exercise] Alarming on Log Errors</p><p style='margin: 0'>This example will listen to an Apache access log stream and raise an alarm if too many errors are returned by the server in real time.</p></td></tr><tr><td style='padding: 15px;'><p>3. [Exercise] Integrating Spark Streaming with Spark SQL</p><p style='margin: 0'>We will integrate Spark Streaming with Spark SQL, allowing us to run SQL queries on data as it is streamed in! Again, we will use Apache logs as an example.</p></td></tr><tr><td style='padding: 15px;'><p>4. Introduction to Structured Streaming</p><p style='margin: 0'>Spark 2.0 introduced experimental support for Structured Streaming, a new DataSet-based API for Spark Streaming that is bound to become increasingly important. Learn how it works.</p></td></tr><tr><td style='padding: 15px;'><p>5. [Activity] Analyzing Apache Log files with Structured Streaming</p><p style='margin: 0'>As an example, we will stream Apache access logs in from a directory and use Structured Streaming to count up status codes over a one-hour moving window.</p></td></tr></tbody></table><br><h4>6. Integrating with Other Systems</h4><p>In this section, we will examine how Spark Streaming can be integrated with other popular data science systems. We will see how it can work with Apache Kafka to listen to and process messages in real time; how Apache Flume can work with Spark Streaming to process huge amounts of log data; using Amazon Kinesis with Spark Streaming to process live data on an AWS/EC2 cluster; and using Cassandra to store massive datasets and analyze them at speed. We will also cover how to write your own Data Receiver class in order to connect Spark Streaming to other proprietary systems.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Integrating with Apache Kafka</p><p style='margin: 0'>Apache Kafka is a popular and robust technology for publishing messages across a cluster on a large scale. We will show how to get Spark Streaming to listen to Kafka topics and process them in real time.</p></td></tr><tr><td style='padding: 15px;'><p>2. Integrating with Apache Flume</p><p style='margin: 0'>Flume is a popular technology for publishing log information at large scale, especially on a Hadoop cluster. We will illustrate how to set up both push-based and pull-based Flume configurations with Spark Streaming and discuss the tradeoffs of each.</p></td></tr><tr><td style='padding: 15px;'><p>3. Integrating with Amazon Kinesis</p><p style='margin: 0'>Amazon's Kinesis Streaming service is basically Kafka on AWS. If you're working with an AWS/EC2 cluster, you will want to know how to integrate Spark Streaming with Kinesis - and that's what this lecture covers.</p></td></tr><tr><td style='padding: 15px;'><p>4. [Activity] Writing Custom Data Receivers</p><p style='margin: 0'>What if you need to integrate Spark Streaming with some proprietary system that does not have an existing connection library? Well, you can always write your own Receiver class. This example shows you how and actually lets you build and run one.</p></td></tr><tr><td style='padding: 15px;'><p>5. Integrating with Cassandra</p><p style='margin: 0'>Cassandra is a popular "NoSQL" database that can be used to provide fast access to massive data sets to real-time applications. Dumping data transformed by Spark Streaming into a Cassandra database can expose that data to your larger, real-time services. We will show you how, and actually run a simple example.</p></td></tr></tbody></table><br><h4>7. Advanced Spark Streaming Examples</h4><p>In this section, we will run through some more advanced real-world examples of Spark Streaming. We will examine tracking stateful information, machine learning with K-Means clustering, and feeding data into linear regression models.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. [Exercise] Stateful Information in Spark Streams</p><p style='margin: 0'>Spark has the ability to track arbitrary state across streams of data as they come in, such as web sessions, running totals and so on. This example shows you how it all works, and challenges you to track your own state using our example as a baseline.</p></td></tr><tr><td style='padding: 15px;'><p>2. [Activity] Streaming K-Means Clustering</p><p style='margin: 0'>Spark Streaming integrates with some of Spark's MLLib (Machine Learning Library) capabilities. This example creates a real-time K-Means clustering example; it does unsupervised machine learning that continually gets better as more training data feeds into it.</p></td></tr><tr><td style='padding: 15px;'><p>3. [Activity] Streaming Linear Regression</p><p style='margin: 0'>Spark Streaming can also feed data in real-time to linear regression models, that get better over time as more data is fed into them. This example shows linear regression in action with Spark Streaming.</p></td></tr></tbody></table><br><h4>8. Spark Streaming in Production</h4><p>This section covers how to move your applications into a production environment, using the spark-submit command to process scripts from the command line. We will also look at packaging scripts and dependencies with SBT and running code on a real cluster with Amazon's Elastic MapReduce (EMR). Finally, we will troubleshoot some code. We will see what can go wrong, and how you can identify and debug issues and tune your Spark applications for peak performance.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. [Activity] Packaging and Running Spark Code in Production</p><p style='margin: 0'>Your production applications won't be run from within the Scala IDE; you will need to run them from a command line, and potentially on a cluster. The spark-submit command is used for this. We will show you how to package up your application and run it using spark-submit from a command prompt.</p></td></tr><tr><td style='padding: 15px;'><p>2. [Activity] Packaging Your Code with SBT</p><p style='margin: 0'>If your Spark Streaming application has external library dependencies that might not be already present on every machine in your cluster, the SBT tool can manage those dependencies for you, and package them into the JAR file you run with spark-submit. We will show you how it works with a real example.</p></td></tr><tr><td style='padding: 15px;'><p>3. Running on a Real Hadoop Cluster with EMR</p><p style='margin: 0'>We will run our simple word count example on a real cluster, using Amazon's Elastic MapReduce service! This just shows you what's involved in running a Spark Streaming job on a real cluster as opposed to your desktop; there are a few parameters to Spark-submit you need to worry about and getting your scripts and data in the right place is also something you need to deal with.</p></td></tr><tr><td style='padding: 15px;'><p>4. Troubleshooting and Tuning Spark Jobs</p><p style='margin: 0'>Spark jobs rarely run perfectly, if at all, on the first try - some tuning and debugging is usually required and arriving at the right scale of your cluster is also necessary. We will cover some performance tips, and how to troubleshoot what's going on with a Spark Streaming job running on a cluster.</p></td></tr></tbody></table><br><h4>9. You Made It!</h4><p>This final section runs through some recommendations on where to learn more about Spark Streaming.</p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Learning More</p><p style='margin: 0'>Want to learn more about Spark Streaming? Here are a few books and other resources I've found valuable.</p></td></tr></tbody></table>

In this course, we will process massive streams of real-time data using Spark Streaming and create Spark applications using the Scala programming language (v2.12). We will also get our hands-on with some real live Twitter data, simulated streams of Apache access logs, and even data used to train machine learning models.
In this course, you will learn the basics of the Scala programming language; learn how Apache Spark operates on a cluster; set up discretized streams with Spark Streaming and transform them as data is received; analyze streaming data over sliding windows of time; maintain stateful information across streams of data; connect Spark Streaming with highly scalable sources of data, including Kafka, Flume, and Kinesis; dump streams of data in real-time to NoSQL databases such as Cassandra; run SQL queries on streamed data in real-time; train machine learning models in real-time with streaming data, and use them to make predictions that keep getting better over time; and also, package, deploy, and run self-contained Spark Streaming code to a real Hadoop cluster using Amazon Elastic MapReduce. This course is very hands-on, filled with achievable activities and exercises to reinforce your learning. By the end of this course, you will be confidently creating Spark Streaming scripts in Scala and be prepared to tackle massive streams of data in a whole new way. You will be surprised at how easy Spark Streaming makes it! All the codes and supporting files for this course are available at https://github.com/packtpublishing/streaming-big-data-with-spark-streaming-scala-and-spark-3-
What You Will Learn
Process large amounts of real-time data using the Spark Streaming module
Create efficient Spark applications using the Scala programming language
Integrate Spark Streaming with various data sources
Integrate Spark Streaming with Spark SQL to query your data in real time
Train machine learning models with streaming data, and use for real-time predictions
Maintain stateful data across a continuous stream of input data
Audience
If you are a student who wants to learn how to use Apache Spark or a big data professional who wants to process large amounts of data on a real-time basis, this course is for you. Some basic programming and scripting experience is required to get the most out of the course.
Approach
This course is very hands-on, filled with achievable activities and exercises to reinforce your learning. In this course, we will cover some real live Twitter data, simulated streams of Apache access logs, and even data used to train machine learning models! You will write and run real Spark Streaming jobs right at home on your own PC.
Key Features
Process streams of real-time data from various sources with Spark Streaming * Query your streaming data in real-time using Spark SQL * A comprehensive tutorial with practical examples to help you develop real-time Spark applications
Github Repo
https://github.com/packtpublishing/streaming-big-data-with-spark-streaming-scala-and-spark-3-
About the Author
Frank Kane
Frank Kane has spent nine years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers all the time. He holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology and teaches others about big data analysis.
Course Outline
1. Getting Started
In this first section, we will introduce the course and set up the development environment for Spark and Scala. We will then create our first Spark Streaming application to display live Twitter feeds.
1. Introduction, and Getting Set Up
A brief introduction to the course, and then we will get your development environment for Spark and Scala all set up on your desktop. A quick test application will confirm Spark is working on your system!
2. [Activity] Stream Live Tweets with Spark Streaming!
Get set up with a Twitter developer account and run your first Spark Streaming application to listen to and print out live Tweets as they happen!
2. A Crash Course in Scala
In this section, we will take a deep dive into Scala. Even if you've never used Scala before, we will cover all of the syntax and structure that you will need to know in order to begin mining your data. We will run through flow control, functions, and data structures with numerous examples throughout so that you can see everything in action and make sure you're understanding it.
1. [Activity] Scala Basics
We start our crash course in the Scala programming language by covering some basics of the language: types and variables, printing, and Boolean comparisons.
2. [Exercise] Flow Control in Scala
Our Scala crash course continues, illustrating various means of flow control in Scala. For loops, do/while loops, while loops and so on.
3. [Exercise] Functions in Scala
Scala is a functional programming language, and so understanding how functions work and are treated in Scala is hugely important! This lecture covers the fundamentals, and lets you put it into practice.
4. [Exercise] Data Structures in Scala
We wrap up our Scala crash course with commonly used data structures using in Spark with Scala. Tuples, lists, and maps.
3. Spark Streaming Concepts
In this section, we will introduce the core concepts of Spark and Spark Streaming. We will define and work with RDDs (Resilient Distributed Datasets) and show you examples of what they can do. RDDs are the key to understanding Spark, and therefore also Spark Streaming. This is also where we will dive deeper into Spark Streaming itself, understanding its inner workings and how the concept of windowing will enable you to aggregate and analyze data over time. Finally, we will take a look at reliability and fault tolerance with regards to Spark Streaming and a few ways in which you can maximize reliability.
1. Introduction to Spark
Before you can learn about Spark Streaming, you need to understand how Spark itself works at a high level! This video covers the why and how of Apache Spark, of which Spark Streaming is a component.
2. The Resilient Distributed Dataset (RDD)
The fundamental object of Spark programming is the Resilient Distributed Dataset (RDD), and this is used not just in Spark but also within Spark Streaming scripts. This lecture explains what they are and what you can do with them.
3. [Activity] RDD's in Action: Simple Word Count Application
Let's walk through and actually run a simple Spark script that counts the number of occurrences of each word in a book.
4. Introduction to Spark Streaming
We finally have all the pr-requisite knowledge to start talking about Spark Streaming itself in more detail! We will cover how it works, what it's for, and its architecture.
5. [Activity] Revisiting the PrintTweets application
Now that we know more, let's go revisit that first Spark Streaming application we ran in lecture two, and dive into how it really works.
6. Windowing: Aggregating data over longer time spans
Windowing allows you to analyze streaming data over a sliding window of time, which lets you do much more than just transform streaming data and store it someplace else. We will cover the concepts of the batch, window, and slide intervals, and how they work together to let you aggregate streaming data over some period of time.
7. Fault Tolerance in Spark Streaming
How can Spark Streaming do so much work continuously in a reliable manner? We will uncover some of its tricks for reliability, as well as tips for configuring Spark Streaming to be as reliable as possible.
4. Spark Streaming Examples with Twitter
Now that we have seen the theory behind Spark Streaming, this section is dedicated to putting all of that into action. We will look at three practical examples of taking Twitter data and applying our new learning to it with file output, windowing, and live tracking, and analysis.
1. [Exercise] Saving Tweets to Disk
We will build on our "print tweets" example to actually store the incoming tweets to disk and illustrate how Spark Streaming can handle file output.
2. [Exercise] Tracking the Average Tweet Length
Compute the average length of a tweet, using windowing in Spark Streaming.
3. [Exercise] Tracking the Most Popular Hashtags
This is a fun one! We will track the most popular hashtags in Twitter over time and watch how they change in real time!
5. Spark Streaming Examples with Clickstream / Apache Access Log Data
This section will continue to look at some of the more advanced aspects of Spark Streaming, this time using Apache access logs to monitor website traffic in real time, showing the most accessed pages and flagging up and log errors as they occur. We will also use SparkSQL to query our data live. Finally, we will look at Spark 2's new structured streaming approach, and how we can use DataSets to streamline our analysis.
1. [Exercise] Tracking the Top URL's Requested
We will simulate an incoming stream of Apache access logs and use Spark Streaming to keep track of the most-requested web pages in real time!
2. [Exercise] Alarming on Log Errors
This example will listen to an Apache access log stream and raise an alarm if too many errors are returned by the server in real time.
3. [Exercise] Integrating Spark Streaming with Spark SQL
We will integrate Spark Streaming with Spark SQL, allowing us to run SQL queries on data as it is streamed in! Again, we will use Apache logs as an example.
4. Introduction to Structured Streaming
Spark 2.0 introduced experimental support for Structured Streaming, a new DataSet-based API for Spark Streaming that is bound to become increasingly important. Learn how it works.
5. [Activity] Analyzing Apache Log files with Structured Streaming
As an example, we will stream Apache access logs in from a directory and use Structured Streaming to count up status codes over a one-hour moving window.
6. Integrating with Other Systems
In this section, we will examine how Spark Streaming can be integrated with other popular data science systems. We will see how it can work with Apache Kafka to listen to and process messages in real time; how Apache Flume can work with Spark Streaming to process huge amounts of log data; using Amazon Kinesis with Spark Streaming to process live data on an AWS/EC2 cluster; and using Cassandra to store massive datasets and analyze them at speed. We will also cover how to write your own Data Receiver class in order to connect Spark Streaming to other proprietary systems.
1. Integrating with Apache Kafka
Apache Kafka is a popular and robust technology for publishing messages across a cluster on a large scale. We will show how to get Spark Streaming to listen to Kafka topics and process them in real time.
2. Integrating with Apache Flume
Flume is a popular technology for publishing log information at large scale, especially on a Hadoop cluster. We will illustrate how to set up both push-based and pull-based Flume configurations with Spark Streaming and discuss the tradeoffs of each.
3. Integrating with Amazon Kinesis
Amazon's Kinesis Streaming service is basically Kafka on AWS. If you're working with an AWS/EC2 cluster, you will want to know how to integrate Spark Streaming with Kinesis - and that's what this lecture covers.
4. [Activity] Writing Custom Data Receivers
What if you need to integrate Spark Streaming with some proprietary system that does not have an existing connection library? Well, you can always write your own Receiver class. This example shows you how and actually lets you build and run one.
5. Integrating with Cassandra
Cassandra is a popular "NoSQL" database that can be used to provide fast access to massive data sets to real-time applications. Dumping data transformed by Spark Streaming into a Cassandra database can expose that data to your larger, real-time services. We will show you how, and actually run a simple example.
7. Advanced Spark Streaming Examples
In this section, we will run through some more advanced real-world examples of Spark Streaming. We will examine tracking stateful information, machine learning with K-Means clustering, and feeding data into linear regression models.
1. [Exercise] Stateful Information in Spark Streams
Spark has the ability to track arbitrary state across streams of data as they come in, such as web sessions, running totals and so on. This example shows you how it all works, and challenges you to track your own state using our example as a baseline.
2. [Activity] Streaming K-Means Clustering
Spark Streaming integrates with some of Spark's MLLib (Machine Learning Library) capabilities. This example creates a real-time K-Means clustering example; it does unsupervised machine learning that continually gets better as more training data feeds into it.
3. [Activity] Streaming Linear Regression
Spark Streaming can also feed data in real-time to linear regression models, that get better over time as more data is fed into them. This example shows linear regression in action with Spark Streaming.
8. Spark Streaming in Production
This section covers how to move your applications into a production environment, using the spark-submit command to process scripts from the command line. We will also look at packaging scripts and dependencies with SBT and running code on a real cluster with Amazon's Elastic MapReduce (EMR). Finally, we will troubleshoot some code. We will see what can go wrong, and how you can identify and debug issues and tune your Spark applications for peak performance.
1. [Activity] Packaging and Running Spark Code in Production
Your production applications won't be run from within the Scala IDE; you will need to run them from a command line, and potentially on a cluster. The spark-submit command is used for this. We will show you how to package up your application and run it using spark-submit from a command prompt.
2. [Activity] Packaging Your Code with SBT
If your Spark Streaming application has external library dependencies that might not be already present on every machine in your cluster, the SBT tool can manage those dependencies for you, and package them into the JAR file you run with spark-submit. We will show you how it works with a real example.
3. Running on a Real Hadoop Cluster with EMR
We will run our simple word count example on a real cluster, using Amazon's Elastic MapReduce service! This just shows you what's involved in running a Spark Streaming job on a real cluster as opposed to your desktop; there are a few parameters to Spark-submit you need to worry about and getting your scripts and data in the right place is also something you need to deal with.
4. Troubleshooting and Tuning Spark Jobs
Spark jobs rarely run perfectly, if at all, on the first try - some tuning and debugging is usually required and arriving at the right scale of your cluster is also necessary. We will cover some performance tips, and how to troubleshoot what's going on with a Spark Streaming job running on a cluster.
9. You Made It!
This final section runs through some recommendations on where to learn more about Spark Streaming.
1. Learning More
Want to learn more about Spark Streaming? Here are a few books and other resources I've found valuable.

Apache Spark

In this course, we will process massive streams of real-time data using Spark Streaming and create Spark applications using the Scala programming language (v2.12). We will also get our hands-on with some real live Twitter data, simulated streams of Apache access logs, and even data used to train machine learning models.

Streaming Big Data with Spark Streaming, Scala, and Spark 3!

This is a comprehensive and practical Apache Spark course. In this course, you will learn and master the art of framing data analysis problems as Spark problems through 20+ hands-on examples, and then scale them up to run on cloud computing services. Explore Spark 3, IntelliJ, Structured Streaming, and a stronger focus on the DataSet API.
'Big data' analysis is a hot and highly valuable skill-and this course will teach you the hottest technology in big data: Apache Spark. Employers including Amazon, eBay, NASA JPL, and Yahoo all use Spark to quickly extract meaning from massive datasets across a fault-tolerant Hadoop cluster. You will learn those same techniques using your own Windows system right at home. It is easier than you think, and you will learn from an ex-engineer and senior manager from Amazon and IMDb. In this course, you will learn the concepts of Spark's Resilient Distributed Datasets, DataFrames, and datasets. We will also cover a crash course in the Scala programming language that will help you with the course. You will learn how to develop and run Spark jobs quickly using Scala, IntelliJ, and SBT. You will learn how to translate complex analysis problems into iterative or multi-stage Spark scripts. You will learn how to scale up to larger datasets using Amazon's Elastic MapReduce service and understand how Hadoop YARN distributes Spark across computing clusters. We will also be practicing using other Spark technologies, such as Spark SQL, DataFrames, DataSets, Spark Streaming, Machine Learning, and GraphX. By the end of this course, you will be running code that analyzes gigabytes worth of information-in the cloud-in a matter of minutes. All the codes and supporting files for this course are available at https://github.com/PacktPublishing/Apache-Spark-with-Scala---Hands-On-with-Big-Data-
What You Will Learn
Learn the concepts of Spark's RDD, DataFrames, and Datasets
Get a crash course in the Scala programming language
Develop and run Spark jobs quickly using Scala, IntelliJ, and SBT
Translate complex analysis problems into iterative or multi-stage Spark scripts
Scale up to larger datasets using Amazon's Elastic MapReduce service
Understand how Hadoop YARN distributes Spark across computing clusters
Audience
This course is designed for software engineers who want to expand their skills into the world of big data processing on a cluster. It is necessary to have some prior programming or scripting knowledge.
Approach
This course is very hands-on; you will spend most of your time following along with the instructor as we write, analyze, and run real code together-both on your own system and in the cloud using Amazon's Elastic MapReduce service. Over eight hours of video content is included, with over 20 real examples of increasing complexity that you can build, run, and study yourself. Move through them at your own pace, on your own schedule.
Key Features
Understand the fundamentals of Scala and the Apache Spark ecosystem * Develop distributed code using the Scala programming language * Include practical examples to help you develop real-world Big Data applications with Spark with Scala
Github Repo
https://github.com/PacktPublishing/Apache-Spark-with-Scala---Hands-On-with-Big-Data-
About the Author
Frank Kane
Frank Kane has spent nine years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers all the time. He holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology and teaches others about big data analysis.
Course Outline
1. Getting Started
This section introduces the course, and you will get the development environment for Spark and Scala and create a histogram of real movie ratings with Spark.
1. Introduction and Installing the Course Materials, IntelliJ, and Scala
A brief introduction to the course, and then we will get your development environment for Spark and Scala all set up on your desktop, using IntelliJ and SBT. A quick test application will confirm that Spark is working on your system!
2. Introduction to Apache Spark
This video is a brief introduction to Apache Spark.
2. Scala Crash Course (Optional)
This section is about Scala. You will learn the basics, flow control, functions, and data structures in Scala.
1. (Activity) Scala Basics
We will go over the basic syntax and structure of Scala code with lots of examples. It's backward from most other languages, but you quickly get used to it.
2. (Exercise) Flow Control in Scala
We will go over the basic syntax and structure of Scala code with lots of examples. It's backward from most other languages, but you quickly get used to it. In this video, you will get some hands-on practice at the end.
3. (Exercise) Functions in Scala
Scala is a functional programming language and so, functions are central to the language. We will go over the many ways functions can be declared and used in Scala, and practice what you have learned.
4. (Exercise) Data Structures in Scala
We will cover the common data structures in Scala such as Map and List and put them into practice.
3. Using Resilient Distributed Datasets (RDDs)
In this section, you will learn how to use Spark RDD.
1. The Resilient Distributed Dataset
The core object of Spark programming is the Resilient Distributed Dataset, or RDD. Once you know how to use RDDs, you know how to use Spark. We will go over what they are and what you can do with them.
2. Ratings Histogram Example
Now that we understand Scala and have the theory of Spark behind us, let's start with a simple example of using RDDs to count up how many of each rating exists in the MovieLens dataset.
3. Spark Internals
How does Spark convert your script into a Directed Acyclic Graph and figure out how to distribute it on a cluster? Understanding how this process works under the hood can be important in writing optimal Spark driver scripts.
4. Key / Value RDDs, and the Average Friends by Age Example
RDDs that contain a tuple of two values are key/value RDDs, and you can use them much like you might use a NoSQL data store. We will use key/value RDDs to figure out the average number of friends by age in some fake social network data.
5. (Activity) Running the Average Friends by Age Example
We will run the average friends by age example on your desktop and give you some ideas for further extending this script on your own.
6. Filtering RDDs, and the Minimum Temperature by Location Example
We will cover how to filter data out of an RDD efficiently and illustrate this with a new example that finds the minimum temperature by location using real weather data.
7. (Activity) Running the Minimum Temperature Example, and Modifying It for Maximum
We will run our minimum temperature by location example and modify it to find maximum temperatures as well. Plus, some ideas for extending this script on your own.
8. (Activity) Counting Word Occurrences Using Flatmap()
flatmap() on an RDD can return variable amounts of new entries into the resulting RDD. We will use this as part of a hands-on example that finds how often each word is used inside a real book's text.
9. (Activity) Improving the Word Count Script with Regular Expressions
We extend the previous lecture's example using regular expressions to better extract words from our book.
10. (Activity) Sorting the Word Count Results
Finally, we sort the final results to see what the most common words in this book really are! And some ideas to extend this script on your own.
11. (Exercise) Find the Total Amount Spent by Customer
Your assignment is to write a script that finds the total amount spent per customer using some fabricated ecommerce data, using what you have learned so far.
12. (Exercise) Check Your Results and Sort Them by Total Amount Spent
We will review my solution to the previous lecture's assignment and challenge you further to sort your results to find the biggest spenders.
13. Check Your Results and Implementation Against Mine
Check your results for finding the biggest spenders in our ecommerce data against my own solution.
4. SparkSQL, DataFrames, and DataSets
This section is about SparkSQL, DataFrames, and DataSets.
1. Introduction to SparkSQL
Understand SparkSQL and the DataFrame and DataSet APIs used for querying structured data in an efficient, scalable manner.
2. (Activity) Using SparkSQL
We will revisit our fabricated social network data but load it into a DataFrame and analyze it with actual SQL queries!
3. (Activity) Using DataSets
We will analyze our social network data another way - this time using SQL-like functions on a DataSet, instead of actual SQL query strings.
4. (Exercise) Implement the Friends by Age Example Using DataSets
Earlier, we broke down the average number of friends by age using RDDs; see if you can do it using DataSets instead!
5. Exercise Solution: Friends by Age, with DataSets
Let's check the solution for the Friends by Age exercise using DataSets.
6. (Activity) Word Count Example Using DataSets
You will learn how to implement the word count activity with DataSets using SQL functions.
7. (Activity) Revisiting the Minimum Temperature Example, with DataSets
You will learn how to implement the weather data activity with DataSets using withColumn() functions.
8. (Exercise) Implement the Total Spent by Customer Problem with DataSets
We will have a look at the total spent by customer problem activity with DataSets.
9. Exercise Solution: Total Spent by Customer with DataSets
We will have a look at the possible solution for the total spent by customer problem activity with DataSets.
5. Advanced Examples of Spark Programs
In this section, we will be working on some advanced examples of Spark programming.
1. (Activity) Find the Most Popular Movie
We will revisit our movie ratings dataset and start off with a simple example to find the most-rated movie.
2. (Activity) Use Broadcast Variables to Display Movie Names
Broadcast variables can be used to share small amounts of data to all of the machines on your cluster. We will use them to share a lookup table of movie IDs to movie names and use that to get movie names in our final results.
3. (Activity) Find the Most Popular Superhero in a Social Graph
We introduce the Marvel superhero social network dataset and write a script to find the most-connected superhero in it. It's not who you might think!
4. (Exercise) Find the Most Obscure Superheroes
This exercise is all about finding the most obscure superheroes from the MostPopularSuperheroDataset.
5. Exercise Solution: Find the Most Obscure Superheroes
We will work on the possible solution to find the most obscure superheroes.
6. Superhero Degrees of Separation: Introducing Breadth-First Search
As a more complex example, we will apply a breadth-first-search (BFS) algorithm to the Marvel dataset to compute the degrees of separation between any two superheroes. In this lecture, we go over how BFS works.
7. Superhero Degrees of Separation: Accumulators, and Implementing BFS in Spark
We will go over our strategy for implementing BFS within a Spark script that can be distributed and introduce the use of Accumulators to maintain running totals that are synced across a cluster.
8. (Activity) Superhero Degrees of Separation: Review the Code and Run It!
Finally, we will review the code for finding the degrees of separation using breadth-first search, run it, and see the results!
9. Item-Based Collaborative Filtering in Spark, cache(), and persist()
Back to our movie ratings data-we will discover movies that are similar to each other just based on user ratings. We will cover the algorithm and how to implement it as a Spark script.
10. (Activity) Running the Similar Movies Script Using Spark's Cluster Manager
We will run our movie similarities script and see the results. In doing so, we will introduce the process of exporting your Spark script as a JAR file that can be run from the command line using the spark-submit script (instead of running from within the Scala IDE).
11. (Exercise) Improve the Quality of Similar Movies
Your challenge is to make the movie similarity results even better! Here are some ideas for you to try out.
6. Running Spark on a Cluster
This section is about running Spark on a cluster.
1. (Activity) Using spark-submit to Run Spark Driver Scripts
In a production environment, you will use spark-submit to start your driver scripts from a command line, cron job, or the like. We will cover the details on what you need to do differently in this case.
2. (Activity) Packaging Driver Scripts with SBT
Spark / Scala scripts that have external dependencies can be bundled up into self-contained packages using the SBT tool. We will use SBT to package up our movie similarities script as an exercise.
3. (Exercise) Package a Script with SBT and Run It Locally with spark-submit
We will work on an exercise where we will package a script with SBT and run it locally with spark-submit.
4. Exercise Solution: Using SBT and spark-submit
We will have a look at the possible solution for this exercise.
5. Introducing Amazon Elastic MapReduce
Amazon Web Services (AWS) offers the Elastic MapReduce service (EMR), which gives us a way to rent time on a Hadoop cluster of our choosing-with Spark pre-installed on it. We will use EMR to illustrate running a Spark script on a real cluster, so let's go over what EMR is and how it works first.
6. Creating Similar Movies from One Million Ratings on EMR
Let's compute movie similarities on a real cluster in the cloud, using one million user ratings!
7. Partitioning
Explicitly partitioning your datasets and RDDs can be an important optimization; we will go over when and how to do this.
8. Best Practices for Running on a Cluster
Other tips and tricks for taking your script to a real cluster and getting it to run as you expect.
9. Troubleshooting and Managing Dependencies
This video is on how to troubleshoot Spark jobs on a cluster using the Spark UI and logs, and more on managing dependencies of your script and data.
7. Machine Learning with Spark ML
This section is about machine learning with MLLib.
1. Introducing MLLib
MLLib offers several distributed machine learning algorithms that you can run on a Spark cluster. We will cover what MLLib can do and how it fits in.
2. (Activity) Using MLLib to Produce Movie Recommendations
We will use MLLib's Alternating Least Squares recommender algorithm to produce movie recommendations using our MovieLens ratings data. The results areunexpected!
3. Linear Regression with MLLib
A brief overview of what linear regression is and how it works, followed by a hands-on example of finding a regression and applying it to fabricated page speed versus revenue data.
4. (Activity) Running a Linear Regression with Spark
We will run our Spark ML example of linear regression using DataFrames.
5. (Exercise) Predict Real Estate Values with Decision Trees in Spark
This exercise is all about predicting real estate values with decision trees in Spark.
6. Exercise Solution: Predicting Real Estate with Decision Trees in Spark
We will have a look at the possible solution for this exercise.
8. Introduction to Spark Streaming
This section is about Spark Streaming.
1. The DStream API for Spark Streaming
Spark Streaming allows you to create Spark driver scripts that run indefinitely, continually processing data as it streams in! We will cover how it works and what it can do, using the original DStream micro-batch API.
2. (Activity) Real-Time Monitoring of the Most Popular Hashtags on Twitter
You will learn how to do real-time monitoring of the most popular hashtags on Twitter.
3. Structured Streaming
Structured Streaming is a newer DataFrame-based API in Spark for writing continuous applications.
4. (Activity) Using Structured Streaming for Real-Time Log Analysis
You will learn how to use Structured Streaming for real-time log analysis.
5. (Exercise) Windowed Operations with Structured Streaming
We will work on an exercise to search the top URLs that are streamed in the past 30 seconds.
6. Exercise Solution: Top URLs in a 30-Second Window
We will look at the possible solution for this exercise.
9. Introduction to GraphX
This section is about GraphX.
1. GraphX, Pregel, and Breadth-First Search with Pregel
We will cover Spark's GraphX library and how it works.
2. Using the Pregel API with Spark GraphX
We will revisit our "superhero degrees of separation" example and see how its breadth-first search algorithm could be implemented using Pregel and GraphX.
3. (Activity) Superhero Degrees of Separation Using GraphX
We will use GraphX and Pregel to recreate our earlier results analyzing the superhero social network data-but with a lot less code!
10. You Made It! Where to Go from Here
This section is about learning more and career tips.
1. Learning More, and Career Tips
You made it to the end! Here are some book recommendations if you want to learn more, as well as some career advice on landing a job in "big data".

Data Processing

This is a comprehensive and practical Apache Spark course. In this course, you will learn and master the art of framing data analysis problems as Spark problems through 20+ hands-on examples, and then scale them up to run on cloud computing services. Explore Spark 3, IntelliJ, Structured Streaming, and a stronger focus on the DataSet API.

Apache Spark with Scala - Hands-On with Big Data!

<strong>This course does not require any prior knowledge of Apache Spark or Hadoop. The author explains Spark architecture and fundamental concepts to help you come up to speed and grasp the content of this course.

The course will help you understand Spark programming and apply that knowledge to build data engineering solutions.</strong>
        <p>Apache Spark is a lightning-fast unified analytics engine for big data and machine learning. Since its release, Apache Spark has seen rapid adoption by enterprises across a wide range of industries. Internet powerhouses such as Netflix, Yahoo, and eBay have deployed Spark at a massive scale. It has quickly become the largest open-source community in big data. So, mastering Apache Spark opens a wide range of professional opportunities.

This course starts with an introduction to Apache Spark where you see what Apache Spark is in brief. Then, you will be installing and using Apache Spark. After that, you will look at the Spark execution model and architecture in detail. Next, you will learn the Spark programming model and developer experience. Following that, you will look at the Spark Structured API foundation, and Spark data sources and sinks. Then, you will explore Spark Data frame and dataset transformations along with aggregations in Apache Spark. Finally, you will look at the Spark Data frame joins in detail.

By the end of this course, you will understand Spark programming and apply that knowledge to build data engineering solutions.

All the resource files are uploaded on the GitHub repository at https://github.com/PacktPublishing/Spark-Programming-in-Scala-for-Beginners-with-Apache-Spark-3</p>
        <h2>What You Will Learn</h2><p>Learn Apache Spark and Spark architecture<br>Look at data engineering and data processing in Spark<br>Work with data sources and sinks<br>Work with data frames, datasets, and Spark SQL<br>Use IntelliJ IDEA for Spark development and debugging<br>Understand unit testing, managing application logs, and cluster deployment</p>
        <h2>Audience</h2><p>This course is designed for software engineers willing to develop a data engineering pipeline and application using Apache Spark. It is also for data architects and data engineers who are responsible for designing and building the organization's data-centric infrastructure. It will also be beneficial for the managers and architects who do not directly work with Spark implementation, and still, they work with the people who implement Apache Spark at the ground level.<br><br>Before proceeding with the course, you will need basic knowledge of the Scala programming language.</p>
        <h2>Approach</h2><p>This is a hands-on and well-balanced course with theoretical and practical content. The course is example-driven and follows a working session-like approach. The author will be taking a live coding approach and explaining all the concepts needed along the way. The section will have a summary video at the end to revise the learned concepts.</p>
        <h2>Key Features</h2><p>A comprehensive course designed for the beginner-level for Spark programming in Scala * Deep dive into Spark 3 architecture and data engineering * Complete tested source code and examples used on Apache Spark 3.0.0 open-source distribution from the author's end</p>
        <h2>Github Repo</h2><p>https://github.com/PacktPublishing/Spark-Programming-in-Scala-for-Beginners-with-Apache-Spark-3</p>
        <h2>About the Author</h2><h3>Scholar Nest</h3><p>ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills.

Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects.

It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.</p>
        <h2>Course Outline</h2><h4>1. Apache Spark Introduction</h4><p></p><br><h4>2. Installing and Using Apache Spark</h4><p></p><br><h4>3. Spark Execution Model and Architecture</h4><p></p><br><h4>4. Spark Programming Model and Developer Experience</h4><p></p><br><h4>5. Spark Structured API Foundation</h4><p></p><br><h4>6. Spark Data Sources and Sinks</h4><p></p><br><h4>7. Spark Dataframe and Dataset Transformations</h4><p></p><br><h4>8. Aggregations in Apache Spark</h4><p></p><br><h4>9. Spark Dataframe Joins</h4><p></p><br><h4>10. Keep Learning</h4><p></p>

This course does not require any prior knowledge of Apache Spark or Hadoop. The author explains Spark architecture and fundamental concepts to help you come up to speed and grasp the content of this course. The course will help you understand Spark programming and apply that knowledge to build data engineering solutions.
Apache Spark is a lightning-fast unified analytics engine for big data and machine learning. Since its release, Apache Spark has seen rapid adoption by enterprises across a wide range of industries. Internet powerhouses such as Netflix, Yahoo, and eBay have deployed Spark at a massive scale. It has quickly become the largest open-source community in big data. So, mastering Apache Spark opens a wide range of professional opportunities. This course starts with an introduction to Apache Spark where you see what Apache Spark is in brief. Then, you will be installing and using Apache Spark. After that, you will look at the Spark execution model and architecture in detail. Next, you will learn the Spark programming model and developer experience. Following that, you will look at the Spark Structured API foundation, and Spark data sources and sinks. Then, you will explore Spark Data frame and dataset transformations along with aggregations in Apache Spark. Finally, you will look at the Spark Data frame joins in detail. By the end of this course, you will understand Spark programming and apply that knowledge to build data engineering solutions. All the resource files are uploaded on the GitHub repository at https://github.com/PacktPublishing/Spark-Programming-in-Scala-for-Beginners-with-Apache-Spark-3
What You Will Learn
Learn Apache Spark and Spark architecture
Look at data engineering and data processing in Spark
Work with data sources and sinks
Work with data frames, datasets, and Spark SQL
Use IntelliJ IDEA for Spark development and debugging
Understand unit testing, managing application logs, and cluster deployment
Audience
This course is designed for software engineers willing to develop a data engineering pipeline and application using Apache Spark. It is also for data architects and data engineers who are responsible for designing and building the organization's data-centric infrastructure. It will also be beneficial for the managers and architects who do not directly work with Spark implementation, and still, they work with the people who implement Apache Spark at the ground level.

Before proceeding with the course, you will need basic knowledge of the Scala programming language.
Approach
This is a hands-on and well-balanced course with theoretical and practical content. The course is example-driven and follows a working session-like approach. The author will be taking a live coding approach and explaining all the concepts needed along the way. The section will have a summary video at the end to revise the learned concepts.
Key Features
A comprehensive course designed for the beginner-level for Spark programming in Scala * Deep dive into Spark 3 architecture and data engineering * Complete tested source code and examples used on Apache Spark 3.0.0 open-source distribution from the author's end
Github Repo
https://github.com/PacktPublishing/Spark-Programming-in-Scala-for-Beginners-with-Apache-Spark-3
About the Author
Scholar Nest
ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects. It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.
Course Outline
1. Apache Spark Introduction

2. Installing and Using Apache Spark

3. Spark Execution Model and Architecture

4. Spark Programming Model and Developer Experience

5. Spark Structured API Foundation

6. Spark Data Sources and Sinks

7. Spark Dataframe and Dataset Transformations

8. Aggregations in Apache Spark

9. Spark Dataframe Joins

10. Keep Learning


Hadoop

This course does not require any prior knowledge of Apache Spark or Hadoop. The author explains Spark architecture and fundamental concepts to help you come up to speed and grasp the content of this course.

The course will help you understand Spark programming and apply that knowledge to build data engineering solutions.

Spark Programming in Scala for Beginners with Apache Spark 3

<strong>Learn the process to design and develop big data engineering projects using Apache Spark. This example-driven advanced-level course will help you understand real-time stream processing using Apache Spark and you can apply that knowledge to build real-time stream processing solutions.</strong>
        <p>Since its inception, Apache Spark has seen rapid adoption by enterprises across a wide range of industries. So, mastering Apache Spark opens a wide range of professional opportunities. If you are a software engineer or architect and want to design or build your own projects, then this is the right course for you.

This is a hands-on, example-driven, advanced course with demonstrations and coding sessions. This course will help you understand real-time stream processing using Apache Spark and later, you will be able to apply that knowledge to build real-time stream processing solutions.

This course covers everything from scratch, which involves installing Apache Spark and seeing how to set up and run Apache Kafka. Furthermore, it introduces stream processing and how to work with files and directories. You will also explore Kafka serialization and deserialization for Spark and how to work with Kafka AVRO Source.  And finally, the course wraps up with streaming Watermark and outer joints.

By the end of this course, you will be able to design and develop big data engineering projects. You will be able to create real-time stream processing applications with Apache Spark. This course will also help you further your growth in real-time stream processing.

All resources and code files are placed here: https://github.com/PacktPublishing/Spark-Streaming-In-Scala</p>
        <h2>What You Will Learn</h2><p>Create arbitrary streaming sinks<br>Explore Kafka Source and integrate Spark with Kafka<br>Learn state-less and state-full streaming transformations<br>Learn to handle memory problems with streaming joins<br>Learn to work with file streams<br>Explore windowing aggregates using Spark Stream</p>
        <h2>Audience</h2><p>This course is designed for software engineers and architects who aspire to develop big data engineering projects using Apache Spark. Also, if you are a programmer and developer who wants to grow and learn data engineering using Apache Spark, then this course is for you. Another group of people that can opt for this course are the managers and architects who might not directly work with Spark implementation but still work with the people who implement Apache Spark at the ground level.</p>
        <h2>Approach</h2><p>This is a comprehensive, hands-on, example-driven course, and it follows a working session-like approach. There is a combination of both theory and demonstration. The theory part explains all the concepts needed and then demonstrations to help you master real-time stream processing. It provides real-world examples to give you practical learning experience.</p>
        <h2>Key Features</h2><p>Deep dive into Spark structured streaming APIs and architecture * Discover streaming joins and aggregation * Explore real-time stream processing concepts</p>
        <h2>Github Repo</h2><p>https://github.com/PacktPublishing/Spark-Streaming-In-Scala</p>
        <h2>About the Author</h2><h3>Scholar Nest</h3><p>ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills.

Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects.

It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.</p>
        <h2>Course Outline</h2><h4>1. Before you start</h4><p></p><br><h4>2. Setup your Environment</h4><p></p><br><h4>3. Getting started with Spark Structured Streaming</h4><p></p><br><h4>4. Spark Streaming with Kafka</h4><p></p><br><h4>5. Windowing and Aggregates</h4><p></p><br><h4>6. Stream Processing and Joins</h4><p></p><br><h4>7. Keep Learning</h4><p></p>

Learn the process to design and develop big data engineering projects using Apache Spark. This example-driven advanced-level course will help you understand real-time stream processing using Apache Spark and you can apply that knowledge to build real-time stream processing solutions.
Since its inception, Apache Spark has seen rapid adoption by enterprises across a wide range of industries. So, mastering Apache Spark opens a wide range of professional opportunities. If you are a software engineer or architect and want to design or build your own projects, then this is the right course for you. This is a hands-on, example-driven, advanced course with demonstrations and coding sessions. This course will help you understand real-time stream processing using Apache Spark and later, you will be able to apply that knowledge to build real-time stream processing solutions. This course covers everything from scratch, which involves installing Apache Spark and seeing how to set up and run Apache Kafka. Furthermore, it introduces stream processing and how to work with files and directories. You will also explore Kafka serialization and deserialization for Spark and how to work with Kafka AVRO Source. And finally, the course wraps up with streaming Watermark and outer joints. By the end of this course, you will be able to design and develop big data engineering projects. You will be able to create real-time stream processing applications with Apache Spark. This course will also help you further your growth in real-time stream processing. All resources and code files are placed here: https://github.com/PacktPublishing/Spark-Streaming-In-Scala
What You Will Learn
Create arbitrary streaming sinks
Explore Kafka Source and integrate Spark with Kafka
Learn state-less and state-full streaming transformations
Learn to handle memory problems with streaming joins
Learn to work with file streams
Explore windowing aggregates using Spark Stream
Audience
This course is designed for software engineers and architects who aspire to develop big data engineering projects using Apache Spark. Also, if you are a programmer and developer who wants to grow and learn data engineering using Apache Spark, then this course is for you. Another group of people that can opt for this course are the managers and architects who might not directly work with Spark implementation but still work with the people who implement Apache Spark at the ground level.
Approach
This is a comprehensive, hands-on, example-driven course, and it follows a working session-like approach. There is a combination of both theory and demonstration. The theory part explains all the concepts needed and then demonstrations to help you master real-time stream processing. It provides real-world examples to give you practical learning experience.
Key Features
Deep dive into Spark structured streaming APIs and architecture * Discover streaming joins and aggregation * Explore real-time stream processing concepts
Github Repo
https://github.com/PacktPublishing/Spark-Streaming-In-Scala
About the Author
Scholar Nest
ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects. It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.
Course Outline
1. Before you start

2. Setup your Environment

3. Getting started with Spark Structured Streaming

4. Spark Streaming with Kafka

5. Windowing and Aggregates

6. Stream Processing and Joins

7. Keep Learning


Kafka

Learn the process to design and develop big data engineering projects using Apache Spark. This example-driven advanced-level course will help you understand real-time stream processing using Apache Spark and you can apply that knowledge to build real-time stream processing solutions.

Real-Time Stream Processing Using Apache Spark 3 for Scala Developers

<strong>A complete course on Sqoop, Flume, and Hive: Ideal for achieving CCA175 and Hortonworks Spark Certification</strong>
        <p>In this course, you will start by learning about the Hadoop Distributed File System (HDFS) and the most common Hadoop commands required to work with HDFS. Next, you'll be introduced to Sqoop Import, which will help you gain insights into the lifecycle of the Sqoop command and how to use the import command to migrate data from MySQL to HDFS, and from MySQL to Hive. 

In addition to this, you will get up to speed with Sqoop Export for migrating data effectively, along with using Apache Flume to ingest data. As you progress, you will delve into Apache Hive, external and managed tables, working with different files, and Parquet and Avro. Toward the concluding section, you will focus on Spark DataFrames and Spark SQL. 

By the end of this course, you will have gained comprehensive insights into big data ingestion and analytics with Flume, Sqoop, Hive, and Spark. 

All code and supporting files are available at - https://github.com/PacktPublishing/Master-Big-Data-Ingestion-and-Analytics-with-Flume-Sqoop-Hive-and-Spark</p>
        <h2>What You Will Learn</h2><p>Explore the Hadoop Distributed File System (HDFS) and commands <br>Get to grips with the lifecycle of the Sqoop command <br>Use the Sqoop Import command to migrate data from MySQL to HDFS and Hive <br>Understand split-by and boundary queries <br>Use the incremental mode to migrate data from MySQL to HDFS <br>Employ Sqoop Export to migrate data from HDFS to MySQL <br>Discover Spark DataFrames  and gain insights into  working with different file formats and compression</p>
        <h2>Audience</h2><p>This course is for anyone who wants to learn Sqoop and Flume or those looking to achieve CCA and HDP certification.</p>
        <h2>Approach</h2><p>A complete course packed with step-by-step instructions, working examples, and helpful advice. This course is systematically divided into small sections that will help you understand each part individually and learn at your own pace.</p>
        <h2>Key Features</h2><p>Learn Sqoop, Flume, and Hive and successfully achieve CCA175 and Hortonworks Spark Certification  * Understand the Hadoop Distributed File System (HDFS), along with exploring Hadoop commands to work effectively with HDFS</p>
        <h2>Github Repo</h2><p>https://github.com/packtpublishing/master-big-data-ingestion-and-analytics-with-flume-sqoop-hive-and-spark</p>
        <h2>About the Author</h2><h3>Navdeep Kaur</h3><p>Navdeep Kaur - Technical Trainer
Navdeep Kaur is a big data professionals with 11 years of industry experience in different technologies and domains. She has a keen interest in providing training in new technologies. She has received CCA175 Hadoop and Spark developer certification and AWS solution architect certification. She loves guiding people and helping them achieves new goals.</p>
        <h2>Course Outline</h2><h4>1. Hadoop Introduction</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. HDFS and Hadoop Commands</p><p style='margin: 0'>Hadoop Introduction: HDFS and Hadoop Commands</p></td></tr></tbody></table><br><h4>2. Sqoop Import</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Sqoop Introduction</p><p style='margin: 0'>Sqoop Import: Sqoop Introduction</p></td></tr><tr><td style='padding: 15px;'><p>2. Managing Target Directories</p><p style='margin: 0'>Sqoop Import: Managing Target Directories</p></td></tr><tr><td style='padding: 15px;'><p>3. Working with Different File Formats</p><p style='margin: 0'>Sqoop Import: Working with Different File Formats</p></td></tr><tr><td style='padding: 15px;'><p>4. Working with Different Compressions</p><p style='margin: 0'>Sqoop Import: Working with Different Compressions</p></td></tr><tr><td style='padding: 15px;'><p>5. Conditional Imports</p><p style='margin: 0'>Sqoop Import: Conditional Imports</p></td></tr><tr><td style='padding: 15px;'><p>6. Split-by and Boundary Queries</p><p style='margin: 0'>Sqoop Import: Split-by and Boundary Queries</p></td></tr><tr><td style='padding: 15px;'><p>7. Field delimeters</p><p style='margin: 0'>Sqoop Import: Field delimeters</p></td></tr><tr><td style='padding: 15px;'><p>8. Incremental Appends</p><p style='margin: 0'>Sqoop Import: Incremental Appends</p></td></tr><tr><td style='padding: 15px;'><p>9. Sqoop Hive Import</p><p style='margin: 0'>Sqoop Import: Sqoop Hive Import</p></td></tr><tr><td style='padding: 15px;'><p>10. Sqoop List Tables/Database</p><p style='margin: 0'>Sqoop Import: Sqoop List Tables/Database</p></td></tr><tr><td style='padding: 15px;'><p>11. Sqoop Import Practice1</p><p style='margin: 0'>Sqoop Import: Sqoop Import Practice1</p></td></tr><tr><td style='padding: 15px;'><p>12. Sqoop Import Practice2</p><p style='margin: 0'>Sqoop Import: Sqoop Import Practice2</p></td></tr><tr><td style='padding: 15px;'><p>13. Sqoop Import Practice3</p><p style='margin: 0'>Sqoop Import: Sqoop Import Practice3</p></td></tr></tbody></table><br><h4>3. Sqoop Export</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Export from Hdfs to Mysql</p><p style='margin: 0'>Sqoop Export: Export from Hdfs to Mysql</p></td></tr><tr><td style='padding: 15px;'><p>2. Export from Hive to Mysql</p><p style='margin: 0'>Sqoop Export: Export from Hive to Mysql</p></td></tr></tbody></table><br><h4>4. Apache Flume</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Flume Introduction & Architecture</p><p style='margin: 0'>Apache Flume: Flume Introduction & Architecture</p></td></tr><tr><td style='padding: 15px;'><p>2. Exec Source and Logger Sink</p><p style='margin: 0'>Apache Flume: Exec Source and Logger Sink</p></td></tr><tr><td style='padding: 15px;'><p>3. Moving data from Twitter to HDFS</p><p style='margin: 0'>Apache Flume: Moving data from Twitter to HDFS</p></td></tr><tr><td style='padding: 15px;'><p>4. Moving data from NetCat to HDFS</p><p style='margin: 0'>Apache Flume: Moving data from NetCat to HDFS</p></td></tr><tr><td style='padding: 15px;'><p>5. Flume Interceptors</p><p style='margin: 0'>Apache Flume: Flume Interceptors</p></td></tr><tr><td style='padding: 15px;'><p>6. Flume Interceptor Example</p><p style='margin: 0'>Apache Flume: Flume Interceptor Example</p></td></tr><tr><td style='padding: 15px;'><p>7. Flume Multi-Agent Flow</p><p style='margin: 0'>Apache Flume: Flume Multi-Agent Flow</p></td></tr><tr><td style='padding: 15px;'><p>8. Flume Consolidation</p><p style='margin: 0'>Apache Flume: Flume Consolidation</p></td></tr></tbody></table><br><h4>5. Apache Hive</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Hive Introduction</p><p style='margin: 0'>Apache Hive: Hive Introduction</p></td></tr><tr><td style='padding: 15px;'><p>2. Hive Database</p><p style='margin: 0'>Apache Hive: Hive Database</p></td></tr><tr><td style='padding: 15px;'><p>3. Hive Managed Tables</p><p style='margin: 0'>Apache Hive: Hive Managed Tables</p></td></tr><tr><td style='padding: 15px;'><p>4. Hive External Tables</p><p style='margin: 0'>Apache Hive: Hive External Tables</p></td></tr><tr><td style='padding: 15px;'><p>5. Hive Inserts</p><p style='margin: 0'>Apache Hive: Hive Inserts</p></td></tr><tr><td style='padding: 15px;'><p>6. Hive Analytics</p><p style='margin: 0'>Apache Hive: Hive Analytics</p></td></tr><tr><td style='padding: 15px;'><p>7. Working with Parquet</p><p style='margin: 0'>Apache Hive: Working with Parquet</p></td></tr><tr><td style='padding: 15px;'><p>8. Compressing Parquet</p><p style='margin: 0'>Apache Hive: Compressing Parquet</p></td></tr><tr><td style='padding: 15px;'><p>9. Working with Fixed File Format</p><p style='margin: 0'>Apache Hive: Working with Fixed File Format</p></td></tr><tr><td style='padding: 15px;'><p>10. Alter Command</p><p style='margin: 0'>Apache Hive: Alter Command</p></td></tr><tr><td style='padding: 15px;'><p>11. Hive String Functions</p><p style='margin: 0'>Apache Hive: Hive String Functions</p></td></tr><tr><td style='padding: 15px;'><p>12. Hive Date Functions</p><p style='margin: 0'>Apache Hive: Hive Date Functions</p></td></tr><tr><td style='padding: 15px;'><p>13. Hive Partitioning</p><p style='margin: 0'>Apache Hive: Hive Partitioning</p></td></tr><tr><td style='padding: 15px;'><p>14. Hive Bucketing</p><p style='margin: 0'>Apache Hive: Hive Bucketing</p></td></tr></tbody></table><br><h4>6. Spark Introduction</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Spark Introduction</p><p style='margin: 0'>Spark Introduction: Spark Introduction</p></td></tr><tr><td style='padding: 15px;'><p>2. Resilient Distributed Datasets</p><p style='margin: 0'>Spark Introduction: Resilient Distributed Datasets</p></td></tr><tr><td style='padding: 15px;'><p>3. Cluster Overview</p><p style='margin: 0'>Spark Introduction: Cluster Overview</p></td></tr><tr><td style='padding: 15px;'><p>4. Directed Acyclic Graph (DAG) & Stages</p><p style='margin: 0'>Spark Introduction: Directed Acyclic Graph (DAG) & Stages</p></td></tr></tbody></table><br><h4>7. Spark Transformations & Actions</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Map/FlatMap Transformation</p><p style='margin: 0'>Spark Transformations & Actions: Map/FlatMap Transformation</p></td></tr><tr><td style='padding: 15px;'><p>2. Filter/Intersection</p><p style='margin: 0'>Spark Transformations & Actions: Filter/Intersection</p></td></tr><tr><td style='padding: 15px;'><p>3. Union/Distinct Transformation</p><p style='margin: 0'>Spark Transformations & Actions: Union/Distinct Transformation</p></td></tr><tr><td style='padding: 15px;'><p>4. GroupByKey/ Group people based on Birthday months</p><p style='margin: 0'>Spark Transformations & Actions: GroupByKey/ Group people based on Birthday months</p></td></tr><tr><td style='padding: 15px;'><p>5. ReduceByKey / Total Number of students in each Subject</p><p style='margin: 0'>Spark Transformations & Actions: ReduceByKey / Total Number of students in each Subject</p></td></tr><tr><td style='padding: 15px;'><p>6. SortByKey / Sort students based on their rollno</p><p style='margin: 0'>Spark Transformations & Actions: SortByKey / Sort students based on their rollno</p></td></tr><tr><td style='padding: 15px;'><p>7. MapPartition / MapPartitionWithIndex</p><p style='margin: 0'>Spark Transformations & Actions: MapPartition / MapPartitionWithIndex</p></td></tr><tr><td style='padding: 15px;'><p>8. Change number of Partitions</p><p style='margin: 0'>Spark Transformations & Actions: Change number of Partitions</p></td></tr><tr><td style='padding: 15px;'><p>9. Join / Join email address based on customer name</p><p style='margin: 0'>Spark Transformations & Actions: Join / Join email address based on customer name</p></td></tr><tr><td style='padding: 15px;'><p>10. Spark Actions</p><p style='margin: 0'>Spark Transformations & Actions: Spark Actions</p></td></tr></tbody></table><br><h4>8. Spark RDD Practice</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Scala Tuples</p><p style='margin: 0'>Spark RDD Practice: Scala Tuples</p></td></tr><tr><td style='padding: 15px;'><p>2. Extract Error Logs from log files</p><p style='margin: 0'>Spark RDD Practice: Extract Error Logs from log files</p></td></tr><tr><td style='padding: 15px;'><p>3. Frequency of word in Text File</p><p style='margin: 0'>Spark RDD Practice: Frequency of word in Text File</p></td></tr><tr><td style='padding: 15px;'><p>4. Population of each City</p><p style='margin: 0'>Spark RDD Practice: Population of each City</p></td></tr><tr><td style='padding: 15px;'><p>5. Orders placed by Customers</p><p style='margin: 0'>Spark RDD Practice: Orders placed by Customers</p></td></tr><tr><td style='padding: 15px;'><p>6. Movie Average Rating greater than 3</p><p style='margin: 0'>Spark RDD Practice: Movie Average Rating greater than 3</p></td></tr></tbody></table><br><h4>9. Spark Dataframes & Spark SQL</h4><p></p><table width='100%'><tbody><tr><td style='padding: 15px;'><p>1. Dataframe Intro</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Dataframe Intro</p></td></tr><tr><td style='padding: 15px;'><p>2. Dafaframe from Json Files</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Dafaframe from Json Files</p></td></tr><tr><td style='padding: 15px;'><p>3. Dataframe from Parquet Files</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Dataframe from Parquet Files</p></td></tr><tr><td style='padding: 15px;'><p>4. Dataframe from CSV Files</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Dataframe from CSV Files</p></td></tr><tr><td style='padding: 15px;'><p>5. Dataframe from Avro/XML Files</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Dataframe from Avro/XML Files</p></td></tr><tr><td style='padding: 15px;'><p>6. Working with Different Compressions</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Working with Different Compressions</p></td></tr><tr><td style='padding: 15px;'><p>7. DataFrame API Part1</p><p style='margin: 0'>Spark Dataframes & Spark SQL: DataFrame API Part1</p></td></tr><tr><td style='padding: 15px;'><p>8. DataFrame API Part2</p><p style='margin: 0'>Spark Dataframes & Spark SQL: DataFrame API Part2</p></td></tr><tr><td style='padding: 15px;'><p>9. Spark SQL</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Spark SQL</p></td></tr><tr><td style='padding: 15px;'><p>10. Working with Hive Tables in Spark</p><p style='margin: 0'>Spark Dataframes & Spark SQL: Working with Hive Tables in Spark</p></td></tr></tbody></table>

A complete course on Sqoop, Flume, and Hive: Ideal for achieving CCA175 and Hortonworks Spark Certification
In this course, you will start by learning about the Hadoop Distributed File System (HDFS) and the most common Hadoop commands required to work with HDFS. Next, you'll be introduced to Sqoop Import, which will help you gain insights into the lifecycle of the Sqoop command and how to use the import command to migrate data from MySQL to HDFS, and from MySQL to Hive. In addition to this, you will get up to speed with Sqoop Export for migrating data effectively, along with using Apache Flume to ingest data. As you progress, you will delve into Apache Hive, external and managed tables, working with different files, and Parquet and Avro. Toward the concluding section, you will focus on Spark DataFrames and Spark SQL. By the end of this course, you will have gained comprehensive insights into big data ingestion and analytics with Flume, Sqoop, Hive, and Spark. All code and supporting files are available at - https://github.com/PacktPublishing/Master-Big-Data-Ingestion-and-Analytics-with-Flume-Sqoop-Hive-and-Spark
What You Will Learn
Explore the Hadoop Distributed File System (HDFS) and commands 
Get to grips with the lifecycle of the Sqoop command 
Use the Sqoop Import command to migrate data from MySQL to HDFS and Hive 
Understand split-by and boundary queries 
Use the incremental mode to migrate data from MySQL to HDFS 
Employ Sqoop Export to migrate data from HDFS to MySQL 
Discover Spark DataFrames and gain insights into working with different file formats and compression
Audience
This course is for anyone who wants to learn Sqoop and Flume or those looking to achieve CCA and HDP certification.
Approach
A complete course packed with step-by-step instructions, working examples, and helpful advice. This course is systematically divided into small sections that will help you understand each part individually and learn at your own pace.
Key Features
Learn Sqoop, Flume, and Hive and successfully achieve CCA175 and Hortonworks Spark Certification * Understand the Hadoop Distributed File System (HDFS), along with exploring Hadoop commands to work effectively with HDFS
Github Repo
https://github.com/packtpublishing/master-big-data-ingestion-and-analytics-with-flume-sqoop-hive-and-spark
About the Author
Navdeep Kaur
Navdeep Kaur - Technical Trainer
Navdeep Kaur is a big data professionals with 11 years of industry experience in different technologies and domains. She has a keen interest in providing training in new technologies. She has received CCA175 Hadoop and Spark developer certification and AWS solution architect certification. She loves guiding people and helping them achieves new goals.
Course Outline
1. Hadoop Introduction

1. HDFS and Hadoop Commands
Hadoop Introduction: HDFS and Hadoop Commands
2. Sqoop Import

1. Sqoop Introduction
Sqoop Import: Sqoop Introduction
2. Managing Target Directories
Sqoop Import: Managing Target Directories
3. Working with Different File Formats
Sqoop Import: Working with Different File Formats
4. Working with Different Compressions
Sqoop Import: Working with Different Compressions
5. Conditional Imports
Sqoop Import: Conditional Imports
6. Split-by and Boundary Queries
Sqoop Import: Split-by and Boundary Queries
7. Field delimeters
Sqoop Import: Field delimeters
8. Incremental Appends
Sqoop Import: Incremental Appends
9. Sqoop Hive Import
Sqoop Import: Sqoop Hive Import
10. Sqoop List Tables/Database
Sqoop Import: Sqoop List Tables/Database
11. Sqoop Import Practice1
Sqoop Import: Sqoop Import Practice1
12. Sqoop Import Practice2
Sqoop Import: Sqoop Import Practice2
13. Sqoop Import Practice3
Sqoop Import: Sqoop Import Practice3
3. Sqoop Export

1. Export from Hdfs to Mysql
Sqoop Export: Export from Hdfs to Mysql
2. Export from Hive to Mysql
Sqoop Export: Export from Hive to Mysql
4. Apache Flume

1. Flume Introduction & Architecture
Apache Flume: Flume Introduction & Architecture
2. Exec Source and Logger Sink
Apache Flume: Exec Source and Logger Sink
3. Moving data from Twitter to HDFS
Apache Flume: Moving data from Twitter to HDFS
4. Moving data from NetCat to HDFS
Apache Flume: Moving data from NetCat to HDFS
5. Flume Interceptors
Apache Flume: Flume Interceptors
6. Flume Interceptor Example
Apache Flume: Flume Interceptor Example
7. Flume Multi-Agent Flow
Apache Flume: Flume Multi-Agent Flow
8. Flume Consolidation
Apache Flume: Flume Consolidation
5. Apache Hive

1. Hive Introduction
Apache Hive: Hive Introduction
2. Hive Database
Apache Hive: Hive Database
3. Hive Managed Tables
Apache Hive: Hive Managed Tables
4. Hive External Tables
Apache Hive: Hive External Tables
5. Hive Inserts
Apache Hive: Hive Inserts
6. Hive Analytics
Apache Hive: Hive Analytics
7. Working with Parquet
Apache Hive: Working with Parquet
8. Compressing Parquet
Apache Hive: Compressing Parquet
9. Working with Fixed File Format
Apache Hive: Working with Fixed File Format
10. Alter Command
Apache Hive: Alter Command
11. Hive String Functions
Apache Hive: Hive String Functions
12. Hive Date Functions
Apache Hive: Hive Date Functions
13. Hive Partitioning
Apache Hive: Hive Partitioning
14. Hive Bucketing
Apache Hive: Hive Bucketing
6. Spark Introduction

1. Spark Introduction
Spark Introduction: Spark Introduction
2. Resilient Distributed Datasets
Spark Introduction: Resilient Distributed Datasets
3. Cluster Overview
Spark Introduction: Cluster Overview
4. Directed Acyclic Graph (DAG) & Stages
Spark Introduction: Directed Acyclic Graph (DAG) & Stages
7. Spark Transformations & Actions

1. Map/FlatMap Transformation
Spark Transformations & Actions: Map/FlatMap Transformation
2. Filter/Intersection
Spark Transformations & Actions: Filter/Intersection
3. Union/Distinct Transformation
Spark Transformations & Actions: Union/Distinct Transformation
4. GroupByKey/ Group people based on Birthday months
Spark Transformations & Actions: GroupByKey/ Group people based on Birthday months
5. ReduceByKey / Total Number of students in each Subject
Spark Transformations & Actions: ReduceByKey / Total Number of students in each Subject
6. SortByKey / Sort students based on their rollno
Spark Transformations & Actions: SortByKey / Sort students based on their rollno
7. MapPartition / MapPartitionWithIndex
Spark Transformations & Actions: MapPartition / MapPartitionWithIndex
8. Change number of Partitions
Spark Transformations & Actions: Change number of Partitions
9. Join / Join email address based on customer name
Spark Transformations & Actions: Join / Join email address based on customer name
10. Spark Actions
Spark Transformations & Actions: Spark Actions
8. Spark RDD Practice

1. Scala Tuples
Spark RDD Practice: Scala Tuples
2. Extract Error Logs from log files
Spark RDD Practice: Extract Error Logs from log files
3. Frequency of word in Text File
Spark RDD Practice: Frequency of word in Text File
4. Population of each City
Spark RDD Practice: Population of each City
5. Orders placed by Customers
Spark RDD Practice: Orders placed by Customers
6. Movie Average Rating greater than 3
Spark RDD Practice: Movie Average Rating greater than 3
9. Spark Dataframes & Spark SQL

1. Dataframe Intro
Spark Dataframes & Spark SQL: Dataframe Intro
2. Dafaframe from Json Files
Spark Dataframes & Spark SQL: Dafaframe from Json Files
3. Dataframe from Parquet Files
Spark Dataframes & Spark SQL: Dataframe from Parquet Files
4. Dataframe from CSV Files
Spark Dataframes & Spark SQL: Dataframe from CSV Files
5. Dataframe from Avro/XML Files
Spark Dataframes & Spark SQL: Dataframe from Avro/XML Files
6. Working with Different Compressions
Spark Dataframes & Spark SQL: Working with Different Compressions
7. DataFrame API Part1
Spark Dataframes & Spark SQL: DataFrame API Part1
8. DataFrame API Part2
Spark Dataframes & Spark SQL: DataFrame API Part2
9. Spark SQL
Spark Dataframes & Spark SQL: Spark SQL
10. Working with Hive Tables in Spark
Spark Dataframes & Spark SQL: Working with Hive Tables in Spark

Database Administration

A complete course on Sqoop, Flume, and Hive: Ideal for achieving CCA175 and Hortonworks Spark Certification

Master Big Data Ingestion and Analytics with Flume, Sqoop, Hive and Spark

<strong>Want to learn how to use Maven and SonarQube effectively for code building and code quality analysis as a DevOps engineer? Then you are in the right place. This learner-centered hands-on course will help you gain confidence in using important DevOps tools such as SVN, Maven, Jenkins, Chef, Puppet, Nagios, Splunk, Selenium, and more. Some basic knowledge of Linux, Git, and AWS EC2 will help you get the most out of this course.</strong>
        <p>Maven is the build automation tool that can be used in applications developed by software languages such as Java, C#, Scala, and other languages. There are lots of DevOps tools such as GitHub, SVN, Maven, Jenkins, Chef, Puppet, Nagios, Splunk, Selenium, and so on. It helps DevOps in providing automation around the build phase of the DevOps life cycle management.

This course helps you to understand how a developer uses Maven and SonarQube for DevOps professionals. You will be creating a maven project on Eclipse, then look at the activities of a DevOps Engineer on Maven. After that, you will create a project using the Maven archetype, and learn Maven coordinates, Maven goals, and Maven repositories. Next, you will learn about pom.xml, commit code from the developer system to GitHub, build projects on Maven, and understand Maven in DevOps workflow.

Finally, you will set up and Integrate Maven with Jenkins along with SonarQube with Jenkins.

By the end of this course, you will be able to learn how to use Maven and SonarQube for code build and code quality analysis from a DevOps engineer's point of view.

All the resource files are uploaded to the GitHub repository at https://github.com/PacktPublishing/maven</p>
        <h2>What You Will Learn</h2><p>Create a Maven project on Eclipse<br>Create a project using the Maven archetype<br>Commit code from developer system to GitHub<br>Understand Maven repositories and build projects on Maven<br>Set up and integrate Maven with Jenkins<br>Set up and integrate SonarQube with Jenkins</p>
        <h2>Audience</h2><p>This course is designed for individuals who are curious to know how to use Maven as a DevOps engineer working in any organization. It will also help beginner-level DevOps enthusiasts who want to explore Maven and SonarQube.<br><br>To gain the most out of this course, you should have basic knowledge of Linux, Git, and AWS EC2.</p>
        <h2>Approach</h2><p>This is a well-structured and comprehensive course keeping in mind the beginner level. The course doesn't deviate and is crisp to the topic. The course comes bundled with code examples and documentation which will be very helpful.<br><br>The course is a quick and useful guide to start with Maven and SonarQube and you will get an opportunity to work on the hands-on labs.</p>
        <h2>Key Features</h2><p>Discover a comprehensive course along with hands-on practical lab * Learn to use Maven in the DevOps workflow * Look at and explore the activities of a DevOps engineer on Maven</p>
        <h2>Github Repo</h2><p>https://github.com/PacktPublishing/maven</p>
        <h2>About the Author</h2><h3>Valaxy Technologies</h3><p>Valaxy Technologies is recognized as one of the top Cloud and DevOps trainers. A.R. Shankar is the owner, founder, and one of the many instructors at Valaxy Technologies. While he has worked with various technologies in his career, he specializes in Cloud and DevOps. 

Valaxy Technologies has exponentially grown ever since its inception in 2013. The instructors at Valaxy Technologies have trained over 100,000 global learners. </p>
        <h2>Course Outline</h2><h4>1. Introduction</h4><p></p><br><h4>2. Setup Maven Environment on Windows</h4><p></p><br><h4>3. Maven Goals, Repositories</h4><p></p><br><h4>4. Maven on Linux</h4><p></p><br><h4>5. Maven in DevOps workflow</h4><p></p><br><h4>6. Additional Resources</h4><p></p><br><h4>7. SonarQube </h4><p></p>

Want to learn how to use Maven and SonarQube effectively for code building and code quality analysis as a DevOps engineer? Then you are in the right place. This learner-centered hands-on course will help you gain confidence in using important DevOps tools such as SVN, Maven, Jenkins, Chef, Puppet, Nagios, Splunk, Selenium, and more. Some basic knowledge of Linux, Git, and AWS EC2 will help you get the most out of this course.
Maven is the build automation tool that can be used in applications developed by software languages such as Java, C#, Scala, and other languages. There are lots of DevOps tools such as GitHub, SVN, Maven, Jenkins, Chef, Puppet, Nagios, Splunk, Selenium, and so on. It helps DevOps in providing automation around the build phase of the DevOps life cycle management. This course helps you to understand how a developer uses Maven and SonarQube for DevOps professionals. You will be creating a maven project on Eclipse, then look at the activities of a DevOps Engineer on Maven. After that, you will create a project using the Maven archetype, and learn Maven coordinates, Maven goals, and Maven repositories. Next, you will learn about pom.xml, commit code from the developer system to GitHub, build projects on Maven, and understand Maven in DevOps workflow. Finally, you will set up and Integrate Maven with Jenkins along with SonarQube with Jenkins. By the end of this course, you will be able to learn how to use Maven and SonarQube for code build and code quality analysis from a DevOps engineer's point of view. All the resource files are uploaded to the GitHub repository at https://github.com/PacktPublishing/maven
What You Will Learn
Create a Maven project on Eclipse
Create a project using the Maven archetype
Commit code from developer system to GitHub
Understand Maven repositories and build projects on Maven
Set up and integrate Maven with Jenkins
Set up and integrate SonarQube with Jenkins
Audience
This course is designed for individuals who are curious to know how to use Maven as a DevOps engineer working in any organization. It will also help beginner-level DevOps enthusiasts who want to explore Maven and SonarQube.

To gain the most out of this course, you should have basic knowledge of Linux, Git, and AWS EC2.
Approach
This is a well-structured and comprehensive course keeping in mind the beginner level. The course doesn't deviate and is crisp to the topic. The course comes bundled with code examples and documentation which will be very helpful.

The course is a quick and useful guide to start with Maven and SonarQube and you will get an opportunity to work on the hands-on labs.
Key Features
Discover a comprehensive course along with hands-on practical lab * Learn to use Maven in the DevOps workflow * Look at and explore the activities of a DevOps engineer on Maven
Github Repo
https://github.com/PacktPublishing/maven
About the Author
Valaxy Technologies
Valaxy Technologies is recognized as one of the top Cloud and DevOps trainers. A.R. Shankar is the owner, founder, and one of the many instructors at Valaxy Technologies. While he has worked with various technologies in his career, he specializes in Cloud and DevOps. Valaxy Technologies has exponentially grown ever since its inception in 2013. The instructors at Valaxy Technologies have trained over 100,000 global learners. 
Course Outline
1. Introduction

2. Setup Maven Environment on Windows

3. Maven Goals, Repositories

4. Maven on Linux

5. Maven in DevOps workflow

6. Additional Resources

7. SonarQube


DevOps

Jenkins

Want to learn how to use Maven and SonarQube effectively for code building and code quality analysis as a DevOps engineer? Then you are in the right place. This learner-centered hands-on course will help you gain confidence in using important DevOps tools such as SVN, Maven, Jenkins, Chef, Puppet, Nagios, Splunk, Selenium, and more. Some basic knowledge of Linux, Git, and AWS EC2 will help you get the most out of this course.

Maven and SonarQube for DevOps Engineers - Beginners Guide

This video covers the essential topics necessary for working with Apache Maven. You will understand the techniques and methods to create multi-module Apache Maven projects from scratch, along with delving into topics needed for testing and deploying Java applications.

Apache Maven: Beginner to Guru

This course will show you why Hadoop is one of the best tools to work with big data. With the help of some real-world data sets, you will learn how to use Hadoop and its distributed technologies, such as Spark, Flink, Pig, and Flume, to store, analyze, and scale big data.
Understanding Hadoop is a highly valuable skill for anyone working at companies that work with large amounts of data. Companies such as Amazon, eBay, Facebook, Google, LinkedIn, IBM, Spotify, Twitter, and Yahoo, use Hadoop in some way to process huge chunks of data. This video course will make you familiar with Hadoop's ecosystem and help you to understand how to apply Hadoop skills in the real world. The course starts by taking you through the installation process of Hadoop on your desktop. Next, you will manage big data on a cluster with Hadoop Distributed File System (HDFS) and MapReduce, and use Pig and Spark to analyze data on Hadoop. Moving along, you will learn how to store and query your data using applications, such as Sqoop, Hive, MySQL, Phoenix, and MongoDB. Next, you will design real-world systems using the Hadoop ecosystem and learn how to manage clusters with Yet Another Resource Negotiator (YARN), Mesos, Zookeeper, Oozie, Zeppelin, and Hue. Towards the end, you will uncover the techniques to handle and stream data in real-time using Kafka, Flume, Spark Streaming, Flink, and Storm. By the end of this course, you will become well-versed with the Hadoop ecosystem and will develop the skills required to store, analyze, and scale big data using Hadoop. All the codes and supporting files for this course are available at - https://github.com/packtpublishing/the-ultimate-hands-on-hadoop
What You Will Learn
Become familiar with Hortonworks and the Ambari User Interface (UI)
Use Pig and Spark to create scripts to process data on a Hadoop cluster
Analyze non-relational data using HBase, Cassandra, and MongoDB
Query data interactively with Drill, Phoenix, and Presto
Publish data to your Hadoop cluster using Kafka, Sqoop, and Flume
Consume streaming data using Spark Streaming, Flink, and Storm
Audience
This video course is designed for people at every level; whether you are a software engineer or a programmer who wants to understand the Hadoop ecosystem, or a project manager who wants to become familiar with the Hadoop's lingo, or a system architect who wants to understand the components available in the Hadoop system. To get started with this course, a basic understanding of Python or Scala and ground-level knowledge of the Linux command line are recommended.
Approach
With the help of a real-world movie dataset, hands-on activities, and interesting exercises, this video course will help you to learn Hadoop and its associated distributed systems. You will find out how easy it is to analyze, store, and scale big data using Hadoop technology.
Key Features
Get to grips with the high-level architecture of Hadoop * Understand the components available in the Hadoop ecosystem, and how they fit together * Get ready to manage big data using Hadoop and related technologies
Github Repo
https://github.com/packtpublishing/the-ultimate-hands-on-hadoop
About the Author
Frank Kane
Frank Kane has spent nine years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers all the time. He holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology and teaches others about big data analysis.
Course Outline
1. Learning All the Buzzwords and Installing the Hortonworks Data Platform Sandbox

1. Introduction and Installation of Hadoop
This video introduces Hadoop. You will learn how to install the Hortonworks Sandbox in a virtual machine on a PC, which is the quickest way to get up and running with Hadoop, so you can start learning and experimenting with it. You will also learn how to download some real movie ratings data and use Hive to analyze it.
2. The Hortonworks and Cloudera Merger and its Effects on the Course
This video focuses on the use of the Hortonworks Data Platform (HDP) and explains why Hortonworks merged with Cloudera works on a Cloudera Data Platform (CDP).
3. Hadoop Overview and History
This video presents an overview of Hadoop history to explain what it is used for and the type of problem it solves.
4. Overview of the Hadoop Ecosystem
This video highlights the concept of the Hadoop ecosystem. You will learn about all the technologies and how they all fit in together, along with the buzzwords used in this course.
2. Using the Hadoop's Core: Hadoop Distributed File System (HDFS) and MapReduce

1. Hadoop Distributed File System (HDFS): What it is and How it Works
This video provides an explanation for storage of massive data sets in Hadoop's Distributed File System (HDFS) across a cluster of commodity computers, in a reliable and scalable manner.
2. Installing the MovieLens Dataset
This video demonstrates how to import real movie ratings data into HDFS using the web-based UI provided by Ambari.
3. Activity - Installing the MovieLens Dataset into Hadoop's Distributed File System (HDFS) using the Command Line
This video demonstrates how to import data from the command prompt and interact with HDFS through the command line interface.
4. MapReduce: What it is and How it Works
This video provides an explanation of how mappers and reducers provide a clever way to analyze massively distributed datasets quickly and reliably.
5. How MapReduce Distributes Processing
This video demonstrates how to scale horizontally across a cluster of computers to find out the power of MapReduce.
6. MapReduce Example: Breaking Down the Movie Ratings by Rating Score
This video demonstrates how to count each rating type that exists in the movie ratings data.
7. Activity - Installing Python, MRJob, and Nano
This video demonstrates how to set up and start with MapReduce using the Python's MRJob package, which allows you to use the MapReduce's streaming feature to write a MapReduce code in Python instead of Java.
8. Activity - Coding Up and Running the Ratings Histogram MapReduce Job
This video demonstrates how to code for building a breakdown of movie ratings and running it on a system.
9. Exercise - Ranking Movies by Their Popularity
This video demonstrates how to write a MapReduce script that sorts movies by how many ratings they have received.
10. Activity - Checking Results
This video demonstrates how to change a script to count movies instead of ratings and review and run a solution for sorting by using the rating count.
3. Programming Hadoop with Pig

1. Introducing Ambari
This video introduces you to the Ambari-Hortonworks' web-based UI as an easy way to experiment with Pig.
2. Introducing the Pig
This video presents an overview of what Pig is used for, who it is for, and how it works.
3. Example - Finding the Oldest Movie with Five-Star Rating Using the Pig
This video demonstrates how to use Pig to script a chain of queries on MovieLens to solve a more complex problem.
4. Activity - Finding the Old Five-Star Movies with Pig
This video demonstrates how to run a program on the Hadoop sandbox to find good and old movies.
5. More Pig Latin
This video focuses on what else Pig Latin can do.
6. Exercise - Finding the Most-Rated One-Star Movie
This video demonstrates how to write a Pig script that finds the most popular "really bad" movie.
7. Pig Challenge - Comparing Results
This video demonstrates how to compare your code results with those of the authors.
4. Programming Hadoop with Spark

1. Why Spark?
This video provides an explanation of how Spark's efficiency and versatility makes Apache Spark one of the hottest Hadoop-related technologies right now, and how it achieves this under the hood.
2. The Resilient Distributed Datasets (RDD)
This video presents an overview of the Resilient Distributed Datasets (RDD) and the functions available in it.
3. Activity - Finding the Movie with the Lowest Average Rating with the Resilient Distributed Datasets (RDD)
This video demonstrates how to write a Spark script to find the movie with the lowest average rating using RDD.
4. Datasets and Spark 2.0
This video provides an explanation of how datasets can make your Spark scripts even faster and easier to write.
5. Activity - Finding the movie with the Lowest Average Rating with DataFrames
This video demonstrates how to write a Spark script to find the movie with the lowest average rating using DataFrames.
6. Activity - Recommending a Movie with Spark's Machine Learning Library (MLLib)
This video demonstrates how to use Spark's machine learning library to produce movie recommendations using the Alternating Least Square (ALS) algorithm.
7. Exercise - Filtering the Lowest-Rated Movies by Number of Ratings
This video demonstrates how to filter the results of movies with a given number of ratings.
8. Activity - Checking Results
This video demonstrates how to review the solution and run the resulting scripts.
5. Using Relational Datastores with Hadoop

1. What is Hive?
This video introduces Apache Hive. You will learn how it enables relational queries on the HDFS-hosted data.
2. Activity - Using Hive to Find the Most Popular Movie
This video demonstrates how to import the MovieLens data set into Hive using the Ambari UI and runs a simple query to find the most popular movies.
3. How Hive Works?
This video provides an explanation of how Hive works under the hood, while using the SQL commands.
4. Exercise - Using Hive to Find the Movie with the Highest Average Rating
This video demonstrates how to use the same Hive database to find the best-rated movie.
5. Comparing Solutions
This video demonstrates how to compare solutions with that of authors.
6. Integrating MySQL with Hadoop
This video presents an overview of MySQL and explains how it might fit into a Hadoop-based work.
7. Activity - Installing MySQL and Importing Movie Data
This video demonstrates how to import the MovieLens data set into MySQL and run a query.
8. Activity - Using Sqoop to Import Data from MySQL to HFDS/Hive
This video demonstrates how to use Sqoop to transfer data from an existing Relational Database Management System (RDBMS) similar to MySQL into Hadoop.
9. Activity - Using Sqoop to Export Data from Hadoop to MySQL
This video demonstrates how to build a new table with Hive and export it back into MySQL.
6. Using Non-Relational Data Stores with Hadoop

1. Why NoSQL?
This video provides an explanation of why NoSQL databases are important for efficiently and scalably vending data.
2. What is HBase?
This video presents an overview of what the HBase, a NoSQL columnar data store, is used for and how it works.
3. Activity - Importing Movie Ratings into HBase
This video demonstrates how to use the RESTful service interface to import movie ratings into HBase.
4. Activity - Using HBase with Pig to Import Data at Scale
This video demonstrates how to integrate HBase with Pig to store big data into HBase.
5. Cassandra - Overview
This video presents an overview of Cassandra, a popular NoSQL database that is appropriate for vending data at massive scale outside of Hadoop.
6. Activity - Installing Cassandra
This video demonstrates how to install Cassandra as it is not part of Hortonworks.
7. Activity - Writing Spark Output into Cassandra
This video demonstrates how to modify an HBase example to write results into a Cassandra database.
8. MongoDB - Overview
This video presents an overview of MongoDB, a popular alternative to Cassandra with different features.
9. Activity - Installing MongoDB and Integrating Spark with MongoDB
This video demonstrates how to install MongoDB on a virtual machine using Ambari, study and run a script to load up a Spark DataFrame of user data and store it into MongoDB.
10. Activity - Using the MongoDB Shell
This video demonstrates how to set up an index using MongoDB's command-line interface.
11. Choosing Database Technology
This video demonstrates how to take an informed decision by analyzing the various attributes of databases.
12. Exercise - Choosing a Database for a Given Problem
This video demonstrates how to compare the decision to choose a database with that of the author.
7. Querying Data Interactively

1. Overview of Drill
This video presents an overview of Drill and the problems it solves.
2. Activity - Setting Up Drill
This video demonstrates how to install the Drill.
3. Activity - Querying Across Multiple Databases with Drill
This video demonstrates how to use Drill to execute a query that spans data on MongoDB and Hive at the same time.
4. Overview of Phoenix
This video presents an overview of Phoenix.
5. Activity - Installing Phoenix and Querying HBase
This video demonstrates how to work with Phoenix and use it to query an HBase Database.
6. Activity - Integrating Phoenix with the Pig
This video demonstrates how to use Phoenix with Pig to store, load the MovieLens user's data, and accelerate queries on it.
7. Overview of Presto
This video presents an overview of Presto and provides its comparison with Drill and Phoenix.
8. Activity - Installing Presto and Querying Hive
This video demonstrates how to install Presto and issue queries on Hive through it.
9. Activity - Querying Both Cassandra and Hive Using Presto
This video demonstrates how to perform a JOIN query that spans both - data in Cassandra and Hive.
8. Managing Your Cluster

1. Yet Another Resource Negotiator (YARN)
This video provides an in-depth explanation on the workings of YARN that controls and allocates the resources of your Hadoop cluster.
2. Tez
This video provides an explanation of the use of Directed Acyclic Graphs by Tez to optimize tasks on a cluster.
3. Activity - Using Hive on Tez and Measuring the Performance Benefit
This video demonstrates how to execute a Hive on Tez.
4. Mesos
This video presents the difference between Mesos and the alternative cluster manager to Hadoop YARN.
5. ZooKeeper
This video focuses on the workings of Zookeeper and the systems that depend on Zookeeper for reliable operation.
6. Activity - Simulating a Failing Master with ZooKeeper
This video demonstrates how to use ZooKeeper's command line interface to explore its workings.
7. Oozie
This video introduces Oozie that allows you to set up complex workflows on a cluster using multiple technologies and schedule them. You will also learn about a few examples of its workings.
8. Activity - Setting Up a Simple Oozie Workflow
This video demonstrates how to use Oozie to import movie data into the Hadoop Distributed File System (HDFS) from MySQL using Sqoop and analyze that data using Hive.
9. Zeppelin - Overview
This video presents an overview of Apache Zeppelin, which provides a notebook-based environment for importing, transforming, and analyzing data.
10. Activity - Using Zeppelin to Analyze Movie Ratings - Part 1
This is the first part of a two-part video that demonstrates how to set up a Zeppelin notebook to load movie ratings and titles into the Spark dataframes and interactively query and visualize them.
11. Activity - Using Zeppelin to Analyze Movie Ratings - Part 2
This is the second part of a two-part video that demonstrates how to set up a Zeppelin notebook to load movie ratings and titles into the Spark dataframes and interactively query and visualize them.
12. Hue - Overview
This video presents an overview of what Hue offers and how it is different than Ambari.
13. Other Technologies Worth Mentioning
This video focuses on Chukwa and Ganglia.
9. Feeding Data to Your Cluster

1. Kafka
This video provides an explanation of how Kafka provides a scalable and reliable means for collecting data across a cluster of computers and broadcasting it for further processing.
2. Activity - Setting Up Kafka and Publishing Data
This video demonstrates how to set up Kafka to publish and consume data from a new topic.
3. Activity - Publishing Web Logs with Kafka
This video demonstrates how to use a Kafka connector to monitor the Apache log files.
4. Flume
This video presents an overview of the sinks and Flume's architecture. You will also learn about how it differs from Kafka.
5. Activity - Setting up Flume and Publishing Logs
This video demonstrates how to connect a source listening to a telnet connection to a sink that logs the information received.
6. Activity - Setting Up Flume to Monitor a Directory and Store its Data in Hadoop Distributed File System (HDFS)
This video demonstrates how to configure Flume to monitor a directory on a local file system for new files, and publish their data into HDFS, organized by the time the data was received.
10. Analyzing Streams of Data

1. Spark Streaming: Introduction
This video provides an explanation of how Spark Streaming allows you to write "continuous applications" that process micro-batches of information in real time.
2. Activity - Analyzing Web Logs Published with Flume using Spark Streaming
This video demonstrates how to write and run a Spark Streaming application that analyzes web logs as they are streamed-in from Flume.
3. Exercise - Monitor Flume-Published Logs for Errors in Real Time
This video demonstrates how to look for status codes in the web log and aggregates on how often different status codes appear.
4. Exercise Solution: Aggregating the Hypertext Transfer Protocol (HTTP) Access Codes with Spark Streaming
This video demonstrates how to review the author's solution to the previous exercise.
5. Apache Storm: Introduction
This video introduces Storm Streaming, which is a better solution to Spark Streaming.
6. Activity - Counting Words with Storm
This video demonstrates how to walk through and run the word count topology sample including Storm.
7. Flink: Overview
This video presents an overview of the difference between Apache Flink and Storm.
8. Activity - Counting Words with Flink
This video demonstrates how to install Flink.
11. Designing Real-World Systems

1. The Best of the Rest
This video focuses on other systems, such as Impala, NiFi, Falcon, Accumulo, Amazon Web Services (AWS), Kinesis, Redis, Ignite, Elasticsearch, and Slider.
2. Review: How the Pieces Fit Together?
This video provides an explanation of how other systems are categorized and fit together.
3. Understanding Your Requirements
This video focuses on the right questions about performance, reliability, and complexity.
4. Sample Application: Consuming Web Server Logs and Keeping Track of Top-Sellers
This video presents an overview of the technologies needed. You will also learn how they will fit together for the sample application.
5. Sample Application: Serving Movie Recommendations to a Website
This video introduces you to the tools needed for the sample application.
6. Exercise - Designing a System to Report Web Sessions Per Day
This video demonstrates how to handle web analytics data for large websites.
7. Exercise Solution: Designing a System to Count Daily Sessions
This video demonstrates how to compare your system design with that of the author.
12. Learning More

1. Books and Online Resources
This video focuses on some of the author's favorite tools and websites for reference.

MongoDB

This course will show you why Hadoop is one of the best tools to work with big data. With the help of some real-world data sets, you will learn how to use Hadoop and its distributed technologies, such as Spark, Flink, Pig, and Flume, to store, analyze, and scale big data.

Educators providing Scala courses in Oxted delivered On Demand