Description

The course helps you learn Snowflake from scratch and explore a few of its important features. You will build automated pipelines with Snowflake and use the AWS cloud with Snowflake as a data warehouse. You will also explore Snowpark to be worked on the data pipelines.

Snowflake is the next big thing, and it is becoming a full-blown data ecosystem. With the level of scalability and efficiency in handling massive volumes of data and also with several new concepts in it, this is the right time to wrap your head around Snowflake and have it in your toolkit. This course not only covers the core features of Snowflake but also teaches you how to deploy Python/PySpark jobs in AWS Glue and Airflow that communicate with Snowflake, which is one of the most important aspects of building pipelines. In this course, you will look at Snowflake, and then the most crucial aspects of Snowflake in an efficient manner. You will be writing Python/Spark Jobs in AWS Glue Jobs for data transformation and seeing real-time streaming using Kafka and Snowflake. You will be interacting with external functions and use cases, and see the security features in Snowflake. Finally, you will look at Snowpark and explore how it can be used for data pipelines and data science. By the end of this course, you will have learned about Snowflake and Snowpark, and learned how to build and architect data pipelines using AWS. You need to have an active AWS account in order to perform the sections related to Python and PySpark. For the rest of the course, a free trial Snowflake account should suffice.

What You Will Learn

Learn about Snowflake and its basics before getting started with labs
Check the crucial aspects of Snowflake in a very practical manner
Write Python/Spark jobs in AWS Glue Jobs for data transformation
Execute real-time streaming using Kafka and Snowflake
Interact with external functions and use cases
Learn and explore the Snowpark library

Audience

This course is ideal for software engineers, aspiring data engineers or data analysts, and data scientists who want to excel in their careers in the IT domain. Apart from them, this course is also good for programmers and database administrators with experience in writing SQL queries.

Prior programming experience in SQL or at least some prior knowledge in writing queries and Python is a must. You should have a basic experience or understanding of cloud services such as AWS along with an active AWS account.

Approach

This course is comprehensive, easy to understand, and designed for intermediate to advance with a basic understanding of programming and SQL knowledge.

You can read the reference links and the official documentation of Snowflake as much as possible, which are attached to the resource links. There are lab exercises and practical content demonstrated after providing theoretical information; therefore, a well-balanced course.

Key Features

Learn from an easy-to-understand and step-by-step course, divided into 85+ videos along with detailed resource files * Integrate real-time streaming data and data orchestration with Airflow and Snowflake * Highly practical explanations and lab exercises to help you grasp the most out of the course

Github Repo

https://github.com/PacktPublishing/Snowflake---Build-and-Architect-Data-Pipelines-using-AWS

About the Author

Siddharth Raghunath

Siddharth Raghunath is a business-oriented engineering manager with a vast experience in the field of software development, distributed processing, and cloud data engineering. He has worked on different cloud platforms such as AWS and GCP as well as on-premise Hadoop clusters. He conducts seminars on distributed processing using Spark, real-time streaming and analytics, and best practices for ETL and data governance. He is passionate about coding and building optimal data pipelines for robust data processing and streaming solutions.

Course Outline

1. Introduction to the Course

1. Course Roadmap

This video provides the course's roadmap.

2. Prerequisites and How to Succeed in This Course

This video talks about the prerequisites and how to succeed in this course.

2. Introduction to Snowflake and AWS

1. What Is Data-Warehouse?

This video talks about the Data-Warehouse.

2. Two Aspects of a Data Ecosystem

This video demonstrates the two aspects of a data ecosystem.

3. Lab - Set Up Snowflake Trial Account

This is a lab video on setting up a Snowflake trial account.

4. Snowflake Architecture

This video explains the Snowflake architecture.

5. Snowflake Object Hierarchy

This video explains about the Snowflake object hierarchy.

6. Snowflake - Virtual Warehouses

This video explains about virtual warehouses in Snowflake.

7. Snowflake - Different Billing Components

This video demonstrates about the different billing components in Snowflake.

8. Snowflake - Track Your Consumption

This video explains how to track your consumption in Snowflake.

9. Snowflake- Resource Monitors

This video explains about resource monitors in Snowflake.

3. Snowflake - Tables

1. Introduction - Different Tables in Snowflake

This video provides an introduction to different tables in Snowflake.

2. Lab - Create Tables in Snowflake

This is a lab video on how to create tables in Snowflake.

3. Snowflake - Views, Materialized Views and Secure Views

This video explains views, materialized views, and secure views in Snowflake.

4. Lab - Create Views in Snowflake

This is a lab video on how to create views in Snowflake.

5. Lab - Create Secure Views in Snowflake

This is a lab video that explains how to create secure views in Snowflake.

6. More about Views in Snowflake

This video explores more about views in Snowflake.

4. Snowflake - Partitioning, Clustering, and Performance Optimization

1. Section Overview

This video provides an overview of the section.

2. Introduction to Partitions and clustering keys

This video demonstrates an introduction to partitions and clustering keys.

3. Lab - Micro-Partitions and Clustering Keys

This video explains micro-partitions and clustering keys.

4. Benefits of Micro-Partitions and Clustering

This video explains the benefits of micro-partitions and clustering.

5. Understanding Clustering Depth and Cluster Overlap

This video helps you in understanding clustering depth and cluster overlap.

6. Lab - Selecting Your Clustering Keys

This is a lab video explains on selecting your clustering keys.

7. Lab - Check Query Profile and History

This is a lab video on checking the query profile and history.

8. Lab - Query Processing and Caching

This is a lab video that explains query processing and caching.

9. Search Optimization Feature

This video explains the search optimization feature.

5. Snowflake - Data Loading/Ingestion and Extraction

1. Section Overview

This video provides an overview to the section.

2. Data Ingestion - Real-World Use Cases

This video shows data ingestion with the help of real world use cases.

3. Lab - Create an Integration Object to Connect Snowflake with AWS S3

This is a lab video that helps create an integration object to connect Snowflake with AWS S3.

4. Lab - Ingest CSV from S3 to Snowflake

This is a lab video to ingest CSV from S3 to Snowflake.

5. Lab - Ingest JSON from S3 to Snowflake

This is a lab video to ingest JSON from S3 to Snowflake.

6. Introduction to Continuous Data Ingestion in Snowflake

This video provides an introduction to continuous data ingestion in Snowflake.

7. Lab - Create and Implement Snow Pipe

This is a lab video that demonstrates the creation and implementation of Snow pipe.

8. Snow pipe - Billing Estimation and Key Considerations for Data Ingestion

This video explains the billing estimation and key considerations for data ingestion in Snow pipe.

9. Lab - Extracting/Unload Data from Snowflake to S3

This is a lab video that demonstrates extracting/unloading data from Snowflake to S3.

6. Snowflake - Tasks and Query Scheduling

1. Section Overview

This video provides an overview of the section.

2. Introduction to Tasks

This video provides an introduction to tasks.

3. Lab - Create Standalone and Dependent Tree of Tasks

This is a lab video that helps create standalone and dependent tree of tasks.

4. Lab - Billing and Query History for Tasks

This s a lab video that demonstrates the billing and query history for tasks.

7. Snowflake - Streams and Change Data Capture

1. Section Overview

This video provides an overview of the section.

2. Introduction to Streams

This video provides an introduction to streams.

3. Lab - Implement Standard Streams

This is a lab video that helps implement standard streams.

4. Lab - Implement Append-Only Streams

This is a lab video to implement append-only streams.

5. Lab - Streams in a Transaction

This is a lab video that explains the streams in a transaction.

6. Streams - Data Retention and Staleness

This video explains about the data retention and staleness in streams.

7. Lab - Change Tracking Using "Changes"

This is a lab video to help you change tracking using "Changes".

8. Project Overview

This video demonstrates the project overview.

9. Lab - Create Streams - Project Solution

This is a lab video that demonstrates how to create streams - project solution.

10. Lab - Create Streams - Continuation

This lab video is a continuation to the previous video on creating streams.

11. Lab - End-to-End Pipeline in Action

This is a lab video that talks about the end-to-end pipeline in action.

8. Snowflake - User-Defined Functions

1. Introduction to User-Defined Functions and UDF Types

This video provides an Introduction to user-defined functions and UDF types.

2. Lab - Write and Implement a Scalar UDF

This is a lab video to write and implement a Scalar UDF.

3. Lab - Write Tabular UDF in SQL

This is a lab video to write Tabular UDF in SQL.

4. Lab - Implement JavaScript UDFs

This is a lab video helps implement JavaScript UDFs.

5. What Is Pushdown in UDF?

This video explains Pushdown in UDF.

6. Lab - How Can Pushdown Expose the Underlying Data?

This is a lab video that explains how pushdown can expose the underlying data.

7. Lab - Write Secure UDFs

This is a lab video to write secure UDFs.

9. Snowflake - External Functions

1. Section Overview

This video provides an overview of the section.

2. Introduction to External Functions

This video provides an introduction to external functions.

3. Lab - Write Deploy AWS Lambda Function

This is a lab video that demonstrates how to write deploy AWS Lambda function.

4. Create IAM Role

This video demonstrates how to create IAM role.

5. Lab - Create API Gateway

This is a lab video to help you create API Gateway.

6. Lab - Secure and Deploy API Gateway

This is a lab video that demonstrates securing and deploying API gateway.

7. Lab - Create External Function in Snowflake

This is a lab video to create an external function in Snowflake.

10. Snowflake with Python, Spark, and Airflow on AWS

1. Section Overview

This video provides an overview of the section.

2. Lab - Connect Python with Snowflake in Your Local Machine

This is a lab video that helps you connect Python with Snowflake in your local machine.

3. Introduction to AWS Glue

This video provides an introduction to AWS Glue.

4. Lab - Deploy and Execute Python Script to AWS Glue

This is a lab video that helps deploy and execute Python script to AWS Glue.

5. Lab - Parameterize Your Python Script on AWS Glue

This is a lab video that helps parameterize your Python script on AWS Glue.

6. Lab - Python Pandas with Snowflake on AWS Glue

This is a lab video on Python Pandas with Snowflake on AWS Glue.

7. What Is Pushdown in Spark 3.1?

This video explains about Pushdown in Spark 3.1.

8. Lab - Deploy a PySpark Script Using AWS Glue

This is a lab video to deploy a PySpark script using AWS Glue.

9. Lab - Set Up Managed Airflow Cluster on AWS

This is a lab video that helps set up a managed airflow cluster on AWS.

10. Lab - Configure Snowflake Connectivity in Airflow

This is a lab video that explains how to configure Snowflake connectivity in airflow.

11. Lab - Deploy a PySpark Transformation job in AWS Glue

This is a lab video that helps deploy a PySpark transformation job in AWS Glue.

12. Lab - Set Up Airflow DAG

This is a lab video to set up airflow DAG.

11. Real-Time Streaming with Kafka and Snowflake

1. Section Overview

This video provides an overview of the section.

2. Lab - Download the Necessary JAR Files

This is a lab video to download the necessary JAR files.

3. Lab - Set Up Kafka in your local system

This is a lab video to set up Kafka in your local system.

4. Lab - Set Up Kafka Snowflake Connector

This is a lab video to set up the Kafka Snowflake Connector.

5. Lab - Set Up Encryption Keys for Kafka-Snowflake Connectivity

This is a lab video that helps you set up encryption keys for Kafka-Snowflake connectivity.

6. Lab - Streaming Data in Action

This is a lab video on streaming data in action.

12. Snowflake - Data Protection and Governance

1. Section Overview

This video provides an overview of the section.

2. What Is Time Travel and Failsafe in Snowflake?

This video explains about Time Travel and Failsafe in Snowflake.

3. Lab - Time Travel and Data Recovery

This is a lab video that talks about Time Travel and data recovery.

4. Lab - Column Level Dynamic Data Masking

This is a lab video that demonstrates about column level dynamic data masking.

5. What Is Row Level Security?

This video explains about row level security.

6. Lab - Create and Implement Row Level Access Policy

This is a lab video to create and implement row level access policy.

13. Wrap Up and More Learning

1. More Updates and What's Next

This video talks about what's next after this course and more updates.

Course Images

Snowflake - Build and Architect Data Pipelines Using AWS

By Packt

Booking options

Highlights

Description

Course Content

About The Provider

Tags

Reviews