Cademy logoCademy Marketplace
  • Professional Development
  • Medicine & Nursing
  • Arts & Crafts
  • Health & Wellbeing
  • Personal Development

3 Apache Airflow courses

Apache Airflow

Data Engineering on Google Cloud

By Nexus Human

Duration 4 Days 24 CPD hours This course is intended for This class is intended for experienced developers who are responsible for managing big data transformations including: Extracting, loading, transforming, cleaning, and validating data. Designing pipelines and architectures for data processing. Creating and maintaining machine learning and statistical models. Querying datasets, visualizing query results and creating reports Overview Design and build data processing systems on Google Cloud Platform. Leverage unstructured data using Spark and ML APIs on Cloud Dataproc. Process batch and streaming data by implementing autoscaling data pipelines on Cloud Dataflow. Derive business insights from extremely large datasets using Google BigQuery. Train, evaluate and predict using machine learning models using TensorFlow and Cloud ML. Enable instant insights from streaming data Get hands-on experience with designing and building data processing systems on Google Cloud. This course uses lectures, demos, and hand-on labs to show you how to design data processing systems, build end-to-end data pipelines, analyze data, and implement machine learning. This course covers structured, unstructured, and streaming data. Introduction to Data Engineering Explore the role of a data engineer. Analyze data engineering challenges. Intro to BigQuery. Data Lakes and Data Warehouses. Demo: Federated Queries with BigQuery. Transactional Databases vs Data Warehouses. Website Demo: Finding PII in your dataset with DLP API. Partner effectively with other data teams. Manage data access and governance. Build production-ready pipelines. Review GCP customer case study. Lab: Analyzing Data with BigQuery. Building a Data Lake Introduction to Data Lakes. Data Storage and ETL options on GCP. Building a Data Lake using Cloud Storage. Optional Demo: Optimizing cost with Google Cloud Storage classes and Cloud Functions. Securing Cloud Storage. Storing All Sorts of Data Types. Video Demo: Running federated queries on Parquet and ORC files in BigQuery. Cloud SQL as a relational Data Lake. Lab: Loading Taxi Data into Cloud SQL. Building a Data Warehouse The modern data warehouse. Intro to BigQuery. Demo: Query TB+ of data in seconds. Getting Started. Loading Data. Video Demo: Querying Cloud SQL from BigQuery. Lab: Loading Data into BigQuery. Exploring Schemas. Demo: Exploring BigQuery Public Datasets with SQL using INFORMATION_SCHEMA. Schema Design. Nested and Repeated Fields. Demo: Nested and repeated fields in BigQuery. Lab: Working with JSON and Array data in BigQuery. Optimizing with Partitioning and Clustering. Demo: Partitioned and Clustered Tables in BigQuery. Preview: Transforming Batch and Streaming Data. Introduction to Building Batch Data Pipelines EL, ELT, ETL. Quality considerations. How to carry out operations in BigQuery. Demo: ELT to improve data quality in BigQuery. Shortcomings. ETL to solve data quality issues. Executing Spark on Cloud Dataproc The Hadoop ecosystem. Running Hadoop on Cloud Dataproc. GCS instead of HDFS. Optimizing Dataproc. Lab: Running Apache Spark jobs on Cloud Dataproc. Serverless Data Processing with Cloud Dataflow Cloud Dataflow. Why customers value Dataflow. Dataflow Pipelines. Lab: A Simple Dataflow Pipeline (Python/Java). Lab: MapReduce in Dataflow (Python/Java). Lab: Side Inputs (Python/Java). Dataflow Templates. Dataflow SQL. Manage Data Pipelines with Cloud Data Fusion and Cloud Composer Building Batch Data Pipelines visually with Cloud Data Fusion. Components. UI Overview. Building a Pipeline. Exploring Data using Wrangler. Lab: Building and executing a pipeline graph in Cloud Data Fusion. Orchestrating work between GCP services with Cloud Composer. Apache Airflow Environment. DAGs and Operators. Workflow Scheduling. Optional Long Demo: Event-triggered Loading of data with Cloud Composer, Cloud Functions, Cloud Storage, and BigQuery. Monitoring and Logging. Lab: An Introduction to Cloud Composer. Introduction to Processing Streaming Data Processing Streaming Data. Serverless Messaging with Cloud Pub/Sub Cloud Pub/Sub. Lab: Publish Streaming Data into Pub/Sub. Cloud Dataflow Streaming Features Cloud Dataflow Streaming Features. Lab: Streaming Data Pipelines. High-Throughput BigQuery and Bigtable Streaming Features BigQuery Streaming Features. Lab: Streaming Analytics and Dashboards. Cloud Bigtable. Lab: Streaming Data Pipelines into Bigtable. Advanced BigQuery Functionality and Performance Analytic Window Functions. Using With Clauses. GIS Functions. Demo: Mapping Fastest Growing Zip Codes with BigQuery GeoViz. Performance Considerations. Lab: Optimizing your BigQuery Queries for Performance. Optional Lab: Creating Date-Partitioned Tables in BigQuery. Introduction to Analytics and AI What is AI?. From Ad-hoc Data Analysis to Data Driven Decisions. Options for ML models on GCP. Prebuilt ML model APIs for Unstructured Data Unstructured Data is Hard. ML APIs for Enriching Data. Lab: Using the Natural Language API to Classify Unstructured Text. Big Data Analytics with Cloud AI Platform Notebooks What's a Notebook. BigQuery Magic and Ties to Pandas. Lab: BigQuery in Jupyter Labs on AI Platform. Production ML Pipelines with Kubeflow Ways to do ML on GCP. Kubeflow. AI Hub. Lab: Running AI models on Kubeflow. Custom Model building with SQL in BigQuery ML BigQuery ML for Quick Model Building. Demo: Train a model with BigQuery ML to predict NYC taxi fares. Supported Models. Lab Option 1: Predict Bike Trip Duration with a Regression Model in BQML. Lab Option 2: Movie Recommendations in BigQuery ML. Custom Model building with Cloud AutoML Why Auto ML? Auto ML Vision. Auto ML NLP. Auto ML Tables.

Data Engineering on Google Cloud
Delivered OnlineFlexible Dates
Price on Enquiry

Data Science Model Deployments and Cloud Computing on GCP

By Packt

Are you interested in learning and deploying applications at scale using Google Cloud platform? Do you lack hands-on exposure when it comes to deploying applications and seeing them in action? Then this course is for you. You will also learn microservices and event-driven architectures with real-world use case implementations.

Data Science Model Deployments and Cloud Computing on GCP
Delivered Online On Demand6 hours 55 minutes
£56.99

Snowflake - Build and Architect Data Pipelines Using AWS

By Packt

The course helps you learn Snowflake from scratch and explore a few of its important features. You will build automated pipelines with Snowflake and use the AWS cloud with Snowflake as a data warehouse. You will also explore Snowpark to be worked on the data pipelines.

Snowflake - Build and Architect Data Pipelines Using AWS
Delivered Online On Demand8 hours 39 minutes
£52.99

Educators matching "Apache Airflow"

Show all 3
Nobleprog

nobleprog

We have a proven record for meeting and exceeding client expectations, even in the most demanding and challenging circumstances. Our consultants are highly experienced expert practitioners in their field, with extensive technical backgrounds, ensuring that the advice you receive will be based on technical expertise combined with real world knowledge.

Message
Nobleprog Pakistan

nobleprog pakistan

NobleProg is an international training and consultancy group, delivering high quality courses to every sector, covering: Artificial Intelligence, IT, Management, Applied Statistics. Over the last 17 years, we have trained more than 50,000 people from over 6000 companies and organisations. Our courses include classroom (both public and closed) and instructor-led online giving you choice and flexibility to suit your time, budget and level of expertise. We practice what we preach – we use a great deal of the technologies and methods that we teach, and continuously upgrade and improve our courses, keeping up to date with all the latest developments. Our trainers are hand picked and have been through rigorous checks and interviews, and all courses are evaluated by delegates ensuring continuous feedback and improvement. NobleProg in numbers 17 + years of experience 15 + offices all over the world 1000 + trainers cooperating with NobleProg 1400 + course outlines offered companies 6100 + companies that entrusted us satisfied participant 58 k. + satisfied participants NobleProg - The World’s Local Training Provider Our mission is to provide comprehensive training and consultancy solutions all over the world, in an effective and accessible way, tailored to consumers’ needs . We offer practical, real-world knowledge supported by a full understanding of the theory. Our expert trainers are skilled in the latest knowledge transfer techniques, blending presentation, demonstration and hands-on learning. We understand that our learners are excited to be gaining new skills and we thrive off that energy to deliver exceptional training events. Investing in upskilling or reskilling with NobleProg means you stay ahead. Our catalogue is constantly evolving and we offer the most in-demand courses, Java, JavaScript, SQL, Visual Basic for Applications (VBA), as well as Apache Spark, OpenStack, TensorFlow, Selenium, Artificial Intelligence, Data Analysis. Our offer consists of more than 1,400 training outlines covering more than 120 technologies. At NobleProg we emphasis a need of not only following the latest technological trends, but also anticipating changes. We focus on delivering professional skills and certifications that will have a real impact. See what sets us apart >> NobleProg's history NobleProg was established in 2005 in Krakow, Poland, and has gradually expanded its operations to other global markets since. In just two years the first international branch was opened in London. The overwhelming potential of NobleProg combined with the rising need for self-development programs, especially in the field of technological skills, prompted the company to change the business model into a franchise. By doing so, in a short period of time the company allowed a number of people passionate about education and new technologies to join the NobleProg Team. With each year the territorial reach of NobleProg was further expanding and we now have offices on every continent. NobleProg is the World's Local Training Provider.

Message
Packt

packt

Birmingham

Founded in 2004 in Birmingham, UK, Packt’s mission is to help the world put software to work in new ways, through the delivery of effective learning and information services to IT professionals. Working towards that vision, we have published over 6,500 books and videos so far, providing IT professionals with the actionable knowledge they need to get the job done – whether that’s specific learning on an emerging technology or optimizing key skills in more established tools. As part of our mission, we have also awarded over $1,000,000 through our Open Source Project Royalty scheme, helping numerous projects become household names along the way.

Message

Search By Location