35 Data Processing courses delivered Live Online

Duration 2 Days 12 CPD hours This course is intended for Experienced DataStage developers seeking training in more advanced DataStage job techniques and who seek techniques for working with complex types of data resources. Overview Use Connector stages to read from and write to database tables Handle SQL errors in Connector stages Use Connector stages with multiple input links Use the File Connector stage to access Hadoop HDFS data Optimize jobs that write to database tables Use the Unstructured Data stage to extract data from Excel spreadsheets Use the Data Masking stage to mask sensitive data processed within a DataStage job Use the Hierarchical stage to parse, compose, and transform XML data Use the Schema Library Manager to import and manage XML schemas Use the Data Rules stage to validate fields of data within a DataStage job Create custom data rules for validating data Design a job that processes a star schema data warehouse with Type 1 and Type 2 slowly changing dimensions This course is designed to introduce you to advanced parallel job data processing techniques in DataStage v11.5. In this course you will develop data techniques for processing different types of complex data resources including relational data, unstructured data (Excel spreadsheets), and XML data. In addition, you will learn advanced techniques for processing data, including techniques for masking data and techniques for validating data using data rules. Finally, you will learn techniques for updating data in a star schema data warehouse using the DataStage SCD (Slowly Changing Dimensions) stage. Even if you are not working with all of these specific types of data, you will benefit from this course by learning advanced DataStage job design techniques, techniques that go beyond those utilized in the DataStage Essentials course. Accessing databases Connector stage overview - Use Connector stages to read from and write to relational tables - Working with the Connector stage properties Connector stage functionality - Before / After SQL - Sparse lookups - Optimize insert/update performance Error handling in Connector stages - Reject links - Reject conditions Multiple input links - Designing jobs using Connector stages with multiple input links - Ordering records across multiple input links File Connector stage - Read and write data to Hadoop file systems Demonstration 1: Handling database errors Demonstration 2: Parallel jobs with multiple Connector input links Demonstration 3: Using the File Connector stage to read and write HDFS files Processing unstructured data Using the Unstructured Data stage in DataStage jobs - Extract data from an Excel spreadsheet - Specify a data range for data extraction in an Unstructured Data stage - Specify document properties for data extraction. Demonstration 1: Processing unstructured data Data masking Using the Data Masking stage in DataStage jobs - Data masking techniques - Data masking policies - Applying policies for masquerading context-aware data types - Applying policies for masquerading generic data types - Repeatable replacement - Using reference tables - Creating custom reference tables Demonstration 1: Data masking Using data rules Introduction to data rules - Using the Data Rules Editor - Selecting data rules - Binding data rule variables - Output link constraints - Adding statistics and attributes to the output information Use the Data Rules stage to valid foreign key references in source data Create custom data rules Demonstration 1: Using data rules Processing XML data Introduction to the Hierarchical stage - Hierarchical stage Assembly editor - Use the Schema Library Manager to import and manage XML schemas Composing XML data - Using the HJoin step to create parent-child relationships between input lists - Using the Composer step Writing Hierarchical data to a relational table Using the Regroup step Consuming XML data - Using the XML Parser step - Propagating columns Topic 6: Transforming XML data - Using the Aggregate step - Using the Sort step - Using the Switch step - Using the H-Pivot step Demonstration 1: Importing XML schemas Demonstration 2: Compose hierarchical data Demonstration 3: Consume hierarchical data Demonstration 4: Transform hierarchical data Updating a star schema database Surrogate keys - Design a job that creates and updates a surrogate key source key file from a dimension table Slowly Changing Dimensions (SCD) stage - Star schema databases - SCD stage Fast Path pages - Specifying purpose codes - Dimension update specification - Design a job that processes a star schema database with Type 1 and Type 2 slowly changing dimensions Demonstration 1: Build a parallel job that updates a star schema database with two dimensions Additional course details: Nexus Humans KM423 IBM InfoSphere DataStage v11.5 - Advanced Data Processing training program is a workshop that presents an invigorating mix of sessions, lessons, and masterclasses meticulously crafted to propel your learning expedition forward. This immersive bootcamp-style experience boasts interactive lectures, hands-on labs, and collaborative hackathons, all strategically designed to fortify fundamental concepts. Guided by seasoned coaches, each session offers priceless insights and practical skills crucial for honing your expertise. Whether you're stepping into the realm of professional skills or a seasoned professional, this comprehensive course ensures you're equipped with the knowledge and prowess necessary for success. While we feel this is the best course for the KM423 IBM InfoSphere DataStage v11.5 - Advanced Data Processing course and one of our Top 10 we encourage you to read the course outline to make sure it is the right content for you. Additionally, private sessions, closed classes or dedicated events are available both live online and at our training centres in Dublin and London, as well as at your offices anywhere in the UK, Ireland or across EMEA.

  The IAPP offers the most encompassing, up-to-date and sought-after global training and certification program for privacy and data protection. The Certified Information Privacy Professional (CIPP) helps organizations around the world bolster compliance and risk mitigation practices, and arms practitioners with the insight needed to add more value to their businesses. Skilled privacy pros are in high demand and IAPP certification is what employers want. When you earn an IAPP credential, you earn the right to be recognized as part of an elite group of knowledgeable, capable and dedicated privacy professionals. With the GDPR effective as of May 2018, among its mandates was the requirement to appoint knowledgeable DPOs (data protection officers) tasked with monitoring compliance, managing internal data protection activities, training data processing staff, conducting internal audits and more. There's a lot to know, there's a lot at stake and there's a lot of opportunity for privacy professionals with the right training and education. Achieving a CIPP/E credential shows you have the comprehensive GDPR knowledge, perspective and understanding to ensure compliance and data protection success in Europe-and to take advantage of the career opportunity this sweeping legislation represents. About This Course   Delivered in a Modular format, the course includes; Module 1: Data Protection Laws Introduces key European data protection laws and regulatory bodies, describing the evolution toward a Harmonised European Legislative Framework. Module 2: Personal Data Defines and differentiates between types of data-including personal, anonymous, pseudo-anonymous and special categories. Module 3: Controllers and Processors Describes the roles and relationships of controllers and processors. Module 4: Processing Personal Data Defines data processing and GDPR processing principles, Explains the application of the GDPR and outlines the legitimate bases for processing personal data. Module 5: Information provision Explains controller obligations for providing information about data processing activities to data subjects and Supervisory Authorities. Module 6: Data Subjects 'Rights Describes data subjects' rights, applications of rights and obligations controller and processor. Module 7: Security or Processing Discusses considerations and duties of controllers and processors for Ensuring security of personal data and providing notification of data breaches. Module 8: Accountability Investigates accountability requirements, data protection management systems, data protection impact assessments, privacy policies and the role of the data protection officer. Module 9: International Data Transfers Outlines options and obligations for transferring data outside the European Economic Area, Decisions adequacy and appropriateness safeguards and derogations. Module 10: Supervision and Enforcement Describes the role, powers and procedures or Supervisory Authorities; the composition and tasks of the European Data Protection Board; the role of the European Data Protection Supervisor; and remedies, liabilities and penalties for non-compliance. Module 11: Compliance Discusses the applications of European data protection law, legal bases and compliance requirements for processing personal data in practice, employers-including processing employee data, surveillance, direct marketing, Internet technology and communications and outsourcing. Prerequisites   There are no prerequisites for this course but candidates would benefit from reading the freely available materials found on the IAPP website. What's Included?   1 years membership of the IAPP  Breakfast, Lunch and refreshments (Classroom courses only) Official Study Guide (European Data Protection, Law & Practice)* Participant Guide* Official Exam Q&A* Official Practice Exam Official Practice Exam* The Exam Fees * In electronic format for Live Online and hard copy for Classroom delegates     Who Should Attend?   The CIPP/E is ideal for IT and information security leaders responsible for applying best practices to cloud security architecture, design, operations and service orchestration, including those in the following positions: Cybersecurity Analysts Data Analysts Security Administrators Aspiring Data Protection Officers Accreditation Our Guarantee   We are an approved IAPP Training Partner. You can learn wherever and whenever you want with our robust classroom and interactive online training courses. Our courses are taught by qualified practitioners with a minimum of 25 years commercial experience. We strive to give our delegates the hands-on experience. Our courses are all-inclusive with no hidden extras.  The one-off cost covers the training, all course materials, and exam voucher. Our aim: To achieve a 100% first time pass rate on all our instructor-led courses. Our Promise: Pass first time or 'train' again for FREE. *FREE training offered for retakes - come back within a year and only pay for the exam.

Duration 1 Days 6 CPD hours This course is intended for The audience for this course is individuals who want to learn the fundamentals of database concepts in a cloud environment, get basic skilling in cloud data services, and build their foundational knowledge of cloud data services within Microsoft Azure. Overview Describe core data concepts Identify considerations for relational data on Azure Describe considerations for working with non-relational data on Azure Describe an analytics workload on Azure In this course, students will gain foundational knowledge of core data concepts and related Microsoft Azure data services. Students will learn about core data concepts such as relational, non-relational, big data, and analytics, and build their foundational knowledge of cloud data services within Microsoft Azure. Students will explore fundamental relational data concepts and relational database services in Azure. They will explore Azure storage for non-relational data and the fundamentals of Azure Cosmos DB. Students will learn about large-scale data warehousing, real-time analytics, and data visualization. 1 - Explore core data concepts Identify data formats Explore file storage Explore databases Explore transactional data processing Explore analytical data processing 2 - Explore data roles and services Explore job roles in the world of data Identify data services 3 - Explore fundamental relational data concepts Understand relational data Understand normalization Explore SQL Describe database objects 4 - Explore relational database services in Azure Describe Azure SQL services and capabilities Describe Azure services for open-source databases 5 - Explore Azure Storage for non-relational data Explore Azure blob storage Explore Azure DataLake Storage Gen2 Explore Azure Files Explore Azure Tables 6 - Explore fundamentals of Azure Cosmos DB Describe Azure Cosmos DB Identify Azure Cosmos DB APIs 7 - Explore fundamentals of large-scale data warehousing Describe data warehousing architecture Explore data ingestion pipelines Explore analytical data stores 8 - Explore fundamentals of real-time analytics Understand batch and stream processing Explore common elements of stream processing architecture Explore Azure Stream Analytics Explore Apache Spark on Microsoft Azure 9 - Explore fundamentals of data visualization Describe Power BI tools and workflow Describe core concepts of data modeling Describe considerations for data visualization

Duration 1 Days 6 CPD hours This course is intended for This class is intended for the following: Data analysts, Data scientists, Business analysts getting started with Google Cloud Platform. Individuals responsible for designing pipelines and architectures for data processing, creating and maintaining machine learning and statistical models, querying datasets, visualizing query results and creating reports. Executives and IT decision makers evaluating Google Cloud Platform for use by data scientists. Overview This course teaches students the following skills:Identify the purpose and value of the key Big Data and Machine Learning products in the Google Cloud Platform.Use Cloud SQL and Cloud Dataproc to migrate existing MySQL and Hadoop/Pig/Spark/Hive workloads to Google Cloud Platform.Employ BigQuery and Cloud Datalab to carry out interactive data analysis.Train and use a neural network using TensorFlow.Employ ML APIs.Choose between different data processing products on the Google Cloud Platform. This course introduces participants to the Big Data and Machine Learning capabilities of Google Cloud Platform (GCP). It provides a quick overview of the Google Cloud Platform and a deeper dive of the data processing capabilities. Introducing Google Cloud Platform Google Platform Fundamentals Overview. Google Cloud Platform Big Data Products. Compute and Storage Fundamentals CPUs on demand (Compute Engine). A global filesystem (Cloud Storage). CloudShell. Lab: Set up a Ingest-Transform-Publish data processing pipeline. Data Analytics on the Cloud Stepping-stones to the cloud. Cloud SQL: your SQL database on the cloud. Lab: Importing data into CloudSQL and running queries. Spark on Dataproc. Lab: Machine Learning Recommendations with Spark on Dataproc. Scaling Data Analysis Fast random access. Datalab. BigQuery. Lab: Build machine learning dataset. Machine Learning Machine Learning with TensorFlow. Lab: Carry out ML with TensorFlow Pre-built models for common needs. Lab: Employ ML APIs. Data Processing Architectures Message-oriented architectures with Pub/Sub. Creating pipelines with Dataflow. Reference architecture for real-time and batch data processing. Summary Why GCP? Where to go from here Additional Resources Additional course details: Nexus Humans Google Cloud Platform Big Data and Machine Learning Fundamentals training program is a workshop that presents an invigorating mix of sessions, lessons, and masterclasses meticulously crafted to propel your learning expedition forward. This immersive bootcamp-style experience boasts interactive lectures, hands-on labs, and collaborative hackathons, all strategically designed to fortify fundamental concepts. Guided by seasoned coaches, each session offers priceless insights and practical skills crucial for honing your expertise. Whether you're stepping into the realm of professional skills or a seasoned professional, this comprehensive course ensures you're equipped with the knowledge and prowess necessary for success. While we feel this is the best course for the Google Cloud Platform Big Data and Machine Learning Fundamentals course and one of our Top 10 we encourage you to read the course outline to make sure it is the right content for you. Additionally, private sessions, closed classes or dedicated events are available both live online and at our training centres in Dublin and London, as well as at your offices anywhere in the UK, Ireland or across EMEA.

About this training The Seismic Uncertainty Evaluation (SUE) course has evolved after a number of years of work experience in the sub-surface domain. A common question closely related to well planning is the quantification and qualification of depth uncertainty and robust estimation of the volumetric ranges, and this course addresses these topics. Training Objectives Upon completion of this course, participants will be able to: Define a structured approach toward seismic depth uncertainty analysis Construct data analytics on seismic products (well logs, velocities, and seismic) Classify advance vertical ray tomography on FWI models to assure a drill ready depth seismic, faults, surfaces, and logs Interpret probabilistic volumetric and automatic spill point control, amplitude conformance closures De-risk the depth uncertainty by providing drilling and completion with a risking score card Target Audience This course is intended for individuals who needs to understand the basic theory and procedures for assessment/ quantification/qualification of all drill-ready products (seismic, faults, horizons, etc.) Geologist Geophysicist Reservoir engineer Drilling engineer Course Level Intermediate Trainer Your expert course leader is a cross-functional Geoscientist and Published Author with 27 years of international experience working in Upstream Petroleum Exploration and Production for Oil and Gas Companies in Australia, India, Singapore, Saudi Arabia, and Oman. During his career he actively supported field development, static & dynamic reservoir modelling & well planning, 3D Seismic data acquisition with Schlumberger & SVUL, 3D seismic data processing with CGG & interpretation, Q.I. and field development with Woodside, Applied Geoscience, and Reliance. POST TRAINING COACHING SUPPORT (OPTIONAL) To further optimise your learning experience from our courses, we also offer individualized 'One to One' coaching support for 2 hours post training. We can help improve your competence in your chosen area of interest, based on your learning needs and available hours. This is a great opportunity to improve your capability and confidence in a particular area of expertise. It will be delivered over a secure video conference call by one of our senior trainers. They will work with you to create a tailor-made coaching program that will help you achieve your goals faster. Request for further information post training support and fees applicable Accreditions And Affliations

  European Data Protection   Principles of Data Protection in Europe covers the essential pan-European and national data protection laws, as well as industry-standard best practices for corporate compliance with these laws. Those taking this course will gain an understanding of the European model for privacy enforcement, key privacy terminology and practical concepts concerning the protection of personal data and trans-border data flows.   The training is based on the body of knowledge for the IAPP's ANSI-accredited Certified Information Privacy Professional/Europe (CIPP/E) certification program. Privacy Programme Management   Principles of Privacy Management is the how-to training on implementing a privacy program framework, managing the privacy program operational lifecycle and structuring a knowledgeable, high-performing privacy team. Those taking this course will learn the skills to manage privacy in an organisation through process and technology-regardless of jurisdiction or industry.   The Principles of Privacy Program Management training is based on the body of knowledge for the IAPP's ANSI-accredited Certified Information Privacy Manager (CIPM) certification programme. Make a difference in your organization and in your career. The CIPM designation says that you're a leader in privacy program administration and that you've got the goods to establish, maintain and manage a privacy program across all stages of its lifecycle.  About This Course   Delivered in a modular format, this four day course covers   Days 1 & 2   Module 1: Data Protection Laws Introduces key European data protection laws and regulatory bodies, describing the evolution toward a Harmonised European Legislative Framework.  Module 2: Personal Data Defines and differentiates between types of data-including personal, anonymous, pseudo-anonymous and special categories.  Module 3: Controllers and Processors Describes the roles and relationships of controllers and processors.  Module 4: Processing Personal Data Defines data processing and GDPR processing principles, Explains the application of the GDPR and outlines the legitimate bases for processing personal data. Module 5: Information provision Explains controller obligations for providing information about data processing activities to data subjects and Supervisory Authorities.  Module 6: Data Subjects 'Rights Describes data subjects' rights, applications of rights and obligations controller and processor.  Module 7: Security or Processing Discusses considerations and duties of controllers and processors for Ensuring security of personal data and providing notification of data breaches.  Module 8: Accountability Investigates accountability requirements, data protection management systems, data protection impact assessments, privacy policies and the role of the data protection officer.  Module 9: International Data Transfers Outlines options and obligations for transferring data outside the European Economic Area, Decisions adequacy and appropriateness safeguards and derogations.  Module 10: Supervision and Enforcement Describes the role, powers and procedures or Supervisory Authorities; the composition and tasks of the European Data Protection Board; the role of the European Data Protection Supervisor; and remedies, liabilities and penalties for non-compliance.  Module 11: Compliance Discusses the applications of European data protection law, legal bases and compliance requirements for processing personal data in practice, employers-including processing employee data, surveillance, direct marketing, Internet technology and communications and outsourcing.    Days 3 & 4   Module 1: Introduction to privacy program management Identifies privacy program management responsibilities, and describes the role of accountability in privacy program management.  Module 2: Privacy governance Examines considerations for developing and implementing a privacy program, including the position of the privacy function within the organization, role of the DPO, program scope and charter, privacy strategy, support and ongoing involvement of key functions and privacy frameworks.  Module 3: Applicable laws and regulations Discusses the regulatory environment, common elements across jurisdictions and strategies for aligning compliance with organizational strategy.  Module 4: Data assessments Relates practical processes for creating and using data inventories/maps, gap analyses, privacy assessments, privacy impact assessments/data protection impact assessments and vendor assessments.  Module 5: Policies Describes common types of privacy-related policies, outlines components and offers strategies for implementation.  Module 6: Data subject rights Discusses operational considerations for communicating and ensuring data subject rights, including privacy notice, choice and consent, access and rectification, data portability, and erasure and the right to be forgotten.  Module 7: Training and awareness Outlines strategies for developing and implementing privacy training and awareness programs.  Module 8: Protecting personal information Examines a holistic approach to protecting personal information through privacy by design.  Module 9: Data breach incident plans Provides guidance on planning for and responding to a data security incident or breach.  Module 10: Measuring, monitoring and auditing program performance Relates common practices for monitoring, measuring, analyzing and auditing privacy program performance Prerequisites   There are no prerequisites for this course but attendees would benefit from a review of the materials on the IAPP SITE What's Included?   1 years membership of the IAPP Breakfast, Lunch, mid-morning and afternoon snacks, teas, coffees Official Study Guides* Official Participant Guides* Official Exam Q&A's* Both exam fees * In electronic format for Live Online and hard copy for Classroom delegates     Who Should Attend?   This course is suitable for aspiring Data Protection Officers, as well as Information Security Managers, Lawyers, Data Managers, Analysts and Risk Teams. Provided by Our Guarantee   We are an approved IAPP Training Partner. You can learn wherever and whenever you want with our robust classroom and interactive online training courses. Our courses are taught by qualified practitioners with a minimum of 25 years commercial experience. We strive to give our delegates the hands-on experience. Our courses are all-inclusive with no hidden extras.  The one-off cost covers the training, all course materials, and exam voucher. Our aim: To achieve a 100% first time pass rate on all our instructor-led courses. Our Promise: Pass first time or 'train' again for FREE. *FREE training offered for retakes - come back within a year and only pay for the exam.

Duration 1 Days 6 CPD hours This course is intended for This course is intended for: Data platform engineers Architects and operators who build and manage data analytics pipelines Overview In this course, you will learn to: Compare the features and benefits of data warehouses, data lakes, and modern data architectures Design and implement a batch data analytics solution Identify and apply appropriate techniques, including compression, to optimize data storage Select and deploy appropriate options to ingest, transform, and store data Choose the appropriate instance and node types, clusters, auto scaling, and network topology for a particular business use case Understand how data storage and processing affect the analysis and visualization mechanisms needed to gain actionable business insights Secure data at rest and in transit Monitor analytics workloads to identify and remediate problems Apply cost management best practices In this course, you will learn to build batch data analytics solutions using Amazon EMR, an enterprise-grade Apache Spark and Apache Hadoop managed service. You will learn how Amazon EMR integrates with open-source projects such as Apache Hive, Hue, and HBase, and with AWS services such as AWS Glue and AWS Lake Formation. The course addresses data collection, ingestion, cataloging, storage, and processing components in the context of Spark and Hadoop. You will learn to use EMR Notebooks to support both analytics and machine learning workloads. You will also learn to apply security, performance, and cost management best practices to the operation of Amazon EMR. Module A: Overview of Data Analytics and the Data Pipeline Data analytics use cases Using the data pipeline for analytics Module 1: Introduction to Amazon EMR Using Amazon EMR in analytics solutions Amazon EMR cluster architecture Interactive Demo 1: Launching an Amazon EMR cluster Cost management strategies Module 2: Data Analytics Pipeline Using Amazon EMR: Ingestion and Storage Storage optimization with Amazon EMR Data ingestion techniques Module 3: High-Performance Batch Data Analytics Using Apache Spark on Amazon EMR Apache Spark on Amazon EMR use cases Why Apache Spark on Amazon EMR Spark concepts Interactive Demo 2: Connect to an EMR cluster and perform Scala commands using the Spark shell Transformation, processing, and analytics Using notebooks with Amazon EMR Practice Lab 1: Low-latency data analytics using Apache Spark on Amazon EMR Module 4: Processing and Analyzing Batch Data with Amazon EMR and Apache Hive Using Amazon EMR with Hive to process batch data Transformation, processing, and analytics Practice Lab 2: Batch data processing using Amazon EMR with Hive Introduction to Apache HBase on Amazon EMR Module 5: Serverless Data Processing Serverless data processing, transformation, and analytics Using AWS Glue with Amazon EMR workloads Practice Lab 3: Orchestrate data processing in Spark using AWS Step Functions Module 6: Security and Monitoring of Amazon EMR Clusters Securing EMR clusters Interactive Demo 3: Client-side encryption with EMRFS Monitoring and troubleshooting Amazon EMR clusters Demo: Reviewing Apache Spark cluster history Module 7: Designing Batch Data Analytics Solutions Batch data analytics use cases Activity: Designing a batch data analytics workflow Module B: Developing Modern Data Architectures on AWS Modern data architectures

Duration 4 Days 24 CPD hours This course is intended for This class is intended for experienced developers who are responsible for managing big data transformations including: Extracting, loading, transforming, cleaning, and validating data. Designing pipelines and architectures for data processing. Creating and maintaining machine learning and statistical models. Querying datasets, visualizing query results and creating reports Overview Design and build data processing systems on Google Cloud Platform. Leverage unstructured data using Spark and ML APIs on Cloud Dataproc. Process batch and streaming data by implementing autoscaling data pipelines on Cloud Dataflow. Derive business insights from extremely large datasets using Google BigQuery. Train, evaluate and predict using machine learning models using TensorFlow and Cloud ML. Enable instant insights from streaming data Get hands-on experience with designing and building data processing systems on Google Cloud. This course uses lectures, demos, and hand-on labs to show you how to design data processing systems, build end-to-end data pipelines, analyze data, and implement machine learning. This course covers structured, unstructured, and streaming data. Introduction to Data Engineering Explore the role of a data engineer. Analyze data engineering challenges. Intro to BigQuery. Data Lakes and Data Warehouses. Demo: Federated Queries with BigQuery. Transactional Databases vs Data Warehouses. Website Demo: Finding PII in your dataset with DLP API. Partner effectively with other data teams. Manage data access and governance. Build production-ready pipelines. Review GCP customer case study. Lab: Analyzing Data with BigQuery. Building a Data Lake Introduction to Data Lakes. Data Storage and ETL options on GCP. Building a Data Lake using Cloud Storage. Optional Demo: Optimizing cost with Google Cloud Storage classes and Cloud Functions. Securing Cloud Storage. Storing All Sorts of Data Types. Video Demo: Running federated queries on Parquet and ORC files in BigQuery. Cloud SQL as a relational Data Lake. Lab: Loading Taxi Data into Cloud SQL. Building a Data Warehouse The modern data warehouse. Intro to BigQuery. Demo: Query TB+ of data in seconds. Getting Started. Loading Data. Video Demo: Querying Cloud SQL from BigQuery. Lab: Loading Data into BigQuery. Exploring Schemas. Demo: Exploring BigQuery Public Datasets with SQL using INFORMATION_SCHEMA. Schema Design. Nested and Repeated Fields. Demo: Nested and repeated fields in BigQuery. Lab: Working with JSON and Array data in BigQuery. Optimizing with Partitioning and Clustering. Demo: Partitioned and Clustered Tables in BigQuery. Preview: Transforming Batch and Streaming Data. Introduction to Building Batch Data Pipelines EL, ELT, ETL. Quality considerations. How to carry out operations in BigQuery. Demo: ELT to improve data quality in BigQuery. Shortcomings. ETL to solve data quality issues. Executing Spark on Cloud Dataproc The Hadoop ecosystem. Running Hadoop on Cloud Dataproc. GCS instead of HDFS. Optimizing Dataproc. Lab: Running Apache Spark jobs on Cloud Dataproc. Serverless Data Processing with Cloud Dataflow Cloud Dataflow. Why customers value Dataflow. Dataflow Pipelines. Lab: A Simple Dataflow Pipeline (Python/Java). Lab: MapReduce in Dataflow (Python/Java). Lab: Side Inputs (Python/Java). Dataflow Templates. Dataflow SQL. Manage Data Pipelines with Cloud Data Fusion and Cloud Composer Building Batch Data Pipelines visually with Cloud Data Fusion. Components. UI Overview. Building a Pipeline. Exploring Data using Wrangler. Lab: Building and executing a pipeline graph in Cloud Data Fusion. Orchestrating work between GCP services with Cloud Composer. Apache Airflow Environment. DAGs and Operators. Workflow Scheduling. Optional Long Demo: Event-triggered Loading of data with Cloud Composer, Cloud Functions, Cloud Storage, and BigQuery. Monitoring and Logging. Lab: An Introduction to Cloud Composer. Introduction to Processing Streaming Data Processing Streaming Data. Serverless Messaging with Cloud Pub/Sub Cloud Pub/Sub. Lab: Publish Streaming Data into Pub/Sub. Cloud Dataflow Streaming Features Cloud Dataflow Streaming Features. Lab: Streaming Data Pipelines. High-Throughput BigQuery and Bigtable Streaming Features BigQuery Streaming Features. Lab: Streaming Analytics and Dashboards. Cloud Bigtable. Lab: Streaming Data Pipelines into Bigtable. Advanced BigQuery Functionality and Performance Analytic Window Functions. Using With Clauses. GIS Functions. Demo: Mapping Fastest Growing Zip Codes with BigQuery GeoViz. Performance Considerations. Lab: Optimizing your BigQuery Queries for Performance. Optional Lab: Creating Date-Partitioned Tables in BigQuery. Introduction to Analytics and AI What is AI?. From Ad-hoc Data Analysis to Data Driven Decisions. Options for ML models on GCP. Prebuilt ML model APIs for Unstructured Data Unstructured Data is Hard. ML APIs for Enriching Data. Lab: Using the Natural Language API to Classify Unstructured Text. Big Data Analytics with Cloud AI Platform Notebooks What's a Notebook. BigQuery Magic and Ties to Pandas. Lab: BigQuery in Jupyter Labs on AI Platform. Production ML Pipelines with Kubeflow Ways to do ML on GCP. Kubeflow. AI Hub. Lab: Running AI models on Kubeflow. Custom Model building with SQL in BigQuery ML BigQuery ML for Quick Model Building. Demo: Train a model with BigQuery ML to predict NYC taxi fares. Supported Models. Lab Option 1: Predict Bike Trip Duration with a Regression Model in BQML. Lab Option 2: Movie Recommendations in BigQuery ML. Custom Model building with Cloud AutoML Why Auto ML? Auto ML Vision. Auto ML NLP. Auto ML Tables.

Duration 2 Days 12 CPD hours This course is intended for This training is ideally suited for data analysts, IT professionals, and software developers who seek to augment their data processing and analytics capabilities. It will also benefit system administrators and data engineers who wish to harness Elastic Stack's functionalities for efficient system logging, monitoring, and robust data visualization. With a focus on practical application, this course is perfect for those aspiring to solve complex data challenges in real-time environments across diverse industry verticals. Overview This course combines engaging instructor-led presentations and useful demonstrations with valuable hands-on labs and engaging group activities. Throughout the course you'll explore: New features and updates introduced in Elastic Stack 7.0 Fundamentals of Elastic Stack including Elasticsearch, Logstash, and Kibana Useful tips for using Elastic Cloud and deploying Elastic Stack in production environments How to install and configure an Elasticsearch architecture How to solve the full-text search problem with Elasticsearch Powerful analytics capabilities through aggregations using Elasticsearch How to build a data pipeline to transfer data from a variety of sources into Elasticsearch for analysis How to create interactive dashboards for effective storytelling with your data using Kibana How to secure, monitor and use Elastic Stack's alerting and reporting capabilities The Elastic Stack is a powerful combination of tools for techniques such as distributed search, analytics, logging, and visualization of data. Elastic Stack 7.0 encompasses new features and capabilities that will enable you to find unique insights into analytics using these techniques. Geared for experienced data analysts, IT professionals, and software developers who seek to augment their data processing and analytics capabilities, Working with Elasticsearch will explore how to use Elastic Stack and Elasticsearch efficiently to build powerful real-time data processing applications. Throughout the two-day hands-on course, you?ll explore the power of this robust toolset that enables advanced distributed search, analytics, logging, and visualization of data, enabled by new features in Elastic Stack 7.0. You?ll delve into the core functionalities of Elastic Stack, understanding the role of each component in constructing potent real-time data processing applications. You?ll gain proficiency in Elasticsearch for distributed searching and analytics, Logstash for logging, and Kibana for compelling data visualization. You?ll also explore the art of crafting custom plugins using Kibana and Beats, and familiarize yourself with Elastic X-Pack, a vital extension for effective security and monitoring. The course also covers essentials like Elasticsearch architecture, solving full-text search problems, data pipeline building, and creating interactive Kibana dashboards. Learn how to deploy Elastic Stack in production environments and explore the powerful analytics capabilities offered through Elasticsearch aggregations. The course will also touch upon securing, monitoring, and utilizing Elastic Stack's alerting and reporting capabilities. Hands-on labs, captivating demonstrations, and interactive group activities enrich your learning journey throughout the course. Introducing Elastic Stack What is Elasticsearch, and why use it? Exploring the components of the Elastic Stack Use cases of Elastic Stack Downloading and installing Getting Started with Elasticsearch Using the Kibana Console UI Core concepts of Elasticsearch CRUD operations Creating indexes and taking control of mapping REST API overview Searching - What is Relevant The basics of text analysis Searching from structured data Searching from the full text Writing compound queries Modeling relationships Analytics with Elasticsearch The basics of aggregations Preparing data for analysis Metric aggregations Bucket aggregations Pipeline aggregations Substantial Lab and Case Study Analyzing Log Data Log analysis challenges Using Logstash The Logstash architecture Overview of Logstash plugins Ingest node Visualizing Data with Kibana Downloading and installing Kibana Preparing data Kibana UI Timelion Using plugins

Duration 1 Days 6 CPD hours This course is intended for This course is intended for the following participants:Cloud professionals interested in taking the Data Engineer certification exam.Data engineering professionals interested in taking the Data Engineer certification exam. Overview This course teaches participants the following skills: Position the Professional Data Engineer Certification Provide information, tips, and advice on taking the exam Review the sample case studies Review each section of the exam covering highest-level concepts sufficient to build confidence in what is known by the candidate and indicate skill gaps/areas of study if not known by the candidate Connect candidates to appropriate target learning This course will help prospective candidates plan their preparation for the Professional Data Engineer exam. The session will cover the structure and format of the examination, as well as its relationship to other Google Cloud certifications. Through lectures, quizzes, and discussions, candidates will familiarize themselves with the domain covered by the examination, to help them devise a preparation strategy. Rehearse useful skills including exam question reasoning and case comprehension. Tips and review of topics from the Data Engineering curriculum. Understanding the Professional Data Engineer Certification Position the Professional Data Engineer certification among the offerings Distinguish between Associate and Professional Provide guidance between Professional Data Engineer and Associate Cloud Engineer Describe how the exam is administered and the exam rules Provide general advice about taking the exam Sample Case Studies for the Professional Data Engineer Exam Flowlogistic MJTelco Designing and Building (Review and preparation tips) Designing data processing systems Designing flexible data representations Designing data pipelines Designing data processing infrastructure Build and maintain data structures and databases Building and maintaining flexible data representations Building and maintaining pipelines Building and maintaining processing infrastructure Analyzing and Modeling (Review and preparation tips) Analyze data and enable machine learning Analyzing data Machine learning Machine learning model deployment Model business processes for analysis and optimization Mapping business requirements to data representations Optimizing data representations, data infrastructure performance and cost Reliability, Policy, and Security (Review and preparation tips) Design for reliability Performing quality control Assessing, troubleshooting, and improving data representation and data processing infrastructure Recovering data Visualize data and advocate policy Building (or selecting) data visualization and reporting tools Advocating policies and publishing data and reports Design for security and compliance Designing secure data infrastructure and processes Designing for legal compliance Resources and next steps Resources for learning more about designing data processing systems, data structures, and databases Resources for learning more about data analysis, machine learning, business process analysis, and optimization Resources for learning more about data visualization and policy Resources for learning more about reliability design Resources for learning more about business process analysis and optimization Resources for learning more about reliability, policies, security, and compliance Additional course details: Nexus Humans Preparing for the Professional Data Engineer Examination training program is a workshop that presents an invigorating mix of sessions, lessons, and masterclasses meticulously crafted to propel your learning expedition forward. This immersive bootcamp-style experience boasts interactive lectures, hands-on labs, and collaborative hackathons, all strategically designed to fortify fundamental concepts. Guided by seasoned coaches, each session offers priceless insights and practical skills crucial for honing your expertise. Whether you're stepping into the realm of professional skills or a seasoned professional, this comprehensive course ensures you're equipped with the knowledge and prowess necessary for success. While we feel this is the best course for the Preparing for the Professional Data Engineer Examination course and one of our Top 10 we encourage you to read the course outline to make sure it is the right content for you. Additionally, private sessions, closed classes or dedicated events are available both live online and at our training centres in Dublin and London, as well as at your offices anywhere in the UK, Ireland or across EMEA.