• Follow Us On :

Course Content
Introduction to Data Engineering
0/12
Explore the role of a data engineer
Analyze data engineering challenges
Introduction to BigQuery
Data Lakes and Data Warehouses
Demo: Federated Queries with BigQuery
Transactional Databases vs Data Warehouses
Website Demo: Finding PII in your dataset with DLP API
Partner effectively with other data teams
Manage data access and governance
Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
Building a Data Lake
0/9
Introduction to Data Lakes
Data Storage and ETL options on GCP
Building a Data Lake using Cloud Storage
Optional Demo: Optimizing cost with Google Cloud Storage classes and Cloud Functions
Securing Cloud Storage
Storing All Sorts of Data Types
Video Demo: Running federated queries on Parquet and ORC files in BigQuery
Cloud SQL as a relational Data Lake
Lab: Loading Taxi Data into Cloud SQL
Building a Data Warehouse
0/16
The modern data warehouse
Introduction to BigQuery
Demo: Query TB+ of data in seconds
Getting Started
Loading Data
Video Demo: Querying Cloud SQL from BigQuery
Lab: Loading Data into BigQuery
Exploring Schemas
Demo: Exploring BigQuery Public Datasets with SQL using INFORMATION_SCHEMA
Schema Design
Nested and Repeated Fields
Demo: Nested and repeated fields in BigQuery
Lab: Working with JSON and Array data in BigQuery
Optimizing with Partitioning and Clustering
Demo: Partitioned and Clustered Tables in BigQuery
Preview: Transforming Batch and Streaming Data
Introduction to Building Batch Data Pipelines
0/6
EL, ELT, ETL
Quality considerations
How to carry out operations in BigQuery
Demo: ELT to improve data quality in BigQuery
Shortcomings
ETL to solve data quality issues
Executing Spark on Cloud Dataproc
0/5
The Hadoop ecosystem
Running Hadoop on Cloud Dataproc
GCS instead of HDFS
Optimizing Dataproc
Lab: Running Apache Spark jobs on Cloud Dataproc
Serverless Data Processing with Cloud Dataflow
0/8
Cloud Dataflow
Why customers value Dataflow
Dataflow Pipelines
Lab: A Simple Dataflow Pipeline (Python/Java)
Lab: MapReduce in Dataflow (Python/Java)
Lab: Side Inputs (Python/Java)
Dataflow Templates
Dataflow SQL
Manage Data Pipelines with Cloud Data Fusion and Cloud Composer
0/13
Building Batch Data Pipelines visually with Cloud Data Fusion
Components
UI Overview
Building a Pipeline
Exploring Data using Wrangler
Lab: Building and executing a pipeline graph in Cloud Data Fusion
Orchestrating work between GCP services with Cloud Composer
Apache Airflow Environment
DAGs and Operators
Workflow Scheduling
Optional Long Demo: Event-triggered Loading of data with Cloud Composer, Cloud Functions, Cloud Storage, and BigQuery
Monitoring and Logging
Lab: An Introduction to Cloud Composer
Introduction to Processing Streaming Data
0/1
Processing Streaming Data
Serverless Messaging with Cloud Pub/Sub
0/2
Cloud Pub/Sub
Lab: Publish Streaming Data into Pub/Sub
Cloud Dataflow Streaming Features
0/2
Cloud Dataflow Streaming Features
Lab: Streaming Data Pipelines
High-Throughput BigQuery and Bigtable Streaming Features
0/4
BigQuery Streaming Features
Lab: Streaming Analytics and Dashboards
Cloud Bigtable
Lab: Streaming Data Pipelines into Bigtable
Advanced BigQuery Functionality and Performance
0/7
Analytic Window Functions
Using WITH Clauses
GIS Functions
Demo: Mapping Fastest Growing Zip Codes with BigQuery GeoViz
Performance Considerations
Lab: Optimizing your BigQuery Queries for Performance
Optional Lab: Creating Date-Partitioned Tables in BigQuery
BigQuery: Advanced Features and Use Cases
0/20
Advanced SQL Capabilities: Using CTEs (Common Table Expressions) for modular queries
Advanced SQL Capabilities: Analytical functions for ranking, windowing, and statistical computations
Advanced SQL Capabilities: Perform advanced joins, unions, and subqueries
BigQuery BI Engine: Accelerating dashboards and reports using Business Intelligence Engine
BigQuery BI Engine: Integration with tools like Google Data Studio, Tableau, and Looker
Geospatial Data Analysis: Using GIS functions for spatial data queries
Geospatial Data Analysis: Demo: Visualizing geospatial insights with BigQuery GeoViz and Google Maps
BigQuery ML: Train and deploy machine learning models directly using SQL
BigQuery ML: Supported models: Linear regression, logistic regression, k-means clustering, and more
BigQuery ML: Lab: Predicting outcomes with regression and classification models in BigQuery ML
BigQuery Omni: Analyze data across multiple cloud platforms seamlessly
BigQuery Omni: Demo: Federated queries across AWS and Azure datasets
BigQuery Security: Manage data access controls with IAM roles
BigQuery Security: Implement row-level and column-level security for sensitive data
Optimization Techniques: Partitioning and clustering for performance tuning
Optimization Techniques: Use materialized views and caching for cost-efficient queries
Optimization Techniques: Monitoring and cost management with BigQuery’s billing reports
BigQuery Use Cases: Real-time analytics for e-commerce or IoT devices
BigQuery Use Cases: Enterprise data lake and warehouse integration
BigQuery Use Cases: Data science workflows for predictive analytics
Databricks
0/12
Introduction to Databricks and Its Architecture
Overview, features, benefits, and integration with Apache Spark
Setting Up and Managing Databricks Environment
Workspace navigation, cluster setup, and account configuration
Working with Databricks Notebooks
Notebook features, multi-language support, and collaboration workflows
Data Processing with Apache Spark in Databricks
Data ingestion, transformation, SQL, and performance optimization
Delta Lake and Advanced Data Management
Creating Delta tables, versioning, time travel, and BI tool integration
Cloud Integration with Databricks
Connecting to AWS, Azure, GCP for storage, ETL pipelines, and security compliance
GCP Data Engineer
Previous
Next