Mastering PySpark Programming

Master Data Engineering skills with PySpark | Beginner to Pro

What you'll learn

Mastering PySpark is a comprehensive course that will help you become proficient in PySpark Programming, Spark SQL, Dataframe APIs, Spark Architecture, Performance Tuning and Join Optimization, Advanced Concepts such as AQE, DPP, Memory Management and Unit Testing

PySpark Programming - Data Engineering and Data Processing using PySpark and Spark SQL
Spark Architecture - Understanding Spark internals, Performance optimization, Memory Management
Advanced Concepts - Data Sources and Sinks, Adaptive Query Execution, Dynamic Partition Pruning, Unit Testing

Course Prerequisite

What do you need to know before you start this course

Programming Knowledge Using Python Programming Language and SQL Fundamentals
A Recent 64-bit Windows/Mac Machine with 8 GB RAM & Internet Connection

Hear out instructor

Course Content

30 hours video - Capstone Project

1. Introduction to Apache Spark
2. Spark System Architecture
3. Spark Platform and Spark Development Environments
4. What is Databricks Platform
5. Create Databricks Free Account
6. Setup your hands-on environment
7. Download Resources
1. Starting Point - Spark Session
2. Dataframe - A View to Structured Data
3. Dataframe Transformations and Actions
4. Dataframe Concepts
5. Exploring Dataframe Transformations
6. Creating Spark Dataframe
7. Review Rating
1. Spark Data Types
2. Schema on read
3. Correcting Data Types
4. EDA and Schema Correction
1. Add Remove Rename Columns
2. Column Expressions
3. Filtering and Removing Duplicates
4. Sorting Limiting and Collecting
5. Unstructured Data Processing
6. Transforming data using LLM
1. Working with Nulls
2. Working with Numbers
3. Manipulating Strings
4. Working with Dates
5. Working with Timestamp
6. Handling Timezone Information
7. Working with Complex Types
8. Working with JSON Data
9. Working with VARIANT Types
1. Introduction to Joins
2. Inner Joins in Spark
3. Outer Joins in Spark
4. Lateral Joins in Spak
5. Other Types of Joins
1. Simple Aggregation
2. Grouping Aggregations
3. Multilevel Aggregations
4. Window Aggregations
1. Introduction to Spark UDF
2. Vectorized UDF
3. User Defined Table Functions
4. Unit Testing Spark Application
1. Setup Local Machine for Spark Development
2. Developing Spark in Local IDE
3. Spark Connect-Connecting to remote cluster
1. Project Scope and Background
2. Transformation Requirement
3. Setup your starter project
4. Download Starter Project
5. Download Final Project
6. Test Your Starter Project
7. Setup your source control and process
8. Creating Your Project CI/CD Pipeline
9. Develop Code
10. Write Test Cases
11. Working with Kafka Integration
12. Estimating resources for your application
13. An end is the new beginning
1. Before your start
2. Setup your Databricks Free Account
3. How to use Databricks Free Account for this course
4. Download Resources
1. Ch1: Downloadable Documents
2. Ch1: Downloadable Materials
3. Big Data - Overview and Objectives
  FREE PREVIEW
4. What is Big Data and How it Started
  FREE PREVIEW
5. What is Big Data and How it Started - Notes
6. Hadoop Architecture, History, and Evolution
7. Hadoop Architecture, History, and Evolution - Notes
8. What is Data Lake and How it works
9. What is Data Lake and How it works - Notes
10. Introducing Apache Spark and Databricks Cloud
  FREE PREVIEW
11. Introducing Apache Spark and Databricks Cloud - Notes
12. Spark Development Environment - Overview and Objectives
  FREE PREVIEW
13. Spark Development Environments
  FREE PREVIEW
14. Spark Development Environments - Notes
15. Setup your Databricks Community Cloud Environment
16. Setup your Databricks Community Cloud Environment - Notes
17. Introduction to Databricks Workspace
18. Introduction to Databricks Workspace - Notes
19. Create Your First Spark Application in Databricks Cloud
20. Create Your First Spark Application in Databricks Cloud - Notes
21. Setup Your Local Development IDE
22. Setup your Local Development IDE - Notes
23. Mac Users - Setup your Local Development IDE
24. Create Your First Spark Application using IDE
  FREE PREVIEW
25. Create Your First Spark Application using IDE - Notes
26. Ch1: Assignment
27. Ch1: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
28. Review Rating
1. Ch2: Downloadable Documents
2. Ch2: Downloadable Materials
3. Introduction to Spark Data Frames and Tables
4. Introduction to Spark Data Frames and Tables - Notes
5. Creating Spark Dataframe
6. Creating Spark Dataframe - Notes
7. Creating Spark Tables
8. Creating Spark Tables - Notes
9. Common Problem With Databricks Community
10. Common Problem With Databricks Community - Notes
11. Data Caching Essentials
12. Data Caching Essentials - Notes
13. Working With Spark SQL
14. Working With Spark SQL - Notes
15. Dataframe Transformations and Actions
16. Dataframe Transformations and Actions - Notes
17. Applying Transformations
18. Applying Transformations - Notes
19. Querying Spark Dataframe
20. Querying Spark Dataframe - Notes
21. More Dataframe Transformations
22. More Dataframe Transformations - Notes
23. Micro Project - Problem Statement
24. Micro Project Problem Statement - Notes
25. Ch2: Assignments
26. Ch2: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
27. Review Rating
1. Ch3: Downloadable Documents
2. Ch3: Downloadable Materials
3. Ch2: Assignment Solutions
4. Creating Spark Session
5. Creating Spark Session - Notes
6. Programmatically Creating Data Frames
7. Programmatically Creating Data Frames - Notes
8. Working with Spark Data Frame Schema
9. Working with Spark Data Frame Schema - Notes
10. Writing Spark Data Frames
11. Writing Spark Data Frames - Notes
12. Managed Vs External Tables
13. Managed Vs External Tables - Notes
14. Introducing Spark Web UI
15. Introducing Spark Web UI - Notes
16. Ch3: Assignment
17. Ch3: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
1. Ch4: Downloadable Documents
2. Ch4: Downloadable Materials
3. Ch3 Assignment Solutions
4. Working with Spark Data Frame Rows
5. Working with Spark Data Frame Rows - Notes
6. Working with Spark Data Frame Columns
7. Working with Spark Data Frame Columns - Notes
8. Working with Select and SelectExpr
9. Working with Select and SelectExpr - Notes
10. Working with Literal Values
11. Working with Literal Values - Notes
12. Adding, Renaming and Removing Columns
13. Adding, Renaming and Removing Columns - Notes
14. Filtering Records and Finding Unique Rows
15. Filtering Records and Finding Unique Rows - Notes
16. Ch4: Assignment
17. Ch4: Assignment Resource for DOWNLOAD
18. Ch4: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
1. Ch5: Downloadable Documents
2. Ch5: Downloadable Materials
3. Ch4 Assignment Solutions
4. Ch4 Assignment Solution Notebooks for DOWNLOAD
5. Sampling and Splitting
6. Sampling and Splitting - Notes
7. Sorting and Limiting
8. Sorting and Limiting - Notes
9. Nulls in Apache Spark
10. Nulls in Apache Spark - Notes
11. Working with Nulls
12. Working with Nulls - Notes
13. Spark Data Types
14. Spark Data Types - Notes
15. Casting and Aliases
16. Casting and Aliases - Notes
17. Ch5: Assignment
18. Ch5: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
1. Ch6: Downloadable Documents
2. Ch6: Downloadable Materials
3. Ch5: Assignment Solutions
4. Ch5: Assignment Solution Notebooks for DOWNLOAD
5. Working with Numbers
6. Working with Numbers - Notes
7. Boolean Expressions
8. Boolean Expressions - Notes
9. Manipulating Strings
10. Manipulating Strings - Notes
11. Working with Dates and Timestamps
12. Working with Dates and Timestamps - Notes
13. Working with Struct
14. Working with Struct - Notes
15. Working with Arrays
16. Working with Arrays - Notes
17. Working with Maps
18. Working with Maps - Notes
19. Ch6: Assignment
20. Ch6: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
1. Ch7: Downloadable Documents
2. Ch7: Downloadable Materials
3. Ch6: Assignment Solutions
4. Ch6: Assignment Solution Notebooks for DOWNLOAD
5. Simple Aggregations in Spark
6. Simple Aggregations in Spark - Notes
7. Grouping Aggregation in Spark
8. Grouping Aggregation in Spark - Notes
9. Windowing Aggregations in Spark
10. Windowing Aggregations in Spark - Notes
11. Top N Aggregation
12. Top N Aggregation - Notes
13. Pivoting and Aggregation
14. Pivoting and Aggregation - Notes
15. Ch7: Assignment
16. Ch7: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
1. Ch8: Downloadable Documents
2. Ch8: Downloadable Materials
3. Ch7: Assignment Solutions
4. Ch7: Assignment Solution Notebooks for DOWNLOAD
5. Introduction to Data Frame Joins
6. Introduction to Data Frame Joins - Notes
7. Outer Joins in Spark
8. Outer Joins in Spark - Notes
9. Data Frame Joins and Column Name Ambiguity
10. Data Frame Joins and Column Name Ambiguity - Notes
11. More on Data Frame Joins
12. More on Data Frame Joins - Notes
13. Ch8: Assignments
14. Ch8: Assignment Resource for Download
15. Ch8: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
1. Ch9: Downloadable Documents
2. Ch9: Downloadable Materials
3. Ch8: Assignment Solutions
4. Ch8: Assignment Solution Notebooks for DOWNLOAD
5. Introduction to Spark Data Sources and Sinks
6. Introduction to Spark Data Sources and Sinks - Notes
7. Understanding Spark Dataframe Reader
8. Understanding Spark Dataframe Reader - Notes
9. Reading CSV, JSON, Parquet, and AVRO File
10. Reading CSV, JSON, Parquet, and AVRO File - Notes
11. Reading Data with Schema-on-read
12. Reading Data with Schema-on-read - Notes
13. DataFrameReader Options for CSV, JSON and Others
14. DataFrameReader Options for CSV, JSON and Others - Notes
15. Understanding Spark DataframeWriter
16. Understanding Spark DataframeWriter - Notes
17. Writing CSV, JSON, Parquet, and AVRO Files
18. Writing CSV, JSON, Parquet, and AVRO Files - Notes
19. Managing Your Data Layout
20. Managing Your Data Layout - Notes
21. Reading Partitioned Dataframes
22. Reading Partitioned Dataframes - Notes
23. Ch9: Assignments
24. Ch9: Assignment Resource Download
25. Ch9: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
1. Ch10: Downloadable Documents
2. Ch10: Downloadable Materials
3. Ch9: Assignment Solutions
4. Ch9: Assignment Solution Notebooks for DOWNLOAD
5. Evolution of Spark APIs
6. Evolution of Spark APIs - Notes
7. Dataframes, RDDs and Distributed Partitions
8. Dataframes, RDDs and Distributed Partitions - Notes
9. Internals of Transformations
10. Internals of Transformations - Notes
11. Narrow and Wide Transformations
12. Narrow and Wide Transformations - Notes
13. Spark Jobs - Stages, Shuffle, Task, and Slots
14. Spark Jobs - Stages, Shuffle, Task, and Slots - Notes
15. Spark SQL Engine and Query Planning
16. Spark SQL Engine and Query Planning - Notes
17. Ch10: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
1. Ch11: Downloadable Documents
2. Ch11: Downloadable Materials
3. Understanding Spark Cluster
4. Understanding Spark Cluster - Notes
5. Running Spark Application on Cluster
6. Running Spark Application on Cluster - Notes
7. Spark Submit and Important Options
8. Spark Submit and Important Options - Notes
9. Spark Deployment Modes
10. Spark Deployment Modes - Notes
11. Resource Allocation to Spark Application
12. Resource Allocation to Spark Application - Notes
13. Spark Cluster and Runtime Architecture
14. Spark Cluster and Runtime Architecture - Notes
15. Ch11: Assignment
16. Ch11: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
1. Ch12: Downloadable Documents
2. Ch12: Downloadable Materials
3. Ch11: Assignment Solutions
4. Ch12: Internals of Spark Joins and Shuffle
5. Ch12: Internals of Spark Joins and Shuffle - Notes
6. Ch12: Optimizing Your Joins Using Broadcast
7. Ch12: Optimizing Your Joins Using Broadcast - Notes
8. Implementing Bucket Joins
9. Implementing Bucket Joins - Notes
10. Spark Adaptive Query Execution
11. Spark Adaptive Query Execution - Notes
12. Spark AQE Dynamic Join Optimization
13. Spark AQE Dynamic Join Optimization - Notes
14. Handling Data Skew in Spark Joins
15. Handling Data Skew in Spark Joins - Notes
16. Spark Dynamic Partition Pruning
17. Spark Dynamic Partition Pruning - Notes
18. Ch12: Assignment
19. Ch12: Assignment Resource for Download
20. Ch12: Test Your Knowledge [Requires 60% passing grade - Retake allowed]
1. Ch13: Downloadable Documents
2. Spark Memory Allocation
3. Spark Memory Allocation - Notes
4. Spark Memory Management
5. Spark Memory Management - Notes
6. Data Caching in Spark
7. Data Caching in Spark - Notes
8. Data Frame Hints
9. Data Frame Hints - Notes
10. Repartition Vs Coalesce
11. Repartition Vs Coalesce - Notes
12. Broadcast Variables in Apache Spark
13. Broadcast Variables in Apache Spark - Notes
14. Spark Accumulators
15. Spark Accumulators - Notes
16. Spark Speculative Execution
17. Spark Speculative Execution - Notes
18. Dynamic Resource Allocation in Spark
19. Dynamic Resource Allocation in Spark - Notes
20. Spark Schedulers
21. Spark Schedulers - Notes
1. Ch14: Downloadable Documents
2. Ch14: Downloadable Materials
3. Creating and Using UDFs
4. Creating and Using UDFs - Notes
5. Unit Testing in Spark
6. Unit Testing in Spark - Notes

Course Features

Hands-On Learning
Source Code & Data
Total Support

Course Reviews

In-Depth and Comprehensive!

Siva Das

This course covers everything you need to know about PySpark. The depth and clarity are remarkable, making complex topics easy to understand. Highly recommend it for anyone serious about mastering PySpark!

Read Less

Excellent course to become the master on PySpark at enterprise level

Amarnath Reddy Danthuluru

Excellent course to become the master on PySpark at enterprise level

Read Less