Databricks Data Engineering with AWS

Build a production Lakehouse & Deploy with DAB and CI/CD

Who Should Take This Course?

Designed for experienced Data Engineers seeking depth, this course goes beyond basic feature walkthroughs. Learn to build, govern, orchestrate, and deploy enterprise-grade Lakehouse solutions on Databricks using modern engineering practices.

Course Curriculum

Designed for experienced Data Engineers to build enterprise-grade Lakehouse solutions on Databricks.

1. About the Course
  FREE PREVIEW
2. Course Prerequisites
  FREE PREVIEW
3. Download Course Material
1. Introduction to Data Engineering
  FREE PREVIEW
2. Apache Spark to Data Engineering Platform
  FREE PREVIEW
3. Introduction to Databricks Platform
  FREE PREVIEW
4. Databricks Platform Architecture
5. Databricks Platform Access - Paid vs Free
6. Creating Databricks Free Account
1. Overview of Databricks Premium Platform Access
2. Creating AWS Account
3. Creating AWS IAM User Account
4. Managing your AWS Cost
5. Creating AWS Databricks Service Contract
6. Creating AWS Databricks Serverless Workspace
7. Creating AWS Databricks Classic Workspace
8. Delete and Cleanup Databricks Workspace
1. Introduction to Databricks Workspace
2. Introduction to Databricks Notebooks
3. Notebook Magic Commands
4. Databricks Utilities and Widgets
5. How to Debug Notebooks
6. Workspace Files vs Git Folders
7. Databricks Compute Cluster
8. Check Your Knowledge
1. What is Delta Lake and why it matters
2. Creating and Managing Delta tables
3. Reading and writing Delta — batch and streaming
4. Time Travel — querying historical versions
5. OPTIMIZE, VACUUM and Data Retention
6. RESTORE and Rollback Strategies
7. MERGE INTO — the Delta Upsert Engine
8. DELETE, UPDATE and Idempotent writes
9. Schema Enforcement and Schema Evolution
10. Type Widening and the Variant Data Type
11. Table constraints — NOT NULL, CHECK, and Identity Columns
12. Check Your Knowledge
1. Why Unity Catalog — Architecture
2. Implementing Unity Catalog Architecture
3. Catalogs, External Locations and Storage Credentials
4. Identity in Unity Catalog — Users, Groups, and Service Principals
5. Implementing Users, Groups, and Access Control
6. Unity Catalog Permissions Model - GRANT
7. Implementing Unity Catalog Permission for Catalog and Schema
8. Unity Catalog Permissions for Tables and Volumns
9. Check Your Knowledge
1. What is Medallion Architecture
2. Designing the Bronze Layer
3. Designing the Silver Layer
4. Designing the Gold Layer
5. Architecture Review and Design Decisions
6. Check Your Knowledge
1. What is Lakeflow Connect — The Ingestion Pillar
2. Ingesting Data to Bronze Layer from a SaaS Application
3. Incremental Ingestion from Database Using Query Based Connector
4. Incremental CDC from Database Using Managed Connector
5. File-Based Ingestion with Auto Loader
6. Incremental Ingestion, Schema Evolution and Bad Records in Production
7. Choosing the Right Ingestion Tool
8. Check Your Knowledge
1. What is Lakeflow SDP — Why it Replaces Manual Pipelines
2. Core concepts — Pipelines, Flows, Destinations, and the Python API
3. SDP API — Flow types, Destination Types, and Syntax
4. Building the Bronze layer — Auto Loader as a Streaming Table
5. Building the Silver layer — AUTO CDC and SCD Type 2
6. Building the Gold layer — Materialized Views and Incremental Refresh
7. SDP Design Decisions and Production Patterns
8. Check Your Knowledge
1. What is Lakeflow Jobs — and where it fits
2. Your first job — pipeline task, schedule, and job-level config
3. Parameterizing jobs — Passing Parameters to Notebook Task
4. Passing Parameters to Python Script Task, Task Value, Repair Runs
5. Multi-task DAGs — Control Flow, Retries, and Backfill
6. Automating Jobs — REST API and Databricks CLI
7. Jobs Design Decisions and Production Patterns
8. Check Your Knowledge
1. What are we building-StepRight Architecture Walkthrough
2. Project Structure - Planning the Initial Project Structure
3. Environment Setup - Repo, Catalog, Schema, and Volume
4. Test Data Strategy - Planning your Test Data Preparation
5. Seed the Project - Data Generator and Loader Notebook

About this course

81 lessons
19.5 hours of video content
PDF & Source Code
+91 93534 65988 (WhatsApp)

What You'll Learn

Build a strong foundation in Databricks Data Engineering on AWS by progressing from platform fundamentals to implementing a production-ready Lakehouse architecture.

By the end of this section, you'll have the foundational knowledge required to build and manage enterprise-grade Lakehouse solutions on Databricks using AWS, following modern engineering and governance practices.

Set up and navigate the Databricks Workspace on AWS.
Understand Delta Lake internals and implement reliable, ACID-compliant data lakes.
Govern enterprise data securely using Unity Catalog.
Design scalable Medallion Architecture for modern Lakehouse platforms.
Build robust ingestion pipelines with Lakeflow Connect.
Develop declarative data transformation pipelines using Lakeflow Spark Declarative Pipelines.
Orchestrate end-to-end workflows with Lakeflow Jobs.
Apply industry best practices through a real-world production project (StepRight) that brings all concepts together.

FAQ

How long can I access the course material?

Course access is available for the validity period selected during enrollment (1 yr or 3 yrs), including all updates released during your active subscription.
How do you provide support?

We provide support through our dedicated learner community, where you can ask questions, participate in discussions, and learn from fellow professionals. Our team actively monitors the community to assist with course-related queries whenever required.
Do you have a refund policy?

You may request a refund within 7 days of purchase or before completing 15% of the course, whichever comes first. A 6% payment processing fee will be deducted from the refund amount.

Related Courses:

Schedule a call with course coordinator for bundles, discounts and live sessions

View more courses