Azure Databricks & Spark for Data Engineers:Hands-on Project

Descripción

Course Fully Refreshed for 2026 This course has been completely rebuilt for 2026 using the latest Azure Databricks features and best practices. Instead of relying on legacy approaches such as Hive Metastore and external orchestration tools, this course focuses on modern Databricks capabilities like Unity Catalog, Lakeflow Jobs, Databricks SQL Dashboards, and Genie. Welcome! In this course, you will build a complete end-to-end data engineering project using Azure Databricks and Apache Spark based on Formula 1 Motor Racing data. You won’t just learn individual concepts. You will design and implement a cloud data platform from scratch, following the same approach used in real-world data engineering and data platform projects. What You Will Build Throughout the course, you will: • Design a modern Data Lakehouse architecture using Azure Databricks • Implement the Medallion Architecture (Bronze, Silver, Gold) for scalable data pipelines • Ingest, transform, and model data using Apache Spark (PySpark and Spark SQL) • Store and manage data using Delta Lake in Databricks • Organise and govern data using Unity Catalog in Azure Databricks • Build and orchestrate pipelines using Lakeflow Jobs in Databricks • Create analytical views and dashboards using Databricks SQL and Dashboards • Enhance the pipeline with incremental data processing using Delta Lake By the end of the course, you will have built a production-ready data engineering pipeline on Azure Databricks. Technologies You Will Use As part of building the project, you will learn: • Azure Databricks • Apache Spark using PySpark and Spark SQL • Delta Lake and modern Lakehouse architecture • Unity Catalog for data governance and organisation in Databricks • Databricks SQL and Dashboards for analytics and reporting How You Will Learn This is a hands-on, project-based Azure Databricks course. • You will build the solution step by step • Concepts are explained in the context of a real-world project • Each section builds on the previous one This approach ensures that you not only understand the concepts, but also know how to apply them in real-world data engineering scenarios. I value your time as much as I do mine. So, I’ve designed this course to be focused, practical, and to the point. The lessons are explained in simple English, without unnecessary jargon, and we start from the basics. By the end of the course, you will be confident building real-world data engineering solutions. How This Course Supports Certification Preparation This course can help you build many of the core skills required for the following certifications: • Databricks Certified Data Engineer Associate • Databricks Certified Associate Developer for Apache Spark • Microsoft Exam DP-750: Implementing Data Engineering Solutions Using Azure Databricks • Databricks Certified Data Engineer Professional The hands-on project will strengthen your practical understanding of key Databricks and Spark concepts tested in these exams. However, this course is not designed as a certification preparation course and does not cover all exam topics. What’s Included (and What’s Not) • This course focuses on core Spark and Databricks concepts • It does not cover Spark Streaming, Spark ML, and Lakeflow Declarative Pipelines • Spark is taught using PySpark and Spark SQL (not Scala or Java) Final Outcome By the end of this course, you will have built a complete, production-ready data engineering solution using Azure Databricks and Spark, and gained the confidence to apply these skills in real-world projects.

Lo que aprenderás

• You will learn how to build a real world data project using Azure Databricks and Spark Core. This course has been taught using real world data.
• You will acquire professional level data engineering skills in Azure Databricks, Delta Lake, Spark Core, Azure Data Lake Gen2 and Azure Data Factory (ADF)
• You will learn how to create notebooks, dashboards, clusters, cluster pools and jobs in Azure Databricks
• You will learn how to ingest and transform data using PySpark in Azure Databricks
• You will learn how to transform and analyse data using Spark SQL in Azure Databricks
• You will learn about Data Lake architecture and Lakehouse Architecture. Also, you will learn how to implement a Lakehouse architecture using Delta Lake.
• You will learn how to build and orchestrate data pipelines using Lakeflow Jobs in Databricks.
• You will learn how to implement incremental data processing using Delta Lake.
• You will gain a comprehensive understanding of Unity Catalog and how it is used to organise and manage data in Databricks.
• You will gain practical experience working with modern Databricks features and best practices used in real-world data engineering projects.
• You will build practical skills that support certifications such as Databricks Certified Data Engineer Associate and Databricks Certified Associate Developer fo
• You will strengthen your understanding of key concepts commonly tested in Databricks and Azure Databricks certification exams.

Requisitos

• All the code and step-by-step instructions are provided, but the skills below will greatly benefit your journey
• Basic Python programming experience will be required
• Basic SQL knowledge will be required
• Knowledge of cloud fundamentals will be beneficial, but not necessary
• Azure subscription will be required, If you don't have one we will create a free account in the course
• No prior experience with Azure Databricks is required.