Azure End-To-End Data Engineering Project for Beginners (FREE Account) | SQL DB Tutorial

Video Description

Join my group for people learning Azure :) https://www.skool.com/azure-skool-4999 ✅ Get the resources (Github) 👉 https://www.lukejbyrne.com/c/azure-data-eng-e2e-repo This project addresses a critical business need by building a comprehensive data pipeline on Azure. The goal is to extract customer and sales data from an on-premises SQL database, transform it in the cloud, and generate actionable insights through a Power BI dashboard. The dashboard will highlight key performance indicators (KPIs) related to gender distribution and product category sales, allowing stakeholders to filter and analyze data by date, product category, and gender. Tech Stack: On-Prem SQL DB, Data Factory, Data Lake Gen 2, Databricks, Synapse Analytics, Power BI, Entra ID (Active Directory), Key Vault -------------------- 📊 Learn Data & AI (50% OFF): https://datacamp.pxf.io/yqr3jb 🤖 Early access to the Applied AI Community: https://kickofflabs.com/waitlist/f795d9da 📮 Join the Newsletter: https://lukejbyrne.com/subscribe 🔗 Follow me on Linkedin: https://linkedin.com/in/lukejbyrne -------------------- 🏅 RECOMMENDED COURSES: (All under one subscription) *Top AI Courses:* AI Engineer for Developers Course - https://datacamp.pxf.io/gO6WRB AI Engineer for Data Scientists Course - https://datacamp.pxf.io/XmnX4a Developing AI Applications Course - https://datacamp.pxf.io/Bny25L *Top Data Courses:* Data Engineer in Python - https://datacamp.pxf.io/aOAWNZ Associate Data Scientist in Python - https://datacamp.pxf.io/AP3Eg1 Data Analyst with Python - https://datacamp.pxf.io/GKVZbk Associate Data Analyst in SQL - https://datacamp.pxf.io/Z6KyV1 -------------------- Further detail on setting up SSMS and SQL: https://youtu.be/z7o5Wju-PZg?si=QnMF0AVB5DxNf182 How to set up PowerBI without work email: https://www.youtube.com/watch?v=9RB5xic9BiY Mr Ks original vid: https://youtu.be/iQ41WqhHglk?si=drPsOQDhLy-gPIbN -------------------- Overview --- 00:00:00 - Introduction 00:01:18 - Setting Up the Azure Environment 00:05:45 - SQL Database Configuration 00:10:30 - Overview of Azure Data Lake Storage SSMS --- 00:15:23 - Configuring Azure Data Factory 00:25:11 - Copying Data from SQL to Data Lake 00:38:05 - Debugging Initial Pipeline Issues Azure Data Factory --- 00:45:13 - ForEach Activity in Azure Data Factory 00:55:30 - Testing the SQL-to-Bronze Pipeline 01:05:30 - Recap of SQL-to-Bronze Process 01:08:41 - Debugging the Pipeline 01:10:04 - Monitoring Pipeline Runs 01:10:28 - Verifying Data in Bronze Layer 01:11:14 - Completion of the Bronze Data Layer Databricks --- 01:11:53 - Starting Databricks Configuration 01:14:43 - Creating a Databricks Cluster 01:17:29 - Mounting Data Lake Storage in Databricks 01:23:00 - Transformation in Databricks (Bronze to Silver) 01:33:06 - Automating Data Transformations 01:37:03 - Integrating Databricks with Data Factory 01:41:33 - Pipeline Testing and Monitoring Synapse Analytics --- 01:45:25 - Loading Data into Synapse Analytics 01:50:07 - Creating Views in Synapse 01:54:40 - Integrating Synapse Views into Data Factory Pipelines Power BI --- 01:57:57 - Power BI Dashboard Setup 02:03:11 - Building Relationships in Power BI 02:06:48 - Dashboard Filters and Slicers 02:10:01 - Publishing and Sharing Power BI Dashboards Automation and Active Directory --- 02:13:03 - Automating the Entire Pipeline 02:17:11 - Active Directory (Entra ID) Integration 02:21:33 - Triggering and Monitoring Automated Pipelines 02:29:43 - Final Dashboard Refresh and Validation Closing --- 02:30:07 - Closing Remarks and Next Steps -------------------- Business inquiries: [email protected]