Databricks Data Engineering Project for Beginners (FREE Account) | Azure Tutorial

Video Description

Join my group for people learning Azure :) https://www.skool.com/azure-skool-4999 👉 Get the resources (Github): https://www.lukejbyrne.com/c/azure-data-eng-e2e-repo Let's make an End-to-End Data Pipeline in Databricks. -------------------- 📊 Learn Data & AI (50% OFF): https://datacamp.pxf.io/yqr3jb 🤖 Early access to the Applied AI Community: https://kickofflabs.com/waitlist/f795d9da 📮 Join the Newsletter: https://lukejbyrne.com/subscribe 🔗 Follow me on Linkedin: https://linkedin.com/in/lukejbyrne -------------------- 🏅 RECOMMENDED COURSES: (All under one subscription) *Top AI Courses:* AI Engineer for Developers Course - https://datacamp.pxf.io/gO6WRB AI Engineer for Data Scientists Course - https://datacamp.pxf.io/XmnX4a Developing AI Applications Course - https://datacamp.pxf.io/Bny25L *Top Data Courses:* Data Engineer in Python - https://datacamp.pxf.io/aOAWNZ Associate Data Scientist in Python - https://datacamp.pxf.io/AP3Eg1 Data Analyst with Python - https://datacamp.pxf.io/GKVZbk Associate Data Analyst in SQL - https://datacamp.pxf.io/Z6KyV1 -------------------- 0:00 - Introduction 3:47 - What is Databricks and its Architecture 5:33 - Medallion Architecture Explained 10:47 - Setting up Databricks in Azure 19:04 - Access Control and Security 29:04 - Connecting Databricks to Storage 36:07 - Introduction to Databricks Notebooks 44:24 - Setting up Compute Resources 45:54 - Configuring Compute Resources 48:02 - Installing Required Libraries 50:34 - Setting Up Notebooks 53:05 - Bronze Notebook: Fetching Raw Data 1:00:00 - Verifying Data in Bronze Layer 1:02:14 - Silver Notebook: Data Transformation 1:10:09 - Gold Notebook: Advanced Transformations 1:18:01 - Writing to Gold Layer 1:23:19 - Creating a Workflow in Databricks 1:27:25 - Configuring Task Dependencies 1:30:34 - Testing the Workflow 1:33:27 - Conclusion and Next Steps -------------------- Business inquiries: [email protected]