How to build and automate a python ETL pipeline with airflow on AWS EC2 | Data Engineering Project

Video Description

In this data engineering project, we will learn how to build and automate an ETL process that can extract current weather data from open weather map API, transform the data and load the data into an S3 bucket using Apache Airflow. Apache Airflow is an open-source platform used for orchestrating and scheduling workflows of tasks and data pipelines. This project will be entirely carried out on AWS cloud platform. We will cover the fundamental concepts of Apache Airflow such as DAG and Operators and I will show you how to install Apache airflow from scratch and schedule your ETL pipeline. I will also show you how to use sensor in your ETL pipeline. As this is a hands-on project, I highly encourage you to first watch the video in its entirety without following along so that you can better understand the concepts and the workflows after which you should either try to replicate the example I showed without watching the video but consult the video when you are stuck or you could watch the video again the second time in its entirety while also following along this time. Remember the best way to learn is by doing it yourself – Get your hands dirty! If you have any questions or comments, ok to ask or leave comments in the comment section below. *Books I recommend* 1. Grit: The Power of Passion and Perseverance https://amzn.to/3EZKSgb 2. Think and Grow Rich!: The Original Version, Restored and Revised: https://amzn.to/3Q2K68s 3. The Book on Rental Property Investing: How to Create Wealth With Intelligent Buy and Hold Real Estate Investing: https://amzn.to/3LLpXRy 4. How to Invest in Real Estate: The Ultimate Beginner's Guide to Getting Started: https://amzn.to/48RbuOb 5. Introducing Python: Modern Computing in Simple Packages https://amzn.to/3Q4driR 6. Python for Data Analysis: Data Wrangling with pandas, NumPy, and Jupyter 3rd Edition: https://amzn.to/3rGF73G ***************** Commands used in this video ***************** sudo apt update sudo apt install python3-pip sudo apt install python3.10-venv python3 -m venv airflow_venv sudo pip install pandas sudo pip install s3fs sudo pip install apache-airflow airflow standalone sudo apt install awscli aws configure aws sts get-session-token ***************** USEFUL LINKS ***************** Extract current weather data from Open Weather Map API using python on AWS EC2: https://www.youtube.com/watch?v=0_caTDCZnd0&t=13s How to remotely SSH (connect) Visual Studio Code to AWS EC2: https://www.youtube.com/watch?v=sQQjMnEkGjs&t=1224s PostgreSQL Playlist: https://www.youtube.com/watch?v=oFaLUCWRnRE&list=PLACD_PaYcVF09khO58CISr08Uy6w3cAIF Weather Map API: https://openweathermap.org/api Github Repo: https://github.com/YemiOla/data_engineering_project_openweathermap_api_airflow_etl_aws Please don’t forget to LIKE, SHARE, COMMENT and SUBSCRIBE to our channel for more AWESOME videos. DISCLAIMER: This video and description has affiliate links. This means when you buy through one of these links, we will receive a small commission and this is at no cost to you. This will help support us to continue making awesome and valuable contents for you.