Llama 4 From Scratch in PyTorch - Vision Language Models + MoE

Video Description

Code: https://github.com/priyammaz/PyTorch-Adventures/tree/main/PyTorch%20for%20NLP/Llama4 Today we do a full implementation of the Llama4 Model! This is basically a guided walkthrough of the Huggingface 🤗 implementation so all credit goes to the authors and the researchers from Meta of course! My goal here is to just take a look under the hood to see exactly whats going on! Now... to inference this model, you need... ahem... an H100 GPU... w/ Int4 Quantization!! Excuse Me?? Well, I obviously cannot inference it so we will do small test cases probing the model and seeing how tensors flow through all of it! (Also the downloaded weights for Scout is like 200GB... I miss the 7B model days) Sorry the video is a little rough (you get to see me debug annoying stuff), I just wanted to get this information out ASAP so its helpful for everyone starting to look at Llama4! I am learning this along with all of yall! Also, I promise to make individual videos going in depth into a lot of these topics (MoE, KV Cache, Rotary Embeddings, etc...) I just wanted to skip ahead a little! Resources incase you feel lost! @umarjamilai has incredible resources to learn about Llama! @AndrejKarpathy has an awesome video about reproducing GPT2! @SebastianRaschka has an entire series on LLMs! Happy Learning! Timestamps 00:00:00 Introduction 00:05:00 Whats New? 00:08:00 MoE + Shared Expert 01:16:38 1D Rotary Embeddings 01:48:10 RMSNorm 01:52:40 KV Cache 02:00:45 Grouped Query Attention (GQA) 02:11:04 Scalable Softmax 02:25:50 Text Decoder Layer 02:35:50 Llama4TextModel 02:44:00 Causal Mask 03:01:50 Chunked Attention Mask 03:15:00 Llama4ForCausalLM 03:18:29 Vision MLP 03:22:48 MultiModal Projection 03:25:35 PixelShuffle 03:37:00 2D Rotary Embeddings 03:57:40 Vision Attention 04:03:20 Vision Encoder Layer 04:07:08 Vision Encoder 04:09:30 UnfoldConvolution for Patch Embeddings 04:14:24 Llama4VisionModel 04:24:00 VisionLanguageModel (Put it Together!) 04:46:05 Testing a Forward Pass! Socials! X https://twitter.com/data_adventurer Instagram https://www.instagram.com/nixielights/ Linkedin https://www.linkedin.com/in/priyammaz/ Discord https://discord.gg/bU9fKymp 🚀 Github: https://github.com/priyammaz 🌐 Website: https://www.priyammazumdar.com/