Mixture of Experts: How LLMs get bigger without getting slower

Julia Turc • April 23, 2025

Julia Turc

About

AI explainer videos from a former Google Research engineer, now startup founder. Expect American optimism with a solid dose of Eastern European cynicism. Find (free) supplemental materials (slide decks, paper reading lists, Colab notebooks) on Patreon 👉 https://www.patreon.com/c/JuliaTurc

Mixture of Experts: How LLMs get bigger without getting slower

Julia Turc

About

Latest Posts

Transformers & Diffusion LLMs: What's the connection?

Text diffusion: A new paradigm for LLMs

The physics behind diffusion models

How LLMs survive in low precision | Quantization Fundamentals

You May Also Like

Upgrade Your LLM Setup Now

MSI Ultra-Slim Thin 15 VR-Ready High FPS Gaming Laptop, 15.6 FHD 144Hz, Intel Core i5-13420H, NVIDIA GeForce RTX 4060, 32GB RAM, 2TB SSD, Backlit KB, Wi-Fi 6, Bundle with PCO Notebook Fold Radiator

acer Nitro 50 N50-620-UA91 Gaming Desktop | 11th Gen Intel Core i5-11400F 6-Core Processor | NVIDIA GeForce GTX 1650 | 8GB DDR4 | 512GB NVMe M.2 SSD | Intel Wi-Fi 6 AX201 | Keyboard and Mouse

GIGABYTE - AORUS Elite 16 Gaming Laptop - 165Hz 2560x1600 WQXGA - NVIDIA GeForce RTX 5070 - Intel Core Ultra 9 275HX - 1TB SSD with 32GB DDR5 RAM - Windows 11 Home AD (AORUS Elite 16 BWHC3USC64SH)

Seasonic Focus V4 GX-1000 (ATX3) - 1000W - 80+ Gold - ATX 3.0 &amp; PCIe 5.1 Ready -Full-Modular -ATX Form Factor -Premium Japanese Capacitor -10 Year Warranty -Nvidia RTX 30/40 Super &amp; AMD GPU Compatible

Seasonic Focus V4 GX-1000 (ATX3) - 1000W - 80+ Gold - ATX 3.0 &amp; PCIe 5.1 Ready -Full-Modular -ATX Form Factor -Premium Japanese Capacitor -10 Year Warranty -Nvidia RTX 30/40 Super &amp; AMD GPU Compatible

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge R660XS R760XS

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge XE9680

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge R6625 R760 R7615 R7625

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge HS5610 HS5620

Crucial 64GB DDR5 RAM, 5600MHz (or 5200MHz or 4800MHz) Desktop Memory Kit, UDIMM 288-Pin, Compatible with 13th Gen Intel Core and AMD Ryzen 7000 - CT2K32G56C46U5

BOSGAME P3 Lite Mini PC, AMD Ryzen 7 6800H Mini Computer (8C/16T, up to 4.7GHz), 24GB DDR4 1TB M.2 NVMe SSD, Triple Display Desktop Gaming PC, Pre-in Win 11 Pro, Dual LAN, WiFi 6E, BT 5.2

ORIGIMAGIC N1 Mini PC, Ryzen 7 5825U(8C/16T, UP to 4.5GHz) 32GB RAM 1TB M.2 NVMe SSD, Mini Computer 4K Quad Display/Dual USB-C(Support Data/PD3.0/DP) /WiFi 6E/BT5.2, Light Gaming/Office

ORIGIMAGIC N1 Mini PC Ryzen 7 5825U(8C/16T, UP to 4.5GHZ), Mini Computers 16GB DDR4 RAM 512GB M.2 NVMe SSD, Support 4K Quad Display/WiFi 6E/BT5.2/2.5G LAN/Dual USB-C(Data/PD3.0/DP),Home/Office

origimagic N1 Gaming Mini PC, AMD Ryzen 7 5825U(8C/16T, UP to 4.5GHz) 32GB RAM 1TB M.2 NVMe SSD, Mini Computer Support 4K Quad Display/WiFi 6E/BT5.2/Dual USB-C(Support Data/PD3.0/DP) Gaming/Office

SAMSUNG 870 EVO SATA SSD 500GB 2.5” Internal Solid State Drive, Upgrade PC or Laptop Memory and Storage for IT Pros, Creators, Everyday Users, MZ-77E500B/AM [Canada Version]

SAMSUNG 870 EVO SATA III SSD 4TB 2.5” Internal Solid State Drive, Upgrade PC or Laptop Memory and Storage for IT Pros, Creators, Everyday Users, MZ-77E4T0B/AM [Canada Version]

Samsung 990 EVO Plus - 4TB PCIe Gen4. X4, Gen5. X2 NVMe 2.0 - M.2 Internal SSD, Speed Up to 7,250 MBs, Upgrade Storage for PC-Laptops, HMB Technology and Intelligent Turbowrite (MZ-V9S4T0B/AM)

SAMSUNG EVO Select Micro SD-Memory-Card + Adapter, 128GB microSDXC 160MB/s Full HD &amp; 4K UHD, UHS-I, U3, A2, V30, for Android Smartphones, Tablets, Nintendo-Switch (MB-ME128SA/AM) [Canada Version]

130W USB C Laptop Charger Compatible with Dell Precision 5550 5560 3560 3590 3550 5540 5570 5750 5530 5520 5510 Latitude 5520 5510 5420 7410 7310 7210 9410 9510 9520 XPS 15 17 AC Adapter Power Cord

UCANUUP 130W Tip 4.5mm Compatible with Dell XPS 15 7590 9530 9550 9560 9570 9575 Inspiron 7348 Precision M3800 5510 5520 5530 06TTY6 AC Adapter Power Replacement Supply Cord

Replacement for Dell 130W Laptop Charger USB C - XPS 17 15 7590 9700 9500 9510 Precision 5560 3560 5540 5570 5550 3561 3550 5510 5520 Latitude 7410 7310 7210 Type C Computer AC Adapter Power Cord

Loading...

Seasonic Focus V4 GX-1000 (ATX3) - 1000W - 80+ Gold - ATX 3.0 & PCIe 5.1 Ready -Full-Modular -ATX Form Factor -Premium Japanese Capacitor -10 Year Warranty -Nvidia RTX 30/40 Super & AMD GPU Compatible

Seasonic Focus V4 GX-1000 (ATX3) - 1000W - 80+ Gold - ATX 3.0 & PCIe 5.1 Ready -Full-Modular -ATX Form Factor -Premium Japanese Capacitor -10 Year Warranty -Nvidia RTX 30/40 Super & AMD GPU Compatible

SAMSUNG EVO Select Micro SD-Memory-Card + Adapter, 128GB microSDXC 160MB/s Full HD & 4K UHD, UHS-I, U3, A2, V30, for Android Smartphones, Tablets, Nintendo-Switch (MB-ME128SA/AM) [Canada Version]