Deep Dive: Optimizing LLM inference

Video Description

Open-source LLMs are great for conversational applications, but they can be difficult to scale in production and deliver latency and throughput that are incompatible with your cost-performance objectives. In this video, we zoom in on optimizing LLM inference, and study key mechanisms that help reduce latency and increase throughput: the KV cache, continuous batching, and speculative decoding, including the state-of-the-art Medusa approach. Slides: https://fr.slideshare.net/slideshow/julien-simon-deep-dive-optimizing-llm-inference-69d3/270921961 ⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. Follow me on Medium at https://julsimon.medium.com or Substack at https://julsimon.substack.com. ⭐️⭐️⭐️ 00:00 Introduction 01:15 Decoder-only inference 06:05 The KV cache 11:15 Continuous batching 16:17 Speculative decoding 25:28 Speculative decoding: small off-the-shelf model 26:40 Speculative decoding: n-grams 30:25 Speculative decoding: Medusa