Deepseek R1 Rewards EXPLAINED: A Complete Breakdown

Video Description

In this video, chris looks at how DeepSeek and O1 performs rule based ai reward modelling using verifiers in training DeepSeek R1 / O3 and how you can create your own rewards, to train your own reasoning based model. Verifiers is the key to building reinforcement learning based reasoning models. You will be able to create your own verifiers and train your own R1 style model locally by applying the same concepts. This video walks through various verifiers—format, accuracy, boxed, limerick, and more—to show how they shape and refine AI-generated outputs. You will also be able to extend the framework to build your own models on your local machine. We’ll also explore different sampling strategies (greedy vs. top-p), demonstrate how to generate custom verifier datasets, and reveal the step-by-step process of collecting prompts from a Deepseek teacher model. We use the same techniques as DeepSeek use in the paper to evoke long chain of thoughts from the models, which we can use to train our model in reasoning Watch as we fine-tune our model (SFT training) on these collected prompts, then evaluate its performance and chain-of-thought quality. Whether you’re an AI enthusiast or a seasoned practitioner, this deep dive into Deepseek verifiers and reward strategies will provide valuable insights into creating more reliable, nuanced, and creative AI outputs. Github repo for fine tuning: https://github.com/chrishayuk/chuk-finetune Github repo for verifiers: https://github.com/chrishayuk/verifiers Github repo for generating math synthetic data: https://github.com/chrishayuk/chuk-math Video on Cold Start Reasoning: https://www.youtube.com/watch?v=Pabqg33sUrg&t=4s Chapters ---- 00:00 - intro 00:53 - deepseek reward modelling 03:20 - format reward verifier 07:31 - accuracy reward verifier 09:43 - boxed reward verifier 12:11 - verifier answer verifier 13:39 - limerick verifier 16:25 - llm verifiers 18:29 - evoking outputs deepseek style 19:07 - greedy sampling 23:10 - top p sampling 30:00 - generating verifier datasets 33:00 - collecting prompts from teacher model (deepseek) 37:00 - sft training on collected prompts 37:33 - inferring from trained model 38:50 - chain of thought quality