Build a small language model from scratch: Data pre-processing

Vizuara • October 1, 2025

View Channel

About

No channel description available.

Latest Posts

PT4M

Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models

Vizuara4 months ago

6045

PT4M

The Principles of Diffusion Models - New Course Launch

Vizuara5 months ago

6294

PT4M

Introduction to Vision Language Models (VLM)

Vizuara5 months ago

9493

PT4M

How the VLLM inference engine works?

Vizuara7 months ago

10668

Video Description

Dr. Raj Dandekar (MIT PhD) conducted a 7-hour small language model workshop. This is part 1 of that workshop. In this, we cover the following four aspects: 1. 00:00 Introduction to SLM 2. 40:15 Dataset loading 3. 53:36 Tokenization 4. 01:24:42 Creating input-output pairs If you want to get access to the detailed lecture notes, register here: https://vizuara.ai/courses/build-slm-from-scratch-workshop-pro/

Build a small language model from scratch: Data pre-processing

About

Latest Posts

Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models

The Principles of Diffusion Models - New Course Launch

Introduction to Vision Language Models (VLM)

How the VLLM inference engine works?

Video Description

You May Also Like

How the VLLM inference engine works?

I pre-trained Gemma3 270M from scratch

MobileNet - Deep Learning that fits in your pocket | Convolutional Neural Network | Computer Vision

Multi-Head Latent Attention From Scratch | One of the major DeepSeek innovation

I left USA 2 days after finishing PhD at MIT. Why? Story of an IIT-MIT graduate

Lecture 2: Large Language Models (LLM) Basics

Lecture 7 - Explainable AI on image data using LIME | Colab implementation | CNN explained

Boost Your Language Model Project

AocBook 15.6&#x27;&#x27; FHD Laptop, Intel N95, Nvidia GTX 1060 4GB, 32GB DDR4 RAM, M.2 SSD, Sleek Notebook with Type-C, HDMI, RJ45 Ethernet, Backlit Keyboard, Fingerprint (32GB DDR4 | 1TB SSD)

acer Nitro 50 N50-620-UA91 Gaming Desktop | 11th Gen Intel Core i5-11400F 6-Core Processor | NVIDIA GeForce GTX 1650 | 8GB DDR4 | 512GB NVMe M.2 SSD | Intel Wi-Fi 6 AX201 | Keyboard and Mouse

Skytech Archangel Gaming PC Desktop – AMD Ryzen 5 3600 3.6 GHz, NVIDIA RTX 3060, 1TB NVME SSD, 16GB DDR4 RAM 3200, 600W Gold PSU, 11AC Wi-Fi, Windows 11 Home 64-bit

Skytech Archangel Gaming PC Desktop – AMD Ryzen 5 3600 3.6 GHz, NVIDIA RTX 3060, 1TB NVME SSD, 16GB DDR4 RAM 3200, 600W Gold PSU, 11AC Wi-Fi, Windows 11 Home 64-bit

Skytech Blaze 3.0 Gaming PC Desktop – Intel Core i5 12400F 2.5 GHz, NVIDIA RTX 3060, 500GB NVME SSD, 16GB DDR4 RAM 3200, 600W Gold PSU, 11AC Wi-Fi, Windows 11 Home 64-bit

Beelink SER3 Mini PC, AMD Ryzen 3 3200U(14nm, 2C/4T) up to 3.5GHz, Mini Gaming Computer 16GB DDR4 RAM 500GB PCIE3.0 X4 SSD, Micro PC 4K@60Hz Dual Display, Mini Computer WiFi6/BT5.2/HTPC/W-11 Pro

Beelink SER3 Mini PC, AMD Ryzen 3 3200U(up to 3.5GHz) 2C/4T, Mini Computer 16GB DDR4 RAM 500GB SSD, Mini Desktop Computer 4K@60Hz Dual HDMI Display WiFi6/BT5.2/WOL/HTPC/W-11 Pro

Samsung 990 EVO Plus - 4TB PCIe Gen4. X4, Gen5. X2 NVMe 2.0 - M.2 Internal SSD, Speed Up to 7,250 MBs, Upgrade Storage for PC-Laptops, HMB Technology and Intelligent Turbowrite (MZ-V9S4T0B/AM)

SAMSUNG 870 EVO SATA SSD 500GB 2.5” Internal Solid State Drive, Upgrade PC or Laptop Memory and Storage for IT Pros, Creators, Everyday Users, MZ-77E500B/AM [Canada Version]

SAMSUNG 870 EVO SATA III SSD 4TB 2.5” Internal Solid State Drive, Upgrade PC or Laptop Memory and Storage for IT Pros, Creators, Everyday Users, MZ-77E4T0B/AM [Canada Version]

SAMSUNG EVO Select Micro SD-Memory-Card + Adapter, 128GB microSDXC 160MB/s Full HD &amp; 4K UHD, UHS-I, U3, A2, V30, for Android Smartphones, Tablets, Nintendo-Switch (MB-ME128SA/AM) [Canada Version]

Western Digital 2TB WD Blue SA510 SATA Internal Solid State Drive SSD - SATA III 6 Gb/s, 2.5&quot;/7mm, Up to 560 MB/s - WDS200T3B0A

Western Digital Black 2TB P10 Game Drive for Xbox One, Portable External Hard Drive with 1-Month Xbox Game Pass - WDBA6U0020BBK-WESN

Anker USB C Cable, [2-Pack, 6 ft] Type C Charger Premium Nylon USB Cable, USB A to Type C Charging Cable Fast Charge for Samsung Galaxy S10 S10+ / Note 8, LG V20 and Other USB C Charger (Black)

Anker USB C Charger Cable (6ft 100W, 2Pack), USB 2.0 Type C Fast Charging Cable for iPhone 15 / 15Pro / 15Plus / 15ProMax MacBook Pro 2020, iPad Pro 2020, iPad Air 4, Samsung Galaxy S23+/S23 Ultra ClimatePartner certified

Anker Nano USB C Wall Charger,45W Fast Charging Smart Display Charger,with 180°Foldable Plug,Smart Recognition,Built-in Care Mode,for iPhone17/16/15 (Non-Battery,One USB-C Port,No Cable Included) ClimatePartner certified

Anker 332 USB-C Hub (5-in-1) with 4K HDMI Display, 5Gbps - and 2 5Gbps USB-A Data Ports and for MacBook Pro, MacBook Air, Dell XPS, Lenovo Thinkpad, HP Laptops and More

Logitech M185 Wireless Mouse, 2.4GHz with USB Mini Receiver, 12-Month Battery Life, 1000 DPI Optical Tracking, Ambidextrous, Compatible with PC, Mac, Laptop - Black

Logitech G203 Wired Gaming Mouse, 8,000 DPI, Rainbow Optical Effect LIGHTSYNC RGB, 6 Programmable Buttons, On-Board Memory, Screen Mapping, PC/Mac Computer and Laptop Compatible - Black

Logitech G305 Lightspeed Wireless Gaming Mouse, Hero 12K Sensor, 12,000 DPI, Lightweight, 6 Programmable Buttons, 250h Battery Life, On-Board Memory, PC/Mac - Black

Logitech G502 Hero High Performance Wired Gaming Mouse, Hero 25K Sensor, 25,600 DPI, RGB, Adjustable Weights, 11 Programmable Buttons, On-Board Memory, PC/Mac, Black

Dell 27 Plus 4K Monitor - S2725QS - 27-inch 4K (3840 x 2160) 120Hz 16:9 Display, IPS Panel, AMD FreeSync Premium, sRGB 99%, Integrated Speakers, 1500:1 Contrast Ratio, Comfortview Plus - Ash White

8GB MP4 Player with 5 Inch HD Touch Screen for Android, Portable Streaming MP4 Music Player HiFi Sound MP3 Player with Speaker and Camera,Expandable Up to 128GB (Black)

Mini Karaoke Machine for Kids Adults, Portable Bluetooth Speaker with 2 Wireless Microphones,Toys Birthday Gifts for Girls Ages 4, 5, 6, 7, 8,9,10,11,12+ Years Old Toddler Teens (Lightpink)

Monster Mini Karaoke Machine for Kids Adults, Portable Bluetooth Speakers with 2 Wireless Microphones and Lights, Christmas Birthday Gifts Toys for Boys Girls 4, 5, 6, 7, 8, 9, 10, 12+ Toddler

CORSAIR Hydro X Series iCUE Link XH405i Custom Cooling Kit – Hardline Water Cooling Loop – XC7 Elite CPU Water Block – XD5 Elite D5 Pump Res – XR5 360mm Radiator – 3X QX120 RGB Fans

Thermalright Aqua Elite 240 V3 Liquid CPU Cooler, 240 Cooling Row Size, 2 x 120mm PWM Fans, S-FDB Bearings,for AMD/AM4/AM5, Intel LGA1150/1151/1155/1156/1200/2011/1700,Desktop CPU Cooler AIO

MSI MAG CORELIQUID A15 240 - AIO ARGB CPU Liquid Cooler - 240mm Radiator - AM5, LGA 1851 Ready - Dual 120mm ARGB PWM Fans, Black

MSI MAG Coreliquid A13 360 - AIO ARGB CPU Liquid Cooler - 360mm Radiator - LGA 1851 Ready - Triple 120mm ARGB PWM Fans, Black

Loading...

AocBook 15.6'' FHD Laptop, Intel N95, Nvidia GTX 1060 4GB, 32GB DDR4 RAM, M.2 SSD, Sleek Notebook with Type-C, HDMI, RJ45 Ethernet, Backlit Keyboard, Fingerprint (32GB DDR4 | 1TB SSD)

SAMSUNG EVO Select Micro SD-Memory-Card + Adapter, 128GB microSDXC 160MB/s Full HD & 4K UHD, UHS-I, U3, A2, V30, for Android Smartphones, Tablets, Nintendo-Switch (MB-ME128SA/AM) [Canada Version]

Western Digital 2TB WD Blue SA510 SATA Internal Solid State Drive SSD - SATA III 6 Gb/s, 2.5"/7mm, Up to 560 MB/s - WDS200T3B0A