Mamba - a replacement for Transformers?
Samuel Albanie
@samuelalbanie1About
AI research. Note that opinions expressed are my own. However, for conflict of interest purposes, please note that I'm employed by Google DeepMind (GDM). This almost certainly biases my judgment to some degree. I think GDM is pretty great. Other related content: - misc/outdoor stuff channel: https://www.youtube.com/@samuelalbaniemisc - https://x.com/SamuelAlbanie - https://bsky.app/profile/samuelalbanie.bsky.social FAQ: Software used to make videos - keynote on mac (to make slides) - Adobe Premiere Pro for editing
Latest Posts
No results found. Try different keywords.
Video Description
Mamba is a new neural network architecture proposed by Albert Gu and Tri Dao. Timestamps: 00:00 - Mamba - a replacement for Transformers? 00:19 - The Long Range Arena benchmark 01:20 - Legendre Memory Units 02:07 - HiPPO: Recurrent Memory with Optimal Polynomial Projections 02:38 - Combining Recurrent, Convolutional and Continuous-time Models with Linear State-Space Layers 03:28 - Efficiently Modeling Long Sequences with Structured State Spaces (S4) 05:46 - The Annotated S4 06:13 - Mamba: Linear-Time Sequence Modeling with Selective State Spaces 07:42 - Motivation: Why selection is needed 09:59 - S5 12:00 - Empirical evaluation The paper can be found here: https://arxiv.org/abs/2312.00752 Topics: #mamba #foundation References for papers mentioned in the video can be found at https://samuelalbanie.com/digests/2023-12-08-mamba/ For related content: - Twitter: https://twitter.com/SamuelAlbanie - personal webpage: https://samuelalbanie.com/ - YouTube: https://www.youtube.com/@SamuelAlbanie1
Transform Your Workflow Today
AI-recommended products based on this video
Cmokifuly GMK67 65% Mechanical Keyboard Kit Tri-Mode South-Facing RGB LED for 3/5pin Switches,66 Keys+1 Knob Hotswap Socket PCB Gasket Mounted Plate Gaming DIY Keyboard Kit (White)
MSI Gaming Backlit RGB, Dedicated Hotkeys, Anti-Ghosting, Water Resistant Mechanical Feel Gaming Keyboard, Crisp Typing +12 Million Click Life Span, Steady Gaming Base Support (Vigor GK30 US)
Asus ROG Azoth Wireless Mechanical Gaming Keyboard – 75% TKL, Tri-Mode Connectivity, Hot Swappable Pre-lubed ROG NX Snow Linear Switches, PBT Keycaps, RGB, OLED Display, PC/Mac Support - Black
Logitech G203 Wired Gaming Mouse, 8,000 DPI, Rainbow Optical Effect LIGHTSYNC RGB, 6 Programmable Buttons, On-Board Memory, Screen Mapping, PC/Mac Computer and Laptop Compatible - Black
Logitech G305 LIGHTSPEED Wireless Gaming Mouse, Hero 12K Sensor, 12,000 DPI, Lightweight, 6 Programmable Buttons, 250h Battery Life, On-Board Memory, PC/Mac - Black
Logitech K400 Plus Wireless Touch TV Keyboard With Easy Media Control and Built-in Touchpad, HTPC Keyboard for PC-connected TV, Windows, Android, Chrome OS, Laptop, Tablet - Black
Logitech M185 Wireless Mouse, 2.4GHz with USB Mini Receiver, 12-Month Battery Life, 1000 DPI Optical Tracking, Ambidextrous, Compatible with PC, Mac, Laptop - Black
T8Plus Mini PC W-11 Pro, 12th Gen Ιntel Alder Lake N100 (up to 3.40Ghz) Mini Computers,12GB DDR5 256GB M.2 SSD Mini Office Desktop PC, Dual Ethernet/4K HDMI*3, 2,4/5 G WiFi,BT4.2
Razer Universal Quick Charging Stand for Xbox Series X|S: Magnetic Secure Charging - Matches Forza Horizon 5 Limited Edition Xbox Wireless Controller - USB Powered - (Controller Sold Separately)
Logitech M185 Wireless Mouse, 2.4GHz with USB Mini Receiver, 12-Month Battery Life, 1000 DPI Optical Tracking, Ambidextrous, Compatible with PC, Mac, Laptop - Black
Logitech G203 Wired Gaming Mouse, 8,000 DPI, Rainbow Optical Effect LIGHTSYNC RGB, 6 Programmable Buttons, On-Board Memory, Screen Mapping, PC/Mac Computer and Laptop Compatible - Black
Logitech G305 LIGHTSPEED Wireless Gaming Mouse, Hero 12K Sensor, 12,000 DPI, Lightweight, 6 Programmable Buttons, 250h Battery Life, On-Board Memory, PC/Mac - Black
Logitech G502 HERO High Performance Wired Gaming Mouse, HERO 25K Sensor, 25,600 DPI, RGB, Adjustable Weights, 11 Programmable Buttons, On-Board Memory, PC / Mac, Black
Asus ROG Delta II Wireless Gaming Headset with Detachable Mic – Tri-Mode (BT, 2.4GHz, 3.5mm), 50mm Titanium-Plated Drivers, 110Hr Battery, Lightweight, for PC, PS5, PS4, Xbox, Switch, Mobile - Black
Turtle Beach Stealth 600 Wireless Multiplatform Amplified Gaming Headset for Xbox Series X|S, Xbox One, PC, PS5, PS4, Nintendo Switch, & Mobile – Bluetooth, 80-Hr Battery, Noise-Cancelling Mic – Black
Turtle Beach Recon 50 Gaming Headset for Nintendo Switch, Xbox Series X|S, Xbox One, PS5, PS4, PlayStation, Mobile, & PC with 3.5mm – Removable Mic, 40mm Speakers – Red/Blue
