TensorRT vs vLLM on DGX Spark: Why Benchmarks Alone Don’t Work Part II.

Video Description

40 tokens per second is useless if you lose your train of thought waiting 4 minutes for the model to load.** Project Gepetto Lock Entry 02: We push the NVIDIA DGX Spark to its absolute limits. With the new software updates, NIVIDIAS DGX Spark finally got native support for **NVFP4 quantization**. The promise? Massive speed and reduced memory usage. I wanted to floor it. I wanted to replace my reliable Ollama setup with a high-performance TensorRT-LLM stack. The benchmarks looked incredible: 39.5 tok/s on a 30B model. But then reality hit. We discovered that raw speed comes with a massive "commitment tax." We ran into the "Configuration Wall," struggled with the open *MXFP4* standard on the massive **GPT-OSS-120B**, and learned a hard lesson about software maturity vs. hardware capability. *In this video, we debug the assumptions of Local AI:* *The Productive Stack:* Why we use Qwen3, Phi-4, and Llama-3.3 for different cognitive gears. *The Crash:* How running 3 TensorRT containers in parallel collapsed performance by 300%. *The vLLM Surprise:* Why the "industry darling" failed at first (110GB VRAM leak) but redeemed itself with the 120B Architect model. This is not a benchmark review. This is a field report on engineering a thinking environment that actually works for me. --- *⏱️ Timestamps*(For the complete video) 0:00 - Intro: Explorer vs. Caretaker 0:19 - Act I. - The Itch 0:55 - INTERMEZZO - The New Landscape 1:35 - Act II. - One human, many gears 4:21 - Act IIa. - The Euphoric Part 7:10 - Act 2b. - The Clash of the Architects 9:10 - Act 3. - The configuration wall 10:57 - Final Curtain --- *🛠️ The Stack & Hardware* *System:* NVIDIA DGX Spark (Blackwell Architecture, 128GB Unified Memory) *Worker Fast:* Qwen3-30B-A3B (NVFP4) - MoE Throughput King *Worker Heavy:* Qwen3-32B (NVFP4) - Dense Anchor *Thinker:* Phi-4-Reasoning-Plus (NVFP4) - Logic Specialist *Architect:* GPT-OSS-120B (MXFP4) & Llama-3.3-70B(NVFP4) *Runtimes tested:* TensorRT-LLM (v0.12.0rc6), vLLM (v25.12.post1-py3) --- *🔗 Links & Resources* NVIDIA Spark Playbook vLLM: https://build.nvidia.com/spark/vllm NVIDIA Spark Playbook Tensor RT: https://build.nvidia.com/spark/trt-llm Full Episode (Building Stability): https://www.youtube.com/watch?v=31jBDLEV7Mg #AI #NVIDIA #MachineLearning #Engineering #DevLog #TensorRT #vLLM #DGXSpark