How Engineers and PMs should collaborate on Evals

Video Description

Join the AI Evals Course starting Jan 27, 2026: https://maven.com/parlance-labs/evals?promoCode=tf-yt-c4 . Every business is trying to implement AI - but how many are actually doing this successfully? You may have heard evals are the secret sauce to successful AI products (heck, you're already a step ahead of the curve by taking this course). The thing is - if what you're evaluating doesn't matter to the business, your AI product might be "really good", but it's optimizing for the wrong thing. In this session, I'll be sharing what the companies that actually have AI in production and successfully implemented evals have in common - they are representative of business metrics. We'll talk about adopting evals beyond a test harness (using proxy business metrics), communicating AI value with non-technical stakeholders, and how to build a business case for evals at your company. 0:00 | Introduction: Engineers & PMs in AI Introduction to the session on collaboration between engineers and product managers in the AI space. 0:49 | Speaker Intro: Aman Khan's Background Aman Khan introduces himself as an AI Product Manager at Arize, sharing his experience from self-driving cars and Spotify's machine learning platform. 1:22 | AI Product Manager Flavors (Venn Diagram) Aman outlines three types of AI Product Managers: AI Product PM, AI Platform PM, and AI Powered PM, discussing their distinct responsibilities and overlapping skill sets. 6:45 | The "Vibe Coding" Workflow (Initial Stage) A look at the early-stage AI product development workflow, where teams quickly build, "vibe check" outputs, and ship, often leading to user complaints. 9:05 | The "Write an Eval" Workflow Aman introduces a more structured evaluation workflow: build, write an eval, trust the eval, and look at the data to ensure product quality. 10:00 | Who Writes Evals & Labels? (The Core Debate) A deep dive into the critical question of who is responsible for writing evaluations and labeling data, highlighting the importance of domain expertise and collaboration between PMs and AI Engineers. 15:39 | 3 Crucial Questions for Your AI Team Aman presents three key questions for teams developing AI products: how many human labels are needed, what to do when evals are good but human labels disagree, and what to do when evals are good but business metrics decline. 25:28 | Evals in Development & Production Workflow Discussion on how robust evaluation processes in development and production can lead to more users and provide clearer insights into product performance. 29:29 | Upskilling for AI PMs (Aman's Maven Course) Aman shares insights on the evolving role of AI PMs and announces his upcoming Maven course, "Prototype to Production: The AI PM Playbook," designed to equip product leaders with practical AI skills.