Google Cloud Professional Machine Learning Engineer Walkthrough 17 Retraining Triggers, Continuous T

Video Description

Google Cloud Professional Machine Learning Engineer Walkthrough 17 Retraining (Triggers, Continuous Training, Best Practices) - November 24 VIEW RECORDING: https://fathom.video/share/VD59wA4b_AAv3oTt7Cfi6hdA-zgLJdsW Meeting Purpose Review GCP machine learning retraining strategies and best practices. Key Takeaways - Retraining is essential because models trained on historical data degrade as the world changes (e.g., pandemics, recessions), requiring updates to reflect current reality. - Combine retraining triggers for optimal results: use a monthly schedule for baseline stability, but also trigger immediate retraining if data drift or performance drops below critical thresholds. - Automate retraining with continuous pipelines (orchestrated by Kubeflow on Vertex AI) to scale the process, eliminating manual work and ensuring consistent, reliable updates. - Use best practices like data windowing (e.g., train on the last 12 months) to focus on current patterns and hyperparameter retuning with Vertex AI Vizier to maintain model performance. Topics Why Retraining Is Essential - Models are "snapshots" of the world at training time, learning patterns from historical data. - Performance degrades when future data no longer resembles the past due to real-world changes (e.g., pandemics, recessions, new competitors). - Retraining on current data is the only fix for these outdated patterns. Retraining Triggers - Production systems need clear policies for when to retrain. - Schedule-based: - Trigger: Fixed cadence (e.g., weekly, monthly). - Pro: Simple, predictable resources. - Con: Retrains unnecessarily or misses urgent changes between intervals. - Performance-based: - Trigger: Performance drops below a threshold. - Pro: Directly tied to business value; retrains only when needed. - Con: Reactive; business impact has already occurred by the time degradation is detected. - Drift-based: - Trigger: Data or concept drift exceeds thresholds. - Pro: Proactive; catches issues before performance drops. Vertex AI Model Monitoring automates this. - Con: Drift doesn't always cause performance problems. - Event-based: - Trigger: Significant business events (e.g., product launch, regulatory change). - Pro: Incorporates domain knowledge. - Con: Relies on human judgment to identify relevant events. Continuous Training Pipelines - Automates the retraining workflow, as manual processes don't scale. - Pipeline Stages (Orchestrated by Kubeflow on Vertex AI): 1. Data Validation: Verify new data quality (schema, nulls, distributions) to "fail fast" on bad data. 2. Model Training: Execute training code on new data batches, maintaining consistency in preprocessing and hyperparameters. 3. Model Evaluation: Calculate performance metrics on validation data. 4. Model Comparison: Compare new model vs. current production model. - If better (e.g., 2% accuracy gain) → proceed to deployment. - If not → keep current model and investigate why. 5. Model Registration: Save approved model to Vertex AI Model Registry with metadata (training data, metrics, data version). 6. Model Deployment: Deploy the new model, potentially using traffic splitting for gradual rollout. 7. Monitoring: Begin monitoring the new model's production performance and data drift. - Triggering Options: - Cloud Scheduler: On a cron schedule (e.g., every Monday at 2 a.m.). - Cloud Functions: On new data arrival in Cloud Storage. - Pub/Sub: On messages (e.g., monitoring alert, campaign launch). - Manual: Ad hoc trigger via console or API. Best Practices - Data Provenance: Track which data was used for each model version to diagnose performance differences. - Incremental Training: For some algorithms (e.g., warm-start NNs), continue training an existing model on new data. - Pro: Faster, cheaper. - Con: Risks "catastrophic forgetting" (losing old knowledge). - Data Windowing: Use a sliding window (e.g., last 12 months) to focus on current patterns and drop obsolete history. - Hyperparameter Retuning: Re-evaluate hyperparameters during retraining. Vertex AI Vizier can automate this search. - Validate Before Deployment: Never auto-deploy. Always compare new vs. old models; new models might perform worse due to data quality issues.