How to Test Machine Learning Models in Production: A/B Testing, Bandits & Progressive Deployment

Offline evaluation, no matter how rigorous, cannot fully predict how a model will perform in production. Holdout sets are static snapshots…