The Risk of "New"
You've trained a new model that outperforms the current one on your offline test set. But deploying it directly to all users is a significant risk. Offline metrics (accuracy, F1 score) are measured on historical data. Real-world traffic has distribution shifts, edge cases, and user behaviors that your test set may not capture. A model that looks great offline can fail in subtle, expensive ways in production.
Shadow Mode (Dark Launch)
Shadow Mode is a deployment pattern that lets you validate a new model against real production traffic with zero risk to users.
The mechanics:
- Deploy the new model (Model B) alongside the current production model (Model A).
- When a user request arrives, route it to both models simultaneously.
- Return Model A's prediction to the user. The user experience is unchanged.
- Silently log Model B's prediction to a database or data warehouse.
- After a sufficient observation period (days or weeks), analyze Model B's logged predictions against real-world outcomes.
Shadow mode answers the question: "Is Model B safe and accurate on real data?" It does not answer whether Model B produces better business outcomes, because users never act on its predictions.