Ship a model and catch it failing

#Brief

The bank's call data runs from May 2008 to November 2010, and the customers called changed dramatically: a model trained on the first 30,000 calls scores about 0.76 ROC AUC on its own period but about 0.53 on the last 5,000 calls. Build the system that would have caught this.

#What to build

1. Train on the first 20,000 rows in time order. Save the pipeline and metadata.

2. Serve it with FastAPI in a Docker container, with input validation, /health and /score, and tests in CI.

3. Replay the remaining rows in order through the API in weekly batches, logging inputs, scores and (delayed) outcomes.

4. Compute PSI per feature and performance per window. Set alert thresholds and show when each would have fired.

5. Retrain on a sliding window, compare against the live model on the next period, and promote only if it wins.

#Deliverables

• Repository with service/, replay/, monitoring/, RETRAINING.md, RUNBOOK.md

#How it is graded

Reviewers run the replay and read the monitoring report. The key question: would your alerts have fired early enough, and would your policy have fixed the model safely?

#Stretch

Deploy the service on a free-tier cloud service and replay against it, with a cost estimate.