Project · Advanced · about 24 hours
Safe deploys with canaries and automatic rollback
Ship a bad release on purpose and show that a canary with automated analysis stops it before most users see it, then roll back without a human.
Skills:CI/CD, Monitoring and observability, Kubernetes
Part of: DevOps / Platform Engineer
Data and licences
- Open Food Facts product database ↗Open Database License (ODbL); images CC BY-SA
Brief
Deploys are the most common cause of incidents. A canary sends a small share of traffic to the new version, measures it, and only continues if it is healthy. Prove your setup catches a bad release of the catalogue service.
What to build
Install Argo Rollouts (or Flagger) and Prometheus on a local cluster
Define a rollout with steps (for example 10%, 25%, 50% traffic) and analysis between steps based on error rate and p95 latency
Generate steady traffic with a load generator
Release a version that returns errors for 5% of requests and another that adds 400 ms latency; show both are rolled back automatically
Release a good version and show it promotes to 100%
Deliverables
Repository with
rollouts/,analysis/,load/andREPORT.mdwith graphs
How it is graded
Reviewers push their own bad release and measure how many requests failed before rollback.
Stretch
Add feature flags so a risky feature can be turned off without any deploy.
Deliverables
- A progressive delivery setup (Argo Rollouts or Flagger) on a local cluster
- Analysis templates that compare the canary's error rate and latency with the stable version
- A load generator and a deliberately bad release
- A report with graphs showing the failed canary and automatic rollback
How a reviewer grades it
| Criterion | Weight | What good looks like |
|---|---|---|
| Rollout steps and analysis thresholds are justified, not defaults | 25% | |
| A bad release is stopped automatically with limited user impact | 35% | |
| A good release promotes without manual steps | 20% | |
| Clear report with evidence | 20% |