Y

Project · Advanced · about 24 hours

High availability and disaster recovery you have tested

Run a demo shop across availability zones, define recovery objectives, and prove them by failing a zone and restoring from backups in another region.

Skills:Cloud platforms (AWS, Azure, Google Cloud), Monitoring and observability, Kubernetes, Infrastructure as code

Part of: Cloud and Network Engineer

Data and licences

Brief

Availability claims mean nothing until you test them. Use Online Boutique on a managed Kubernetes service or VMs to design, build and test for failure. Cost warning: this project uses paid cloud resources. Set a budget alert before you start, stay in the free tier where you can, and destroy everything at the end of each session.

What to build

  1. Deploy the app across two zones: node groups per zone, replicated services, a managed database or Redis with a standby, and a multi-zone load balancer

  2. Define RTO and RPO with a made-up business owner, and write why

  3. Back up state to another region on a schedule

  4. Fail a zone (cordon and drain nodes, or stop instances) under load and measure errors and recovery time

  5. Restore the app in the second region from backups and measure the real RTO and RPO

Deliverables

  • Repository with infra/, dr/ runbook, test logs and REPORT.md with graphs

How it is graded

Reviewers compare your measured RTO and RPO with your targets and follow the runbook step by step.

Stretch

Automate the regional restore so it runs monthly and reports its duration.

Deliverables

  • Infrastructure spread across at least two zones with no single point of failure you know of
  • Backups and replication to a second region, with the recovery time and recovery point objectives written down
  • A zone failure test and a regional restore test with measured results
  • A disaster recovery runbook

How a reviewer grades it

CriterionWeightWhat good looks like
Architecture removes single points of failure and says which remain25%
RTO and RPO are defined and measured, not assumed30%
Failure and restore tests are realistic and documented30%
Runbook is usable under stress15%