Y

Cloud platforms (AWS, Azure, Google Cloud) · Advanced

Design for failure, and for the bill

3 min read

Why it matters

At senior level you own outcomes: whether the service stays up when a data centre fails, and whether the monthly bill surprises finance. Cloud architect and platform interviews use design questions to test both. Most real outages and cost overruns come from designs that ignored one of them.

The concept

The AWS Well-Architected pillars (Azure and Google Cloud have equivalent frameworks):

  1. Operational excellence: automate, observe, learn from incidents.

  2. Security: least privilege, encryption, traceability.

  3. Reliability: recover from failure automatically, scale with demand.

  4. Performance efficiency: right services and sizes for the workload.

  5. Cost optimisation: pay only for what delivers value.

  6. Sustainability: use fewer resources for the same outcome.

Reliability patterns: spread instances across at least two availability zones behind a load balancer, use managed databases with a standby in another zone, keep servers stateless, and set health checks so failed instances are replaced.

Cost controls: tag every resource with owner, project and environment; create budgets with alerts; turn off non-production resources out of hours; choose storage classes by access pattern; review the biggest line items monthly.

Worked example: guard rails for a new project

Before building anything, create a budget alert. Save as budget.json:

budget.json
{
  "BudgetName": "taxi-pipeline-monthly",
  "BudgetLimit": { "Amount": "50", "Unit": "USD" },
  "TimeUnit": "MONTHLY",
  "BudgetType": "COST",
  "CostFilters": { "TagKeyValue": ["user:project$taxi-pipeline"] }
}
guard-rails.sh
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# Email the owner at 80% of forecast spend
aws budgets create-budget --account-id "$ACCOUNT_ID" --budget file://budget.json \
  --notifications-with-subscribers '[{"Notification":{"NotificationType":"FORECASTED","ComparisonOperator":"GREATER_THAN","Threshold":80,"ThresholdType":"PERCENTAGE"},"Subscribers":[{"SubscriptionType":"EMAIL","Address":"owner@example.com"}]}]'

# Find resources that break the tagging rule (no project tag)
aws resourcegroupstaggingapi get-resources --region ap-south-1 \
  --query 'ResourceTagMappingList[?!(Tags[?Key==`project`])].ResourceARN'

Activate project as a cost allocation tag in the billing console so the budget filter and cost reports can group by it.

Then review a design against the pillars. For the event-driven CSV pipeline from An event-driven function with a least-privilege role:

pillarquestionanswer
reliabilityWhat happens if a file fails?Retries, then a dead-letter queue with an alarm
securityWho can read raw data?Only the function role and the data team group
costWhat grows over time?Raw files: lifecycle rule expires them after 90 days
operationsHow do we know it works?Metrics for invocations, errors and duration, with alerts

Common mistakes

  • Single-zone "production". One data centre problem becomes a full outage.

  • No budget until the first bill. Set alerts on day one; forecasts warn before money is spent.

  • Untagged resources. Nobody knows who owns them, so nobody turns them off.

  • Over-provisioning "to be safe". Measure utilisation and right-size; use autoscaling for variable load.

Interview angle

System design rounds for cloud roles often end with "how would you make this highly available, and what would it cost?". Walk through zones, load balancing, managed data stores, backups and recovery objectives, then the main cost drivers and how you would monitor them.

Practice task

  1. Draw a two-zone architecture for a web app with a database, and mark what fails over automatically.

  2. Use the AWS Pricing Calculator to estimate it, and list the three largest costs.

  3. Write a one-page review of a design against the six pillars.

Further reading