Y

CI/CD · Advanced

Build once, promote safely, roll back fast

3 min read

Why it matters

Continuous delivery is how DevOps and platform teams turn merged code into running software many times a day without drama. Interviews for these roles ask you to design a deployment pipeline and explain what happens when a release goes wrong. The answers that stand out are about safety, not speed.

The concept

  • Build once: produce one versioned artefact (a container image tagged with the commit SHA) and promote that same artefact through environments. Rebuilding per environment means production runs something you never tested.

  • Environments (staging, production) with their own secrets and protection rules: required reviewers, wait timers, branch restrictions.

  • Verification after every deploy: health checks, smoke tests and key metrics.

  • Release strategies:

    • rolling: replace instances gradually

    • blue-green: run the new version beside the old and switch traffic at once, keeping the old ready for instant rollback

    • canary: send a small share of traffic to the new version, watch metrics, then increase

  • Rollback: redeploy the previous image tag. Database changes must be backward compatible (expand, migrate, contract) so the old version still works.

  • Feature flags separate deploying code from releasing features.

Worked example: staging then production with approval

.github/workflows/deploy.yml
name: Deploy

on:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write

concurrency:
  group: deploy-production
  cancel-in-progress: false

jobs:
  image:
    runs-on: ubuntu-latest
    outputs:
      tag: ${{ steps.meta.outputs.tag }}
    steps:
      - uses: actions/checkout@v4
      - id: meta
        run: echo "tag=ghcr.io/${{ github.repository }}:${{ github.sha }}" >> "$GITHUB_OUTPUT"
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - run: |
          docker build -t "${{ steps.meta.outputs.tag }}" .
          docker push "${{ steps.meta.outputs.tag }}"

  staging:
    needs: image
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/deploy.sh staging "${{ needs.image.outputs.tag }}"
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
      - run: curl --fail --retry 5 --retry-delay 10 https://staging.example.com/health

  production:
    needs: [image, staging]
    runs-on: ubuntu-latest
    environment: production          # add required reviewers to this environment in the repository settings
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/deploy.sh production "${{ needs.image.outputs.tag }}"
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
      - run: curl --fail --retry 5 --retry-delay 10 https://example.com/health

scripts/deploy.sh stands for your platform's deploy command (for example kubectl set image, aws ecs update-service or a Helm upgrade).

Notice:

  • The image is tagged with the commit SHA, built once in image, and both environments deploy that exact tag.

  • concurrency with cancel-in-progress: false queues deployments instead of cancelling one halfway.

  • The production environment pauses for a named reviewer before running, and has its own DEPLOY_TOKEN secret that staging jobs cannot read.

  • Each deploy is followed by a health check that fails the job if the service is not healthy.

Rollback is a deploy of the previous tag, not a rebuild: keep a record of which SHA is live in each environment.

Common mistakes

  • Rebuilding for production. Dependencies drift between builds; ship what you tested.

  • Deploying on latest tags. You cannot tell what is running or roll back precisely.

  • Breaking database migrations. Dropping a column in the same release that stops using it makes rollback impossible. Expand first, contract later.

  • No post-deploy checks. A deploy that "succeeds" while the service returns errors is a failed deploy.

Interview angle

"Design a deployment pipeline for a web service with zero downtime." Cover build once and tag by SHA, staging with automated tests, approval gates, a canary or blue-green release with automatic rollback on error-rate alerts, backward-compatible migrations and feature flags. Then describe what you would do at 2 a.m. when the canary's error rate jumps.

Practice task

  1. Add a rollback workflow (workflow_dispatch with a tag input) that redeploys a chosen image tag.

  2. Write the expand and contract steps for renaming a database column without downtime.

  3. Sketch a canary release: traffic steps, metrics watched, and the automatic rollback rule.

Further reading