Your first Terraform: a secure data lake bucket

4 min read

Why it matters

Infrastructure as code appeared in 88% of the Cloud and Network and 84% of the DevOps job descriptions we read. Code-defined infrastructure is reviewed like software, repeatable across environments and recoverable after mistakes. Interviews often include reading a Terraform snippet or writing a small module.

The concept

Providers talk to platforms (AWS, Azure, Google Cloud, GitHub, Cloudflare). Pin their versions.
Resources are things Terraform creates and manages; data sources read things that already exist.
Variables make configurations reusable; validation blocks catch bad values early. Outputs expose useful values.
terraform plan shows what would change; terraform apply makes it so. Always read the plan.
State records what Terraform manages and maps it to real resources. It can contain secrets, so it must be protected (next module).

Terraform is declarative: you describe the end state, and it works out the steps.

Worked example: a private, encrypted lake bucket and a scoped policy

providers.tf

terraform {
  required_version = ">= 1.6"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  region = var.region
  default_tags {
    tags = {
      project     = var.project
      environment = var.environment
      managed_by  = "terraform"
    }
  }
}

variables.tf

variable "region" {
  type    = string
  default = "ap-south-1"
}

variable "project" {
  type    = string
  default = "taxi-pipeline"
}

variable "environment" {
  type = string
  validation {
    condition     = contains(["dev", "prod"], var.environment)
    error_message = "environment must be dev or prod."
  }
}

main.tf

resource "aws_s3_bucket" "lake" {
  bucket = "${var.project}-${var.environment}-lake-2026"
}

resource "aws_s3_bucket_public_access_block" "lake" {
  bucket                  = aws_s3_bucket.lake.id
  block_public_acls       = true
  ignore_public_acls      = true
  block_public_policy     = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_versioning" "lake" {
  bucket = aws_s3_bucket.lake.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "lake" {
  bucket = aws_s3_bucket.lake.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "lake" {
  bucket = aws_s3_bucket.lake.id
  rule {
    id     = "expire-raw"
    status = "Enabled"
    filter {
      prefix = "raw/"
    }
    expiration {
      days = 90
    }
  }
}

data "aws_iam_policy_document" "pipeline" {
  statement {
    sid       = "ReadRaw"
    actions   = ["s3:GetObject"]
    resources = ["${aws_s3_bucket.lake.arn}/raw/*"]
  }
  statement {
    sid       = "WriteClean"
    actions   = ["s3:PutObject"]
    resources = ["${aws_s3_bucket.lake.arn}/clean/*"]
  }
  statement {
    sid       = "ListLake"
    actions   = ["s3:ListBucket"]
    resources = [aws_s3_bucket.lake.arn]
  }
}

resource "aws_iam_policy" "pipeline" {
  name   = "${var.project}-${var.environment}-pipeline"
  policy = data.aws_iam_policy_document.pipeline.json
}

output "lake_bucket" {
  value = aws_s3_bucket.lake.bucket
}

workflow.sh

terraform init                          # downloads the pinned provider
terraform fmt -check && terraform validate
terraform plan -var environment=dev     # read this before applying
terraform apply -var environment=dev
terraform plan -var environment=qa      # fails: "environment must be dev or prod."

Practise without an AWS account. Run the moto mock server (pip install "moto[server]", then moto_server -p 5000) and add this override file, which Terraform merges into the provider block:

local_override.tf

provider "aws" {
  access_key                  = "test"
  secret_key                  = "test"
  skip_credentials_validation = true
  skip_requesting_account_id  = true
  skip_metadata_api_check     = true
  s3_use_path_style           = true
  endpoints {
    s3  = "http://localhost:5000"
    iam = "http://localhost:5000"
    sts = "http://localhost:5000"
  }
}

With it, terraform apply -var environment=dev creates all seven resources against the mock and prints lake_bucket = "taxi-pipeline-dev-lake-2026". Delete the override file before pointing at a real account. A mock is good for learning the workflow; it does not reproduce every behaviour of the real service.

Notice the patterns: every security setting is its own reviewed resource, default tags make cost reporting work (see the cloud architecture lesson), and the IAM policy is generated from the bucket's ARN, so there is no hand-typed resource name to get wrong.

Common mistakes

Unpinned providers. A new major version can change behaviour. Pin with ~> and commit .terraform.lock.hcl.
Applying without reading the plan. Look for anything marked destroy or replace.
Committing state or .tfvars with secrets. Keep state in a protected remote backend; keep secrets in a secrets manager.
Clicking in the console to fix things. The next apply will undo or conflict with it. Change the code.

Interview angle

"What is Terraform state and why does it matter?" Explain that state maps configuration to real resources, enables plans and must be stored remotely with locking and restricted access. Follow-up: "What happens if someone changes a resource by hand?" Terraform detects the drift on the next plan and proposes to change it back.

Practice task

Add a second bucket for query results with objects expiring after 7 days.
Turn the lake into a module with inputs project, environment and raw_retention_days.
Run terraform destroy and read what it removes.

#Why it matters

#The concept

#Worked example: a private, encrypted lake bucket and a scoped policy

#Common mistakes

#Interview angle

#Practice task

#Further reading