Y

Security compliance and data protection · Advanced

Compliance as code: evidence that collects itself

3 min read

Why it matters

Audits for SOC 2 and ISO 27001 look at whether controls operated throughout a period, not only on audit day. Collecting screenshots by hand does not scale past a few systems. Senior cloud and security engineers automate checks that run continuously, alert on drift and store evidence. Tools like AWS Config, Azure Policy and open-source scanners do this at scale; writing a small version yourself teaches what they do.

The concept

  • Express each control as a testable rule ("every bucket blocks public access, is encrypted by default and is versioned").

  • Run the rule on a schedule and on every infrastructure change.

  • Store results as evidence: what was checked, when, by which code version, and the result per resource.

  • Alert on failures and track them to resolution; an exception needs an owner, a reason and an expiry date.

  • Prefer preventive controls (policies that block a public bucket) over detective ones (checks that find it), and use both.

Worked example: storage control evidence

evidence.py
"""Collect compliance evidence for every S3 bucket: public access blocked, encryption on, versioning on."""
import json
from datetime import datetime, timezone

import boto3
from botocore.exceptions import ClientError

s3 = boto3.client("s3")


def check_bucket(name: str) -> dict:
    result = {"bucket": name}
    try:
        block = s3.get_public_access_block(Bucket=name)["PublicAccessBlockConfiguration"]
        result["public_access_blocked"] = all(block.values())
    except ClientError:
        result["public_access_blocked"] = False              # no block configured at all
    try:
        s3.get_bucket_encryption(Bucket=name)
        result["encrypted_by_default"] = True
    except ClientError:
        result["encrypted_by_default"] = False
    result["versioning"] = s3.get_bucket_versioning(Bucket=name).get("Status") == "Enabled"
    result["compliant"] = all(v for k, v in result.items() if k != "bucket")
    return result


report = {
    "control": "Storage is private, encrypted and recoverable (maps to ISO 27001 A.8.12, A.8.24, A.8.13)",
    "collected_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
    "results": [check_bucket(b["Name"]) for b in s3.list_buckets()["Buckets"]],
}
report["failing"] = [r["bucket"] for r in report["results"] if not r["compliant"]]
print(json.dumps(report, indent=1))

Tested offline with moto against two buckets, one configured properly and one left at defaults:

evidence.json
{
 "control": "Storage is private, encrypted and recoverable (maps to ISO 27001 A.8.12, A.8.24, A.8.13)",
 "collected_at": "2026-09-30T06:34:42+00:00",
 "results": [
  {"bucket": "finance-reports", "public_access_blocked": true, "encrypted_by_default": true, "versioning": true, "compliant": true},
  {"bucket": "ml-scratch", "public_access_blocked": false, "encrypted_by_default": false, "versioning": false, "compliant": false}
 ],
 "failing": ["ml-scratch"]
}

On real AWS, every bucket has had default server-side encryption since January 2023, so the encryption check above always passes there. If your policy requires customer-managed KMS keys, check the algorithm is aws:kms and the key id is one you approve. Controls must match your actual policy, not a generic one.

Run it daily in CI with a read-only role, store each JSON report in a write-once location (for example an S3 bucket with object lock), and alert when failing is not empty.

Common mistakes

  • Checks nobody reads. Failures must create a ticket or alert with an owner.

  • Evidence that can be edited. Store it where it cannot be changed afterwards, with timestamps and the checking code's version.

  • Permanent exceptions. Every exception needs a reason, an approver and an expiry.

  • Only detective controls. Prevent the obvious mistakes with policies (service control policies, Azure Policy deny rules) as well.

Interview angle

"How would you prove to an auditor that all storage has been encrypted all year?" Describe the automated check, its schedule, immutable evidence storage, alerting, exception handling and the preventive policy that stops new violations. Mention managed tools (AWS Config rules, Security Hub, Azure Policy) and when you would write your own.

Practice task

  1. Extend the script to check that bucket policies deny requests without TLS (aws:SecureTransport).

  2. Add a --fail-on-violation flag so CI fails when any bucket is non-compliant.

  3. Write the same control as an AWS Config managed rule or an Azure Policy definition, and compare.

Further reading