Y

Security compliance and data protection · Intermediate

Pseudonymise and minimise personal data in a pipeline

4 min read

Why it matters

Privacy laws such as India's DPDP Act and the EU's GDPR expect organisations to collect only what they need, protect it, and delete it when the purpose ends. Data engineers implement this in pipelines. Interviewers increasingly ask "how would you share this dataset with analysts safely?".

The concept

  • Classify columns: direct identifiers (name, phone, email, customer id), quasi-identifiers (postcode, birth date, exact timestamps) that can identify people in combination, and non-personal data.

  • Minimise: drop columns and precision the use case does not need (month instead of exact time, age band instead of birth date).

  • Pseudonymise: replace identifiers with tokens. A keyed hash (HMAC) gives the same token for the same person, so joins and counts still work, but tokens cannot be reversed or recomputed without the key. Pseudonymised data is still personal data under GDPR, because the key holder can re-identify; treat it with care.

  • Anonymise only when re-identification is truly impossible, which is hard; aggregation with minimum group sizes is the usual route.

  • Retain for a defined period, then delete, and prove it.

A plain unsalted hash of a phone number is not protection. Indian mobile numbers are 10 digits starting with 6 to 9, so there are at most 4 billion of them; hashing every one with SHA-256 takes under an hour on a single laptop core, even in plain Python (about 0.4 seconds per million).

Worked example: share retail data with the analytics team

pseudonymise.py
"""Pseudonymise customer identifiers and minimise columns before data leaves the production zone."""
import hashlib
import hmac
import os

import pandas as pd

KEY = os.environ["PSEUDONYM_KEY"].encode()           # from a secrets manager, never in code


def pseudonym(value) -> str | None:
    """Keyed hash: the same customer always maps to the same token, but it cannot be reversed without the key."""
    if pd.isna(value):
        return None
    return hmac.new(KEY, str(int(value)).encode(), hashlib.sha256).hexdigest()[:16]


sales = pd.read_csv("online_retail_2010_2011.csv", dtype={"Invoice": str, "StockCode": str})
shared = (sales
          .assign(customer_token=sales["Customer ID"].map(pseudonym),
                  invoice_month=pd.to_datetime(sales["InvoiceDate"]).dt.to_period("M").astype(str))
          .drop(columns=["Customer ID", "InvoiceDate"])   # minimise: drop direct identifiers and exact times
          [["Invoice", "StockCode", "Quantity", "Price", "Country", "invoice_month", "customer_token"]])

print(shared.head(3).to_string(index=False))
print("customers:", sales["Customer ID"].nunique(), "tokens:", shared["customer_token"].nunique())

Run with PSEUDONYM_KEY=... python pseudonymise.py (a CSV export of the UCI Online Retail II 2010 to 2011 sheet):

output
Invoice StockCode  Quantity  Price        Country invoice_month   customer_token
 536365    85123A         6   2.55 United Kingdom       2010-12 a9b01981371a1f4e
 536365     71053         6   3.39 United Kingdom       2010-12 a9b01981371a1f4e
 536365    84406B         8   2.75 United Kingdom       2010-12 a9b01981371a1f4e
customers: 4372 tokens: 4372

4,372 distinct customers become 4,372 distinct tokens, so cohort and RFM analyses work exactly as before, but nobody downstream sees a real customer id or an exact purchase time. Your token values will differ, because they depend on your key.

Operational rules that make this real:

  • The key lives in a secrets manager, with access limited to the pipeline; rotating it breaks joins across periods, so plan rotation.

  • Small groups can still identify people (one customer in a small country). Before sharing aggregates publicly, suppress groups below a threshold, such as 10.

  • Record what was shared, with whom, for what purpose and until when.

Common mistakes

  • Unsalted hashes of phone numbers or emails. Easily reversed by hashing all candidates.

  • Keeping the key next to the data. Anyone with both can re-identify.

  • Forgetting free-text columns, which often contain names, phone numbers and addresses.

  • "Anonymised" data with exact timestamps and locations. A few points identify most people.

Interview angle

"How would you give analysts customer data while respecting privacy?" Describe classification, minimisation, keyed pseudonymisation with a managed key, access controls, small-group suppression for published results, retention and an access log. Distinguish pseudonymisation from anonymisation.

Practice task

  1. Add a retention step that deletes shared rows older than 24 months, and a test that proves it.

  2. Build a country-level summary that suppresses countries with fewer than 10 customers.

  3. Scan the Description column for anything that looks like an email address or phone number.

Further reading