Y

Apache Airflow and orchestration · Advanced

Backfills, idempotency and alerting

3 min read

Why it matters

Pipelines fail, logic changes, and sources correct old data. When that happens you rerun history: a backfill. If tasks are not idempotent, a backfill duplicates or corrupts data, and nobody notices until a report is wrong. Senior data engineering interviews ask how you would handle exactly this.

The concept

  • Idempotent task: running it twice for the same period gives the same result as running it once. Achieve it by overwriting a partition (write to month=2025-01/ replacing what is there), or by merging on a unique key, never by blind appends.

  • Deterministic inputs: a task reads only data for its own period, derived from the run's logical date, never "latest" or "now".

  • Backfill: create runs for past logical dates. In Airflow 3, airflow backfill create schedules them through the normal scheduler, with a limit on parallel runs.

  • Alerting: failure callbacks or notifiers send failures to chat, email or on-call; track duration too, because a job that silently takes three times longer is often about to fail.

  • Data-aware scheduling: downstream DAGs can run when an upstream asset is updated instead of on a clock.

Worked example: backfilling the taxi DAG

The DAG from A monthly pipeline as an Airflow DAG is already idempotent: the raw file is written once per month and the clean partition is overwritten on every run. That lets you backfill safely:

backfill.sh
# Preview which runs would be created
airflow backfill create --dag-id taxi_monthly \
  --from-date 2025-02-01 --to-date 2025-12-31 --dry-run

# Run them, two at a time, reprocessing months that already completed
airflow backfill create --dag-id taxi_monthly \
  --from-date 2025-02-01 --to-date 2025-12-31 \
  --max-active-runs 2 --reprocess-behavior completed

Each run derives its own month from its logical date, so the February 3 run processes January, the March 3 run processes February, and so on. Rerunning any of them overwrites the same partition.

Add alerting at the DAG level:

alerting.py
def notify_failure(context):
    ti = context["ti"]
    message = f"{ti.dag_id}.{ti.task_id} failed for {context['logical_date']:%Y-%m-%d} (try {ti.try_number})"
    # Replace with your chat webhook or paging tool; keep secrets in an Airflow connection, not in code.
    print(message)


@dag(
    schedule="0 6 3 * *",
    start_date=datetime(2025, 2, 1),
    catchup=False,
    default_args={"retries": 3, "retry_delay": timedelta(hours=6), "on_failure_callback": notify_failure},
)
def taxi_monthly():
    ...

Checklist before any backfill:

  • Is every task idempotent? Check writes: overwrite by partition or merge by key.

  • Are inputs derived from the logical date?

  • Will parallel runs overwhelm the source or the warehouse? Set --max-active-runs.

  • Who is told when it finishes, and how will you verify it (row counts per period against the source)?

Common mistakes

  • Append-only loads. Every rerun doubles the data. Overwrite partitions or merge.

  • "Latest file" logic. A backfill for March quietly processes September's file.

  • Alerts to a shared inbox nobody reads. Route to on-call or a monitored channel with an owner.

  • Backfilling everything at once. Throttle, and verify the first few periods before releasing the rest.

Interview angle

"You fixed a bug in a transformation. How do you correct the last six months?" Answer: confirm idempotency, backfill with limited parallelism, verify counts and key metrics per period against the source, and tell downstream owners. If a task was not idempotent, fix that first.

Practice task

  1. Change transform to append instead of overwrite, run it twice for the same month and measure the duplicate rows. Then revert.

  2. Write a verification task that compares clean row counts with raw row counts per month and fails if more than 5% are dropped.

  3. Convert the downstream summary DAG to run on an asset update instead of a schedule.

Further reading