Project · Beginner · about 8 hours

Monthly data quality report for taxi trips

Build a repeatable data quality report for any month of NYC taxi data, with thresholds and a clear verdict.

Skills:Data cleaning and quality, Python, SQL

Part of: Data Engineer

Data and licences

Brief

Every month the taxi data arrives, and every month someone asks "can we trust it?". Automate the answer.

What to build

  1. A script quality.py --month 2025-01 that runs at least ten checks with DuckDB directly on the TLC Parquet file.

  2. Measure normal rates over several months (negative fares, missing passenger counts, pickups outside the month) and set warn and fail thresholds from them.

  3. Include the consistency check between total_amount and its parts, and report the pattern of differences.

  4. Output a Markdown report with a verdict, a table of checks, and month-on-month changes.

Deliverables

  • quality.py, three reports, escalations.md

How it is graded

Reviewers run the script on a month you did not include. The report must work unchanged and give a sensible verdict.

Stretch

Run it as a GitHub Actions job on the 5th of every month and open an issue when the verdict is fail.

Deliverables

  • A script that takes a month and produces a Markdown quality report
  • Reports for three consecutive months
  • A short note on the issues you would escalate to the data owner

How a reviewer grades it

CriterionWeightWhat good looks like
Checks cover completeness, validity, consistency, uniqueness and timeliness35%
Thresholds are set from observed normal rates, not guessed25%
Report is readable and gives a clear pass, warn or fail verdict25%
Escalation note is specific15%