Project · Beginner · about 8 hours
Monthly data quality report for taxi trips
Build a repeatable data quality report for any month of NYC taxi data, with thresholds and a clear verdict.
Skills:Data cleaning and quality, Python, SQL
Part of: Data Engineer
Data and licences
- NYC TLC Yellow Taxi Trip Records ↗NYC Open Data public data (TLC terms of use, no warranty)
Brief
Every month the taxi data arrives, and every month someone asks "can we trust it?". Automate the answer.
What to build
A script
quality.py --month 2025-01that runs at least ten checks with DuckDB directly on the TLC Parquet file.Measure normal rates over several months (negative fares, missing passenger counts, pickups outside the month) and set warn and fail thresholds from them.
Include the consistency check between
total_amountand its parts, and report the pattern of differences.Output a Markdown report with a verdict, a table of checks, and month-on-month changes.
Deliverables
quality.py, three reports,escalations.md
How it is graded
Reviewers run the script on a month you did not include. The report must work unchanged and give a sensible verdict.
Stretch
Run it as a GitHub Actions job on the 5th of every month and open an issue when the verdict is fail.
Deliverables
- A script that takes a month and produces a Markdown quality report
- Reports for three consecutive months
- A short note on the issues you would escalate to the data owner
How a reviewer grades it
| Criterion | Weight | What good looks like |
|---|---|---|
| Checks cover completeness, validity, consistency, uniqueness and timeliness | 35% | |
| Thresholds are set from observed normal rates, not guessed | 25% | |
| Report is readable and gives a clear pass, warn or fail verdict | 25% | |
| Escalation note is specific | 15% |