A year of taxi trips in Spark

#Brief

The data team is moving from single-machine scripts to Spark. Show what a well-behaved Spark job looks like on about 40 million trips.

#What to build

1. Read twelve monthly files, handle schema differences between months (new columns appear during the year), and clean consistently.

2. Enrich with the zone lookup using a broadcast join.

3. Aggregate: trips and revenue by day and borough, and the top zone pairs per month.

4. Write partitioned Parquet by month; show that a filtered read touches only one partition.

5. Measure: with and without broadcast, different shuffle partition counts, and caching. Include plans and Spark UI screenshots.

6. Reconcile one month's totals against DuckDB.

#Deliverables

• job.py, PERFORMANCE.md, reconciliation output

#How it is graded

Reviewers run the job locally on three months and read the performance report. Claims must be backed by plans and timings.

#Stretch

Run the job on a managed Spark service's free trial and compare cost and time.