Y

Apache Spark · Intermediate · 6 h

Spark DataFrames

Sessions, lazy transformations, actions, partitions and writing partitioned Parquet.

Why employers care: Most batch pipelines at scale run on Spark or Databricks. Interviews check you know what actually runs when.

In the roadmaps for: Data Scientist, Data Engineer, AI/ML Engineer

Lessons, in order

  1. Spark DataFrames and lazy evaluationClean two months of NYC taxi trips, 7 million rows, with PySpark, and learn what runs when.3 min

Next shelf: Joins, shuffles and skew