Library · tool

Apache Spark

Process data too big for one machine with Spark DataFrames, and keep jobs fast by understanding partitions, shuffles and joins.

Spark appeared in 78% of the Data Engineer job descriptions we read and 40% of the AI/ML Engineer ones. Scala appeared in 26% of Data Engineer postings; the concepts here apply in Scala and Python alike, and the examples use PySpark.

Everything runs in local mode on a laptop against the NYC TLC trip files, so you can see real plans and timings without a cluster.

1Spark DataFramesSpark DataFrames and lazy evaluation3 min read