Y

Library · practice

ETL and data pipelines

Move data from sources into an analysis-ready shape, reliably and repeatably. The core of data engineering and a growing part of analyst work.

ETL was named in 96% of the Data Engineer and 22% of the Data Analyst job descriptions we read.

The lessons build a real pipeline over the NYC TLC monthly trip files in plain Python and DuckDB. Orchestration (Airflow) and distributed processing (Spark) have their own shelves.

  1. 1Build your first pipelineA small, real pipeline for monthly taxi files4 min read

ETL and data pipelines lessons

Loading lessons…

    Further reading

    • NYC TLC trip record data ↗dataset · free · NYC Open Data public data (TLC terms of use)

      Monthly Parquet files of every yellow and green taxi trip in New York City.