What a Data Engineer does

Data engineers move data from where it is produced to where it is useful, reliably, on time and at a sensible cost. They build ingestion, transformation, warehouses, orchestration and streaming, and they are accountable when numbers are late or wrong.

This roadmap is built from 50 job descriptions posted in the last 90 days. It starts with SQL, Python and data modelling, builds pipelines, warehouses with dbt and orchestration with Airflow, then scales out with Spark, Kafka, the cloud and infrastructure as code.

At about 12 hours a week, most people reach Job-ready in 8 to 12 months; analysts and backend engineers often get there faster.

We read 50 Data Engineer job descriptions posted between July and September 2026: 32 in India and 18 global remote roles.

• Pipelines are the job. ETL and data pipelines appear in 96% of postings.

• SQL and Python are the languages. 80% and 78%. Scala appears in 26%, mostly alongside Spark.

• Spark is the default engine. 78% of postings name it.

• Modelling and warehousing are Must-haves. Data modelling 72%, warehousing 68% (84% of junior postings), Snowflake 34%.

• Quality is part of the role. 66% mention data quality, validation or reconciliation.

• Cloud matters, Azure most of all here. Azure 54%, AWS 48%, Google Cloud 20%.

• Stakeholders and communication. 58% and 44%: data engineers work with analysts, scientists and product teams daily.

• Engineering practice. CI/CD 38%, Git 28%, streaming with Kafka 30%, orchestration 22%, and machine learning support 28%.

Method, full frequency tables and every job link are in the research file linked from this page.

Many free data engineering courses teach tools in isolation. The job descriptions ask for reliable systems. This roadmap adds:

• Idempotency, backfills and late data in every pipeline, with tests that compare incremental loads with full rebuilds.

• Data quality as code: thresholds, dbt tests and escalation, measured on real data with real problems.

• Performance you can explain: columnar layouts, partitioning and Spark plans, measured rather than asserted.

• Streaming correctness: consumer groups, committed offsets, idempotent sinks and dead letters, tested by replaying events.

• Cloud with guard rails: infrastructure as code, least privilege, budgets and privacy by design.