Apache Airflow
Apache Airflow is the widely adopted open-source orchestrator where pipelines are Python-defined DAGs, with a large provider ecosystem and multiple managed offerings.
Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows. Pipelines are Python-defined DAGs of tasks; the scheduler executes them on intervals or triggers, with retries, backfills, and SLA monitoring handled by the platform.
A large provider ecosystem supplies pre-built operators for warehouses, clouds, and data tools, and the web UI gives operational visibility into runs, task state, and logs.
Airflow is the incumbent general-purpose data orchestrator, running everything from ELT schedules to ML pipelines. It can be self-hosted or consumed via managed services (Amazon MWAA, Google Cloud Composer, Astronomer). Dagster and Prefect are the most common alternatives, differing mainly in development model and data-awareness.
