Apache Spark
Apache Spark is an open-source distributed compute engine for batch processing, SQL, machine learning, and stream processing across large clusters.
Apache Spark is an open-source distributed compute engine for large-scale data processing. It offers a unified programming model across batch, SQL (Spark SQL), machine learning (MLlib), and micro-batch streaming (Structured Streaming), with APIs in Python, Scala, Java, and R.
Spark executes work as a DAG of tasks over a cluster, with in-memory processing and a mature query optimizer. It reads and writes most storage systems and table formats in the ecosystem, including Iceberg and Delta Lake.
It remains the default heavy-duty transformation engine in many lakehouse platforms — self-managed, through cloud services such as EMR and Dataproc, or via managed platforms like Databricks. Lighter-weight engines such as DuckDB are increasingly used where single-node processing suffices.
