Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance. Originally developed at the University of California, Berkeley's AMPLab starting in 2009, in 2013, the Spark codebase was donated to the Apache Software Foundation, which has maintained it since.
Overview Apache Spark has its architectural foundation in the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way. The Dataframe API was released as an abstraction on top of the RDD, followed by the Dataset API. In Spark 1.x, the RDD was the primary application programming interface (API), but as of Spark 2.x use of the Dataset API is encouraged even though the RDD API is not deprecated. The RDD technology still underlies the Dataset API.
Spark and its RDDs were developed in 2012 in response to limitations in the MapReduce cluster computing paradigm, which forces a particular linear dataflow structure on distributed programs: MapReduce programs read input data from disk, map a function across the data, reduce the results of the map, and store reduction results on disk. Spark's RDDs function as a working set for distributed programs that offers a (deliberately) restricted form of distributed shared memory.
Inside Apache Spark the workflow is managed as a directed acyclic graph (DAG). Nodes represent RDDs while edges represent the operations on the RDDs.
Spark facilitates the implementation of both iterative algorithms, which visit their data set multiple times in a loop, and interactive/exploratory data analysis, i.e., the repeated database-style querying of data. The latency of such applications may be reduced by several orders of magnitude compared to Apache Hadoop MapReduce implementation. Among the class of iterative algorithms are the training algorithms for machine learning systems, which formed the initial impetus for developing Apache Spark.
Apache Spark requires a cluster manager and a distributed storage system. For cluster management, Spark supports standalone native Spark, Hadoop YARN, Apache Mesos or Kubernetes. A standalone native Spark cluster can be launched manually or by the launch scripts provided by the install package. It is also possible to run the daemons on a single machine for testing. For distributed storage Spark can interface with a wide variety of distributed systems, including Alluxio, Hadoop Distributed File System (HDFS),Figure showing Spark in relation to other open-source Software projects including Hadoop MapR File System (MapR-FS),MapR ecosystem support matrix Cassandra, OpenStack Swift, Amazon S3, Kudu, Lustre file system, or a custom solution can be implemented. Spark also supports a pseudo-distributed local mode, usually used only for development or testing purposes, where distributed storage is not required and the local file system can be used instead; in such a scenario, Spark is run on a single machine with one executor per CPU core.
Spark Core Spark Core is the foundation of the overall project. It provides distributed task dispatching, scheduling, and basic I/O functionalities, exposed through an application programming interface (for Java, Python, Scala, .NET and R) centered on the RDD abstraction (the Java API is available for other JVM languages, but is also usable for some other non-JVM languages that can connect to the JVM, such as Julia). This interface mirrors a functional/higher-order model of programming: a "driver" program invokes parallel operations such as map, filter or reduce on an RDD by passing a function to Spark, which then schedules the function's execution in parallel on the cluster. These operations, and additional ones such as joins, take RDDs as input and produce new RDDs. RDDs are immutable and their operations are lazy; fault-tolerance is achieved by keeping track of the "lineage" of each RDD (the sequence of operations that produced it) so that it can be reconstructed in the case of data loss. RDDs can contain any type of Python, .NET, Java, or Scala objects.
Besides the RDD-oriented functional style of programming, Spark provides two restricted forms of shared variables: broadcast variables reference read-only data that needs to be available on all nodes, while accumulators can be used to program reductions in an imperative style.
A typical example of RDD-centric functional programming is the following Scala program that computes the frequencies of all words occurring in a set of text files and prints the most common ones. Each , (a variant of ) and takes an anonymous function that performs a simple operation on a single data item (or a pair of items), and applies its argument to transform an RDD into a new RDD.
import org.apache.spark.{SparkConf, SparkContext} import org.apache.spark.rdd.RDD
val conf: SparkConf = new SparkConf().setAppName("wiki_test") // create a spark config object val sc: SparkContext = new SparkContext(conf) // Create a spark context val data: RDD[String] = sc.textFile("/path/to/somedir") // Read files from "somedir" into an RDD of (filename, content) pairs. val tokens: RDD[String] = data.flatMap(_.split(" ")) // Split each file into a list of tokens (words). val wordFreq: RDD[(String, Int)] = tokens.map((_, 1)).reduceByKey(_ + _) // Add a count of one to each token, then sum the counts per word type. val topWords: Array[(Int, String)] = wordFreq.sortBy(s => -s._2).map(x => (x._2, x._1)).top(10) // Get the top 10 words. Swap word and count to sort by count.
PySpark PySpark is the Python API for Apache Spark, introduced in Spark 0.7, released in February 2013. It allows Spark applications to be written in Python and provides an interactive shell for exploratory analysis. PySpark code executes under the standard CPython runtime and can call native Python libraries such as NumPy and SciPy; communication between Python user code and Spark's JVM-based core has historically been mediated by the Py4J library.
PySpark provides Python interfaces to most of Spark's components, including Spark Core, Spark SQL and the DataFrame API, Structured Streaming, and MLlib; since Spark 4.1 it also exposes the Spark Declarative Pipelines API.
A pandas-compatible API, known as pandas API on Spark, was integrated into PySpark in Apache Spark 3.2.0, released in October 2021. It provides equivalents to the pandas data-analysis API that execute against the Spark engine, allowing pandas-style code to be run on distributed data. The component originated as Koalas, a separate project open-sourced by Databricks in 2019, which was subsequently merged into PySpark.
PySpark also provides a client for Spark Connect, a client–server architecture that enables remote connectivity to Spark clusters from PySpark applications.
A typical PySpark program uses a SparkSession to load data as a DataFrame and apply transformations:
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("example").getOrCreate()
df = spark.read.json("path/to/people.json") df.groupBy("age").count().show()
Spark SQL Spark SQL is a component on top of Spark Core that introduced a data abstraction called DataFrames, which provides support for structured and semi-structured data. Spark SQL provides a domain-specific language (DSL) to manipulate DataFrames in Scala, Java, Python or .NET. It also provides SQL language support, with command-line interfaces and ODBC/JDBC server. Although DataFrames lack the compile-time type-checking afforded by RDDs, as of Spark 2.0, the strongly typed DataSet is fully supported by Spark SQL as well.
import org.apache.spark.sql.{DataFrame, SparkSession}
val url: String = "jdbc:mysql://yourIP:yourPort/test?user=yourUsername;password=yourPassword" // URL for your database server. val spark: SparkSession = SparkSession.builder().getOrCreate() // Create a Spark session object
val df: DataFrame = spark .read .format("jdbc") .option("url", url) .option("dbtable", "people") .load()
df.printSchema() // Looks at the schema of this DataFrame. val countsByAge: DataFrame = df.groupBy("age").count() // Counts people by age Or alternatively via SQL: df.createOrReplaceTempView("people") val countsByAge: DataFrame = spark.sql("SELECT age, count(*) FROM people GROUP BY age")
Spark Declarative Pipelines Spark Declarative Pipelines (SDP) is a declarative framework for defining and running extract, transform, load (ETL) data pipelines, added in Apache Spark 4.1.0, which was released on December 16, 2025. Developers declare the datasets a pipeline should produce and the queries that define them, rather than writing imperative code specifying how each dataset is computed and in what order; Spark constructs a dataflow graph and manages dependency ordering, parallelism, checkpointing, and retries.
The framework extends Spark's lazy, declarative execution model, previously applied to individual queries through Spark SQL, to pipelines spanning multiple datasets. Pipelines are defined in SQL or Python and are organized around streaming tables and materialized views, with the underlying transformations expressed as "flows"; batch and streaming computations can be combined within a single pipeline.
A pipeline using the syntax can declare a streaming table that ingests records and a materialized view that aggregates them, using the Python API:
from pyspark import pipelines as dp
@dp.table def orders(): return spark.readStream.table("orders_source")
@dp.materialized_view def daily_orders_by_state(): return (spark.table("orders") .groupBy("state", "order_date") .count())
The same datasets can be defined in SQL:
CREATE STREAMING TABLE orders AS SELECT * FROM STREAM orders_source;
CREATE MATERIALIZED VIEW daily_orders_by_state AS SELECT state, order_date, count(*) AS order_count FROM orders GROUP BY state, order_date;
Spark Structured Streaming Spark Structured Streaming uses Spark Core's fast scheduling capability to perform streaming analytics. It ingests data in micro-batches and performs RDD transformations on those micro-batches of data. This design enables the same set of application code written for batch analytics to be used in streaming analytics, thus facilitating easy implementation of lambda architecture. However, this convenience comes with the penalty of latency equal to the mini-batch duration. Other streaming data engines that process event by event rather than in micro-batches include Storm and the streaming component of Flink. Spark Streaming has support built-in to consume from Kafka, Flume, Twitter, ZeroMQ, Kinesis, and TCP/IP sockets.
In Spark 2.x, a separate technology based on Datasets, called Structured Streaming, that has a higher-level interface is also provided to support streaming.
Apache Spark 4.x added a Real-Time Mode (RTM) execution model for Structured Streaming, with the stated goal of lowering end-to-end latency to support operational and batch workloads on a single engine. Existing Structured Streaming queries written against the DataFrame and Dataset APIs can be executed under the new mode without modification to the query.
Spark can be deployed in a traditional on-premises data center as well as in the cloud.
Spark Connect Spark Connect is a decoupled client–server architecture for Apache Spark that allows remote applications to connect to a Spark cluster from any environment, introduced in Apache Spark 3.4, released in April 2023. The architecture separates the client application from the Spark driver, which historically ran in the same process; under Spark Connect, the client is a thin library that can be embedded in application servers, IDEs, notebooks and other programming environments.
The Spark Connect client translates DataFrame operations into unresolved logical query plans, encodes them using