An RDD is an immutable distributed collection split into partitions. The API looks like local collection programming, but performance comes from where partitions live, which dependencies cross machines, and when Spark materializes work.
Transformations build a plan; actions run it
map, filter, and flatMap are lazy transformations. They extend a lineage graph. An action such as count, collect, or save asks the scheduler to execute enough of that graph to produce a result.
Lineage is also the recovery mechanism: if a partition is lost, Spark can recompute it from ancestors instead of replicating every intermediate result.
Narrow and wide dependencies
A narrow dependency lets each child partition read a small number of parent partitions. It can usually pipeline within one stage. A wide dependency—groupByKey, reduceByKey, repartition, many joins—moves records across the network and creates a shuffle boundary.
input partitions -> narrow maps -> SHUFFLE -> reduce partitionsShuffles add serialization, network transfer, disk spill, sorting, and straggler risk. Prefer combinable operations such as reduceByKey over sending every value through groupByKey.
Partitions are the unit of parallel work
Too few partitions leave executors idle; too many create scheduler overhead and tiny tasks. Skew is worse: one hot key can make a single task dominate the job. Inspect task duration and input size distributions, not only the average.
Cache only reused, expensive-to-recompute data. Caching everything consumes memory, spills useful blocks, and can slow the application. Use collect() only when the result is provably small because it brings all data to the driver.
In plain terms: Spark is fast when computation stays near balanced partitions. It becomes expensive when data must be regrouped across the cluster.
Review card
- RDDs are immutable, partitioned, and recoverable through lineage.
- Transformations are lazy; actions trigger execution.
- Wide dependencies introduce shuffle boundaries.
- Partition count and skew determine usable parallelism.
- Persist only reused lineage whose recomputation is costly.