An RDD is an immutable distributed collection split into partitions. The API looks like local collection programming, but performance comes from where partitions live, which dependencies cross machines, and when Spark materializes work.
Transformations build a plan; actions run it
map, filter, and flatMap are lazy transformations. They extend a lineage graph. An action such as count, collect, or save asks the scheduler to execute enough of that graph to produce a result.
Lineage is also the recovery mechanism: if a partition is lost, Spark can recompute it from ancestors instead of replicating every intermediate result.
Narrow and wide dependencies
A narrow dependency lets each child partition read a small number of parent partitions. It can usually pipeline within one stage. A wide dependency—groupByKey, reduceByKey, repartition, many joins—moves records across the network and creates a shuffle boundary.
input partitions -> narrow maps -> SHUFFLE -> reduce partitionsShuffles add serialization, network transfer, disk spill, sorting, and straggler risk. Prefer combinable operations such as reduceByKey over sending every value through groupByKey.
Partitions are the unit of parallel work
Too few partitions leave executors idle; too many create scheduler overhead and tiny tasks. Skew is worse: one hot key can make a single task dominate the job. Inspect task duration and input size distributions, not only the average.
Cache only reused, expensive-to-recompute data. Caching everything consumes memory, spills useful blocks, and can slow the application. Use collect() only when the result is provably small because it brings all data to the driver.
In plain terms: Spark is fast when computation stays near balanced partitions. It becomes expensive when data must be regrouped across the cluster.
Review card
- RDDs are immutable, partitioned, and recoverable through lineage.
- Transformations are lazy; actions trigger execution.
- Wide dependencies introduce shuffle boundaries.
- Partition count and skew determine usable parallelism.
- Persist only reused lineage whose recomputation is costly.
RDD 是切成 partitions 的 immutable distributed collection。API 看起來像 local collection programming，但 performance 取決於 partition 在哪裡、dependency 是否跨機器，以及 Spark 何時真正 materialize work。
Transformation 建 plan；action 才執行
map、filter、flatMap 是 lazy transformation，只會延伸 lineage graph。count、collect、save 等 action 才要求 scheduler 執行足夠的 graph 以產生結果。
Lineage 同時是 recovery mechanism：partition 遺失時，Spark 可從 ancestor recompute，不必 replicate 每個 intermediate result。
Narrow 與 wide dependency
Narrow dependency 的 child partition 只讀少量 parent partitions，通常能在同一 stage pipeline。Wide dependency，例如 groupByKey、reduceByKey、repartition 與許多 join，會把 record 跨 network 移動並建立 shuffle boundary。
input partitions -> narrow maps -> SHUFFLE -> reduce partitionsShuffle 會增加 serialization、network、disk spill、sorting 與 straggler risk。能在 map side combine 時，reduceByKey 通常比把所有 values 都送進 groupByKey 合理。
Partition 是 parallel work 的單位
Partition 太少會讓 executor 閒置；太多則造成 scheduler overhead 與 tiny tasks。Skew 更麻煩：一個 hot key 就可能讓單一 task 拖住整個 job。要看 task duration 與 input size distribution，不能只看平均值。
只 cache 會重複使用、且 recompute 昂貴的資料。全部 cache 會吃掉 memory、讓有用 block spill，反而更慢。collect() 會把所有資料拉到 driver，只有 result 明確很小時才能使用。
白話來說： Computation 留在均衡 partitions 附近時 Spark 才快；資料必須跨 cluster 重新分組時，成本就上來了。
複習卡
- RDD immutable、partitioned，靠 lineage recovery。
- Transformation lazy；action trigger execution。
- Wide dependency 形成 shuffle boundary。
- Partition count 與 skew 決定真正 parallelism。
- 只 persist 重複使用且 recomputation 昂貴的 lineage。