An RDD tells Spark how to transform objects. A DataFrame also tells Spark what columns and types mean. That extra structure lets the engine rewrite a logical plan, prune columns, push filters, and choose physical operators.
Read plans in three layers
- The parsed logical plan reflects the query.
- The optimized logical plan applies equivalent rewrites.
- The physical plan chooses scans, exchanges, joins, and aggregation algorithms.
Use explain() before guessing. Look for full scans, unnecessary Exchange nodes, sort operations, and estimated sizes that contradict reality.
Select less and filter early
Column pruning avoids reading unused fields from columnar formats. Predicate pushdown lets Parquet or another source skip irrelevant row groups. Both require expressions the engine understands; opaque Python UDFs can block optimization and introduce serialization overhead.
Joins decide where data moves
A small table can be broadcast to executors, avoiding a shuffle of the large side. Large-large joins usually require repartitioning by key. Incorrect size estimates can choose the wrong strategy, while skew can overload one reduce partition even when the plan is otherwise sensible.
result = (
orders.filter("status = 'paid'")
.select("customer_id", "amount")
.join(customers.select("customer_id", "country"), "customer_id")
.groupBy("country").sum("amount")
)
result.explain("formatted")In plain terms: higher-level APIs are often faster not because SQL is shorter, but because Spark can see and rearrange the work.
Review card
- Schema gives the optimizer semantic information.
- Logical plans describe meaning; physical plans describe execution.
- Projection and predicate pushdown reduce I/O.
- UDFs can hide work from Catalyst.
- Join strategy is largely a data-movement decision.
RDD 告訴 Spark 如何轉換 object；DataFrame 還告訴 Spark column 與 type 代表什麼。多出來的 structure 讓 engine 能 rewrite logical plan、prune columns、push filters，並選 physical operators。
用三層讀 execution plan
- Parsed logical plan 對應原始 query。
- Optimized logical plan 套用等價 rewrite。
- Physical plan 選 scan、exchange、join 與 aggregation algorithm。
不要先猜，先跑 explain()。檢查 full scan、不必要的 Exchange、sort，以及和真實資料差太多的 estimated size。
少選 column、提早 filter
Column pruning 讓 columnar format 不必讀 unused fields。Predicate pushdown 讓 Parquet 等 source 跳過無關 row groups。前提是 expression 必須能被 engine 理解；opaque Python UDF 常阻擋 optimization，也增加 serialization overhead。
Join 決定資料搬到哪裡
Small table 可以 broadcast 到 executors，避免 large side shuffle。Large-large join 通常要依 key repartition。錯誤 size estimate 可能選錯 strategy；skew 則會讓單一 reduce partition 過載。
result = (
orders.filter("status = 'paid'")
.select("customer_id", "amount")
.join(customers.select("customer_id", "country"), "customer_id")
.groupBy("country").sum("amount")
)
result.explain("formatted")白話來說： Higher-level API 常更快，不只是 SQL 比較短，而是 Spark 看得懂並能重新安排工作。
複習卡
- Schema 給 optimizer semantic information。
- Logical plan 描述意義；physical plan 描述執行。
- Projection／predicate pushdown 減少 I/O。
- UDF 可能把工作藏在 Catalyst 看不到的地方。
- Join strategy 很大程度是 data movement decision。