An RDD tells Spark how to transform objects. A DataFrame also tells Spark what columns and types mean. That extra structure lets the engine rewrite a logical plan, prune columns, push filters, and choose physical operators.
Read plans in three layers
- The parsed logical plan reflects the query.
- The optimized logical plan applies equivalent rewrites.
- The physical plan chooses scans, exchanges, joins, and aggregation algorithms.
Use explain() before guessing. Look for full scans, unnecessary Exchange nodes, sort operations, and estimated sizes that contradict reality.
Select less and filter early
Column pruning avoids reading unused fields from columnar formats. Predicate pushdown lets Parquet or another source skip irrelevant row groups. Both require expressions the engine understands; opaque Python UDFs can block optimization and introduce serialization overhead.
Joins decide where data moves
A small table can be broadcast to executors, avoiding a shuffle of the large side. Large-large joins usually require repartitioning by key. Incorrect size estimates can choose the wrong strategy, while skew can overload one reduce partition even when the plan is otherwise sensible.
result = (
orders.filter("status = 'paid'")
.select("customer_id", "amount")
.join(customers.select("customer_id", "country"), "customer_id")
.groupBy("country").sum("amount")
)
result.explain("formatted")In plain terms: higher-level APIs are often faster not because SQL is shorter, but because Spark can see and rearrange the work.
Review card
- Schema gives the optimizer semantic information.
- Logical plans describe meaning; physical plans describe execution.
- Projection and predicate pushdown reduce I/O.
- UDFs can hide work from Catalyst.
- Join strategy is largely a data-movement decision.