Spark extends the same distributed execution ideas into streaming, graph processing, and machine learning. The APIs differ, but partitions, shuffles, state, and recovery still determine whether the system behaves well.
Streaming turns an unbounded input into repeated computation
The older DStream model represents a stream as a sequence of RDDs. Structured Streaming expresses an incremental query over an unbounded table. Either way, production design needs more than a source and a transformation:
- event time versus processing time;
- watermark and acceptable lateness;
- checkpoint location and restart semantics;
- idempotent or transactional sinks;
- state size for windows and aggregations.
Exactly-once claims are end-to-end properties. A replayable source does not prevent duplicate external side effects.
Graphs are vertices and edges with repeated communication
GraphFrames models vertices and edges as DataFrames. Algorithms such as PageRank and connected components repeatedly exchange information along edges. Graph partitioning and high-degree vertices therefore matter as much as the algorithm formula.
ML pipelines protect transformations, not just models
Spark ML’s DataFrame API composes transformers and estimators: clean input, encode features, fit a model, and apply the same fitted stages later. Splitting data before fitting vocabulary, scaling, or imputation prevents leakage.
Distributed ML helps only when data or computation warrants its coordination cost. A local library is often simpler for a dataset that fits comfortably in memory.
In plain terms: Spark provides one operating substrate, not a reason to make every workload distributed.
Review card
- Streaming correctness includes time, state, replay, and sink behavior.
- Checkpoints recover progress; they do not make arbitrary side effects idempotent.
- Graph workloads amplify skew around high-degree vertices.
- Pipelines keep training and inference transformations aligned.
- Distribution must earn its operational cost.
Spark 把同一套 distributed execution idea 延伸到 streaming、graph processing 與 machine learning。API 不同，但 partitions、shuffles、state 與 recovery 仍決定 system 是否可靠。
Streaming 把 unbounded input 變成反覆計算
早期 DStream 把 stream 表示成一連串 RDD；Structured Streaming 則把它表示成對 unbounded table 的 incremental query。無論哪種，production design 都不能只有 source 與 transformation：
- event time 或 processing time；
- watermark 與能接受的 lateness；
- checkpoint location 與 restart semantics；
- idempotent／transactional sink；
- window 與 aggregation 的 state size。
Exactly-once 是 end-to-end property。Source 可以 replay，不代表 external side effect 就不會重複。
Graph 是 vertices、edges 與反覆 communication
GraphFrames 用 DataFrame 表示 vertices 與 edges。PageRank、connected components 等 algorithm 會沿 edge 反覆交換資訊，因此 graph partitioning 與 high-degree vertex 和公式本身同樣重要。
ML pipeline 保護的不只是 model
Spark ML 的 DataFrame API 組合 transformer 與 estimator：clean input、encode features、fit model，再於之後套用相同 fitted stages。必須先 split data，再 fit vocabulary、scaling 或 imputation，才能避免 leakage。
Distributed ML 只有在 data 或 computation 足以抵銷 coordination cost 時才合理。資料能舒服地放進 memory 時，local library 通常更簡單。
白話來說： Spark 提供共同 operating substrate，不代表每個 workload 都應該 distributed。
複習卡
- Streaming correctness 包含 time、state、replay 與 sink behavior。
- Checkpoint 能恢復 progress，不能讓任意 side effect 自動 idempotent。
- Graph workload 會放大 high-degree vertex 周圍的 skew。
- Pipeline 讓 training 與 inference transformation 對齊。
- Distribution 必須值得它的 operational cost。