Spark extends the same distributed execution ideas into streaming, graph processing, and machine learning. The APIs differ, but partitions, shuffles, state, and recovery still determine whether the system behaves well.
Streaming turns an unbounded input into repeated computation
The older DStream model represents a stream as a sequence of RDDs. Structured Streaming expresses an incremental query over an unbounded table. Either way, production design needs more than a source and a transformation:
- event time versus processing time;
- watermark and acceptable lateness;
- checkpoint location and restart semantics;
- idempotent or transactional sinks;
- state size for windows and aggregations.
Exactly-once claims are end-to-end properties. A replayable source does not prevent duplicate external side effects.
Graphs are vertices and edges with repeated communication
GraphFrames models vertices and edges as DataFrames. Algorithms such as PageRank and connected components repeatedly exchange information along edges. Graph partitioning and high-degree vertices therefore matter as much as the algorithm formula.
ML pipelines protect transformations, not just models
Spark ML’s DataFrame API composes transformers and estimators: clean input, encode features, fit a model, and apply the same fitted stages later. Splitting data before fitting vocabulary, scaling, or imputation prevents leakage.
Distributed ML helps only when data or computation warrants its coordination cost. A local library is often simpler for a dataset that fits comfortably in memory.
In plain terms: Spark provides one operating substrate, not a reason to make every workload distributed.
Review card
- Streaming correctness includes time, state, replay, and sink behavior.
- Checkpoints recover progress; they do not make arbitrary side effects idempotent.
- Graph workloads amplify skew around high-degree vertices.
- Pipelines keep training and inference transformations aligned.
- Distribution must earn its operational cost.