MapReduce turns a large batch job into two user functions and a runtime-managed data movement phase.
map(k1, v1) → [(k2, v2)]
shuffle groups all values by k2
reduce(k2, [v2]) → outputFor word count, mappers emit (word, 1) and reducers sum counts. The simple API hides input splitting, scheduling, partitioning, transfer, retry, and output commit.
The shuffle is the expensive middle
A partitioner decides which reducer owns each intermediate key. All equal keys must reach the same reducer. Combiners can aggregate locally before transfer when the operation is associative and commutative, reducing network bytes.
Data locality schedules map tasks near their input blocks. Reduce tasks still need their partitions from every mapper, so shuffle, skew, and the slowest partition often dominate elapsed time.
Fault tolerance through deterministic tasks
The coordinator tracks tasks and re-executes lost work. Mapper output on a failed worker can be recomputed from replicated input. Reducer output becomes visible only after an atomic commit. This works best when tasks are deterministic and side-effect free.
Speculative execution runs duplicate copies of unusually slow tasks; the first valid result wins. It helps random stragglers but wastes capacity when slowness comes from systematic skew.
In plain terms: MapReduce often stores the recipe and input instead of trying to preserve every intermediate calculation.
Review card
- Map transforms records; reduce aggregates one key’s values.
- Shuffle is distributed grouping and network movement.
- Partitioning determines reducer balance.
- Combiners are safe only for suitable operations.
- Deterministic tasks make retry-based fault tolerance practical.
- Measure skew and tail tasks, not only average worker time.