Data-Cleaning Workflows: Automation Is Not Reproducibility

A script can automate a cleaning job and still be impossible to reproduce. It may read an unversioned file, depend on hidden state, overwrite its input, or omit the manual rule that changed the result. A workflow makes the data and control dependencies explicit.

Think in products and transformations

raw data ──> profile ──> cleaning ──> cleaned data
    │                         │              │
    └──────────────> validation <────────────┘
                              │
                         evidence report

Each step should declare:

• inputs and outputs;
• parameters and rule files;
• executable version;
• success criteria;
• logs and quality metrics;
• whether it is safe to retry.

The course summarizes workflow value as ASAP: automation, scaling, abstraction, and provenance. Those benefits appear only when boundaries are explicit.

Idempotence makes reruns safe

An idempotent stage produces the same output when rerun with the same inputs and configuration. Prefer writing versioned outputs or rebuilding a target from scratch over mutating the only copy.

output identity = hash(input + code + rules + parameters)

The hash is not a full provenance model, but it helps detect when two runs are not actually comparable.

Profile feedback without hiding decisions

Profiling often informs thresholds. For example, a location-to-city mapping may require support of at least three rows and confidence of at least 0.75. That creates a feedback edge from profile to rules.

Do not bury the chosen numbers inside code. Store them in versioned configuration and record the profiling evidence used to choose them. Otherwise a rerun is mechanically repeatable but scientifically unexplained.

Validation is a gate, not a final decoration

Useful gates include:

• input schema and checksum match expectations;
• row count reconciles;
• key and referential violations are zero;
• distributions remain within reviewed bounds;
• unresolved and rejected counts are reported;
• the intended analytical query executes on both raw and cleaned data.

A pipeline should fail loudly on broken invariants. Warnings are appropriate for expected uncertainty, not for missing core evidence.

Separate orchestration from cleaning logic

Small functions should implement parsing and rules; an orchestrator should call stages in order and pass explicit paths. This makes unit tests possible and prevents a notebook’s hidden execution order from becoming part of the algorithm.

Notebooks can still serve as an executable narrative, but the authoritative work should run from a clean process without relying on cells executed earlier.

Review card

• Automation reruns commands; reproducibility explains the result.
• Declare inputs, outputs, rules, parameters, versions, and gates.
• Preserve raw data and make stages idempotent.
• Version thresholds and the evidence behind them.
• Validation belongs inside the workflow.
• Separate testable cleaning functions from orchestration and presentation.