Data-Cleaning Workflows: Automation Is Not Reproducibility

A script can automate a cleaning job and still be impossible to reproduce. It may read an unversioned file, depend on hidden state, overwrite its input, or omit the manual rule that changed the result. A workflow makes the data and control dependencies explicit.

Think in products and transformations

raw data ──> profile ──> cleaning ──> cleaned data
    │                         │              │
    └──────────────> validation <────────────┘
                              │
                         evidence report

Each step should declare:

inputs and outputs;
parameters and rule files;
executable version;
success criteria;
logs and quality metrics;
whether it is safe to retry.

The course summarizes workflow value as ASAP: automation, scaling, abstraction, and provenance. Those benefits appear only when boundaries are explicit.

Idempotence makes reruns safe

An idempotent stage produces the same output when rerun with the same inputs and configuration. Prefer writing versioned outputs or rebuilding a target from scratch over mutating the only copy.

output identity = hash(input + code + rules + parameters)

The hash is not a full provenance model, but it helps detect when two runs are not actually comparable.

Profile feedback without hiding decisions

Profiling often informs thresholds. For example, a location-to-city mapping may require support of at least three rows and confidence of at least 0.75. That creates a feedback edge from profile to rules.

Do not bury the chosen numbers inside code. Store them in versioned configuration and record the profiling evidence used to choose them. Otherwise a rerun is mechanically repeatable but scientifically unexplained.

Validation is a gate, not a final decoration

Useful gates include:

input schema and checksum match expectations;
row count reconciles;
key and referential violations are zero;
distributions remain within reviewed bounds;
unresolved and rejected counts are reported;
the intended analytical query executes on both raw and cleaned data.

A pipeline should fail loudly on broken invariants. Warnings are appropriate for expected uncertainty, not for missing core evidence.

Separate orchestration from cleaning logic

Small functions should implement parsing and rules; an orchestrator should call stages in order and pass explicit paths. This makes unit tests possible and prevents a notebook’s hidden execution order from becoming part of the algorithm.

Notebooks can still serve as an executable narrative, but the authoritative work should run from a clean process without relying on cells executed earlier.

Review card

Automation reruns commands; reproducibility explains the result.
Declare inputs, outputs, rules, parameters, versions, and gates.
Preserve raw data and make stages idempotent.
Version thresholds and the evidence behind them.
Validation belongs inside the workflow.
Separate testable cleaning functions from orchestration and presentation.

Script 可以自動執行 cleaning，結果仍可能無法 reproduce：它可能讀 unversioned file、依賴 hidden state、覆寫 input，或漏掉改變結果的人工規則。Workflow 的價值是把 data 與 control dependency 明確化。

用 data product 與 transformation 思考

raw data ──> profile ──> cleaning ──> cleaned data
    │                         │              │
    └──────────────> validation <────────────┘
                              │
                         evidence report

每個 step 應宣告：

input／output；
parameter／rule file；
executable version；
success criteria；
log／quality metric；
是否能安全 retry。

課程把 workflow 價值整理成 ASAP：automation、scaling、abstraction、provenance。只有 boundary 明確時，這些好處才真的存在。

Idempotence 讓 rerun 安全

Idempotent stage 在相同 input 與 configuration 下重跑，會產生相同 output。應優先寫 versioned output 或重建 target，不要原地修改唯一副本。

output identity = hash(input + code + rules + parameters)

Hash 不是完整 provenance model，但能幫助判斷兩次 run 是否真的可比較。

Profile feedback 不應藏起決策

Profiling 常用來選 threshold。例如 location-to-city mapping 可能要求 support 至少 3、confidence 至少 0.75，這形成 profile 回饋 rules 的 edge。

不要把數字埋在 code 裡。把它放進 versioned configuration，並保存選 threshold 的 profiling evidence。否則 rerun 只是機械上可重複，科學上仍無法解釋。

Validation 是 gate，不是最後裝飾

有用 gate 包含：

input schema／checksum 符合預期；
row count 能 reconcile；
key／referential violation 為零；
distribution 留在 review 過的範圍；
unresolved／rejected count 有報告；
intended analytical query 能在 raw／cleaned data 都執行。

Core invariant 壞掉時 pipeline 應明確 fail；warning 適合預期的不確定性，不適合缺少核心證據。

Orchestration 與 cleaning logic 分開

小 function 實作 parser 與 rule；orchestrator 依序呼叫 stage 並傳入明確 path。這讓 unit test 可行，也避免 notebook 的 hidden execution order 變成 algorithm 一部分。

Notebook 可以保留為 executable narrative，但 authoritative work 應能從 clean process 執行，不依賴之前曾跑過哪些 cell。

複習卡

Automation 重跑 command；reproducibility 解釋結果。
宣告 input、output、rule、parameter、version 與 gate。
保留 raw data，stage 應 idempotent。
Threshold 與支持它的 evidence 都要 version。
Validation 要放在 workflow 裡。
可測 cleaning function 與 orchestration／presentation 分開。