Provenance is the origin and processing history of data. A final value without provenance may be correct, but it is difficult to trust, debug, update, or reproduce. For cleaning, lineage should connect each derived value to source evidence and the rule that transformed it.
Prospective and retrospective provenance
| Form | Describes | Example question |
|---|---|---|
| Prospective | Intended workflow structure | Which step consumes the profile and rule files? |
| Retrospective | What happened in one run | Which file version and rule produced this city? |
A workflow diagram is prospective: it shows possible dependencies. A run log is retrospective: it records actual inputs, outputs, times, and decisions. One cannot substitute for the other.
In plain terms: prospective provenance is the recipe; retrospective provenance is the kitchen record from tonight’s service.
Ask provenance questions before choosing granularity
Lineage can be recorded at dataset, table, row, cell, or operation level. Finer detail costs more storage and instrumentation.
Start from questions:
- Why does this output row exist?
- Which inputs contributed to this aggregate?
- Which outputs would change if this source file were corrected?
- Which rule and version generated this canonical value?
- Was the value parsed, copied, inferred, or manually reviewed?
Dataset-level lineage cannot answer a cell-level “why.” Cell-level lineage may be unnecessary for a simple file copy.
Why-, where-, and how-provenance
- Why-provenance identifies contributing input witnesses.
- Where-provenance identifies the source location from which a value was copied.
- How-provenance records how contributions combined through operations.
For an inferred city, city_source = learned_crosswalk and city_rule = location majority, support 5, confidence .83 are more useful than a generic cleaned = true flag.
Reveal hidden workflows in scripts
Script-based pipelines often contain a workflow without declaring one. YesWorkflow-style annotations identify programs, inputs, outputs, and data links while leaving the executable language unchanged:
# @begin clean_menu
# @in raw_menu
# @in city_rules
# @out clean_menu
# @out change_log
...
# @end clean_menuAnnotations can generate a prospective graph. Retrospective capture still requires runtime identifiers, checksums, parameters, timestamps, and status.
Provenance must survive repair
Do not overwrite place with a guessed city. Prefer:
place_raw
canonical_city
city_source
city_rule
rule_version
confidence
review_statusThis design supports reprocessing when a rule changes and allows consumers to filter inferred values. A change log should use stable record IDs and distinguish filled, rewritten, cleared, unchanged, and still_blank.
Provenance has its own quality
Lineage can be incomplete or wrong. Validate that every derived non-null value has a source and rule, referenced artifacts exist, hashes match, and workflow edges correspond to real files. Provenance is not decorative metadata; it is data with constraints.
Review card
- Prospective provenance describes the plan; retrospective provenance records a run.
- Choose granularity from the questions lineage must answer.
- Why, where, and how provenance capture different explanations.
- Script annotations expose dependencies but do not replace runtime capture.
- Preserve raw values beside derived values and rules.
- Validate provenance with constraints like any other dataset.
Provenance 是資料的 origin 與 processing history。沒有 provenance 的 final value 可能正確，卻很難信任、debug、更新或 reproduce。對 cleaning 而言，lineage 應把每個 derived value 連回 source evidence 與 transformation rule。
Prospective 與 retrospective provenance
| 類型 | 描述 | 問題例子 |
|---|---|---|
| Prospective | 預期 workflow structure | 哪個 step 使用 profile 與 rule file？ |
| Retrospective | 某次 run 實際發生的事 | 哪個 file version 與 rule 產生這個 city？ |
Workflow diagram 是 prospective，顯示可能 dependency；run log 是 retrospective，記錄實際 input、output、時間與 decision。兩者不能互相取代。
白話來說： Prospective provenance 是食譜；retrospective provenance 是今晚廚房真正做過什麼的紀錄。
先問問題，再選 granularity
Lineage 可以記到 dataset、table、row、cell 或 operation。越細，storage 與 instrumentation 成本越高。
先列問題：
- 為什麼這個 output row 存在？
- 哪些 input 貢獻了這個 aggregate？
- 修正某個 source file 會影響哪些 output？
- 哪個 rule／version 產生 canonical value？
- Value 是 parsed、copied、inferred 或 manually reviewed？
Dataset-level lineage 回答不了 cell-level 的「為什麼」；單純 file copy 又不一定需要 cell-level lineage。
Why、where、how provenance
- Why-provenance： 找出貢獻 input witness。
- Where-provenance： 指出 value 從 source 哪個位置 copy。
- How-provenance： 記錄 contribution 如何經 operation 組合。
對 inferred city，city_source = learned_crosswalk 加上 city_rule = location majority, support 5, confidence .83，遠比 cleaned = true 有用。
把 script 裡隱藏的 workflow 顯示出來
Script pipeline 常已有 workflow，只是沒有宣告。YesWorkflow 式 annotation 能標示 program、input、output 與 data link，同時不改 executable language：
# @begin clean_menu
# @in raw_menu
# @in city_rules
# @out clean_menu
# @out change_log
...
# @end clean_menuAnnotation 可以產生 prospective graph；retrospective capture 仍需 runtime identifier、checksum、parameter、timestamp 與 status。
Repair 後 provenance 也要留下
不要把 place 直接覆寫成猜測的 city，應保存：
place_raw
canonical_city
city_source
city_rule
rule_version
confidence
review_statusRule 改變時才能 reprocess，consumer 也能 filter inferred value。Change log 應使用 stable record ID，並分清 filled、rewritten、cleared、unchanged、still_blank。
Provenance 也有品質
Lineage 也可能 incomplete 或錯誤。應驗證每個 derived non-null value 都有 source／rule、referenced artifact 存在、hash 相符、workflow edge 對得上真實 file。Provenance 不是裝飾 metadata，而是同樣需要 constraint 的 data。
複習卡
- Prospective provenance 描述 plan；retrospective provenance 記錄 run。
- 從 lineage 要回答的問題選 granularity。
- Why、where、how provenance 提供不同解釋。
- Script annotation 顯示 dependency，但不取代 runtime capture。
- Raw value、derived value 與 rule 並存。
- 像驗證其他 dataset 一樣驗證 provenance。