Provenance is the origin and processing history of data. A final value without provenance may be correct, but it is difficult to trust, debug, update, or reproduce. For cleaning, lineage should connect each derived value to source evidence and the rule that transformed it.
Prospective and retrospective provenance
| Form | Describes | Example question |
|---|---|---|
| Prospective | Intended workflow structure | Which step consumes the profile and rule files? |
| Retrospective | What happened in one run | Which file version and rule produced this city? |
A workflow diagram is prospective: it shows possible dependencies. A run log is retrospective: it records actual inputs, outputs, times, and decisions. One cannot substitute for the other.
In plain terms: prospective provenance is the recipe; retrospective provenance is the kitchen record from tonight’s service.
Ask provenance questions before choosing granularity
Lineage can be recorded at dataset, table, row, cell, or operation level. Finer detail costs more storage and instrumentation.
Start from questions:
- Why does this output row exist?
- Which inputs contributed to this aggregate?
- Which outputs would change if this source file were corrected?
- Which rule and version generated this canonical value?
- Was the value parsed, copied, inferred, or manually reviewed?
Dataset-level lineage cannot answer a cell-level “why.” Cell-level lineage may be unnecessary for a simple file copy.
Why-, where-, and how-provenance
- Why-provenance identifies contributing input witnesses.
- Where-provenance identifies the source location from which a value was copied.
- How-provenance records how contributions combined through operations.
For an inferred city, city_source = learned_crosswalk and city_rule = location majority, support 5, confidence .83 are more useful than a generic cleaned = true flag.
Reveal hidden workflows in scripts
Script-based pipelines often contain a workflow without declaring one. YesWorkflow-style annotations identify programs, inputs, outputs, and data links while leaving the executable language unchanged:
# @begin clean_menu
# @in raw_menu
# @in city_rules
# @out clean_menu
# @out change_log
...
# @end clean_menuAnnotations can generate a prospective graph. Retrospective capture still requires runtime identifiers, checksums, parameters, timestamps, and status.
Provenance must survive repair
Do not overwrite place with a guessed city. Prefer:
place_raw
canonical_city
city_source
city_rule
rule_version
confidence
review_statusThis design supports reprocessing when a rule changes and allows consumers to filter inferred values. A change log should use stable record IDs and distinguish filled, rewritten, cleared, unchanged, and still_blank.
Provenance has its own quality
Lineage can be incomplete or wrong. Validate that every derived non-null value has a source and rule, referenced artifacts exist, hashes match, and workflow edges correspond to real files. Provenance is not decorative metadata; it is data with constraints.
Review card
- Prospective provenance describes the plan; retrospective provenance records a run.
- Choose granularity from the questions lineage must answer.
- Why, where, and how provenance capture different explanations.
- Script annotations expose dependencies but do not replace runtime capture.
- Preserve raw values beside derived values and rules.
- Validate provenance with constraints like any other dataset.