Regular expressions describe patterns in text. In data cleaning they are useful for three distinct jobs: validating a shape, extracting components, and transforming a known representation. Problems begin when a shape match is treated as proof of meaning.
Read a pattern in layers
For an IPv4-like shape:
^(\d{1,3}\.){3}\d{1,3}$^and$anchor the whole value;\d{1,3}matches one to three digits;\.matches a literal dot;{3}repeats the grouped prefix three times.
It accepts 999.999.999.999, so it validates syntax but not the numeric domain. Parse the four components and enforce 0..255 separately.
In plain terms: regex can recognize the shape of an address on an envelope. It cannot prove the address exists.
Match, extract, and replace are different operations
import re
raw = "2026/9/7"
m = re.fullmatch(r"(\d{4})/(\d{1,2})/(\d{1,2})", raw)
if m:
year, month, day = map(int, m.groups())After extraction, use a date library to validate month lengths and leap years. Only then emit an ISO value such as 2026-09-07.
A replacement should target what was captured:
re.sub(r"^([^,]+),\s*([^,]+)$", r"\2 \1", "Lin, Johnny")This transformation is safe only if the source convention is known. Names are not universally two comma-separated tokens.
Avoid the common traps
| Trap | Symptom | Better approach |
|---|---|---|
| Missing anchors | Valid substring inside invalid text passes | Use full match for validation |
| Greedy wildcard | One group consumes too much | Use explicit delimiters or non-greedy match |
| Catastrophic backtracking | Runtime spikes on crafted input | Simplify nested repetition and bound length |
| One giant pattern | Impossible to review or explain | Parse in stages with named groups |
| Engine assumptions | Pattern works in one tool only | Record flavor and flags |
Unicode also matters. \w, case folding, and character classes behave differently across engines and flags. Test accents, non-Latin scripts, and composed versus decomposed characters when the dataset contains them.
Build a test matrix before mass editing
| Case | Input | Expected |
|---|---|---|
| Normal | john@example.com |
accept |
| Boundary | one-character local part | accept if policy allows |
| Malformed | missing domain | reject |
| Near miss | trailing comment or space | normalize or reject explicitly |
| Adversarial | very long repeated text | finish quickly |
Store the examples beside the cleaning rule. A regex without positive and negative fixtures is an undocumented guess.
Know when to stop using regex
Use a parser for nested formats, CSV quoting, JSON, URLs, dates, and programming-language syntax. Use reference data or business rules for identity and geography. Regex remains valuable at the boundary: extracting a candidate or rejecting an impossible shape before stronger validation.
Review card
- Anchor validation patterns to the whole value.
- Separate syntactic matching from domain and semantic validation.
- Extract components first; canonicalize only after validation.
- Record regex flavor, flags, and Unicode assumptions.
- Test valid, invalid, boundary, near-miss, and adversarial inputs.
- Prefer a real parser when the format has nested grammar.