Skip to content

Add FakeJson and RebaseUrl transformers - #222

Draft
MustafaAldelaimi wants to merge 2 commits into
mainfrom
anon-local-dev/fake-json-rebase-url
Draft

MustafaAldelaimi wants to merge 2 commits into
mainfrom
anon-local-dev/fake-json-rebase-url

Conversation

@MustafaAldelaimi

@MustafaAldelaimi MustafaAldelaimi commented Jul 3, 2026

Copy link
Copy Markdown

What

Two new transformers, motivated by the anonymised-local-dev epic's principle that local environments must run unmodified services — the strategy adapts to the product, never the reverse.

FakeJson — shape-preserving JSON anonymisation

Fakes a JSON value while preserving its shape: object keys, array lengths and value types are kept; string leaves become fake words of similar length; numbers, booleans and null pass through. Unparseable input collapses to {} (the EmptyJson behaviour) so nothing fails open.

Optional arg preserve_string_max_len: string leaves of at most N characters pass through unfaked, for enum-like option values ("Yes"/"No", option labels) that consuming services compare against. Free text (long strings) is always faked.

Why: strategies currently use EmptyJson on JSON columns whose fields downstream code reads (e.g. aurora Answers.answerBody.answerText, read by both the survey UI and the programme-recommendation scoring). Emptying the object crashes/short-circuits unmodified consumers; faking the leaves preserves consumability while stripping PII.

RebaseUrl — rebase stored URLs onto a local origin

Replaces a URL's scheme+host(+port) with a base arg, keeping path and query (path segments are Identity-preserved ids, so deep links keep resolving locally). Optional only_hosts arg (comma-separated): only URLs on a listed host are rebased; anything else — third-party links (survey tools, video links) or non-URL values — passes through unchanged. Without only_hosts, values with no recognisable origin become base alone, so a production host never survives. Optional template arg rebuilds the value from sibling columns of the same row (same mechanism as id_column), e.g. {base}/survey/skills-scan/{external_id}.

Why: URL columns are currently either scrambled (dead links) or passed through (links to production). Both break local journeys; per-service post-restore SQL rewrites are the current workaround. Real data mixes service URLs with third-party ones in a single column (measured in user_home user_tasks.action_url: ~84.6k multiverse-host rows vs ~1.2k qualtrics), hence only_hosts.

Notes

  • Both transformers handle Postgres COPY text escapes (\\, \n, \t, \r): FakeJson unescapes before serde_json parsing and re-escapes on output.
  • No new dependencies (serde_json and fake were already in the tree).
  • 15 new unit tests; cargo test 188 passed, clippy and fmt clean.
  • Functionally smoke-tested against real data: 221 real answerBody rows anonymised → all valid JSON, key/type/array-length distribution identical before/after, zero original strings survive.
  • Strategy-side adopters queued in the epic: aurora Answers.answerBody/detailsFakeJson; user_home user_skills_scans.url + user_tasks.action_urlRebaseUrl.

🤖 Generated with Claude Code

MustafaAldelaimi and others added 2 commits July 3, 2026 15:52
FakeJson anonymises a JSON value while preserving its shape: object keys,
array lengths and value types survive; string leaves become fake words of
similar length; numbers, booleans and null pass through; unparseable input
collapses to {} so nothing fails open. An optional preserve_string_max_len
arg keeps short enum-like option values (e.g. "Yes"/"No") that consuming
services compare against. This lets strategies stop emptying JSON columns
(EmptyJson) that downstream code reads fields from.

RebaseUrl rebases a stored URL onto a different origin, keeping the path
and query so identity-preserved ids in the path keep resolving. An optional
template arg rebuilds the value from sibling columns of the same row
(e.g. {base}/survey/skills-scan/{external_id}). This replaces per-service
post-restore SQL that rewrote URL columns after anonymisation.

Both handle Postgres COPY text escapes: FakeJson unescapes before parsing
and re-escapes on output, matching how JSON cells appear in dumps.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
URL columns can mix service links with third-party links (survey tools,
video links) in one column. only_hosts takes a comma-separated host list:
URLs on a listed host are rebased onto base; everything else — including
values that are not URLs at all — passes through unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant