Add FakeJson and RebaseUrl transformers - #222
Draft
MustafaAldelaimi wants to merge 2 commits into
Draft
MustafaAldelaimi wants to merge 2 commits into
MustafaAldelaimi wants to merge 2 commits into
Conversation
FakeJson anonymises a JSON value while preserving its shape: object keys,
array lengths and value types survive; string leaves become fake words of
similar length; numbers, booleans and null pass through; unparseable input
collapses to {} so nothing fails open. An optional preserve_string_max_len
arg keeps short enum-like option values (e.g. "Yes"/"No") that consuming
services compare against. This lets strategies stop emptying JSON columns
(EmptyJson) that downstream code reads fields from.
RebaseUrl rebases a stored URL onto a different origin, keeping the path
and query so identity-preserved ids in the path keep resolving. An optional
template arg rebuilds the value from sibling columns of the same row
(e.g. {base}/survey/skills-scan/{external_id}). This replaces per-service
post-restore SQL that rewrote URL columns after anonymisation.
Both handle Postgres COPY text escapes: FakeJson unescapes before parsing
and re-escapes on output, matching how JSON cells appear in dumps.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
URL columns can mix service links with third-party links (survey tools, video links) in one column. only_hosts takes a comma-separated host list: URLs on a listed host are rebased onto base; everything else — including values that are not URLs at all — passes through unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two new transformers, motivated by the anonymised-local-dev epic's principle that local environments must run unmodified services — the strategy adapts to the product, never the reverse.
FakeJson— shape-preserving JSON anonymisationFakes a JSON value while preserving its shape: object keys, array lengths and value types are kept; string leaves become fake words of similar length; numbers, booleans and null pass through. Unparseable input collapses to
{}(theEmptyJsonbehaviour) so nothing fails open.Optional arg
preserve_string_max_len: string leaves of at most N characters pass through unfaked, for enum-like option values ("Yes"/"No", option labels) that consuming services compare against. Free text (long strings) is always faked.Why: strategies currently use
EmptyJsonon JSON columns whose fields downstream code reads (e.g. auroraAnswers.answerBody.answerText, read by both the survey UI and the programme-recommendation scoring). Emptying the object crashes/short-circuits unmodified consumers; faking the leaves preserves consumability while stripping PII.RebaseUrl— rebase stored URLs onto a local originReplaces a URL's scheme+host(+port) with a
basearg, keeping path and query (path segments are Identity-preserved ids, so deep links keep resolving locally). Optionalonly_hostsarg (comma-separated): only URLs on a listed host are rebased; anything else — third-party links (survey tools, video links) or non-URL values — passes through unchanged. Withoutonly_hosts, values with no recognisable origin becomebasealone, so a production host never survives. Optionaltemplatearg rebuilds the value from sibling columns of the same row (same mechanism asid_column), e.g.{base}/survey/skills-scan/{external_id}.Why: URL columns are currently either scrambled (dead links) or passed through (links to production). Both break local journeys; per-service post-restore SQL rewrites are the current workaround. Real data mixes service URLs with third-party ones in a single column (measured in user_home
user_tasks.action_url: ~84.6k multiverse-host rows vs ~1.2k qualtrics), henceonly_hosts.Notes
\\,\n,\t,\r):FakeJsonunescapes beforeserde_jsonparsing and re-escapes on output.serde_jsonandfakewere already in the tree).cargo test188 passed, clippy and fmt clean.answerBodyrows anonymised → all valid JSON, key/type/array-length distribution identical before/after, zero original strings survive.Answers.answerBody/details→FakeJson; user_homeuser_skills_scans.url+user_tasks.action_url→RebaseUrl.🤖 Generated with Claude Code