feat(compute-gateway): turn the read-path masking PDP ON - #1387
Merged
Conversation
⚠️ PRODUCTION BEHAVIOUR CHANGE. graph-query/graph-stats reads will begin returning masked direct identifiers to callers that previously received them in cleartext. masking.py has been mounted in the dispatch path since it landed, but with no policy configured it returned outputs unchanged. The 2026-08-04 audit found GATEWAY_MASKING_POLICY set nowhere under deploy/ — not a missing mechanism, an unused one. The moat existed and was switched off, so the WKC-parity claim rested on code that never ran. The policy masks direct identifiers on read (email/phone/msisdn hmac_pseudonym, ssn/national_id one_way_hash, patient_mrn chameleon_token, dob generalize, name/street redact) and encodes the no_health_adtech veto as a forbidden_mixture: an adtech-realm requester reading health-topic records is DENIED and the records are WITHHELD, not masked. The masking decision is appended as a sealed ComputeOutput so the Ed25519 receipt attests exactly what the caller received — the property WKC-style dynamic masking cannot produce, because there the enforcement leaves no verifiable artifact. tests/test_masking_policy_values.py parses GATEWAY_MASKING_POLICY out of the values file itself and exercises THAT string, not a fixture written to agree with it. A YAML typo now fails CI instead of silently reverting the gateway to passthrough — which is exactly how this ended up off in the first place. Covered: identifiers masked and non-identifiers preserved, the decision emitted, the veto withholding records, and an unknown scheme failing closed to redact rather than passing raw through. FOUND WHILE TESTING, and asserted rather than assumed: masking._iter_records reads `data["nodes"]` and returns [] for anything else, so a result shaped as rows/table/edges is returned UNMASKED even with this policy active. test_masking_coverage_boundary_is_nodes_only pins that limit so the next person reads it from a failing assertion rather than an incident. Widening it means changing _iter_records, which is a bigger change than turning the policy on and is deliberately not bundled here. 249 compute-gateway tests pass; preflight_deploy_contract green.
mdheller
pushed a commit
that referenced
this pull request
Aug 4, 2026
Closes a LIVE hole. The masking PDP went on in #1387, and `_iter_records` walked `data["nodes"]` and nothing else — so a result shaped as rows / table / edges was returned UNMASKED while the decision output reported the policy active. `rows` is a shape this estate actually emits (hellgraph + compute-gateway both do), so personal data was being served in cleartext on those paths with masking nominally enforced. I pinned that limit with a test rather than fixing it when the policy landed. That was the wrong call for something already enforcing in production. The walker is now shape-agnostic and depth-bounded: every nested dict is a candidate record. An allowlist of container keys only moves the hole to the next shape somebody adds. This is safe because _mask_record rewrites ONLY keys named in the policy's mask_fields — a dict with no configured field comes back untouched — so over-walking does nothing while under-walking serves personal data. test_over_walking_is_harmless asserts a payload with no configured field is returned byte-identical, so widening coverage cannot corrupt unrelated envelopes. output.data itself is excluded: it is the envelope, not a record, and masking its top level would rewrite container keys that happen to collide with a field name. Tests: every shape (nodes/rows/edges/records) asserted to leak nothing; nested payloads reached; over-walking harmless; READ_KINDS unchanged so widening the walker does not widen the kinds governed. 252 pass.
mdheller
pushed a commit
that referenced
this pull request
Aug 4, 2026
Closes a LIVE hole. The masking PDP went on in #1387, and `_iter_records` walked `data["nodes"]` and nothing else — so a result shaped as rows / table / edges was returned UNMASKED while the decision output reported the policy active. `rows` is a shape this estate actually emits (hellgraph + compute-gateway both do), so personal data was being served in cleartext on those paths with masking nominally enforced. I pinned that limit with a test rather than fixing it when the policy landed. That was the wrong call for something already enforcing in production. The walker is now shape-agnostic and depth-bounded: every nested dict is a candidate record. An allowlist of container keys only moves the hole to the next shape somebody adds. This is safe because _mask_record rewrites ONLY keys named in the policy's mask_fields — a dict with no configured field comes back untouched — so over-walking does nothing while under-walking serves personal data. test_over_walking_is_harmless asserts a payload with no configured field is returned byte-identical, so widening coverage cannot corrupt unrelated envelopes. output.data itself is excluded: it is the envelope, not a record, and masking its top level would rewrite container keys that happen to collide with a field name. Tests: every shape (nodes/rows/edges/records) asserted to leak nothing; nested payloads reached; over-walking harmless; READ_KINDS unchanged so widening the walker does not widen the kinds governed. 252 pass.
mdheller
added a commit
that referenced
this pull request
Aug 4, 2026
Closes a LIVE hole. The masking PDP went on in #1387, and `_iter_records` walked `data["nodes"]` and nothing else — so a result shaped as rows / table / edges was returned UNMASKED while the decision output reported the policy active. `rows` is a shape this estate actually emits (hellgraph + compute-gateway both do), so personal data was being served in cleartext on those paths with masking nominally enforced. I pinned that limit with a test rather than fixing it when the policy landed. That was the wrong call for something already enforcing in production. The walker is now shape-agnostic and depth-bounded: every nested dict is a candidate record. An allowlist of container keys only moves the hole to the next shape somebody adds. This is safe because _mask_record rewrites ONLY keys named in the policy's mask_fields — a dict with no configured field comes back untouched — so over-walking does nothing while under-walking serves personal data. test_over_walking_is_harmless asserts a payload with no configured field is returned byte-identical, so widening coverage cannot corrupt unrelated envelopes. output.data itself is excluded: it is the envelope, not a record, and masking its top level would rewrite container keys that happen to collide with a field name. Tests: every shape (nodes/rows/edges/records) asserted to leak nothing; nested payloads reached; over-walking harmless; READ_KINDS unchanged so widening the walker does not widen the kinds governed. 252 pass. Co-authored-by: mdheller <charles.peterson@socioprophet.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The moat existed and was switched off
masking.pyhas been mounted in the compute-gateway dispatch path since it landed. With no policy configured it returns outputs unchanged. The 2026-08-04 audit foundGATEWAY_MASKING_POLICYset nowhere underdeploy/— not a missing mechanism, an unused one.So the WKC-parity claim rested on code that never ran. This turns it on.
What the policy does
email,email_address,phone,msisdnhmac_pseudonymssn,national_idone_way_hashpatient_mrnchameleon_tokendate_of_birth,dobgeneralizefull_name,street_addressredactPlus the
no_health_adtechveto as aforbidden_mixture: an adtech-realm requester reading health-topic records is denied and the records withheld, not masked.The masking decision is appended as a sealed
ComputeOutput, so the Ed25519 receipt attests exactly what the caller received. That's the property WKC-style dynamic masking cannot produce — there, enforcement leaves no verifiable artifact.The policy is under test, from the values file
tests/test_masking_policy_values.pyparsesGATEWAY_MASKING_POLICYout ofdeploy/values/compute-gateway.yamland exercises that string — not a fixture written to agree with it. A YAML typo now fails CI instead of silently reverting the gateway to passthrough, which is exactly how this ended up off in the first place.Covered: identifiers masked while non-identifiers survive; the decision emitted; the veto withholding records entirely; an unknown scheme failing closed to redact rather than passing raw through.
Found while testing — asserted, not assumed
masking._iter_recordsreadsdata["nodes"]and returns[]for anything else. A result shaped as rows / table / edges is returned UNMASKED even with this policy active.test_masking_coverage_boundary_is_nodes_onlypins that limit so the next person reads it from a failing assertion rather than from an incident. Widening it means changing_iter_records— a bigger change than turning the policy on, deliberately not bundled here.Verification
249 compute-gateway tests pass ·
preflight_deploy_contract.pygreen.Pairs with SocioProphet/sociosphere#581, which binds
masking-decision.v1to a classification stance so the PDP can tell "confidently public" from "we have no idea what this is".