Skip to content

Keep report privacy replacements consistent and synchronize review panes - #1699

Merged
elkimek merged 17 commits into
elkimek:mainfrom
bhargavmehta01:feat/consistent-privacy-redaction
Nov 2, 2026
Merged

elkimek merged 17 commits into
elkimek:mainfrom
bhargavmehta01:feat/consistent-privacy-redaction

Conversation

@bhargavmehta01

@bhargavmehta01 bhargavmehta01 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem and behavior

The existing regex path already uses fixed Czech patient aliases, but other identifying values receive fresh replacements per occurrence, and some report layouts retain names or consume neighboring fields. This change keeps replacements consistent throughout one report while preserving clinical text and avoiding a permanent cross-report pseudonym key.

  • Recognized patient names use conventional fictional names: Jan Novák / Jana Nováková for Czech reports, Peter Horváth / Petra Horváthová for Slovak reports, and John Doe / Jane Doe for English reports. Specific Czech/Slovak report labels supply the language hint; shared, mixed or unknown clues use the English alias set without suppressing name removal. Reported sex, including multiline fields, takes precedence over profile sex. Aliases also avoid original labeled first/surname components so local-model final validation does not reject an intentional replacement. If every conventional and anonymous alias collides, a non-name mask is used.
  • Repeated identifiers use HMAC-SHA256 with a fresh, lazy, in-memory report key. The SHA-256 implementation comes from the existing vendored library. The key is neither stored nor synced.
  • Explicit name fields handle short and lowercase names, including Slovak Meno and Priezvisko without a profile-name hint. Slovak sex, DOB, clinician and address labels are supported; whitespace-separated physician/address labels remain supported. Standalone name matches preserve measured clinical rows such as White blood cell count and Free T4. Empty identifying fields cannot consume the next result line.
  • Recognized DOBs retain their year and agree across repeated copies, including Birth Date labels and strict multiline date values. Empty DOB fields cannot consume clinical rows. Age stays unchanged, and labeled collection/report dates remain intact even when their value equals the DOB.
  • Local-model preprocessing and finalization share the captured profile hints, alias and key. Streaming review displays and sends the finalized result with its actual character count.
  • Review panes synchronize vertical and horizontal scrolling while the prepared text stays editable.

Merged current main and regenerated the module map and DOM-sink inventory. Public documentation is prepared separately in getbased-docs #26, to land with or after this change.

Validation

Using Node.js 24.22.0 and the project's TypeScript 7 toolchain:

  • Four focused unit suites: 146 passed (pii-redaction, pii-scroll-sync, pii-review-presentation, pii-review-styles).
  • Existing node tests/test-pii.js checks: 47 passed.
  • Focused Chromium PII and PDF-import specs: 14 passed, including Slovak aliases across repeated pages, real pane scrolling, clinical-text preservation and edited Send payloads.
  • Application/test migration typechecks, Svelte checks, architecture check, DOM-sink audit, and quality guardrails passed.
  • The PR Actions run passed on 3b28960e: compiler/quality and production-artifact preparation, unit tests, native browser checks, all four selected Chromium shipping shards, resource limits and the required test gate.
  • CodeQL passed. Greptile reviewed the same commit with 5/5 confidence, all three numbered findings addressed and no new actionable issues. No unresolved review threads remain.

Verification used synthetic reports and controlled provider responses. This PR run used affected selection with native Chromium and four Chromium shipping shards. Additional browser engines and combined coverage belong to nightly/full release-evidence runs and were not part of this scoped run. These checks do not establish recognition of every real report layout or live model behavior.

bhargavmehta01 and others added 3 commits October 30, 2026 16:59
Regex redaction before the analysis model now uses keyed, deterministic
replacements (HMAC-SHA256 with a random per-profile key), so a value maps
to the same replacement on every page and in every report of a profile:

- Patient names are removed ([PATIENT]) in every written form ("Last,
  First", "First M Last", upper case), detected from labels, header
  layouts before DOB, and the profile name when it is a full name.
- Dates of birth keep the real year; only month/day are replaced, the
  same way everywhere (including the profile DOB in any format). Age is
  left unchanged so age-based analysis stays correct.
- IDs are format-preserving; physicians, addresses, cities/ZIPs, phones
  and emails get neutral keyed placeholders instead of random Czech data.
- Labeled values stop at the next column/label, so labels such as
  "Patient Report" or "Specimen Details" are never treated as values and
  neighbouring fields on the same line survive.
- The local-model path pre-redacts deterministically first, so repeated
  identifiers stay consistent there too.

The Review & Edit panes now scroll together (vertical and horizontal),
following the textarea, diff preview, or pane itself, with echo
suppression to avoid feedback loops.
@greptile-apps

greptile-apps Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The reviewed changes appear safe to merge, with all three numbered findings addressed.

Summary

Keeps privacy replacements consistent within each report and synchronizes the review panes.

  • Report redaction keeps repeated replacements matched and protects clinical details.
  • Streamed review shows and sends the finalized text.
  • The original and prepared report panes scroll together.

No new actionable issues were found in the changes since the previous review. All three numbered findings are addressed. Related repositories showed no incompatible code consumers. Checks here were based on source inspection; tests were not rerun.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Report and captured profile hints] --> B[Report-local redaction]
  B --> C[Local model]
  C --> D[Final privacy check]
  B -->|Keep supplied replacements| D
  D --> E[Editable review]
  E --> F[Send finalized or edited text]
Loading

Reviews (12) · Last reviewed commit: "Fix privacy alias collisions and preserv..." · Reviewed by Greptile

Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii.ts Resolved
Comment thread js/pii-redaction.ts Resolved
- Accept English name labels without a colon (full names only, at line or column start) and lowercase full names behind explicit labels
- Replace uppercase month DOBs and keep their case
- Replace letters-only and dotted labeled IDs; route labeled birth numbers through the birth-number replacement
- Keep labeled phone matches within one field and stop treating pH as a phone label
- Keep the deterministic pre-pass replacements in the final pass and send the finalized text from streaming review
Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
@greptile-apps

greptile-apps Bot commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Want your agent to iterate on Greptile's feedback? Try greploops.

Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
@bhargavmehta01

Copy link
Copy Markdown
Contributor Author

Addressed all the review feedback and merged the latest main (966727b), so the conflict is gone. CI is green except one unit test, coverage-producer.test.ts, which hit its 30s timeout (31.4s here; it passed at 27.3s on the previous run of this branch and doesn't touch the redaction code). @elkimek could you re-run that job and take another look when you get a chance? Thanks!

Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
…ates only beside their label

- A surname like Low is kept only when it trails a number on a result line
  with a unit, never right after a label (Surname: Low  Age: 27).
- An 8-digit date is kept only when it sits directly beside a collection or
  report label, so a later Order: number on the same line is still redacted.
- Regression tests use the exact review examples.
Comment thread js/pii-redaction.ts Resolved
Patient and physician names written as "Smith, Jane Anne Marie" kept only
the first middle name, so "Marie" survived. All given names after the comma
are now kept, and each middle name or its initial can appear in any written
form of the name.
@elkimek elkimek changed the title feat: Make privacy redaction consistent and sync the review panes Keep report privacy replacements consistent and synchronize review panes Nov 2, 2026
Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
Comment thread js/pii-redaction.ts Resolved
@elkimek
elkimek merged commit 8f689c6 into elkimek:main Nov 2, 2026
23 checks passed
elkimek added a commit that referenced this pull request Nov 3, 2026
#1711)

* Keep every dated medication course visible in supplement AI context

The supplements section of the AI context capped compact mode at 12
products (24 in detail), ranked active records first, and summarised the
rest as an unnamed '+N more therapy records' line: the 'Other stored
therapy records' list was diffed against the pre-cap set, so it was empty
whenever the inventory itself was the input. A medication stopped before
the latest blood draw lost to any current supplement, and the model
received neither its name nor its dates. Users who log each dose change as
a separate record (common for titrated medications) lose exactly the
course that explains their lab history, and the model then fills the gap
with invented dates.

- Fill the character budget instead of a fixed product cap (cap kept as a
  safety net: 40 compact / 60 detail).
- Rank records named in the question first, then medications before
  supplements; within each, current before lab-range overlapping before
  unrelated.
- List every record without full detail by name, type, status and date
  ranges.
- Match named records case/diacritic-insensitively with a leading-word
  stem, including genericName and brand, so inflected mentions open
  detail mode; add folded Slovak/Czech therapy vocabulary.
- Add a completeness rule: never fill a therapy timeline gap by assumption.
- Report 'N of M records detailed' in the chat context receipt.

Fixes #1710

* Keep report privacy replacements consistent and synchronize review panes (#1699)

* Make privacy redaction consistent and sync the review panes

Regex redaction before the analysis model now uses keyed, deterministic
replacements (HMAC-SHA256 with a random per-profile key), so a value maps
to the same replacement on every page and in every report of a profile:

- Patient names are removed ([PATIENT]) in every written form ("Last,
  First", "First M Last", upper case), detected from labels, header
  layouts before DOB, and the profile name when it is a full name.
- Dates of birth keep the real year; only month/day are replaced, the
  same way everywhere (including the profile DOB in any format). Age is
  left unchanged so age-based analysis stays correct.
- IDs are format-preserving; physicians, addresses, cities/ZIPs, phones
  and emails get neutral keyed placeholders instead of random Czech data.
- Labeled values stop at the next column/label, so labels such as
  "Patient Report" or "Specimen Details" are never treated as values and
  neighbouring fields on the same line survive.
- The local-model path pre-redacts deterministically first, so repeated
  identifiers stay consistent there too.

The Review & Edit panes now scroll together (vertical and horizontal),
following the textarea, diff preview, or pane itself, with echo
suppression to avoid feedback loops.

* Skip profile labels as patient names and create the pseudonym key lazily

* Expect the regex fallback to redact the patient name instead of faking one

* Close redaction gaps from review and keep pre-pass replacements stable

- Accept English name labels without a colon (full names only, at line or column start) and lowercase full names behind explicit labels
- Replace uppercase month DOBs and keep their case
- Replace letters-only and dotted labeled IDs; route labeled birth numbers through the birth-number replacement
- Keep labeled phone matches within one field and stop treating pH as a phone label
- Keep the deterministic pre-pass replacements in the final pass and send the finalized text from streaming review

* Keep the obfuscatePDFText return shape compatible for review callers

* Keep empty ID labels from consuming the next line

* Redact titled physician names after a label without a colon

* fix: redact odd DOB formats and spaced Czech addresses; keep result flags and compact collection dates

* fix: keep DOB labels on one line, narrow result-flag and compact-date exceptions

* fix: keep result-flag surnames only after a measured value, compact dates only beside their label

- A surname like Low is kept only when it trails a number on a result line
  with a unit, never right after a label (Surname: Low  Age: 27).
- An 8-digit date is kept only when it sits directly beside a collection or
  report label, so a later Order: number on the same line is still redacted.
- Regression tests use the exact review examples.

* fix: keep every given name after the comma in Last, First Middle names

Patient and physician names written as "Smith, Jane Anne Marie" kept only
the first middle name, so "Marie" survived. All given names after the comma
are now kept, and each middle name or its initial can appear in any written
form of the name.

* Add Slovak privacy aliases and recognize Slovak report fields

* Fix privacy alias collisions and preserve labeled birth-date copies

---------

Co-authored-by: bhargavmehta01 <bhargavmehta01@users.noreply.github.com>
Co-authored-by: bhargavmehta01 <16927944+bhargavmehta01@users.noreply.github.com>
Co-authored-by: Elkim <36666630+elkimek@users.noreply.github.com>

* Retry reports after privacy-review outages and retire stale model output (#1700)

* Retry auto-imported reports when the local privacy model is unavailable

A report whose privacy review could not run, or was cancelled after the
local privacy model failed (or while a configured model is unreachable),
was recorded as an explicit skip, so the auto-import source never offered
it again. Report it as failed instead so it is retried on a later check.
Other review callers are unchanged: the new result is opt-in.

* Ignore stale privacy streams and cover batch retry outcomes

A stream aborted for the regex fallback could reject later with a plain
Error (the reachability probe rewraps the abort), which marked the local
model as failed again and turned an explicit cancel into a retry. Ignore
results from a stream that was aborted or replaced.

Add tests through the real handleBatchPDFs path: an unavailable review and
a cancel while the configured local model is unreachable report 'failed';
ordinary cancels report 'skipped'.

---------

Co-authored-by: bhargavmehta01 <bhargavmehta01@users.noreply.github.com>
Co-authored-by: bhargavmehta01 <16927944+bhargavmehta01@users.noreply.github.com>
Co-authored-by: Elkim <36666630+elkimek@users.noreply.github.com>

* Add medication-aware demos and proportionate chat safety

* Bound therapy context by actual coverage and align browser fixtures

* Select demo refresh behavior tests in affected CI plans

* Preserve specimen duration and volume during privacy redaction

* Cache demo treatment data and refresh affected test fixtures

* Stabilize owned chat fixtures and retain wellness prompt contracts

* Cancel dashboard auto-open before discussion fixture setup

* Enable paused AI before asserting the shipping chat composer

* Wait for the delayed profile write before replacing its view

* Persist native conversation fixtures before testing chat controls

---------

Co-authored-by: Bhargav Mehta <bhargav.mehta01@gmail.com>
Co-authored-by: bhargavmehta01 <bhargavmehta01@users.noreply.github.com>
Co-authored-by: bhargavmehta01 <16927944+bhargavmehta01@users.noreply.github.com>
Co-authored-by: Elkim <36666630+elkimek@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants