📔 feat: Preserve Owner Text During PII Redaction - #16311
lia-by-librechat[bot] wants to merge 10 commits into
Conversation
|
Review handoff for exact remote head PR 2 adds opt-in text-only interactive redaction, atomic canonical text plus encrypted owner sidecar storage, a tenant/owner-authorized batch view, and a display-only UI context. Normal history, sharing, exports, request execution, and diagnostics do not receive the original. The browser test inspects the actual mock-provider message projection, then exercises owner reload, network retry, sharing, unauthorized retrieval, ciphertext storage, and deletion. Verification: 681 focused tests passed (139 API, 142 data-schemas, 14 filter-schema, 74 client, 312 legacy/controller); the final browser test passed; all four changed TypeScript workspace checks, package builds, and staged static checks passed. The synthetic PII canary was absent from the successful browser-run server log. Invariant self-review covered ingress ordering, initial and fallback writers, persistence failure/admission, retry identity, tenant and owner isolation, canonical/share readers, display-cache invalidation, expiry/deletion, and rollback. No independent review has arrived for this head yet. Explicit limits: the default is unchanged; non-text and alternate ingress retain blocking; no historical migration or bound-secret protection is enabled here. The sidecar shares message deletion/retention. CREDS_KEY rotation without retaining the old key makes prior originals unreadable; canonical filtered content remains usable. Local Lighthouse was attempted before and after UI changes but did not produce a passing result: the initial full-browser launch lacked system libraries and the later audit exceeded the 90-second tool budget. No performance score is claimed. Light/dark screenshots were captured, but GitHub media upload rejected App authentication and they are not attached. CI and fresh review still need to cover this SHA. |
|
Lighthouse CI failed. The last 80 log lines contain the measured budgets and assertion failures. |
|
Review handoff for exact remote head |
|
@codex review the latest head |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5634f3fa08
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Review handoff for exact remote head |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 28ed3d1ea8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
danny-avila
added
🗺️ Moderation Sec
# Conflicts: # api/server/controllers/agents/__tests__/request.partialDisconnect.spec.js # api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js # client/src/locales/en/translation.json
|
Review handoff for exact remote head |
|
@codex review the latest head |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 27bd2242a1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Exact-head review handoff: |
|
New exact-head review handoff: |
|
Exact-head review handoff: |
|
Exact-head review handoff: |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 22734756b2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (shouldSaveMessage) { | ||
| await saveMessage( | ||
| await savePrivateTextMessage( |
There was a problem hiding this comment.
When a PII-containing first turn is rejected by checkBan or a message limiter, the client sends no conversation ID and uses NO_PARENT, so shouldSaveMessage is false and this protected save is skipped. The preceding created event nevertheless replaces the client's optimistic plaintext with the canonical text and privacyRevision; OwnerTextProvider then queries a sidecar that was never stored and permanently displays “Original unavailable,” including after Retry. Preserve the owner view for this unsaved failure path or avoid stamping an unrecoverable revision.
AGENTS.md reference: AGENTS.md:L42-L44
Useful? React with 👍 / 👎.
| const text = result.content.replace( | ||
| /\[(EMAIL|PHONE|NAME|CREDENTIAL|CUSTOM)_(\d+)\]/g, | ||
| (marker, category: string, index: string) => | ||
| body.text.includes(marker) ? marker : `[${category}_${index}_${revision}]`, |
There was a problem hiding this comment.
When the redact policy includes a custom pattern that matches the generated 32-character hexadecimal revision (for example, a credential rule matching [a-f0-9]{32}), this suffix introduces a new PII match that was not present in the submitted text. The normal chat middleware at api/server/routes/agents/chat.js:75-80 subsequently inspects the transformed body again, treats the redact finding as blocking, and rejects an otherwise supported submission with a content-filter error. Mark captured text as already transformed or prevent the downstream inspector from evaluating the generated placeholders.
AGENTS.md reference: AGENTS.md:L42-L44
Useful? React with 👍 / 👎.
| <OwnerTextProvider | ||
| messages={messages} | ||
| conversationId={conversationId} | ||
| isSubmitting={chatHelpers.isSubmitting} |
There was a problem hiding this comment.
During a successful first turn, this prop is still the route value new; the created event has already replaced the optimistic message with a protected message whose own conversationId is the newly allocated UUID, but navigation to that UUID occurs only in the final handler. The provider therefore posts to /api/messages/new/owner-text, which the UUID validation rejects, and shows “Original unavailable” for the entire model generation even though the sidecar is persisted before the provider call. Resolve the owner-text scope from the acknowledged message/conversation ID so the original can load while the first response is streaming.
AGENTS.md reference: AGENTS.md:L42-L44
Useful? React with 👍 / 👎.
This branch has not been deployed
2 participants
Summary
With
filters.messages.pii.action: redact, fresh text-only interactive Agent chat submissions now continue with a filtered model-facing copy. The owner can still read the submitted original after reload. Other ingress and unsupported submission shapes retain the existing blocking behavior. The default action is unchanged.The filtered text is canonical in request execution, message storage, ordinary message reads, sharing, and exports. Originals use a separately authorized display endpoint and React context, not the ordinary message/query cache, edit callbacks, or prompt history. Browser request diagnostics now log identifiers rather than submitted text.
How it works
The sidecar is a
select: false, unindexed field on the message, rather than a separate collection. This keeps canonical text and the encrypted original atomic and gives them the same deletion/TTL lifecycle, without orphan cleanup. AES-GCM authenticates the owner, tenant, conversation, message, revision, and canonical text. The key is derived fromCREDS_KEYwith domain separation. Keyed, submission-specific revisions preserve network-retry identity and prevent unrelated placeholders from aliasing across history.The owner endpoint batches up to 50 IDs, returns
Cache-Control: private, no-store, and never grants original access from sharing or administrator status. It loads up to three batches concurrently and reuses only unchanged owner-, tenant-, conversation-, revision- and text-matched originals. A missing result while a turn is still being persisted is retried once when submission settles; failures and unavailable originals retain a localized manual retry action. Missing keys, tampering, stale rows, and decryption failures fall back to filtered text. Persistence failure prevents the main provider call.Every text-bearing Agent chat POST loads the configured policy before early ban denial; only supported fresh interactive turns are transformed. Denials with matching untransformed text or missing policy fail closed before emitting or saving it. Protected Stop carries the nonsecret revision through preliminary and created-event job metadata to the abort final event; it verifies the existing sidecar without rewriting it. For older revisionless job records, the ordinary prerequisite is insert-only and cannot overwrite an existing protected row; identified filtered placeholders require an existing protected row. Canonical edits atomically discard stale private fields; imports, forks, and bulk copies cannot transplant private fields to new identities. Browser diagnostics project only identifiers.
Scope and rollout
redactaction with selected text patterns and retain a validCREDS_KEY.CREDS_KEYmakes existing originals unavailable. Preserve the key with encrypted backups. Canonical filtered messages remain usable.blockstops transformation without restoring raw text into canonical storage. Older clients can still read filtered messages. This does not retroactively remove PII from historical records or infrastructure body capture.Type of change
Testing
packages/api,packages/data-schemas,packages/data-provider, andclient; package builds and touched-file static checks.Subsystem self-review covered the early ingress boundary, ban and message-limit denial, Stop persistence, ordinary and fallback user-message writers, retry identity, owner/tenant authorization, canonical and shared readers, display/cache isolation, expiration/deletion, key loss, and rollback. Codex reviewed the earlier
27bd2242a1head; its three newly reported lifecycle findings are addressed on the pushed head22734756b2, which awaits independent review.Screenshots
The owner view shows the original while canonical storage and sharing retain typed placeholders. The test also verifies reload and captures light and dark mode. Files remain in the worktree under
e2e/specs/.test-results/private-text-*/owner-text-{light,dark}.png; no uploaded image is claimed.Title verification: 📔 has 1 prior subject-leading use in 5,540 indexed LibreChat commits through 2026-09-23 23:08:36. The earlier use was a hosting guide, not the owner-original frame; the sentinel passed.