Skip to content

data-removal.md

Latest commit

 

History

History
129 lines (111 loc) · 7.09 KB

data-removal.md

File metadata and controls

129 lines (111 loc) · 7.09 KB
Raw

Data removal

NeoMail includes an autonomous data-removal (privacy) engine that scans data-broker and people-search sites for a user's personal information and tracks opt-out requests to confirmation. It is a modular subsystem — its own tables, services, routes, and Flutter view — that works alongside the mail system without being entangled in its data model.

UI flow

The Flutter view walks the user through four steps: Identity (create or pick a profile) → Scan → Review (select exposures) → Track (removal status). Each step only unlocks once the previous one has enough state to act on, and the whole flow can be revisited from the step header.

Identity is write-only

  • A user creates an identity profile (name, aliases, contact info, addresses) that is encrypted at rest.
  • Only the first name is ever readable again — it's the one plaintext field, stored separately so the app can show something to pick from without ever decrypting the rest. Every other field (last name, aliases, emails, phones, DOB, addresses) can be set once at creation and is never returned by any route, MCP tool, or the audit log afterward.
  • There is no update endpoint. The only supported change to an existing profile is full deletion (DELETE /api/privacy/profiles/:id), which removes the encrypted identity and cascades to its scans, exposures, and removals.
  • The full identity is only ever decrypted in-process, for the specific scan or removal operation that needs it (getProfileIdentity in services/dataremoval/profiles.js) — never serialized back to a client.
  • Every route in server/routes/dataremoval.js and server/routes/api_v1.js requires an authenticated session or a scoped API key/OAuth token; nothing in this subsystem is reachable unauthenticated.

How scanning works

  • A scan builds a search-vector fan-out: every name/alias combined with each known city/state, plus each phone and email, filtered to what a given broker's search actually supports. Different vectors surface different listings for the same person (a maiden name, an old address).
  • A broker is scanned via either a hand-verified search URL template, or — for brokers where we only have a homepage — a small set of common people-search URL conventions tried against that domain (candidateTemplates in matcher.js). Both paths run through the exact same corroboration check below, so a wrong URL guess just comes back not_found/blocked; it can never produce a false positive. Exposures found via a guessed URL are flagged patternVerified: false in the API so the UI can ask the user to double-check before requesting removal.
  • A match requires corroboration — the subject's name must appear together with a known address, phone, email, or age. Name-only pages are never recorded as a positive; they are escalated to a single AI confirmation pass, and if that is inconclusive the broker is left not_found.
  • A fetch that errors, times out, or is bot-blocked is recorded as blocked, never as a found exposure. NeoMail does not assume presence on failure.
  • Broker fetches run with bounded concurrency (NEOMAIL_PRIVACY_SCAN_CONCURRENCY, default 8) so a full-catalog scan against ~886 scannable brokers finishes in a reasonable time instead of running fully serially.
  • Autonomous re-scans run on a schedule (NEOMAIL_PRIVACY_SWEEP_INTERVAL_MS, default daily) for users who opt into privacy.autonomyScans. Scans are the only thing that runs autonomously — removals always require the user to select and approve them.

How removal works

  • For brokers with an opt-out email, NeoMail sends a least-disclosure CCPA/GDPR/generic request from the user's own connected mail account (reusing the existing IMAP/SMTP credentials), so brokers see a request from the real mailbox owner rather than a third-party relay.
  • For simple opt-out forms, NeoMail submits the form directly.
  • Brokers that require phone verification, government ID, or a photo upload (Tier 3) are never automated — NeoMail surfaces the exact opt-out URL and a copy-ready request so the user can complete it manually.
  • Every removal moves through a validated state machine: submitted → verification_pending → awaiting_processing → confirmed_removed, with blocked and manual_pending side states. A removal is only marked confirmed after a re-scan proves the listing is gone.

Data stored

  • privacy_profiles — encrypted identity used for searches.
  • privacy_scans — one row per scan run, with progress counters.
  • privacy_exposures — a broker match with encrypted evidence and a confidence score.
  • privacy_removals — the opt-out request lifecycle, with encrypted response snippets and a full history.

All tables cascade-delete with the owning user, same as the mail schema.

Broker catalog

server/services/dataremoval/data/brokers.json ships with ~890 brokers, split into two tiers:

  • Scannable (~886 brokers). Actively scanned on every run. Split further by confidence, both surfaced via patternVerified on each exposure:
    • Hand-verified (~24 brokers). A confirmed search URL pattern and a checked list of identifiers its search supports (search.by).
    • Homepage-probed (~860 brokers). No verified pattern, only a homepage (search.homepage) — scanned by trying common people-search URL conventions against that domain. Most of this set was imported from the retired NeoDataRemoval project's broker data, which only ever recorded a homepage + opt-out method, never a search pattern. Guessing the URL is safe here precisely because the corroboration check above is what decides detected, not the URL itself — a wrong guess only costs a few wasted HTTP requests.
  • Opt-out-only (~5 brokers). No homepage and no verified pattern at all — pure B2B/ad-tech data brokers (e.g. Acxiom, LexisNexis, Oracle Data Cloud) with no consumer-facing person-search feature to probe in the first place. These are never scanned; they're still offered as preemptive opt-out targets since removing a real, working opt-out mechanism would be a net loss of privacy protection for no gain — they were never going to be scannable regardless of effort.

The catalog can be grown further in two ways:

  • Drop in an updated broker export at the same path (it fully replaces the bundled file).
  • Configure NEOMAIL_PRIVACY_REGISTRY_URL to point at a public broker registry feed — JSON or CSV (e.g. the California Data Broker Registry CSV export). POST /api/privacy/brokers/refresh merges new entries into a runtime cache without touching the bundled file.

API and MCP

REST endpoints live under /api/v1/privacy/*, gated by the privacy:read and privacy:write API-key scopes. The MCP host exposes privacy_start_scan, privacy_list_exposures, privacy_submit_removal, and privacy_removal_status so an agent can drive the same approval-gated workflow on the user's behalf. See API and MCP.