Skip to content

Add persistent dataset search - #3

Merged
KingAkeem merged 10 commits into
mainfrom
agent/add-dataset-search
Jul 17, 2026
Merged

KingAkeem merged 10 commits into
mainfrom
agent/add-dataset-search

Conversation

@KingAkeem

@KingAkeem KingAkeem commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor

What changed

  • Add a persistent SQLite dataset service for normalized profile and phone records.
  • Add a unified sparse entity ingestion model that accepts a profile identifier, phone identifier, or both and preserves their association.
  • Migrate existing SQLite databases in place while retaining legacy typed import compatibility.
  • Add schema discovery, mapped imports, per-row validation, provenance, raw source retention, fuzzy profile matching, exact normalized phone matching, dataset listing, record browsing, and deletion.
  • Route the service through nginx, persist it in a Compose volume, and include it in desktop readiness checks.
  • Fix the phone service Compose entry point and expose explicit routed health/readiness endpoints.
  • Replace opaque profile-inspection 500 errors with actionable X provider responses.
  • Resolve Compose service names through Docker DNS at request time so recreated containers cannot inherit one another's stale nginx routes.
  • Preserve public username-validation evidence and display richer live and imported result summaries without requiring paid X profile credits.
  • Include fictional profile CSV and phone JSONL fixtures that exercise canonical fields, rich metadata, and raw-field retention.
  • Present profile and phone investigations in one Search workspace with Auto/Profile/Phone modes and provider-aware source filters.
  • Publish the Who Is It? UI as a platform submodule and point both changed submodules at their reviewable commits.
  • Document and expose provider integration status and protected credential replacement through the Electron application.
  • Rewrite the platform, environment, changelog, and dataset documentation around the current architecture.

Why

Live profile and phone providers cover only part of an investigation and can be unavailable or constrained by provider access. A normalized local source makes authorized private, licensed, and curated datasets searchable without obscuring their provenance, while the source-aware UI keeps successful results usable when another provider fails.

Dependencies

The platform branch references the commits from those PR branches so every submodule revision is already available from GitHub.

Validation

  • .venv/bin/pytest -q — 14 backend, entity-association, and routing tests passed.
  • Created data with the pre-entity service, migrated that SQLite database, verified both legacy record types remained searchable, and imported a linked entity.
  • npm test in whoisit — 12 parser, packaged-path, integration-configuration, search-mode, and source-filter tests passed.
  • npm run package in whoisit — Electron package completed successfully.
  • Rebuilt and started all Compose services.
  • Verified all routed health checks through nginx.
  • Verified an import, local search, and deletion cycle through port 80.

@KingAkeem
KingAkeem force-pushed the agent/add-dataset-search branch from 87d8f94 to fb16867 Compare July 17, 2026 16:16
@KingAkeem
KingAkeem marked this pull request as ready for review July 17, 2026 20:06
@KingAkeem
KingAkeem merged commit 303ffad into main Jul 17, 2026
@KingAkeem
KingAkeem deleted the agent/add-dataset-search branch July 17, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant