Add persistent dataset search

.env.example

@@ -1,8 +1,14 @@
 # Copy this file to .env and fill in the required values before running docker compose.
 
-# profile_search
+# profile_search: X API v2 bearer token used by the Tweepy client.
+# Create it in an X Developer Portal project/app. Keep it secret.
 TWEEPY_BEARER_TOKEN=your_tweepy_bearer_token
 
-# phone_search
+# phone_search: Twilio API credentials from the Twilio Console.
+# The Account SID normally starts with AC. Keep the auth token secret.
 TWILIO_ACCOUNT_SID=your_twilio_account_sid
 TWILIO_AUTH_TOKEN=your_twilio_auth_token
+
+# dataset_service: Compose already persists this path in the dataset_data volume.
+# Override only when you intentionally want a different in-container location.
+# DATASET_DB_PATH=/data/datasets.db

.gitignore

@@ -0,0 +1,4 @@
+.env
+__pycache__/
+*.py[cod]
+finder.log

.gitmodules

@@ -7,3 +7,6 @@
 [submodule "phone_search"]
 	path = phone_search
 	url = git@github.com:osint-services/phone_search
+[submodule "whoisit"]
+	path = whoisit
+	url = git@github.com:osint-services/whoisit.git

CHANGELOG-2026-07-17.md

@@ -0,0 +1,34 @@
+# Changelog for 2026-07-17
+
+## Added
+
+- Sparse entities that can contain a profile identifier, a phone identifier, or both, with associations visible from either search.
+- Persistent SQLite-backed profile and phone datasets with mapped imports and local search.
+- CSV, JSON, JSONL, and NDJSON parsing with automatic field suggestions and previews.
+- Per-row validation, rejected-row reporting, provenance, observation time, confidence, and raw source retention.
+- Search history and combined live/local results in the Electron application.
+- Integration status and credential management for X/Tweepy and Twilio Lookup.
+- Rich username result cards that remain informative when paid X profile inspection is unavailable.
+- Fictional sample profile and phone files for testing the dataset workflow without real personal data.
+- Backend and renderer parser regression tests.
+
+## Changed
+
+- Replaced the desktop record-type selector with one entity ingestion and field-mapping pipeline.
+- Added an in-place SQLite schema migration while retaining legacy profile and phone import compatibility.
+- Unified username and phone lookup into a Search workspace with Auto, Profile, and Phone modes plus source filters that avoid unnecessary provider calls.
+- Reorganized the desktop application into Search, Datasets, Integrations, and History workspaces.
+- Expanded profile and caller details and made source provenance visible throughout the UI.
+- Kept API failures scoped to their own source and search workflow.
+- Added dataset routing, persistence, and readiness checks to the deployed architecture.
+- Restricted saved provider credentials to the operating-system user and prevented existing secrets from being returned to the UI.
+- Fixed platform-root discovery in packaged Electron layouts so integration settings use the correct `.env` file.
+- Mapped X API failures to explicit profile-inspection responses, including `402` for depleted credits and `429` for rate limits.
+- Made nginx resolve Compose service names at request time, preventing cross-service 404s after containers receive new IP addresses.
+
+## Documentation
+
+- Rewrote the root architecture and setup guide.
+- Expanded environment-variable guidance.
+- Rewrote the desktop UI guide and added a dataset service guide.
+- Updated the OSINT Services organization profile README.

CHANGELOG.md

@@ -5,12 +5,28 @@ All notable changes to this project will be documented in this file.
 ## [Unreleased]
 
 ### Added
-- Added lightweight `/healthz` endpoints to the FastAPI services for smoke checks and container health probes.
-- Added an initial regression test suite covering the new health endpoints.
+- Added a sparse entity model that associates optional profile and phone identifiers while preserving their type-specific metadata.
+- Added a persistent SQLite dataset service for mapped profile and phone imports.
+- Added CSV, JSON, JSONL, and NDJSON parsing, previews, automatic field suggestions, per-row validation, provenance, and raw-record retention.
+- Added combined live and imported-source searching, local search history, raw JSON inspection, and dataset management to the Electron UI.
+- Added an Integrations workspace for X/Tweepy, Twilio Lookup, and local dataset connection status with protected credential replacement.
+- Expanded username result cards with public validation evidence, direct profile links, dataset provenance, observation time, and confidence.
+- Added fictional profile CSV and phone JSONL datasets for testing imports, mapping, local search, and rich result rendering.
+- Expanded regression coverage for dataset import, fuzzy profile search, normalized phone search, deletion, health endpoints, and client-side parsers.
 
 ### Changed
-- Parallelized profile checker requests with bounded concurrency to reduce scan latency across site checks.
-- Removed the obsolete Compose `version` field from the deployment configuration to avoid startup warnings.
+- Replaced separate profile and phone dataset ingestion with one entity mapping pipeline; legacy API imports remain compatible and existing SQLite databases migrate in place.
+- Unified username and phone lookup into one Search workspace with conservative auto-detection, explicit Profile and Phone modes, and source filters that control which APIs and datasets are queried.
+- Reworked the Electron interface into cohesive Search, Datasets, Integrations, and History workspaces with type-specific master/detail results.
+- Isolated live-provider failures so other sources can still return results and errors remain in the correct search workspace.
+- Expanded profile and phone detail views with provider metadata, provenance, freshness, confidence, metrics, and raw source records.
+- Added the dataset service to Docker Compose, nginx routing, persistent storage, and desktop readiness probes.
+- Increased the desktop window size and stopped opening DevTools in normal runs.
+- Kept stored provider secrets out of the renderer and applied credential changes through atomic, owner-only `.env` updates.
+- Fixed packaged Electron builds so they locate the parent platform configuration and report the real integration state.
+- Replaced opaque profile-inspection 500 errors with actionable X provider responses for depleted credits, rate limits, invalid access, and missing profiles.
+- Switched nginx service routing to Docker's runtime DNS resolver so container recreation cannot silently send requests to reassigned service IPs.
 
 ### Documentation
-- Updated the platform README to describe the new health checks and operational improvements.
+- Updated the platform, environment, desktop UI, dataset service, and organization profile documentation for the current architecture.
+- Added [2026-07-17 release notes](CHANGELOG-2026-07-17.md).

Dockerfile

@@ -8,6 +8,7 @@ WORKDIR /app
 COPY ./profile_checker /app/profile_checker
 COPY ./profile_search /app/profile_search
 COPY ./phone_search /app/phone_search
+COPY ./dataset_service /app/dataset_service
 
 # Install the selected service dependencies
 ARG APP_DIR=profile_checker

README.md

@@ -1,54 +1,97 @@
-# platform
+# OSINT Services Platform
 
-Stands up multiple microservices so they can be contacted using a single URI with `nginx` and `docker compose`.
-Each service is stored as a git submodule; see each service directory for service-specific details.
+The platform combines live public-data APIs with user-supplied datasets behind one nginx endpoint and one Electron investigation workspace.
+
+## Architecture
+
+```text
+whoisit desktop app
+        |
+        v
+nginx :80
+  |-- /scan/*          -> profile_checker  (public username availability)
+  |-- /focus*          -> profile_search   (X profile metadata via Tweepy)
+  |-- /phone_search*   -> phone_search     (caller-name data via Twilio)
+  `-- /datasets*       -> dataset_service  (SQLite entity imports and local search)
+```
+
+Docker Compose runs the four FastAPI services, nginx, and a persistent `dataset_data` volume. The Electron UI provides one search workspace with Auto, Profile, and Phone modes plus All, Live APIs, and Imported datasets source filters. It queries enabled sources independently and combines successful results, so one unavailable provider does not prevent other sources from being searched.
 
 ## Requirements
 
-- [`docker`](https://www.docker.com/)
-- Docker Compose support (`docker compose`)
+- Docker with `docker compose`
+- Node.js 18 or newer to build or run the Electron UI
+- An X API bearer token for profile inspection
+- Twilio Account SID and Auth Token for live phone lookup
 
 ## Setup
 
-1. Initialize submodules if needed:
-
 ```bash
 git submodule update --init --recursive
+cp .env.example .env
 ```
 
-2. Build and start all services:
+Add your provider credentials to `.env`, then start the API stack:
 
 ```bash
 ./scripts/start.sh
 ```
 
-3. Stop the stack:
+Run the desktop UI from a second terminal:
 
 ```bash
-./scripts/stop.sh
+cd whoisit
+npm install
+npm start
 ```
 
-4. Rebuild the services:
+The UI checks service readiness on launch and can attempt to start the Compose stack when Docker is available. Its Integrations workspace shows the configured and reachable state for X/Tweepy, Twilio Lookup, and local datasets. Users can replace provider credentials there without exposing existing values to the renderer. Packaged builds are created with `npm run package`.
 
-```bash
-./scripts/build.sh
-```
+## Environment values
 
-5. Stream logs:
+| Variable | Required | Used by | Meaning |
+| --- | --- | --- | --- |
+| `TWEEPY_BEARER_TOKEN` | For live profile inspection | `profile_search` | X API v2 application bearer token. Despite the variable name, the credential is issued by X; Tweepy is the Python client. |
+| `TWILIO_ACCOUNT_SID` | For live phone lookup | `phone_search` | Twilio account identifier, normally beginning with `AC`. |
+| `TWILIO_AUTH_TOKEN` | For live phone lookup | `phone_search` | Secret used with the Account SID to authenticate Twilio API calls. |
+| `DATASET_DB_PATH` | No | `dataset_service` | SQLite file location. Compose sets this to `/data/datasets.db` inside its persistent volume. |
+
+Do not commit `.env`. Imported datasets may contain sensitive or licensed information; only import data you are authorized to retain and use.
+
+## API routes
+
+| Route | Purpose |
+| --- | --- |
+| `GET /scan/{username}` | Check supported sites for a public username |
+| `GET /focus?url=https://x.com/{username}` | Retrieve expanded X profile metadata |
+| `GET /phone_search?phone_number=+18135551212` | Retrieve live caller-name metadata |
+| `GET /datasets` | List imported datasets |
+| `GET /datasets/schema/entity` | Get the unified sparse entity schema and sample |
+| `POST /datasets/import` | Import mapped entities containing profile identifiers, phone identifiers, or both |
+| `GET /datasets/search/profiles?query=...&fuzzy=true` | Search imported profiles |
+| `GET /datasets/search/phones?phone_number=...` | Search imported phone records |
+| `DELETE /datasets/{dataset_id}` | Delete a dataset and its records |
+
+The import API accepts sparse entity rows after the UI parses CSV, JSON, JSONL, or NDJSON. Each row needs at least one username, profile URL, or E.164 phone number; profile and phone fields can coexist and remain associated in search results. A single import is limited to 10,000 records and the UI limits files to 10 MB. Canonical records preserve their source row, dataset provenance, observation time, and optional confidence score. Legacy typed imports remain API-compatible and existing databases migrate in place.
+
+## Operations and tests
 
 ```bash
-./scripts/logs.sh
+./scripts/build.sh      # rebuild images
+./scripts/logs.sh       # stream Compose logs
+./scripts/stop.sh       # stop the stack
+.venv/bin/pytest -q     # backend tests
+cd whoisit && npm test  # importer parser tests
 ```
 
-## Notes
-
-- The proxy listens on port `80` and routes `/scan/...` to `profile_checker`, `/focus` to `profile_search`, and `/phone_search` to `phone_search`.
-- `profile_search` requires `TWEEPY_BEARER_TOKEN` to be available in the environment when running the Compose stack.
-- `phone_search` requires `TWILIO_ACCOUNT_SID` and `TWILIO_AUTH_TOKEN` to be set.
-- Copy `.env.example` to `.env` and fill in the required values before starting Compose.
-- Each service now exposes a lightweight `/healthz` endpoint for smoke checks and container health probes.
+Liveness endpoints are available at `/scan/healthz`, `/phone_search/healthz`, and `/datasets/healthz`. Profile inspection also exposes `/focus/readyz`, which verifies that its X credential is configured.
 
-## Changelog
+## Documentation
 
-- See [CHANGELOG.md](CHANGELOG.md) for the current release notes.
-- See [CHANGELOG-2026-07-16.md](CHANGELOG-2026-07-16.md) for the latest dated entry.
+- [Desktop UI guide](whoisit/README.md)
+- [Dataset service guide](dataset_service/README.md)
+- [Sample combined entity dataset](examples/datasets/sample_entities.csv)
+- [Sample profile dataset](examples/datasets/sample_profiles.csv)
+- [Sample phone dataset](examples/datasets/sample_phone_records.jsonl)
+- [Changelog](CHANGELOG.md)
+- [2026-07-17 release notes](CHANGELOG-2026-07-17.md)

dataset_service/README.md

@@ -0,0 +1,46 @@
+# Dataset service
+
+The dataset service provides normalized, persistent identity sources for the platform's profile and phone searches. It uses SQLite and has no external provider dependency.
+
+## Supported input
+
+The Electron UI parses `.csv`, `.json`, `.jsonl`, and `.ndjson` files, then maps every import through the entity schema returned by:
+
+```http
+GET /datasets/schema/entity
+```
+
+Each sparse entity row requires at least one searchable identifier: a username, profile URL, or E.164 phone number such as `+12025550101`. All other fields are optional. When both profile and phone fields exist, the service stores typed identifiers under one entity so a match for either identifier exposes the association. Invalid booleans, negative metrics, confidence outside `0..1`, and malformed identifiers are rejected per row without discarding valid rows.
+
+The legacy `profile` and `phone` schemas and import values remain accepted for API compatibility, but the desktop application uses only the entity pipeline. Existing SQLite databases are migrated in place when the service starts.
+
+Each accepted entity includes:
+
+- its dataset and source;
+- an observation timestamp;
+- optional confidence;
+- normalized searchable identifiers; and
+- the complete original source row for audit.
+
+## Storage and limits
+
+Compose stores `/data/datasets.db` in the named `dataset_data` volume. Deleting a dataset cascades to its entities and typed identifiers. Each import accepts 1–10,000 JSON object rows; the desktop UI additionally enforces a 10 MB file limit.
+
+The API is intentionally local and currently has no authentication layer. Do not expose the nginx proxy to an untrusted network without adding access control.
+
+## Sample data
+
+The platform repository includes a combined fictional entity dataset plus separate fixtures for exercising compatibility and source-specific fields:
+
+- [`examples/datasets/sample_entities.csv`](../examples/datasets/sample_entities.csv)
+- [`examples/datasets/sample_profiles.csv`](../examples/datasets/sample_profiles.csv)
+- [`examples/datasets/sample_phone_records.jsonl`](../examples/datasets/sample_phone_records.jsonl)
+
+Import any of them from the same desktop Datasets workflow; canonical headers are auto-mapped. The combined file demonstrates a username and phone on one entity, a profile-only entity, and a phone-only entity. Search for `demo_ada_1843` or `+12025550101` to see the association in both directions. The phone values use the reserved North American `555-01xx` fictional range and do not identify real subscribers.
+
+## Development
+
+```bash
+uvicorn dataset_service.server:app --reload --port 8000
+.venv/bin/pytest -q tests/test_dataset_service.py
+```

dataset_service/__init__.py

@@ -0,0 +1 @@
+"""Local dataset import and search service."""

dataset_service/models.py

@@ -0,0 +1,56 @@
+from enum import Enum
+from typing import Any
+
+from pydantic import BaseModel, Field
+
+
+class RecordType(str, Enum):
+    entity = "entity"
+    profile = "profile"
+    phone = "phone"
+
+
+class DatasetImportRequest(BaseModel):
+    name: str = Field(min_length=1, max_length=120)
+    record_type: RecordType = RecordType.entity
+    filename: str | None = Field(default=None, max_length=255)
+    mapping: dict[str, str] = Field(default_factory=dict)
+    rows: list[dict[str, Any]] = Field(min_items=1, max_items=10_000)
+
+
+class RejectedRow(BaseModel):
+    row_number: int
+    reason: str
+
+
+class DatasetImportResult(BaseModel):
+    dataset_id: str
+    imported: int
+    rejected: int
+    rejected_rows: list[RejectedRow]
+    profile_identifiers: int = 0
+    phone_identifiers: int = 0
+
+
+class DatasetSummary(BaseModel):
+    id: str
+    name: str
+    record_type: RecordType
+    filename: str | None
+    imported_at: str
+    row_count: int
+    rejected_count: int
+
+
+class FieldDefinition(BaseModel):
+    key: str
+    label: str
+    required: bool = False
+    description: str
+
+
+class DatasetSchema(BaseModel):
+    record_type: RecordType
+    fields: list[FieldDefinition]
+    sample: dict[str, Any]
+    identifier_fields: list[str] = Field(default_factory=list)