docs: prioritize product improvements from implementation evidence

CHANGELOG.md

@@ -31,6 +31,7 @@ All notable changes follow Keep a Changelog and Semantic Versioning.
 - Scan coreference words once per block using dictionary lookup instead of rebuilding
   a regular expression containing every learned token. Preserve case and word boundaries.
 ### Fixed
+
 - Include the hosted-service extra in release verification and isolated installation
   audits, while keeping service frameworks out of the dependency-free base import.
 

HANDOVER.md

@@ -529,3 +529,21 @@ Reusable lesson assessment: match learned literal words through word scanning
 and dictionary lookup when semantics allow it; preserve a boundary-equivalence
 test before replacing a dynamically constructed regex. No new global policy is
 needed for this local algorithm substitution.
+
+## Product direction review (2026-10-02)
+
+The proposed development priorities are recorded in
+`docs/decisions/1.35.0_product_review.md`, linked from VISION, ROADMAP and the docs
+navigation. The review maps existing implementation paths to product value,
+acceptance conditions and unresolved evidence. It separates five implemented PRs
+from future proposals; it is not a claim that those proposals have shipped.
+The roadmap's optional-extra list now matches `pyproject.toml`.
+
+Observed checks: strict docs build and `git diff --check` passed. Implementation
+PRs carry their own regression tests and CI evidence. The attempted additional
+privacy validation sandbox failed during network namespace setup, so those
+source-grounded proposals have no new dynamic validation claim.
+
+Reusable lesson assessment: the existing evidence-before-claims policy already
+covers the findings. This review applies it to concrete product priorities rather
+than creating another general standard or duplicating the shared runtime.

ROADMAP.md

@@ -49,6 +49,21 @@ Alpha releases optimize for the cleanest safe architecture, not backward compati
 remove, rename, or replace public APIs without aliases or shims. Material changes are documented,
 but compatibility guarantees start only with `0.1.0`.
 
+## Active development review
+
+The [1.35 product review](docs/decisions/1.35.0_product_review.md) compares the
+vision, existing implementation and evidence gaps. It proposes the next order:
+
+1. Review the isolated correctness, performance, onboarding and measurement fixes.
+2. Resolve ambiguous policy, archive, remote-response and diagnostic contracts.
+3. Reconcile quality-gate methodology, compatibility and platform evidence.
+4. Add bounded ML configuration, caller-framed streaming and lazy record processing.
+5. Resume measured detector improvements under the existing experimental protocol.
+
+These are proposed priorities, not published capabilities or verified security
+findings. The review records the source paths, acceptance conditions and validation
+limits; individual PRs and HANDOVER carry implementation evidence.
+
 ## Release gate for every milestone
 
 - Ruff formatting and linting pass.
@@ -634,9 +649,10 @@ independent-corpus evaluation, and paired uncertainty show a real benefit.
 
 ## Optional dependency policy
 
-Extras appear only with the release that owns them: `ml`, `pdf`, `office`, `ocr`, `documents`,
-`docling`, and `remote`. An `all` extra may exist for CI and integration testing, but user
-documentation recommends the narrowest installation that satisfies the workload.
+The currently declared extras are `ml`, `pdf`, `office`, `ocr`, `remote`, `html`, and `service`.
+`pyproject.toml` is authoritative. `documents`, `docling`, and `all` are not currently
+published extras. Recommend the narrowest installation that satisfies the workload;
+new extras require an owning capability and independent installed-wheel checks.
 
 ## Deliberately uncommitted work
 

VISION.md

@@ -116,6 +116,14 @@ The project succeeds when an application can place a narrow local boundary aroun
 content, choose the appropriate transformation, inspect what happened without leaking what was
 found, and add only the format or detection dependencies it actually needs.
 
+## Current product review
+
+The [1.35 development review](docs/decisions/1.35.0_product_review.md) retains this
+vision and prioritizes predictable contracts, reproducible evidence and bounded
+resource use across the capabilities already shipped. Proposed additions should
+reduce integration work without expanding the trusted core or implying stronger
+privacy guarantees than the evidence supports.
+
 ## Future Exploration & Open Proposals
 
 The following architectural and visionary proposals are under consideration for the long-term future:

docs/decisions/1.35.0_product_review.md

@@ -0,0 +1,198 @@
+# Product review for the 1.35 development cycle
+
+Status: proposed priorities, with five separately implemented improvements.
+Original reviewed base: `bc18792`. Review date: 2026-10-02.
+Refreshed against `17e8fcc` on 2026-10-09; development version is `1.37.0`.
+The historical filename identifies the original review cycle.
+
+## Decision
+
+Keep the local-first, dependency-free privacy-boundary vision. The strongest next
+investment is to make existing behavior predictable, measurable and inexpensive
+to integrate. Adding more providers, formats or semantic heuristics would expand
+the surface before the current contracts are consistently demonstrated.
+
+This is a product and implementation review, not a security certification or a
+new detection-quality benchmark. Existing release labels are historical claims;
+this review does not independently certify every published release or model-card
+number. No held-out corpus was used to tune detectors.
+
+## Implemented proposals
+
+Each change has its own branch, tests, changelog entry and handover evidence:
+
+1. [Bloom filter correctness, PR 91](https://github.com/ma2za/pseudonymize/pull/91):
+   invalidate cached misses after insertion and validate filter sizing. This
+   prevents stale membership results and zero-bit filters. Four regression tests
+   failed before the fix; 11 focused tests passed afterwards.
+2. [Coreference lookup cost, PR 92](https://github.com/ma2za/pseudonymize/pull/92):
+   replace a growing regex alternation with a Unicode word scan and dictionary
+   lookup. Generated equivalence tests preserve boundary semantics. A comparative
+   benchmark covers 100, 1,000 and 10,000 learned tokens; small scopes need not be
+   faster. This changes computational cost, not the detection-quality target.
+3. [Executable onboarding, PR 93](https://github.com/ma2za/pseudonymize/pull/93):
+   repair README assertions that expected obsolete long aliases, exercise its
+   Python examples, distinguish overlap from exact-boundary scoring, and remove
+   an unconditional streaming-safety promise.
+4. [Local benchmark integrity, PR 94](https://github.com/ma2za/pseudonymize/pull/94):
+   correct one-to-one match accounting, cover empty and disjoint inputs, keep
+   explanation logs free of source values, and defer optional ML imports.
+5. [Required release evidence, PR 95](https://github.com/ma2za/pseudonymize/pull/95): reject absent external evaluation, invalid external
+   F1, incomplete critical-entity counts and incomplete approval reports. The
+   comparator handles zero true positives without a division error. This does
+   not yet reconcile the roadmap's stricter exact-boundary improvement protocol
+   with the existing tolerance gate.
+
+These are reviewable increments, not merged features. Follow the PR checks and
+branch handovers for current acceptance status. Do not assume they are present
+in a release because this review describes them.
+
+## Implementation findings by product layer
+
+### Core, identity and structured payloads
+
+The core already provides explicit scopes, HMAC namespaces, opt-in mappings,
+immutable document blocks and typed locations. Preserve those contracts.
+
+`Policy.allows_path` skips every path containing names such as `description`,
+`title`, `id` or `error` when schema preservation is enabled, including ordinary
+application data. This is broader than recognizing an actual schema or protocol
+envelope. Proposed next change: distinguish structural schema locations from
+content fields, with an explicit migration path for existing callers. Acceptance
+must cover nested payloads, schema descriptions containing sensitive content,
+protocol identifiers and explicit include/exclude policies.
+
+`MemoryBoundary.write` accepts `namespace` without applying it, and `read` accepts
+`restore` without using it. Resolve these misleading controls before promoting
+the boundary as a tenant-isolation or reversible-memory integration. Prefer a
+small explicit contract over adding a token vault, which remains a non-goal.
+
+`process_data` recursively processes mappings, lists and tuples. Plan explicit
+cycle/depth behavior and tests before advertising arbitrary application objects.
+For large independent records, a lazy record iterator would avoid materializing
+all results; document that shared alias state can still grow with unique values.
+
+### Streaming and LLM integration
+
+`TextStreamProcessor.push` chooses heuristic punctuation boundaries and can cut
+at a fixed position in long unbroken text. Context retained after emission cannot
+retract already emitted bytes. Proposed feature: caller-framed processing, where
+only complete bounded messages are emitted, alongside clearly labeled heuristic
+streaming. Test every chunk split, long secrets, Unicode boundaries, cancellation
+and failure before publication. Do not promise arbitrary streaming equivalence
+with whole-input detection.
+
+Keep session restoration and token formats stable. Use executable round trips
+as the integration contract instead of adding framework-specific wrappers now.
+
+### Detection, gazetteers and model inference
+
+Structured rules, checksums, contextual heuristics, gazetteers and ONNX already
+cover substantial scope. Further recall work should follow the grouped error
+atlas and fixed experimental protocol, not more unmeasured heuristics.
+
+Gazetteer loaders use `data/gazetteer` relative to the working directory and
+silently return no dictionary when loading fails. Propose explicit resource
+configuration and distinguish an intentionally absent dictionary from invalid
+configuration. Test installed-wheel behavior from another working directory.
+Do not bundle external dictionaries until their provenance and license are
+established.
+
+`DAWG` is currently a trie implementation. Measure its memory before claiming
+compressed graph behavior or replacing it with another structure.
+
+ONNX inference batches up to 32 windows and retains up to 1,024 cached inference
+inputs. Propose configurable window batch size and cache capacity, explicit cache
+clearing, and measurements of latency, peak memory and retained input volume.
+The local full-suite run ended with exit 137 during ONNX tests; this is a symptom,
+not proof that batching caused it. Use bounded real-model measurements before
+choosing defaults. Remove dead allocations and placeholder loops only with
+unchanged output evidence.
+
+### Files, documents and archives
+
+Keep native document structure and atomic safe-copy publication. Existing PDF
+redaction and Office metadata paths deserve format-specific acceptance evidence,
+including extraction of the output and checks of hidden content. Rendering a
+visually clean page alone is insufficient.
+
+`ArchiveAdapter` skips unsupported/corrupt member extraction and can copy those
+members unchanged during rendering. Propose an explicit policy for unprocessed
+members, a value-free manifest, and limits on nesting, member count and expanded
+bytes. A strict mode should refuse publication if required members were not
+processed. Test actual nested archives and ensure rejection leaves no output.
+
+HTML extraction handles string-valued attributes, while some parsed attributes
+are lists. Document the supported surface and add realistic round-trip coverage
+before claiming all attributes are sanitized. For plain JSONL and CSV, the current
+adapters materialize whole files. A record-oriented interface is more useful than
+adding another file extension before memory behavior is measured.
+
+### Remote processing and observability
+
+The engine checks network policy and backend consent before invocation. Preserve
+that boundary. Remote response parsing currently skips malformed detections and
+can interpret missing response fields as no detections. Propose explicit response
+contract validation so provider failures cannot masquerade as a clean document.
+Pair configurable payload/response limits with finite timeouts and bounded retries.
+
+Source paths in `invoke_backend` and remote errors retain exception causes.
+`DlpLoggingFilter` processes messages and arguments but not exception traceback
+text; telemetry integration also needs evidence with the actual optional SDK and
+exporter ordering. Proposed acceptance: capture complete formatted diagnostics
+and traces using synthetic values, including chained exceptions, rather than
+checking only `str(error)`. Do not claim these paths were dynamically validated
+by this review.
+
+### Quality evidence, packaging and developer experience
+
+The zero-runtime-dependency wheel and narrow extras remain valuable. Keep frozen
+lockfiles, installed-wheel checks and optional-import isolation as release gates.
+
+The quality gate uses overlap deltas and tolerates a slightly negative interval,
+while the roadmap calls for a positive exact-boundary improvement interval.
+Reconcile this explicitly in a separate methodology change: either implement the
+roadmap criterion or document a deliberately different gate with approval and
+versioned evidence. Never relabel overlap scores as exact-boundary scores.
+
+The release matrix tests Linux and Windows, while the roadmap also names macOS.
+Add macOS evidence or revise that promise explicitly. Compatibility documentation
+still describes the `0.1` line while package metadata is `1.37.0`; reconcile the
+public compatibility scope without silently changing existing guarantees.
+
+The README smoke test is a starting point. Extend executable examples to CLI
+recipes and the built wheel, and make the documented installation extras match
+package metadata. Avoid duplicated implementation inventories in README,
+ROADMAP and HANDOVER; keep product direction in VISION, planned acceptance in
+ROADMAP and dated verification evidence in HANDOVER.
+
+## Proposed order and acceptance
+
+1. Review required quality evidence (PR 95), local benchmark integrity (PR 94),
+   and Bloom mutation correctness (PR 91) first. Then review executable examples
+   (PR 93) and coreference performance (PR 92). Require fresh CI and package
+   evidence against current main; merge independently after review.
+2. Correct contract ambiguity in structured policies, remote response handling,
+   unprocessed archive members and diagnostic boundaries. These have higher
+   user value than new surface area. Validate the source-grounded candidates
+   below before presenting them as confirmed security findings.
+3. Reconcile release-quality methodology, public compatibility and supported
+   platforms. Publish reproducible evidence and retain unsuccessful results.
+4. Add caller-framed streaming, configurable ML resource budgets and lazy record
+   processing, with explicit latency/memory and semantic acceptance criteria.
+5. Resume multilingual recall and model comparisons only when grouped development
+   evidence identifies the limiting error class. Preserve the independent lockbox.
+
+Measure progress through successful executable examples, deterministic behavior,
+complete value-free failure reports, peak memory by workload, and quality metrics
+with pinned provenance. A larger feature list is not an acceptance criterion.
+
+## Evidence limits
+
+The implementation PRs include real regression tests and observed local checks;
+CI supplied full-suite and installed-wheel evidence for the implementation PRs. The
+additional privacy-boundary items above are source-grounded proposals, not
+confirmed exploit findings. An attempted isolated validation harness could not
+start because network namespace setup was denied. No deployment or external
+service was probed. PDF/OCR behavior on new artifacts, real OpenTelemetry exporter
+behavior, and new held-out detection-quality comparisons were not rerun here.

mkdocs.yml

@@ -31,6 +31,7 @@ nav:
   - Model Card: model_card.md
   - Contributing: contributing.md
   - Decisions:
+      - Product Review (1.35 development): decisions/1.35.0_product_review.md
       - Dependency-free core: adr/0001-dependency-free-core.md
       - Token format: adr/0002-token-format.md
       - No reversible storage: adr/0003-no-reversible-storage.md