feat(archive): index v46 wire-evidence batch, free-threaded-only runtime, parse-failure recovery - #3390
Merged
Merged
Conversation
Sinity
changed the title
📝 WalkthroughWalkthroughPython support is narrowed to 3.14, parser and schema evidence is expanded, delegation mapping is rebuilt around content identity, replay concurrency is simplified, and storage retention and repair flows are added. ChangesPlatform and verification
Schema generation and promotion
Structured parsing
Delegation and archive behavior
Storage retention and repair
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e86093ff45
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for statement in ( | ||
| "DELETE FROM raw_authority_census_plans WHERE census_id = ?", | ||
| "DELETE FROM raw_authority_census_post_plans WHERE census_id = ?", | ||
| "DELETE FROM raw_authority_blockers WHERE census_id = ?", |
There was a problem hiding this comment.
Once a blocker’s census falls outside the eight-census plan window, this unconditionally deletes the blocker even when resolved_at_ms IS NULL. That makes unresolved_raw_authority_blockers() and readiness report the obligation as cleared, invalidates any blocker ID already presented to an operator, and discards the evidence needed by resolve_raw_authority_blocker() without an explicit resolution. Only resolved blockers should be eligible for retention pruning; unresolved durable obligations and their supporting census data must remain.
AGENTS.md reference: AGENTS.md:L165-L168
Useful? React with 👍 / 👎.
| generation = IndexGeneration(**json.loads(metadata_path.read_text(encoding="utf-8"))) | ||
| except (OSError, ValueError, TypeError): | ||
| continue # unreadable metadata: retain | ||
| if generation.state == "promoting": |
There was a problem hiding this comment.
When a paused or failed resumable rebuild has an inactive generation, this filter admits it into the same pool as previously active generations. A later promotion can therefore delete that resumable candidate, and because retention is ordered only by creation time, a newer inactive candidate can consume the single retained slot and cause the actual previous active rollback target to be deleted. Filter candidates by promotion history/transaction ownership rather than treating every non-active, non-promoting generation as superseded.
Useful? React with 👍 / 👎.
| role=turn.role, | ||
| text=turn.text, | ||
| timestamp=turn.timestamp, | ||
| blocks=[_browser_capture_parsed_block(block) for block in turn.blocks], |
There was a problem hiding this comment.
For captures containing both rendered turn.text and structured blocks, assigning the blocks here causes the text to disappear when the session is written: _message_blocks() returns message.blocks whenever nonempty and only synthesizes a text block from message.text when there are no blocks. The newly added test payload itself demonstrates this with text="calling search" plus a tool-use block, but it stops at the parsed model and never observes that persistence drops the text. Include the rendered text as a text block when it is not already represented; the Claude fallback has the same issue when it replaces content with generated block segments.
Useful? React with 👍 / 👎.
| "Topic :: Software Development :: Libraries", | ||
| ] | ||
| requires-python = ">=3.11" | ||
| requires-python = ">=3.14" |
There was a problem hiding this comment.
With this metadata floor, pip refuses to install the built wheel on Python 3.11–3.13, but .github/workflows/release.yml still runs the installed-smoke and installed-smoke-mcp jobs on python-version: ["3.11", "3.12", "3.13", "3.14"] and installs that wheel in each leg. Consequently every release workflow now has six guaranteed failing installation legs across Ubuntu and macOS; update those release matrices alongside the supported-version change.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 19
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Line 72: Update the comment describing the Python test matrix to state that
the suite runs on one Python version, matching the current python-version matrix
entry for 3.14.
In `@CLAUDE.md`:
- Around line 169-177: Update the later “Schema-touching changes” workflow
guidance in CLAUDE.md to align with the derived-tier policy: permit declared
non-semantic deltas to use index_fast_forward_plan() in place, while requiring
polylogue ops reset --index and polylogued run only for SEMANTIC_REPARSE. Remove
or revise the blanket prohibition on upgrade helpers and ensure undeclared bumps
remain policy violations.
In `@docs/plans/demo-corpus-construct-audit.md`:
- Around line 11-12: The demo-corpus audit must not present complete coverage
while verification fails: update the seed/query and embedding fixtures so the
required Claude Code session is included and synthetic_message_embedding_rows
and embedding_status_rows each contain at least one row, then regenerate the
audit; alternatively, explicitly retain and track these gaps and revise the
datasheet’s coverage claims.
In `@polylogue/browser_capture/models.py`:
- Around line 74-79: Update coerce_object_field to distinguish the validated
field via Pydantic ValidationInfo: keep null as None for optional tool_input,
but coerce explicit null metadata to an empty dictionary so the non-optional
metadata field validates successfully. Add the required ValidationInfo import
and preserve existing JSON coercion for non-null values.
In `@polylogue/schemas/promotion_audit.py`:
- Around line 247-327: Extend _catalog_coherence_findings to compare matching
element records in catalog and manifest, not only _element_kinds and
package-level fields. For each element, validate sample_count, first_seen,
last_seen, and bundle_scope_count, reporting catalog_incoherent findings with an
element-specific JSON path and values when they differ. Preserve the existing
package-level and kind checks.
In `@polylogue/schemas/providers/chatgpt/catalog.json`:
- Around line 2-39: Regenerate the ChatGPT schema artifacts using the current
serializer so both polylogue/schemas/providers/chatgpt/catalog.json (lines 2-39)
and polylogue/schemas/providers/chatgpt/versions/v1/package.json (lines 4-29)
use the current schema shape, including package- and package-element-level
bundle_scope_identities and workload_profile_file fields.
In `@polylogue/sources/parsers/browser_capture.py`:
- Around line 206-209: Update the block conversion logic handling
BlockType.THINKING to also accept BlockType.REASONING, preserving the existing
thinking payload shape and empty-text behavior so reasoning blocks are emitted
instead of falling through to None.
- Around line 226-227: Update the return mapping in the IMAGE/DOCUMENT branch so
block.metadata is expanded first, then explicitly set type and media_type from
the block afterward; preserve the existing metadata fields while preventing them
from overriding the segment discriminator used by normalize_chat_messages.
In `@polylogue/sources/parsers/claude/code_parser.py`:
- Around line 989-1004: Update the ParsedSessionEvent construction in the
delegation_progress loop so source_message_provider_id is not populated with
parent_tool_use_id, since that value is a tool_use identifier rather than a
provider message reference. Preserve parent_tool_use_id in the payload and keep
the existing delegation progress statistics and event ordering unchanged.
- Around line 193-201: Update the pr-link branch in the record parser to return
None when both prNumber and prUrl are absent, matching the documented behavior
and sibling branches. Preserve the existing event construction when either field
is present, including summary formatting.
In `@polylogue/sources/revision_backfill.py`:
- Around line 1188-1195: Update the census_parse_worker docstring to remove the
outdated claim that it is dispatched through ProcessPoolExecutor as a GIL-build
fallback, including the reference to _parse_unique_retained_raws. Keep the
remaining accurate description of census_parse_worker unchanged.
In `@polylogue/storage/index_generation.py`:
- Around line 539-553: The prune_superseded_generations candidate filter must
include only generations whose state is "active", leaving inactive unpromoted
candidates for discard_if_inactive. Update
polylogue/storage/index_generation.py:539-553 accordingly. Add a regression test
in tests/unit/storage/test_index_generation.py:439-465 that creates an inactive
generation, performs unrelated create-and-promote cycles, and verifies the
inactive generation survives pruning and promote().
- Around line 556-574: Track retired-* marker deletions in the same removal
state used by the pruning method, updating the removed/fsync gate around the
marker cleanup loop. Ensure _fsync_directory(self.generations_root) runs
whenever either generation directories or marker directories were deleted, while
preserving the existing generation IDs for the log message.
- Around line 555-557: Update the candidate ordering in the generation-pruning
flow before the `for _created_at_ms, generation_id, directory in
candidates[keep:]` loop to use a monotonic creation-order tiebreaker when
`created_at_ms` values match. Ensure `create()` records or exposes that sequence
consistently, then sort by timestamp and the monotonic value so retention and
pruning reflect true creation order rather than glob ordering.
In `@polylogue/storage/raw_authority.py`:
- Around line 909-923: Enforce the retention invariant in the configuration or
cleanup logic around the header-retention calculation in raw_authority_censuses:
ensure RAW_AUTHORITY_CENSUS_HEADER_RETENTION is always greater than or equal to
RAW_AUTHORITY_CENSUS_PLAN_RETENTION, failing fast if the constants violate this
relationship. Preserve the existing header deletion behavior in the cleanup
flow.
- Line 1117: Capture the `(plan_rows, headers)` result from
`prune_raw_authority_census_history(conn)` and log a concise message when either
count is non-zero, including both pruned counts. Follow the existing logging
pattern used by `IndexGenerationStore.prune_superseded_generations` without
changing the pruning behavior.
- Around line 889-908: Bound the stale census cleanup performed in
record_raw_authority_census to a fixed maximum per call, limiting the initial
SELECT and subsequent deletions to that batch size. Preserve chunked deletion
and allow later calls to process remaining stale census_ids, while keeping
cleanup within the existing write transaction.
In `@tests/unit/core/test_schema_laws.py`:
- Around line 231-260: Update
test_observed_structure_merge_never_drops_a_named_property to include an
explicit anyOf composite schema containing a null branch and an object branch
with named properties, then merge it with a plain observed-structure schema.
Assert the object branch’s property paths remain reachable after merging,
ensuring _flatten_composite_branches is exercised rather than only the existing
type/properties union path.
In `@tests/unit/storage/test_index_generation.py`:
- Around line 439-465: Add regression coverage for an unpromoted, inactive
generation created through create_transaction alongside unrelated
create-and-promote cycles. Assert that promote() and
prune_superseded_generations() do not remove the in-progress inactive candidate,
while preserving existing cleanup behavior for eligible superseded generations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 16c6d5ca-94bf-4d9b-9f10-3af1c0db7f0d
⛔ Files ignored due to path filters (8)
polylogue/schemas/providers/antigravity/versions/v1/elements/session_document.schema.json.gzis excluded by!**/*.gz,!**/*.json.gzpolylogue/schemas/providers/claude-ai/versions/v1/elements/session_document.schema.json.gzis excluded by!**/*.gz,!**/*.json.gzpolylogue/schemas/providers/claude-code/versions/v1/elements/session_record_stream.schema.json.gzis excluded by!**/*.gz,!**/*.json.gzpolylogue/schemas/providers/claude-code/versions/v1/elements/subagent_session_stream.schema.json.gzis excluded by!**/*.gz,!**/*.json.gzpolylogue/schemas/providers/codex/versions/v1/elements/session_record_stream.schema.json.gzis excluded by!**/*.gz,!**/*.json.gzpolylogue/schemas/providers/gemini-cli/versions/v1/elements/session_document.schema.json.gzis excluded by!**/*.gz,!**/*.json.gzpolylogue/schemas/providers/hermes/versions/v1/elements/session_document.schema.json.gzis excluded by!**/*.gz,!**/*.json.gzuv.lockis excluded by!**/*.lock,!**/*.lock
📒 Files selected for processing (49)
.beads/issues.jsonl.github/workflows/ci.yml.gitignoreCLAUDE.mddevtools/verify.pydocs/configuration.mddocs/design/convergence-simplification-inventory.mddocs/plans/demo-corpus-construct-audit.mddocs/test-quality-workflows.mdflake.nixpolylogue/browser_capture/models.pypolylogue/config.pypolylogue/schemas/generation/dynamic_keys.pypolylogue/schemas/promotion_audit.pypolylogue/schemas/providers/antigravity/catalog.jsonpolylogue/schemas/providers/antigravity/manifest.jsonpolylogue/schemas/providers/antigravity/versions/v1/package.jsonpolylogue/schemas/providers/chatgpt/catalog.jsonpolylogue/schemas/providers/chatgpt/versions/v1/package.jsonpolylogue/schemas/providers/claude-ai/catalog.jsonpolylogue/schemas/providers/claude-ai/versions/v1/package.jsonpolylogue/schemas/providers/claude-code/catalog.jsonpolylogue/schemas/providers/claude-code/versions/v1/package.jsonpolylogue/schemas/providers/codex/catalog.jsonpolylogue/schemas/providers/codex/versions/v1/package.jsonpolylogue/schemas/providers/gemini-cli/catalog.jsonpolylogue/schemas/providers/gemini-cli/manifest.jsonpolylogue/schemas/providers/gemini-cli/versions/v1/package.jsonpolylogue/schemas/providers/gemini/catalog.jsonpolylogue/schemas/providers/gemini/versions/v1/package.jsonpolylogue/schemas/providers/hermes/catalog.jsonpolylogue/schemas/providers/hermes/manifest.jsonpolylogue/schemas/providers/hermes/versions/v1/package.jsonpolylogue/sources/parsers/browser_capture.pypolylogue/sources/parsers/chatgpt.pypolylogue/sources/parsers/claude/code_parser.pypolylogue/sources/revision_backfill.pypolylogue/storage/index_generation.pypolylogue/storage/raw_authority.pypyproject.tomltests/unit/core/test_config_resolution_regression.pytests/unit/core/test_schema_laws.pytests/unit/core/test_schema_promotion_audit.pytests/unit/sources/test_browser_capture.pytests/unit/sources/test_claude_code_sidecar_evidence.pytests/unit/sources/test_parsers_chatgpt.pytests/unit/sources/test_revision_backfill.pytests/unit/storage/test_index_generation.pytests/unit/storage/test_raw_authority_ledger.py
💤 Files with no reviewable changes (4)
- docs/configuration.md
- polylogue/config.py
- tests/unit/core/test_config_resolution_regression.py
- tests/unit/sources/test_revision_backfill.py
Comment on lines
+247
to
+327
| def _catalog_coherence_findings(root: Path) -> list[PromotionAuditFinding]: | ||
| """Require each provider's catalog.json to agree with its packages. | ||
|
|
||
| ``catalog.json`` is the resolution authority: ``runtime_registry`` reads it | ||
| to pick a package and then reads that package's element list. A promotion | ||
| that rewrites ``versions/<v>/package.json`` without regenerating the | ||
| catalog is therefore *inert* -- the new elements and profile families are | ||
| never resolved, while the tree looks promoted. That happened once | ||
| already, so it is a blocker rather than a review item. Writing through | ||
| ``SchemaRegistry.replace_provider_packages`` keeps the two in step. | ||
| """ | ||
| findings: list[PromotionAuditFinding] = [] | ||
| for catalog_path in sorted(root.rglob("catalog.json")): | ||
| provider_dir = catalog_path.parent | ||
| relative = str(catalog_path.relative_to(root)) | ||
| try: | ||
| catalog = _load_artifact(catalog_path) | ||
| except Exception: | ||
| continue # already reported as malformed_artifact | ||
| if not isinstance(catalog, dict): | ||
| continue | ||
| packages = catalog.get("packages") | ||
| if not isinstance(packages, list): | ||
| continue | ||
| catalogued = { | ||
| str(entry.get("version")): entry for entry in packages if isinstance(entry, dict) and entry.get("version") | ||
| } | ||
| on_disk = {path.parent.name for path in provider_dir.glob("versions/*/package.json") if path.is_file()} | ||
| for missing in sorted(on_disk - set(catalogued)): | ||
| findings.append( | ||
| PromotionAuditFinding( | ||
| severity="blocker", | ||
| category="catalog_incoherent", | ||
| artifact=relative, | ||
| json_path="$.packages", | ||
| value=f"version={missing};reason=package_on_disk_absent_from_catalog", | ||
| ) | ||
| ) | ||
| for stale in sorted(set(catalogued) - on_disk): | ||
| findings.append( | ||
| PromotionAuditFinding( | ||
| severity="blocker", | ||
| category="catalog_incoherent", | ||
| artifact=relative, | ||
| json_path="$.packages", | ||
| value=f"version={stale};reason=catalogued_version_has_no_package", | ||
| ) | ||
| ) | ||
| for version in sorted(on_disk & set(catalogued)): | ||
| manifest_path = provider_dir / "versions" / version / "package.json" | ||
| try: | ||
| manifest = _load_artifact(manifest_path) | ||
| except Exception: | ||
| continue | ||
| if not isinstance(manifest, dict): | ||
| continue | ||
| entry = catalogued[version] | ||
| catalog_kinds = _element_kinds(entry.get("elements")) | ||
| manifest_kinds = _element_kinds(manifest.get("elements")) | ||
| if catalog_kinds != manifest_kinds: | ||
| findings.append( | ||
| PromotionAuditFinding( | ||
| severity="blocker", | ||
| category="catalog_incoherent", | ||
| artifact=relative, | ||
| json_path=f"$.packages[version={version}].elements", | ||
| value=f"catalog={catalog_kinds};package={manifest_kinds}", | ||
| ) | ||
| ) | ||
| for field in ("sample_count", "first_seen", "last_seen"): | ||
| if entry.get(field) != manifest.get(field): | ||
| findings.append( | ||
| PromotionAuditFinding( | ||
| severity="blocker", | ||
| category="catalog_incoherent", | ||
| artifact=relative, | ||
| json_path=f"$.packages[version={version}].{field}", | ||
| value=f"catalog={entry.get(field)!r};package={manifest.get(field)!r}", | ||
| ) | ||
| ) | ||
| return findings |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win
Consider extending coherence checks to element-level metadata.
catalog_kinds = _element_kinds(entry.get("elements")) manifest_kinds = _element_kinds(manifest.get("elements")) if catalog_kinds != manifest_kinds: ... for field in ("sample_count", "first_seen", "last_seen"): if entry.get(field) != manifest.get(field): This only checks element kinds (names) and package-level aggregate fields. A provider with multiple element kinds could have per-element sample_count/first_seen/last_seen/bundle_scope_count drift between catalog and package while element kinds and the package-level aggregates still match, which this audit would miss — the same "inert promotion" failure mode the docstring describes, just at finer granularity.
♻️ Sketch: compare per-element records, not just kinds
entry = catalogued[version]
catalog_kinds = _element_kinds(entry.get("elements"))
manifest_kinds = _element_kinds(manifest.get("elements"))
if catalog_kinds != manifest_kinds:
findings.append(...)
+ catalog_elements = {
+ str(e.get("element_kind")): e for e in entry.get("elements", []) if isinstance(e, dict)
+ }
+ manifest_elements = {
+ str(e.get("element_kind")): e for e in manifest.get("elements", []) if isinstance(e, dict)
+ }
+ for kind, catalog_element in catalog_elements.items():
+ manifest_element = manifest_elements.get(kind, {})
+ for field in ("sample_count", "first_seen", "last_seen"):
+ if catalog_element.get(field) != manifest_element.get(field):
+ findings.append(...)Also applies to: 368-368
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@polylogue/schemas/promotion_audit.py` around lines 247 - 327, Extend
_catalog_coherence_findings to compare matching element records in catalog and
manifest, not only _element_kinds and package-level fields. For each element,
validate sample_count, first_seen, last_seen, and bundle_scope_count, reporting
catalog_incoherent findings with an element-specific JSON path and values when
they differ. Preserve the existing package-level and kind checks.
Comment on lines
+889
to
+908
| plan_rows = 0 | ||
| if plan_floor_row is not None: | ||
| floor = int(plan_floor_row[0]) | ||
| stale = [ | ||
| str(row[0]) | ||
| for row in conn.execute( | ||
| "SELECT census_id FROM raw_authority_censuses WHERE sequence_no < ?", | ||
| (floor,), | ||
| ) | ||
| ] | ||
| for start in range(0, len(stale), 512): | ||
| chunk = stale[start : start + 512] | ||
| for statement in ( | ||
| "DELETE FROM raw_authority_census_plans WHERE census_id = ?", | ||
| "DELETE FROM raw_authority_census_post_plans WHERE census_id = ?", | ||
| "DELETE FROM raw_authority_blockers WHERE census_id = ?", | ||
| ): | ||
| cursor = conn.executemany(statement, ((census,) for census in chunk)) | ||
| plan_rows += cursor.rowcount if cursor.rowcount and cursor.rowcount > 0 else 0 | ||
|
|
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
Unbounded first-run backlog cleanup runs synchronously inside the write transaction.
stale is built from every census_id older than the retention floor with no cap, then deleted via chunked (512-row) executemany across three tables. On an archive that has already accumulated a large backlog under the old unbounded behavior (the motivating case cited in the docstring above — multi-GB, 99.98% carried_forward rows, ~990 MB/day growth), the very first call to record_raw_authority_census after this ships could need to process many thousands of stale census_ids inline, inside the same write transaction as the new census insert. Given "the daemon is the sole SQLite writer," this holds the only writer for a potentially long duration on that first call, stalling all other daemon writes queued behind it.
Consider bounding the amount of backlog processed per call (letting subsequent calls make further progress), matching the "automagic... maintained automatically" philosophy already stated for this function without risking a pathological first-run stall.
⚡ Proposed fix: cap per-call backlog processing
+_MAX_CENSUS_PRUNE_ROWS_PER_CALL = 5_000 # bound worst-case first-run backlog cleanup per call
+
def prune_raw_authority_census_history(conn: sqlite3.Connection) -> tuple[int, int]:
...
if plan_floor_row is not None:
floor = int(plan_floor_row[0])
stale = [
str(row[0])
for row in conn.execute(
- "SELECT census_id FROM raw_authority_censuses WHERE sequence_no < ?",
- (floor,),
+ "SELECT census_id FROM raw_authority_censuses WHERE sequence_no < ? LIMIT ?",
+ (floor, _MAX_CENSUS_PRUNE_ROWS_PER_CALL),
)
]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@polylogue/storage/raw_authority.py` around lines 889 - 908, Bound the stale
census cleanup performed in record_raw_authority_census to a fixed maximum per
call, limiting the initial SELECT and subsequent deletions to that batch size.
Preserve chunked deletion and allow later calls to process remaining stale
census_ids, while keeping cleanup within the existing write transaction.
| """, | ||
| ((census_id, plan.plan_id, ordinal) for ordinal, plan in enumerate(plans)), | ||
| ) | ||
| prune_raw_authority_census_history(conn) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
Prune counts are computed but never surfaced.
prune_raw_authority_census_history returns (plan_rows, headers), but the call site discards it, and unlike the sibling IndexGenerationStore.prune_superseded_generations (which logs pruned counts), nothing here logs when rows are actually deleted. Given this runs on every census and is expected to do meaningful work on first activation of archives with a large pre-existing backlog, logging non-zero counts would aid observability of that migration.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@polylogue/storage/raw_authority.py` at line 1117, Capture the `(plan_rows,
headers)` result from `prune_raw_authority_census_history(conn)` and log a
concise message when either count is non-zero, including both pruned counts.
Follow the existing logging pattern used by
`IndexGenerationStore.prune_superseded_generations` without changing the pruning
behavior.
…owser-wins Problem: three coalescing tests on master asserted a browser-capture title should outrank a coexisting direct/GDPR export title (test_browser_capture_raw_payload_coalesces_with_chatgpt_export x2 parametrizations, test_browser_capture_raw_payload_coalesces_with_claude_ai_export). All three failed on a clean origin/master checkout, order-independently. Investigation established this is a stale test contract, not a regression: #3179 (b473d92, polylogue-z1c6) intentionally added a "direct export always wins" mirror rule to browser_capture_precedence() in polylogue/storage/sqlite/archive_tiers/ingest_precedence.py, so a browser capture can never shadow a genuine export once it arrives (only backfill ahead of one). That PR predates these three tests' last edit and updated the sibling proof test_archive_tiers_archive_facade_export_vs_native_precedence_is_order_independent in tests/unit/storage/test_archive_tiers_archive.py (already asserting export-wins) but missed these three in test_browser_capture.py. Live-archive check (read-only, /realm/db/polylogue): 0 sessions currently have raw_sessions rows with more than one distinct capture_mode for the same (origin, native_id), so no live session's title is affected either way -- production has been export-wins all along, only the tests were stale. What changed: updated the three title assertions to the current, intentional rule and added a comment citing the precedence function, #3179/polylogue-z1c6, and the sibling order-independence test. Ref polylogue-lrdh Verification: devtools test tests/unit/sources/test_browser_capture.py -k coalesces -> 3 passed devtools test tests/unit/sources/test_browser_capture.py -> full file green devtools test tests/unit/storage/test_archive_tiers_archive.py -k precedence -> 13 passed devtools verify --quick -> exit_code 0 Co-Authored-By: Claude <noreply@anthropic.com>
Problem: detect_context_compaction read item.get("compact_metadata") and
meta.get("preserved_segment")/.get("anchor_uuid") (snake_case), but every
live Claude Code compact_boundary record uses camelCase
(compactMetadata/preservedSegment/anchorUuid) -- confirmed against the real
corpus 2026-07-29. trigger/pre_tokens/preserved_segment_id were silently
null on every compaction event ever ingested. The existing tests encoded the
same wrong assumption, matching the buggy reader rather than the wire shape.
Separately, microcompact_boundary (Claude Code's incremental "trim old tool
results" compaction, distinct from a full recompact) was not detected at all
and fell through to ordinary message parsing, keeping only a placeholder
"Context microcompacted" string and losing trigger/preTokens/tokensSaved/
compactedToolIds/clearedAttachmentUUIDs entirely.
What changed: compactMetadata/preservedSegment/anchorUuid are now read as
the primary (camelCase) shape with the old snake_case kept as a defensive
fallback. Added detect_micro_compaction + a new micro_compaction session
event wired into code_parser.py.
Verification: devtools test tests/unit/sources/test_compaction.py -- 42 passed.
Problem
`count_search_sessions` and `search_session_ids` joined `sessions` unconditionally
so that an OPTIONAL session-level filter, built with alias `s`, would have
something to reference. Neither query projects an `s.*` column, and
`blocks.session_id` already carries the value. For the common case -- a bare
`polylogue find "docker"` with no session filter -- every matched block paid a
sessions primary-key probe that resolved nothing, on the hot path of every
search.
Measured on the live archive (18,871 sessions / 4.9M messages), controlled A/B
on two never-previously-queried terms of comparable match volume:
with the join 2.50 pread64 syscalls per matched block
without 1.29
Dropping it roughly halves random reads per match, and cuts warm wall time ~18%
(10.1ms -> 8.2ms, 5-run mean, same connection). Cold, the exact-count query for
a common term was ~1.15s.
This is not a missing index. EXPLAIN already chose correct index paths on both
sides (`SEARCH b USING INTEGER PRIMARY KEY`, `SEARCH s USING INDEX
sqlite_autoindex_sessions_1`). It is a join that should not have been emitted.
What changed
`_sessions_join_if_filtered` emits the join only when the rendered WHERE clause
references alias `s`, keyed on the WHERE text itself so the join can never be
elided while something still references it. Both callers' explicit `session_id`
predicate moves from `s.session_id = ?` to `b.session_id = ?`, so that predicate
alone no longer drags the join back in -- `blocks.session_id` is the same value.
Behaviour is unchanged: with any session-level filter present the emitted SQL is
identical to before.
Verification
devtools test tests/unit/storage/test_archive_tiers_search_guard.py \
tests/unit/cli/test_query_exec_laws.py \
tests/unit/mcp/test_query_request_contracts.py \
tests/unit/storage/test_fts_repair_sql.py 115 passed
ruff check / mypy on the changed module clean
Measurement credit: the read-path investigation lane, which located and
quantified this but could not land it -- `archive_tiers/archive.py` was outside
its write scope.
…e evidence Problem: parser-diff triage (bd polylogue-2qx.3/polylogue-cgfy) over the claude-code parser found several record types and message-scoped fields carrying real corpus signal that nothing read: - custom-title (1,056 live occurrences) and file-history-delta (304) are two sidecar record types outside the original twelve in _SKIPPED_SIDECAR_RECORD_TYPES; they carry no message/text so they fell through to ordinary parsing and were silently skipped (continue, no event). - toolUseResult (Claude Code's own structured tool-result sidecar, parallel to the Anthropic-protocol tool_result content block) was read for only two of its ~60 observed subfields (backgroundTaskId, retrieval_status/task). sandbox/interrupted/file/structuredPatch/newTodos-oldTodos and friends were read nowhere. - message.ttftMs, stop_reason/stop_sequence, usage.cache_creation's 1h/5m split, usage.service_tier/inference_geo, and message.diagnostics. cache_miss_reason were all unread despite carrying real values on the live corpus. - sessionKind and gitBranch (stamped on every JSONL record) were read nowhere in the primary parse path; gitBranch was only ever populated via a separate, often-absent legacy sessions-index.json sidecar file. What changed: - custom-title -> claude_custom_title event; an explicit user rename now also wins the session title over the provider-suggested ai-title (stronger intent signal). - file-history-delta -> claude_file_history_delta event. - New claude_tool_execution_result event projects toolUseResult's bounded structural facts (sandbox, interrupted, isImage, userModified, numFiles, file path/line extents, structuredPatch hunk/line counts, bounded filename list). Deliberately excludes stdout/stderr/output free text: that duplicates the tool_result content block already captured and risks unbounded payload size. - New claude_todo_state event projects TodoWrite's accepted before/after state (content/status/activeForm/priority) -- the call *input* was already captured wholesale via tool_input; this is the *result*. - _message_usage_event_payload gained ttft_ms, stop_reason, stop_sequence, cache_creation_by_ttl, service_tier, inference_geo, iterations, speed, and cache_miss_reason, each recorded only when present. - New claude_session_kind event; ParsedSession.git_branch now set directly from the record stream instead of only via the legacy index sidecar. Verification: devtools test tests/unit/sources/test_claude_code_sidecar_evidence.py tests/unit/sources/test_parsers_claude_code_artifacts.py tests/unit/sources/test_assembly_claude_code_history.py tests/unit/sources/test_claude_code_normalization_laws.py -- all passed. mypy polylogue/sources/parsers/claude/ -- no issues. Ref polylogue-2qx.3
…omote-schemas-and-wire-gates
…omote-schemas-and-wire-gates
…on summary Problem: parser-diff triage over the claude-ai parser (bd polylogue-2qx.3) ranked chat_messages[].content[].start_timestamp/stop_timestamp, approval_key/approval_options, display_content, integration_name/ integration_icon_url at 85-99% document coverage, and none were read. These are Claude AI web connected-app (MCP) tool_use/tool_result segment fields: per-block wall-clock timing, which integration served the call, the human-in-the-loop approval gate shown before running it, and the provider's own rendered summary of the call. content_blocks_from_segments (base_support.py) is shared with Claude Code, which never emits these fields, so they belong in the claude-specific layer, not the shared one. Separately, the top-level ``summary`` field (Claude's own generated conversation summary, distinct from name/title) was read nowhere despite carrying real multi-paragraph text on the live corpus. What changed: _claude_ai_web_tool_evidence (common.py) projects the above fields into ParsedContentBlock.metadata for tool_use/tool_result segments; parse_ai (ai_parser.py) emits a claude_ai_conversation_summary session event for a non-empty top-level summary. Also recorded (as code comments, common.py): several parser-diff rows are false positives already read via shared helpers outside the tool's per- provider module scan (attachment_from_meta, content_blocks_from_segments, the wholesale tool_input dict copy), one deliberately-deferred cluster (nested Drive/doc-citation content[].content[] records -- needs its own construct-projection design, filed as a to-acquire item, not silently dropped), and one deliberately-dropped cluster (single low-frequency fields duplicating display_content/tool_input with no corpus evidence of unique signal; top-level ``account`` -- provider identity, out of scope and a PII risk to conflate into content evidence). Verification: devtools test tests/unit/sources/test_parsers_claude_ai_catalog.py -- 43 passed. mypy polylogue/sources/parsers/claude/ -- no issues. Ref polylogue-2qx.3
…omote-schemas-and-wire-gates
…omote-schemas-and-wire-gates
Both lanes ran mypy over polylogue/ but not over their own test files, so four `object is not indexable` / `not Sized` errors only surfaced once the branch gate ran mypy across the whole tree. Cast the JSON payload reads at the assertion site; no assertion semantics change. Verification: mypy on both files clean; devtools test on both -> 110 passed.
Problem
Committed schema packages stamp x-polylogue-element-first-seen/-last-seen
at the artifact-kind (schema-per-element) level, all generated the same
microsecond. There is no per-FIELD first/last-seen, so the drift sentinel
can only say a field is new relative to a package's promotion date, not
when it actually first appeared in the corpus (polylogue-2qx.3 AC4).
What changed
The generation pass already threads one session_id per flattened schema
sample through _collect_field_stats (MembershipSessionIds) for enum
session-evidence capping. This adds a parallel observed_ats collection
(MembershipObservedAts / JournalMemberships.iter_observed_ats /
ObservationJournal.iter_membership_observed_ats), reusing the exact same
per-unit SchemaUnit.observed_at value the package-level window already
reads -- no second timestamp scan. FieldStats gains
field_first_seen/field_last_seen (observe_field_timestamp), rolled up
during the existing per-sample walk in _collect_field_stats. annotate_schema
emits x-polylogue-field-first-seen/-last-seen per property when present.
Cost of a live regeneration: unchanged from before this PR -- this only
adds an annotation to the existing generation pass output, so any full
schema regeneration (already scoped as promote-and-report per AC1) picks it
up for free; no separate corpus pass is needed. Verified against a small
synthetic corpus per instruction, not the 2.1M-sample live archive.
Verification
devtools test tests/unit/core/test_field_stats.py tests/unit/core/test_schema_annotation_contracts.py
-> 130 passed
devtools verify --quick -> exit 0
Ref polylogue-2qx.3
…-parser coverage join
Problem
The drift sentinel (polylogue-da1) classifies UNSEEN_SHAPE, NEW_FIELD, and
FIELD_CHANGED -- all three ask what the SCHEMA does not know. There is no
classification for "schema knows this field, the parser ignores it": a
payload that validates cleanly, resolves to a real package, and carries no
unknown fields silently classifies as no drift at all, even when it
carries a field the committed schema has observed for months that no
parser module ever reads (polylogue-2qx.3 AC5).
What changed
- polylogue/schemas/drift_sentinel.py: new KNOWN_FIELD_UNREAD
classification, added to RISKY_CLASSIFICATIONS (so the existing
status.py windowed risky-rate gate picks it up with no further wiring --
it already reads RISKY_CLASSIFICATIONS via ops_write.py). classify_schema_drift
gains an `unread_known_fields` parameter, checked last (behind
FIELD_CHANGED / UNSEEN_SHAPE / NEW_FIELD, which are each a stronger,
more specific signal).
- polylogue/schemas/schema_parser_coverage.py (new): the static join
itself -- schema_known_field_names() walks committed schema packages'
`properties`; parser_referenced_field_names() AST-parses each provider's
configured parser modules for string constants; unread_field_names()
is the difference; payload_unread_field_names() intersects one
in-memory payload's own keys against that set (what the sentinel needs
at ingest time without a full corpus re-walk).
- polylogue/pipeline/services/ingest_worker.py: _classify_plan_drift now
computes payload_unread_field_names for the resolved provider (only
when the payload would otherwise classify as clean) and threads it
into classify_schema_drift; the SchemaDriftObservation's
unseen_key_signature carries the unread field names for this
classification (drift_warnings is empty in this branch).
- devtools/schema_parser_diff.py (new) + command_catalog.py: the
committed, runnable schema-vs-parser diff CLI (`devtools lab schema
parser-diff`) that scopes a parser batch by evidence (AC2's stated
goal). Includes the hermes triage lane's confirmed correction: local_agent.py
defines both parse_gemini_cli and parse_hermes, so it must map to BOTH
providers in PROVIDER_PARSERS -- mapping it to gemini-cli alone produced
21 false-positive "unread" rows for hermes (including last_updated).
Also documents two accuracy limits the same lane hand-confirmed:
name-collision over-counting (call_id/response_item_id masking a
genuinely-unread nested path) and verbatim-object under-counting
(parameters.properties.<argname> rows where the parser retains the
whole object). polylogue/schemas/schema_parser_coverage.py's own
PROVIDER_PARSERS mirrors the same correction.
Verification against the live archive's committed schema packages:
against my checkout's currently-committed provider packages (which
predate the x-polylogue-observed-distribution annotation and AC1's
promotion), `devtools lab schema parser-diff` reports 0 rows for every
provider -- there is no per-key document/encountered count to rank by
yet. This is not a defect in the corrected mapping: verified directly
that the local_agent.py fix changes the referenced-string set (adds 63
strings including "last_updated", confirmed absent from the pre-fix
hermes-only set) and that schema_known_field_names()/unread_field_names()
work correctly against synthetic fixtures (see
tests/unit/schemas/test_schema_parser_coverage.py). Real per-provider
totals need a schema promotion (AC1) that lands x-polylogue-observed-distribution
into the committed packages this diff reads.
Verification
devtools test tests/unit/schemas/test_drift_sentinel.py tests/unit/schemas/test_schema_parser_coverage.py tests/unit/pipeline/test_schema_drift_sentinel_ingest.py
-> 32 passed
ruff check / mypy --strict (dmypy) -> clean
devtools render devtools-reference && devtools render topology-projection && devtools render topology-status
devtools render all --check -> OK
devtools verify --quick -> exit 0
Ref polylogue-2qx.3, polylogue-cgfy, polylogue-da1
…omote-schemas-and-wire-gates # Conflicts: # devtools/schema_parser_diff.py
Problem: ~/.codex holds five live SQLite databases Polylogue never acquires (polylogue-0jf4). threads.title and thread_spawn_edges have no other evidence source -- verified empirically that no Codex rollout JSONL session_meta record carries a curated title or an orchestration- level parent/child spawn relationship. What changed: a new, independent parser module polylogue/sources/parsers/codex_state.py classifies each of the five databases (acquire / acquire-partial / out-of-scope, with reasons -- CODEX_STATE_FIDELITY) and parses threads/thread_spawn_edges, thread_goals, and stage1_outputs structural facts from a *snapshot*, reusing sqlite_snapshot.py's existing consistent-backup mechanism (snapshot_sqlite_database/snapshot_sqlite_to_blob) rather than adding a second acquisition mechanism -- the same path parsers/hermes_state.py already uses for Hermes. sqlite_snapshot.py gains codex_state_raw_id(), mirroring hermes_profile_raw_id() but scoped to a single-instance install (no profile multiplicity). spawn_edges_as_session_events() produces ParsedSessionEvent rows shaped for session_events' existing "any new event_type, no schema change" path. Deliberately not done here (see codex_state.py's module docstring for detail): wiring the new detector into sources/dispatch.py, widening the "codex" WatchSource root in sources/live/watcher.py from ~/.codex/sessions to ~/.codex, and folding thread_spawn_edges into the existing codex-session rows' session_events without a full-replace data loss. Those touch shared, actively-contended files outside this change's write scope (sources/live/**, this new parser module, sqlite_snapshot.py) -- dispatch.py in particular, and title assembly specifically, overlaps polylogue-ih67 (in progress, separate lane, INDEX_SCHEMA_VERSION 45 already claimed there). logs_2.sqlite (627 MB runtime tracing) and codex-dev.db (empty local automation scheduling config) are classified out-of-scope, not acquired. Verification: - devtools test tests/unit/sources/parsers/test_codex_state.py tests/unit/sources/parsers/test_codex_state_schema_canary.py -> 19 passed, including a concurrent-writer test proving the snapshot never blocks a live Codex and never observes an uncommitted mid-transaction write, and a content-hash idempotency test. - python3 -m mypy polylogue/sources/parsers/codex_state.py polylogue/sources/sqlite_snapshot.py -> no issues. - devtools verify --quick -> exit 0 (required adding a hash-boundary-registry.yaml entry for the new hashlib.sha256 call site in codex_state_raw_id). - devtools render topology-projection && devtools render topology-status; devtools render all --check -> no "out of sync". - Parsed the real live ~/.codex/state_5.sqlite (snapshotted to /realm/tmp, never modified in place): 3,054 threads, 2,771 with a non-empty title, 1,030 spawn edges, 26 thread_goals, 30 stage1_outputs rows -- matches the bead's measured counts exactly. Ref polylogue-0jf4
…omote-schemas-and-wire-gates
Problem: docs/design/convergence-simplification-inventory.md item 4 named _RAW_MATERIALIZATION_CENSUS_BATCH_LIMIT + the census_mode escalation switch as one number doing double duty -- a parse-throughput knob AND a writer-hold-duration knob -- which cannot serve both jobs well. What changed: polylogue/daemon/cli.py's periodic raw-materialization loop no longer escalates the writer-held pass's own limit via a census_mode guess. The parse-stage warmer (_maybe_warm_raw_materialization_parse_stage, already running off the writer hold via DaemonParseStage) now always prefetches up to a dedicated, independent _RAW_MATERIALIZATION_PARSE_STAGE_WARM_LIMIT (64) candidates, and the writer-held pass widens its own limit only to match how many raws the warmer *actually* admitted this tick (bounded between the 16-floor and the 64-ceiling), instead of guessing from the prior pass's outcome. Consuming an already-parsed prefetch hit costs a receipt write, not a reparse, so this does not meaningfully extend the writer hold. With daemon_parse_stage_split off (today's default), the warmer is a no-op and the pass limit is simply the unchanged floor. Item 5 (per-pass candidate requery in polylogue/storage/repair.py) was investigated but not landed: a PRAGMA data_version / SQLite file-header change-counter memoization attempt broke ~29 tests with genuine staleness, root-caused to WAL-mode checkpoint behavior (verified via a standalone repro). That change was fully reverted; the doc records the dead end so a future attempt does not repeat it. Verification: devtools test polylogue/daemon/cli.py tests/unit/daemon/test_daemon_cli.py (117 passed); mypy --strict polylogue/daemon/cli.py (success); devtools verify --quick (exit 0). Ref polylogue-m6tp Co-Authored-By: Claude <noreply@anthropic.com>
…omote-schemas-and-wire-gates
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@polylogue/sources/parsers/hermes_spans.py`:
- Around line 1108-1133: Update the span-building function containing the
hermes_tool_availability_span return so it emits that event only when tools is a
non-empty list; return no event for tool-less llm_request values. Keep tool
metadata generation unchanged for requests with tools, and route any retained
instruction, tool-choice, or reasoning metadata through the existing
hermes_llm_request_span instead.
In `@polylogue/storage/raw_retention.py`:
- Around line 562-572: Update the _application_matches_head docstring to clarify
that logical_source_key is the implicit eighth join column, already matched by
using it to look up head; retain the existing seven explicit comparisons and the
parallel with _active_index_raw_authority.
- Around line 698-707: Update the head-row cache in the stale-candidate loop to
use head.accepted_raw_id as its key instead of logical_source_key, and adjust
the cache lookup and assignment accordingly. Preserve the existing
_raw_revision_rows lookup and head-row behavior while allowing shared head raws
across logical sources to reuse one cached result.
- Around line 524-559: The _read_superseded_applications query currently loads
all historical superseded receipts; pre-filter them in SQL by joining
raw_revision_applications to raw_revision_heads using the same seven fields
checked by _application_matches_head, retaining only stale or no-head groups
needed by plan_stale_supersession_reissue. Add a companion COUNT(*) query for
already-current receipts so already_current_count remains accurate without
Python-side tallying, while preserving validation and returned application data.
- Around line 836-842: In the per-item guard around
record_revision_application_sync, catch sqlite3.Error in addition to
RuntimeError and ValueError so SQL failures are recorded in errors for that
raw_id and processing continues to index_conn.commit(). Preserve the existing
reissued count and error-message behavior for successful and failed receipts.
In `@polylogue/storage/repair.py`:
- Around line 5823-5833: Remove the unused preview_stale_supersession_receipts
helper, or register and invoke it through PREVIEW_HANDLERS so the preview path
uses it consistently. Ensure stale supersession preview output is rendered in
one place rather than duplicated in repair_stale_supersession_receipts().detail.
In `@polylogue/storage/sqlite/archive_tiers/index.py`:
- Around line 1437-1459: Update the child_identity_text CTE to select only the
live message variant by adding the is_active_path = 1 predicate to both the
outer messages query and its MIN(position) subquery. Preserve the existing
first-user-turn and text-block selection while ensuring each child_session_id
produces at most one identity row.
In `@tests/unit/storage/test_raw_retention.py`:
- Around line 2018-2019: Update the test invocation of
reissue_stale_supersession_receipts to open and pass a separate SQLite
connection to index_db_path as index_conn instead of reusing source_conn. Keep
source_conn for the source database and preserve the dry_run assertion flow so
any unintended write targets the distinct index connection and fails visibly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 735f573e-fd7f-4c0d-843c-de4fcee97b1f
📒 Files selected for processing (29)
.beads/issues.jsonldevtools/command_catalog.pydevtools/schema_parser_diff.pydocs/devtools.mddocs/openapi/search.yamldocs/schemas/cli-output/query-unit-envelope.schema.jsonpolylogue/annotations/join.pypolylogue/api/archive.pypolylogue/core/refs.pypolylogue/insights/delegation_work_evidence.pypolylogue/sources/parsers/codex.pypolylogue/sources/parsers/hermes_spans.pypolylogue/storage/raw_retention.pypolylogue/storage/repair.pypolylogue/storage/sqlite/archive_tiers/archive.pypolylogue/storage/sqlite/archive_tiers/index.pypolylogue/storage/sqlite/archive_tiers/write.pypolylogue/storage/sqlite/lifecycle.pypolylogue/surfaces/payloads.pytests/unit/annotations/test_join.pytests/unit/api/test_facade_contracts.pytests/unit/pipeline/test_hermes_raw_identity.pytests/unit/sources/parsers/test_hermes_spans.pytests/unit/sources/test_browser_capture.pytests/unit/sources/test_parsers_codex.pytests/unit/storage/test_delegation_facts_bulk_rebuild.pytests/unit/storage/test_delegations_view.pytests/unit/storage/test_raw_retention.pywebui/src/api/generated.ts
💤 Files with no reviewable changes (2)
- docs/schemas/cli-output/query-unit-envelope.schema.json
- docs/openapi/search.yaml
Comment on lines
+1108
to
+1133
| llm_request = json_document(extra.get("llm_request")) | ||
| if not llm_request: | ||
| return [] | ||
| payload: dict[str, object] = {"step_index": index} | ||
| if step_id is not None: | ||
| payload["step_id"] = step_id | ||
| tools = json_document_list(llm_request.get("tools")) | ||
| if tools: | ||
| payload["tool_count"] = len(tools) | ||
| payload["tools"] = [ | ||
| {"name": tool.get("name"), "description": tool.get("description"), "parameters": tool.get("parameters")} | ||
| for tool in tools | ||
| ] | ||
| instructions = llm_request.get("instructions") | ||
| payload["has_instructions"] = isinstance(instructions, str) and bool(instructions) | ||
| payload["has_input"] = llm_request.get("input") is not None | ||
| tool_choice = llm_request.get("tool_choice") | ||
| if tool_choice is not None: | ||
| payload["tool_choice"] = tool_choice | ||
| parallel_tool_calls = llm_request.get("parallel_tool_calls") | ||
| if isinstance(parallel_tool_calls, bool): | ||
| payload["parallel_tool_calls"] = parallel_tool_calls | ||
| reasoning_effort = json_document(llm_request.get("reasoning")).get("effort") | ||
| if isinstance(reasoning_effort, str) and reasoning_effort: | ||
| payload["reasoning_effort"] = reasoning_effort | ||
| return [ParsedSessionEvent(event_type="hermes_tool_availability_span", payload=payload)] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
hermes_tool_availability_span is emitted even when no tools were offered.
Any step with an extra.llm_request yields a span, including ones with no tools list. Those tool-less spans still increment tool_availability_spans (Line 1295) and flip the tool_availability capability to exact, which overstates a capability whose detail claims a captured tool-definition schema.
🛠️ Gate the span on a non-empty tool list
llm_request = json_document(extra.get("llm_request"))
if not llm_request:
return []
+ tools = json_document_list(llm_request.get("tools"))
+ if not tools:
+ return []
payload: dict[str, object] = {"step_index": index}
if step_id is not None:
payload["step_id"] = step_id
- tools = json_document_list(llm_request.get("tools"))
- if tools:
- payload["tool_count"] = len(tools)
- payload["tools"] = [
- {"name": tool.get("name"), "description": tool.get("description"), "parameters": tool.get("parameters")}
- for tool in tools
- ]
+ payload["tool_count"] = len(tools)
+ payload["tools"] = [
+ {"name": tool.get("name"), "description": tool.get("description"), "parameters": tool.get("parameters")}
+ for tool in tools
+ ]If tool-less llm-request configuration (instructions/tool_choice/reasoning flags) is worth keeping, fold it into the existing hermes_llm_request_span payload instead.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| llm_request = json_document(extra.get("llm_request")) | |
| if not llm_request: | |
| return [] | |
| payload: dict[str, object] = {"step_index": index} | |
| if step_id is not None: | |
| payload["step_id"] = step_id | |
| tools = json_document_list(llm_request.get("tools")) | |
| if tools: | |
| payload["tool_count"] = len(tools) | |
| payload["tools"] = [ | |
| {"name": tool.get("name"), "description": tool.get("description"), "parameters": tool.get("parameters")} | |
| for tool in tools | |
| ] | |
| instructions = llm_request.get("instructions") | |
| payload["has_instructions"] = isinstance(instructions, str) and bool(instructions) | |
| payload["has_input"] = llm_request.get("input") is not None | |
| tool_choice = llm_request.get("tool_choice") | |
| if tool_choice is not None: | |
| payload["tool_choice"] = tool_choice | |
| parallel_tool_calls = llm_request.get("parallel_tool_calls") | |
| if isinstance(parallel_tool_calls, bool): | |
| payload["parallel_tool_calls"] = parallel_tool_calls | |
| reasoning_effort = json_document(llm_request.get("reasoning")).get("effort") | |
| if isinstance(reasoning_effort, str) and reasoning_effort: | |
| payload["reasoning_effort"] = reasoning_effort | |
| return [ParsedSessionEvent(event_type="hermes_tool_availability_span", payload=payload)] | |
| llm_request = json_document(extra.get("llm_request")) | |
| if not llm_request: | |
| return [] | |
| tools = json_document_list(llm_request.get("tools")) | |
| if not tools: | |
| return [] | |
| payload: dict[str, object] = {"step_index": index} | |
| if step_id is not None: | |
| payload["step_id"] = step_id | |
| payload["tool_count"] = len(tools) | |
| payload["tools"] = [ | |
| {"name": tool.get("name"), "description": tool.get("description"), "parameters": tool.get("parameters")} | |
| for tool in tools | |
| ] | |
| instructions = llm_request.get("instructions") | |
| payload["has_instructions"] = isinstance(instructions, str) and bool(instructions) | |
| payload["has_input"] = llm_request.get("input") is not None | |
| tool_choice = llm_request.get("tool_choice") | |
| if tool_choice is not None: | |
| payload["tool_choice"] = tool_choice | |
| parallel_tool_calls = llm_request.get("parallel_tool_calls") | |
| if isinstance(parallel_tool_calls, bool): | |
| payload["parallel_tool_calls"] = parallel_tool_calls | |
| reasoning_effort = json_document(llm_request.get("reasoning")).get("effort") | |
| if isinstance(reasoning_effort, str) and reasoning_effort: | |
| payload["reasoning_effort"] = reasoning_effort | |
| return [ParsedSessionEvent(event_type="hermes_tool_availability_span", payload=payload)] |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@polylogue/sources/parsers/hermes_spans.py` around lines 1108 - 1133, Update
the span-building function containing the hermes_tool_availability_span return
so it emits that event only when tools is a non-empty list; return no event for
tool-less llm_request values. Keep tool metadata generation unchanged for
requests with tools, and route any retained instruction, tool-choice, or
reasoning metadata through the existing hermes_llm_request_span instead.
Comment on lines
+524
to
+559
| def _read_superseded_applications(index_db_path: Path) -> tuple[_StaleSupersededApplication, ...]: | ||
| if not index_db_path.is_file(): | ||
| raise RawRetentionSafetyError(f"index tier is unavailable: {index_db_path}") | ||
| try: | ||
| uri = f"{index_db_path.resolve().as_uri()}?mode=ro" | ||
| with closing(sqlite3.connect(uri, uri=True)) as conn: | ||
| conn.execute("PRAGMA query_only = ON") | ||
| rows = conn.execute( | ||
| """SELECT raw_id, session_id, logical_source_key, source_revision, | ||
| accepted_raw_id, accepted_source_revision, accepted_content_hash, | ||
| acquisition_generation, append_end_offset, decided_at_ms | ||
| FROM raw_revision_applications | ||
| WHERE decision = 'superseded'""" | ||
| ).fetchall() | ||
| except (OSError, sqlite3.Error) as exc: | ||
| raise RawRetentionSafetyError(f"index tier raw authority is unreadable: {exc}") from exc | ||
| applications: list[_StaleSupersededApplication] = [] | ||
| for row in rows: | ||
| accepted_content_hash = row[6] | ||
| if accepted_content_hash is not None and not isinstance(accepted_content_hash, bytes): | ||
| raise RawRetentionSafetyError(f"superseded application has a non-blob accepted_content_hash: {row[0]!r}") | ||
| applications.append( | ||
| _StaleSupersededApplication( | ||
| raw_id=str(row[0]), | ||
| session_id=str(row[1]), | ||
| logical_source_key=str(row[2]), | ||
| source_revision=str(row[3]), | ||
| accepted_raw_id=str(row[4]) if row[4] is not None else None, | ||
| accepted_source_revision=str(row[5]) if row[5] is not None else None, | ||
| accepted_content_hash=accepted_content_hash, | ||
| acquisition_generation=int(row[7]), | ||
| append_end_offset=int(row[8]) if row[8] is not None else None, | ||
| decided_at_ms=int(row[9]), | ||
| ) | ||
| ) | ||
| return tuple(applications) |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
Consider pre-filtering superseded receipts in SQL.
_read_superseded_applications materializes every superseded row in the archive into Python dataclasses, then discards the overwhelming majority (already_current / no-head groups) in plan_stale_supersession_reissue. On a full archive this table grows with every retention decision ever made, so both memory and the group-building loop scale with total history rather than with the stale set. The staleness predicate is expressible as a join against raw_revision_heads on the same seven columns _application_matches_head compares, which lets SQLite drop already-current receipts before they cross into Python.
Note the plan still needs the already_current_count, so that would become a COUNT(*) companion query rather than a Python tally.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@polylogue/storage/raw_retention.py` around lines 524 - 559, The
_read_superseded_applications query currently loads all historical superseded
receipts; pre-filter them in SQL by joining raw_revision_applications to
raw_revision_heads using the same seven fields checked by
_application_matches_head, retaining only stale or no-head groups needed by
plan_stale_supersession_reissue. Add a companion COUNT(*) query for
already-current receipts so already_current_count remains accurate without
Python-side tallying, while preserving validation and returned application data.
Comment on lines
+562
to
+572
| def _application_matches_head(app: _StaleSupersededApplication, head: _CurrentRawRevisionHead) -> bool: | ||
| """Mirror the exact eight-column join ``_active_index_raw_authority`` uses.""" | ||
| return ( | ||
| app.session_id == head.session_id | ||
| and app.accepted_raw_id == head.accepted_raw_id | ||
| and app.accepted_source_revision == head.accepted_source_revision | ||
| and app.accepted_content_hash == head.accepted_content_hash | ||
| and app.acquisition_generation == head.acquisition_generation | ||
| and app.append_end_offset == head.append_end_offset | ||
| and app.decided_at_ms == head.decided_at_ms | ||
| ) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
Docstring says eight columns; the body compares seven.
logical_source_key is the implicit eighth (it is the lookup key for head), which is fine, but the docstring reads as if all eight are checked here. A half-line note that the key column is matched by construction would keep the parallel with _active_index_raw_authority auditable.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@polylogue/storage/raw_retention.py` around lines 562 - 572, Update the
_application_matches_head docstring to clarify that logical_source_key is the
implicit eighth join column, already matched by using it to look up head; retain
the existing seven explicit comparisons and the parallel with
_active_index_raw_authority.
Comment on lines
+698
to
+707
| # One lookup per distinct head raw, cached across every stale raw that | ||
| # shares a logical source (typically many-to-one). | ||
| head_row_cache: dict[str, sqlite3.Row | None] = {} | ||
| eligible: list[StaleSupersessionCandidate] = [] | ||
| for (raw_id, session_id, logical_source_key), rep in stale.items(): | ||
| head = heads[logical_source_key] | ||
| if logical_source_key not in head_row_cache: | ||
| fetched = _raw_revision_rows(source_conn, {head.accepted_raw_id}, allow_missing=True) | ||
| head_row_cache[logical_source_key] = fetched.get(head.accepted_raw_id) | ||
| head_row = head_row_cache[logical_source_key] |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
Cache comment and cache key disagree.
The comment claims one lookup per distinct head raw, but the cache is keyed by logical_source_key. Behaviourally equivalent today (one head per logical source), yet a reader auditing this safety-critical pass has to re-derive that invariant. Keying on head.accepted_raw_id would make the comment literally true and de-duplicate across logical sources that share a head raw.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@polylogue/storage/raw_retention.py` around lines 698 - 707, Update the
head-row cache in the stale-candidate loop to use head.accepted_raw_id as its
key instead of logical_source_key, and adjust the cache lookup and assignment
accordingly. Preserve the existing _raw_revision_rows lookup and head-row
behavior while allowing shared head raws across logical sources to reuse one
cached result.
Comment on lines
+836
to
+842
| try: | ||
| record_revision_application_sync(index_conn, receipt, decided_at_ms=head.decided_at_ms) | ||
| except (RuntimeError, ValueError) as exc: | ||
| errors.append(f"{item.raw_id[:16]}: {exc}") | ||
| else: | ||
| reissued += 1 | ||
| index_conn.commit() |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
sqlite3.Error escapes the per-item guard and discards every successful insert.
Only RuntimeError/ValueError are caught, but record_revision_application_sync executes SQL: a sqlite3.OperationalError (locked index tier, disk I/O) or IntegrityError propagates out of the loop, skips index_conn.commit(), and the caller in polylogue/storage/repair.py (Line 5803) closes the connection — rolling back every receipt already written in this pass, with no StaleSupersessionReissueResult to report what happened. Since the receipts are independent and idempotent by decision_id, per-item isolation is the intended behaviour; widen the guard so a single bad row degrades to an errors entry.
♻️ Proposed fix
try:
record_revision_application_sync(index_conn, receipt, decided_at_ms=head.decided_at_ms)
- except (RuntimeError, ValueError) as exc:
+ except (sqlite3.Error, RuntimeError, ValueError) as exc:
errors.append(f"{item.raw_id[:16]}: {exc}")
else:
reissued += 1Note sqlite3.OperationalError is not a RuntimeError subclass, so ordering here is only for readability.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| try: | |
| record_revision_application_sync(index_conn, receipt, decided_at_ms=head.decided_at_ms) | |
| except (RuntimeError, ValueError) as exc: | |
| errors.append(f"{item.raw_id[:16]}: {exc}") | |
| else: | |
| reissued += 1 | |
| index_conn.commit() | |
| try: | |
| record_revision_application_sync(index_conn, receipt, decided_at_ms=head.decided_at_ms) | |
| except (sqlite3.Error, RuntimeError, ValueError) as exc: | |
| errors.append(f"{item.raw_id[:16]}: {exc}") | |
| else: | |
| reissued += 1 | |
| index_conn.commit() |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@polylogue/storage/raw_retention.py` around lines 836 - 842, In the per-item
guard around record_revision_application_sync, catch sqlite3.Error in addition
to RuntimeError and ValueError so SQL failures are recorded in errors for that
raw_id and processing continues to index_conn.commit(). Preserve the existing
reissued count and error-message behavior for successful and failed receipts.
Comment on lines
+1437
to
+1459
| -- The child's own first user turn: for a subagent this literally IS the | ||
| -- text Claude Code/Codex injected as the dispatch's prompt, not a human | ||
| -- turn (material_origin is 'generated_context_pack', never | ||
| -- 'human_authored' -- see polylogue-1vpm.7 corpus audit), so this | ||
| -- deliberately does not filter on material_origin. | ||
| child_identity_text AS ( | ||
| SELECT | ||
| m.session_id AS child_session_id, | ||
| ( | ||
| SELECT b.text FROM blocks b | ||
| WHERE b.message_id = m.message_id AND b.block_type = 'text' | ||
| ORDER BY b.position LIMIT 1 | ||
| ) AS first_text | ||
| FROM messages m | ||
| WHERE m.role = 'user' | ||
| AND m.message_type = 'message' | ||
| AND m.position = ( | ||
| SELECT MIN(m2.position) FROM messages m2 | ||
| WHERE m2.session_id = m.session_id | ||
| AND m2.role = 'user' | ||
| AND m2.message_type = 'message' | ||
| ) | ||
| ), |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
child_identity_text can fan out on regenerated/edited first-turn variants.
The subquery picks the child's first user messages row by m.position = MIN(...), but messages is keyed (session_id, position, variant_index) and the schema explicitly models regenerated/edited variants at the same position (see idx_messages_active_path partial index on is_active_path = 1, dedicated to selecting the live variant). If the child's first user turn ever has more than one variant (edited-and-resent, or a fork at turn 0), this CTE yields multiple (child_session_id, first_text) rows for the same child instead of one, which can inflate child_match_counts and cause an otherwise-unique content match to be excluded by the cmc.n = 1 uniqueness gate in identity_pairs — silently pushing a resolvable dispatch into unresolved/edge_only.
Restricting to the active variant makes the "first user turn" selection deterministic:
🐛 Proposed fix
child_identity_text AS (
SELECT
m.session_id AS child_session_id,
(
SELECT b.text FROM blocks b
WHERE b.message_id = m.message_id AND b.block_type = 'text'
ORDER BY b.position LIMIT 1
) AS first_text
FROM messages m
WHERE m.role = 'user'
AND m.message_type = 'message'
+ AND m.is_active_path = 1
AND m.position = (
SELECT MIN(m2.position) FROM messages m2
WHERE m2.session_id = m.session_id
AND m2.role = 'user'
AND m2.message_type = 'message'
+ AND m2.is_active_path = 1
)
),📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| -- The child's own first user turn: for a subagent this literally IS the | |
| -- text Claude Code/Codex injected as the dispatch's prompt, not a human | |
| -- turn (material_origin is 'generated_context_pack', never | |
| -- 'human_authored' -- see polylogue-1vpm.7 corpus audit), so this | |
| -- deliberately does not filter on material_origin. | |
| child_identity_text AS ( | |
| SELECT | |
| m.session_id AS child_session_id, | |
| ( | |
| SELECT b.text FROM blocks b | |
| WHERE b.message_id = m.message_id AND b.block_type = 'text' | |
| ORDER BY b.position LIMIT 1 | |
| ) AS first_text | |
| FROM messages m | |
| WHERE m.role = 'user' | |
| AND m.message_type = 'message' | |
| AND m.position = ( | |
| SELECT MIN(m2.position) FROM messages m2 | |
| WHERE m2.session_id = m.session_id | |
| AND m2.role = 'user' | |
| AND m2.message_type = 'message' | |
| ) | |
| ), | |
| -- The child's own first user turn: for a subagent this literally IS the | |
| -- text Claude Code/Codex injected as the dispatch's prompt, not a human | |
| -- turn (material_origin is 'generated_context_pack', never | |
| -- 'human_authored' -- see polylogue-1vpm.7 corpus audit), so this | |
| -- deliberately does not filter on material_origin. | |
| child_identity_text AS ( | |
| SELECT | |
| m.session_id AS child_session_id, | |
| ( | |
| SELECT b.text FROM blocks b | |
| WHERE b.message_id = m.message_id AND b.block_type = 'text' | |
| ORDER BY b.position LIMIT 1 | |
| ) AS first_text | |
| FROM messages m | |
| WHERE m.role = 'user' | |
| AND m.message_type = 'message' | |
| AND m.is_active_path = 1 | |
| AND m.position = ( | |
| SELECT MIN(m2.position) FROM messages m2 | |
| WHERE m2.session_id = m.session_id | |
| AND m2.role = 'user' | |
| AND m2.message_type = 'message' | |
| AND m2.is_active_path = 1 | |
| ) | |
| ), |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@polylogue/storage/sqlite/archive_tiers/index.py` around lines 1437 - 1459,
Update the child_identity_text CTE to select only the live message variant by
adding the is_active_path = 1 predicate to both the outer messages query and its
MIN(position) subquery. Preserve the existing first-user-turn and text-block
selection while ensuring each child_session_id produces at most one identity
row.
Comment on lines
+2018
to
+2019
| with sqlite3.connect(source_db) as source_conn: | ||
| result = reissue_stale_supersession_receipts(source_conn, source_conn, index_db_path=index_db, dry_run=True) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Pass a real index connection so a dry-run regression fails loudly.
source_conn is handed in as index_conn. If the dry_run short-circuit ever regresses, receipts get inserted into source.db — the index.db count assertion still passes and the test stays green while writes land in the wrong tier. Opening a separate index_db connection makes the intended contract explicit and turns that regression into a visible failure.
💚 Proposed change
- with sqlite3.connect(source_db) as source_conn:
- result = reissue_stale_supersession_receipts(source_conn, source_conn, index_db_path=index_db, dry_run=True)
+ with sqlite3.connect(source_db) as source_conn, sqlite3.connect(index_db) as index_conn:
+ result = reissue_stale_supersession_receipts(source_conn, index_conn, index_db_path=index_db, dry_run=True)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| with sqlite3.connect(source_db) as source_conn: | |
| result = reissue_stale_supersession_receipts(source_conn, source_conn, index_db_path=index_db, dry_run=True) | |
| with sqlite3.connect(source_db) as source_conn, sqlite3.connect(index_db) as index_conn: | |
| result = reissue_stale_supersession_receipts(source_conn, index_conn, index_db_path=index_db, dry_run=True) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/unit/storage/test_raw_retention.py` around lines 2018 - 2019, Update
the test invocation of reissue_stale_supersession_receipts to open and pass a
separate SQLite connection to index_db_path as index_conn instead of reusing
source_conn. Keep source_conn for the source database and preserve the dry_run
assertion flow so any unintended write targets the distinct index connection and
fails visibly.
Problem: `devtools test tests/unit/cli/test_import.py -k demo_wait` had two red tests reporting the entire ~40-construct demo corpus at zero (0/19 sessions, 0/31 messages). This looked like the demo seeder itself was broken, but bisecting to b473d92 (#3179, 2026-07-20) showed the production `import --demo --wait` flow changed to run `apply_demo_post_ingest_augmentation` and the real `_verify_demo_now` unconditionally after the wait step (previously `_verify_demo_now` only ran on the `--with-overlays` branch, and the success banner printed a hardcoded `sessions=3 messages=19`). tests/unit/cli/test_import.py was never updated for that call-sequence change, so both tests let the real augmentation/verifier hit the empty per-test archive (urlopen and the daemon wait are mocked; no real ingest ever runs in these unit tests) — one crashed on `no such table: sessions`, the other correctly reported the archive as empty. Also fixed a pre-existing, unrelated staleness in `test_import_demo_materializes_fixture_world_before_daemon_request`: its expected source-directory listing and JSON fixture count predate the gemini-cli/antigravity/hermes demo origins added in #3182/#2674, so it asserted 6 directories where 9 now exist and 4 fixture files where 7 now exist. What changed: - test_import_demo_wait_verifies_after_daemon_acceptance: mock `apply_demo_post_ingest_augmentation` and `_verify_demo_now` (matching the with-overlays test's existing pattern), assert the call sequence daemon -> wait -> augment -> verify, and assert the banner against the fake DemoVerifyResult's counts instead of hitting the real (unseeded) archive. - test_import_demo_wait_with_overlays_seeds_after_convergence: add the missing `apply_demo_post_ingest_augmentation` mock the production code already calls unconditionally; update the expected event order to daemon -> wait-base -> augment -> seed-overlays -> verify-overlays. - test_import_demo_materializes_fixture_world_before_daemon_request: update the expected directory listing (+antigravity, +gemini-cli, +hermes) and fixture-file count (4 -> 7) to match the current demo corpus. This is a test-only fix: no production seeder, daemon, or storage code changed. The declared demo-construct contract (`polylogue/demo/constructs.py`) is untouched and still enforced end-to-end by tests/unit/demo/test_demo_seed_verify.py and tests/integration/test_demo_daemon_convergence.py; nothing here weakens what those constructs require. Verification: - `devtools test tests/unit/cli/test_import.py -k demo_wait` -> 2 passed - `devtools test tests/unit/cli/test_import.py` -> 19 passed - `devtools verify --quick` -> exit 0 Co-Authored-By: Claude <noreply@anthropic.com>
…omote-schemas-and-wire-gates
polylogue/sources/parsers/codex_state.py landed with the Codex SQLite acquisition lane; adding a module under polylogue/ requires regenerating the projection or render all --check fails.
Problem: session_provider_usage_events.payload_json survived only as a projection of 8 Hermes billing-provenance keys (estimated_cost_usd, actual_cost_usd, cost_status, cost_source, pricing_version, billing_provider, billing_base_url, billing_mode) -- 104 of 4,030,168 live rows carry any value -- yet the column still forced a JSON CHECK and a json_extract read path in _reextract_provider_usage_tail_db (polylogue-c3ip follow-up from the v42 session_events slimming audit). What changed: the column is dropped in favor of 8 nullable typed columns on the same table (archive_tiers/index.py), the writer (_provider_usage_event_row) populates them directly instead of building a JSON blob, and _reextract_provider_usage_tail_db's predicates now reference the typed columns instead of json_extract(payload_json, ...). Classification: this delta is, in isolation, clone-safe (CONSTRAINT_ONLY/REPLACE_TABLE -- values already live in the row, backfill is a json_extract of already-persisted data, not a raw reparse). It is folded into v45 rather than given its own version because IndexDeltaDeclaration classifies one version as a whole, and v45 already carries the delegation_facts identity-join SEMANTIC_REPARSE. Declaring v45 partially fast-forwardable would misstate it: every v45 archive goes through `polylogue ops reset --index && polylogued run` regardless of which of the two changes triggered the bump. The CONSTRAINT_ONLY-shaped sub-delta is recorded in prose in lifecycle.py so a future per-object declaration split (polylogue-9rw0.1) can recover it. Verification: - devtools test tests/unit/storage/test_lineage_normalization.py tests/unit/storage/test_provider_usage_report.py tests/unit/storage/test_archive_tiers_assertions.py tests/unit/storage/test_archive_tiers_ddl.py tests/unit/pipeline/test_hermes_raw_identity.py tests/unit/storage/test_schema_policy_contracts.py -> 132 passed - devtools lab policy schema-versioning -> "Schema evolution policy intact." (0 undeclared index schema deltas) - devtools render all --check -> no drift - ruff check / format --check on touched files -> clean - mypy on touched files -> no issues Co-Authored-By: Claude <noreply@anthropic.com>
Problem: polylogue-m6tp item 5 names repair_raw_materialization's per-tick full O(backlog) _raw_materialization_candidate_ids() rescan as deletable. Attempted a generation-counted, explicitly write-path-invalidated cache scoped to this bead's write grant (repair.py, raw_authority.py, revision_application.py, daemon/**). What changed: nothing in the shipped tree -- the cache was built, then reverted, after two tests in tests/unit/storage/test_repair.py (index-tier reset, out-of-band census write) proved it goes stale via writers this bead cannot instrument (sources/live/*, storage/repository/**, storage/sqlite/archive_tiers/archive.py) -- the same observable failure shape the prior PRAGMA data_version attempt hit, reached through a different mechanism. Documents the finding in the inventory doc and files polylogue-iy3n so the next attempt starts from the persistent backlog-iterator design (phase c) instead of re-trying a smaller cache. Verification: git diff is empty against origin/master for every source file; tests/unit/storage/test_repair.py + test_raw_authority_scale_proof.py reconfirmed green (86 passed) on the reverted tree. Ref polylogue-m6tp, polylogue-iy3n
…fidence Follow-up to the SESSION_COLUMNS fix (declaring title_ref/title_confidence in archive/query/discovery.py): docs/search.md is generated from that column declaration and needed a matching regeneration. Verification: devtools render all --check -> no "out of sync" lines.
…d-wire-gates' into worktree-agent-a95a4f5a5eb38cf3c # Conflicts: # tests/unit/storage/test_incremental_rebuild_equivalence.py
…d-wire-gates' into feature/chore/promote-schemas-and-wire-gates
Both are fallout from correct changes landed earlier today, not defects.
docs/internals.md referenced polylogue/storage/block_anchor.py, deleted
2026-07-30 as a two-sided citation-anchor gap (nothing emitted an anchor, so
nothing could resolve one; the wiring is deferred to polylogue-bby.11). The
drift checker flags any backticked path that no longer exists, correctly.
Rewrote the passage to record what actually happened -- the column
blocks.content_hash is live and still written on every block; only the helper
module went -- without naming a path that is gone.
test_operator_inference derived baseline_versions from the BUNDLED catalog and
then asserted them in a listing read from the workspace-ISOLATED registry.
That passed only because write_schema_version() silently inherited bundled
versions into an isolated registry -- a real isolation leak fixed earlier
today. With the leak closed, an isolated registry contains exactly what was
written to it, so the listing is [v3, v4] and not [v1, v2, v3, v4]. The
baseline read stays: it derives non-colliding version NAMES, which is why the
test survives future provider promotions.
Suite: 207 -> 72 -> 39 -> 2 -> 0 failures over the branch.
Verification
pytest tests/unit/devtools/test_verify_docs_drift.py 12 passed
pytest tests/unit/core/test_operator_inference.py 5 passed
mypy + ruff clean
Sinity
changed the title
Sinity
added a commit
that referenced
this pull request
Jul 31, 2026
o4j2's runSettings-storage scope predates this bead (PR #3390); the remaining pendingInputs gap is fixed on #3415. AC2 (query-DSL numeric predicates over run_settings) is genuinely deferred -- filed as polylogue-mgf6 with the two concrete blockers (grammar INT-only literals, SQL-builder plain-column assumption). Ref polylogue-o4j2 Ref polylogue-mgf6
This was referenced Jul 31, 2026
Sinity
added a commit
that referenced
this pull request
Jul 31, 2026
Summary: parses AI Studio's chunkedPrompt.pendingInputs (unsent textbox drafts) into ParsedSession.pending_drafts, deliberately kept outside session_revision_projection's identity/comparison axes (polylogue-aggz Invariant 1) after a reviewer traced a P1 on the original draft_input session_event design -- drafts are mutable, and session_events feed append-only revision comparison, so editing/submitting a draft would misclassify revision membership (same defect class as polylogue-bu1i / polylogue-nuec). Also documents that the bead's runSettings claim was already stale (fixed by PR #3390) and files polylogue-j8yo (browser-capture SKIP verdict, with live CDP-Network evidence) and polylogue-mgf6 (query-DSL follow-up). Verification: devtools test (parser + storage round-trip + revision membership suites) all green; mypy --strict clean; devtools verify --quick 19/19 steps ok; devtools lab policy schema-versioning clean (index v47 delta declared). Ref polylogue-o4j2 Ref polylogue-j8yo Ref polylogue-mgf6
This was referenced Jul 31, 2026
Sinity
added a commit
that referenced
this pull request
Jul 31, 2026
…3431) ## Summary Closes the two named, still-open gaps from tonight's polylogue-pbuh and polylogue-cijx.4 investigations: (1) verifies and finishes the read-side wiring for typed session→PR evidence (pbuh AC4), and (2) wires the `root:` session-structure filter end-to-end across the query DSL, CLI, and Python API (cijx.4 AC4 / its follow-up polylogue-oqib). ## Problem **Gap 1 (pbuh AC4).** polylogue-pbuh's parser-side fix (index v46, PR #3390) persists Claude Code's `pr-link` sidecar record as typed `session_refs` evidence, but nothing on the CLI/insights/MCP surface read it — the four dependent beads (212.2/xyel/kph/fs1.4) were blocked on an inference mechanism (`session_commits`, 0 readers, 2,989 rows of a narrower fact) instead of the typed evidence the provider already supplies. **Gap 2 (cijx.4 AC4 / oqib).** `sessions.parent_session_id` and `Session.is_root` are correct, and a plan-level `root: bool | None` field plus `.is_root()` builder already existed, but `root` had no `spec_attr`, no DSL grammar case, and no CLI flag — completely unreachable from any query surface, so a default `find` mixes 66.1% root sessions with 33.9% subagent/branch children unlabeled. ## Solution **Gap 1**: A sibling lane's PR #3425 (fix/insights/session-commit-typed-evidence) had landed the actual read-side fix — `build_correlation_result` now consumes `session_refs`/`claude_bridge_session` typed evidence as authoritative, falling back to regex/time-window heuristics only where no typed evidence exists, and surfacing disagreements. It was open but unmerged when this pass started; triaged its 3 non-blocking CodeRabbit P2 findings (filed as follow-up polylogue-2vor) and merged it (5525446). Verifying the now-merged surface against the live archive (`find id:<session> then read --view correlation`) surfaced a **second, independent, pre-existing bug**: `_enrich_with_github_api` (`polylogue/insights/correlation_view.py`) referenced `SessionCorrelationResult` at runtime while only importing it under `TYPE_CHECKING` — every call with the default `github_api=True` and any issue/PR ref present raised `NameError`. This predates PR #3425 (present since `ac84f734f`); the existing test suite only exercised `github_api=False`, so it was never caught. Fixed by importing the class at runtime alongside the existing `GitHubRef` import, plus a regression test. **Gap 2**: - `SessionQuerySpec.root: bool | None`, wired through `build_query_spec_from_params`/`query_spec_to_plan` (new `optional_bool` tri-state parser in `archive/query/spec.py`). - DSL: `root:true`/`root:false` field clause (`archive/query/expression.py`); `-root:` negation is rejected pointing at `root:false` instead (the value already carries polarity). - CLI: `--root/--no-root` flag, added last in `cli()`'s signature per this repo's "new Click params go last" convention. - `EXPRESSION_FIELD_REGISTRY["root"]` + regenerated `docs/cli-reference.md`, `docs/search.md`. Two deeper bugs found while making this reachable (neither was "just unreachable" — both were silent no-ops even where a plan/spec did carry a value): 1. The CLI's actual browse/search path (`cli/archive_query.py`'s `_ArchiveFilterKwargs` → `ArchiveStore.list_summaries`/`search_summaries`/ `count_sessions`/`count_search_sessions`/`search_session_ids`/ `semantic_summaries`/`stats`/`stats_by`) is a SQL-level filter path entirely separate from the `SessionQueryPlan`/`apply_common_filters` post-filter machinery `root`'s field descriptor (`requires_post_filter=True`) was designed against. None of those eight `ArchiveStore` methods accepted a `root` kwarg. Fixed by pushing `root` into `_session_filter_clause` as a direct SQL predicate (`sessions.parent_session_id IS [NOT] NULL` — trivially SQL-pushable, unlike `continuation`/`sidechain` which derive from `branch_type`) and threading it through all eight methods. 2. Even the `SessionQueryPlan` post-filter path (Python API's `list_summaries_archive`/`list_archive`) was independently broken: `ArchiveSessionSummary` never carried `parent_id` (the SELECT never projected `sessions.parent_session_id`, `_summary_from_row` never read it), so `is_root` was `True` for every summary row regardless of actual parent — a `root:true` filter would have silently returned everything even once reachable. Fixed by adding `parent_id` to `ArchiveSessionSummary`, projecting the column in both `read_summary` and `list_summaries`, and threading it through `_summary_to_domain`. **Default-behavior decision**: did **not** flip any surface's default. `find`/Python API `list()`/MCP query/daemon HTTP all continue to return every session unless `root:`/`--root`/`.is_root()` is given explicitly. Flipping even the narrower "CLI `find` verb only" option still has real blast radius (every existing test/saved-query/demo-script assuming today's "everything" default needs re-auditing), and reachability is the load-bearing wedge — `polylogue find repo:polylogue root:true` now produces the named, non-fanout view AC4's proof text asks for. The default question is left open as a deliberate, separately-reviewable follow-up (narrowed onto polylogue-oqib). ## AC disposition **polylogue-pbuh** (typed session→PR evidence): AC4 **satisfied**. Typed session→PR linkage is reachable from `read --view correlation` (CLI/API), verified live against the real archive. AC6 (before/after UUID-title/PR-link census) is untouched — out of this gap's declared scope, still open. **polylogue-cijx.1** (repo-identity bead whose notes tracked the session-commits/pr-link consumer question): the specific blocking concern its notes raised for 212.2/xyel/kph/fs1.4 ("the producer does not work" / "0 readers") is resolved. Its own titled AC (repo_id fragmentation) is unrelated and was already addressed by cijx.4. **212.2 / xyel / kph / fs1.4**: unblocked, not closed — each still needs its own concrete deliverable (demo build, CI hook, CLI/report regen) beyond "the data is now readable." Noted individually on each bead. **polylogue-cijx.4** AC4: reachability **satisfied**. Default-behavior question **deferred**, narrowed onto polylogue-oqib (priority lowered — remaining scope is a design decision, not plumbing). **polylogue-oqib**: reachability + both deeper bugs **satisfied and fixed**. Default-behavior flip **not attempted**, left open and explicitly narrowed to that one remaining decision. ## Verification - `devtools test tests/unit/cli/test_correlate_view.py` — 4 passed (including new NameError regression test). - `devtools test tests/unit/cli/test_query_expression.py tests/unit/core/test_query_fields.py tests/unit/cli/test_archive_query.py tests/unit/archive/test_archive_execution_filters.py tests/unit/cli/test_query_exec_laws.py tests/unit/archive/ tests/unit/storage/test_archive_tiers_archive.py tests/unit/archive/query/test_discovery.py` — all green (new `test_root_filter_partitions_top_level_and_subagent_sessions` covers all eight `ArchiveStore` methods plus the `parent_id`/`is_root` wiring bug). - `mypy --strict` on every touched module — no issues. - `devtools render all --check` — sync OK. - `devtools verify --quick` — exit 0 (also ran automatically via the pre-push hook). - Manual live verification (read-only, `/realm/db/polylogue/index.db`): `find id:<session> then read --view correlation --format json` returns typed `pr_refs` (`source=typed_session_ref`) plus a `disagreements` list; `find repo:polylogue --root` → total 1906; `find repo:polylogue --no-root` → total 3206; 1906+3206=5112, the unfiltered total. ## Not done / follow-ups - polylogue-2vor: 3 CodeRabbit P2 findings on PR #3425's typed-evidence path (PR #0 coercion, cross-repo number collision, foreign-trailer false-disagreement). - polylogue-oqib: the default-behavior decision for `root:` (which surfaces, if any, should default to top-level-only). - `root` is not wired into `query_unit_session_filters` (the `with <units>` projection's separate session-filter adapter), and daemon HTTP's `_build_query_spec_params` named-param allowlist has no dedicated `?root=` query param (the existing `?query=root:true` DSL path already covers it). `continuation`/`sidechain`/`has_branches` remain exactly as unreachable as before this PR. Ref polylogue-pbuh, polylogue-cijx.1, polylogue-cijx.4, polylogue-oqib. --------- Co-authored-by: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Jul 31, 2026
… label (#3421) ## Summary Fixes the one concrete production bug found while auditing polylogue-cijx.4 ("Repo identity, path normalization and readable labels are ONE batch"): the session structural-label projection (decision 3 of that bead) was wired into the summary read path but never actually fired, because the "does this session have a real title" check only asked whether `sessions.title` was non-blank — not whether it came from a real source. ## Problem `polylogue-cijx.4` decisions 1-3 (repo identity keyed on the normalized remote, repo-relative paths, and a read-time structural-label projection) turned out to already be substantially landed on `master` before this lane started — they shipped as part of PR #3390's larger diff (commit `5e23e6abf`, "index v46 wire-evidence batch"), with real test coverage (`tests/unit/archive/test_repo_identity.py`, `tests/unit/insights/test_session_label.py`). Auditing that existing implementation against a read-only copy of the live archive (`/realm/db/polylogue/index.db`) found the label was dead in production: `_summary_from_row` in `storage/sqlite/archive_tiers/archive.py` treated any non-blank `sessions.title` as a genuine provider title. But Claude Code's parser (`sources/parsers/claude/code_parser.py`, out of this lane's write scope — owned by the polylogue-pbuh lane) initializes `title` to the raw composed session id (a bare UUID, or `"<uuid>:agent-<hash>"` for a subagent) and only promotes `title_source` off `UNKNOWN` when a real signal (human message, `agent-name`, `ai-title`, `custom-title`) is found. So a `title_source='unknown'` row still carries a non-blank title — exactly the raw-id echo the bead's motivating text complains about ("`agent-ad682bc849a1cd0f0 - 27f - 499m` — worse than the UUID it replaces"). Measured live, read-only: **7,501 of 15,401 root sessions (48.7%)** carry `title_source='unknown'`. The structural-label fallback never fired for any of them. ## Solution `_summary_from_row` now only treats a non-blank `sessions.title` as a real title when `title_source` is `origin`, `heuristic`, or `user`. `unknown` (and the label's own prior `path` output, for rebuild idempotency) fall through to `session_structural_label_for_session`. Modules touched: `polylogue/storage/sqlite/archive_tiers/archive.py` (the fix), `tests/unit/storage/test_title_source_queryable.py` (new regression test reproducing the exact Claude Code raw-id-fallback shape), `.beads/issues.jsonl` (AC disposition + a new follow-up bead). **AC4 of polylogue-cijx.4 (default result unit = top-level session) is explicitly NOT addressed here** — investigated and found to require a separate, higher-blast-radius change (Lark DSL grammar wiring for a `root:` field, which currently doesn't exist on any query surface despite the underlying `Session.is_root`/`parent_session_id` plumbing already being correct, plus a default-behavior decision affecting every unfiltered `find`/`list()`/MCP `query` call). Split out as `polylogue-oqib` rather than folded into this diff. Full AC1-5 disposition recorded as a comment on `polylogue-cijx.4`. Not a `polylogue-pbuh` overlap: pbuh's typed `ai-title`/`agent-name`/ `pr-link` sidecar-record work is a different, complementary fix (making `title_source` legitimately `origin` more often) — this PR fixes the read path's *consumption* of `title_source`, regardless of how it got there. ## Verification ``` devtools test tests/unit/storage/test_title_source_queryable.py tests/unit/insights/test_session_label.py tests/unit/archive/test_repo_identity.py # 29 passed in ~19s devtools test tests/unit/archive/ tests/unit/cli/ tests/unit/mcp/ tests/unit/api/ -k title # 21 passed devtools verify --quick # 19 steps, exit 0 (ruff format/check, mypy --strict, render all --check, topology/layering/closure-matrix, schema policy/roundtrip) ``` Live collision-rate re-measurement (AC5 of polylogue-cijx.4), read-only against `/realm/db/polylogue/index.db` (15,401 root sessions), after this fix: - Among sessions with resolved file-touch evidence (a dominant repo-relative path — the population the bead's original 3.5%/max-10 baseline was measured against): **3.28% collision (78/2,377), max group 37**. - Raw collision across all 13,219 title-less sessions: 76.46%, but that figure is dominated by a 5,233-session cluster of genuinely evidence-free (zero-message, no repo, no file touch) sessions collapsing to the honest label `"0 msgs"` — not a labeling defect. Full numbers and a note about whether 5,233 zero-message root sessions is itself a data-quality question are on the `polylogue-cijx.4` comment. Not run: the full non-quick `devtools verify --all` (testmon wasn't seeded on this worktree; ran the exact touched-file selection instead per repo convention). Ref polylogue-cijx.4
Sinity
added a commit
that referenced
this pull request
Jul 31, 2026
…#3419) ## Summary Closes the remaining measured gap in polylogue-pbuh ("Claude Code sidecar records are discarded at parse: 1,172,890 records"): per-type coverage reporting (AC5), plus honest bd bookkeeping on what was already done, what this pass did, and what is still open. ## Problem polylogue-pbuh listed 6 acceptance criteria. Auditing the bead against the current codebase (not re-deriving it from scratch, per the bead's own method note) found that AC1 ("classify every skip type"), AC2 ("persist typed evidence for ai-title/agent-name/pr-link/bridge-session/ file-history-snapshot"), and AC3 ("titles/agent names reach read surfaces") were already fully satisfied by PR #3390 ("index v46 wire-evidence batch"), already merged to master before this pass started. Re-implementing that work would have been redundant and risked silently regressing it. What PR #3390 did NOT do: AC5 (per-type coverage reporting: seen vs. parsed vs. persisted, so a future silent skip is visible) had no implementation anywhere in `sources/parsers/claude/` or `devtools/`. ## Solution `polylogue/sources/parsers/claude/code_parser.py`: - `_parse_code_records` now counts, per skipped sidecar record type, how many records were **seen** vs. how many actually turned into **persisted** evidence (a `session_event`, a `session_ref`, a title override, or a delegation edge) -- these can genuinely diverge, e.g. a `bash_progress` tick under `progress` is seen but never persisted (see the classification comment above `_SKIPPED_SIDECAR_RECORD_TYPES`). - It also samples record types that reached ordinary message parsing but carried no text/blocks and were dropped there (`empty_dropped_by_record_type`) -- the pre-#1617 failure mode the bead's method note warns against repeating by assumption. - One bounded `claude_parse_coverage` `session_event` is emitted per session when either counter is non-empty (no event for the common no-sidecar-activity case, so this doesn't bloat every session). - `session_events.event_type` has no CHECK-constrained vocabulary (`storage/sqlite/archive_tiers/index.py`), so this is additive data, not a schema change -- no migration, no index bump. - `_accumulate_delegation_progress` now returns whether a record folded into a genuine dispatch edge, feeding the persisted counter for `progress`. `tests/unit/sources/test_claude_code_sidecar_evidence.py`: - Added a `_typed_events()` helper so the 9 pre-existing exact-event-list assertions look through the new coverage event instead of hard-coding it into every one (it's orthogonal to what each of those tests actually pins). - Two new tests: `test_parse_coverage_event_reports_seen_and_persisted_counts` (pins the seen/persisted divergence on a real example) and `test_parse_coverage_event_absent_when_only_ordinary_messages_parsed` (pins the no-bloat case). **Not done in this pass, recorded honestly on the bead instead of claimed:** - **AC4** (pr-link becomes the session->PR producer, four consumer beads unblocked/re-scoped): the producer is real (`session_refs` table, `storage/sqlite/queries/session_refs.py`) via PR #3390, but nothing on the CLI/insights/MCP surface reads `session_refs` yet, so `polylogue-cijx.1` and its four dependents (`212.2`/`xyel`/`kph`/`fs1.4`) are not actually unblocked. Noted on both `polylogue-pbuh` and `polylogue-cijx.1` with the specific gap (reader, not producer). Consumer wiring is insights/CLI/MCP territory, outside this pass's declared surface (`sources/parsers/claude/`, `assembly_claude_code.py`, `providers/claude_code*.py`). - **AC6** (reprocess existing raws, report before/after UUID-title/PR-link census): PR #3390's body recorded *expected* post-rebuild numbers, not an actual measured before/after census, and whether the v46 SEMANTIC_REPARSE rebuild has actually run against the real corpus since merge is an operational fact about the live archive, unverifiable from a sandboxed worktree. Recorded on the bead as an explicit open item for whoever has archive access next. No inflation risk: this shape (typed `session_events`/`session_refs`, one row per real fact, plus one bounded coverage event per session) does not add sessions or messages -- the same producer/consumer shape PR #3390 already established and that this PR only extends with counters. ## Verification ``` devtools test tests/unit/sources/test_claude_code_sidecar_evidence.py \ tests/unit/sources/test_parsers_claude_code_artifacts.py \ tests/unit/sources/test_claude_code_unread_wire_fields.py \ tests/unit/sources/test_compaction.py \ tests/unit/sources/test_dispatch_payloads.py \ tests/unit/sources/test_parsed_session_typed_context.py \ tests/unit/sources/test_parser_crashlessness.py \ tests/unit/sources/test_parsers_base.py \ tests/unit/sources/test_source_laws.py \ tests/unit/sources/test_tool_result_sidecars.py \ tests/unit/sources/test_parsers_props.py # -> 397 passed devtools verify --quick # -> exit_code 0 (ruff format/check, mypy --strict, render all --check, # layering, closure-matrix, schema-versioning, doc-commands, docs-coverage, # test-clock-hygiene, hash-boundary-census all green) ``` Ref polylogue-pbuh --------- Co-authored-by: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Jul 31, 2026
…rofiles (#3492) ## Summary Bounded/degraded large-session profiles now compute a real `terminal_state` from a bounded tail read, instead of hardcoding `"unknown"` unconditionally. ## Problem Live evidence (triage-verified 2026-07-31): all 1,575 `bounded_large_session` profiles carried `terminal_state="unknown"` (100%) — `build_large_session_insight_record_bundle_{sync,async}` in `storage/insights/session/rebuild.py` hardcoded `terminal_state="unknown"`, `terminal_state_confidence=0.0`, `terminal_state_method="bounded_materialization"` unconditionally, regardless of any structural evidence in the session. Message `stop_reason` is now persisted (index v46, PR #3390), but the bounded profile path never read the tail data needed to derive terminal state at all. This left the longest, most failure-prone sessions (message-count decile 9) terminal-state-blind, gating campaign D3 in the annotation launch order (per polylogue-wofr notes). `_terminal_state` (`archive/session/runtime.py`) is structurally O(session tail): it only needs the last message, last tool outcomes, and trailing session events — not a full-session scan — so excluding it from the bounded path was unnecessary caution, not a real cost tradeoff. ## Solution - Added a bounded tail read (`_tail_session_sync`/`_tail_session_async` in `polylogue/storage/insights/session/rebuild.py`): last 50 messages + their non-text (`tool_use`/`tool_result`) blocks + last 200 trailing session events for one session, via new `_SESSION_INSIGHT_TAIL_MESSAGE_SQL`/`_SESSION_INSIGHT_TAIL_BLOCK_SQL_TEMPLATE`/`_SESSION_INSIGHT_TAIL_EVENT_SQL` queries (each LIMIT-bounded, ordered ascending via a subquery so no Python-side reversal is needed). - These hydrate a minimal in-memory `Session` (`session_from_records`) for just that tail window. - `_bounded_session_terminal_state_{sync,async}` then call `build_session_analysis` + the exact same `_terminal_state` function (`archive/session/runtime.py`) the unbounded profile path already uses — imported directly, not reimplemented, per the existing precedent of importing private runtime helpers into rebuild.py (e.g. `_primary_model`). - `_large_session_profile_record_from_row` now accepts an optional `terminal_state_result` and threads the real `(terminal_state, confidence, evidence, method)` into the evidence payload, inference payload, and stored `SessionProfileRecord` fields. The `bounded_materialization` provenance marker is kept in `terminal_state_evidence` alongside the real structural evidence, so a reader can still tell the record came from the degraded path. - All other bounded/degraded reductions (workflow_shape, cost, work events, phases) are unchanged — this is scoped strictly to `terminal_state`. A session whose real terminal event falls outside the tail window degrades to whatever `_terminal_state` reports from the window it was given (typically `"unknown"`/`"no_signal"`) — the same honest behavior as before for sessions genuinely lacking structural evidence in scope. ## Verification - `python -m devtools test tests/unit/storage/test_session_insight_refresh.py` — 32 passed, including two new tests: - `test_large_session_rebuild_derives_terminal_state_from_bounded_tail`: a bounded-profile fixture whose tail message's final tool outcome is a typed error asserts `terminal_state="error_left"`/`method="action_outcome"` via the bounded path with `load_sync_batch` patched to raise (proving no full hydration). Anti-vacuity: manually reverted the derivation to the old hardcoded `"unknown"` and confirmed both this test and the parity test below fail (`assert 'unknown' == 'error_left'`), then restored the fix. - `test_bounded_and_unbounded_terminal_state_agree_on_shared_fixture`: the same fixture (2 messages, well within the 50-message tail window) materialized once through the ordinary full-analysis path and once through the bounded/degraded path (threshold monkeypatched to 1) derives identical `terminal_state`/`terminal_state_method`/`terminal_state_confidence` via both routes. - `python -m devtools verify --quick` — exit 0 (format, lint, `mypy --strict`, `render all --check`, layering, schema-versioning, and other policy gates). - `python -m mypy polylogue/storage/insights/session/rebuild.py` — Success: no issues found in 1 source file. Not run: `devtools verify --all` / the full non-integration suite (out of scope for this focused fix; the touched surface's own test file — 32 tests — is green, and per-PR CI runs `lint` + skips the heavy `test` suite until merge). ## Acceptance criteria (polylogue-wofr) 1. Synthetic heavy session with a typed tool error at the tail → `terminal_state="error_left"` with evidence, via the bounded rebuild path. **Satisfied** (async twin shares the same helper functions, exercised by the existing async bounded-path tests plus manual code-path symmetry — no separate async-specific terminal-state test was added since `_bounded_session_terminal_state_async` is a direct twin of the sync version with identical logic). 2. Clean-tail heavy session yields the same terminal_state as its unbounded equivalent. **Satisfied** via the parity test. 3. Bounded-path cost stays O(tail): the existing `fail_full_load`/`guarded_load` monkeypatches (asserting `load_sync_batch`/`load_async_batch` are never called) continue to pass, and the new tail queries are separate LIMIT-bounded SQL, not the full per-session batch loader. No new explicit row-count-budget assertion was added beyond the existing "must not call the full loader" guard — deferring a dedicated cost-pinning test as a possible follow-up if live telemetry ever shows otherwise. 4. Existing bounded-profile tests stay green. **Satisfied** — all 32 tests in the file pass, none modified beyond the two additions. Ref polylogue-wofr Co-authored-by: Claude <noreply@anthropic.com>
This was referenced Aug 1, 2026
Sinity
added a commit
that referenced
this pull request
Aug 2, 2026
Investigated docs/design/convergence-simplification-inventory.md's remaining "deletable once X" items now that both preconditions (3.14t free-threaded deploy, daemon_parse_stage_split flag removal) hold. Item 1 (process-pool machinery) and item 3 (64 MiB daemon blob limit) both looked deletable on the doc's own terms but are STILL-NEEDED: process_pool.py has three live unconditional call sites the doc never described (ingest_batch/_core.py, validation_flow.py, archive_ingest.py), and the blob limit now feeds polylogue-t93b's whale-pass escalation tiering, a capability added after the doc was written. Item 6 (rebuild-index CLI) is already covered by rpuqn/mkk0/4jsk. Items 4/5 were already resolved by the doc itself. Filed polylogue-gzyqk (update the two stale doc items) and polylogue-btv32 (unresolved merge-conflict marker found at convergence-simplification-inventory.md:288, landed via 5e23e6a/#3390) as discovered-from follow-ups. No code changed; find-and-classify only. Co-Authored-By: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Aug 2, 2026
Problem: two doc-accuracy bugs found in docs/design/convergence-simplification-inventory.md during the polylogue-iiu6r "automatic path already covers this" audit. 1. A literal unresolved diff3 merge-conflict marker (`||||||| b64a074`) landed via the 268-commit squash merge 5e23e6a (#3390), sitting between two duplicate "What it is" paragraphs in item 5's history section (the newer paragraph restates the same finding with current line numbers, the marker-adjacent one is the stale ancestor text). 2. Items 1 and 3 described process-pool machinery and the daemon's 64 MiB blob-limit constant as "deletable once X lands," and X (the 3.14t free-threaded deploy, and daemon_parse_stage_split's removal) has now landed -- but re-checking the current tree shows both items remain load-bearing for reasons the doc's original text didn't anticipate. What changed: - Removed the conflict marker and the duplicate stale-ancestor paragraph in item 5, keeping the more detailed, currently-accurate text. - Item 1: added a status note recording that only the one call site the row names (_parse_unique_retained_raws in revision_backfill.py) was retired. Three other unconditional call sites remain -- ingest_batch/_core.py:1053, validation_flow.py:185 (measured Threads(24)=160MB/s vs Process(8)=605MB/s, a 3.7x win independent of the GIL/free-threaded argument), and archive_ingest.py:279 -- none gated on parallel_threads_effective(), so the free-threaded deploy does not collapse them. - Item 3: added a status note recording that raw_materialization_whale_pass_candidate (polylogue-t93b, added after this row was written) now threads _RAW_MATERIALIZATION_DAEMON_BLOB_LIMIT_BYTES through as ordinary_max_payload_bytes, the boundary distinguishing the daemon's ordinary trickle envelope from its escalation-tier whale pass -- a capability unrelated to DaemonParseStage's in-flight budget. The original design-time reasoning under each item is kept for history, with the new status notes marking it as superseded rather than deleting it, matching item 5's existing convention for documenting a failed or outdated deletion attempt in place. Verification: devtools render all --check (exit 0, no "out of sync" lines); devtools verify --quick (exit 0, all steps pass). Ref polylogue-gzyqk, polylogue-btv32 Co-Authored-By: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Aug 2, 2026
…e inventory (#3578) ## Summary Two doc-accuracy fixes to `docs/design/convergence-simplification-inventory.md`, both surfaced by the polylogue-iiu6r "automatic path already covers this, manual surface never deleted" audit. ## Problem 1. A literal unresolved diff3 merge-conflict marker (`||||||| b64a074`) landed in item 5's history section via the 268-commit squash merge 5e23e6a (#3390) -- a duplicate, stale-ancestor "What it is" paragraph sat between the marker and the rest of the section. 2. Items 1 and 3 described process-pool machinery and the daemon's 64 MiB blob-limit constant as "deletable once X lands." X has now landed (the 3.14t free-threaded deploy; `daemon_parse_stage_split`'s removal), but re-checking the current tree shows both items remain load-bearing for reasons the original text didn't anticipate. ## Solution - Removed the conflict marker and the duplicate stale paragraph in item 5, keeping the more detailed, currently-line-numbered text that follows it. - Item 1 (process-pool machinery): added a status note. Only the one call site the row names (`_parse_unique_retained_raws` in `revision_backfill.py`) was retired. Three other unconditional call sites remain -- `pipeline/services/ingest_batch/_core.py:1053`, `pipeline/services/validation_flow.py:185` (measured `Threads(24)=160MB/s` vs `Process(8)=605MB/s`, a 3.7x win independent of the GIL/free-threaded argument), and `pipeline/services/archive_ingest.py:279` -- none gated on `parallel_threads_effective()`, so the free-threaded deploy does not collapse them. - Item 3 (64 MiB daemon blob-limit constant): added a status note. `raw_materialization_whale_pass_candidate` (polylogue-t93b, added after this row was written) now threads `_RAW_MATERIALIZATION_DAEMON_BLOB_LIMIT_BYTES` through as `ordinary_max_payload_bytes`, the boundary distinguishing the daemon's ordinary trickle envelope from its escalation-tier whale pass -- a capability unrelated to `DaemonParseStage`'s in-flight budget. The original design-time reasoning under each item is kept for history (matching item 5's existing convention for documenting a superseded/failed deletion attempt in place), with new status notes marking it stale rather than deleting it. ## Verification - `devtools render all --check` -- exit 0, no "out of sync" lines. - `devtools verify --quick` -- exit 0, all 19 steps pass (also ran automatically via the pre-push hook). Ref polylogue-gzyqk, polylogue-btv32
Sinity
added a commit
that referenced
this pull request
Aug 2, 2026
Problem: polylogue-wkc6 (census-plan bookkeeping 89% of source.db, growing ~1GB/day) overlapped in scope with the in-flight raw-authority redesign (polylogue-lb39z Phase 1, polylogue-w6hql Phase 2), so it needed an investigation to determine whether it was already covered or needed its own action. What changed: verified live against /realm/db/polylogue that the bead's urgent retention ask already shipped in a prior session (PR #3390 wires prune_raw_authority_census_history into the census-record path; PR #3530 adds _delete_orphaned_raw_authority_plans). Current live state: raw_authority_census_plans/post_plans steady at 761,602 rows each (down from 6,621,562/6,621,527 on 2026-07-29), source.db shrunk 6.6GB->1.48GB, freelist_count=0. Closed wkc6 with that evidence. Its two remaining architectural DO items (stop recording carried_forward rows at all; decide whether census bookkeeping belongs in the durable vs disposable tier) are real but non-urgent and distinct from w6hql's "collapse verdict vocabulary" scope, so they're split into a new bead (polylogue-ubdxf) rather than lost. Cross-referenced the finding on w6hql's notes. No source code changed this session -- this was a verification-only investigation confirming a previous fix already landed and is holding. Co-Authored-By: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Aug 2, 2026
Problem: polylogue-wkc6 (census-plan bookkeeping 89% of source.db, growing ~1GB/day) overlapped in scope with the in-flight raw-authority redesign (polylogue-lb39z Phase 1, polylogue-w6hql Phase 2), so it needed an investigation to determine whether it was already covered or needed its own action. What changed: verified live against /realm/db/polylogue that the bead's urgent retention ask already shipped in a prior session (PR #3390 wires prune_raw_authority_census_history into the census-record path; PR #3530 adds _delete_orphaned_raw_authority_plans). Current live state: raw_authority_census_plans/post_plans steady at 761,602 rows each (down from 6,621,562/6,621,527 on 2026-07-29), source.db shrunk 6.6GB->1.48GB, freelist_count=0. Closed wkc6 with that evidence. Its two remaining architectural DO items (stop recording carried_forward rows at all; decide whether census bookkeeping belongs in the durable vs disposable tier) are real but non-urgent and distinct from w6hql's "collapse verdict vocabulary" scope, so they're split into a new bead (polylogue-ubdxf) rather than lost. Cross-referenced the finding on w6hql's notes. No source code changed this session -- this was a verification-only investigation confirming a previous fix already landed and is holding. Co-Authored-By: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Aug 2, 2026
## Summary Investigation bead polylogue-wkc6 (census-plan bookkeeping at 89% of source.db, growing ~1GB/day) is closed: its urgent retention ask already shipped in a prior session and is verified holding live. Remaining architectural items split into a new bead so they aren't lost. ## Problem wkc6 overlapped in scope with the in-flight raw-authority redesign (polylogue-lb39z Phase 1, polylogue-w6hql Phase 2), so it needed investigation to determine whether it was fully covered by that track or needed its own action. ## Solution No source code changed. Verified live against /realm/db/polylogue that PR #3390 (wires `prune_raw_authority_census_history` into the census-record path) and PR #3530 (`_delete_orphaned_raw_authority_plans`) already landed and are holding: `raw_authority_census_plans`/`_post_plans` steady at 761,602 rows each (down from 6,621,562/6,621,527 measured 2026-07-29), `source.db` shrunk 6.6GB -> 1.48GB, `PRAGMA freelist_count`=0. Closed wkc6 with that evidence. Its two remaining architectural DO items (stop recording carried_forward rows at all; decide durable vs disposable tier placement) are real but non-urgent, and distinct from w6hql's "collapse the verdict vocabulary to one closed enum" scope -- split into polylogue-ubdxf and cross-referenced on w6hql's notes. ## Verification - `bd show polylogue-wkc6 --json` / `bd show polylogue-ubdxf --json` / `bd show polylogue-w6hql --json` reflect the new state. - Live query against `/realm/db/polylogue/source.db` (read-only): census/plan/post-plan row counts and file size as stated above. - `gh pr view 3390/3530 --json state,mergedAt` confirm both MERGED.
Sinity
added a commit
that referenced
this pull request
Aug 3, 2026
…dence notes zoek0 closed: the 'unconditional routing never ran' evidence window (Jul 22-29) predates PR #3390 (Jul 30 01:31) which actually made the routing unconditional; the running daemons were flag-gated builds. mkk0 note: equivalence AC should become coverage-census gating. lkrc note: live census shows 7,200 logical sources / 30 GiB unindexed, 4,157 unresolved quarantine blockers from Jul 31, all mechanisms silent. Report: /realm/data/derived/reports/polylogue-convergence-redesign-2026-08-03.html Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lo4gGibHP94JeF62vivvwA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A pre-reindex batch: everything that had to land before a full index rebuild is worth running. Four independent defects, each found by measuring the live archive rather than reading code — plus the schema promotion the rebuild will consume, and the two provider-parser fixes that make the rebuild read data it currently discards.
Problem
Measured on the live archive (
/realm/db/polylogue, 41,363 raws / 92.22 GiB of payload):promote()retired the pointer into a marker but left the superseded ~35 GB directory forever. Nine had accumulated: 370 GB total, ~262 GB reclaimable. One 218 GB directory was an operator hand-quarantine — the leak had been hit and worked around by hand.raw_sessions+blob_refs), 99.98% of rows recordingcarried_forward— that nothing happened — growing ~990 MB/day with noDELETEanywhere in the tree._merge_observed_structure_pairnever readanyOf/oneOf/allOf, so merging a nullable object against a plain one discarded every property in the branches. Merging was not a union: adding an input could remove names.Separately, the committed provider schemas were 134 days old while a 2026-07-16 inference run sat unpromoted, and its promotion had been written to
versions/*/package.jsonwithout regeneratingcatalog.json— the resolution authority — leaving it inert.Solution
>=3.14does not imply free-threaded, and GIL parse threads inflate a concurrent writer's commit latency ~5000×.process_pool_executoris untouched —validation_flowmeasured 605 MB/s against 160 MB/s for threads.prune_superseded_generationson promotion, failing closed on active/promoting/unreadable and pruning nothing when the pointer will not resolve.prune_raw_authority_census_historyinside the recording transaction, two windows: per-plan rows keep 8 censuses (they serve one per-census inspection pager), headers keep 256 (they carry convergence history, and are bounded too sinceresidual_jsonaverages 144 KB).replace_provider_packagesso packages and catalog cannot drift apart. Acatalog_incoherentblocker in the promotion audit makes that class of drift impossible.ai-titlenow resolves titles. Browser-capture blocks channel — plus thetool_idfix for ChatGPT captures, where 100% oftool_use/tool_resultblocks hadtool_id IS NULL, so no pairing ever existed.flake.nixdeclaredpython314butnix develophanded out 3.13.13, becauseuv venvonly ran when.venvwas absent and the sync fingerprint readpython --versionafter activation. The 3.14t shell now auto-syncs adev-freethreadedextra.Verification
Anti-vacuity, each checked by reverting the implementation and confirming the test fails:
test_observed_structure_merge_never_drops_a_named_propertycatalog_incoherentfindings against the pre-fix treeparallel_threads_effective()is False on the standard shell, True under.#freethreadedProperty-path coverage after re-promotion, zero losses: claude-code 627 → 1060 (the
anyOfbug alone had cost 433), codex 207 → 1153, claude-ai 190 → 595, gemini-cli 118 → 126.Known-broken baseline
tests/unit/sources/test_browser_capture.pyhas 3 failing title-precedence tests. Reproduced at pureorigin/masterwith these changes stashed — inherited, not introduced. Tracked aspolylogue-lrdh.render all --checkalso reported two surfaces out of sync before this branch; both are now regenerated.Follow-ups
Filed, not fixed here:
polylogue-ktwa(multi-session raws never acquire alogical_source_key, so 50 GiB is stuckrevision_kind='unknown'),polylogue-ei0d(1.28 GiB of write-onlypayload_json),polylogue-wkc6(the accumulated 6 GB needs a one-off prune + VACUUM once this deploys),polylogue-mkk0(the archive-scale equivalence receipt is blocked behind an unrelated 3.14t deploy bead).🤖 Generated with Claude Code
https://claude.ai/code/session_0182HDxDpJpsbn2qcKWK6Fsf
Summary by CodeRabbit
New Features
Improvements
anyOf/oneOf/allOf).Compatibility
ambiguous(nowresolved,unresolved,edge_only,quarantined).