From 3b7a9930531d8561c3b65ac5b867587c174df009 Mon Sep 17 00:00:00 2001 From: Thor Whalen <1906276+thorwhalen@users.noreply.github.com> Date: Tue, 8 Sep 2026 22:12:20 +0200 Subject: [PATCH 1/3] Evidence layer: persist analysis into a lacing store MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The committed StepDocument is the contract; the machine evidence behind it — the speech/music split, the metric grid, the beats, the transcript, the step candidates, the derivation recipes and the lineage between them — now lives in a lacing store, and the document is a projection of it (docs/07 §6.0). - paces/bodies/: eight body schemas, every URI paces owns prefixed `guide-`. register_body_schema is last-write-wins and silent on a flat, fleet-wide namespace, and annot://schema/beat/v1 is already reelee's narrative beat — so one rule rather than a per-name audit. word/v1 is lacing's and reused. - paces/evidence.py: to_store / from_store / recipes_from_store. Annotation ids are uuid5 over a stable evidence key, so docs/07 §6.4 steps 1-3 hold by construction and Origin.annotationId is reproducible; an unchanged value digest leaves the row completely alone; a row a re-run no longer produces is pruned rather than left to be resurrected. - Times cross as RationalTime ticks at an injectable rate=, raising rather than rounding. Intervals live on the reference, never in a body. - locks, questions, artifacts and span excerpt windows are document-layer and never flow into the store — §6.0's one-way rule. - [lacing] extra, mirrored into [dev]. `import paces` still imports only pydantic; paces.to_store resolves through a module __getattr__. - URI set and each JSON Schema pinned against a committed snapshot. Claude-Session: https://claude.ai/code/session_01UDdKG2wFtDzsY2fgBGEjVK --- .claude/CLAUDE.md | 20 +- README.md | 61 +- docs/07-annotation-model.md | 93 ++- paces/__init__.py | 30 + paces/bodies/__init__.py | 111 +++ paces/bodies/_base.py | 24 + paces/bodies/derivation.py | 76 ++ paces/bodies/document.py | 118 ++++ paces/bodies/signal.py | 122 ++++ paces/bodies/step.py | 155 ++++ paces/edits.py | 8 +- paces/evidence.py | 1208 ++++++++++++++++++++++++++++++++ paces/model.py | 5 +- pyproject.toml | 15 +- tests/data/body_schemas.json | 746 ++++++++++++++++++++ tests/test_evidence.py | 571 +++++++++++++++ tests/test_evidence_schemas.py | 94 +++ 17 files changed, 3421 insertions(+), 36 deletions(-) create mode 100644 paces/bodies/__init__.py create mode 100644 paces/bodies/_base.py create mode 100644 paces/bodies/derivation.py create mode 100644 paces/bodies/document.py create mode 100644 paces/bodies/signal.py create mode 100644 paces/bodies/step.py create mode 100644 paces/evidence.py create mode 100644 tests/data/body_schemas.json create mode 100644 tests/test_evidence.py create mode 100644 tests/test_evidence_schemas.py diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index e15ff8e..fae9575 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -7,6 +7,7 @@ | 1 | `segmenter=` on `segment()` | auto-select from present facts (`grid-placed` / `explicit`) | the strategies catalogued in `docs/alignment/07` (novelty-k, align-to-steps); `kodokan.segment`; `mixing.audio.find_segments` | | 2 | `renderer=` on `tools.render()` | `"html"` practice page (stdlib) | the POC page (metronome+clips) at `docs/poc-reference/render/`; PDF/deck per `docs/03 §2.3` | | 3 | `grid=` values on `segment()` | caller-supplied tempo/origin; measured from local media when absent (`grid-measured`: mixing speech/music split + beat_grid, origin estimated + flagged) | better origin fitting (issue #2 follow-ons; `kodokan.estimate_period`) | +| 4 | `store=` on `evidence.to_store()` / `from_store()` | none — injected, always: `lacing.MemoryStore` in tests, `SqliteStore` for a project sidecar | any `dol` MutableMapping conforming to `lacing.IntervalAnnotationStore` | Surface for v1: **CLI** (`argh` over `tools._dispatch_funcs`). MCP (`py2mcp` string refs), HTTP (`qh`), shipped skills, frontend: questions answered — none @@ -26,6 +27,19 @@ template internals, CLI parsing, storage (v1 reads/writes explicit paths). by `resolve()`, never stored. - A segmenter may never return a step count it has no evidence for (honesty states in `paces/segment.py` docstring). -- Named next steps: evidence layer → lacing (`docs/07 §6.3`); `paces/genre.py` - registering `step_by_step` into `nw` (ADR-0004); intrinsic segmenter - combinators when a third intrinsic segmenter needs them (ADR-0003). +- The evidence layer (`paces/evidence.py` + `paces/bodies/`, issue #4) owns + the lacing boundary. Its rules, in one line each: intervals live on + `Annotation.reference`, never in a body; annotation ids are `uuid5` over a + stable evidence key, so a re-derivation *is* the same row; an unchanged + value digest leaves the row completely alone; `locks`, `questions`, + `artifacts` and span `excerpt` windows are document-layer and never flow + into the store. +- **Body schema URIs are a flat, fleet-wide namespace and + `register_body_schema` overwrites silently.** Every URI paces owns is + prefixed `guide-`; `word/v1` is lacing's and reused. Before adding one, + check the fleet (`rg 'annot://schema/'` across `$PP`) and pin it in + `tests/test_evidence_schemas.py`. +- Named next steps: `paces/genre.py` registering `step_by_step` into `nw` + (ADR-0004); intrinsic segmenter combinators when a third intrinsic + segmenter needs them (ADR-0003); a CLI/MCP surface over `to_store` + (needs no core change — the store's on-disk shape is the open question). diff --git a/README.md b/README.md index 7a9df89..89b470a 100644 --- a/README.md +++ b/README.md @@ -114,6 +114,60 @@ run-through and the breakdown are the same step seen twice). Uncertainty is content (`OpenQuestion`), and human edits are protected from regeneration (`Lock`). +## The evidence layer + +The document is the *contract*; the machine evidence behind it — the +speech/music split, the metric grid, the beats, the transcript, the step +candidates, the crop recipes, and the lineage between them — lives in a +[`lacing`](https://github.com/thorwhalen/lacing) store, and the document is a +**projection** of it. The document is derivable from the store; the store is +not derivable from the document. + +```python +from lacing import MemoryStore # pip install 'paces[lacing]' +from paces import segment, from_store, to_store +from paces.model import dumps_document + +seg = segment(None, steps=[("Mise en place", 2), ("Déhanchés", 8)], grid=GRID) + +store = MemoryStore() # or SqliteStore('project.annot') +write = to_store(seg, store=store, asset_id=asset_sha256, + doc_id="que-calor", title="Que Calor", domain="dance", + source="https://youtu.be/q_TUyxUhoEw") + +dumps_document(from_store(store, asset_id=asset_sha256)) == dumps_document(write.document) +# True — the projection is exact + +to_store(seg, store=store, asset_id=asset_sha256, doc_id="que-calor", + title="Que Calor", domain="dance", + source="https://youtu.be/q_TUyxUhoEw").written +# 0 — a re-run of the same analysis writes nothing +``` + +The store is injected, never constructed for you: `MemoryStore()` in tests, a +`SqliteStore` for a project sidecar, any `dol` store that conforms. Annotation +ids are derived (`uuid5`) from the evidence they stand for, so a re-derivation +*is* the same annotation — which is what keeps `was_derived_from` lineage +resolving and lets `write.document`'s `Origin.annotationId` be committed. Rows +whose value digest did not change are left completely alone, so freshness does +not fire on a no-op re-run. + +`to_store` is a **re-derivation** of the guide, not a merge into it: a step +that no longer exists is dropped rather than left to be resurrected by the +next projection. It only ever prunes this asset, this `doc_id`, and the tiers +that call actually wrote to — so omitting `passes=` leaves an earlier +speech/music split standing. `prune=False` accumulates instead. + +Analysis a `Segmentation` cannot carry rides along as keywords: `passes=` +(the speech/music split), `beats=`, `transcript=`, `cues=`, `recipes=`. What +stays on the document and never flows back into the store: `locks`, +`questions`, `artifacts`, and span `excerpt` windows — those are human +decisions, not measurements. Full table and rationale in +`docs/07-annotation-model.md` §6. + +`import paces` does not import `lacing`: the core stays pydantic-only, and +`paces.to_store` resolves the extra on first use. + ## The pieces | you want | reach for | @@ -126,6 +180,7 @@ content (`OpenQuestion`), and human edits are protected from regeneration | the committed artifact | `to_document(seg, ...)` → `StepDocument` | | real clips/gifs/posters for the page | `derive_document(doc, media=..., doc_path=...)` / `paces derive` (`pip install paces[media]`) | | auto-crop those clips to the people in frame | `derive(..., subject_locator=paces.pose.rtmlib_pose)` / `--subject-locator paces.pose:rtmlib_pose` (`pip install paces[pose]`) | +| persist the analysis behind a document | `to_store(seg, store=..., asset_id=...)` / `from_store(store, asset_id=...)` (`pip install paces[lacing]`) | | a practice page | `render_html(doc)` | | wall-clock times from counts | `resolve(doc)` | | sanity checks | `validate_document(doc)` | @@ -137,6 +192,6 @@ content (`OpenQuestion`), and human edits are protected from regeneration Young and moving. The document schema is validated by round-tripping a real proof of concept ([an interactive dance-practice page](https://thorwhalen.com/que_calor_dance/)) through it — see -`tests/test_roundtrip_poc.py`. Media derivation (auto-cropped looping clips), -intrinsic segmenters (scene/beat/speech detection), and the evidence layer -are designed (see `docs/`) and arrive next. +`tests/test_roundtrip_poc.py`. Media derivation (auto-cropped looping clips) +and the evidence layer have landed; intrinsic segmenters (scene/beat/speech +detection) are designed (see `docs/`) and arrive next. diff --git a/docs/07-annotation-model.md b/docs/07-annotation-model.md index 43821e8..df93321 100644 --- a/docs/07-annotation-model.md +++ b/docs/07-annotation-model.md @@ -551,34 +551,74 @@ class StepDocument(_Base): | survives re-generation; human edits not lost | `Lock` list per node + `Origin.value_digest`; regeneration is `reelee.edits.regen`'s algorithm — re-derive, **adopt the superseded id**, skip locked paths, and use `annotation_value_digest` to stop propagating when nothing changed | | plain JSON, git-diffable | see §6.5 | -### 6.3 Projection contract: lacing tiers → document +### 6.3 Projection contract: lacing tiers → document — **SHIPPED** (issue #4) -Register these body schemas (`lacing.schema.register_body_schema`) in the new package's -`bodies/`, alongside the ones that already exist: +Implemented in `paces/bodies/` (the schemas) and `paces/evidence.py` (`to_store`, +`from_store`), behind the `[lacing]` extra. The table below is what actually ships; the +column on the right records where it differs from this file's original proposal and why. | tier | stereotype | parent | body schema | holds | |---|---|---|---|---| -| `source.pass` | `NONE` | — | `annot://schema/media-pass/v1` | *"[45s, 215s] is a performance pass; [220s, 520s] is an instruction pass"* — **detecting this is a first-class Phase-1 job** and is what made the POC's dual timestamps possible | -| `step` | `NONE` | — | `annot://schema/step/v1` | one annotation **per (step, span)**; body carries `step_id`, `role`, `name`, `description` | -| `step.sub` | `INCLUDED_IN` | `step` | `annot://schema/step/v1` | sub-steps | -| `cue` | `INCLUDED_IN` | `step` | `annot://schema/cue/v1` | lyric/audio landmarks | -| `transcript.segment` / `transcript.word` | `NONE` | — | `word/v1` (**exists**) | whisper output | -| `beat` | `NONE` | — | `annot://schema/beat/v1` | `beats.npy` | -| `derivation` | `SYMBOLIC_ASSOCIATION` | `step` | `annot://schema/derivation-recipe/v1` | crop box, fps, palette — the *parameters*, so a re-run is stable and a human override survives | - -`step/v1` body ≈ `{step_id, parent_step_id, role, name, description, duration_value, -duration_unit, ordinal}`. Note the shape mirrors `nw/bodies/section.py`'s -`SectionBodyV1{section_id, label, energy, mood}` — the `*_id` field is *"Stable id within a -project… **Distinct from the annotation id**"*. Same discipline: document ids are slugs, -store ids are UUIDs, and they are related by a body field, never conflated. +| `document` | `NONE` | — | `annot://schema/guide-doc/v1` | the guide's envelope: id, title, lang, domain, plus the run's own `method` and honesty `flags` | +| `source` | `NONE` | — | `annot://schema/guide-source/v1` | one per `Source` | +| `source.pass` | `NONE` | — | `annot://schema/guide-pass/v1` | *"[0s, 51.2s] is speech; [51.2s, 400s] is music"* — the split `paces.measure` gets from `mixing.find_segments`, and what made the POC's dual timestamps possible | +| `step` | `NONE` | — | `annot://schema/guide-step/v1` | one annotation **per (step, span)**; `step_id`, `parent_step_id`, `ordinal`, `span_index`, name, description, duration | +| `step.sub` | `INCLUDED_IN` | `step` | `annot://schema/guide-step/v1` | sub-steps | +| `cue` | `INCLUDED_IN` | `step` | `annot://schema/guide-cue/v1` | lyric/audio landmarks | +| `grid` | `NONE` | — | `annot://schema/guide-grid/v1` | the metric grid, **one** annotation over the region it governs; the document's `metric` is projected straight back out of it | +| `beat` | `NONE` | — | `annot://schema/guide-beat/v1` | one point annotation per *measured* beat | +| `transcript.word` | `NONE` | — | `word/v1` (**lacing's, reused unchanged**) | whisper output | +| `derivation` | `SYMBOLIC_ASSOCIATION` | `step` | `annot://schema/guide-recipe/v1` | the crop box, window, locator and policy params — the *parameters*, so a re-run is stable and a human override survives | + +**Why `guide-`, and not the bare names this file first proposed.** +`lacing.schema.register_body_schema` is last-write-wins and silent, and the URI namespace +is one flat, fleet-wide dict populated by import side effects. `annot://schema/beat/v1` is +**already** `reelee/bodies/beat.py`'s *narrative* beat — claiming it would have replaced +reelee's validator in any process importing both. `braidio` had already hit this and named +its own `narrative-beat/v1`. So paces takes one rule rather than a per-name judgement: +every URI it owns is prefixed `guide-` (the thing it describes, reusable by any other +producer of a step guide), and the one URI it does not own — `word/v1` — is reused. The +URI set and each schema's content are pinned in `tests/test_evidence_schemas.py` against a +committed snapshot, so a rename or a field change fails the build. Tier names are +unprefixed: tiers are store-local and cost nothing. + +`guide-step/v1`'s `step_id` mirrors `nw/bodies/section.py`'s `SectionBodyV1{section_id, +…}` — *"Stable id within a project… **Distinct from the annotation id**"*. Same +discipline: document ids are slugs, store ids are UUIDs, related by a body field, never +conflated. + +**Two rules the implementation holds absolutely.** No interval ever appears in a body — +these are standoff annotations, and the interval lives on `Annotation.reference`; and no +float crosses the boundary — times become `RationalTime` ticks at an injectable `rate=` +(default 24000), where a value the rate cannot hold exactly raises rather than rounding. + +**What does not flow into the store**: `locks`, `questions`, `artifacts`, and span +`excerpt` windows. Those are document-layer records — a human edit, an open question, a +built file, a hand-picked loop — and §6.0's one-way rule is what says so. +`from_store` returns them empty, which is exactly what `to_document` emits, so the round +trip is byte-exact. ### 6.4 Regeneration algorithm (adapt `reelee/edits.py`, don't reinvent) -1. Re-run analysis → new annotations in the store, each with fresh `uuid4` + provenance. -2. For each, compute `annotation_value_digest`. Unchanged digest ⇒ **stop propagating**; the - downstream artifacts are still valid and cost nothing. -3. Changed digest ⇒ the regenerated annotation **adopts the id of the one it supersedes** - (`_adopt_output_identity`), so downstream `was_derived_from` edges keep resolving. +Steps 1–3 are the store side and **shipped** with issue #4; steps 4–6 are the document +side and ship with the edit-protection work (`paces/edits.py`). + +1. Re-run analysis → annotations in the store. paces derives each id as + `uuid5(PACES_NAMESPACE, f"{tier}\n{evidence_key}")` rather than minting a fresh `uuid4`, + where the evidence key is the span address `paces/derivation.py` already uses. +2. For each, compute `annotation_value_digest`. Unchanged digest ⇒ **stop propagating** — + and paces leaves the row *completely* untouched, provenance included, so + `generated_at_time` does not churn and freshness does not fire. A second `to_store` of + the same input writes zero annotations. +3. Changed digest ⇒ the regenerated annotation replaces the one it supersedes **under the + same id**, so downstream `was_derived_from` edges keep resolving. `reelee` does this + procedurally (`_adopt_output_identity`); with a derived id it holds by construction, and + `to_store` is reproducible from a fresh store as a bonus. + A row the re-run no longer produces is *removed*, not left behind — `reelee`'s + *"the graph is a set of nodes whose values are re-derivable, not an append-only log"*, + applied. Pruning is scoped to the asset, the `doc_id`, and the tiers that run actually + wrote to, so omitting an optional evidence kwarg cannot delete an earlier producer's + rows. 4. Re-project to a candidate `StepDocument`. 5. Three-way merge against the committed document: for every `Lock.path` on a node, keep the committed value and record `Origin.value_digest` of what was rejected. For everything else, @@ -598,8 +638,13 @@ holds at the document layer too), and that append to the op-log. for a model, since declaration order is the semantic order. - **No floats anywhere on the wire.** Decimal strings (`"231.3"`) — exact via `Fraction`, no `0.30000000000000004`, and they diff as one token. Parse to `RationalTime` on load. -- **No absolute local paths, no per-run timestamps, no uuid4s** in the committed document. - `Lock.at` is the only timestamp, and it changes only when a human edits. +- **No absolute local paths, no per-run timestamps, and no *churning* ids** in the + committed document. `Lock.at` is the only timestamp, and it changes only when a human + edits. This bullet used to say "no uuid4s", which contradicted §6.1's + `Origin.annotation_id`; issue #4 settled it on the rule's actual reason. A `uuid4` is + barred because it is different on every run; paces' annotation ids are `uuid5` over a + stable evidence key, so they never churn and `Origin.annotationId` / + `Origin.valueDigest` are committed. - `ArtifactRef.uri` is relative to the document. `asset_id` is the durable identity. - Sidecars: `project.annot` (SQLite lacing store, **gitignored**), `media/` (artifacts, either gitignored or LFS), `.recipes.json` (crop recipes, committed), `document.json` @@ -640,7 +685,7 @@ Three are declined; see §7 for the one-line reasons: optional `SourceSpan.id` ( | **A `kind`/`type` discriminated union on `Step`** | the POC needed exactly one step type. `Beat`-like non-content steps (a title card, a rest) can be `Step(tags=["rest"], spans=[])`. Add the union at the third real variant, not the first. | | **`TimeInterval` on the document's spans** | `{"start":{"v":231300,"r":1000},"end":…}` is correct and unreadable. Decimal strings on the wire, `RationalTime` in memory, one converter in the loader. Inside the lacing store, rational all the way. | | **A DSL / Markdown authoring layer** | `an/ir/sync.py` proves it can be done and its bidirectional round-trip is the riskiest code in that package. JSON + typed patches first; add a Markdown face only after the JSON contract has survived two renderers. | -| **Optional `SourceSpan.id`** (issue #12) | the real cure for the one residual mis-identification case a structural match can't beat (identity-swapping content: a locked span whose start moves while a same-`(source, role)` sibling assumes its exact old start). Fits #4's evidence-layer timeline (store-side identity adoption wants ids anyway) — add it there, as a schema change, not as an unscoped addition here. | +| **Optional `SourceSpan.id`** (issue #12) | the real cure for the one residual mis-identification case a structural match can't beat (identity-swapping content: a locked span whose start moves while a same-`(source, role)` sibling assumes its exact old start). This row expected #4 to want it ("store-side identity adoption wants ids anyway"); it turned out not to. Store-side identity is the span address `{step_id}/{source}/{role}/{start}` — already `paces/derivation.py`'s key, and already protected from collision by `validate_document`. So #4 declined it and it stays a #12 document-layer fix, for the lock-reapplication case it was actually diagnosed for. | | **Refusing selection-boost scoring on fact presence vs. usability** (issue #12) | boundaries + N names + N matching chapters can pick `explicit` (unnamed spans + mismatch flag) where `chapters` would have named everything — honestly flagged today (the mismatch flag says so), never silently wrong. Revisiting the scoring function meaningfully needs a third real case to generalize from; the issue itself defers this to when a fourth segmenter lands. | --- diff --git a/paces/__init__.py b/paces/__init__.py index 1d5631a..0f821dc 100644 --- a/paces/__init__.py +++ b/paces/__init__.py @@ -116,6 +116,36 @@ # editing & regeneration "apply_edits", "merge_regenerated", + # the evidence layer (issue #4) — resolved lazily, needs [lacing] + "to_store", + "from_store", + "StoreWrite", # rendering "render_html", ] + +#: The evidence layer's public names and where they live. Resolved on first +#: attribute access rather than imported here, so ``import paces`` keeps the +#: pydantic-only core of ADR-0004 — installing ``paces`` must not pull +#: ``lacing`` in, and asking for ``paces.to_store`` without the ``[lacing]`` +#: extra must fail with an install line rather than an AttributeError. +_LAZY = { + "to_store": "paces.evidence", + "from_store": "paces.evidence", + "StoreWrite": "paces.evidence", + "recipes_from_store": "paces.evidence", +} + + +def __getattr__(name: str): + """Resolve the evidence layer on demand (PEP 562).""" + module_name = _LAZY.get(name) + if module_name is None: + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") + import importlib + + return getattr(importlib.import_module(module_name), name) + + +def __dir__() -> list[str]: + return sorted({*globals(), *_LAZY}) diff --git a/paces/bodies/__init__.py b/paces/bodies/__init__.py new file mode 100644 index 0000000..e9a6936 --- /dev/null +++ b/paces/bodies/__init__.py @@ -0,0 +1,111 @@ +"""The body schemas paces registers with ``lacing`` — the evidence layer's vocabulary. + +Importing this package registers every URI paces owns into +:mod:`lacing.schema`'s registry. That registry is **one flat, fleet-wide +dict**, ``register_body_schema`` is last-write-wins and silent, and +``annot://schema/beat/v1`` is already ``reelee``'s *narrative* beat — so +every URI paces owns is prefixed ``guide-`` (the thing it describes: a step +guide), and the one URI paces does not own, ``annot://schema/word/v1``, is +reused from :mod:`lacing.bodies` unchanged. The reasoning is issue #4's +decision comment and ``docs/07-annotation-model.md §6.3``. + +Tier names are *not* prefixed: tiers are store-local, so ``step``, +``step.sub``, ``cue``, ``beat`` cost nothing and stay exactly as +``docs/07 §6.3`` wrote them. + +Two rules hold across every body here: + +- **No interval in a body.** Annotations are standoff; the interval lives on + ``Annotation.reference``. A body that wants to say *when* says it by being + attached somewhere, never by carrying a number. +- **No floats where the document would refuse one.** Times, durations and + box coordinates cross as decimal strings, matching the document's + no-floats wire (``docs/07 §6.5``). ``confidence`` is the one exception the + document itself already makes. +""" + +from paces.bodies.derivation import ( + RECIPE_BODY_SCHEMA_URI, + RECIPE_TIER, + GuideRecipeBodyV1, +) +from paces.bodies.document import ( + DOC_BODY_SCHEMA_URI, + DOC_TIER, + SOURCE_BODY_SCHEMA_URI, + SOURCE_TIER, + GuideDocBodyV1, + GuideSourceBodyV1, +) +from paces.bodies.signal import ( + BEAT_BODY_SCHEMA_URI, + BEAT_TIER, + GRID_BODY_SCHEMA_URI, + GRID_TIER, + PASS_BODY_SCHEMA_URI, + PASS_TIER, + WORD_BODY_SCHEMA_URI, + WORD_TIER, + GuideBeatBodyV1, + GuideGridBodyV1, + GuidePassBodyV1, +) +from paces.bodies.step import ( + CUE_BODY_SCHEMA_URI, + CUE_TIER, + STEP_BODY_SCHEMA_URI, + STEP_TIER, + SUB_STEP_TIER, + GuideCueBodyV1, + GuideStepBodyV1, +) + +#: Every ``body_schema_uri`` paces *owns*. ``word/v1`` is deliberately absent: +#: it is lacing's, reused rather than claimed. Pinned by +#: ``tests/test_evidence_schemas.py`` against a committed JSON Schema snapshot, +#: so a rename or a field change fails the build (issue #4). +PACES_BODY_SCHEMA_URIS = ( + DOC_BODY_SCHEMA_URI, + SOURCE_BODY_SCHEMA_URI, + PASS_BODY_SCHEMA_URI, + STEP_BODY_SCHEMA_URI, + CUE_BODY_SCHEMA_URI, + GRID_BODY_SCHEMA_URI, + BEAT_BODY_SCHEMA_URI, + RECIPE_BODY_SCHEMA_URI, +) + +__all__ = [ + "PACES_BODY_SCHEMA_URIS", + # document envelope + "DOC_TIER", + "DOC_BODY_SCHEMA_URI", + "GuideDocBodyV1", + "SOURCE_TIER", + "SOURCE_BODY_SCHEMA_URI", + "GuideSourceBodyV1", + # steps and cues + "STEP_TIER", + "SUB_STEP_TIER", + "STEP_BODY_SCHEMA_URI", + "GuideStepBodyV1", + "CUE_TIER", + "CUE_BODY_SCHEMA_URI", + "GuideCueBodyV1", + # measured signal + "PASS_TIER", + "PASS_BODY_SCHEMA_URI", + "GuidePassBodyV1", + "GRID_TIER", + "GRID_BODY_SCHEMA_URI", + "GuideGridBodyV1", + "BEAT_TIER", + "BEAT_BODY_SCHEMA_URI", + "GuideBeatBodyV1", + "WORD_TIER", + "WORD_BODY_SCHEMA_URI", + # derivation + "RECIPE_TIER", + "RECIPE_BODY_SCHEMA_URI", + "GuideRecipeBodyV1", +] diff --git a/paces/bodies/_base.py b/paces/bodies/_base.py new file mode 100644 index 0000000..279f3a1 --- /dev/null +++ b/paces/bodies/_base.py @@ -0,0 +1,24 @@ +"""Shared conventions for every paces body schema. + +One base class and one type alias, so the three rules that make a body +round-trippable are stated once rather than re-typed per module: frozen, +``extra="forbid"`` (lacing non-negotiable #3), and decimal strings where the +document would refuse a float. +""" + +from __future__ import annotations + +from typing import Annotated + +from pydantic import BaseModel, Field + +#: Exact decimal number as a string ("231.3") — the document's wire form for +#: any quantity, mirrored here so a body never carries a float the document +#: itself would reject (``docs/07 §6.5``). +Decimal = Annotated[str, Field(pattern=r"^-?\d+(\.\d+)?$")] + + +class BodyBase(BaseModel): + """Frozen, closed-world body model — lacing validates against it by URI.""" + + model_config = {"frozen": True, "extra": "forbid"} diff --git a/paces/bodies/derivation.py b/paces/bodies/derivation.py new file mode 100644 index 0000000..83c169f --- /dev/null +++ b/paces/bodies/derivation.py @@ -0,0 +1,76 @@ +"""Body schema for derivation recipes — the POC's ``crops.json``, promoted. + +``annot://schema/guide-recipe/v1``, tier ``derivation`` +(``SYMBOLIC_ASSOCIATION`` under ``step``): the *parameters* a derived file +was made with, so a re-run is stable and a human override survives it. +``docs/07 §6.2``'s split, stated once: the document's +:class:`~paces.model.ArtifactRef` says **what** a file is; this says **how** +it was made. + +The recipe's ``window`` is not a field — it is the annotation's reference +interval, which is exactly what a window is. Everything else is +:class:`paces.derivation.CropRecipe` as it already exists on disk, with the +box's coordinates crossing as decimal strings; the sidecar is committed, so +it has always been float-free and the store keeps it that way. +""" + +from __future__ import annotations + +from lacing.schema import register_body_schema +from pydantic import Field + +from paces.bodies._base import BodyBase, Decimal + +RECIPE_TIER = "derivation" +RECIPE_BODY_SCHEMA_URI = "annot://schema/guide-recipe/v1" + + +class GuideRecipeBodyV1(BodyBase): + """One span's derivation parameters, keyed by the span address.""" + + doc_id: str = Field(..., description="Which guide this recipe belongs to.") + span_key: str = Field( + ..., + description=( + "The span address ``{step_id}/{source}/{role}/{start}`` — the same " + "key ``paces.derivation`` writes into the committed sidecar, so " + "the two layers join without a translation table." + ), + ) + step_id: str = Field(..., description="The step this recipe derives media for.") + box: list[Decimal] | None = Field( + None, + description=( + "The resolved crop box (x, y, w, h) in pixels, as decimal " + "strings. None means full frame — a recorded decision, not a gap." + ), + ) + frame_width: int = Field(..., description="Source frame width the box is in.") + frame_height: int = Field(..., description="Source frame height the box is in.") + source_asset_id: str | None = Field( + None, description="Hash of the media cut from. None = identity unverified." + ) + locator: str = Field( + ..., description="Which subject locator resolved the box ('full_frame', ...)." + ) + params: dict[str, Decimal] = Field( + default_factory=dict, + description=( + "Policy parameters the box was resolved under (aspect, pad). A " + "change here re-locates an unlocked entry." + ), + ) + locked: bool = Field( + False, description="A human set this box; reconcile must not re-derive it." + ) + media_digest: str | None = Field( + None, + description=( + "What the stored media was actually cut with (box + window + " + "source hash). A hand-edited box or a re-timed window changes it " + "and forces a re-encode instead of serving stale bytes." + ), + ) + + +register_body_schema(RECIPE_BODY_SCHEMA_URI, GuideRecipeBodyV1) diff --git a/paces/bodies/document.py b/paces/bodies/document.py new file mode 100644 index 0000000..75cffc5 --- /dev/null +++ b/paces/bodies/document.py @@ -0,0 +1,118 @@ +"""Body schemas for the document envelope and its sources. + +- ``annot://schema/guide-doc/v1`` (tier ``document``) — the one annotation + that says *which guide this is*: id, title, language, domain, plus the + analysis run's own report (``method``, ``flags``). Everything ordered or + timed hangs off the other tiers. +- ``annot://schema/guide-source/v1`` (tier ``source``) — one per + :class:`~paces.model.Source`. + +Neither body carries an interval; both annotations attach as point +annotations at tick 0, which is lacing's spelling of "about the asset, not +about a region of it". + +What is deliberately **not** here: ``locks``, ``questions``, ``artifacts`` +and span ``excerpt`` windows. Those are document-layer records — a human +edit, an open question, a built file, a hand-picked loop — and +``docs/07 §6.0`` is explicit that nothing flows document → store. The +evidence layer holds what analysis produced; the document holds what people +did to it. +""" + +from __future__ import annotations + +from typing import Any + +from lacing.schema import register_body_schema +from pydantic import Field + +from paces.bodies._base import BodyBase, Decimal + +DOC_TIER = "document" +DOC_BODY_SCHEMA_URI = "annot://schema/guide-doc/v1" + +SOURCE_TIER = "source" +SOURCE_BODY_SCHEMA_URI = "annot://schema/guide-source/v1" + + +class GuideDocBodyV1(BodyBase): + """The guide's envelope: identity, presentation language, and the run.""" + + doc_id: str = Field( + ..., + description=( + "Stable slug of the guide within a project. Distinct from the " + "annotation id, and the field that lets one store hold several " + "guides over the same asset." + ), + ) + title: str = Field(..., description="Human title of the guide.") + lang: str = Field("en", description="BCP-47 language tag of the guide's prose.") + domain: str = Field( + "generic", description="Domain slug: dance | recipe | workout | ..." + ) + schema_version: str = Field( + ..., description="Version of the StepDocument schema this was projected at." + ) + credits: str | None = Field( + None, description="Attribution line, when there is one." + ) + method: str = Field( + "", + description=( + "Which segmentation capability produced the steps " + "(``Segmentation.method``). Empty when unrecorded." + ), + ) + flags: list[str] = Field( + default_factory=list, + description=( + "The run's honesty flags (``Segmentation.flags``) — " + "'naming-abstained', 'origin-estimated: ...', and friends. " + "Evidence about the run, never projected into the document." + ), + ) + attrs: dict[str, Any] = Field( + default_factory=dict, description="Namespaced open bag, as on the document." + ) + + +class GuideSourceBodyV1(BodyBase): + """One input the guide's spans refer into.""" + + doc_id: str = Field( + ..., + description=( + "Which guide this source belongs to — one store can hold several " + "guides over the same asset, and a projection has to read back " + "only its own." + ), + ) + ordinal: int = Field( + ..., description="Position in the document's source list, 0-based." + ) + source_id: str = Field( + ..., description="Slug of the source within the guide. Not the annotation id." + ) + kind: str = Field(..., description="video | audio | image | document | url.") + uri: str = Field(..., description="Where the source lives.") + asset_id: str | None = Field( + None, description="Content hash of a local copy, if any." + ) + duration_s: Decimal | None = Field( + None, description="Source duration in seconds, as a decimal string." + ) + title: str | None = Field(None, description="Source title.") + attribution: str | None = Field(None, description="Who made it.") + rights: str | None = Field( + None, + description=( + "Usage rights. First-class in this domain (``docs/07 §8.7``): a " + "guide that restyles someone's choreography has to say so." + ), + ) + attrs: dict[str, Any] = Field(default_factory=dict, description="Open bag.") + + +register_body_schema(DOC_BODY_SCHEMA_URI, GuideDocBodyV1) +register_body_schema(SOURCE_BODY_SCHEMA_URI, GuideSourceBodyV1) diff --git a/paces/bodies/signal.py b/paces/bodies/signal.py new file mode 100644 index 0000000..a24531d --- /dev/null +++ b/paces/bodies/signal.py @@ -0,0 +1,122 @@ +"""Body schemas for what was measured off the media: passes, the grid, beats. + +- ``annot://schema/guide-pass/v1`` (tier ``source.pass``) — *"[45 s, 215 s] + is a music pass; [220 s, 520 s] is speech"*. The speech/music split + ``paces.measure`` gets from ``mixing.find_segments``, which is what made + the POC's dual timestamps possible in the first place. +- ``annot://schema/guide-grid/v1`` (tier ``grid``) — the metric grid, **one** + annotation over the region it governs. +- ``annot://schema/guide-beat/v1`` (tier ``beat``) — one point annotation per + *measured* beat. + +Why the grid is one annotation and the beats are separate: paces' grid is +closed form (``tempo_bpm`` + ``origin`` + ``subdivisions``), so materialising +N beats *from* it would be a denormalised cache with no invalidation story — +the same argument ``docs/07 §7`` uses to refuse materialising Allen +relations. Beat annotations are therefore written only when a producer hands +over times it actually measured, and they carry a ``was_derived_from`` edge +to the grid so a re-measure invalidates them. + +Transcripts reuse lacing's own ``annot://schema/word/v1`` unchanged, on tier +``transcript.word``; the URIs and tier name live here so every evidence tier +is named in one place. +""" + +from __future__ import annotations + +import lacing.bodies # noqa: F401 — registers the word/v1 paces reuses +from lacing.schema import register_body_schema +from pydantic import Field + +from paces.bodies._base import BodyBase, Decimal + +PASS_TIER = "source.pass" +PASS_BODY_SCHEMA_URI = "annot://schema/guide-pass/v1" + +GRID_TIER = "grid" +GRID_BODY_SCHEMA_URI = "annot://schema/guide-grid/v1" + +BEAT_TIER = "beat" +BEAT_BODY_SCHEMA_URI = "annot://schema/guide-beat/v1" + +#: Transcripts are lacing's, not paces'. Named here so the tier table is +#: complete in one file; NOT registered here, and NOT in +#: ``PACES_BODY_SCHEMA_URIS`` — ``lacing.bodies.word`` owns it. +WORD_TIER = "transcript.word" +WORD_BODY_SCHEMA_URI = "annot://schema/word/v1" + + +class GuidePassBodyV1(BodyBase): + """A stretch of one source with a single character (music, speech, ...).""" + + label: str = Field( + ..., + description=( + "What this stretch is: 'music' | 'speech' | '' when the detector " + "found a segment it would not name." + ), + ) + detector: str = Field( + "", + description=( + "What produced the split ('mixing.find_segments:speech_music'). " + "Empty when the caller supplied passes without saying." + ), + ) + + +class GuideGridBodyV1(BodyBase): + """How the domain unit relates to wall-clock time, over the region it governs. + + The document's ``metric`` is projected straight back out of this — one + grid, one source of truth, no second copy in the document envelope. + """ + + doc_id: str = Field( + ..., + description=( + "Which guide this grid governs. The grid is per-guide, not " + "per-asset: two guides can count the same video differently." + ), + ) + unit: str = Field(..., description="The domain unit counted: eight | bar | rep.") + subdivisions: int = Field(1, ge=1, description="Beats per unit (8 for an 8-count).") + tempo_bpm: Decimal | None = Field( + None, + description=( + "Tempo as a decimal string. None spells 'tempo unknown' — never '0'." + ), + ) + origin: Decimal | None = Field( + None, description="Seconds into ``origin_source`` where unit 0 starts." + ) + origin_source: str | None = Field( + None, description="Which source ``origin`` is measured in." + ) + beat_count: int | None = Field( + None, + description=( + "How many beats were actually tracked, when the grid was measured " + "rather than supplied. None means nobody counted." + ), + ) + method: str = Field( + "", + description=( + "Which capability produced the grid ('grid-measured', " + "'grid-placed'). Empty when the caller supplied it outright." + ), + ) + + +class GuideBeatBodyV1(BodyBase): + """One measured beat. The time is the reference's point interval.""" + + index: int = Field( + ..., description="Position in the measured beat sequence, 0-based." + ) + + +register_body_schema(PASS_BODY_SCHEMA_URI, GuidePassBodyV1) +register_body_schema(GRID_BODY_SCHEMA_URI, GuideGridBodyV1) +register_body_schema(BEAT_BODY_SCHEMA_URI, GuideBeatBodyV1) diff --git a/paces/bodies/step.py b/paces/bodies/step.py new file mode 100644 index 0000000..4842098 --- /dev/null +++ b/paces/bodies/step.py @@ -0,0 +1,155 @@ +"""Body schemas for steps, sub-steps, and cues. + +- ``annot://schema/guide-step/v1`` — one annotation per **(step, span)** + (``docs/07 §6.3``), on tier ``step`` for top-level steps and tier + ``step.sub`` (``INCLUDED_IN`` ``step``) for sub-steps. A step that appears + in three places is three annotations sharing one ``step_id``; + ``parent_step_id``, ``ordinal`` and ``span_index`` are what put the tree + and its order back together. +- ``annot://schema/guide-cue/v1`` — one per cue, tier ``cue`` + (``INCLUDED_IN`` ``step``). + +The span's ``start`` and ``end`` are **not** fields here: they are the +annotation's reference interval. What stays in the body is everything about +the span that is not a time — which source, which role, the label and the +caption a human will edit. + +A step with no spans still gets exactly one annotation, with +``span_index=None`` and a point interval at tick 0. "This step exists, its +extent is not recorded" is a state the segmenters can legitimately return +(``paces/segmenters.py``'s honesty rules), so it is a state the store has to +be able to hold. +""" + +from __future__ import annotations + +from typing import Any + +from lacing.schema import register_body_schema +from pydantic import Field + +from paces.bodies._base import BodyBase, Decimal + +STEP_TIER = "step" +SUB_STEP_TIER = "step.sub" +STEP_BODY_SCHEMA_URI = "annot://schema/guide-step/v1" + +CUE_TIER = "cue" +CUE_BODY_SCHEMA_URI = "annot://schema/guide-cue/v1" + + +class GuideStepBodyV1(BodyBase): + """One step's presence in one span (or its absence from every span).""" + + doc_id: str = Field(..., description="Which guide this step belongs to.") + step_id: str = Field( + ..., + description=( + "Stable id within the guide ('b4'). Distinct from the annotation " + "id — document ids are slugs, store ids are UUIDs, and the two " + "are related by this field, never conflated." + ), + ) + parent_step_id: str | None = Field( + None, description="Enclosing step's id; None for a top-level step." + ) + ordinal: int = Field( + ..., description="Position among siblings, 0-based. Order is semantic." + ) + name: str = Field( + "", + description="The step's name. Empty means found-but-not-named, a valid state.", + ) + description: str = Field("", description="Prose about the step.") + duration_value: Decimal = Field( + ..., description="Duration in the domain's own unit, as a decimal string." + ) + duration_unit: str = Field( + ..., description="The domain unit: eight | rep | second." + ) + repeat: int = Field(1, description="How many times the children cycle.") + optional: bool = Field(False, description="'Facultatif — si tu le sens'.") + variant_of: str | None = Field( + None, description="Another angle on that step, not a new step." + ) + tags: list[str] = Field(default_factory=list, description="Open tag vocabulary.") + attrs: dict[str, Any] = Field( + default_factory=dict, description="Namespaced open bag, as on the step." + ) + + span_index: int | None = Field( + None, + description=( + "Which of the step's spans this annotation is, 0-based. None " + "means the step has no spans at all and this is its only row." + ), + ) + span_count: int = Field( + 0, description="How many spans the step has, so a reader can tell rows apart." + ) + span_source: str | None = Field( + None, description="Slug of the source the span refers into." + ) + span_role: str | None = Field( + None, description="performance | instruction | closeup | ... (open vocab)." + ) + span_open_ended: bool = Field( + False, + description=( + "The span recorded a start but no end — a deep link, not an " + "interval. The reference interval is then a point at the start, " + "and this flag is what keeps that distinct from a zero-length span." + ), + ) + span_label: str | None = Field(None, description="Short label for this span.") + span_caption: str | None = Field( + None, description="Prose about THIS span, not about the step." + ) + span_confidence: float | None = Field( + None, ge=0.0, le=1.0, description="Confidence in this span specifically." + ) + span_attrs: dict[str, Any] = Field( + default_factory=dict, description="Open bag on the span." + ) + + +class GuideCueBodyV1(BodyBase): + """A landmark the learner can hear or watch for, anchored to a step.""" + + doc_id: str = Field(..., description="Which guide this cue belongs to.") + ordinal: int = Field( + ..., description="Position in the document's cue list, 0-based." + ) + cue_id: str = Field(..., description="Stable id within the guide.") + kind: str = Field(..., description="lyric | count | audio-landmark | caption | ...") + text: str = Field(..., description="What the learner hears or reads.") + step_id: str = Field(..., description="The step this cue is anchored to.") + offset_value: Decimal | None = Field( + None, + description=( + "Offset into the step, in the domain unit. RELATIVE, always — an " + "absolute cue time is computed by ``resolve()``, never stored." + ), + ) + offset_unit: str | None = Field(None, description="Unit of ``offset_value``.") + duration_value: Decimal | None = Field( + None, description="How long the cue lasts, in the domain unit." + ) + duration_unit: str | None = Field(None, description="Unit of ``duration_value``.") + source: str | None = Field( + None, description="Which source the cue was heard in, when known." + ) + at_resolved: bool = Field( + False, + description=( + "True when the cue's wall-clock position in that source is known, " + "in which case the reference is a point at it. False means the " + "annotation sits at tick 0 because there is nothing better — the " + "one bit that keeps 'at second zero' distinct from 'unknown'." + ), + ) + attrs: dict[str, Any] = Field(default_factory=dict, description="Open bag.") + + +register_body_schema(STEP_BODY_SCHEMA_URI, GuideStepBodyV1) +register_body_schema(CUE_BODY_SCHEMA_URI, GuideCueBodyV1) diff --git a/paces/edits.py b/paces/edits.py index 04b2e2b..30efc56 100644 --- a/paces/edits.py +++ b/paces/edits.py @@ -45,9 +45,11 @@ a merge across multiple regenerations; only the most recent ``/id`` lock is consulted (issue #12). -Not yet recorded anywhere: the fresh values a merge *rejects* -(``Origin.value_digest`` and the op-log arrive with the evidence layer, -issue #4). +Not yet recorded anywhere: the fresh values a merge *rejects*. The evidence +they would be recorded against now exists — ``paces.evidence`` fills +``Origin.annotation_id`` / ``Origin.value_digest`` on every projected step +(issue #4) — but wiring the *rejected* value into the merge is document-side +work and belongs with the rest of issue #12. """ from __future__ import annotations diff --git a/paces/evidence.py b/paces/evidence.py new file mode 100644 index 0000000..9b15c14 --- /dev/null +++ b/paces/evidence.py @@ -0,0 +1,1208 @@ +"""The evidence layer: persist analysis into a ``lacing`` store, project it back. + +Two layers, one direction of flow (``docs/07 §6.0``). The committed +:class:`~paces.model.StepDocument` is the contract renderers read; the +machine evidence — the speech/music split, the metric grid, the beats, the +transcript, the step candidates, the derivation recipes and the lineage +between them — lives in a ``lacing`` store, and the document is a +*projection* of it. **The document is derivable from the store; the store is +not derivable from the document.** + +Three functions:: + + seg = segment(...) # analysis + write = to_store(seg, store=store, asset_id=asset) # evidence ← here + doc = from_store(store, asset_id=asset) # projection ← here + dumps_document(doc) == dumps_document(write.document) + +What crosses into the store, and what deliberately does not: + +============================ ========================================== +crosses stays on the document +============================ ========================================== +sources, passes, the grid, ``locks``, ``questions``, ``artifacts``, +beats, transcript words, span ``excerpt`` windows +steps + their spans, cues, +derivation recipes +============================ ========================================== + +The right column is not an omission. Those four are *document-layer* +records — a human edit, an open question, a built file, a hand-picked loop — +and ``docs/07 §6.0`` is explicit that nothing flows document → store. +:func:`from_store` therefore returns them empty, which is exactly what +:func:`paces.projection.to_document` emits, so the round trip is exact. + +Three properties this module is built to hold, each with a test: + +- **Deterministic identity.** An annotation's id is + ``uuid5(PACES_NAMESPACE, f"{tier}\\n{evidence_key}")``, not a fresh + ``uuid4`` plus a procedural adoption pass. ``docs/07 §6.4`` steps 1–3 then + hold by construction: a re-derivation of the same evidence *is* the same + id, so ``was_derived_from`` edges keep resolving and ``to_store`` is + reproducible from a fresh store. +- **Idempotence, digest-gated.** A row whose + :func:`lacing.annotation_value_digest` is unchanged is left *completely* + alone — provenance included — so ``generated_at_time`` does not churn and + freshness does not fire. A second :func:`to_store` of the same input + writes nothing. +- **Re-derivation, not accumulation.** A row this run superseded — a step + that no longer exists, a cue that was dropped — is removed, scoped to this + asset, this guide, and the tiers this call actually wrote to. `reelee`'s + phrasing: *"the graph is a set of nodes whose values are re-derivable, not + an append-only log"*. A stale row left behind would come back out of + :func:`from_store` as a step nobody analysed. ``prune=False`` opts out. +- **No floats across the boundary.** Every time crosses as + ``RationalTime`` ticks at ``rate=`` (lacing non-negotiable #1); + :meth:`RationalTime.from_seconds` raises rather than rounding, so a + document carrying finer time than the rate can hold is a loud error with + ``rate=`` as the answer. Reading back yields the canonical minimal decimal, + so a hand-written ``"231.30"`` normalises to ``"231.3"``. + +Intervals live on ``Annotation.reference`` and never in a body: these are +standoff annotations, and a body that carried its own times would be a +second, unindexed interval store. + +Requires the ``[lacing]`` extra. ``import paces`` does not import this +module — the core stays pydantic-only (ADR-0004); ``paces.to_store`` and +``paces.from_store`` resolve it on first attribute access. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable, Iterator, Mapping, Sequence +from dataclasses import dataclass +from fractions import Fraction +from typing import Any +from uuid import NAMESPACE_URL, UUID, uuid5 + +try: # pragma: no cover - the message is the point, not the branch + from lacing import ( + DEFAULT_RATE, + Annotation, + MediaRef, + Provenance, + RationalTime, + Tier, + TierStereotype, + TimeInterval, + annotation_value_digest, + ) +except ImportError as error: # pragma: no cover + raise ImportError( + "paces' evidence layer needs lacing: `pip install 'paces[lacing]'`. " + "The rest of paces does not — the core is pydantic-only (ADR-0004)." + ) from error + +from paces.bodies import ( + BEAT_BODY_SCHEMA_URI, + BEAT_TIER, + CUE_BODY_SCHEMA_URI, + CUE_TIER, + DOC_BODY_SCHEMA_URI, + DOC_TIER, + GRID_BODY_SCHEMA_URI, + GRID_TIER, + PASS_BODY_SCHEMA_URI, + PASS_TIER, + RECIPE_BODY_SCHEMA_URI, + RECIPE_TIER, + SOURCE_BODY_SCHEMA_URI, + SOURCE_TIER, + STEP_BODY_SCHEMA_URI, + STEP_TIER, + SUB_STEP_TIER, + WORD_BODY_SCHEMA_URI, + WORD_TIER, + GuideBeatBodyV1, + GuideCueBodyV1, + GuideDocBodyV1, + GuideGridBodyV1, + GuidePassBodyV1, + GuideRecipeBodyV1, + GuideSourceBodyV1, + GuideStepBodyV1, +) +from paces.model import ( + SCHEMA_VERSION, + Anchor, + Cue, + Measure, + MetricGrid, + Origin, + Source, + SourceSpan, + Step, + StepDocument, +) +from paces.projection import to_document +from paces.segmenters import Segmentation + +#: The namespace every paces annotation id is derived under. Derived from the +#: repository URL rather than typed as a literal, so it is reproducible from +#: something a reader can check. +PACES_NAMESPACE = uuid5(NAMESPACE_URL, "https://github.com/thorwhalen/paces") + +#: Default ``provenance.was_generated_by`` for rows that are not a step's own +#: (a step's carries its ``Origin.generated_by`` instead). +DFLT_GENERATED_BY = "processor:paces.to_store" + +#: Default ``provenance.was_attributed_to``. +DFLT_ATTRIBUTED_TO = "paces" + +#: The tiers this module writes, with the ELAN stereotypes of ``docs/07 §6.3``. +#: Registered on the store by :func:`register_tiers`, which :func:`to_store` +#: calls — a store that cannot hold tiers (the Protocol makes them optional) +#: is used without them rather than refused. +TIERS: tuple[Tier, ...] = ( + Tier(DOC_TIER), + Tier(SOURCE_TIER), + Tier(PASS_TIER), + Tier(GRID_TIER), + Tier(BEAT_TIER), + Tier(WORD_TIER), + Tier(STEP_TIER), + Tier(SUB_STEP_TIER, stereotype=TierStereotype.INCLUDED_IN, parent=STEP_TIER), + Tier(CUE_TIER, stereotype=TierStereotype.INCLUDED_IN, parent=STEP_TIER), + Tier( + RECIPE_TIER, + stereotype=TierStereotype.SYMBOLIC_ASSOCIATION, + parent=STEP_TIER, + ), +) + +_HEX64 = re.compile(r"^[0-9a-f]{64}$") + + +# ── time at the boundary ──────────────────────────────────────────────────── + + +def _ticks(value: str, *, rate: int) -> RationalTime: + """A decimal-string second count as exact ticks. Lossy input raises.""" + return RationalTime.from_seconds(value, rate) + + +def _decimal(time: RationalTime) -> str: + """The canonical minimal decimal string for *time*. + + Exact: the tick rate's denominator is factored into 2s and 5s and the + numerator scaled by the larger power of ten, so nothing is rounded. A + rate whose denominator survives that (a third of a second) has no finite + decimal and says so rather than rounding into the document. + + >>> _decimal(RationalTime.from_seconds('51.2', 24000)) + '51.2' + >>> _decimal(RationalTime.from_seconds('72', 24000)) + '72' + """ + fraction = time.to_fraction() + numerator, denominator = fraction.numerator, fraction.denominator + rest, twos, fives = denominator, 0, 0 + while rest % 2 == 0: + rest //= 2 + twos += 1 + while rest % 5 == 0: + rest //= 5 + fives += 1 + if rest != 1: + raise ValueError( + f"{time!r} is {fraction} seconds, which has no finite decimal form; " + "the document's wire is decimal strings, so this tick rate cannot " + "be projected without rounding" + ) + places = max(twos, fives) + scaled = numerator * 10**places // denominator + sign = "-" if scaled < 0 else "" + digits = str(abs(scaled)).rjust(places + 1, "0") + if places == 0: + return sign + digits + return sign + f"{digits[:-places]}.{digits[-places:]}".rstrip("0").rstrip(".") + + +def _interval(start: str, end: str | None, *, rate: int) -> TimeInterval: + """The half-open span for a document span. ``end=None`` → a point.""" + begin = _ticks(start, rate=rate) + return TimeInterval(begin, begin if end is None else _ticks(end, rate=rate)) + + +def _point(seconds: str | None, *, rate: int) -> TimeInterval: + """A point interval, at *seconds* or at tick 0 when there is nothing better.""" + at = RationalTime.zero(rate) if seconds is None else _ticks(seconds, rate=rate) + return TimeInterval.point(at) + + +def _asset_lineage(asset_id: str) -> list[str]: + """The asset as a ``was_derived_from`` ref, when it is one. + + ``Provenance`` accepts only a bare 64-hex SHA-256 as an artifact ref + (``lacing.model.AssetId``). A ``MediaRef.asset_id`` is free-form, so a + project keyed by a readable name still works — it just contributes no + lineage edge, rather than raising. + """ + return [asset_id] if _HEX64.match(asset_id) else [] + + +# ── writing ───────────────────────────────────────────────────────────────── + + +@dataclass(frozen=True, slots=True) +class StoreWrite: + """What :func:`to_store` did, and the document it projects to. + + ``document`` is the projection with its evidence back-references filled + in: every :class:`~paces.model.Origin` carries the ``annotation_id`` and + ``value_digest`` of the row it came from. That is the document + :func:`from_store` reproduces byte for byte. + """ + + document: StepDocument + written: int # rows that did not exist + updated: int # rows whose value digest changed + unchanged: int # rows left completely alone — the idempotence count + removed: int # rows this re-derivation superseded and dropped + annotation_ids: tuple[str, ...] # every row touched, in write order + + @property + def total(self) -> int: + return self.written + self.updated + self.unchanged + + +class CollidingEvidenceKey(ValueError): + """Two rows in one run would derive the same annotation id.""" + + +class _Writer: + """Digest-gated upsert over one store. Not part of the public surface.""" + + def __init__(self, store, *, attributed_to: str, generated_at: RationalTime): + self._store = store + self._attributed_to = attributed_to + self._generated_at = generated_at + self._index: dict[UUID, Annotation] = {a.id: a for a in store.all()} + self.written = 0 + self.updated = 0 + self.unchanged = 0 + self.removed = 0 + self.ids: list[str] = [] + self.touched: set[UUID] = set() + + def put( + self, + *, + tier: str, + key: str, + interval: TimeInterval, + asset_id: str, + body: Mapping[str, Any], + schema_uri: str, + generated_by: str = DFLT_GENERATED_BY, + derived_from: Iterable[Any] = (), + confidence: float | None = None, + ) -> tuple[UUID, str]: + """Upsert one annotation; return its ``(id, value_digest)``.""" + annotation_id = uuid5(PACES_NAMESPACE, f"{tier}\n{key}") + if annotation_id in self.touched: + raise CollidingEvidenceKey( + f"two {tier!r} rows in one run share the evidence key {key!r}. " + "Derived ids make that a silent overwrite, so it is refused " + "here instead. Two spans with the same (source, role, start) " + "are the usual cause — `validate_document` reports the same " + "collision on the document side." + ) + candidate = Annotation( + id=annotation_id, + tier=tier, + reference=MediaRef(asset_id=asset_id, interval=interval), + body=dict(body), + body_schema_uri=schema_uri, + provenance=Provenance( + was_generated_by=generated_by, + was_attributed_to=self._attributed_to, + was_derived_from=list(derived_from), + generated_at_time=self._generated_at, + activity="derive", + ), + confidence=confidence, + ) + digest = annotation_value_digest(candidate) + existing = self._index.get(annotation_id) + if existing is not None: + if annotation_value_digest(existing) == digest: + self.unchanged += 1 + self._record(annotation_id) + return annotation_id, digest + self._store.remove(annotation_id) + self.updated += 1 + else: + self.written += 1 + self._store.add(candidate) + self._index[annotation_id] = candidate + self._record(annotation_id) + return annotation_id, digest + + def _record(self, annotation_id: UUID) -> None: + self.ids.append(str(annotation_id)) + self.touched.add(annotation_id) + + def prune(self, *, asset_id: str, doc_id: str, tiers: set[str]) -> None: + """Drop rows on *tiers* this run superseded — a re-derivation, not a merge. + + Scoped three ways so it can only ever remove what this call is + authoritative for: the asset it was given, the guide it was given + (a row carrying another guide's ``doc_id`` is left alone), and the + tiers this call actually wrote to. + """ + for annotation in list(self._store.all()): + if annotation.tier not in tiers or annotation.id in self.touched: + continue + if getattr(annotation.reference, "asset_id", None) != asset_id: + continue + if annotation.body.get("doc_id") not in (None, doc_id): + continue + self._store.remove(annotation.id) + self._index.pop(annotation.id, None) + self.removed += 1 + + +def register_tiers(store) -> None: + """Declare :data:`TIERS` on *store*, if it keeps tiers at all. + + ``IntervalAnnotationStore`` is a Protocol: a dict-shaped store that + conforms structurally may not implement ``add_tier``. Tier metadata is + descriptive, so its absence is not a reason to refuse the store. + """ + add_tier = getattr(store, "add_tier", None) + if add_tier is None: + return + for tier in TIERS: + add_tier(tier) + + +def to_store( + seg: Segmentation, + *, + store, + asset_id: str, + doc_id: str = "guide", + title: str = "", + source=None, + domain: str = "generic", + lang: str = "en", + cues: Sequence[Cue | Mapping[str, Any]] = (), + passes: Sequence[Sequence[Any]] = (), + beats: Sequence[float | str] = (), + transcript: Sequence[Mapping[str, Any] | Sequence[Any]] = (), + recipes: Mapping[str, Any] | Any = None, + detector: str = "", + attributed_to: str = DFLT_ATTRIBUTED_TO, + generated_at: RationalTime | None = None, + prune: bool = True, + rate: int = DEFAULT_RATE, +) -> StoreWrite: + """Persist an analysis result into *store* as standoff annotations. + + *seg* is the analysis result; the projection arguments (*doc_id*, + *title*, *source*, *domain*, *lang*) are :func:`paces.to_document`'s, + unchanged. Everything else is evidence a segmenter does not carry on + :class:`~paces.segmenters.Segmentation` but a pipeline may have: + + Args: + store: Any ``lacing.IntervalAnnotationStore`` — ``MemoryStore()`` for + tests, ``SqliteStore`` for a project sidecar, or a ``dol`` store + that conforms. Injected, never constructed here: nothing in paces + decides where a user's evidence lives. + asset_id: Identity of the media every reference points into. A bare + 64-hex SHA-256 also becomes a lineage edge; any other string is + a usable key that simply contributes no edge. + cues: :class:`~paces.model.Cue` records (or their wire mappings). + Cue detection is analysis (``docs/07 §6.3``), and a + ``Segmentation`` has nowhere to carry it, so it arrives here. + passes: ``(start_s, end_s, label)`` triples — exactly the shape of + ``paces.measure.GridMeasurement.evidence['segments']``. + beats: Measured beat times in seconds. Written **only** when given; + beats are never synthesised from the grid (see the module + docstring of ``paces/bodies/signal.py``). + transcript: ``(start_s, end_s, text)`` triples or + ``{'start', 'end', 'text', 'speaker'}`` mappings, landing on + lacing's own ``word/v1``. + recipes: A ``paces.derivation.RecipesFile``, or its ``entries`` + mapping of span address → recipe. + detector: What produced *passes*, recorded on each pass row. + generated_at: ``provenance.generated_at_time``; defaults to now. + Unchanged rows keep the timestamp they already had, so this only + stamps rows that actually changed. + prune: Drop this guide's rows that this run superseded — a step that + no longer exists, a cue that was dropped. On by default, because + ``to_store`` is a **re-derivation** of the guide, not a merge + into it: a stale row left behind would be resurrected by + :func:`from_store` as a step nobody analysed. Only tiers this + call actually wrote to are pruned, and only for this asset and + this ``doc_id``. Pass ``False`` to accumulate instead. + rate: Ticks per second for every interval. Raises rather than + rounding if a time cannot be represented exactly. + + Returns: + A :class:`StoreWrite` whose ``document`` is the projection with real + ``Origin`` back-references. + """ + document = to_document( + seg, doc_id=doc_id, title=title, source=source, domain=domain, lang=lang + ) + if cues: + document = document.model_copy(update={"cues": [_as_cue(cue) for cue in cues]}) + + register_tiers(store) + writer = _Writer( + store, + attributed_to=attributed_to, + generated_at=generated_at or RationalTime.now(), + ) + base = _asset_lineage(asset_id) + put = _bind_put(writer, asset_id=asset_id) + + put( + tier=DOC_TIER, + key=f"doc:{doc_id}", + interval=_point(None, rate=rate), + body=GuideDocBodyV1( + doc_id=document.id, + title=document.title, + lang=document.lang, + domain=document.domain, + schema_version=document.schema_version, + credits=document.credits, + method=seg.method, + flags=list(seg.flags), + attrs=dict(document.attrs), + ).model_dump(mode="json"), + schema_uri=DOC_BODY_SCHEMA_URI, + derived_from=base, + ) + + for ordinal, src in enumerate(document.sources): + put( + tier=SOURCE_TIER, + key=f"source:{doc_id}/{src.id}", + interval=_point(None, rate=rate), + body=GuideSourceBodyV1( + doc_id=doc_id, + ordinal=ordinal, + source_id=src.id, + kind=src.kind, + uri=src.uri, + asset_id=src.asset_id, + duration_s=src.duration_s, + title=src.title, + attribution=src.attribution, + rights=src.rights, + attrs=dict(src.attrs), + ).model_dump(mode="json"), + schema_uri=SOURCE_BODY_SCHEMA_URI, + derived_from=base, + ) + + written_passes = _write_passes(put, passes, detector=detector, base=base, rate=rate) + grid_id = _write_grid( + put, + document.metric, + doc_id=doc_id, + method=seg.method, + beat_count=len(beats) or None, + span=_document_span(document), + base=[*base, *(annotation_id for annotation_id, _, _ in written_passes)], + rate=rate, + ) + _write_beats(put, beats, base=[grid_id, *base] if grid_id else base, rate=rate) + _write_transcript(put, transcript, base=base, rate=rate) + + step_base = [grid_id, *base] if grid_id else list(base) + step_rows: dict[str, tuple[UUID, str]] = {} + span_rows: dict[str, tuple[UUID, str]] = {} + steps = _write_steps( + put, + document.steps, + doc_id=doc_id, + parent_id=None, + parent_ann=None, + base=step_base, + passes=written_passes, + rate=rate, + step_rows=step_rows, + span_rows=span_rows, + ) + _write_cues(put, document.cues, doc_id=doc_id, step_rows=step_rows, rate=rate) + _write_recipes( + put, recipes, doc_id=doc_id, span_rows=span_rows, base=base, rate=rate + ) + + if prune: + writer.prune( + asset_id=asset_id, + doc_id=doc_id, + tiers=_prunable_tiers( + passes=passes, + beats=beats, + transcript=transcript, + cues=document.cues, + recipes=recipes, + ), + ) + + return StoreWrite( + document=document.model_copy(update={"steps": steps}), + written=writer.written, + updated=writer.updated, + unchanged=writer.unchanged, + removed=writer.removed, + annotation_ids=tuple(writer.ids), + ) + + +#: Tiers every ``to_store`` writes, because they are projected from the +#: ``Segmentation`` itself — a re-derivation is authoritative over all of them. +_ALWAYS_WRITTEN_TIERS = frozenset( + {DOC_TIER, SOURCE_TIER, GRID_TIER, STEP_TIER, SUB_STEP_TIER} +) + + +def _prunable_tiers(*, passes, beats, transcript, cues, recipes) -> set[str]: + """Which tiers this call may prune: the ones it is authoritative over. + + A tier fed by a keyword is prunable only when that keyword was supplied. + Omitting ``passes=`` therefore leaves an earlier speech/music split + standing rather than silently deleting it — "a re-derivation replaces + what it re-derives", and it re-derives what it was given. + """ + tiers = set(_ALWAYS_WRITTEN_TIERS) + for supplied, tier in ( + (passes, PASS_TIER), + (beats, BEAT_TIER), + (transcript, WORD_TIER), + (cues, CUE_TIER), + (recipes, RECIPE_TIER), + ): + if supplied: + tiers.add(tier) + return tiers + + +def _bind_put(writer: _Writer, *, asset_id: str): + """``writer.put`` with the asset pinned — every reference names one asset.""" + + def put(**kwargs) -> tuple[UUID, str]: + return writer.put(asset_id=asset_id, **kwargs) + + return put + + +def _as_cue(cue: Cue | Mapping[str, Any]) -> Cue: + return cue if isinstance(cue, Cue) else Cue.model_validate(dict(cue)) + + +def _document_span(document: StepDocument) -> tuple[str, str] | None: + """The outermost ``(start, end)`` any span in *document* covers.""" + starts, ends = [], [] + for step in _walk(document.steps): + for span in step.spans: + starts.append(Fraction(span.start)) + ends.append(Fraction(span.end if span.end is not None else span.start)) + if not starts: + return None + return _fraction_decimal(min(starts)), _fraction_decimal(max(ends)) + + +def _fraction_decimal(value: Fraction) -> str: + """A ``Fraction`` of seconds back as a decimal string, exactly.""" + return _decimal(RationalTime(value.numerator, value.denominator)) + + +def _walk(steps: Sequence[Step]) -> Iterator[Step]: + for step in steps: + yield step + yield from _walk(step.steps) + + +def _write_passes( + put, passes, *, detector: str, base, rate: int +) -> list[tuple[UUID, TimeInterval, str]]: + written = [] + for row in passes: + start, end, label = _pass_row(row) + interval = _interval(start, end, rate=rate) + annotation_id, _ = put( + tier=PASS_TIER, + key=f"pass:{label}/{start}/{end}", + interval=interval, + body=GuidePassBodyV1(label=label, detector=detector).model_dump( + mode="json" + ), + schema_uri=PASS_BODY_SCHEMA_URI, + derived_from=base, + ) + written.append((annotation_id, interval, label)) + return written + + +def _pass_row(row) -> tuple[str, str, str]: + if isinstance(row, Mapping): + start, end, label = row["start"], row["end"], row.get("label", "") + else: + start, end, label = (list(row) + [""])[:3] + return _seconds(start), _seconds(end), str(label or "") + + +def _seconds(value) -> str: + """A time as a decimal string, whatever the caller had it as. + + Floats are accepted *here* and only here — they are what the analysis + libraries return — and are converted through ``repr`` so no precision is + invented, then re-normalised by the tick round trip. + """ + if isinstance(value, str): + return value + return _fraction_decimal(Fraction(str(value))) + + +def _write_grid( + put, + grid: MetricGrid | None, + *, + doc_id: str, + method: str, + beat_count: int | None, + span: tuple[str, str] | None, + base, + rate: int, +) -> UUID | None: + if grid is None: + return None + interval = ( + _point(None, rate=rate) + if span is None + else _interval(span[0], span[1], rate=rate) + ) + annotation_id, _ = put( + tier=GRID_TIER, + key=f"grid:{doc_id}", + interval=interval, + body=GuideGridBodyV1( + doc_id=doc_id, + unit=grid.unit, + subdivisions=grid.subdivisions, + tempo_bpm=grid.tempo_bpm, + origin=grid.origin, + origin_source=grid.origin_source, + beat_count=beat_count, + method=method, + ).model_dump(mode="json"), + schema_uri=GRID_BODY_SCHEMA_URI, + derived_from=base, + ) + return annotation_id + + +def _write_beats(put, beats, *, base, rate: int) -> None: + for index, at in enumerate(beats): + seconds = _seconds(at) + put( + tier=BEAT_TIER, + key=f"beat:{seconds}", + interval=_point(seconds, rate=rate), + body=GuideBeatBodyV1(index=index).model_dump(mode="json"), + schema_uri=BEAT_BODY_SCHEMA_URI, + derived_from=base, + ) + + +def _write_transcript(put, transcript, *, base, rate: int) -> None: + for index, row in enumerate(transcript): + if isinstance(row, Mapping): + start, end = _seconds(row["start"]), _seconds(row["end"]) + text, speaker = row["text"], row.get("speaker") + else: + start, end, text = (list(row) + [None])[:3] + start, end, speaker = _seconds(start), _seconds(end), None + put( + tier=WORD_TIER, + key=f"word:{start}/{end}/{index}", + interval=_interval(start, end, rate=rate), + body={"text": text, "speaker": speaker}, + schema_uri=WORD_BODY_SCHEMA_URI, + derived_from=base, + ) + + +def _write_steps( + put, + steps: Sequence[Step], + *, + doc_id: str, + parent_id: str | None, + parent_ann: UUID | None, + base, + passes: Sequence[tuple[UUID, TimeInterval, str]], + rate: int, + step_rows: dict[str, tuple[UUID, str]], + span_rows: dict[str, tuple[UUID, str]], +) -> list[Step]: + """Write one sibling level, recurse, and return the steps with back-refs.""" + tier = STEP_TIER if parent_id is None else SUB_STEP_TIER + out: list[Step] = [] + for ordinal, step in enumerate(steps): + generated_by = ( + step.origin.generated_by + if step.origin and step.origin.generated_by + else DFLT_GENERATED_BY + ) + confidence = step.origin.confidence if step.origin else None + lineage = [*base] if parent_ann is None else [parent_ann, *base] + rows: list[tuple[UUID, str]] = [] + for span_index, span in _spans_of(step): + key = _span_key(doc_id, step.id, span) + interval = ( + _point(None, rate=rate) + if span is None + else _interval(span.start, span.end, rate=rate) + ) + annotation_id, digest = put( + tier=tier, + key=key, + interval=interval, + body=_step_body( + step, + doc_id=doc_id, + parent_id=parent_id, + ordinal=ordinal, + span_index=span_index, + span=span, + ), + schema_uri=STEP_BODY_SCHEMA_URI, + generated_by=generated_by, + derived_from=[*lineage, *_covering(passes, interval)], + confidence=confidence, + ) + rows.append((annotation_id, digest)) + if span is not None: + span_rows[_span_address(step.id, span)] = (annotation_id, digest) + head = rows[0] + step_rows[step.id] = head + children = _write_steps( + put, + step.steps, + doc_id=doc_id, + parent_id=step.id, + parent_ann=head[0], + base=base, + passes=passes, + rate=rate, + step_rows=step_rows, + span_rows=span_rows, + ) + out.append( + step.model_copy( + update={ + "steps": children, + "origin": _with_backref( + step.origin, head, generated_by=generated_by + ), + } + ) + ) + return out + + +def _spans_of(step: Step) -> list[tuple[int | None, SourceSpan | None]]: + """``(span_index, span)`` pairs — one ``(None, None)`` row when spanless. + + A step with no spans is still a step: it gets exactly one annotation, so + the store can hold "found, extent not recorded" rather than dropping it. + """ + if not step.spans: + return [(None, None)] + return list(enumerate(step.spans)) + + +def _span_address(step_id: str, span: SourceSpan) -> str: + """``{step_id}/{source}/{role}/{start}`` — ``paces.derivation``'s key.""" + return f"{step_id}/{span.source}/{span.role}/{span.start}" + + +def _span_key(doc_id: str, step_id: str, span: SourceSpan | None) -> str: + """The evidence key one ``(step, span)`` row is identified by. + + The step id is in it even when the span is not: a spanless step is still + its own row, and keying every spanless step on the same string would + collapse them all onto one annotation id. + """ + address = f"{step_id}/-" if span is None else _span_address(step_id, span) + return f"step:{doc_id}/{address}" + + +def _step_body( + step: Step, + *, + doc_id: str, + parent_id: str | None, + ordinal: int, + span_index: int | None, + span: SourceSpan | None, +) -> dict: + return GuideStepBodyV1( + doc_id=doc_id, + step_id=step.id, + parent_step_id=parent_id, + ordinal=ordinal, + name=step.name, + description=step.description, + duration_value=step.duration.value, + duration_unit=step.duration.unit, + repeat=step.repeat, + optional=step.optional, + variant_of=step.variant_of, + tags=list(step.tags), + attrs=dict(step.attrs), + span_index=span_index, + span_count=len(step.spans), + span_source=None if span is None else span.source, + span_role=None if span is None else span.role, + span_open_ended=span is not None and span.end is None, + span_label=None if span is None else span.label, + span_caption=None if span is None else span.caption, + span_confidence=None if span is None else span.confidence, + span_attrs={} if span is None else dict(span.attrs), + ).model_dump(mode="json") + + +def _covering(passes: Sequence[tuple[UUID, TimeInterval, str]], interval) -> list[UUID]: + """The pass rows a span sits inside. + + Lineage, not decoration: this is the edge that makes a re-run of the + speech/music split mark the steps it framed stale + (``nw.freshness.stale_after`` walks ``was_derived_from``). Containment is + tested on the interval's own endpoints — an exact test through + ``lacing.allen`` would need the span strictly inside, and a step that + starts exactly where the music does is still framed by it. + """ + return [ + annotation_id + for annotation_id, span, _ in passes + if span.start <= interval.start and interval.end <= span.end + ] + + +def _with_backref( + origin: Origin | None, row: tuple[UUID, str], *, generated_by: str +) -> Origin: + """The step's ``Origin`` with its evidence back-reference filled in. + + A step that arrived without an ``Origin`` leaves with one: the store now + knows where it came from, and saying so is the point of the field. + """ + annotation_id, digest = row + fields = {} if origin is None else origin.model_dump() + return Origin( + **{ + **fields, + "generated_by": generated_by, + "annotation_id": str(annotation_id), + "value_digest": digest, + } + ) + + +def _write_cues(put, cues: Sequence[Cue], *, doc_id: str, step_rows, rate: int) -> None: + for ordinal, cue in enumerate(cues): + parent = step_rows.get(cue.anchor.step) + put( + tier=CUE_TIER, + key=f"cue:{doc_id}/{cue.id}", + interval=_point(cue.at_s, rate=rate), + body=GuideCueBodyV1( + doc_id=doc_id, + ordinal=ordinal, + cue_id=cue.id, + kind=cue.kind, + text=cue.text, + step_id=cue.anchor.step, + offset_value=None + if cue.anchor.offset is None + else cue.anchor.offset.value, + offset_unit=None + if cue.anchor.offset is None + else cue.anchor.offset.unit, + duration_value=None if cue.duration is None else cue.duration.value, + duration_unit=None if cue.duration is None else cue.duration.unit, + source=cue.source, + at_resolved=cue.at_s is not None, + attrs=dict(cue.attrs), + ).model_dump(mode="json"), + schema_uri=CUE_BODY_SCHEMA_URI, + derived_from=[] if parent is None else [parent[0]], + ) + + +def _write_recipes(put, recipes, *, doc_id: str, span_rows, base, rate: int) -> None: + for address, recipe in _recipe_entries(recipes): + entry = ( + recipe if isinstance(recipe, Mapping) else recipe.model_dump(mode="json") + ) + window = entry["window"] + parent = span_rows.get(address) + put( + tier=RECIPE_TIER, + key=f"derivation:{doc_id}/{address}", + interval=_interval(window[0], window[1], rate=rate), + body=GuideRecipeBodyV1( + doc_id=doc_id, + span_key=address, + step_id=address.split("/", 1)[0], + box=None if entry.get("box") is None else list(entry["box"]), + frame_width=entry["frameWidth"] + if "frameWidth" in entry + else entry["frame_width"], + frame_height=entry["frameHeight"] + if "frameHeight" in entry + else entry["frame_height"], + source_asset_id=entry.get( + "sourceAssetId", entry.get("source_asset_id") + ), + locator=entry["locator"], + params=dict(entry.get("params", {})), + locked=bool(entry.get("locked", False)), + media_digest=entry.get("mediaDigest", entry.get("media_digest")), + ).model_dump(mode="json"), + schema_uri=RECIPE_BODY_SCHEMA_URI, + derived_from=[*([] if parent is None else [parent[0]]), *base], + ) + + +def _recipe_entries(recipes) -> Iterator[tuple[str, Any]]: + if recipes is None: + return + entries = getattr(recipes, "entries", None) + if entries is None and isinstance(recipes, Mapping): + entries = recipes.get("entries", recipes) + yield from (entries or {}).items() + + +# ── reading back ──────────────────────────────────────────────────────────── + + +class DocumentNotInStore(KeyError): + """No guide in this store matches the asset (and doc id) asked for.""" + + +class AmbiguousDocument(ValueError): + """Several guides share the asset and no ``doc_id`` said which.""" + + +def from_store(store, *, asset_id: str, doc_id: str | None = None) -> StepDocument: + """Project the guide held in *store* back into a :class:`StepDocument`. + + The inverse of :func:`to_store`: given the same store, it reproduces + ``StoreWrite.document`` byte for byte under + :func:`paces.model.dumps_document`. Document-layer records (``locks``, + ``questions``, ``artifacts``, span ``excerpt`` windows) come back empty, + because they never went in — see the module docstring. + + Args: + store: The ``lacing`` store written by :func:`to_store`. + asset_id: Which media's annotations to read. + doc_id: Which guide, when the store holds more than one for that + asset. Optional when there is exactly one. + + Raises: + DocumentNotInStore: nothing matches. + AmbiguousDocument: several guides match and none was named. + """ + rows = [ + annotation + for annotation in store.all() + if getattr(annotation.reference, "asset_id", None) == asset_id + ] + envelope = _envelope(rows, asset_id=asset_id, doc_id=doc_id) + guide_id = envelope.body["doc_id"] + + def mine(tier: str) -> list[Annotation]: + return [ + row + for row in rows + if row.tier == tier and row.body.get("doc_id") == guide_id + ] + + sources = [ + Source( + id=row.body["source_id"], + kind=row.body["kind"], + uri=row.body["uri"], + asset_id=row.body["asset_id"], + duration_s=row.body["duration_s"], + title=row.body["title"], + attribution=row.body["attribution"], + rights=row.body["rights"], + attrs=dict(row.body["attrs"]), + ) + for row in sorted(mine(SOURCE_TIER), key=lambda r: r.body["ordinal"]) + ] + grids = mine(GRID_TIER) + metric = _metric(grids[0]) if grids else None + step_rows = mine(STEP_TIER) + mine(SUB_STEP_TIER) + cues = [ + _cue(row) for row in sorted(mine(CUE_TIER), key=lambda r: r.body["ordinal"]) + ] + return StepDocument( + id=guide_id, + schema_version=envelope.body.get("schema_version", SCHEMA_VERSION), + title=envelope.body["title"], + lang=envelope.body["lang"], + domain=envelope.body["domain"], + credits=envelope.body["credits"], + metric=metric, + sources=sources, + steps=_steps(step_rows, parent_id=None), + cues=cues, + attrs=dict(envelope.body["attrs"]), + ) + + +def _envelope(rows, *, asset_id: str, doc_id: str | None) -> Annotation: + envelopes = [row for row in rows if row.tier == DOC_TIER] + if doc_id is not None: + envelopes = [row for row in envelopes if row.body["doc_id"] == doc_id] + if not envelopes: + raise DocumentNotInStore( + f"no guide for asset {asset_id!r}" + + ("" if doc_id is None else f" with doc_id {doc_id!r}") + ) + if len(envelopes) > 1: + found = sorted(row.body["doc_id"] for row in envelopes) + raise AmbiguousDocument( + f"asset {asset_id!r} holds {len(envelopes)} guides ({found}); " + "pass doc_id= to say which" + ) + return envelopes[0] + + +def _metric(row: Annotation) -> MetricGrid: + body = row.body + return MetricGrid( + unit=body["unit"], + subdivisions=body["subdivisions"], + tempo_bpm=body["tempo_bpm"], + origin=body["origin"], + origin_source=body["origin_source"], + ) + + +def _cue(row: Annotation) -> Cue: + body = row.body + offset = ( + None + if body["offset_value"] is None + else Measure(value=body["offset_value"], unit=body["offset_unit"]) + ) + duration = ( + None + if body["duration_value"] is None + else Measure(value=body["duration_value"], unit=body["duration_unit"]) + ) + return Cue( + id=body["cue_id"], + kind=body["kind"], + text=body["text"], + anchor=Anchor(step=body["step_id"], offset=offset), + duration=duration, + source=body["source"], + at_s=_decimal(row.reference.interval.start) if body["at_resolved"] else None, + attrs=dict(body["attrs"]), + ) + + +def _steps(rows: Sequence[Annotation], *, parent_id: str | None) -> list[Step]: + """Rebuild one sibling level from its annotation rows, then recurse.""" + level = [row for row in rows if row.body["parent_step_id"] == parent_id] + by_step: dict[str, list[Annotation]] = {} + for row in level: + by_step.setdefault(row.body["step_id"], []).append(row) + ordered = sorted(by_step.items(), key=lambda kv: kv[1][0].body["ordinal"]) + return [ + _step(step_id, sorted(step_rows, key=_span_order), rows=rows) + for step_id, step_rows in ordered + ] + + +def _span_order(row: Annotation) -> int: + index = row.body["span_index"] + return -1 if index is None else index + + +def _step(step_id: str, step_rows: Sequence[Annotation], *, rows) -> Step: + head = step_rows[0] + body = head.body + return Step( + id=step_id, + name=body["name"], + duration=Measure(value=body["duration_value"], unit=body["duration_unit"]), + description=body["description"], + spans=[_span(row) for row in step_rows if row.body["span_index"] is not None], + steps=_steps(rows, parent_id=step_id), + repeat=body["repeat"], + optional=body["optional"], + variant_of=body["variant_of"], + tags=list(body["tags"]), + origin=Origin( + annotation_id=str(head.id), + value_digest=annotation_value_digest(head), + generated_by=head.provenance.was_generated_by, + confidence=head.confidence, + ), + attrs=dict(body["attrs"]), + ) + + +def _span(row: Annotation) -> SourceSpan: + body = row.body + interval = row.reference.interval + return SourceSpan( + source=body["span_source"], + role=body["span_role"], + start=_decimal(interval.start), + end=None if body["span_open_ended"] else _decimal(interval.end), + label=body["span_label"], + caption=body["span_caption"], + confidence=body["span_confidence"], + attrs=dict(body["span_attrs"]), + ) + + +def recipes_from_store(store, *, asset_id: str, doc_id: str | None = None) -> dict: + """The derivation recipes held for one guide, as the sidecar's shape. + + ``{span address: recipe dict}`` — the same keys + ``paces.derivation.RecipesFile.entries`` uses, so a store round trip and + the committed sidecar are directly comparable. + """ + rows = [ + row + for row in store.all() + if row.tier == RECIPE_TIER + and getattr(row.reference, "asset_id", None) == asset_id + and (doc_id is None or row.body["doc_id"] == doc_id) + ] + return { + row.body["span_key"]: { + "box": row.body["box"], + "window": [ + _decimal(row.reference.interval.start), + _decimal(row.reference.interval.end), + ], + "frameWidth": row.body["frame_width"], + "frameHeight": row.body["frame_height"], + "sourceAssetId": row.body["source_asset_id"], + "locator": row.body["locator"], + "params": dict(row.body["params"]), + "locked": row.body["locked"], + "mediaDigest": row.body["media_digest"], + } + for row in rows + } diff --git a/paces/model.py b/paces/model.py index bd252b7..583e3ab 100644 --- a/paces/model.py +++ b/paces/model.py @@ -130,8 +130,9 @@ class SourceSpan(_Base): class ArtifactRef(_Base): """WHAT a derived file is, never HOW it was made — the recipe lives in the - evidence layer, keyed by ``asset_id`` (interim: the span-address-keyed - ``.recipes.json`` sidecar, ADR-0005 §3, until issue #4 lands).""" + evidence layer (``paces.evidence``, tier ``derivation``) and in the + committed span-address-keyed ``.recipes.json`` sidecar (ADR-0005 §3), + which stay in step: both are keyed by the same span address.""" role: Slug # clip | gif | poster | thumbnail | waveform | audio uri: str # relative to the document — deploy-portable diff --git a/pyproject.toml b/pyproject.toml index 846be54..d8dc41b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -99,6 +99,16 @@ pose = [ "rtmlib>=0.0.16,<0.1", "onnxruntime", ] +# The evidence layer (issue #4): analysis persisted as standoff annotations in +# a lacing store, with the StepDocument as its projection. MIT, and its own +# closure is pydantic + intervaltree + dol — nothing heavy, nothing copyleft. +# Floor 0.0.40: that release made wire-form time validation raise +# ValidationError rather than escaping a TypeError, and made `from_wire` refuse +# unknown keys — both of which this layer's round trip relies on to fail loudly +# instead of silently normalising. +lacing = [ + "lacing>=0.0.40", +] dev = [ "pytest>=7.0", "pytest-cov>=4.0", @@ -112,6 +122,9 @@ dev = [ # [media] (the >=0.0.39 floor wins over [audio]'s >=0.0.36). "mixing[beats,audio]>=0.0.39", "numba>=0.59", + # The evidence-layer tests run against real, dict-backed lacing stores — + # mirrored with [lacing] so they run rather than skip. + "lacing>=0.0.40", # tests/test_pose.py reads pyproject.toml to guard the licence perimeter; # stdlib tomllib is 3.11+, and a skipped guard guards nothing. "tomli>=2.0; python_version < '3.11'", @@ -166,7 +179,7 @@ install.windows = "choco install ffmpeg -y --no-progress" # -------------------------------------------------------------------------- [tool.wads.licence] enabled = false -include-extras = ["cli", "audio", "media", "pose"] +include-extras = ["cli", "audio", "media", "pose", "lacing"] [tool.hatch.build.targets.wheel] packages = ["paces"] diff --git a/tests/data/body_schemas.json b/tests/data/body_schemas.json new file mode 100644 index 0000000..daab7d5 --- /dev/null +++ b/tests/data/body_schemas.json @@ -0,0 +1,746 @@ +{ + "annot://schema/guide-beat/v1": { + "additionalProperties": false, + "description": "One measured beat. The time is the reference's point interval.", + "properties": { + "index": { + "description": "Position in the measured beat sequence, 0-based.", + "title": "Index", + "type": "integer" + } + }, + "required": [ + "index" + ], + "title": "GuideBeatBodyV1", + "type": "object" + }, + "annot://schema/guide-cue/v1": { + "additionalProperties": false, + "description": "A landmark the learner can hear or watch for, anchored to a step.", + "properties": { + "at_resolved": { + "default": false, + "description": "True when the cue's wall-clock position in that source is known, in which case the reference is a point at it. False means the annotation sits at tick 0 because there is nothing better — the one bit that keeps 'at second zero' distinct from 'unknown'.", + "title": "At Resolved", + "type": "boolean" + }, + "attrs": { + "additionalProperties": true, + "description": "Open bag.", + "title": "Attrs", + "type": "object" + }, + "cue_id": { + "description": "Stable id within the guide.", + "title": "Cue Id", + "type": "string" + }, + "doc_id": { + "description": "Which guide this cue belongs to.", + "title": "Doc Id", + "type": "string" + }, + "duration_unit": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Unit of ``duration_value``.", + "title": "Duration Unit" + }, + "duration_value": { + "anyOf": [ + { + "pattern": "^-?\\d+(\\.\\d+)?$", + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "How long the cue lasts, in the domain unit.", + "title": "Duration Value" + }, + "kind": { + "description": "lyric | count | audio-landmark | caption | ...", + "title": "Kind", + "type": "string" + }, + "offset_unit": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Unit of ``offset_value``.", + "title": "Offset Unit" + }, + "offset_value": { + "anyOf": [ + { + "pattern": "^-?\\d+(\\.\\d+)?$", + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Offset into the step, in the domain unit. RELATIVE, always — an absolute cue time is computed by ``resolve()``, never stored.", + "title": "Offset Value" + }, + "ordinal": { + "description": "Position in the document's cue list, 0-based.", + "title": "Ordinal", + "type": "integer" + }, + "source": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Which source the cue was heard in, when known.", + "title": "Source" + }, + "step_id": { + "description": "The step this cue is anchored to.", + "title": "Step Id", + "type": "string" + }, + "text": { + "description": "What the learner hears or reads.", + "title": "Text", + "type": "string" + } + }, + "required": [ + "doc_id", + "ordinal", + "cue_id", + "kind", + "text", + "step_id" + ], + "title": "GuideCueBodyV1", + "type": "object" + }, + "annot://schema/guide-doc/v1": { + "additionalProperties": false, + "description": "The guide's envelope: identity, presentation language, and the run.", + "properties": { + "attrs": { + "additionalProperties": true, + "description": "Namespaced open bag, as on the document.", + "title": "Attrs", + "type": "object" + }, + "credits": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Attribution line, when there is one.", + "title": "Credits" + }, + "doc_id": { + "description": "Stable slug of the guide within a project. Distinct from the annotation id, and the field that lets one store hold several guides over the same asset.", + "title": "Doc Id", + "type": "string" + }, + "domain": { + "default": "generic", + "description": "Domain slug: dance | recipe | workout | ...", + "title": "Domain", + "type": "string" + }, + "flags": { + "description": "The run's honesty flags (``Segmentation.flags``) — 'naming-abstained', 'origin-estimated: ...', and friends. Evidence about the run, never projected into the document.", + "items": { + "type": "string" + }, + "title": "Flags", + "type": "array" + }, + "lang": { + "default": "en", + "description": "BCP-47 language tag of the guide's prose.", + "title": "Lang", + "type": "string" + }, + "method": { + "default": "", + "description": "Which segmentation capability produced the steps (``Segmentation.method``). Empty when unrecorded.", + "title": "Method", + "type": "string" + }, + "schema_version": { + "description": "Version of the StepDocument schema this was projected at.", + "title": "Schema Version", + "type": "string" + }, + "title": { + "description": "Human title of the guide.", + "title": "Title", + "type": "string" + } + }, + "required": [ + "doc_id", + "title", + "schema_version" + ], + "title": "GuideDocBodyV1", + "type": "object" + }, + "annot://schema/guide-grid/v1": { + "additionalProperties": false, + "description": "How the domain unit relates to wall-clock time, over the region it governs.\n\nThe document's ``metric`` is projected straight back out of this — one\ngrid, one source of truth, no second copy in the document envelope.", + "properties": { + "beat_count": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "description": "How many beats were actually tracked, when the grid was measured rather than supplied. None means nobody counted.", + "title": "Beat Count" + }, + "doc_id": { + "description": "Which guide this grid governs. The grid is per-guide, not per-asset: two guides can count the same video differently.", + "title": "Doc Id", + "type": "string" + }, + "method": { + "default": "", + "description": "Which capability produced the grid ('grid-measured', 'grid-placed'). Empty when the caller supplied it outright.", + "title": "Method", + "type": "string" + }, + "origin": { + "anyOf": [ + { + "pattern": "^-?\\d+(\\.\\d+)?$", + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Seconds into ``origin_source`` where unit 0 starts.", + "title": "Origin" + }, + "origin_source": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Which source ``origin`` is measured in.", + "title": "Origin Source" + }, + "subdivisions": { + "default": 1, + "description": "Beats per unit (8 for an 8-count).", + "minimum": 1, + "title": "Subdivisions", + "type": "integer" + }, + "tempo_bpm": { + "anyOf": [ + { + "pattern": "^-?\\d+(\\.\\d+)?$", + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Tempo as a decimal string. None spells 'tempo unknown' — never '0'.", + "title": "Tempo Bpm" + }, + "unit": { + "description": "The domain unit counted: eight | bar | rep.", + "title": "Unit", + "type": "string" + } + }, + "required": [ + "doc_id", + "unit" + ], + "title": "GuideGridBodyV1", + "type": "object" + }, + "annot://schema/guide-pass/v1": { + "additionalProperties": false, + "description": "A stretch of one source with a single character (music, speech, ...).", + "properties": { + "detector": { + "default": "", + "description": "What produced the split ('mixing.find_segments:speech_music'). Empty when the caller supplied passes without saying.", + "title": "Detector", + "type": "string" + }, + "label": { + "description": "What this stretch is: 'music' | 'speech' | '' when the detector found a segment it would not name.", + "title": "Label", + "type": "string" + } + }, + "required": [ + "label" + ], + "title": "GuidePassBodyV1", + "type": "object" + }, + "annot://schema/guide-recipe/v1": { + "additionalProperties": false, + "description": "One span's derivation parameters, keyed by the span address.", + "properties": { + "box": { + "anyOf": [ + { + "items": { + "pattern": "^-?\\d+(\\.\\d+)?$", + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The resolved crop box (x, y, w, h) in pixels, as decimal strings. None means full frame — a recorded decision, not a gap.", + "title": "Box" + }, + "doc_id": { + "description": "Which guide this recipe belongs to.", + "title": "Doc Id", + "type": "string" + }, + "frame_height": { + "description": "Source frame height the box is in.", + "title": "Frame Height", + "type": "integer" + }, + "frame_width": { + "description": "Source frame width the box is in.", + "title": "Frame Width", + "type": "integer" + }, + "locator": { + "description": "Which subject locator resolved the box ('full_frame', ...).", + "title": "Locator", + "type": "string" + }, + "locked": { + "default": false, + "description": "A human set this box; reconcile must not re-derive it.", + "title": "Locked", + "type": "boolean" + }, + "media_digest": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "What the stored media was actually cut with (box + window + source hash). A hand-edited box or a re-timed window changes it and forces a re-encode instead of serving stale bytes.", + "title": "Media Digest" + }, + "params": { + "additionalProperties": { + "pattern": "^-?\\d+(\\.\\d+)?$", + "type": "string" + }, + "description": "Policy parameters the box was resolved under (aspect, pad). A change here re-locates an unlocked entry.", + "title": "Params", + "type": "object" + }, + "source_asset_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Hash of the media cut from. None = identity unverified.", + "title": "Source Asset Id" + }, + "span_key": { + "description": "The span address ``{step_id}/{source}/{role}/{start}`` — the same key ``paces.derivation`` writes into the committed sidecar, so the two layers join without a translation table.", + "title": "Span Key", + "type": "string" + }, + "step_id": { + "description": "The step this recipe derives media for.", + "title": "Step Id", + "type": "string" + } + }, + "required": [ + "doc_id", + "span_key", + "step_id", + "frame_width", + "frame_height", + "locator" + ], + "title": "GuideRecipeBodyV1", + "type": "object" + }, + "annot://schema/guide-source/v1": { + "additionalProperties": false, + "description": "One input the guide's spans refer into.", + "properties": { + "asset_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Content hash of a local copy, if any.", + "title": "Asset Id" + }, + "attribution": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Who made it.", + "title": "Attribution" + }, + "attrs": { + "additionalProperties": true, + "description": "Open bag.", + "title": "Attrs", + "type": "object" + }, + "doc_id": { + "description": "Which guide this source belongs to — one store can hold several guides over the same asset, and a projection has to read back only its own.", + "title": "Doc Id", + "type": "string" + }, + "duration_s": { + "anyOf": [ + { + "pattern": "^-?\\d+(\\.\\d+)?$", + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Source duration in seconds, as a decimal string.", + "title": "Duration S" + }, + "kind": { + "description": "video | audio | image | document | url.", + "title": "Kind", + "type": "string" + }, + "ordinal": { + "description": "Position in the document's source list, 0-based.", + "title": "Ordinal", + "type": "integer" + }, + "rights": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Usage rights. First-class in this domain (``docs/07 §8.7``): a guide that restyles someone's choreography has to say so.", + "title": "Rights" + }, + "source_id": { + "description": "Slug of the source within the guide. Not the annotation id.", + "title": "Source Id", + "type": "string" + }, + "title": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Source title.", + "title": "Title" + }, + "uri": { + "description": "Where the source lives.", + "title": "Uri", + "type": "string" + } + }, + "required": [ + "doc_id", + "ordinal", + "source_id", + "kind", + "uri" + ], + "title": "GuideSourceBodyV1", + "type": "object" + }, + "annot://schema/guide-step/v1": { + "additionalProperties": false, + "description": "One step's presence in one span (or its absence from every span).", + "properties": { + "attrs": { + "additionalProperties": true, + "description": "Namespaced open bag, as on the step.", + "title": "Attrs", + "type": "object" + }, + "description": { + "default": "", + "description": "Prose about the step.", + "title": "Description", + "type": "string" + }, + "doc_id": { + "description": "Which guide this step belongs to.", + "title": "Doc Id", + "type": "string" + }, + "duration_unit": { + "description": "The domain unit: eight | rep | second.", + "title": "Duration Unit", + "type": "string" + }, + "duration_value": { + "description": "Duration in the domain's own unit, as a decimal string.", + "pattern": "^-?\\d+(\\.\\d+)?$", + "title": "Duration Value", + "type": "string" + }, + "name": { + "default": "", + "description": "The step's name. Empty means found-but-not-named, a valid state.", + "title": "Name", + "type": "string" + }, + "optional": { + "default": false, + "description": "'Facultatif — si tu le sens'.", + "title": "Optional", + "type": "boolean" + }, + "ordinal": { + "description": "Position among siblings, 0-based. Order is semantic.", + "title": "Ordinal", + "type": "integer" + }, + "parent_step_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Enclosing step's id; None for a top-level step.", + "title": "Parent Step Id" + }, + "repeat": { + "default": 1, + "description": "How many times the children cycle.", + "title": "Repeat", + "type": "integer" + }, + "span_attrs": { + "additionalProperties": true, + "description": "Open bag on the span.", + "title": "Span Attrs", + "type": "object" + }, + "span_caption": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Prose about THIS span, not about the step.", + "title": "Span Caption" + }, + "span_confidence": { + "anyOf": [ + { + "maximum": 1.0, + "minimum": 0.0, + "type": "number" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Confidence in this span specifically.", + "title": "Span Confidence" + }, + "span_count": { + "default": 0, + "description": "How many spans the step has, so a reader can tell rows apart.", + "title": "Span Count", + "type": "integer" + }, + "span_index": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Which of the step's spans this annotation is, 0-based. None means the step has no spans at all and this is its only row.", + "title": "Span Index" + }, + "span_label": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Short label for this span.", + "title": "Span Label" + }, + "span_open_ended": { + "default": false, + "description": "The span recorded a start but no end — a deep link, not an interval. The reference interval is then a point at the start, and this flag is what keeps that distinct from a zero-length span.", + "title": "Span Open Ended", + "type": "boolean" + }, + "span_role": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "performance | instruction | closeup | ... (open vocab).", + "title": "Span Role" + }, + "span_source": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Slug of the source the span refers into.", + "title": "Span Source" + }, + "step_id": { + "description": "Stable id within the guide ('b4'). Distinct from the annotation id — document ids are slugs, store ids are UUIDs, and the two are related by this field, never conflated.", + "title": "Step Id", + "type": "string" + }, + "tags": { + "description": "Open tag vocabulary.", + "items": { + "type": "string" + }, + "title": "Tags", + "type": "array" + }, + "variant_of": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Another angle on that step, not a new step.", + "title": "Variant Of" + } + }, + "required": [ + "doc_id", + "step_id", + "ordinal", + "duration_value", + "duration_unit" + ], + "title": "GuideStepBodyV1", + "type": "object" + } +} diff --git a/tests/test_evidence.py b/tests/test_evidence.py new file mode 100644 index 0000000..d28980b --- /dev/null +++ b/tests/test_evidence.py @@ -0,0 +1,571 @@ +"""The evidence layer's acceptance tests (issue #4). + +Offline and synthetic throughout: every store is a dict-backed +``lacing.MemoryStore``, never a file, never the user data dir. +""" + +from __future__ import annotations + +import subprocess +import sys +from pathlib import Path + +import pytest + +lacing = pytest.importorskip("lacing") + +from lacing import ( # noqa: E402 + DEFAULT_RATE, + LossyTimeConversionError, + MemoryStore, + RationalTime, + TierStereotype, + annotation_value_digest, +) +from lacing.schema import validate as validate_body # noqa: E402 + +from paces import SegStep, Segmentation, segment # noqa: E402 +from paces.bodies import ( # noqa: E402 + BEAT_TIER, + CUE_TIER, + DOC_TIER, + GRID_TIER, + PASS_TIER, + RECIPE_TIER, + SOURCE_TIER, + STEP_TIER, + SUB_STEP_TIER, + WORD_TIER, +) +from paces.evidence import ( # noqa: E402 + TIERS, + AmbiguousDocument, + CollidingEvidenceKey, + DocumentNotInStore, + from_store, + recipes_from_store, + to_store, +) +from paces.model import Cue, dumps_document # noqa: E402 +from paces.projection import to_document # noqa: E402 + +ASSET = "a" * 64 # a bare 64-hex asset id, so lineage edges are representable +GRID = {"unit": "eight", "subdivisions": 8, "tempoBpm": "129.2", "origin": "51.2"} +STEPS = [ + {"name": "Mise en place", "duration": 2}, + { + "name": "Déhanchés", + "duration": 8, + "children": [ + {"name": "sur place", "duration": 4}, + {"name": "en avançant", "duration": 4}, + ], + }, + {"name": "Soleil avec les bras", "duration": 4}, +] + +# The speech/music split's own shape: measure_grid puts exactly these triples +# in GridMeasurement.evidence["segments"]. +PASSES = [(0.0, 51.2, "speech"), (51.2, 400.0, "music")] +BEATS = ["51.2", "51.664", "52.128"] +TRANSCRIPT = [ + {"start": "60.0", "end": "60.4", "text": "se", "speaker": "singer"}, + {"start": "60.4", "end": "61.0", "text": "hace", "speaker": "singer"}, +] +RECIPES = { + "dehanches/source/performance/58.63": { + "box": ["120", "40", "480", "640"], + "window": ["58.63", "88.36"], + "frameWidth": 1280, + "frameHeight": 720, + "sourceAssetId": ASSET, + "locator": "full_frame", + "params": {"aspect": "0.75", "pad": "0.12"}, + "locked": False, + "mediaDigest": "d" * 64, + } +} + + +def a_segmentation(steps=STEPS): + return segment(None, steps=steps, grid=GRID) + + +def a_write(store=None, *, seg=None, **kwargs): + kwargs.setdefault("doc_id", "que-calor") + kwargs.setdefault("title", "Que Calor") + kwargs.setdefault("source", "https://youtu.be/q_TUyxUhoEw") + kwargs.setdefault("domain", "dance") + return to_store( + seg or a_segmentation(), + store=store if store is not None else MemoryStore(), + asset_id=ASSET, + **kwargs, + ) + + +# ── the acceptance test ───────────────────────────────────────────────────── + + +def test_roundtrip_reproduces_the_document_byte_for_byte(): + store = MemoryStore() + write = a_write(store) + assert dumps_document(from_store(store, asset_id=ASSET)) == dumps_document( + write.document + ) + + +def test_roundtrip_with_every_tier_present(): + store = MemoryStore() + write = a_write( + store, + cues=[ + Cue( + id="calor", + kind="lyric", + text="se hace difícil respirar", + anchor={"step": "dehanches", "offset": {"value": "2", "unit": "eight"}}, + at_s="60.5", + ) + ], + passes=PASSES, + beats=BEATS, + transcript=TRANSCRIPT, + recipes=RECIPES, + detector="mixing.find_segments:speech_music", + ) + assert dumps_document(from_store(store, asset_id=ASSET)) == dumps_document( + write.document + ) + + +def test_the_projection_only_gains_its_evidence_backreferences(): + """``to_store``'s document is ``to_document``'s, plus the ``Origin`` refs.""" + seg = a_segmentation() + write = a_write(seg=seg) + plain = to_document( + seg, + doc_id="que-calor", + title="Que Calor", + source="https://youtu.be/q_TUyxUhoEw", + domain="dance", + ) + assert [s.name for s in write.document.steps] == [s.name for s in plain.steps] + for enriched, bare in zip(write.document.steps, plain.steps): + assert enriched.origin.generated_by == bare.origin.generated_by + assert enriched.origin.confidence == bare.origin.confidence + assert bare.origin.annotation_id is None + assert enriched.origin.annotation_id is not None + assert enriched.origin.value_digest is not None + + +# ── idempotence and identity ──────────────────────────────────────────────── + + +def test_a_second_to_store_writes_nothing(): + store = MemoryStore() + first = a_write(store) + before = len(list(store.all())) + second = a_write(store) + assert (second.written, second.updated) == (0, 0) + assert second.unchanged == first.total + assert len(list(store.all())) == before + + +def test_ids_are_deterministic_across_fresh_stores(): + one, two = a_write(), a_write() + assert one.annotation_ids == two.annotation_ids + assert dumps_document(one.document) == dumps_document(two.document) + + +def test_a_changed_step_updates_in_place_keeping_its_id(): + """A change that keeps the step id rewrites that row, id and all.""" + store = MemoryStore() + base = a_write(store) + longer = [dict(STEPS[0], duration=3), *STEPS[1:]] + changed = a_write(store, seg=a_segmentation(longer)) + annotation_id = base.document.steps[0].origin.annotation_id + assert annotation_id in changed.annotation_ids # same id + assert changed.updated >= 1 # new value + assert ( + changed.document.steps[0].origin.value_digest + != base.document.steps[0].origin.value_digest + ) + + +def test_a_step_that_no_longer_exists_is_pruned_not_resurrected(): + store = MemoryStore() + a_write(store) + shorter = a_write(store, seg=a_segmentation(STEPS[:2])) + assert shorter.removed == 1 # "Soleil avec les bras" is gone + back = from_store(store, asset_id=ASSET) + assert [step.id for step in back.steps] == [ + step.id for step in shorter.document.steps + ] + assert dumps_document(back) == dumps_document(shorter.document) + + +def test_prune_false_accumulates_instead(): + store = MemoryStore() + a_write(store) + shorter = a_write(store, seg=a_segmentation(STEPS[:2]), prune=False) + assert shorter.removed == 0 + assert len([a for a in store.all() if a.tier == STEP_TIER]) == 3 + + +def test_pruning_only_touches_tiers_this_call_wrote_to(): + """Omitting ``passes=`` leaves an earlier speech/music split standing.""" + store = MemoryStore() + a_write(store, passes=PASSES) + a_write(store) # same guide, no passes supplied + assert len([a for a in store.all() if a.tier == PASS_TIER]) == len(PASSES) + + +def test_pruning_leaves_another_guide_on_the_same_asset_alone(): + store = MemoryStore() + a_write(store, doc_id="one") + a_write(store, doc_id="two", seg=a_segmentation(STEPS[:1])) + assert len(from_store(store, asset_id=ASSET, doc_id="one").steps) == 3 + assert len(from_store(store, asset_id=ASSET, doc_id="two").steps) == 1 + + +def test_origin_backreferences_resolve_to_real_annotations(): + store = MemoryStore() + write = a_write(store) + by_id = {str(a.id): a for a in store.all()} + for step in write.document.steps: + annotation = by_id[step.origin.annotation_id] + assert annotation.tier == STEP_TIER + assert annotation.body["step_id"] == step.id + assert step.origin.value_digest == annotation_value_digest(annotation) + + +# ── the tier table ────────────────────────────────────────────────────────── + + +def test_tiers_land_as_the_table_says(): + store = MemoryStore() + a_write( + store, + cues=[ + Cue( + id="calor", + kind="lyric", + text="se hace difícil respirar", + anchor={"step": "dehanches"}, + ) + ], + passes=PASSES, + beats=BEATS, + transcript=TRANSCRIPT, + recipes=RECIPES, + ) + counts: dict[str, int] = {} + for annotation in store.all(): + counts[annotation.tier] = counts.get(annotation.tier, 0) + 1 + assert counts == { + DOC_TIER: 1, + SOURCE_TIER: 1, + PASS_TIER: len(PASSES), + GRID_TIER: 1, + BEAT_TIER: len(BEATS), + WORD_TIER: len(TRANSCRIPT), + STEP_TIER: 3, # one per (step, span); three top-level steps, one span each + SUB_STEP_TIER: 2, + CUE_TIER: 1, + RECIPE_TIER: 1, + } + + +def test_declared_stereotypes_match_docs_07_section_6_3(): + by_name = {tier.name: tier for tier in TIERS} + assert by_name[SUB_STEP_TIER].stereotype is TierStereotype.INCLUDED_IN + assert by_name[SUB_STEP_TIER].parent == STEP_TIER + assert by_name[CUE_TIER].stereotype is TierStereotype.INCLUDED_IN + assert by_name[CUE_TIER].parent == STEP_TIER + assert by_name[RECIPE_TIER].stereotype is TierStereotype.SYMBOLIC_ASSOCIATION + assert by_name[RECIPE_TIER].parent == STEP_TIER + assert by_name[STEP_TIER].stereotype is TierStereotype.NONE + + +def test_every_body_validates_against_its_registered_schema(): + store = MemoryStore() + a_write(store, passes=PASSES, beats=BEATS, transcript=TRANSCRIPT, recipes=RECIPES) + for annotation in store.all(): + validate_body(annotation.body, annotation.body_schema_uri) + + +# ── provenance / lineage ──────────────────────────────────────────────────── + + +def _one(store, tier): + rows = [a for a in store.all() if a.tier == tier] + assert len(rows) == 1, f"{tier}: expected one row, got {len(rows)}" + return rows[0] + + +def test_lineage_edges_make_freshness_work(): + store = MemoryStore() + a_write(store, passes=PASSES, beats=BEATS, recipes=RECIPES) + grid = _one(store, GRID_TIER) + passes = {a.id for a in store.all() if a.tier == PASS_TIER} + steps = [a for a in store.all() if a.tier == STEP_TIER] + subs = [a for a in store.all() if a.tier == SUB_STEP_TIER] + beats = [a for a in store.all() if a.tier == BEAT_TIER] + recipe = _one(store, RECIPE_TIER) + + assert passes <= set(grid.provenance.was_derived_from) + assert ASSET in grid.provenance.was_derived_from + for step in steps: + assert grid.id in step.provenance.was_derived_from + step_ids = {a.id for a in steps} + for sub in subs: + assert step_ids & set(sub.provenance.was_derived_from) + for beat in beats: + assert grid.id in beat.provenance.was_derived_from + assert step_ids & set(recipe.provenance.was_derived_from) + + +def test_a_step_is_derived_from_the_pass_that_frames_it(): + store = MemoryStore() + a_write(store, passes=PASSES) + music = next( + a for a in store.all() if a.tier == PASS_TIER and a.body["label"] == "music" + ) + speech = next( + a for a in store.all() if a.tier == PASS_TIER and a.body["label"] == "speech" + ) + steps = [a for a in store.all() if a.tier == STEP_TIER] + assert steps # the routine starts at the grid origin, inside the music pass + for step in steps: + assert music.id in step.provenance.was_derived_from + assert speech.id not in step.provenance.was_derived_from + + +def test_a_non_hex_asset_id_still_works_and_simply_carries_no_edge(): + store = MemoryStore() + to_store(a_segmentation(), store=store, asset_id="local-video", doc_id="g") + doc = _one(store, DOC_TIER) + assert doc.provenance.was_derived_from == [] + assert from_store(store, asset_id="local-video").id == "g" + + +# ── the no-floats boundary ────────────────────────────────────────────────── + + +def _leaf_values(node, path=""): + if isinstance(node, dict): + for key, value in node.items(): + yield from _leaf_values(value, f"{path}/{key}") + elif isinstance(node, (list, tuple)): + for index, value in enumerate(node): + yield from _leaf_values(value, f"{path}/{index}") + else: + yield path, node + + +def test_no_float_reaches_a_body(): + store = MemoryStore() + a_write(store, passes=PASSES, beats=BEATS, transcript=TRANSCRIPT, recipes=RECIPES) + offenders = [ + (annotation.tier, path, value) + for annotation in store.all() + for path, value in _leaf_values(annotation.body) + if isinstance(value, float) and not path.endswith("confidence") + ] + assert offenders == [] + + +def test_every_interval_is_rational_ticks(): + store = MemoryStore() + a_write(store, passes=PASSES, beats=BEATS) + for annotation in store.all(): + interval = annotation.reference.interval + assert isinstance(interval.start, RationalTime) + assert isinstance(interval.start.value, int) + assert interval.start.rate == DEFAULT_RATE + + +def test_a_time_the_rate_cannot_hold_raises_rather_than_rounding(): + # 1/7 s is not representable at any decimal-friendly rate. + with pytest.raises(LossyTimeConversionError): + to_store( + a_segmentation(), + store=MemoryStore(), + asset_id=ASSET, + passes=[("0.0", "0.14285714", "music")], + ) + + +def test_a_non_canonical_decimal_comes_back_normalised(): + """The store speaks ticks, so ``"60.50"`` reads back as ``"60.5"``. + + Documented in ``paces/evidence.py``: a projection is always canonical, so + this only shows up on a hand-written value, and it is a normalisation + rather than a loss. + """ + store = MemoryStore() + a_write( + store, + cues=[ + Cue( + id="c", + kind="lyric", + text="x", + anchor={"step": "dehanches"}, + at_s="60.50", + ) + ], + ) + assert from_store(store, asset_id=ASSET).cues[0].at_s == "60.5" + + +def test_a_finer_rate_is_a_keyword_not_a_rewrite(): + store = MemoryStore() + to_store( + a_segmentation(), + store=store, + asset_id=ASSET, + doc_id="g", + passes=[("0.0", "1.00048", "music")], + rate=1_000_000, + ) + assert any(a.tier == PASS_TIER for a in store.all()) + + +# ── shapes the segmenters are allowed to return ───────────────────────────── + + +def test_a_step_with_no_spans_is_one_row_not_zero(): + store = MemoryStore() + spanless = Segmentation( + steps=( + SegStep(id="intro", name="Intro", confidence=0.4), + SegStep(id="verse", name="Verse", confidence=0.4), + ), + unit="seconds", + method="explicit", + flags=("extent-unrecorded",), + ) + write = to_store(spanless, store=store, asset_id=ASSET, doc_id="g") + rows = [a for a in store.all() if a.tier == STEP_TIER] + assert len(rows) == 2 + assert all(row.body["span_index"] is None for row in rows) + assert all(row.body["span_count"] == 0 for row in rows) + assert dumps_document(from_store(store, asset_id=ASSET)) == dumps_document( + write.document + ) + + +def test_cues_and_recipes_come_back_out(): + store = MemoryStore() + cue = Cue( + id="calor", + kind="lyric", + text="se hace difícil respirar", + anchor={"step": "dehanches", "offset": {"value": "2", "unit": "eight"}}, + source="source", + at_s="60.5", + ) + a_write(store, cues=[cue], recipes=RECIPES) + back = from_store(store, asset_id=ASSET) + assert back.cues == [cue] + assert recipes_from_store(store, asset_id=ASSET) == RECIPES + + +# ── errors that name the fix ──────────────────────────────────────────────── + + +def test_two_spans_at_one_address_are_refused_not_merged(): + """Derived ids make a duplicate span address a silent overwrite.""" + doubled = Segmentation( + steps=(SegStep(id="b1", name="One", spans=((0.0, 10.0), (0.0, 20.0))),), + unit="seconds", + method="explicit", + ) + with pytest.raises(CollidingEvidenceKey): + to_store(doubled, store=MemoryStore(), asset_id=ASSET, doc_id="g") + + +def test_reading_an_asset_with_nothing_in_it_says_so(): + with pytest.raises(DocumentNotInStore): + from_store(MemoryStore(), asset_id=ASSET) + + +def test_two_guides_on_one_asset_need_a_doc_id(): + store = MemoryStore() + a_write(store, doc_id="one") + a_write(store, doc_id="two") + with pytest.raises(AmbiguousDocument): + from_store(store, asset_id=ASSET) + assert from_store(store, asset_id=ASSET, doc_id="two").id == "two" + + +# ── the core stays pydantic-only (ADR-0004) ───────────────────────────────── + + +def test_importing_paces_does_not_import_lacing(): + code = "import sys, paces; print('lacing' in sys.modules)" + out = subprocess.run( + [sys.executable, "-c", code], capture_output=True, text=True, check=True + ) + assert out.stdout.strip() == "False" + + +def test_to_store_is_reachable_from_the_package_root(): + import paces + + assert paces.to_store is to_store + assert "to_store" in dir(paces) + with pytest.raises(AttributeError): + paces.no_such_thing + + +# ── the README's example, run ─────────────────────────────────────────────── + + +def test_the_readme_example_actually_works(): + """The block under README's "The evidence layer", executed. + + Also checks the README still contains the lines this mirrors, so the two + cannot drift apart silently. + """ + asset_sha256 = ASSET + seg = segment(None, steps=[("Mise en place", 2), ("Déhanchés", 8)], grid=GRID) + + store = MemoryStore() + write = to_store( + seg, + store=store, + asset_id=asset_sha256, + doc_id="que-calor", + title="Que Calor", + domain="dance", + source="https://youtu.be/q_TUyxUhoEw", + ) + + assert dumps_document(from_store(store, asset_id=asset_sha256)) == dumps_document( + write.document + ) + assert ( + to_store( + seg, + store=store, + asset_id=asset_sha256, + doc_id="que-calor", + title="Que Calor", + domain="dance", + source="https://youtu.be/q_TUyxUhoEw", + ).written + == 0 + ) + + readme = (Path(__file__).resolve().parents[1] / "README.md").read_text( + encoding="utf-8" + ) + for line in ( + "from lacing import MemoryStore", + "write = to_store(seg, store=store, asset_id=asset_sha256,", + "dumps_document(from_store(store, asset_id=asset_sha256))", + ): + assert line in readme, f"README no longer shows: {line}" diff --git a/tests/test_evidence_schemas.py b/tests/test_evidence_schemas.py new file mode 100644 index 0000000..f938d5b --- /dev/null +++ b/tests/test_evidence_schemas.py @@ -0,0 +1,94 @@ +"""The body-schema URIs paces claims, pinned with their content. + +``lacing.schema``'s registry is one flat, fleet-wide dict and +``register_body_schema`` is last-write-wins and silent, so a URI is a claim +on a shared namespace, not a local name. This file is the guard on both +halves of that claim: + +- **which URIs** paces owns (a rename, an addition or a drop fails here), and + that they do not collide with a name another fleet package already owns; +- **what is inside them** — the committed JSON Schema snapshot in + ``tests/data/body_schemas.json``. New URIs are additive and need no + migration; a *changed* one is a wire break for every already-written store, + and this test is what makes that a decision rather than an accident. + +Regenerating the snapshot deliberately:: + + python -m tests.test_evidence_schemas # writes tests/data/body_schemas.json +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +pytest.importorskip("lacing") + +from lacing.schema import json_schema, registered_uris # noqa: E402 + +from paces.bodies import PACES_BODY_SCHEMA_URIS, WORD_BODY_SCHEMA_URI # noqa: E402 + +SNAPSHOT = Path(__file__).parent / "data" / "body_schemas.json" + +#: Names other fleet packages already own on this shared namespace, checked +#: here so paces never quietly takes one over. ``beat/v1`` is the live +#: example: it is reelee's *narrative* beat, which is why paces' musical one +#: is ``guide-beat/v1``. +FOREIGN_URIS = ( + "annot://schema/beat/v1", # reelee — a narrative beat + "annot://schema/step/v1", # unclaimed, but too generic to take + "annot://schema/cue/v1", # unclaimed, but too generic to take + "annot://schema/section/v1", # nw + "annot://schema/shot/v1", # nw + "annot://schema/clip/v1", # reelee + "annot://schema/source/v1", # braidio + "annot://schema/named-entity/v1", # lacing +) + + +def _snapshot() -> dict: + return json.loads(SNAPSHOT.read_text(encoding="utf-8")) + + +def test_the_uri_set_is_exactly_what_is_pinned(): + assert sorted(PACES_BODY_SCHEMA_URIS) == sorted(_snapshot()) + + +def test_every_pinned_schema_still_has_its_pinned_content(): + snapshot = _snapshot() + for uri, expected in snapshot.items(): + assert json_schema(uri) == expected, ( + f"{uri} changed shape. New URIs are additive and free; CHANGING " + "one breaks every store already written against it. If the change " + "is intended, bump to /v2 and register a migration — do not " + "re-snapshot in place." + ) + + +def test_paces_claims_no_name_another_package_owns(): + assert not set(PACES_BODY_SCHEMA_URIS) & set(FOREIGN_URIS) + + +def test_importing_the_bodies_registers_them(): + registered = set(registered_uris()) + assert set(PACES_BODY_SCHEMA_URIS) <= registered + # word/v1 is reused, not claimed: it must be there, and it must be + # lacing's model, not a paces copy. + assert WORD_BODY_SCHEMA_URI in registered + assert WORD_BODY_SCHEMA_URI not in PACES_BODY_SCHEMA_URIS + + +def _write_snapshot() -> None: + SNAPSHOT.parent.mkdir(parents=True, exist_ok=True) + payload = {uri: json_schema(uri) for uri in sorted(PACES_BODY_SCHEMA_URIS)} + SNAPSHOT.write_text( + json.dumps(payload, indent=2, ensure_ascii=False, sort_keys=True) + "\n", + encoding="utf-8", + ) + + +if __name__ == "__main__": + _write_snapshot() + print(f"wrote {SNAPSHOT}") From 29b83f4129237d83ab29f6d5b8523de3eb319306 Mon Sep 17 00:00:00 2001 From: Thor Whalen <1906276+thorwhalen@users.noreply.github.com> Date: Wed, 9 Sep 2026 02:33:43 +0200 Subject: [PATCH 2/3] Fix three blocking findings from PR #23's review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1. Cross-guide pruning. `source.pass` / `beat` / `transcript.word` bodies carry no doc_id (they describe the asset, and two guides over one video share one transcript), so the old `doc_id not in (None, doc_id)` filter matched them and one guide's re-run deleted evidence a sibling guide's was_derived_from pointed at. Tiers are now split into ASSET_SCOPED_TIERS and DOC_SCOPED_TIERS; only doc-scoped rows are prunable, on a strict doc_id match, and a test asserts the two sets partition every tier written so a new tier must be classified. 2. Stale lineage behind the digest gate. annotation_value_digest excludes provenance, so a row whose value held while its upstream moved was skipped and kept citing the previous run's inputs — re-measuring a pass left the grid and its steps pointing at the old one. Provenance is now compared on every digest-gated skip and the row is rewritten when lineage differs, keeping generated_at_time so freshness still does not fire on an unchanged value. Counted separately as StoreWrite.relinked, so idempotence stays a clean zero. 3. README reformatted by CI. wads runs an unpinned `uvx ruff format .`, ruff >=0.16 formats Python fences in Markdown, and the publish job commits the result — which would have broken the README-drift assertion on main. `*.md` is excluded in [tool.ruff], and the assertions are on format-stable fragments. Also from the review: StoreWrite.document now normalises a caller-supplied cue at_s through the tick round trip, so "byte for byte" is literal rather than approximate; the lineage test says what it actually proves (edges resolve, not nw-verified-fresh — thorwhalen/paces#24); the concurrency and doc_id-rename caveats are documented on to_store; the README example defines the names it uses; and docs/07 §6 records that a time with no finite decimal form is refused at every rate, not only at 24000. Claude-Session: https://claude.ai/code/session_01UDdKG2wFtDzsY2fgBGEjVK --- README.md | 24 +++-- docs/07-annotation-model.md | 35 ++++++- paces/evidence.py | 175 ++++++++++++++++++++++++++--------- pyproject.toml | 6 ++ tests/test_evidence.py | 177 +++++++++++++++++++++++++++++------- 5 files changed, 326 insertions(+), 91 deletions(-) diff --git a/README.md b/README.md index 89b470a..b934d1d 100644 --- a/README.md +++ b/README.md @@ -128,19 +128,20 @@ from lacing import MemoryStore # pip install 'paces[lacing]' from paces import segment, from_store, to_store from paces.model import dumps_document -seg = segment(None, steps=[("Mise en place", 2), ("Déhanchés", 8)], grid=GRID) +grid = {"unit": "eight", "subdivisions": 8, "tempoBpm": "129.2", "origin": "51.2"} +asset_sha256 = "a" * 64 # lacing.hash_file(video) in real use +seg = segment(None, steps=[("Mise en place", 2), ("Déhanchés", 8)], grid=grid) store = MemoryStore() # or SqliteStore('project.annot') -write = to_store(seg, store=store, asset_id=asset_sha256, - doc_id="que-calor", title="Que Calor", domain="dance", - source="https://youtu.be/q_TUyxUhoEw") +guide = dict(doc_id="que-calor", title="Que Calor", domain="dance", + source="https://youtu.be/q_TUyxUhoEw") + +write = to_store(seg, store=store, asset_id=asset_sha256, **guide) dumps_document(from_store(store, asset_id=asset_sha256)) == dumps_document(write.document) # True — the projection is exact -to_store(seg, store=store, asset_id=asset_sha256, doc_id="que-calor", - title="Que Calor", domain="dance", - source="https://youtu.be/q_TUyxUhoEw").written +to_store(seg, store=store, asset_id=asset_sha256, **guide).written # 0 — a re-run of the same analysis writes nothing ``` @@ -154,9 +155,12 @@ not fire on a no-op re-run. `to_store` is a **re-derivation** of the guide, not a merge into it: a step that no longer exists is dropped rather than left to be resurrected by the -next projection. It only ever prunes this asset, this `doc_id`, and the tiers -that call actually wrote to — so omitting `passes=` leaves an earlier -speech/music split standing. `prune=False` accumulates instead. +next projection. Pruning only ever touches this asset, this `doc_id`, and the +guide's own tiers. Evidence *about the asset* — the speech/music split, the +beats, the transcript — is shared by every guide over that asset and is never +pruned, so one guide can't delete what another's lineage points at; a +re-measure adds a row under a new content-derived key and the old one stays. +`prune=False` opts out of the rest. Analysis a `Segmentation` cannot carry rides along as keywords: `passes=` (the speech/music split), `beats=`, `transcript=`, `cues=`, `recipes=`. What diff --git a/docs/07-annotation-model.md b/docs/07-annotation-model.md index df93321..fe5059f 100644 --- a/docs/07-annotation-model.md +++ b/docs/07-annotation-model.md @@ -592,6 +592,24 @@ these are standoff annotations, and the interval lives on `Annotation.reference` float crosses the boundary — times become `RationalTime` ticks at an injectable `rate=` (default 24000), where a value the rate cannot hold exactly raises rather than rounding. +*Consequence worth stating, because it will surprise someone:* the boundary is decimal +strings both ways (§6.5's wire rule), so a time with **no finite decimal form** — a third +of a second, say — is refused with `LossyTimeConversionError` at *every* rate, not just at +24000. Raising the rate does not help; a third of a second is not a decimal at any of them. +This is the wire rule doing exactly what it was written for, and the honest answer is to +record the value the analysis actually measured (`"0.333"`), not a fraction the document +cannot spell. + +**Two scopes, and pruning reads the difference.** A `to_store` is a re-derivation *of one +guide*, so it drops that guide's rows it no longer produces — otherwise `from_store` would +resurrect a step nobody analysed. But `source.pass`, `beat` and `transcript.word` describe +the **asset**, carry no `doc_id` on purpose (two guides over one video share one +transcript), and are therefore never pruned: deleting a pass row out from under a sibling +guide's `was_derived_from` would leave that guide's grid and steps pointing at nothing. A +re-measure adds a row under a new content-derived key and leaves the old one standing, +where provenance still says which run used which. `tests/test_evidence.py` holds the +partition — a new tier must be classified into one scope or the other. + **What does not flow into the store**: `locks`, `questions`, `artifacts`, and span `excerpt` windows. Those are document-layer records — a human edit, an open question, a built file, a hand-picked loop — and §6.0's one-way rule is what says so. @@ -610,15 +628,22 @@ side and ship with the edit-protection work (`paces/edits.py`). and paces leaves the row *completely* untouched, provenance included, so `generated_at_time` does not churn and freshness does not fire. A second `to_store` of the same input writes zero annotations. + **One exception, and it is the digest's own blind spot:** `VALUE_FIELDS` excludes + provenance, so a row can keep its exact value while the thing it was derived from moves + underneath it — re-measure the music pass and the grid's body is unchanged, but the pass + it cites is not the pass that produced it any more. paces compares `was_derived_from` + (and the rest of provenance bar the timestamp) on every digest-gated skip and rewrites + the row when it differs, keeping `generated_at_time` so freshness still does not fire on + a value that did not change. `StoreWrite.relinked` counts those, kept separate from + `updated` so idempotence stays a clean zero. 3. Changed digest ⇒ the regenerated annotation replaces the one it supersedes **under the same id**, so downstream `was_derived_from` edges keep resolving. `reelee` does this procedurally (`_adopt_output_identity`); with a derived id it holds by construction, and `to_store` is reproducible from a fresh store as a bonus. - A row the re-run no longer produces is *removed*, not left behind — `reelee`'s - *"the graph is a set of nodes whose values are re-derivable, not an append-only log"*, - applied. Pruning is scoped to the asset, the `doc_id`, and the tiers that run actually - wrote to, so omitting an optional evidence kwarg cannot delete an earlier producer's - rows. + A doc-scoped row the re-run no longer produces is *removed*, not left behind — + `reelee`'s *"the graph is a set of nodes whose values are re-derivable, not an + append-only log"*, applied. Pruning is scoped to the asset, the `doc_id`, and the + doc-scoped tiers that run actually wrote to; asset-scoped evidence is exempt (§6.3). 4. Re-project to a candidate `StepDocument`. 5. Three-way merge against the committed document: for every `Lock.path` on a node, keep the committed value and record `Origin.value_digest` of what was rejected. For everything else, diff --git a/paces/evidence.py b/paces/evidence.py index 9b15c14..539a6a7 100644 --- a/paces/evidence.py +++ b/paces/evidence.py @@ -44,13 +44,21 @@ :func:`lacing.annotation_value_digest` is unchanged is left *completely* alone — provenance included — so ``generated_at_time`` does not churn and freshness does not fire. A second :func:`to_store` of the same input - writes nothing. -- **Re-derivation, not accumulation.** A row this run superseded — a step - that no longer exists, a cue that was dropped — is removed, scoped to this - asset, this guide, and the tiers this call actually wrote to. `reelee`'s - phrasing: *"the graph is a set of nodes whose values are re-derivable, not - an append-only log"*. A stale row left behind would come back out of - :func:`from_store` as a step nobody analysed. ``prune=False`` opts out. + writes nothing. The one exception is lineage: the value digest excludes + provenance by design, so a row whose answer held but whose + ``was_derived_from`` moved is rewritten (``StoreWrite.relinked``) with its + ``generated_at_time`` preserved — otherwise a re-measured pass would leave + the grid and its steps pointing at the previous run's inputs. +- **Re-derivation, not accumulation — for the guide's own rows.** A + doc-scoped row this run superseded (a step that no longer exists, a cue + that was dropped) is removed: `reelee`'s *"the graph is a set of nodes + whose values are re-derivable, not an append-only log"*, and a stale one + left behind would come back out of :func:`from_store` as a step nobody + analysed. :data:`ASSET_SCOPED_TIERS` — the speech/music split, the beats, + the transcript — are **never** pruned: they describe the asset, are shared + by every guide over it, and deleting one out from under a sibling guide's + ``was_derived_from`` would leave that guide pointing at nothing. + ``prune=False`` opts out of the rest. - **No floats across the boundary.** Every time crosses as ``RationalTime`` ticks at ``rate=`` (lacing non-negotiable #1); :meth:`RationalTime.from_seconds` raises rather than rounding, so a @@ -259,18 +267,36 @@ class StoreWrite: written: int # rows that did not exist updated: int # rows whose value digest changed unchanged: int # rows left completely alone — the idempotence count + relinked: int # rows whose value held but whose upstream moved removed: int # rows this re-derivation superseded and dropped annotation_ids: tuple[str, ...] # every row touched, in write order @property def total(self) -> int: - return self.written + self.updated + self.unchanged + return self.written + self.updated + self.unchanged + self.relinked class CollidingEvidenceKey(ValueError): """Two rows in one run would derive the same annotation id.""" +#: The provenance fields that say where a row came *from*. Compared on every +#: digest-gated skip, because ``annotation_value_digest`` deliberately excludes +#: provenance entirely (``lacing/digest.py``'s ``VALUE_FIELDS``): a row can keep +#: its exact value while the thing it was derived from moves underneath it. +_LINEAGE_FIELDS = ( + "was_generated_by", + "was_attributed_to", + "was_derived_from", + "activity", +) + + +def _same_lineage(one: Provenance, other: Provenance) -> bool: + """Do these two provenances agree on everything except *when*?""" + return all(getattr(one, f) == getattr(other, f) for f in _LINEAGE_FIELDS) + + class _Writer: """Digest-gated upsert over one store. Not part of the public surface.""" @@ -282,6 +308,7 @@ def __init__(self, store, *, attributed_to: str, generated_at: RationalTime): self.written = 0 self.updated = 0 self.unchanged = 0 + self.relinked = 0 self.removed = 0 self.ids: list[str] = [] self.touched: set[UUID] = set() @@ -328,11 +355,32 @@ def put( existing = self._index.get(annotation_id) if existing is not None: if annotation_value_digest(existing) == digest: - self.unchanged += 1 - self._record(annotation_id) - return annotation_id, digest - self._store.remove(annotation_id) - self.updated += 1 + if _same_lineage(existing.provenance, candidate.provenance): + self.unchanged += 1 + self._record(annotation_id) + return annotation_id, digest + # Same answer, different upstream. The value digest cannot see + # this — provenance is excluded from it by design — so the + # gate alone would leave the row pointing at the previous + # run's inputs, which is the one thing the edge exists to + # answer. Rewrite it, but keep ``generated_at_time``: the + # value did not change, so freshness has no business firing. + candidate = candidate.model_copy( + update={ + "provenance": candidate.provenance.model_copy( + update={ + "generated_at_time": ( + existing.provenance.generated_at_time + ) + } + ) + } + ) + self._store.remove(annotation_id) + self.relinked += 1 + else: + self._store.remove(annotation_id) + self.updated += 1 else: self.written += 1 self._store.add(candidate) @@ -348,16 +396,18 @@ def prune(self, *, asset_id: str, doc_id: str, tiers: set[str]) -> None: """Drop rows on *tiers* this run superseded — a re-derivation, not a merge. Scoped three ways so it can only ever remove what this call is - authoritative for: the asset it was given, the guide it was given - (a row carrying another guide's ``doc_id`` is left alone), and the - tiers this call actually wrote to. + authoritative for: the asset it was given, the guide it was given, + and the tiers this call actually wrote to. The ``doc_id`` test is a + strict match on the body's own field, so a row that does not carry + one is never pruned — see :data:`ASSET_SCOPED_TIERS` for why that + matters. """ for annotation in list(self._store.all()): if annotation.tier not in tiers or annotation.id in self.touched: continue if getattr(annotation.reference, "asset_id", None) != asset_id: continue - if annotation.body.get("doc_id") not in (None, doc_id): + if annotation.body.get("doc_id") != doc_id: continue self._store.remove(annotation.id) self._index.pop(annotation.id, None) @@ -435,9 +485,23 @@ def to_store( no longer exists, a cue that was dropped. On by default, because ``to_store`` is a **re-derivation** of the guide, not a merge into it: a stale row left behind would be resurrected by - :func:`from_store` as a step nobody analysed. Only tiers this - call actually wrote to are pruned, and only for this asset and - this ``doc_id``. Pass ``False`` to accumulate instead. + :func:`from_store` as a step nobody analysed. Scoped to this + asset, this ``doc_id``, and the doc-scoped tiers this call + actually wrote to; :data:`ASSET_SCOPED_TIERS` are never pruned. + Pass ``False`` to accumulate instead. + + Note: + ``doc_id`` names the guide, so changing it writes a **second** guide + rather than renaming the first — the old one is still there, and a + later ``from_store(store, asset_id=...)`` without a ``doc_id`` will + say so by raising :class:`AmbiguousDocument`. Renaming a guide means + writing the new one and dropping the old rows deliberately. + + Concurrency: + Not safe against another writer on the same store. The digest index + is snapshotted at entry and pruning reads-then-removes, both without + a lock, so two concurrent ``to_store`` calls on one asset can delete + each other's rows. Serialise them, or give each its own store. rate: Ticks per second for every interval. Raises rather than rounding if a time cannot be represented exactly. @@ -449,7 +513,9 @@ def to_store( seg, doc_id=doc_id, title=title, source=source, domain=domain, lang=lang ) if cues: - document = document.model_copy(update={"cues": [_as_cue(cue) for cue in cues]}) + document = document.model_copy( + update={"cues": [_as_cue(cue, rate=rate) for cue in cues]} + ) register_tiers(store) writer = _Writer( @@ -539,13 +605,7 @@ def to_store( writer.prune( asset_id=asset_id, doc_id=doc_id, - tiers=_prunable_tiers( - passes=passes, - beats=beats, - transcript=transcript, - cues=document.cues, - recipes=recipes, - ), + tiers=_prunable_tiers(cues=document.cues, recipes=recipes), ) return StoreWrite( @@ -553,36 +613,51 @@ def to_store( written=writer.written, updated=writer.updated, unchanged=writer.unchanged, + relinked=writer.relinked, removed=writer.removed, annotation_ids=tuple(writer.ids), ) -#: Tiers every ``to_store`` writes, because they are projected from the -#: ``Segmentation`` itself — a re-derivation is authoritative over all of them. +#: Tiers whose rows describe **the asset**, not one guide over it: the +#: speech/music split of a video, its beat times, its transcript. Their bodies +#: carry no ``doc_id`` on purpose — two guides over the same video share one +#: transcript rather than duplicating it — and that is exactly why they are +#: never pruned. Pruning is per-guide, and a guide has no authority to delete +#: evidence another guide's annotations were derived from; deleting a pass row +#: out from under a sibling guide's ``was_derived_from`` would leave it +#: pointing at nothing. A re-measure that moves a boundary therefore *adds* a +#: row under a new content-derived key and leaves the old one standing, where +#: provenance still says which run used which. +ASSET_SCOPED_TIERS = frozenset({PASS_TIER, BEAT_TIER, WORD_TIER}) + +#: Tiers whose rows belong to exactly one guide (every body carries a +#: ``doc_id``) and which a re-derivation of that guide is therefore +#: authoritative over. +DOC_SCOPED_TIERS = frozenset( + {DOC_TIER, SOURCE_TIER, GRID_TIER, STEP_TIER, SUB_STEP_TIER, CUE_TIER, RECIPE_TIER} +) + +#: The doc-scoped tiers every ``to_store`` writes, because they are projected +#: from the ``Segmentation`` itself. _ALWAYS_WRITTEN_TIERS = frozenset( {DOC_TIER, SOURCE_TIER, GRID_TIER, STEP_TIER, SUB_STEP_TIER} ) -def _prunable_tiers(*, passes, beats, transcript, cues, recipes) -> set[str]: +def _prunable_tiers(*, cues, recipes) -> set[str]: """Which tiers this call may prune: the ones it is authoritative over. - A tier fed by a keyword is prunable only when that keyword was supplied. - Omitting ``passes=`` therefore leaves an earlier speech/music split - standing rather than silently deleting it — "a re-derivation replaces - what it re-derives", and it re-derives what it was given. + Doc-scoped only (:data:`ASSET_SCOPED_TIERS` explains the exclusion), and + within those, a tier fed by a keyword is prunable only when that keyword + was supplied: "a re-derivation replaces what it re-derives", and it + re-derives what it was given. """ tiers = set(_ALWAYS_WRITTEN_TIERS) - for supplied, tier in ( - (passes, PASS_TIER), - (beats, BEAT_TIER), - (transcript, WORD_TIER), - (cues, CUE_TIER), - (recipes, RECIPE_TIER), - ): + for supplied, tier in ((cues, CUE_TIER), (recipes, RECIPE_TIER)): if supplied: tiers.add(tier) + assert tiers <= DOC_SCOPED_TIERS # the invariant this function exists for return tiers @@ -595,8 +670,20 @@ def put(**kwargs) -> tuple[UUID, str]: return put -def _as_cue(cue: Cue | Mapping[str, Any]) -> Cue: - return cue if isinstance(cue, Cue) else Cue.model_validate(dict(cue)) +def _as_cue(cue: Cue | Mapping[str, Any], *, rate: int) -> Cue: + """A cue in the store's own spelling of its time. + + ``at_s`` is the one caller-supplied *time* that reaches the returned + document — spans arrive already canonical from + :func:`~paces.projection.to_document`. Passing it through the tick round + trip here is what keeps ``StoreWrite.document`` and :func:`from_store` + literally identical rather than merely equivalent: a hand-written + ``"60.50"`` becomes ``"60.5"`` on the way in, not on the way back out. + """ + cue = cue if isinstance(cue, Cue) else Cue.model_validate(dict(cue)) + if cue.at_s is None: + return cue + return cue.model_copy(update={"at_s": _decimal(_ticks(cue.at_s, rate=rate))}) def _document_span(document: StepDocument) -> tuple[str, str] | None: diff --git a/pyproject.toml b/pyproject.toml index d8dc41b..6f87f51 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -199,6 +199,12 @@ line-length = 88 target-version = "py310" exclude = [ "**/*.ipynb", + # Markdown is documentation, not source. Ruff >=0.16 formats the Python + # fences inside it, and CI runs an UNPINNED `uvx ruff format .` whose + # publish job commits the result — so a fence would be silently rewritten + # by whichever ruff shipped that week, and README examples that tests + # assert against would drift out from under them on main. + "*.md", ".git", ".venv", "build", diff --git a/tests/test_evidence.py b/tests/test_evidence.py index d28980b..5468cbc 100644 --- a/tests/test_evidence.py +++ b/tests/test_evidence.py @@ -38,10 +38,13 @@ WORD_TIER, ) from paces.evidence import ( # noqa: E402 + ASSET_SCOPED_TIERS, + DOC_SCOPED_TIERS, TIERS, AmbiguousDocument, CollidingEvidenceKey, DocumentNotInStore, + _decimal, from_store, recipes_from_store, to_store, @@ -104,6 +107,24 @@ def a_write(store=None, *, seg=None, **kwargs): ) +def _one(store, tier): + rows = [a for a in store.all() if a.tier == tier] + assert len(rows) == 1, f"{tier}: expected one row, got {len(rows)}" + return rows[0] + + +def _one_where(store, tier, *, end=None, **body): + rows = [ + a + for a in store.all() + if a.tier == tier + and all(a.body.get(k) == v for k, v in body.items()) + and (end is None or _decimal(a.reference.interval.end) == end) + ] + assert len(rows) == 1, f"{tier} {body}: expected one row, got {len(rows)}" + return rows[0] + + # ── the acceptance test ───────────────────────────────────────────────────── @@ -221,6 +242,39 @@ def test_pruning_only_touches_tiers_this_call_wrote_to(): assert len([a for a in store.all() if a.tier == PASS_TIER]) == len(PASSES) +def test_a_re_run_of_one_guide_never_prunes_a_siblings_asset_evidence(): + """Asset-level evidence is shared, so no single guide may delete it. + + ``source.pass`` / ``beat`` / ``transcript.word`` bodies carry no + ``doc_id`` on purpose — two guides over one video share one transcript. + That is exactly why they are exempt from a per-guide prune: deleting a + pass row out from under guide A's ``was_derived_from`` would leave A's + grid and steps pointing at nothing. + """ + store = MemoryStore() + ga = a_write(store, doc_id="ga", passes=PASSES, beats=BEATS, transcript=TRANSCRIPT) + gb = a_write( + store, + doc_id="gb", + passes=[(0.0, 60.0, "speech"), (60.0, 300.0, "music")], + beats=["60.0"], + transcript=[{"start": "70.0", "end": "70.5", "text": "otra"}], + ) + assert gb.removed == 0 + + present = {a.id for a in store.all()} | {ASSET} + for write in (ga, gb): + for step in write.document.steps: + annotation = next( + a for a in store.all() if str(a.id) == step.origin.annotation_id + ) + assert not set(annotation.provenance.was_derived_from) - present + # ga's own projection is untouched by gb's differing measurements. + assert dumps_document( + from_store(store, asset_id=ASSET, doc_id="ga") + ) == dumps_document(ga.document) + + def test_pruning_leaves_another_guide_on_the_same_asset_alone(): store = MemoryStore() a_write(store, doc_id="one") @@ -229,6 +283,42 @@ def test_pruning_leaves_another_guide_on_the_same_asset_alone(): assert len(from_store(store, asset_id=ASSET, doc_id="two").steps) == 1 +def test_a_moved_upstream_relinks_a_row_whose_value_did_not_change(): + """The digest gate cannot see lineage — ``VALUE_FIELDS`` excludes provenance. + + Re-measure the music pass end and the grid's own body is unchanged, so + the digest says "skip". Skipping would leave the grid and its steps + citing the *previous* run's pass — the one question the edge exists to + answer. The row is relinked instead, keeping its ``generated_at_time`` + because the value really did not change. + """ + store = MemoryStore() + first = a_write(store, passes=PASSES) + old_pass = _one_where(store, PASS_TIER, label="music") + grid_before = _one(store, GRID_TIER) + + moved = [(0.0, 51.2, "speech"), (51.2, 401.0, "music")] + second = a_write(store, passes=moved) + + new_pass = _one_where(store, PASS_TIER, label="music", end="401") + grid_after = _one(store, GRID_TIER) + assert grid_after.id == grid_before.id # derived id: same row + assert new_pass.id in grid_after.provenance.was_derived_from + assert old_pass.id not in grid_after.provenance.was_derived_from + # the value held, so freshness must not fire + assert ( + grid_after.provenance.generated_at_time + == grid_before.provenance.generated_at_time + ) + assert annotation_value_digest(grid_after) == annotation_value_digest(grid_before) + assert second.relinked >= 1 and second.updated == 0 + assert first.relinked == 0 + + present = {a.id for a in store.all()} | {ASSET} + for annotation in store.all(): + assert not set(annotation.provenance.was_derived_from) - present + + def test_origin_backreferences_resolve_to_real_annotations(): store = MemoryStore() write = a_write(store) @@ -277,6 +367,26 @@ def test_tiers_land_as_the_table_says(): } +def test_the_two_scopes_partition_every_tier_paces_writes(): + """A new tier has to be classified, because pruning reads the split. + + Doc-scoped rows carry a ``doc_id`` and a re-derivation of that guide owns + them; asset-scoped rows do not and are shared. A tier in neither set (or + in both) would be pruned on a rule nobody chose. + """ + written = {tier.name for tier in TIERS} + assert ASSET_SCOPED_TIERS | DOC_SCOPED_TIERS == written + assert not ASSET_SCOPED_TIERS & DOC_SCOPED_TIERS + + +def test_asset_scoped_bodies_carry_no_doc_id_and_doc_scoped_ones_do(): + store = MemoryStore() + a_write(store, passes=PASSES, beats=BEATS, transcript=TRANSCRIPT, recipes=RECIPES) + for annotation in store.all(): + has_doc_id = "doc_id" in annotation.body + assert has_doc_id == (annotation.tier in DOC_SCOPED_TIERS), annotation.tier + + def test_declared_stereotypes_match_docs_07_section_6_3(): by_name = {tier.name: tier for tier in TIERS} assert by_name[SUB_STEP_TIER].stereotype is TierStereotype.INCLUDED_IN @@ -298,13 +408,15 @@ def test_every_body_validates_against_its_registered_schema(): # ── provenance / lineage ──────────────────────────────────────────────────── -def _one(store, tier): - rows = [a for a in store.all() if a.tier == tier] - assert len(rows) == 1, f"{tier}: expected one row, got {len(rows)}" - return rows[0] - +def test_lineage_edges_resolve(): + """Every ``was_derived_from`` names something that is actually there. -def test_lineage_edges_make_freshness_work(): + This is what a freshness walk (``nw.freshness.stale_after``) needs from + paces, not the whole of it: nw also reads ``verifying-trace`` rows to + call anything *verified* fresh, and paces writes none (thorwhalen/paces#24). + What is asserted here is that the graph is connected and that no edge + dangles. + """ store = MemoryStore() a_write(store, passes=PASSES, beats=BEATS, recipes=RECIPES) grid = _one(store, GRID_TIER) @@ -325,6 +437,11 @@ def test_lineage_edges_make_freshness_work(): assert grid.id in beat.provenance.was_derived_from assert step_ids & set(recipe.provenance.was_derived_from) + present = {a.id for a in store.all()} | {ASSET} + for annotation in store.all(): + dangling = set(annotation.provenance.was_derived_from) - present + assert not dangling, f"{annotation.tier}: {dangling}" + def test_a_step_is_derived_from_the_pass_that_frames_it(): store = MemoryStore() @@ -397,15 +514,16 @@ def test_a_time_the_rate_cannot_hold_raises_rather_than_rounding(): ) -def test_a_non_canonical_decimal_comes_back_normalised(): - """The store speaks ticks, so ``"60.50"`` reads back as ``"60.5"``. +def test_a_non_canonical_decimal_is_normalised_on_the_way_in(): + """The store speaks ticks, so ``"60.50"`` becomes ``"60.5"`` — on BOTH sides. - Documented in ``paces/evidence.py``: a projection is always canonical, so - this only shows up on a hand-written value, and it is a normalisation - rather than a loss. + Normalising only on read-back would make ``StoreWrite.document`` and + ``from_store`` merely equivalent rather than identical, and the + round-trip claim is "byte for byte". So the hand-written value is + converted as it enters, and the two documents agree literally. """ store = MemoryStore() - a_write( + write = a_write( store, cues=[ Cue( @@ -417,7 +535,11 @@ def test_a_non_canonical_decimal_comes_back_normalised(): ) ], ) + assert write.document.cues[0].at_s == "60.5" assert from_store(store, asset_id=ASSET).cues[0].at_s == "60.5" + assert dumps_document(from_store(store, asset_id=ASSET)) == dumps_document( + write.document + ) def test_a_finer_rate_is_a_keyword_not_a_rewrite(): @@ -534,38 +656,29 @@ def test_the_readme_example_actually_works(): seg = segment(None, steps=[("Mise en place", 2), ("Déhanchés", 8)], grid=GRID) store = MemoryStore() - write = to_store( - seg, - store=store, - asset_id=asset_sha256, + guide = dict( doc_id="que-calor", title="Que Calor", domain="dance", source="https://youtu.be/q_TUyxUhoEw", ) + write = to_store(seg, store=store, asset_id=asset_sha256, **guide) assert dumps_document(from_store(store, asset_id=asset_sha256)) == dumps_document( write.document ) - assert ( - to_store( - seg, - store=store, - asset_id=asset_sha256, - doc_id="que-calor", - title="Que Calor", - domain="dance", - source="https://youtu.be/q_TUyxUhoEw", - ).written - == 0 - ) + assert to_store(seg, store=store, asset_id=asset_sha256, **guide).written == 0 readme = (Path(__file__).resolve().parents[1] / "README.md").read_text( encoding="utf-8" ) - for line in ( + # Fragments, not whole lines: `[tool.ruff] exclude` keeps ruff out of + # Markdown so the fence cannot be reflowed under us, and asserting on + # short fragments means a deliberate rewrap would not fail this either. + for fragment in ( "from lacing import MemoryStore", - "write = to_store(seg, store=store, asset_id=asset_sha256,", - "dumps_document(from_store(store, asset_id=asset_sha256))", + "to_store(seg, store=store, asset_id=asset_sha256", + "from_store(store, asset_id=asset_sha256)", + ".written", ): - assert line in readme, f"README no longer shows: {line}" + assert fragment in readme, f"README no longer shows: {fragment}" From 8a1b8fb3c7322e8ca062ea26f43216e5e8aa91f7 Mon Sep 17 00:00:00 2001 From: Thor Whalen <1906276+thorwhalen@users.noreply.github.com> Date: Wed, 9 Sep 2026 02:39:05 +0200 Subject: [PATCH 3/3] Record the retirement gap the asset-scoped exemption leaves MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ASSET_SCOPED_TIERS are never pruned, which is what keeps one guide from deleting evidence a sibling's was_derived_from cites. The cost is that superseded passes/beats/words never retire, so a consumer reading one of those tiers directly sees every generation superimposed. from_store does not read them, so the projection is unaffected — but the gap should be written down where someone will meet it rather than discovered later. Refs #25, filed with the three options (GC verb, run marker, or a ruling that these tiers are append-only by design). Claude-Session: https://claude.ai/code/session_01UDdKG2wFtDzsY2fgBGEjVK --- docs/07-annotation-model.md | 5 ++++- paces/evidence.py | 7 +++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/07-annotation-model.md b/docs/07-annotation-model.md index fe5059f..e3916dc 100644 --- a/docs/07-annotation-model.md +++ b/docs/07-annotation-model.md @@ -608,7 +608,10 @@ transcript), and are therefore never pruned: deleting a pass row out from under guide's `was_derived_from` would leave that guide's grid and steps pointing at nothing. A re-measure adds a row under a new content-derived key and leaves the old one standing, where provenance still says which run used which. `tests/test_evidence.py` holds the -partition — a new tier must be classified into one scope or the other. +partition — a new tier must be classified into one scope or the other. The open cost of +that exemption — superseded asset-level rows never retire, so a consumer reading one of +those tiers directly sees every generation at once — is issue #25; the projection is +unaffected, since `from_store` does not read them. **What does not flow into the store**: `locks`, `questions`, `artifacts`, and span `excerpt` windows. Those are document-layer records — a human edit, an open question, a diff --git a/paces/evidence.py b/paces/evidence.py index 539a6a7..75e9e4c 100644 --- a/paces/evidence.py +++ b/paces/evidence.py @@ -629,6 +629,13 @@ def to_store( #: pointing at nothing. A re-measure that moves a boundary therefore *adds* a #: row under a new content-derived key and leaves the old one standing, where #: provenance still says which run used which. +#: +#: The cost, and it is a real one: superseded rows here have no retirement +#: path, so a consumer reading one of these tiers *directly* sees every +#: generation superimposed. :func:`from_store` never reads them, so the +#: projection is unaffected — but nothing in a row says which measurement is +#: current, and the fix (a GC verb, a run marker, or a ruling that these +#: tiers are append-only by design) is thorwhalen/paces#25, sibling of #24. ASSET_SCOPED_TIERS = frozenset({PASS_TIER, BEAT_TIER, WORD_TIER}) #: Tiers whose rows belong to exactly one guide (every body carries a