feat!: multimodal integration and a content-addressed media package by lloyal-research · Pull Request #118 · lloyal-ai/hdk

lloyal-research · 2026-08-31T22:32:36Z

Two things: the multimodal (vision) integration in the HDK, and the
content-addressed media package the integration needs to be replayable.

Multimodal integration

An embedding ingress rail beside the token rail, converging at the KV cache.
Everything downstream — fork, branch, batched decode, sampling, prune — is
already modality-agnostic, so no per-target code was needed.

Three entry points, one contract, so there is one media pipeline rather than
three:

entry	builder	trace role
user turn	`buildUserDeltaMultimodal`	`warmDelta`
spine / standing reference	`mediaContent` via `withSpine`	`spineHeader`
tool result	`buildToolResultDeltaMultimodal`	`toolResult`

Shared downstream: deltaCells(ctx, delta) for admission cost (an operation on
the delta, so builders stay pure), Branch.prefillMultimodal for landing, and
one replay path that resolves roots → expands → flattens → same call. Origin
matters for replay: an attachment must go back through the builder it came
from, or the rebuilt KV differs. branch:prefill.role already discriminated
the three.

Media items settle on their own rail — cohort dispatch, per-item failure
isolation, defer rather than fail when an image does not fit.

Admission previously could not account for media; deltaCells measures before
the decode and SETTLE charges it. Measuring rather than estimating is required
because image cost is non-additive: Qwen pairs images temporally (1→580,
2→580, 3→1142, 4→1142, 5→1704 cells), so a per-image estimate over-commits ~2×
on even counts.

Verified on real weights: the cat fixture prefills at 579 cells and serves
three forked agents.

Content-addressed media package

A trace records the media marker, not the pixels, so a media-bearing run could
not be replayed or inspected. Content is now stored as an OCI Image Layout —
oras reads the store directory with none of our code in the path.

An attachment references a manifest, never a blob. That is what makes video
(one manifest, N frame representations) and live capture additive rather than a
second pipeline.

Rules the implementation depends on: the store is never dev-gated
(addressability is a replay requirement, not telemetry); index.json is a
discovery catalogue, never the runtime authority; write order is blobs →
manifest → index, so a crash leaves orphan blobs rather than a manifest
pointing at absent content; ai.lloyal.derive.* records what derived a
representation, because one source under two settings is different pixels and
different KV.

The composer uploads over HTTP and sends root descriptors. submit_query has
no base64 field — bytes never go over the socket, whose replay history is sized
for small frames.

Breaking

@lloyal-labs/agents no longer exports the content vocabulary. Attachment,
AttachmentStore, ContentIngress, Descriptor, sniffMediaType,
materialize and 12 others moved to @lloyal-labs/media (469 lines and 21 of
137 exports had been defining content inside the orchestration package).
FileAttachmentStore → @lloyal-labs/media/node; createProjectMediaStore
stays in rig, since where a project keeps content is harness policy while the
layout is format. Agents' public surface: 61 → 43 exports.

packages/media is a dependency root: . is pure and browser-safe (zero
internal deps), ./node holds what needs a runtime (sharp, the filesystem).

Defects fixed

Each with a test confirmed red first, then mutation-verified.


a media failure	tore down the whole pool, not just the one agent
`GET /v1/content`	sent `Content-Length` with an empty body
pass-through	rescued 1 of 6 projector formats
the published package	could not normalize — sharp was a devDependency
a media spine	left no replayable seed; `Branch.prefillMultimodal` never implemented the throw its docs promised, so four callers continued against a poisoned branch
the tool-result trace	was lossy, and `branch:prefill` claimed KV had moved before the dispatch that moves it

Two more surfaced while writing the tests. The stall-break was passing the
policy a cost of zero for media results (prefillTokens is [] on that rail
by construction). And a poisoned agent met every condition the termination
sweep tests, so it was force-recovered on a branch just marked unresumable,
emitting two terminal events for one agent.

Decomplection pass

One concept per change: one home for content; one meaning for Attachment
(branded, so a representation can no longer pass as a root — the confusion that
shipped as the replay marker-guard bug); one failure convention, writes throw
and lookups return nothing (26 non-null assertions deleted); one media-type
authority, the bytes decide, so the HTTP route stops trusting a client's
Content-Type; one unit, cells not tokens (four fields carried the wrong
one); one rail model as a discriminated union; one discard set; one
admission-failure event; one commit sequence for manifests; and useTraceScope
became a resource, removing a close() four callers could forget.

Normalization safety

A decompression-bomb ceiling, a decode timeout, process-wide bounded
concurrency with a bounded wait, header dimensions for formats sharp cannot
read, and forced derivation on non-identity EXIF orientation even under the
pixel ceiling — stb_image has no orientation handling, so a phone photo small
enough to pass through reached the model sideways.

Known limit, recorded in code: a non-sRGB profile forces derivation and the
profile is stripped, giving one consistent interpretation, but the pixels are
not converted. sharp/libvips performs no ICC transform (measured three ways).

Verification

a test-only tsc project — test files were in no program, so a double could
drift from the interface it claimed. It caught six sites in one rename that
the suite would not have seen, since vitest erases types
npm run verify:packed — walks the packed entry's own requires, so
browser-safety is structural rather than conventional
npm run verify:oci — new CI job, seven checks driving oras against a
layout the real ingress wrote and our reader against a layout oras wrote

826 tests / 94 files.

Open — not ready to merge

A served-host segfault is unexplained: with two SessionContexts and the
reranker provisioned, BranchStore::allocate_slot faults on a garbage freelist
index. It reproduces on a clean-room scaffold and on the CLI target, and it is
not multimodal — a text-only boot crashes identically.

Cleared by direct test: the kernel (real-inference tiers + ASan), the binding
(multimodal fan-out on the production pairing), the SDK core, the harness code,
the serving layer, config/sizing, models, and duplicate effection.

The manual end-to-end gate is unmet until that is resolved.

…uilder The SDK counterpart of lloyal.node's embedding rail (feat/mtmd there). Composition mirrors the text path exactly; the one difference is where tokenization lives — mtmd owns it, so the multimodal delta stops at the string stage. - ContextOptions: mmprojPath (fail-loud load), imageMinTokens / imageMaxTokens (per-image budget — the video-frames lever). - SessionContext: supportsVision()/supportsAudio() probes; _storePrefillMultimodal → per-branch MultimodalPrefillResult {tokensDecoded, positionAdvance} (JS can't know multimodal token counts; traces + pressure math need them). - deltas: MEDIA_MARKER ('<__media__>' — the one place the constant lives) + buildUserDeltaMultimodal(ctx, content, images, opts) → {sep, prompt, bitmaps}. Emits structured media_marker content parts (the chat layer's native part type — never spliced into a string); same formatChatSync options as buildUserDelta. - Branch.prefillMultimodal(prompt, bitmaps, sepTokens?) — cohort-of-1 through the store path, like prefill(). - Session.prefillUserMultimodal — trunk turn with images; observer tokenCount from the native counts. Images land as a shared prefix: spine and agents forked from the trunk attend them with zero re-encode. - README: multimodal section.

rig's resolveModel fetches into <projectRoot>/models/<role>/<id>.gguf — the repo root when tests or examples run here — and only the harness-cli template path was ignored, so a 610 MB reranker GGUF sat untracked and stageable. Mirrors lloyal.node's existing `models/` rule.

An image entering a run had no durable identity. A trace records the media marker, never the pixels, so a media-bearing run could not be replayed or inspected — and admission could not account for what an image cost, because image cost is non-additive (measured: 1 and 2 images both cost 580 cells on Qwen3.5, 3 and 4 both 1142). The substrate is an OCI Image Layout, not an approximation of one: `oras` and `crane` read a store directory with none of our code in the path, which keeps distribution a later adapter rather than a rewrite. An attachment references a MANIFEST, never a blob — the seam that makes video (one manifest, N frame representations) and live capture additive rather than a second pipeline. BREAKING: `@lloyal-labs/agents` no longer exports the content vocabulary. `Attachment`, `AttachmentStore`, `ContentIngress`, `Descriptor`, `sniffMediaType`, `materialize` and 12 others now live in `@lloyal-labs/media`. Agents NAMES attachments; it no longer defines them. `FileAttachmentStore` moves to `@lloyal-labs/media/node`; `createProjectMediaStore` stays in rig, because WHERE a project keeps content is harness policy while the layout is format. `packages/media` is a dependency ROOT — `.` is pure and browser-safe (zero internal deps), `./node` holds what needs a runtime (sharp, the filesystem). `verify:packed` proves that structurally, walking the packed entry's own requires rather than trusting that an import succeeded. Defects fixed, each with a test confirmed red first: - a media failure tore down the whole pool, not the one agent - `GET /v1/content` sent Content-Length with an empty body - pass-through rescued 1 of 6 projector formats - the published package could not normalize (sharp was a devDependency) - a media spine left no replayable seed, and `Branch.prefillMultimodal` never implemented the throw its own docs promised — four callers carried on against a POISONED branch - the tool-result trace was lossy, and `branch:prefill` claimed KV had moved before the dispatch that moves it Decomplection, one concept per change: - one home for content · one meaning for `Attachment` (branded, so a representation can no longer pass as a root — the confusion that shipped as the replay marker-guard bug) · one failure convention (writes throw, lookups return nothing; 26 non-null assertions deleted) · one media-type authority (the bytes decide; the HTTP route stops trusting a client's Content-Type) · one unit (`cells`, not `tokens` — four fields carried the wrong one) · one rail model (a discriminated union; the stall-break was telling the policy a media result costs ZERO) · one discard set (a poisoned agent was force-recovered and got two terminal events) · one admission failure event · one commit sequence for manifests Normalization safety, which PR-3 requires: a decompression-bomb ceiling, a decode timeout, process-wide bounded concurrency with a bounded wait, header dimensions for formats sharp cannot read, and an admission policy that forces derivation on non-identity EXIF orientation even under the pixel ceiling — `stb_image` contains no orientation handling, so a phone photo small enough to pass through reached the model sideways. The composer now uploads over HTTP and sends root descriptors; bytes never touch the socket, whose replay history is sized for tiny frames. Verification added: a test-only tsc project (test files were in no program, so a double could drift from the interface it claimed), `verify:packed`, and `verify:oci` — seven checks driving `oras` against a layout the real ingress wrote and our reader against a layout `oras` wrote, in CI.

Copilot

Pull request overview

Introduces replayable multimodal support through a new content-addressed media package, SDK/agent embedding rails, runtime model pairing, and expanded verification.

Changes:

Adds multimodal prefill, admission accounting, tracing, and replay.
Adds OCI-backed media storage, normalization, and HTTP content routes.
Expands type-checking, tests, packaging checks, and developer controls.

Reviewed changes

Copilot reviewed 96 out of 98 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
`tsconfig.test.json`	Adds test type-checking
`scripts/verify-packed-install.sh`	Verifies packed media package
`scripts/verify-oci-conformance.sh`	Verifies OCI interoperability
`.github/workflows/ci.yml`	Adds type and OCI checks
`.gitignore`	Ignores provisioned models
`package.json`	Registers media and verification scripts
`packages/sdk/src/types.ts`	Adds multimodal contracts
`packages/sdk/src/Session.ts`	Adds multimodal trunk prefill
`packages/sdk/src/index.ts`	Exports multimodal APIs
`packages/sdk/src/deltas.ts`	Builds and measures media deltas
`packages/sdk/src/BranchStore.ts`	Adds cohort media prefill
`packages/sdk/src/Branch.ts`	Adds branch media prefill
`packages/sdk/README.md`	Documents vision APIs
`packages/sdk/test/MockSessionContext.ts`	Mocks multimodal behavior
`packages/sdk/test/deltas-multimodal.test.ts`	Tests multimodal deltas
`packages/sdk/test/branch-double-free.test.ts`	Tests stale branch pruning
`packages/sdk/test/rerank-instruction.test.ts`	Preserves fixture narrowing
`packages/media/package.json`	Defines media package
`packages/media/tsconfig.json`	Configures media build
`packages/media/LICENSE`	Adds package license
`packages/media/LICENSE-FAQ.md`	Adds license guidance
`packages/media/README.md`	Documents media format
`packages/media/src/index.ts`	Defines browser-safe entry
`packages/media/src/node.ts`	Defines Node entry
`packages/media/src/attachment.ts`	Defines OCI attachment shapes
`packages/media/src/store.ts`	Defines store contract
`packages/media/src/file-store.ts`	Implements OCI filesystem store
`packages/media/src/ingress.ts`	Adds ingestion and materialization
`packages/media/src/media-type.ts`	Detects projector formats
`packages/media/src/image.ts`	Normalizes and commits images
`packages/media/test/store.test.ts`	Tests null store
`packages/media/test/file-store.test.ts`	Tests OCI layout
`packages/media/test/commit-manifest.test.ts`	Tests manifest commits
`packages/media/test/normalize.test.ts`	Tests normalization safety
`packages/agents/package.json`	Adds media dependency
`packages/agents/tsconfig.json`	References media project
`packages/agents/src/context.ts`	Adds media contexts
`packages/agents/src/init.ts`	Installs attachment storage
`packages/agents/src/index.ts`	Updates public exports
`packages/agents/src/prepare-content.ts`	Adds media admission barrier
`packages/agents/src/agent-pool.ts`	Integrates tool-result media
`packages/agents/src/spine.ts`	Adds multimodal spines
`packages/agents/src/replay.ts`	Replays media-backed spines
`packages/agents/src/Tool.ts`	Defines tool media channel
`packages/agents/src/Agent.ts`	Tracks result cells
`packages/agents/src/AgentPolicy.ts`	Uses cell-based settlement
`packages/agents/src/trace-types.ts`	Extends media trace schema
`packages/agents/src/trace-scope.ts`	Makes scopes resources
`packages/agents/src/trace-writer.ts`	Updates scope documentation
`packages/agents/src/use-agent.ts`	Uses managed trace scopes
`packages/agents/README.md`	Documents shared image prefixes
`packages/agents/test/attachments.test.ts`	Tests content preparation/replay
`packages/agents/test/agent-pool.test.ts`	Extends pool media coverage
`packages/agents/test/tool-media.test.ts`	Tests media extraction
`packages/agents/test/spine-multimodal.test.ts`	Tests multimodal spine behavior
`packages/agents/test/trace-scope-halt.test.ts`	Tests halted scope cleanup
`packages/agents/test/spawn-agents.test.ts`	Reuses format fixture
`packages/agents/test/helpers/raw-ingress.ts`	Adds test ingress
`packages/agents/test/helpers/memory-store.ts`	Adds in-memory test store
`packages/agents/test/helpers/media.ts`	Adds media fixtures
`packages/agents/test/helpers/format-config.ts`	Centralizes format fixture
`packages/agents/test/invariants/harness.ts`	Instruments media operations
`packages/agents/test/invariants/predicates.ts`	Adds failure-isolation predicate
`packages/agents/test/invariants/scenarios/no-projector-says-so.scenario.test.ts`	Tests text-visible media failure
`packages/agents/test/invariants/scenarios/media-prefill-failure-claims-no-kv.scenario.test.ts`	Tests poisoned-prefill tracing
`packages/agents/test/invariants/scenarios/media-ingress-failure-isolated.scenario.test.ts`	Tests ingress isolation
`packages/agents/test/invariants/scenarios/discarded-agent-not-resurrected.scenario.test.ts`	Tests discard permanence
`packages/agents/test/invariants/scenarios/deferred-media-cost-is-honest.scenario.test.ts`	Tests measured media cost
`packages/agents/test/invariants/scenarios/decision-matrix.scenario.test.ts`	Tightens policy typing
`packages/agents/test/invariants/scenarios/agent-cancel-no-sweep-recovery.scenario.test.ts`	Updates discard terminology
`packages/agents/test/authGuard.test.ts`	Updates typed fixtures
`packages/agents/test/AgentPolicy.test.ts`	Updates policy contracts
`packages/agents/test/Agent.test.ts`	Updates cell accounting tests
`packages/rig/package.json`	Adds media dependency
`packages/rig/tsconfig.json`	References media declarations
`packages/rig/src/models.ts`	Pairs LLMs with projectors
`packages/rig/src/media-store.ts`	Defines project media location
`packages/rig/src/content-routes.ts`	Adds HTTP content plane
`packages/rig/src/trace-sink.ts`	Adds managed trace writer
`packages/rig/src/runner.ts`	Carries attachment stores
`packages/rig/src/node.ts`	Exports Node media APIs
`packages/rig/src/tools/delegate.ts`	Uses managed trace scopes
`packages/rig/test/content-routes.test.ts`	Tests content routes
`packages/rig/test/verification-properties.test.ts`	Updates ability fixtures
`packages/rig/test/spine-render.test.ts`	Updates ability fixtures
`packages/rig/test/runner-substrate.test.ts`	Fixes generic inference
`packages/rig/test/reranker-options.test.ts`	Types context mock arguments
`packages/rig/test/registry.test.ts`	Updates ability vocabulary
`packages/rig/test/provision.test.ts`	Types resolver mock
`packages/rig/test/plan-routing-key.test.ts`	Corrects schema casts
`packages/rig/test/models.test.ts`	Corrects response body typing
`packages/rig/test/keyless-search.test.ts`	Corrects resolver narrowing
`packages/rig/test/define-ability.test.ts`	Uses canonical manifest type
`packages/dev-tools/src/index.ts`	Adds image-setting tiers
`packages/dev-tools/src/react.tsx`	Adds stepped sliders
`packages/abilities/corpus/test/bm25.test.ts`	Extends token fixture
`packages/abilities/corpus/test/ability.test.ts`	Completes reranker mock

Suppressed comments (1)

packages/agents/src/trace-types.ts:102

This attachment documentation also refers to tokenCount, but the enclosing branch:prefill event now exposes the whole-prefill cost as cells. Keeping the old name makes the public trace contract internally inconsistent.

       *  Deliberately no per-image cell count: image cost is not additive.
       *  A model that pairs images temporally charges the same cells for two
       *  as for one (measured on Qwen3.5: 1 and 2 images both cost 580 cells,
       *  3 and 4 both cost 1142), so a per-image share would be a fiction.
       *  `tokenCount` above is the whole prefill's real cost.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

`AbilityDescriptor` declared `iconUrl` and `AbilityManifest` carried one, but the builder hardcoded `undefined` — so the field has always been dead and every surface fell back to a glyph, whether or not the ability named a mark. Read it off the manifest, which `describe` already has in hand. A distributed ability can now be recognised by its own mark rather than by its position in a list; absent still means glyph, so nothing that renders one changes.

Copilot

Pull request overview

Copilot reviewed 97 out of 99 changed files in this pull request and generated 1 comment.

Suppressed comments (11)

Previously missed (4) — in code that hasn't changed since the last review.

packages/rig/src/trace-sink.ts:65

Ensure the descriptor is closed even when flushing fails (for example, after the disk fills). As written, a thrown writer.flush() skips closeSync, leaking one file descriptor per affected session during teardown.
packages/media/src/attachment.ts:160
Select only layers explicitly tagged representation. Treating every non-source layer as model input means an untagged or future-role layer is projected during replay; the HTTP representation route can also expose an untagged retained source. The manifest contract and conformance check require explicit roles, so unknown roles should be skipped or rejected.
packages/rig/src/content-routes.ts:305
The upload endpoint is not connected to the repository's composer contract described by this PR. examples/shared/tui-ink/commands.ts:13 still defines submit_query with only query and mode, and Composer.tsx:131 sends that text-only shape; no production caller mounts this route or sends returned root descriptors. Wire upload and attachment roots into the command path before claiming composer multimodal support.
packages/rig/src/content-routes.ts:44
Type this callback as returning Attachment, not any Descriptor. The endpoint promises a root manifest and the media design relies on never treating a representation blob as an attachment; the current API lets an adapter return store.putBlob(...) and still send 201 with an unusable root.

packages/sdk/src/Session.ts:167

A failed warm multimodal prefill poisons this branch by the method's own contract, but the session keeps it as trunk. If a caller catches the exception, later session operations continue against partially mutated KV. Invalidate/prune the trunk on this failure, or prefill a disposable fork and promote only after success.
packages/sdk/src/Session.ts:172
Clean up the newly allocated cold trunk when multimodal prefill throws. It has not been assigned to this._trunk, so Session.dispose() cannot reach it and the native branch slot remains allocated.
packages/agents/src/spine.ts:263
The write-ahead seed still is not replayable when this call fails: attachment roots are only written later on branch:prefill, which is intentionally omitted after an error. extractSpineSeed then sees markers but no references, and reconstructBranch always throws. Record the committed roots on the write-ahead event (or a separate intent event) and extract them from there.

      writeSpineSeed();
      const counts = yield* call(() =>
        spine.prefillMultimodal(formatted.prompt, bitmaps));

packages/agents/src/Tool.ts:234

Always remove the reserved _images field when it is present, even if its value is malformed. With _images: someUint8Array instead of an array, this branch returns the original object and JSON.stringify sends every byte on the token rail—the exact large-payload failure this helper is intended to prevent.

  const { [TOOL_MEDIA_KEY]: raw, ...rest } = result as Record<string, unknown>;
  if (!Array.isArray(raw)) return { media: [], result };
  return {
    media: raw.filter((b): b is Uint8Array => b instanceof Uint8Array),
    result: rest,

packages/rig/src/content-routes.ts:305

The configured timeout and cancellation stop at body ingestion: the timer is cleared on end, and the optional AbortSignal is never passed to ingest. A disconnected client or hanging normalizer can therefore keep occupying the process-wide normalization queue and complete an orphan upload after the request is gone. Keep an abort controller for the full request lifecycle, pass its signal here, and enforce uploadTimeoutMs through ingress completion.
packages/rig/src/models.ts:403
Do not infer a catalog projector when config.path overrides the catalog LLM. In that case modelPath may be any custom model while llmId still names the manifest default, so this pairs incompatible weights and makes context creation fail. Require an explicit config.mmproj for path-based models.
packages/agents/src/agent-pool.ts:1359
Record tool history only after the dispatch succeeds. This currently runs during admission, before either prefill call; a media entry that later returns error is still recorded as settled, and contextAfterPercent is sampled before any admitted cells have entered KV. This makes the new cell-based history inaccurate for both successful and failed media results.

        const postSettle = new ContextPressure(ctx, pressureOpts);
        a.recordToolResult({
          name: item.toolName, args: item.args,
          resultCells: itemCells,
          contextAfterPercent: postSettle.percentAvailable,
          timestamp: performance.now(),
        });

The arc's five packages take prerelease versions under the `alpha` dist-tag — media 0.2.0-alpha.0, sdk 3.2.0-alpha.0, agents 6.0.0-alpha.0 (a MAJOR: the content vocabulary left for @lloyal-labs/media), rig 5.6.0-alpha.0, dev-tools 0.5.0-alpha.0 — with exact internal pins, because semver ranges EXCLUDE prereleases and a caret would produce uninstallable dependents. Untouched siblings keep their stables; the publish loop's npm-view guard skips them. The release workflow learns two things it was missing: media joins the dist-verify, the typecheck list and the publish loop (right after channel-verify — sdk, agents and rig all depend on it), and the dist-tag now follows each package's own version, so an -alpha.N can never move `latest`. scripts/cut-alpha.mjs cuts the next set: bases come from the registry at cut time, so a stable that ships mid-arc self-corrects on the next cut.

Opens with what becomes true when you attach an image — replay from the exact bytes, inspectable forever, a valid OCI Image Layout any registry already hosts — and adds the quickstart the page never had. Two corrections ride along: the admission section no longer claims an unsupported file "fails mid-run on a branch already in flight" (MtmdSource rejects before any decode, branch untouched — admission decides who reports it), and pass-through is now stated once, conditional on the full five-axis policy, instead of a two-axis summary the policy table below contradicted.

Correctness, each with the test that pins it: - a failed multimodal prefill now prunes and clears the Session trunk (warm) and the never-promoted branch (cold) instead of leaving poisoned KV installed or leaking the slot - a literal media marker in user/system/tool text is defanged at the delta builders — the native splitter would count it and desynchronize markers from bitmaps - takeToolMedia strips a MALFORMED media channel instead of serializing byte indices onto the token rail - the spine seed carries its attachment roots WRITE-AHEAD, and extractSpineSeed falls back to them — a spine whose prefill failed is exactly the one replay must rebuild, and it had no roots to rebuild from - probe branch:prefill events are buffered until the batched dispatch lands, keeping the success-only event contract - asAttachment validates the complete untrusted-JSON shape; getManifest validates every layer descriptor instead of crashing representationsOf on a corrupt manifest - one upload deadline now spans body transfer AND ingress via AbortSignal - a custom model path no longer infers a catalog projector for a model that is not running Surface honesty: NoContentIngress keeps the full contract signature; verify:packed generates its fixture instead of reaching into a sibling checkout; the range control tells assistive tech the value, not the index; two stale tokenCount cross-references; the agents README media example now shows the store + ingress installs it actually needs.

`replayTurns` is the delta-replay primitive under `reconstructBranch`, exported on its own: seed REBUILD verifies it can restore the recorded state or throws; delta replay is provenance-blind — the branch may be a fresh rebuild or a fork of live, resident state, and the primitive neither knows nor checks. It owns nothing about lifetime. `reconstructBranch` keeps its exact contract and becomes visibly a composition: seed rebuild, then replayTurns. A caller continuing from a live fork composes forkSync + replayTurns and owns the prefix contract by construction — no mode flag, no guards that conditionally skip. Test pins the fork case, the one reconstructBranch itself never exercises.

…dges A BPE piece is a byte sequence that can end mid-character; converting each piece to a string independently replaced both halves with U+FFFD — every multi-byte character split across tokens (÷, emoji, CJK) streamed as diamonds and was persisted that way in reports and traces. The fix is a value-level fold: splitCompleteUtf8 emits up to the last complete character boundary and holds the 0-3 byte tail; produceSync derives text purely from (held, piece); commit installs the successor tail — including the batched BranchStore.commit the agent pool uses — and forks inherit the tail mid-character. The binding gains one passthrough method (tokenToBytes) so bytes survive to the layer that assembles them. Property tests: any chunking of a corpus reproduces it exactly; junk is decided immediately, never held; produce stays a pure observation. Verified on real weights: "12 �� 4 = 3 ✅ ��" → "12 ÷ 4 = 3 ✅ 📋".

…outcome docs/self-healing.md, wave A. llama_decode's return code arrives as data (decodeRcOf — the one reader; Branch.prefillMultimodal forwards it through its re-wrap) and SETTLE answers each class with the cheapest response that preserves the run: - rc 1 (no KV slot, state restored — the branch is INTACT): the item DEFERS and re-dispatches next tick, riding through the stall-break on its own budget; MAX_DEFER_ATTEMPTS escalates to a per-agent terminal. On the token rail this replaces a latent whole-pool death — the batched settle dispatch was unwrapped, and one capacity reject took every agent down. - rc -1 (invalid input, restored): the item is dropped and the model is told what it did not see, on the same channel as the no-projector note. - rc 2 / < -1 (partial ubatches remain): the existing poison path, unchanged, now with the rc on pool:settleFailed. A tripwire counts consecutive fatals and stops the ladder — a backend in a sticky error state fails every decode, and deferring there burns budget for nothing. SETTLE bookkeeping (settle order, tool history, re-activation, branch:prefill) moved to success-only — it runs after the dispatch lands, so the record only ever describes what happened. New trace event pool:agentDefer carries rc, attempt and a pressure snapshot (the diagnostic that separates honest fullness from fragmentation). Six ladder tests; the ride-through and the tripwire are mutation-verified.

docs/self-healing.md, wave B. The poison costs the agent its branch, not its task: within budget (one heal per lineage) and with the backend healthy, the pool forks the spine — the prefix, seed images included, rides the fork for free — replays the agent's record, and admits the replacement as a NEW agent. The original's agent:failed stands; the lineage rides pool:agentHeal {of, agentId, rc, attempt, pressure}. The record is the trace, held where heal can reach it: every piece was already emitted (agent:turn.rawOutput, tool:result, probeText) — the pool now keeps the same data as one ordered AgentTurnRecord list per agent, alongside the retained spawn spec (the birth certificate the reboot design also wants). replayAgentTurns is the agent-shaped sibling of replayTurns: assistant deltas, tool-result deltas (media-bearing ones resolve through materialize and the throwing single-branch prefill), probe prefills — provenance-blind and lifetime-free. Heals drain at the SPAWN phase, suffix batched with ordinary spawns, on the loop fiber. A replay that cannot land discards the half-built replacement; a replacement that poisons again goes terminal — a second failure on replayed state is evidence, not bad luck. Three tests: the warm respawn end to end, the budget, and the record-replay primitive across all four delta kinds. The enqueue is mutation-verified.

Found on real weights, not in mocks: the record's tail is the poisoned transaction itself — an assistant turn whose tool call never settled — and replaying it left the replacement dangling mid-call (it emitted a stray think and stopped instead of re-driving the tool). The heal enqueue now drops trailing assistant entries, so the replacement regenerates that turn and calls the tool itself. Proven by repro-heal.mjs: injected fatal on the first media cohort → pool:agentHeal → the replacement re-called the tool against the real kernel and answered the image question.

Cut 0's media was burned before the review fixes landed (the publish-loop guard skips existing versions, so pressing the button would have shipped a stale @lloyal-labs/media). One set id for the whole cut, as designed: everything moves to -alpha.1 with exact pins.

The cut script bumped from registry latest, but media's latest IS an alpha (the manual first publish stamps latest — npm behavior). An alpha is not a base: the stable it prefigures hasn't shipped, so its release triple is the pending base, unbumped. Cut 1 for media is 0.2.0-alpha.1, not 0.3.0-alpha.1.

… session/spine timeline The rig's trace writer now mirrors every write onto the dev bus, with attribution carried in the event data itself (agentId/callId); the pool tee and its bridge are gone. The pane folds the mirror into three scope rails — session, spine, lanes — with elbow connectors for every orchestrator shape, spine growth ticks from spine:extend, and a tokens-saved line on the cache-read convention: every fork-inherited token counts; the spine's own build is the cache write.

…prefill pair Session's prefill observer reported turns ENTERING the trunk's KV; nothing reported a trunk leaving it, so Session-level prunes (dispose, promote moving the crown off a live trunk, the multimodal poison path) were invisible — trunk generations appeared in the trace but never left, and diagnosing a doc-switch meant native refcounts and pool-pressure arithmetic. TrunkReleaseObserver fires after each release with the handle and position; initAgents bridges it to the same branch:prune vocabulary the pool already writes. Release info is captured BEFORE the prune (the getters are not for disposed branches). Unit-tested over the mock: dispose fires once then goes silent, release+rebirth yields a fresh generation, promote distinguishes supersede from re-crown. MockSessionContext's relative imports gain .js extensions (proper ESM — required by NodeNext consumers; bundler-resolution consumers unaffected).

The trunk feed was an append-only log of every turn-commit the mirror ever emitted — it spanned documents, so the pane implied KV residency it did not have. With releases now on the wire, the fold deletes a released trunk's turns (handle-matched, never a blanket clear: an agent branch's prune must not touch the feed, and the match is order-independent with the next generation's first commit). The panel now means what it shows: the conversation the model can currently attend. Model test walks fold → unrelated prune → the trunk's own release → a clean next generation.

lloyal-research · 2026-09-02T15:39:24Z

Two additions riding this branch from the trunk-observability arc:

14cc4f8 — sdk/agents: the trunk release observer. Session's prefill observer reported turns entering the KV; nothing reported a trunk leaving it, so dispose/promote were invisible in the trace (diagnosing a doc switch meant native refcounts and pool-pressure arithmetic). TrunkReleaseObserver fires on every Session-level release; initAgents bridges it to the existing branch:prune vocabulary. Unit-tested over the mock (dispose-once, release+rebirth generations, promote supersede-vs-recrown). MockSessionContext imports gain .js extensions — proper ESM, required by NodeNext consumers.
d62f84a — dev-tools: the session panel shows the resident conversation only. The trunk feed was an append-only cross-document log; with releases on the wire, the fold deletes a released trunk's turns (handle-matched), so the panel shows exactly what the model can currently attend.

Full suite: 98 files, 881 passed. Both changes are pure JS over the existing native surface — no GPU relevance.

…ing.js MockSessionContext lived under test/, which the files whitelist never ships — so a scaffolded harness could not run a behavioural suite over it (its import only ever resolved through workspace symlinks). Promoted to src/testing.ts and compiled into dist; the deep path @lloyal-labs/sdk/dist/testing.js resolves plainly (no exports map). Every in-repo consumer (sdk tests, agents suites + invariants) moves to the same entry, so there is ONE mock with one address.

A heal was queued as spec plus records and forked and tokenized only in the executor, at batch time, so the scheduler could not admit it and a replacement that did not fit joined the shared batch, where a no-slot rc ended the whole pool. Now a heal is a spawn request carrying its lineage: forked and measured when queued, through the same setup the orchestrator's spawn uses, admitted against headroom, replayed once its suffix has landed. HealRequest, the pending heal lane and the executor's heal loop are gone; a replacement that cannot fit is dropped with pressure_init like any spawn. A pruned agent now reads disposed: the prune pass transitions it through the lifecycle table and teardown forces it, so the sweep no longer consults the branch beside the status. Recovery without a terminal tool reads the way the voluntary path reads: a designated terminal means its call; otherwise a call's result if the model made one, else its prose. The rig's delegate pools designate no terminal tool, so their recovery turns recover instead of failing. The package-surface scan reads export-from re-exports as well as imports. Each fix landed behind a test in test/invariants that failed first.

…aiting it Items and retries already read their owner's status; the dispatch lane had only the same-tick drop set, so a cancel enacted inside a pause left a parked tool call that went out after play, for an agent that had already failed. One predicate now serves all three lanes: queued work runs only for an agent still awaiting_tool that was not dropped in this schedule. Status is the durable truth across ticks and holds; the drop set covers the window between a decision and its enactment. The schedule no longer reads the ambient clock: the wall is sampled into the tick state with the pressure, and retry parks are due against that sample, so the same state decides the same way whenever it runs. The pressure ledger only spends: minus() rejects a negative argument and clamps remaining at zero the way the kernel's KvPressure does. npm test is an alias of the Vitest suite again, as the README says. Each fix landed behind a test in test/invariants that failed first, the cancel-during-hold scenario re-verified red after its retiming.

…s priced whole Two cancel signals for one agent in one tick decided two drops; the second enactment asked the lifecycle table for idle to idle and the run closed partial. Every drop is now decided through one writer that keeps the drop set, and every step that decides something for an agent consults that set, the cancels included. A heal was admitted for its suffix and then replayed its whole lineage on faith. The admission invariant, stated fully, is that every prefill a request will cause is in its admitted cost: prepareReplay builds each delta of the lineage once, with the same builders the live path uses, and prices it; spawnCells is the one cost of a spawn; forge is the one way a spawn request is made, for the orchestrator and for a heal alike; the executor replays exactly the priced steps. recoveryBudget and maxConcurrentTools are validated at open beside the hardLimit >= nBatch check, with the value named: a non-positive budget cut every report at its first token, a non-positive permit count hung the first fan-out call. The package-surface scan reads fenced code in tracked markdown; the README import blocks name what the index exports, the removed diverge and the pre-rename App names are gone from the documentation and the doc comments. Each fix landed behind a test in test/invariants that failed first.

…s forged after reclamation Four findings from the fifth review, each behind a test that was red first. Thresholds are validated at open. One requireInteger checks the recovery budget, the fan-out permit count, and the resolved soft and hard limits as finite integers; the hard limit keeps its nBatch reason in the message. Serial recovery is exempt from the soft reserve, not from the cache. A serial recovery turn is admitted only when its cells fit what physically remains after this tick's earlier admissions minus the hardLimit reserve its own decode needs, and it spends from the same ledger as everything after it. A turn that cannot fit is carried while a sibling can still free KV; at the stall-break a recovery item is re-decided serial only if that rule admits it and skipped otherwise. T9 now asserts the oversized prefill was never issued. Every drop decides its recovery at the drop. The produce-time idle takes the tick's mode instead of leaving a serial agent to the close, so an orchestrator waiting on it resumes against the recovered result rather than a missing one. The close-time sweep is unreachable and removed, with Schedule.sweep, the applier's recovery-mode dependency and the lifecycle edge idle to awaiting_tool. Serial idles free their KV at the drop instead of at close. A heal is forged after the prune pass. The ladder decides the heal and records its lineage on the original; the loop forges it at the next observe, after reclamation, so the replacement asks for a lease once the poisoned branch has given its own back. An original with live children is forged all the same, once, and admitted or refused by fit like any spawn. The forge prices the lineage before forking, setupAgent forks as its last fallible step, and a forge that throws stands down without taking the pool with it. The sdk mock now refunds the image slack it charged on the embedding rail when a branch is pruned, as BranchStore::release does; I42 on the new media scenario found the gap. Scenarios: chain-serial-recovery, serial-recovery-too-big-is-skipped, heal-forged-after-reclaim, heal-lineage-lost-stands-down, five new threshold cases, and mock-image-slack-refund in the sdk.

waitUntilSettled awaited a native decode's settlement from inside a finally block. Effection's contract forbids a yield inside a finally: a halt unwinds the generator with return(), a yielding finally suspends that frame, and the resume takes it out of unwind mode, so once the cleanup finishes execution continues past the operation that was being halted. A new test reproduces it against the old implementation: the halt waited for the decode to settle, then the halted operation carried on, and was not halted at its next suspension either. For the pool that meant an executor halted mid-prefill kept booking, emitting and stepping against a store being torn down. The settle-wait is now registered with ensure() inside a scoped() boundary, which runs on return, error and halt alike. The rig registry's enable() had the same shape on its failure path and is changed the same way. Two signals were used for messaging between operations, which the contract reserves for Channel and Queue. The loop's wake is now a Queue, added to from the orchestrator fiber, fan-out children and the signal watchers alike and read by the loop alone; the completion mailbox stays an array because a queue has no non-blocking drain. An agent's status signal, sent on every transition and consumed by waitFor through each(), is replaced by one future per agent, Agent.final, resolved the first time the agent is final and never by the idle it is born with; waitFor is one yield on it. A contract test refuses any yielding finally in the agents and rig sources and pins createSignal to its one legitimate site, the bridge from a tool's onProgress callback. Not adopted: the task-buffer extension in place of the fan-out permit gate. Its request queue is not halt-aware, so a tool call queued behind the cap would still run after its agent was cancelled; the action-based gate drops the waiter on halt, which the cancel path depends on.

The stall-break is the last resort for a deferred item, and since round five it is terminal for a recovery turn. It fired on "no agent is decoding", the naive quiescence test: a sibling awaiting a fan-out tool, holding a parked retry, dispatching this tick, or dropped or finished this tick is not decoding, but each of those is a route to KV being freed. It now fires only when nothing can make progress: no admitted prefill or spawn, no decode, no dispatch, no tool in flight, no parked retry, and no drop, finish or abandoned retry decided in this schedule. Every lane counted either advances on its own or is bounded to the schedule that decided it. Two candidate lanes are deliberately not counted, because an outstanding obligation is not a route. A pending prune on a branch with live children cannot be reclaimed while the children are the blocked ones, and would have held the stall open forever; a scenario with a cancelled parent and two children blocked on oversized results now guards that the pool terminates. An item the kernel once refused and that no longer fits admission is an oversized item like any other; the ride-through that carried it indefinitely is gone. Deferred extends carry on the same rule or on a stall decision, and their implicit "any alive sibling" term is gone with it. The loop naps when nothing ran. Counting pending work kept a carried item spinning the loop for as long as a sibling's tool was in flight; every push into the pending record happens inside a tick that ran something or arrives with a wake. The slow-tool scenario bounds the pressure samples taken during a sibling's tool call. Four doc comments still described the close-time sweep removed in round five. The pause doc now says decode is suspended at the tick boundary while cancels still run as hold ticks and the prune pass still reclaims. The Effection floor is raised to 4.1 in agents, rig and host: the settle-wait in waitUntilSettled and the registry's cleanup rely on scoped()'s async teardown, which the library's contract says was only correct from 4.1, and the old range admitted versions predating it.

…e reserve, a batch owns its forks The prune pass runs to a fixpoint. It walked the roster once, so a prune-requested parent skipped for its live child stayed resident after the child was pruned later in the same pass. Three readers take the pass as complete: the pressure sample taken right after it, the stall-break's rule that an outstanding prune is not progress, and the close. A parent unpinned by its own pass is now reclaimed in that observe, before all three, so an oversized result that fits once the parent's cells return is admitted rather than stalled, and no lease is held past the close for teardown to find. An extracting agent is finished when the post-admission pressure is critical, not only when its budget is spent. A serial report's budget is infinite and a cohort's reservation lives only in the tick that admitted it, so a report the model would not end decoded past the hard reserve until the native commit failed and the pool closed partial. Finishing salvages what it produced, as the terminal cap does. The spawn batch owns its forks until they enter the roster. The batch prefill is a native call the loop suspends on, and the window between admission and activation is open to everything that can happen during a yield: wind-down halting the orchestrator, whose requests become discarded; or the pool itself halted, which unwinds the executor without visiting any catch. The batch now runs inside a scope whose cleanup returns every fork not yet handed over, and a request discarded while the batch was in flight is pruned instead of activated and announced into a draining pool. A wind-down that lands while a batch is still activating therefore discards the spawns not yet activated; the wind-down scenario fires once its cohort is active, which is what it was written to test. The harness gains a hook that hands a scenario the signal senders, so a signal can be fired from inside an instrumented native call.

…ts that say what is enforced An explicit recoveryBudget above the adaptive ceiling was honoured by cohort recovery and silently capped at 2048 for an in-flight terminal call. The cap now reads the configured value unchanged; the ceiling applies only when no budget is set. The contract is restated where it is read: the budget caps a cohort recovery turn and a voluntary terminal report in either shape, and serial forced recovery has no configured cap and ends at its stop token or when the pressure turns critical. The policy field had claimed the option was unused by the staggered shape, which the terminal cap contradicted. The reserve assertion in the never-stopping-report scenario compared a value clamped at zero against zero. It now requires every landing to leave at least hardLimit minus one and to reach the boundary, and its header speaks for the extractor's unconditional finish only: a non-extracting agent's critical exit is shouldExit ?? critical, which a policy may veto. cellsUsed is documented as it behaves: it grows on prefill and commit and shrinks when a branch is released. The source base class described a bind lifecycle removed at the start of this work and linked a function that never existed here. Its dead context type parameter and import are gone, the class doc describes constructor injection, and the subclasses were migrated with the change of meaning in mind: the remaining parameter is the chunk type, so two-argument uses keep their chunk type and context-only arguments disappear.

… will still hold it The adaptive cohort budget divides the recovery reserve among the agents that will still hold or take KV after the schedule. The count was taken before the schedule's own cancels were removed from it, so a tick that cancelled several agents and reaped one truncated the survivor's report for sharers that never decode again. The count now excludes agents dropped in step zero. A salvage or skip decided later in the same verdict loop also stops decoding, but a pre-verdict count cannot know that, and one divisor for the whole cohort beats a running one that would hand earlier reaps a smaller share than later ones. The sdk mock accepted a handle twice in one multimodal batch. The binding applies the kernel's distinct-handle rule to that rail before its worker runs, so the mock was the permissive one; it now refuses the duplicate on both rails, with a test for each. The public recoveryBudget contract on the policy interface still said the option was unused by the staggered shape; round eight had corrected the options copy and missed this one. The interface doc is now the one contract, naming both enforced uses and the serial exception, and the options field links to it instead of repeating it.

The header sentence describing what a prune refunds had been extended in round five to include the embedding slack and kept its older tail, so it contradicted itself. The formula is now stated once: the position delta plus the image slack, as BranchStore::release refunds it.

…chedule, the agent states and the invariants packages/agents/docs/scheduler.md replaces the local primer with a page that names only what exists: the five phases tied to the file and function that run them, the Schedule fields in the code's own words, the exact AgentStatus set with extracting and final beside it, the Pending record and its two rules, recovery and the return-code ladder, inline versus fan-out tools, and nine invariants each mapped to the test that holds it. The .gitignore gains one negation so packages/agents/docs is tracked while the root docs/ stays local.

…s the way into the code 280 lines and five conceptual essays become 87 lines. The headline, the one code sample and the licence stay. New: a Start-here that points at the lloyal-ai CLI (npx lloyal-ai new) and Build your first harness; a Reading-the-code section that introduces docs/scheduler.md; one line per concept linking the docs page that owns it. The three properties under the sample now name three mechanisms of this runtime: live-state forks, continuous tree batching, and structured concurrency over KV tenancy. docs/ joins the package files so the loop doc ships with the tarball and the relative link resolves on npm.

agents: the pool becomes a scheduler

Copilot

🔵 Needs a closer look

A critical test-command regression and unresolved media, replay, and descriptor correctness defects remain.

Review details

Files reviewed: 185/280 changed files
Comments generated: 3
Review effort level: Balanced

… true, and an ability's icon is read where it lives Three findings from the review of feat/mtmd, each behind a test that was red first. package.json: `npm test` lost its build step when the root test/ and examples/ leftovers went in the scheduler round; 24 test files import workspace packages by name and resolve to dist, so a clean checkout failed the canonical command while CI, which builds before test:unit, stayed green. The build step is back. media: putBlob trusted a file already at the digest path. A drifted or planted blob there left that content unreplayable for every later upload, while get() kept refusing it. The short-circuit is gone: every put writes temp-then-rename over the path, so the name is made true rather than trusted; identical bytes rewrite idempotently and concurrent puts converge. rig: buildAbilityDescriptors read the icon at manifest.iconUrl through a local ManifestLike shape; the manifest declares it under hints.iconUrl, so no icon ever reached a surface. The synonym type is deleted, the function is typed against AbilityManifest, and the field is read where it lives. The catalog's own worker-resolved icon is a separate source and is named as such.

…ts ability, assets on the run Field Note can attach a PDF. The content plane gains a document ingress (PDFium): one text/markdown representation, the PDF retained as source, a facts-only sidecar (sections, page map, per-page counts, page renders as their own image roots). A new first-party ability, `documents`, searches, reads and views those documents; evidence is cited by content address (`attachment://<12 hex>/page/<n>`). media - `createContentIngress` dispatches on bytes: image → normalizer, %PDF- → document ingress. - `pdf.ts` / `pdf-layout.ts`: PDFium codec per document, one shared gate, deterministic coverage under declared bounds; layout from glyph geometry (advance boxes, text-matrix size, baseline grouping, small-caps headings, corroborated titles). Hardened on two real PDFs (Springer chapter, arXiv paper); fixtures + `matrix.pdf` pin the rules. - Sidecar type + `pagesOf`; renders keep the page's aspect ratio. rig - `fitChunks`, `BM25Index`, ranges lifted to rig; `loadDocuments`; content routes gain `GET /v1/media/<m>`, `/config`, `/source`; the registry seeds the store into abilities. - `createReranker` KV default q4_0 → q8_0. Measured on real document windows: at q4_0 ten identical passages spread 4–6 logits across the leaves and the verdict changed sign; at q8_0 the spread is 0.05–0.12 at the same pass time. `reranker-resolution.test.ts` (weights-gated) holds the default to that floor. agents - Assets available to a run: `AgentPoolOptions.attachments`, `ToolContext.attachments` (staged ∪ admitted by any agent), `Source.promptData(attachments)`; tool media may be a descriptor (`TOOL_ATTACHMENTS_KEY`); rail follows bitmaps; `book()` carries roots. - `agent:prefilled` rides the bus beside the `branch:prefill` trace: a host books and shows admissions from the stream it already consumes (announced from `settle()`, since `emit.trace` refuses bus projections). - Exploit mode takes ONE extra pass: `min(tool score, scoreEntailmentBatch)`. BREAKING: `EntailmentScorer.scoreRelevanceBatch` is removed (its only caller). abilities - `documents` (new, 0.1.0): `search_documents` (top-K within a token budget, explore mode always — the attached document is the on-topic universe), `read_document`, `view_page` (projection rule over sidecar facts). Skill: figures by caption, generic names, no silent substitution, failed tools named. - corpus adopts `fitChunks`; web's `fetch_page` says to attach a PDF instead. Gates: npm test (156 files) green, `tsc -p tsconfig.test.json` clean, verify:packed and verify:oci ok, lockfile reconciled; proved end to end on real weights in the scaffold's web target (evidence in the PR).

…nd a test that was red first - media: image bounds are clipped to the page as they are read and a crop is sized by the same `fitScale` rule as a page, so the side and area ceilings hold for crops (an image painted far past the page no longer asks for a 20,000-pixel bitmap); `readPage` walks Form XObjects with a depth bound and a visit cap, applying the pixel-ceiling check to nested images and carrying their bounds into page space through the composed form matrices. Fixtures `clipped.pdf`, `form.pdf`, `bigimage.pdf`, `bigform.pdf` (hand-written in make.sh) pin all three. - rig: `DelegateTool` forwards the delegating call's `attachments` into the child pool — the run's staged roots plus everything admitted so far — so a delegated agent sees the documents; `delegate-assets.test.ts`. - documents: a repeated `view_page` carries the page again with a note. The tool cannot see whether its last result landed (a settle nudge may have replaced it), so suppression left the model blind; admission is the gate. - rig: `reranker-options.test.ts` states the q8_0 default contract; the earlier gate statement had misattributed this file's failure.

feat(media,rig,agents,abilities)!: documents — PDF ingress, a documents ability, assets on the run

sdk 4.0.0-alpha.4, agents 6.0.0-alpha.4, media 0.2.0-alpha.4, rig 5.6.0-alpha.4, dev-tools 0.5.0-alpha.4. Internal pins exact, since semver ranges exclude prereleases. lloyal.node is HELD at 3.2.0-alpha.3: no changes to the binding or liblloyal this cut, so rig, host and sdk keep the published pin rather than the set id. The cutter's EXTERNAL table names it as a member of the arc and computed 3.2.0-alpha.4, which does not exist; the lockfile regeneration refused that pin, which is how it was caught. The lockfile now regenerates clean, so every pin in the set resolves.

Hosted runner minutes are billed and the org's spending limit is reached, so a dispatch dies at job creation — one second, zero steps, no logs, which reads like a broken workflow rather than a billing problem. Self-hosted minutes are not billed. `runs-on` becomes an input so a release can land on a local runner without the file diverging per branch. The `|| 'ubuntu-latest'` fallback is load-bearing rather than defensive: this workflow also triggers on `push: tags: v*`, where `inputs` is null. An empty label queues forever instead of failing. The dist-tag already falls through to `latest` for any version without a prerelease suffix, so one hand-edited manifest could move production from a manual run. A dispatch may now only ever ship a prerelease; `latest` moves through a v* tag alone. The check sits inside the branch that actually publishes, since placing it before the npm-view guard would abort on every already-published stable the loop was going to skip. NODE_AUTH_TOKEN goes. Publishing authenticates through trusted publishing over OIDC, and the secret behind that line gave npm a second credential competing with the exchange — the shape of aws-actions/configure-aws-credentials#453, where a runner's ambient instance credentials shadowed the OIDC path. Removing it also makes the new probe honest: it reports whether the token endpoint is present, which is worth nothing if a token could be doing the work instead. The probe runs unconditionally, outside the publish gate, so a skip_publish rehearsal proves the plumbing before a real cut depends on it.

npm verifies the sigstore bundle's runner claim and refuses anything but a hosted one: 422 Error verifying sigstore provenance bundle: Unsupported GitHub Actions runner environment: "self-hosted". Only "github-hosted" runners are supported when publishing with provenance. Measured on run 34142877384, which built and typechecked fine, minted an OIDC token, signed the statement and got it into the transparency log — and was then rejected at the registry. Nothing published, so no partial release. The condition reads RUNNER_ENVIRONMENT rather than taking a flag. A hosted run keeps attestation with nothing to remember, and a local one drops only what it cannot have. Production is unaffected either way: a stable version ships through a v* tag, a tag push carries no inputs, so runs-on falls back to ubuntu-latest and the condition passes. The alpha gate closes the other direction by refusing to publish a non-prerelease from a dispatch at all. Unattested and stable are therefore mutually exclusive. An unattested publish is announced as a warning rather than passing quietly, and the probe now prints the runner environment so the condition can be seen to work rather than assumed.

A desktop target cannot mount an HTTP server to serve its own media, and should not: these routes authenticate nothing by design — a digest is identity, not authorization — so a loopback port would make a user's documents readable by every process on the machine. Electron serves them from a custom scheme instead, which opens no socket at all. But it needs the same routes. So the routes split the way the scheduler does, decide then enact: resolveContent({ method, path, ifNoneMatch }, store) -> ContentReply | null createContentRoutes keeps its signature and is reimplemented over it, adding only what is HTTP's own: CORS, the preflight, and the upload. The ingress stays with the transport deliberately — every route in the table is a READ, answerable from the store alone, while an upload needs bytes and limits only a transport can supply. A marked door, not an oversight. ContentReply is ONE shape rather than a union over outcome kind. A 404 and a 200 differ in value, not in concern, and an adapter should not branch on a distinction it never acts on. CORS is deliberately absent from it: that is transport policy — an HTTP host has a configured origin, a custom scheme's requests arrive with none — whereas everything in the shape describes the CONTENT and is byte-identical wherever it is served. The catch moves into the table so every adapter inherits it. A throw reaching a server's request emit would take the process down — the resident model and every live Session with it — and a desktop protocol.handle is no safer. The existing content-routes tests pass UNCHANGED; that is the proof the extraction was faithful. resolve-content.test.ts drives the table with no socket under it, and every claim it makes was checked against a mutation that should break it.

…omething else `decodeSegment` was a homonym twice over in this stack. `decode` is overwhelmingly llama_decode here — the decode set, DecodeError{rc, partial}, decodeErrorOf — and `segment` is the multimodal unit: trace-types compares cells against SegmentSource::cells() and DecodeSegmentsResult::cells. So a reader landing on `decodeSegment(rep[1])` in a routes file could reasonably parse it as decoding a multimodal segment, which is a real type one package over. `decodeURISegment` mirrors the `decodeURIComponent` it wraps — already visible two lines below it — and the qualifier scopes both words at once: a URI segment is unmistakably a path segment, and nobody reads "decode a URI" as inference. No new vocabulary, and the name is legible from this file alone. Module-private; 1 definition and 5 route call sites.

…p skips The note on `resolveContent` said the ingress "on desktop is not an HTTP route at all, but a path over IPC". Measured, and it is: Electron's custom scheme delivers a POST body to `protocol.handle`, so the desktop target serves `POST /v1/media/ingress` on `attachment://` exactly as the HTTP host serves it on a socket. That matters beyond the comment. It is what lets the client half derive the ingress from its origin the same way it derives every read — so the shared view uploads with one call and never learns which target it is running on. The reason the route stays out of the table is unchanged: it is a WRITE, and both its bytes and its limits come from the transport.

`contentError` never consulted the method, so every error reply came back with a JSON body — on a HEAD as much as a GET. Node's HTTP layer strips a HEAD body on the way out, which is exactly why nobody noticed: over a socket the wire was correct while the table was not. An adapter that builds a real response object — a desktop `protocol.handle` returning a `Response` — hands those bytes over. The rule now lives in one place. `resolveContent` became a wrapper owning the three things every route obeys: is this ours, never throw at the caller, and a HEAD carries no body. `route()` underneath answers unconditionally, and `serveBlob` lost its `bodyless` parameter — with the rule applied once above, a second copy of it was a second thing to keep in step. Headers are untouched, `Content-Length` included: it describes the body a GET would have returned, which is what makes a HEAD worth sending. The test that pinned this was mine, and it asserted the wrong thing — it read an error body out of a HEAD and passed. Split: GET asserts the body, HEAD asserts none and keeps the declared length. Removing the strip now reddens two tests. `content-routes.test.ts` is unchanged and still green, which is the evidence the HTTP path never changed behaviour.

feat(rig): the content route table becomes a part — resolveContent

The mirror of the CLI's a0b2406, which has carried these guards since cut 2 while hdk went without them. `--include` is now required and repeatable. The DEPS table says what the arc TOUCHED; it cannot say what moves on any given cut, and treating it as the set is what stamped `3.2.0-alpha.4` for a binding that was not shipping — and pinned three manifests to a version that would never exist. Absence now cuts nothing rather than cutting everything, and an unknown name is refused instead of silently ignored. `unclosed()` refuses a cut whose pins do not close: a member that ships while something still points at the version it is leaving behind is caught before anything is published, not after a consumer cannot install. 16 tests, all green.

rig 5.6.0-alpha.5 carries the content route table as a part (`resolveContent`, shared by the HTTP host and a desktop scheme) and the HEAD-body fix. dev-tools 0.5.0-alpha.5 is pulled in by closure: it pins rig exactly, so the cutter refuses rig alone rather than leave a package pinned to a version that ships. sdk, agents, media and lloyal.node are HELD at alpha.4 — nothing in them moved. The lockfile is workspace-local, so the alpha.5 versions resolve without a registry round-trip.

lloyal-research added 3 commits August 30, 2026 09:15

lloyal-research added the skip-gpu-tests label Aug 31, 2026

lloyal-research changed the title ~~feat(media)!: content addressing gets its own package~~ feat(multimodal)!: one image, three entry points, N agents forked onto it Aug 31, 2026

lloyal-research changed the title ~~feat(multimodal)!: one image, three entry points, N agents forked onto it~~ feat!: multimodal integration and a content-addressed media package Aug 31, 2026

lloyal-research requested a balanced review from Copilot August 31, 2026 22:47

Copilot started reviewing on behalf of lloyal-research August 31, 2026 22:48 View session

Copilot AI reviewed Aug 31, 2026

View reviewed changes

lloyal-research requested a balanced review from Copilot September 1, 2026 03:51

Copilot started reviewing on behalf of lloyal-research September 1, 2026 03:52 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Comment thread packages/sdk/src/deltas.ts

lloyal-research added 13 commits September 1, 2026 19:36

alpha: cut 1

1ca577b

Cut 0's media was burned before the review fixes landed (the publish-loop guard skips existing versions, so pressing the button would have shipped a stale @lloyal-labs/media). One set id for the whole cut, as designed: everything moves to -alpha.1 with exact pins.

lloyal-research requested a balanced review from Copilot September 3, 2026 08:13

Copilot started reviewing on behalf of lloyal-research September 3, 2026 08:13 View session

lloyal-research added 13 commits September 5, 2026 23:13

Merge pull request #119 from lloyal-ai/feat/scheduler

e55321c

agents: the pool becomes a scheduler

lloyal-research requested a balanced review from Copilot September 6, 2026 11:04

Copilot started reviewing on behalf of lloyal-research September 6, 2026 11:05 View session

Copilot AI reviewed Sep 6, 2026

View reviewed changes

Comment thread package.json Outdated

Comment thread packages/media/src/file-store.ts Outdated

Comment thread packages/rig/src/ability-descriptors.ts Outdated

lloyal-research added 14 commits September 6, 2026 21:45

Merge pull request #120 from lloyal-ai/feat/documents

c76f638

feat(media,rig,agents,abilities)!: documents — PDF ingress, a documents ability, assets on the run

Merge pull request #123 from lloyal-ai/feat/desktop-content-plane

0318a4c

feat(rig): the content route table becomes a part — resolveContent

Navigation Menu

Uh oh!

feat!: multimodal integration and a content-addressed media package - #118

feat!: multimodal integration and a content-addressed media package#118
lloyal-research wants to merge 59 commits into
mainfrom
feat/mtmd

lloyal-research commented Aug 31, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

lloyal-research commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

lloyal-research commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Multimodal integration

Content-addressed media package

Breaking

Defects fixed

Decomplection pass

Normalization safety

Verification

Open — not ready to merge

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

lloyal-research commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lloyal-research commented Aug 31, 2026 •

edited

Loading