From 3964dfbc203e17b317fc67d0d484791ff385438e Mon Sep 17 00:00:00 2001 From: Corvin Jaedicke Date: Wed, 12 Aug 2026 16:20:54 +0200 Subject: [PATCH 1/2] Add CLAUDE.md with repo layout, commands, and architecture Documents the two mirrored Python/Node packages, the test layer split and its env vars, the cross-file invariants (loaded-model API, lazy Processor format init, shared VAD inference via frame userdata, fail-open), and the logging and parity conventions. Points at DEVELOPMENT.md as the authoritative long-form reference rather than duplicating it. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..d630d09 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,112 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Repository shape + +Two independently built packages that are functional mirrors of each other: + +- `python/` — `ai-coustics-livekit-plugin`, importable as `livekit.plugins.ai_coustics` (namespace + package under `python/src/livekit/plugins/ai_coustics/`), built on `aic-sdk` 3.1. +- `node/` — `@ai-coustics/livekit-plugin` (`node/src/`), built on `@ai-coustics/aic-sdk` 0.23. + +Run all commands from inside `python/` or `node/`; there is no root-level build. The two packages +are released in lockstep and must always carry the same version. + +`DEVELOPMENT.md` is the authoritative long-form document for architecture rationale, the logging +convention, the local end-to-end environment, release steps, and the planned upstream LiveKit +integrations (`download-files`, an audio observer/tap interface, frame-processor and VAD metrics). +Read it before changing public behavior; update it when that rationale changes. + +## Commands + +Python (`uv`): + +```bash +cd python +uv sync --dev +uv run pytest tests/test_processor.py -q # single unit test file +uv run pytest tests/test_vad.py::test_name -q # single test +uv run pytest tests/test_integration.py -q # needs AIC_SDK_LICENSE +uv run pytest tests/test_e2e_room.py -q # needs AIC_SDK_LICENSE + LiveKit server +uv run ruff check . && uv run ruff format --check . +uv run mypy # strict; CI skips it only on 3.10 +``` + +Node (`npm`): + +```bash +cd node +npm ci +npm test # unit tests only (explicit file list in package.json) +npx vitest run test/vad.test.ts -t "name" # single test +npm run test:integration # needs AIC_SDK_LICENSE +npm run test:e2e # needs AIC_SDK_LICENSE + LiveKit server +npm run check # tsc --noEmit +npm run build # tsup -> dist +``` + +`npm test` names its unit test files explicitly — a new unit test file must be added to that script +or CI will not run it. + +Test environment: `AIC_SDK_LICENSE` is required for integration and e2e (those suites skip without +it rather than fail). Models default to `quail-vf-2.2-s-16khz`, `vad-2.1-xxs-16khz`, and +`tyto-1.1-l-16khz`, overridable via `AIC_INTEGRATION_MODEL_ID`, `AIC_INTEGRATION_VAD_MODEL_ID`, +`AIC_INTEGRATION_ANALYSIS_MODEL_ID`, `AIC_INTEGRATION_MODEL_DIR`. E2E needs a LiveKit server — +`livekit-server --dev` (or the pinned `livekit/livekit-server:v1.13.1` container) with the dev +defaults `ws://127.0.0.1:7880` / `devkey` / `secret`. Never point e2e at a production project. + +## Architecture + +Four public objects, each mirrored across both runtimes: + +- **`Processor`** (`processor.py` / `processor.ts`) — a LiveKit `FrameProcessor` doing SDK speech + enhancement. Takes an already-loaded SDK `Model`, never a model ID or path; applications provision + models themselves via `Model.download` + `Model.from_file`/`fromFile`. Native construction happens + in the constructor (synchronous errors surface immediately, with backend auth continuing in the + SDK grace period); **audio format init is lazy** because LiveKit only supplies stream geometry with + the first frame. One LiveKit frame maps to one fixed-size SDK call. Multichannel input is + downmixed and the enhanced mono signal duplicated back across channels to preserve frame geometry. +- **`VAD` / `VADProcessor`** (`vad.py` / `vad.ts`) — inference runs *once*, in the pass-through + `vad.processor` frame processor, which attaches immutable per-block snapshots (including raw mono + PCM) to `frame.userdata`. Every `VADStream` derives LiveKit events from those shared snapshots, so + extra streams cost no native VADs and no extra inference. Closing a stream must not tear down the + shared SDK session — RoomIO closes `vad.processor`. The wrapper overrides SDK duration defaults + with LiveKit-compatible values (50 ms minimum speech, 250 ms speech hold, the latter satisfying + LiveKit's streaming turn detector); explicit `VADParameters` still win. +- **`Analyzer` / `Collector`** (`analyzer.py` / `analyzer.ts`) — the public `collector` is a + transparent `FrameProcessor` that buffers mono float32; the analyzer runs periodic + `analyze_buffered()`/`analyzeBuffered()` inference off the audio path and emits + `analysis_result` / `analysisResult` events plus aggregate OpenTelemetry instruments. Python uses + an asyncio task + `asyncio.to_thread()`; Node uses a timer around the synchronous SDK call. +- **`FrameProcessorChain`** (`frame_processor_chain.py` / `.ts`) — lets these share RoomIO's single + `noise_cancellation` slot. Order matters: `vad.processor` first, then `analyzer.collector`, then + the enhancement `Processor`, so VAD and analysis see original input audio. + +`ProcessorContext` wraps the SDK context purely to add structured logging around parameter and +bearer-token changes. Node's `sdk.ts` hand-declares structural types because aic-sdk 0.23 ships no +TypeScript declarations. + +**Fail-open is a hard invariant.** Any processing error is logged and the *original* frame is +returned; room audio must keep flowing whatever the SDK does. Repeated failures and +slower-than-realtime warnings are rate-limited, with recovery/close records summarizing affected +frames, RTF, and maximum processing backlog. Keep the model's fixed audio delay conceptually +distinct from processing backlog (scheduling/compute debt) in metrics and logs. + +## Conventions + +**Parity.** A change to one runtime almost always needs the mirrored change in the other, including +tests. The file layouts intentionally correspond one-to-one. + +**Logging** (full rules in `DEVELOPMENT.md`). Messages are `Component: event` +(`Processor: initialized`, `VAD: inference falling behind realtime`). Every record carries +`plugin=ai-coustics` and `component=processor|vad|analyzer|collector`. Put changing values in +structured fields, never in the message string. Python: `livekit.plugins.ai_coustics` logger, +`extra=`, snake_case fields, durations in seconds. Node: LiveKit's Pino logger with the object +before the message, camelCase fields, durations suffixed `Ms`, with a console fallback for objects +created before LiveKit initializes its global logger. Do not add a bracketed provider prefix. + +**Releases.** Bump `project.version` in `python/pyproject.toml` (then `uv lock`) and run +`npm version --no-git-tag-version` in `node/`; merge to `main`, then push an unprefixed +semver tag (`git tag 0.1.0`). The release workflow refuses tags that are not on `main` or that +disagree with either package version. From db2197093427387aed518482db97224f2a41707e Mon Sep 17 00:00:00 2001 From: Corvin Jaedicke Date: Wed, 12 Aug 2026 16:20:54 +0200 Subject: [PATCH 2/2] Publish to PyPI with trusted publishing and check uv.lock in CI The publish-python job now authenticates over OIDC instead of a stored PYPI_API_TOKEN: it gains id-token: write and drops the password input, so PyPI verifies the repository, workflow file, and publish environment and no long-lived PyPI credential exists. The project is not yet on PyPI, so this is registered as a pending publisher that creates it on first publish. npm has no pending-publisher equivalent and only accepts a trusted publisher on an existing package, so the first release still uses the NPM_TOKEN bootstrap secret the workflow already expects. DEVELOPMENT.md records that sequence and that the secret is deleted afterwards. CI also gains uv lock --check. It runs before uv sync --dev, which would otherwise re-lock and mask the drift it is meant to catch. npm ci already fails on a stale package-lock.json, so this gives the Python side the same guarantee and keeps both lockfiles trustworthy as dependency-scanner input. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 4 ++++ .github/workflows/release.yml | 6 +++++- DEVELOPMENT.md | 18 +++++++++++++++--- 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index eb8bc11..927a465 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -20,6 +20,10 @@ jobs: - uses: astral-sh/setup-uv@v5 with: python-version: ${{ matrix.python-version }} + # Keep uv.lock in step with pyproject.toml. Dependency scanners read the + # lockfile, so a stale one silently misreports the dependency tree. This + # runs before `uv sync`, which would otherwise re-lock and mask the drift. + - run: uv lock --check - run: uv sync --dev - run: uv run ruff check . - run: uv run ruff format --check . diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d2301f6..f3432d9 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -105,6 +105,9 @@ jobs: needs: [build-python, build-node] runs-on: ubuntu-latest environment: publish + permissions: + contents: read + id-token: write steps: - name: Download Python distribution uses: actions/download-artifact@v8 @@ -112,11 +115,12 @@ jobs: name: python-dist path: dist + # Authenticates through PyPI trusted publishing (OIDC). No API token is + # stored: PyPI verifies this repository, workflow file, and environment. - name: Publish to PyPI uses: pypa/gh-action-pypi-publish@release/v1 with: packages-dir: dist/ - password: ${{ secrets.PYPI_API_TOKEN }} skip-existing: true publish-node: diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 54950d1..beaa761 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -377,7 +377,19 @@ artifacts. The GitHub release is created only after both registry publications s Repository and registry configuration required by the publish jobs: -- A GitHub `publish` environment, optionally with required reviewers. -- A `PYPI_API_TOKEN` secret available to that environment. +- A GitHub `publish` environment, restricted to `*.*.*` tags, optionally with required reviewers. +- A PyPI trusted publisher for `ai-coustics-livekit-plugin`, restricted to this repository, + `release.yml`, and the `publish` environment. The publish job authenticates over OIDC and + stores no PyPI token. - An npm trusted publisher for `@ai-coustics/livekit-plugin`, restricted to this repository, - `.github/workflows/release.yml`, and the `publish` environment. + `.github/workflows/release.yml`, and the `publish` environment. npm only allows a trusted + publisher on a package that already exists, so the first release authenticates with an + `NPM_TOKEN` secret in the `publish` environment. Configure the trusted publisher and delete + that secret once the package is on the registry. + +## Dependency scanning + +Both lockfiles are committed and are the input our dependency scanning reads: `python/uv.lock` +and `node/package-lock.json`. Keep them in step with their manifests. `npm ci` fails on a stale +`package-lock.json`, and CI runs `uv lock --check` for the same reason on the Python side. Do not +add either lockfile to `.gitignore`.