Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 19 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,14 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
Two independently built packages that are functional mirrors of each other:

- `python/` — `ai-coustics-livekit-plugin`, importable as `livekit.plugins.ai_coustics` (namespace
package under `python/src/livekit/plugins/ai_coustics/`), built on `aic-sdk` 3.1.
- `node/` — `@ai-coustics/livekit-plugin` (`node/src/`), built on `@ai-coustics/aic-sdk` 0.23.
package under `python/src/livekit/plugins/ai_coustics/`), built on `aic-sdk` 3.2.
- `node/` — `@ai-coustics/livekit-plugin` (`node/src/`), built on `@ai-coustics/aic-sdk` 0.24.

Run all commands from inside `python/` or `node/`; there is no root-level build. The two packages
are released in lockstep and must always carry the same version.
are released in lockstep and must always carry the same version. Their SDK pins must resolve to the
same ai-coustics native core, which the bindings report through `get_sdk_version()` / `getVersion()`
rather than through their own package version: `aic-sdk` 3.2 for Python and 0.24 for Node both wrap
core 0.24.

`DEVELOPMENT.md` is the authoritative long-form document for architecture rationale, the logging
convention, the local end-to-end environment, release steps, and the planned upstream LiveKit
Expand Down Expand Up @@ -76,16 +79,25 @@ Four public objects, each mirrored across both runtimes:
LiveKit's streaming turn detector); explicit `VADParameters` still win.
- **`Analyzer` / `Collector`** (`analyzer.py` / `analyzer.ts`) — the public `collector` is a
transparent `FrameProcessor` that buffers mono float32; the analyzer runs periodic
`analyze_buffered()`/`analyzeBuffered()` inference off the audio path and emits
`analyze_buffered()`/`analyzeAsync()` inference off the audio path and emits
`analysis_result` / `analysisResult` events plus aggregate OpenTelemetry instruments. Python uses
an asyncio task + `asyncio.to_thread()`; Node uses a timer around the synchronous SDK call.
an asyncio task + `asyncio.to_thread()`; Node uses a timer around the SDK's own `analyzeAsync()`,
which runs on a libuv worker thread. Both skip a tick whose predecessor is still running, and
both defer session teardown until in-flight inference settles — Node's `Analyzer.close()`
therefore returns a promise.
- **`FrameProcessorChain`** (`frame_processor_chain.py` / `.ts`) — lets these share RoomIO's single
`noise_cancellation` slot. Order matters: `vad.processor` first, then `analyzer.collector`, then
the enhancement `Processor`, so VAD and analysis see original input audio.

`ProcessorContext` wraps the SDK context purely to add structured logging around parameter and
bearer-token changes. Node's `sdk.ts` hand-declares structural types because aic-sdk 0.23 ships no
TypeScript declarations.
bearer-token changes. Node's `sdk.ts` is a thin re-export boundary over the declarations aic-sdk
0.24 ships; it only hand-mirrors `ProcessorParameter` and `VadParameter`, which are declared as
`const enum`s that TypeScript treats as compile-time-only, so the plugin owns the runtime objects
it re-exports rather than leaning on declarations a bundler may erase.

In Node, `close()` on all three components terminates the SDK session and then calls the SDK's
`dispose()`; disposal is idempotent and any later call on a disposed instance throws, so close
clears native references before disposing. Python relies on binding finalization instead.

**Fail-open is a hard invariant.** Any processing error is logged and the *original* frame is
returned; room audio must keep flowing whatever the SDK does. Repeated failures and
Expand Down
35 changes: 23 additions & 12 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ process a throwaway frame to probe the license.

Processor format initialization is lazy because LiveKit supplies the complete stream geometry
with the first frame. Each LiveKit frame is processed in one fixed-size SDK call, avoiding the
additional latency of the SDK's variable-block-size mode. aic-sdk 3.1 for Python and 0.23 for Node
additional latency of the SDK's variable-block-size mode. aic-sdk 3.2 for Python and 0.24 for Node
process mono audio only, so multichannel LiveKit frames are downmixed before processing and the
enhanced signal is duplicated across the original channel count. This preserves the LiveKit frame
geometry and metadata.
Expand Down Expand Up @@ -48,21 +48,31 @@ by LiveKit's streaming turn detector. Because the SDK hold uses a rolling-majori
wrapper also keeps an active LiveKit speech segment open until that much continuous raw silence
has accumulated. Explicit `VADParameters` values still take precedence.

Each `Analyzer` owns one SDK collector/analyzer pair. Its public `collector` is a transparent
Each `Analyzer` owns one SDK analysis instance: a collector/analyzer pair in Python, a single
`Analyzer` carrying both halves in Node since aic-sdk 0.24. Its public `collector` is a transparent
`FrameProcessor` installed in RoomIO's `noise_cancellation` slot: it lazily initializes from the
first frame, downmixes PCM16 input to mono float32, buffers it, and returns the original frame
unchanged. Stream boundaries reset the analyzer. Closing either the analyzer or its collector
stops scheduling and terminates the SDK telemetry session.

In Node, every component's `close()` follows its `terminateSession()` with the SDK's `dispose()`,
releasing the native instance at a known point instead of leaving it to garbage collection.
Disposal is idempotent, and any call on a disposed instance throws, so each `close()` clears its
native references first and every `process()` guards on them. Python has no equivalent call and
relies on the binding's own finalization.

`FrameProcessorChain` forwards stream-info lifecycle hooks and applies any number of enabled
processors in constructor order. It lets a `Processor`, VAD processor, and Collector share
RoomIO's single `noise_cancellation` slot. Placing the Collector before the enhancement Processor
is recommended: analyzing original input audio helps explain how its quality affects the rest of
the pipeline.

Python schedules inference with an asyncio task and runs each blocking `analyze_buffered()` call
through `asyncio.to_thread()`. Shutdown waits for an active inference before terminating the SDK
session. Node uses a timer around the SDK's synchronous `analyzeBuffered()` API. Both runtimes emit
through `asyncio.to_thread()`. Node uses a timer around `analyzeAsync()`, which the SDK runs on a
libuv worker thread; a tick that arrives while an analysis is still running is skipped rather than
queued, and the skip is reported through a rate-limited warning. In both runtimes shutdown waits
for an active inference before terminating the SDK session, because termination and disposal wait
for the analyzer lock. Both runtimes emit
a plugin-level result event after every successful scheduled call without logging the result by
default. They also record aggregate score, inference-duration, and success/error count instruments
through the process-wide OpenTelemetry metrics API; operational errors remain logged and fail-open
Expand Down Expand Up @@ -117,10 +127,11 @@ such as package metadata or an explicit package list. Until those pieces exist,

### First-class streaming Analyzer integration

The aic-sdk streaming analysis API is split into a `Collector` and an `Analyzer`. The collector
accepts mono float32 audio synchronously and is safe to feed from the audio path, while
`analyze_buffered()` / `analyzeBuffered()` runs an expensive model inference and must execute away
from that path. The result contains risk, speaker reverb, speaker loudness, interfering speech,
The aic-sdk streaming analysis API separates buffering from inference: Python splits it across a
`Collector` and an `Analyzer`, Node carries both on one `Analyzer`. Buffering accepts mono float32
audio synchronously, does not take the analyzer lock, and is safe to feed from the audio path,
while `analyze_buffered()` / `analyzeAsync()` runs an expensive model inference and must execute
away from that path. The result contains risk, speaker reverb, speaker loudness, interfering speech,
noise, codec-degradation, and packet-loss scores. `FileAnalyzer` is intended for complete in-memory
signals and is not appropriate for a live agent stream.

Expand Down Expand Up @@ -203,10 +214,10 @@ so `window_duration` must remain optional until aic-sdk provides it. If future a
different context windows, that API will also avoid hard-coding the current five-second window.

Python runs `analyze_buffered()` through `asyncio.to_thread()` because the binding releases the
GIL during inference. Node aic-sdk 0.23 exposes only synchronous `analyzeBuffered()` and
`terminateSession()`, so calling them from a timer would still block the agent's JavaScript event
loop. A production Node integration first needs native asynchronous APIs such as
`analyzeBufferedAsync()` and `terminateSessionAsync()` that execute on a worker pool.
GIL during inference. Node uses the SDK's own `analyzeAsync()`, added in aic-sdk 0.24, which runs
on a libuv worker thread. `terminateSession()` and `dispose()` remain synchronous and wait for the
analyzer lock, so the Node `Analyzer.close()` returns a promise and releases the native instance
only after any in-flight analysis has settled.

### First-class Processor metrics

Expand Down
10 changes: 7 additions & 3 deletions node/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,9 @@ Download models during deployment or container setup:
```ts
import { Model } from "@ai-coustics/livekit-plugin";

const enhancementPath = Model.download("quail-vf-2.2-l-16khz", "./models");
const vadPath = Model.download("vad-2.1-xxs-16khz", "./models");
const analysisPath = Model.download("tyto-1.1-l-16khz", "./models");
const enhancementPath = await Model.download("quail-vf-2.2-l-16khz", "./models");
const vadPath = await Model.download("vad-2.1-xxs-16khz", "./models");
const analysisPath = await Model.download("tyto-1.1-l-16khz", "./models");
```

Enhancement and VAD models are different model types. Make the returned paths available to your
Expand Down Expand Up @@ -170,6 +170,10 @@ rest of the pipeline.
This still uses LiveKit's `noiseCancellation` slot as a temporary integration. RoomIO owns the
chain and closes the processor, collector, and analyzer together.

Analysis inference runs on a worker thread, so it never blocks the agent's event loop. If you own
an `Analyzer` outside RoomIO, `analyzer.close()` returns a promise that resolves once any in-flight
analysis has settled and the SDK session is released; awaiting it is optional.

## Configuration

Set the enhancement level through the Processor context, and configure all SDK VAD parameters on
Expand Down
98 changes: 55 additions & 43 deletions node/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions node/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
"prepack": "npm run build"
},
"dependencies": {
"@ai-coustics/aic-sdk": "^0.23.0",
"@ai-coustics/aic-sdk": "^0.24.0",
"@livekit/typed-emitter": "^3.0.0",
"@opentelemetry/api": "^1.9.0"
},
Expand All @@ -48,9 +48,9 @@
"@livekit/rtc-node": ">=0.13.24 <1"
},
"devDependencies": {
"@types/node": "^22.0.0",
"@livekit/agents": "^1.0.43",
"@livekit/rtc-node": "^0.13.24",
"@types/node": "^22.0.0",
"livekit-server-sdk": "^2.14.1",
"tsup": "^8.5.0",
"typescript": "^5.9.3",
Expand Down
Loading
Loading