Skip to content

[FEATURE] Audio: spatial graph, bus sends and ducking, reverb zones, sample banks, streamed layered music, audio as an asset kind #1687

Description

@Noisemaker111

Problem: audio is a flat gain chain with no spatialization and procedural-only music. Verified on main: packages/shell/src/audio/audioEngine.ts:38 setListenerPose(position) takes position only; no PannerNode in packages/shell/src/audio/; packages/shell/src/audio/musicDirector.ts crossfadeTo fades all other layers to zero; packages/assets/src/manifest.ts AssetSourceKind has no audio kind.

Tasks, one PR each, in order. Audio has no screenshot; each PR includes a headless Web Audio graph test using a mocked AudioContext (see how audioEngine.test.ts mocks it, or add a small mock in packages/shell/src/audio/testAudioContext.ts excluded from the build).

  1. Listener orientation and panners. Change setListenerPose to setListenerPose({ position, forward, up }) (keep accepting a bare Vec3 for one release, mapping to forward [0,0,-1]); set AudioListener.positionX/forwardX/upX params. Add spatial?: { panning: "hrtf" | "equalpower"; refDistance?: number; maxDistance?: number; rolloff?: number; coneInner?: number; coneOuter?: number; coneOuterGain?: number } to SoundDef in packages/core/src/audio/audioFalloff.ts; when set, route the source through a PannerNode instead of the scalar falloff gain. The shell's camera rig feeds forward/up each frame (packages/shell/src/audio/audioWire.ts). Test: a sound to the left of a listener facing -Z gets a panner with the expected position. CHANGELOG Migrate (signature) and Added.

  2. Buses, sends, ducking. In packages/core/src/audio/audioFalloff.ts extend AudioBusDef with sends?: Record<string, number>, insert?: ("compressor" | "lowpass" | "reverb")[], duckedBy?: { bus: string; amount: number; attackMs: number; releaseMs: number }. New pure packages/core/src/audio/busGraph.ts: resolveBusGains(buses, activity: Record<busId, boolean>, previous, dtMs) -> Record<busId, number> implementing ducking envelopes; tests. Shell realizes sends as extra GainNode connections and inserts as DynamicsCompressorNode / BiquadFilterNode / ConvolverNode, and applies resolveBusGains with setTargetAtTime.

  3. Zones and occlusion. New packages/core/src/audio/audioZones.ts: AudioZone { id; bounds: { min: Vec3; max: Vec3 }; reverb?: { irId: string; wet: number }; lowpassHz?: number }, resolveListenerZone(zones, position), occlusionFactor(from, to, raycast: (a, b) => boolean) -> 0 | 1; tests. Add an editor volume kind audioZone in packages/core/src/scene/builtinSceneKinds.ts and a runtime consumer that feeds the shell's zone state; the shell sets the master lowpass and the reverb wet from it. Impulse responses: generate a synthetic exponential-decay IR in code (no asset needed).

  4. Sample banks and voices. Add SoundBankDef { id; sounds: Record<string, SoundDef>; preload: "eager" | "lazy" }, SoundDef.variants?: string[], pitchJitter?: number, maxVoices?: number, priority?: number. New pure packages/core/src/audio/voiceAllocator.ts: createVoiceAllocator({ maxTotal }) with request(soundId, priority) -> { ok: boolean; steal?: voiceId }, release(voiceId), snapshot/restore; tests. Shell preloads eager banks and uses the allocator before creating sources.

  5. Streamed layered music. Add MusicTrackDef { id; url?: string; streaming?: boolean; layers?: { id; url; gain }[]; stingers?: { id; url; quantizeBeats?: number; bpm?: number }[] } to the audio config; playMusic accepts a track id as well as a theme id; streamed layers use createMediaElementSource on <audio> elements started in sync, layer gains driven by MusicStateConfig tiers, stingers as buffers scheduled on the next beat. Test: layer gains follow the state config. Use a tiny synthesized OGG committed under a test fixture (generate with a script, do not download).

  6. Audio as an asset kind. Add "audio" to AssetSourceKind in packages/assets/src/manifest.ts, accept .ogg/.wav/.mp3 in the pull CLI (packages/assets/src/cli/pull.ts) with duration metadata, and index a small CC0 set from freesound.org (record licenses in CREDITS.md). bun run gen.

  7. Editor Audio workspace. Flip supported: false in packages/editor/src/shell/WorkspaceRail.tsx; packages/editor/src/AudioPanel.tsx lists banks, zones, and a preview button per sound. Shots of the panel. Closes #1687.

Rules: wait for bun run agent:bootstrap --check; branch claude/<slug> off origin/main; claim comment first; one task per PR with Refs #1687; bun run gen after export changes (JSDoc on every export; create* factories need snapshot/restore); CHANGELOG bullet; check-types and test on core, shell, assets, editor as touched plus check-stateful-ratchet, check-doc-symbols, check-orphan-ratchet; never add Kenney audio; merge origin/main before pushing; squash auto-merge; fix CI on the same branch.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions