[MateuszSalski]

mixer.init's audio_router_node names an external IInputsObjects node; the orchestrator emits afv_program on every cut/take completion and afv_crossfade per frame during a crossfade, so that node's own gain policy can follow program/preview changes.

Also: optional per-source audio decode (Source.has_audio ->
MixerApplication.audio_edges), preserving each source's native channel layout (mono, stereo, 5.1, ...) rather than forcing stereo, and a Janus Opus audio output helper, for an application that wants to feed and ship that node's mix.

[jpietek]

@MateuszSalski could you review the two commits I pushed to this branch (4652208, d30558b)?

Problem: audio_mixer mixed only once every input held a full block. One stalled or restarting source silenced every bus, and the other inputs' queues grew without bound.

Change: the node is now clocked by the program's video ticks, not by its inputs.

• Blocks stay a fixed frame_samples (960 = 20 ms, Opus's frame). Each one leaves at the deadline of the video tick holding its last sample (tick time + the compositor's latency_ms), together with that tick's picture. Audio and video rates need no common multiple. wire_audio_mixer now takes fps and latency_ms from the builder.
• Samples are placed by their realtime set_pts media time in a per-source SampleQueue (audio_mixer_core.hpp):
  • a frame within 1 ms of the previous one's end continues it;
  • a late or missing span becomes silence, faded out and back in over 3 ms;
  • nothing is held more than 1 s ahead.
• Output pts are the block's media time instead of a counter from 0. Gain events are kept by sample index.
• node.object.get <mixer> status counts dropped / missing / jumps per source.

Tests:

• tests/test_audio_mixer_native.py: two tones, one stopping after 1 s. The mix keeps going at 20 ms blocks, the survivor stays audible, and a scheduled mute lands on its exact sample.
• Plus the core C++ test and the mixer pytest suite, all green on a build container.
• Not run: a GPU show with video, or Janus/browser audio.

Still open for you:

1. Merge develop. The conflicts are in cut.cpp/fade.cpp/wipe.cpp/application.py, where develop reworked takes (lever, TakeClock). Please port the AFV hooks onto that code, and use TakeClock::progress(pts) / take_progress instead of computing x in fade.cpp and wipe.cpp.
2. Lip sync over WebRTC is unverified. Video RTP skips RTCP SR, so the browser has no SR pair to align audio and video. The build_janus_audio_output docstring calling audio RTCP the lip-sync anchor is stale.
3. raise_initial_audio_gains() re-cuts the initial scene at startup. The new test hit the same race: an event sent before the group creates the node is lost. Initial gains belong in the node or in an orchestrator event at start, not in a second take.
4. demos/mixer_audio duplicates demos/mixer (25 files, many identical). It should be a config or flag on the existing demo.
5. Stepped fades: gains during a fade step once per video frame and hold in between.
6. Leftovers: (void)slot; in scene.cpp, (void)kind; in audio_mixer::setObject, and non-const new_slot/old_slot.
7. Wake-ups: the node wakes for every input frame. Fine for a few sources; worth batching (like the compositor's input_poll_ms) before scaling toward 192.

🤖 Generated with Claude Code

[jpietek]

@MateuszSalski I rebased this branch onto latest develop (b98c070d) and force-pushed it. Your 13 commits are squashed into one commit, still authored by you, followed by my stall fix and one test fix.

What changed while resolving the conflicts:

• Take hooks: transition_progress for fade, dip and wipe now sends develop's take progress (TakeClock::progress / take_progress) instead of the hand-rolled x. Audio now follows the lever, pauses and the fade curve the same way video does.
• Inputs: has_audio goes through develop's _add_packet_input, so it also works with input_routing without a demux node, and alongside the frame_metadata / decoded_ahead / capture_identity options.
• Operator page: your program-audio mute button now sits in the viewer header of develop's redesigned page. I haven't checked it in a browser yet.
• Dockerfile: opus-devel / --enable-libopus sits next to develop's fatbin CUDA filter build.
• Tests: tests/mixer/test_audio_routing.py now builds its fake API from test_graph.fake_api, because develop's graph needs MixerDelivery and other new node classes.

Tests: module build, C++ core test, the native audio test and 925 mixer pytest pass on a build container. Not yet run: a GPU show, Janus/browser audio, lip sync. Items 2–7 of my previous comment are still open.

The previous head was d30558b8.

🤖 Generated with Claude Code

[jpietek]

@MateuszSalski one more commit, 21a46f43. It came out of a GPU run: in the setup flow, program audio never reached a browser.

• The gap: develop's tools/janus_mountpoints.py created every mountpoint video-only ("audio": False). The setup's mixer also never passed --audio-router-node, so a has_audio source's audio edge went unread. Once that edge filled, it would block the demux, and the video with it.
• The fix: the show is now the single source of truth.
  • A has_audio source defaults audio_router_node to the in-tree audio_mixer. The option still overrides it.
  • The first dirty Janus program rendition (MixerConfig.audio_rendition) gets the Opus stream on its mountpoint, both when created and when a video-only mountpoint is replaced.
  • Its settings.program_outputs entry carries "audio": true, and the viewer's mute button shows only there.
• Behavior change to your API: an empty audio_router_node no longer means no audio when sources have audio. To turn audio off, drop has_audio.

Verified on an L4 with this branch's images, using a two-source show (flash + beep per second, 880 / 440 Hz):

• The beep pitch follows each cut A → B → A on the first beep after the cut.
• Audio arrives at Janus about 10 ms after video on the wire.
• No missing samples, discontinuities or output drops over 8 minutes.
• Mountpoint sync works against the real Janus, both creating the program mountpoint fresh and adding audio to a video-only one.
• Tests: 927 mixer pytest pass; the new tests/mixer/check_mute.cjs and the deck/outputs page checks pass.

Lip sync in the browser is still open: video has no RTCP SR.

🤖 Generated with Claude Code