Skip to content

fix(macos-helper): set audio probe frameLength before PCM copy - #3821

Merged
thymikee merged 2 commits into
callstack:mainfrom
jipeng6036-del:fix/3771-audio-probe-framelength
Oct 20, 2026
Merged

thymikee merged 2 commits into
callstack:mainfrom
jipeng6036-del:fix/3771-audio-probe-framelength

Conversation

@jipeng6036-del

@jipeng6036-del jipeng6036-del commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

audio-probe reported -90 dBFS because AudioProbeStatusWriter.add(sampleBuffer:) copied PCM while AVAudioPCMBuffer.frameLength was still 0, leaving destination byte sizes at 0 and returning -12731.

Set frameLength before copying. Make the internal writer accessible to its regression through @testable. Two files; no public API change. Fixes #3771.

Validation

  • Base: 256aa126598e8cc67a0ba5daf8388b3c60ead711. Unchanged head: b8e317697a8d6b5cb27a4b41dcc7c28a4d72c383.
  • macOS validation record: commands, JSON, and test output. macOS 14.6.1, Xcode 16.2, Swift 6.0.3: swift test --package-path apple/macos-helper passed 23 XCTest tests, including the PCM regression; swift build --package-path apple/macos-helper passed.
  • Live ScreenCaptureKit probes at that head ran for 3000 ms with Screen Recording granted. Glass.aiff playback produced heard:true, maximum peak -20 dBFS and RMS -35 dBFS. The silence control produced heard:false with every bucket at -90 dBFS. The linked record documents the temporary CLT/VFS toolchain workaround used for the live helper.
  • A passed exact-head pnpm check:affected --run --head receipt is still unavailable. Linux, Apple, and Android CI runs remain action_required, with zero jobs executed. The neutral cubic review is not a test pass.

Maintainer CI approval and review remain pending. The additional stereo/exact-peak test suggestion was explicitly optional and is outside this update.

View guided diff

AVAudioPCMBuffer starts with frameLength 0, so mDataByteSize stays 0
and CMSampleBufferCopyPCMDataIntoAudioBufferList returns -12731. Set
frameLength first, then copy. Add a regression that feeds a real float
PCM sample buffer through add(sampleBuffer:) and asserts a non-silent
peak.

Fixes callstack#3771
@jipeng6036-del
jipeng6036-del marked this pull request as draft October 19, 2026 13:52
@thymikee

Copy link
Copy Markdown
Member

This change is not ready to merge yet. At b8e3176 the fix looks right, but nothing shows it works on a real audio capture. The change affects the ScreenCaptureKit audio path in AudioProbe.swift. The PR body says no Swift tests and no live audio probe ran. Without a live run, a silent drop back to -90 dBFS would pass every CI lane.

Please run swift test --package-path apple/macos-helper on a macOS host (13 or newer, Screen Recording granted) and post the AudioProbeTests pass. Then build the helper from this head and run agent-device-macos-helper audio-probe for about 3 s while system audio plays, for example afplay /System/Library/Sounds/Glass.aiff in a loop. The output must show heard:true and at least one peakDbfs/rmsDbfs bucket above -90. Run it once more in silence and show heard:false. Post the commands and the JSON output in the PR.

Not blocking, and you can take or leave it: AudioProbeTests.swift builds a mono buffer, but ScreenCaptureKit sends two-channel non-interleaved float32, and the checks are loose enough that a copy filling one channel would still pass. Would a stereo case with a different level per channel and an exact peak check (for example -6 dBFS) make the test stronger?

CI shows one check pending and none failing, but I could not confirm it is the macos-helper gate, so a green result may not mean the new test ran. There are no conflicts. Once the macOS test and live run are posted, mark the draft ready.

@jipeng6036-del

Copy link
Copy Markdown
Contributor Author

Validated unchanged commit b8e317697a8d6b5cb27a4b41dcc7c28a4d72c383 on macOS 14.6.1.

  • Xcode 16.2 (16C5032a), Apple Swift 6.0.3.
  • swift test --package-path apple/macos-helper: 23 XCTest tests passed, 0 failures, including AudioProbeTests.testAddSampleBufferRecordsANonSilentPeakFromRealPCM.
  • swift build --package-path apple/macos-helper: passed.
  • Live ScreenCaptureKit probes ran for 3000 ms on the same unchanged commit, with Screen Recording granted. Sound was generated using afplay -v 0.5 /System/Library/Sounds/Glass.aiff in a loop, then stopped before the user-confirmed silent run.
  • The live helper was built using CLT 6.0.3 with a temporary toolchain library copy and VFS overlay to exclude stale local installation files. The subsequent unit tests and build used the full Xcode toolchain with DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer.

Portable sound reproduction command (the recorded run invoked the built executable directly):

"$(swift build --package-path apple/macos-helper --show-bin-path)/agent-device-macos-helper" audio-probe --duration-ms 3000 --out /tmp/sound.json
cat /tmp/sound.json

Output:

{"rmsDbfs":[-90,-44,-90,-35,-64],"source":"system-audio","durationMs":3000,"active":false,"peakDbfs":[-90,-24,-78,-20,-57],"stoppedAt":"2026-10-20T15:12:24Z","startedAt":"2026-10-20T15:12:21Z","bucketMs":1000,"elapsedMs":3000,"reason":"completed","sourceCount":1,"state":"stopped","backend":"macos-screencapturekit","heard":true,"sampleCount":5,"audio":"probe"}

Portable silence reproduction command (the recorded run invoked the built executable directly):

"$(swift build --package-path apple/macos-helper --show-bin-path)/agent-device-macos-helper" audio-probe --duration-ms 3000 --out /tmp/silent.json
cat /tmp/silent.json

Output:

{"audio":"probe","state":"stopped","heard":false,"reason":"completed","backend":"macos-screencapturekit","source":"system-audio","bucketMs":1000,"sampleCount":5,"rmsDbfs":[-90,-90,-90,-90,-90],"elapsedMs":3000,"durationMs":3000,"stoppedAt":"2026-10-20T15:13:10Z","sourceCount":1,"active":false,"startedAt":"2026-10-20T15:13:07Z","peakDbfs":[-90,-90,-90,-90,-90]}

Last 20 lines of the completed unit test run:

	 Executed 6 tests, with 0 failures (0 unexpected) in 0.002 (0.002) seconds
Test Suite 'RectResponseTests' started at 2026-10-20 23:54:20.361.
Test Case '-[AgentDeviceMacOSHelperTests.RectResponseTests testFiniteRectKeepsItsValues]' started.
Test Case '-[AgentDeviceMacOSHelperTests.RectResponseTests testFiniteRectKeepsItsValues]' passed (0.000 seconds).
Test Case '-[AgentDeviceMacOSHelperTests.RectResponseTests testNonFiniteComponentYieldsNoRect]' started.
Test Case '-[AgentDeviceMacOSHelperTests.RectResponseTests testNonFiniteComponentYieldsNoRect]' passed (0.000 seconds).
Test Suite 'RectResponseTests' passed at 2026-10-20 23:54:20.362.
	 Executed 2 tests, with 0 failures (0 unexpected) in 0.001 (0.001) seconds
Test Suite 'ScreenshotResponseTests' started at 2026-10-20 23:54:20.362.
Test Case '-[AgentDeviceMacOSHelperTests.ScreenshotResponseTests testResponseNeverCarriesAFullscreenField]' started.
Test Case '-[AgentDeviceMacOSHelperTests.ScreenshotResponseTests testResponseNeverCarriesAFullscreenField]' passed (0.001 seconds).
Test Suite 'ScreenshotResponseTests' passed at 2026-10-20 23:54:20.362.
	 Executed 1 test, with 0 failures (0 unexpected) in 0.001 (0.001) seconds
Test Suite 'agent-device-macos-helperPackageTests.xctest' passed at 2026-10-20 23:54:20.362.
	 Executed 23 tests, with 0 failures (0 unexpected) in 0.035 (0.037) seconds
Test Suite 'All tests' passed at 2026-10-20 23:54:20.362.
	 Executed 23 tests, with 0 failures (0 unexpected) in 0.035 (0.038) seconds
◇ Test run started.
↳ Testing Library Version: 102 (x86_64-apple-macos13.0)
✔ Test run with 0 tests passed after 0.001 seconds.

The worktree remained clean. GitHub CI remains action_required; these are local validation results.

@jipeng6036-del
jipeng6036-del marked this pull request as ready for review October 20, 2026 16:12
@thymikee

Copy link
Copy Markdown
Member

Thanks, this covers what I asked for at b8e3176. The macOS run shows AudioProbeTests passing, and the live probe reports heard:true with peaks up to -20 dBFS while sound plays and heard:false at -90 dBFS in silence. The code verdict is unchanged: the fix looks right.

GitHub CI has not run on this head yet because the fork workflows wait for approval, so a maintainer needs to approve them. There are no conflicts. I am marking this ready for human review.

Not blocking: the stereo test case with an exact per-channel peak check is still a possible follow-up.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Oct 20, 2026
* origin/main: (28 commits)
  feat(gesture): add gesture tap with 2 to 5 simultaneous contacts (callstack#3847)
  chore: mark 0.21.29 as released
  chore(diagnostics): trace Apple prepare phases and subprocess lifecycle (callstack#3846)
  feat(sdk): start an embedded runtime from the main entry (callstack#3843)
  fix(alert): align local Apple and Limrun button choices (callstack#3825)
  docs(adr): drop the withdrawn host allocator design from ADR 0021 (callstack#3845)
  feat(screenshot): support selector crops on CoreDevice iPhones (callstack#3839)
  feat(scroll): support explicit budgets for long target searches (callstack#3837)
  fix(apple): key the toolchain fingerprint memo on the resolved xcode-select dir (callstack#3726)
  feat(android): hand rotation back to the sensor with orientation auto (callstack#3822)
  fix(sdk): scope released entries and deferred provider calls (callstack#3833)
  test(android): prepare the snapshot helper before landmark waits (callstack#3834)
  perf(ci): overlap iOS simulator boot with the native build (callstack#3836)
  docs(ios): record SpringBoard sessions as verified on a physical iPhone (callstack#3840)
  feat(apple): admit two-finger gestures on physical iOS devices (callstack#3838)
  perf(android): bound target readiness and confirm moved clears (callstack#3815)
  perf(android): reuse live helper for input and capture (callstack#3811)
  chore(gates): forbid process globals outside declared entry points (callstack#3753) (callstack#3823)
  refactor(commands): resolve command inputs and output from the caller host (callstack#3753) (callstack#3829)
  refactor(runtime): use the owning host environment at process boundaries (callstack#3753) (callstack#3828)
  ...
@thymikee
thymikee merged commit f217bd6 into callstack:main Oct 20, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

macOS helper audio-probe always reports -90 dBFS: AVAudioPCMBuffer frameLength set after the copy

2 participants