[EtienneLescot]

Summary

The first frame of a Linux capture encoded with VAAPI can come out with its lower half blurred and blocky. The editor opens a take on frame 0, so that is the picture the user sees first. The cause is the encoder's rate control on the first keyframe, not a partly filled PipeWire buffer. This PR caps the VAAPI QP at 30 (qmax). It is a one-line change in open_backend, small enough to cherry-pick onto release/v2.1.0.

What was observed

Found during the manual e2e pass of the installed v2.1.0-rc.1 .deb on Ubuntu 24.04, GNOME Shell 46 (Wayland), AMD Radeon 610M (VAAPI encode over the dmabuf path), Mesa 25.2.8.

• Window take recording-1792077324409.mp4, 866×628, encoder {"video":"vaapi","rejected":[]}:
  • Frame 0 is a 54 444-byte I-frame. Its top is sharp, and from about 70 % of the height down the terminal text cannot be read.
  • Frames 1 to 4 are P-frames of 13 015, 276, 117 and 96 bytes. Frame 1 already restores the picture.
• ffmpeg -debug qp on frame 0 gives the per-macroblock QP:
  • QP is 13 for the top rows, then rises row after row to 51.
  • The bottom third is entirely at QP 51.
  • The same shape as one sharp picture, not a mix of two pictures.
• Later keyframes in the same file are fine: 94 875 bytes at QP 20 to 25, every 0.5 s.
• The 1920×1080 screen take of the same session has the same QP ramp at a smaller scale: QP 13 to 31 on frame 0.

Root cause

AMD's VCN rate control, reached through Mesa's VA-API driver, gives the first IDR of a session a budget of about 8 times the average frame:

• Each frame gets bitrate / fps.
• For this window that is 3.26 Mbit/s / 60 ≈ 6.8 KB, so frame 0 gets about 54 KB.

When the picture needs more than that budget, the encoder raises the QP of each macroblock row from the top down until the frame fits.

This was reproduced off-line. The input was a sharp still taken from the same file, encoded with h264_vaapi using the helper's exact parameters: 60 fps, -b:v 3263088, -g 30, -bf 0.

Variant	Frame 0	Frame 1	Frame-0 QP
Helper parameters, vendored ffmpeg 8.1	54 258 B	12 050 B	13 → 51
VBV ×2, ×4, full initial occupancy, maxrate ×2/×4/×8	unchanged (54 258 / 52 075 B)		13 → 51
Bitrate ×2 / ×4	105 / 143 KB		moves (budget follows bitrate/fps)
qmax=30	69 993 B	11 111 B	13 → 30

Only the bitrate moved the budget; VBV size, initial occupancy and maxrate did not.

ffmpeg's h264_vaapi defaults qmax to -1. vaapi_encode_init_rate_control then sends max_qp = 0 to the driver, which means a ceiling of 51 (qmax > 0 ? qmax : 0, checked in the 8.0.1 source).

Fix

• Backend::max_qp() returns Some(30) for VAAPI. open_backend sets it as codec_ctx->qmax before avcodec_open2.
• Both VAAPI routes go through open_backend: the zero-copy dmabuf open_importing and the ladder's CPU-upload path. Both get the cap.
• Vulkan and libopenh264 keep their defaults. This was only measured on VAAPI.

Measured cost:

Same still, 60 frames:

• Frame-0 PSNR against the source goes from 28.6 dB to 41.2 dB.
• The total of the 60 frames goes from 177 276 B to 173 634 B. The following P-frames have less to repair.
• Frame 59 is unchanged (47.7 dB).

1080p segment of the real screen take:

• At the default bitrate, the output is effectively unchanged: 7.30 Mbit/s both times and PSNR within 0.02 dB, because QP almost never reaches 30.
• Starved at 3 Mbit/s, the output goes from 4.17 to 5.58 Mbit/s and from 35.2 to 43.1 dB.

Where the cap binds, it trades bitrate for quality. That is the right trade for a file the editor decodes and re-encodes.

In the two real takes, about 3 % of macroblocks were above QP 30, in bursts of change.

Related issue

Refs the v2.1.0-rc.1 Linux manual e2e pass (no issue filed).

Type of change

• Bug fix
• Feature
• Enhancement
• Documentation
• Refactor / maintenance
• Performance
• Security

Release impact

• Patch
• Minor
• Major / breaking change
• No release note needed

Desktop impact

• Windows
• macOS
• Linux
• Installer / packaging
• Not platform-specific

Testing

Unit test only_vaapi_caps_the_qp_and_the_cap_reaches_the_codec:

• Checks that only VAAPI is capped.
• Checks that libopenh264 keeps qmax = -1.
• Where a VA-API encoder opens, checks that the cap is still in the context after avcodec_open2.
• CI has no GPU and skips that last part, as does a machine with libva < 2.22, which the vaMapBuffer2 guard refuses.

Commands run:

• cargo test in electron/native/pipewire-capture: 100 passed, 1 ignored.
• With LD_LIBRARY_PATH pointing at a libva 2.23, the VAAPI branch of the new test ran and passed, and OPENSCREEN_LINUX_ENCODE_TEST=1 ladder_encodes_a_playable_mp4 selected vaapi.

Through the helper's own VideoEncoder (scratch test, not committed):

• Vendored ffmpeg 8.1, libva 2.23, Radeon 610M.
• Input: the sharp window still, 866×628 at 3 263 088 bit/s, 60 fps.
• Before the fix: frame 0 is 54 338 B with QP 13 → 51.
• After the fix: frame 0 is 70 460 B with QP 13 → 30.

Manual verification on GNOME Wayland (AMD/VAAPI)

1. Build the helper (npm run build:native:linux) and run a dev build, or the CI artifact.
2. Open a terminal with a full screen of text, about 870×630.
3. Record a window (the portal's Window tab), pick the terminal, record about 5 s, then stop.
4. In the editor the first frame (t = 0, where the editor opens) must show readable text all the way down.
5. Check from the CLI:

   f=~/.config/openscreen/recordings/recording-<id>.mp4
   ffprobe -v error -select_streams v -show_entries packet=size,flags -of csv=p=0 "$f" | head -3
   ffmpeg -hide_banner -debug qp -i "$f" -frames:v 1 -f null - 2>&1 | sed -n '/type: I/,/nal_unit/p' | head -60

   Every QP row of frame 0 must be ≤ 30. Before the fix, rows reached 51.
6. Record a full screen and check that the file size is in line with an rc.1 take of similar content.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Improvements
  • VAAPI video encoding now uses a maximum quantization parameter of 30. Vulkan and software encoding behavior remains unchanged.

[EtienneLescot]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: e4b97188-66f7-4e1b-8cb0-4085818d0b7f

📥 Commits

Reviewing files that changed from the base of the PR and between 1b7ed86 and 3deb845.

📒 Files selected for processing: 1

• electron/native/pipewire-capture/src/encoder.rs

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

📝 Walkthrough

VAAPI encoder setup now sets codec_ctx.qmax to 30. Vulkan and software retain their existing QP settings. A new test checks the backend-specific limits and verifies the VAAPI limit after encoder initialization when VAAPI is available.

Changes: Backend QP limits

Layer / File(s)	Summary
Select and apply backend QP limits electron/native/pipewire-capture/src/encoder.rs	Backend::max_qp returns a limit of 30 for VAAPI and no limit for Vulkan or software. open_backend applies the limit to codec_ctx.qmax before opening the encoder. Tests check these settings and the opened VAAPI context when available.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 3deb8

VAAPI receives a requested QP ceiling of 30, while Vulkan and software retain their defaults. No concrete failure is established, so the change appears ready for normal checks.

Pre-merge checks: 5

Passed checks: 5 passed

Check name	Status	Explanation
Title check		The title clearly and concisely identifies the Linux VAAPI QP cap and its purpose of keeping the first keyframe legible.
Description check		The description is complete and follows the repository template. It explains the problem, root cause, fix, platform and release impact, testing, and manual verification steps.
Docstring Coverage		Docstring coverage is 83.33% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 1 files.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Some tools did not complete. Review the errors below.

🔧 Clippy (1.98.1)

Clippy execution failed

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}