Skip to content

Add experimental NVENC-backed VA-API encode support - #438

Open
ntoskrnl7 wants to merge 11 commits into
elFarto:masterfrom
ntoskrnl7:nvenc
Open

Add experimental NVENC-backed VA-API encode support#438
ntoskrnl7 wants to merge 11 commits into
elFarto:masterfrom
ntoskrnl7:nvenc

Conversation

@ntoskrnl7

Copy link
Copy Markdown

Summary

This PR adds experimental VA-API encode support backed by NVENC while keeping the existing decode-focused behavior intact.

The new encode path exposes VAEntrypointEncSlice only after runtime NVENC capability probing, so the advertised profiles should match the target GPU, driver, and NVENC API support.

What changed

  • Added experimental NVENC-backed encode support for H.264, HEVC/H.265, and AV1.
  • Added runtime capability probing for codec/profile support before advertising encode entrypoints.
  • Added CBR, VBR, and CQP rate-control mapping from VA-API parameters to NVENC.
  • Added support for capability-dependent profiles such as H.264 High10, HEVC Main10, HEVC Main422_10, HEVC Main444/Main444_10, and AV1 10-bit input where supported.
  • Improved direct backend external surface import paths for encode workloads, including CUDA/DMABUF import and fallback copy paths.
  • Added runtime smoke tests for advertised VA-API encode profiles using vainfo and ffmpeg.
  • Updated documentation for encode support, limitations, testing commands, and Chrome usage through an external hotpatch runtime.

Limitations

This is intentionally experimental and aimed at transcoding-style workloads.

  • NVENC public APIs do not expose HEVC 12-bit encode, so HEVC Main12/Main444_12 remain decode-only.
  • VP8, VP9, MPEG-2, VC-1, MPEG-4, JPEG, and MJPEG encode are not implemented.
  • Some client paths may renegotiate pixel formats before frames reach the driver, especially HEVC rext/high-bit-depth cases.
  • Chrome still requires an external runtime such as ntoskrnl7/chrome-vaapi-hotpatch; this PR does not make stock Chrome use the driver by itself.

Testing

Tested with:

  • meson test
  • vainfo --display drm --device /dev/dri/renderD128
  • ffmpeg VA-API encode/decode checks for H.264 and HEVC
  • Runtime matrix through driver-runtime-vaapi-smoke

Reference validation documented in the README:

  • Functional matrix: 51/51 pass
  • Stress suite: 20/20 pass
  • Tested across H.264, HEVC Main10, HEVC rext444, CBR/VBR/CQP, long-run, 4K, and parallel encode cases

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant