Skip to content

Opt-in hardware video decode for media layers (FOSFORA_HWACCEL) - #301

Merged
kevinraymond merged 3 commits into
kevinraymond:mainfrom
BuddhiLW:feat/hwaccel
Nov 13, 2026
Merged

kevinraymond merged 3 commits into
kevinraymond:mainfrom
BuddhiLW:feat/hwaccel

Conversation

@BuddhiLW

@BuddhiLW BuddhiLW commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Split out of #299, as asked: this PR holds only the hardware video decode.

FOSFORA_HWACCEL=cuda|vaapi|qsv|vulkan|auto adds -hwaccel and a cheaper chroma scaler to the media-layer ffmpeg command. The value must be a plain identifier. Unset or none, nothing changes. On a 16 s 1080p30 H.264 clip, CPU time went from 17.0 s to 6.5 s.

It includes the two lines that add FOSFORA_HWACCEL to the pre-decode path (decode_all_frames in media/video.rs), which were in the REPL media commit in #299. The CHANGELOG entry is the one from #299.

Checked locally: cargo fmt --check, cargo test -p fosfora-app --lib --features video -- hwaccel, cargo clippy -p fosfora-app --all-targets --features video -- -D warnings, scripts/check_changelog.py.

kevinraymond added a commit that referenced this pull request Nov 12, 2026
The attribution section told every agent to commit as Kevin Raymond and to
set git config to that if it differed. The file is tracked, so an agent in
a contributor's clone followed it too: PRs #300 and #301 arrived with two
commits authored as Kevin and all five committed as him. Agents now keep
the identity that is configured and never change it.
@kevinraymond

Copy link
Copy Markdown
Owner

Two of the commits carry my name and email instead of yours, and so does the committer on the others. I think the repo's CLAUDE.md told your agent to set that. I've fixed the file. Please set your own user.name and user.email in your clone, rewrite both branches with git rebase origin/main --exec 'git commit --amend --no-edit --reset-author', and force-push.

If FOSFORA_HWACCEL names a method the machine can't start, ffmpeg exits without decoding a frame. On my machine vaapi and qsv both do this. The streaming path treats that as the end of the clip and starts ffmpeg again right away, so a looping layer stays black and relaunches ffmpeg about 12 times a second with no error shown. ffmpeg only falls back to software when the device works but can't decode the codec, not when the device is missing.

Please check the device once before using it, like:

ffmpeg -v error -init_hw_device <method> -f lavfi -i nullsrc=s=16x16 -frames:v 1 -f null 

This exits 0 when the device works and non-zero when it doesn't. Run that the first time method() is called and cache the result. If it fails, log a warning that names the method and decode in software. auto already falls back by itself, so it can skip the check.

Also, video isn't a default feature, so cargo test -p fosfora-app --lib -- hwaccel compiles none of the new code and runs no tests. Add --features video.

Thanks!

FOSFORA_HWACCEL=<method> (cuda, vaapi, qsv, vulkan, auto) adds -hwaccel
<method> before the clip's input and -sws_flags fast_bilinear to the output.
Unset, empty or `none` leaves the ffmpeg command unchanged. The method must
be a plain identifier, so the variable cannot smuggle extra arguments.

Frames are still downloaded for the RGBA pipe, so decode alone moving to the
GPU saves little: the YUV to RGBA conversion dominates. The clip is never
resized, so fast_bilinear only affects chroma upsampling. Measured on a 16 s
1080p30 H.264 clip (RTX 4070 Laptop, ffmpeg 8.0.1), CPU time:
  stock 17.0 s, -hwaccel cuda 13.2 s, fast_bilinear 8.3 s, both 6.5 s.
scale_cuda cannot convert nv12 to rgb0 in this ffmpeg, so the conversion stays
on the CPU; uploading NV12 and converting in a shader is the larger follow-up.
decode_all_frames (clips up to 60 s) now takes FOSFORA_HWACCEL like the
streaming path, so decode stays off the CPU there as well. Adds the
CHANGELOG entry.
A FOSFORA_HWACCEL method whose device cannot start (vaapi or qsv on a
machine without one) makes ffmpeg exit without decoding a frame. The
streaming path reads that as the end of the clip and restarts ffmpeg at
once, so a looping layer stayed black and relaunched ffmpeg about 12
times a second with nothing logged.

method() now opens the device once per process with
`ffmpeg -init_hw_device <method>` on a generated 16x16 frame and caches
the answer. A method that fails logs a warning naming it and the clip
decodes in software. `auto` falls back by itself and is not checked.
@BuddhiLW

Copy link
Copy Markdown
Contributor Author

Sorry about the author. The clone had a repo-local user.name / user.email override with your identity, set from the old CLAUDE.md. I removed it, re-applied the commits on origin/main with the author reset, and force-pushed. Author and committer are now mine on all of them.

The device check is in the new commit, "Check the hwaccel device once before using it":

  • The first call to method() runs ffmpeg -v error -nostdin -init_hw_device <method> -f lavfi -i nullsrc=s=16x16 -frames:v 1 -f null - and caches the answer for the process.
  • If it fails, a warning names the method and both paths (streaming and pre-decode) run without -hwaccel and without the fast_bilinear scaler, so the command is the stock one.
  • auto is not checked.

On my machine the check exits 0 for cuda and vulkan and 251 for vaapi and qsv, so I see the same failure you do for those two.

You are right about the test command, it compiled none of it. Checked this time with the feature on: cargo test -p fosfora-app --lib --features video -- hwaccel (3 tests, two of them new, for the fallback and for auto skipping the check) and cargo clippy -p fosfora-app --all-targets --features video -- -D warnings. I fixed the command in the description too. The CHANGELOG entry gained one sentence about the fallback.

@BuddhiLW

BuddhiLW commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

To be clear, the wrong author was my mistake. Your assessment is right: my agent followed the old CLAUDE.md, and I did not check the result. I will pay attention to it next time.

@kevinraymond
kevinraymond merged commit 93c3f50 into kevinraymond:main Nov 13, 2026
13 checks passed
BuddhiLW added a commit to BuddhiLW/fosfora that referenced this pull request Nov 14, 2026
hwaccel.rs takes upstream's reviewed version (device probe); media/mod.rs and
video.rs keep integration's extra modules and the -v error flag.
@BuddhiLW
BuddhiLW deleted the feat/hwaccel branch November 14, 2026 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants