Fix NVENC rate control, switch to HEVC Main 10, record the ffmpeg build - #6
Merged
Merged
Conversation
vc2.sh encoded with h264_nvenc but still decoded in software, which cost ~730% CPU on a 1080p h264 source while NVDEC sat idle. Benchmarked on a 10-minute segment (S51E11, 2 runs/arm, best-of): software decode 24.70s wall 1214% CPU 587 fps -hwaccel cuda 22.52s wall 571% CPU 651 fps Wall time only improves ~9% because the pipeline re-binds on the CPU filter chain and loudnorm, but CPU per job drops 53%. Output is byte-identical between the two (md5 e2574057...), as h264 decode is bit-exact by spec. Gated on both h264_nvenc and cuda being available, so --no-gpu and non-NVIDIA hosts produce the same command as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two paths were shared by every invocation: vc2.sh tempOut="$tempDir/converting.mp4" cc_norm.sh outFile="/tmp/sample.json" Both are harmless while encodes run one at a time, but each fails silently rather than loudly once they overlap. Concurrent jobs would write the same converting.mp4, and the following `mv -f "$tempOut" "$outFile"` could move one job's partial file to another job's destination -- wrong content, exit status 0. Likewise two cc_norm runs would clobber each other's measurements and apply one file's loudnorm values to another. vc2.sh now suffixes the temp name with the shell pid and loop index; cc_norm.sh uses mktemp with a trap so early exits stop leaking the scratch file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With decode on NVDEC a single encode uses ~570% CPU, so one job leaves most of a 16-core machine idle while its single-threaded loudnorm filter is the actual ceiling. Running two encodes at once puts each loudnorm on its own core and roughly doubles throughput; the GPU has room either way (one NVENC session draws 44W of a 215W budget). Only the ffmpeg execution is parallelised. probeIt/normalizeIt/setOpts/ getMeta stay serial -- they are cheap, and keeping them serial means cc_norm is never running twice at once. Backgrounding encodeIt is safe against the loop's use of $l and friends because the subshell copies the variable state at fork. Concurrent ffmpeg -stats output would interleave unreadably on the terminal, so with -j > 1 each job's progress goes to its own log under $logDir (removed on success, kept on failure) and the terminal gets start/done lines instead. Per-file completion markers moved into encodeIt so they stay attached to their job, and the size report now names the file it describes. -j 1 restores the previous serial behavior including live progress. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Backgrounding the encodes sent ffmpeg's -stats output to a per-job log, which dropped the fps/speed/elapsed/Lsize line from the terminal. That line is the main throughput feedback during a run, so losing it made -j 2 a downgrade in practice even though it encoded faster. Two jobs cannot share a terminal for live progress, but the final stats line can be recovered: the job log is now scanned for the last frame= record (translating the \r separators -stats writes) and reported in the completion block alongside the source duration. The completion block is also assembled into a single string and emitted with one write, so two jobs finishing at the same moment cannot interleave their report lines. The block now names its file, which matters when completions arrive out of order. Serial mode is unaffected: with no job log there is no stats line to recover, and live progress still prints as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A/B on 6x20min SNL segments, warm cache, two reps: j=1 158.7s 2.27 files/min j=2 157.7s 2.28 files/min j=3 158.0s 2.28 files/min Throughput is flat within 1%. The gate was verified to be working -- three ffmpeg processes ran for the duration of the j=3 arm -- so this is not a measurement artifact. Nothing is saturated: CPU sits 70-75% idle, NVENC and NVDEC report ~57% and ~50% regardless of job count, and the disk supplies 237 MB/s cold against ~28 MB/s demanded. Setting -multipass to fullres, qres or disabled changes neither throughput nor output size, which rules out encoder capacity as the limit. What remains is the GPU itself: each encode holds a separate CUDA context and the GPU time-slices between them instead of running them together. Three jobs drew only 398% CPU against one job's 334% -- they were blocked, not working. This also explains why concurrency did appear to help (~1.25x) before NVDEC decoding landed: back then decode ran on the CPU and there was CPU work to overlap. Moving decode to the GPU removed exactly the work that parallelised, so the two changes are antagonistic. Concurrency remains available via -j for --no-gpu runs, where the CPU does the decoding. Defaulting to 1 also restores live per-file progress. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The GPU path could not run at all. It built -vf "...,scale_cuda=1280:-2",
but this ffmpeg had no scale_cuda (it was configured without cuda-nvcc or
libnpp), so every source wider than 1280 or narrower than 720 died with
"No such filter: 'scale_cuda'" and the script exited 1. Only sources
already 720-1280 wide reached the encoder. The syntax was wrong anyway --
scale_cuda needs w=/h= and rejects -2 -- and the CUDA block dead-coded the
software-scale block above it.
Where it did run it inflated files. On 30s of 720p, -preset fast -cq 18
produced 31.6 MB from a 19.6 MB source (8429 kb/s, SSIM 0.99366): "fast"
selects NVENC's legacy preset table (= hp 1 pass), and cq 18 is far below
anything useful on this scale.
Measured, same clip, SSIM against a common reference:
old -preset fast -cq 18 8429 kb/s SSIM 0.99366
new --mode quality 2739 kb/s SSIM 0.99362 -67% at parity
new default (compression) 1419 kb/s SSIM 0.98937 -83%
Quality mode matches the old output's quality at a third the size.
Encoder ladder, from measurements on this hardware:
hevc_nvenc libx265 libsvtav1
compression cq 29 crf 26 (slow) crf 34 (preset 6)
quality cq 24 crf 20 (slow) crf 28 (preset 5)
NVENC gets -rc vbr -b:v 0 so -cq governs rather than being inherited from
a preset, plus lookahead, spatial/temporal AQ, b_ref_mode and multipass.
-highbitdepth is free: 1419 kb/s at SSIM 0.98937 against 8-bit's 1444 at
0.98823, same speed. SVT-AV1 beats x265 on both axes (1254 kb/s @ 0.99119
in 9.7s vs 1247 @ 0.98954 in 29.2s), so --cpu is an HEVC-compatibility
option, not the quality one.
scale_cuda is deliberately not used even though it now exists: a full-GPU
pipeline measured identical to CPU scaling at both 1080p (9.8s vs 9.8s)
and 4K (3.9s vs 3.9s). The encoder is the bottleneck, not the scaling.
Also fixed: the cuvid case statement is replaced by plain -hwaccel cuda so
ffmpeg falls back to software for codecs the GPU cannot decode instead of
hard-failing; upscaling is gone; fps is capped rather than forced, so 25fps
PAL and 30fps sources stop juddering; pixel format is clamped so Main 10
sources do not abort the encoder; audio downmixes to stereo and passes
through when already AAC/Opus; subtitles (built by cc_probe but previously
never referenced), chapters and metadata are carried; and the command is
built as an array instead of relying on word-splitting.
--verify scores with VMAF. It applies the encode's own scale/fps chain to
the reference first -- comparing a 23.976fps output frame-for-frame against
a 30fps source drifts within the first second and scored 25.6 instead of
94.3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Aia7CvksykLQ3j7Vfj9Uoa
Four problems in the GPU path, all confirmed by measurement.
x264 preset names were being handed to NVENC. "fast"/"medium"/"slow" are
not NVENC's scale; they select its legacy preset table, where "fast" means
"hp 1 pass" -- the worst quality-per-bit setting it has. The series profile
used vPreset='fast', so every GPU encode ran it. On that profile, pinned at
the maxrate ceiling so both runs spend the same bits:
-preset fast 2596 kb/s SSIM 0.98580
-preset p6 2476 kb/s SSIM 0.98948
p5/p6/p7 are indistinguishable for h264_nvenc on this GPU (p7 came out
byte-identical to p6); the whole win is escaping the legacy table. vPreset
still drives the CPU path, where the names are correct.
-b:v was dead. NVENC discards an average bitrate once -cq is set and
honours only -maxrate: -cq 23 -b:v 4000k and -cq 23 -b:v 0 produce
byte-identical output. The target_QF math therefore only ever sized the
ceiling, so it is now labelled as one and -b:v 0 is explicit.
The CPU path was pure ABR with no CRF at all, so --no-gpu produced
different-character output from the GPU path. It now uses -crf with the
same ceiling. Measured 1670 kb/s @ SSIM 0.99005 against the old ABR path's
1805 @ 0.99055.
Switched to HEVC Main 10. Verified from the Plex server's own log that
hevc_nvenc output direct-plays on this setup's PC, phone and Samsung TV
clients; only AV1 was transcoded, and in software (-codec:0 libx264) on a
host with no GPU. Against the H.264 output this script produced before:
GPU 2476 kb/s @ 0.98948 -> hevc_nvenc cq26 2168 @ 0.99110
CPU 1673 kb/s @ 0.99006 -> libx265 crf23 1472 @ 0.99029
Both smaller and better. Real result on a Ted Lasso episode: 2142 -> 1039
kb/s on the video stream, Bits/(Pixel*Frame) 0.109 -> 0.053, with
subtitles, SDH, chapters and audio intact. 10-bit is smaller AND higher
quality than 8-bit on both encoders, so the old yuv420p clamp -- which
existed only because h264_nvenc cannot encode 10-bit -- is now 10-bit.
NVENC's CQ and x265's CRF are different scales, so the shared quality
number is split into nvenc_cq and cpu_crf. A single value worked while both
encoders were H.264 and does not survive the switch.
Two hazards this would otherwise have hit: -tune film is x264 vocabulary
and libx265 rejects it outright ("Error setting preset/tune (null)/film"),
aborting the encode, so -tune is now conditional on libx264; and ffmpeg
tags HEVC in MP4 as hev1 by default, which Apple platforms will not decode,
so -tag:v hvc1 is set for HEVC. The tag is a container label, not a stream
change -- a hev1->hvc1 remux is byte-identical.
The --no-gpu path now requires a multilib x265; against an 8-bit-only build
the 10-bit request silently falls back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Aia7CvksykLQ3j7Vfj9Uoa
The /usr/local ffmpeg these scripts call directly was built by hand with no
recorded recipe. vc1.sh --av1 and --verify, and vc2.sh's 10-bit output, all
depend on features that build carries, so losing it means rediscovering the
configure line. Versioning it alongside the scripts that need it.
Two findings are baked into the script because both cost real time:
- --enable-cuda-nvcc cannot be used on a current Fedora. CUDA 12.9 refuses
gcc 15 outright and, against gcc-14, collides with glibc 2.42's C23
sinpi/cospi/tanpi. --enable-cuda-llvm builds the same filters through
clang's NVPTX backend and needs no CUDA toolkit. --enable-libnpp is gone
too -- upstream removed it, so scale_npp no longer exists anywhere.
- x265 must be built from a real, reachable tag. Its CMakeLists guards the
shared-library install with "# shared library is not installed if a tag
is not found", so a tagless or SHALLOW checkout installs only libx265.a
and leaves the previous .so in place. The symptom is an ffmpeg still
linked against 8-bit x265 after an apparently clean multilib build, plus
the "HEVC encoder version unknown" banner. Fetching tags alone is not
enough on a shallow clone: git describe still fails with "No tags can
describe ...", so the script unshallows and checks out X265_TAG.
Dropped the dead CUDA references while moving it: nothing links out of
/usr/local/cuda-* any more, so the "CUDA not found" precondition would have
aborted the build on a host that simply does not need it. BUILD_ROOT, PREFIX
and X265_TAG are now environment-overridable.
Verification asserts each feature by encoding with it rather than grepping a
listing -- the pix_fmt list is built at runtime from whichever libx265.so is
loaded, which is precisely what was wrong before.
README: vc1.sh's section described the pre-rewrite interface (mp4 output, a
cuvid codec list that no longer exists) and is replaced with the real option
table and quality ladder. vc2.sh's docs now say hevc_nvenc/libx265, note that
the Preset column is CPU-only since NVENC uses p1-p7, and describe Quality
Factor as the rate ceiling it actually is. Added the HEVC Main 10 direct-play
caveat up front, since that is a stronger client requirement than the H.264
these scripts used to emit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Aia7CvksykLQ3j7Vfj9Uoa
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes, each measured rather than assumed. All benchmarks are SSIM against a common reference on the same 30s clip unless stated.
1. Rewrite
vc1.shThe GPU path could not run at all. It built
-vf "...,scale_cuda=1280:-2", but this ffmpeg had noscale_cuda, so every source wider than 1280 or narrower than 720 died withNo such filter: 'scale_cuda'and exited 1. Only sources already 720–1280 wide reached the encoder.Where it did run it inflated files:
-preset fast -cq 18produced 31.6 MB from a 19.6 MB source.fastselects NVENC's legacy preset table (hp 1 pass), and cq 18 is far below anything useful.-preset fast -cq 18--mode qualityQuality mode matches the old output's quality at a third the size.
Also fixed: cuvid case statement replaced by plain
-hwaccel cudaso unsupported codecs fall back instead of hard-failing; no more upscaling; fps is capped not forced, so 25fps PAL and 30fps sources stop juddering; pixel format clamped so Main 10 sources don't abort the encoder; audio downmixes and passes through when already AAC/Opus; subtitles (built bycc_probebut never referenced), chapters and metadata carried through.--verifyscores with VMAF, applying the encode's own scale/fps chain to the reference first — comparing a 23.976fps output frame-for-frame against a 30fps source drifts within the first second and scored 25.6 instead of 94.3.2.
vc2.sh→ HEVC Main 10, and three rate-control bugsx264 preset names were going to NVENC.
fast/medium/slowselect NVENC's legacy table. Theseriesprofile usedvPreset='fast', so every GPU encode ran the worst quality-per-bit setting available. At the same bitrate: SSIM 0.98580 → 0.98948.-b:vwas dead. NVENC discards an average bitrate once-cqis set and honours only-maxrate— verified-cq 23 -b:v 4000kand-cq 23 -b:v 0are byte-identical. Thetarget_QFmath only ever sized the ceiling, so it's now labelled as one.The CPU path had no CRF at all — pure ABR, so
--no-gpuproduced different-character output. Now constant-quality with the same ceiling.Codec switch verified from the Plex server's own log:
hevc_nvencoutput direct-plays on PC, phone and Samsung TV clients here. Only AV1 was transcoded, and in software (-codec:0 libx264) on a host with no GPU.Real result on a TV episode: 2142 → 1039 kb/s on the video stream, Bits/(Pixel*Frame) 0.109 → 0.053, subtitles/SDH/chapters/audio intact.
Two hazards this would otherwise have hit:
-tune filmis x264 vocabulary and libx265 rejects it outright, aborting the encode, so-tuneis now conditional on libx264; and ffmpeg tags HEVC in MP4 ashev1, which Apple platforms will not decode, so-tag:v hvc1is set.3.
tools/build-ffmpeg.shThe
/usr/localffmpeg these scripts call directly had no recorded recipe. Two findings are baked in because both cost real time:--enable-cuda-nvcccannot be used on current Fedora. CUDA 12.9 refuses gcc 15, and against gcc-14 collides with glibc 2.42's C23sinpi/cospi/tanpi.--enable-cuda-llvmbuilds the same filters via clang's NVPTX backend.--enable-libnppis gone too — upstream removed it.# shared library is not installed if a tag is not found, so a tagless or shallow checkout installs onlylibx265.aand leaves the previous.soin place — an ffmpeg still linked against 8-bit x265 after an apparently clean multilib build.Notes for review
--no-gpunow needs a multilib x265; against an 8-bit-only build the 10-bit output silently degrades.scale_cudais deliberately unused despite now existing: a full-GPU pipeline measured identical to CPU scaling at 1080p (9.8s vs 9.8s) and 4K (3.9s vs 3.9s). The encoder is the bottleneck.vc1.sh's section described the pre-rewrite interface.🤖 Generated with Claude Code
https://claude.ai/code/session_01Aia7CvksykLQ3j7Vfj9Uoa