Fix NVENC rate control, switch to HEVC Main 10, record the ffmpeg build

README.md

@@ -1,6 +1,8 @@
 # Video_Convert_2
 
-A collection of bash scripts to batch re-encode video files into portable MP4 format, suitable for playback on any Smart TV or media server (works great with Plex/Serviio). The default settings produce files that play on virtually any device without transcoding.
+A collection of bash scripts to batch re-encode video files into portable MP4 format, suitable for playback on any Smart TV or media server (works great with Plex/Serviio).
+
+The default output is **HEVC Main 10** (`hvc1`-tagged) with stereo AAC. That direct-plays on current TVs, phones and desktop clients, but it is a stronger requirement than the H.264 these scripts used to emit — if you serve older or unusual clients, verify direct play before converting a library, since a client that cannot decode it makes the *server* transcode. AV1 is deliberately not used for this reason.
 
 Make sure to compare the input video with the output using `mediainfo`. You may be surprised by the results — the output is often better quality than the input.
 
@@ -9,16 +11,17 @@ Make sure to compare the input video with the output using `mediainfo`. You may
 | Script | Purpose |
 |--------|---------|
 | `vc2.sh` | Main batch video converter (auto-detects NVIDIA GPU) |
-| `vc1.sh` | Single-file converter (experimental) |
+| `vc1.sh` | Single-file converter, tuned for compression or quality |
 | `cc_probe.sh` | Helper: probe video file and output stream info |
 | `cc_norm.sh` | Helper: analyze audio and output loudnorm parameters |
 | `vc_rename.sh` | Utility: normalize video filenames |
+| `tools/build-ffmpeg.sh` | Rebuild the `/usr/local` ffmpeg these scripts depend on |
 
 ---
 
 ## vc2.sh — Batch Video Converter
 
-Searches a configured directory for video files, probes each one, normalizes audio, and re-encodes to MP4. Automatically uses NVIDIA hardware acceleration (`h264_nvenc`) when an NVIDIA GPU is detected; falls back to `libx264` software encoding otherwise.
+Searches a configured directory for video files, probes each one, normalizes audio, and re-encodes to MP4. Automatically uses NVIDIA hardware acceleration (`hevc_nvenc`) when an NVIDIA GPU is detected; falls back to `libx265` software encoding otherwise. The software path needs a **multilib x265** — against an 8-bit-only build the 10-bit output silently degrades.
 
 ### Synopsis
 
@@ -35,7 +38,7 @@ Searches a configured directory for video files, probes each one, normalizes aud
                       fast preset, with resize.
 
     --no-gpu          Disable hardware acceleration even if an NVIDIA GPU
-                      is detected; forces libx264 software encoding.
+                      is detected; forces libx265 software encoding.
 
     -m, --movie       Feature-length movies  (searches: renamed/features/)
     --mv              Music videos           (searches: renamed/mtv/)
@@ -55,7 +58,11 @@ Searches a configured directory for video files, probes each one, normalizes aud
 | video | 30 | 0.20 | 192k | slow | no |
 | restricted | 25 | 0.08 | 92k | fast | yes |
 
-Video bitrate is calculated dynamically from the quality factor: `bitrate = QF × width × height × FPS / 1000`.  
+The Preset column applies to the **CPU** encoder only. NVENC uses its own `p1`–`p7` scale (`nvenc_preset`, default `p6`) — x264's preset names mean something quite different there, and passing them selects NVENC's legacy table.
+
+Quality Factor is a **rate ceiling**, not a target: it sizes `-maxrate`/`-bufsize`, while `nvenc_cq` / `cpu_crf` decide the actual quality and the encoder spends less on easy content.
+
+The rate ceiling is calculated dynamically from the quality factor: `ceiling = QF × width × height × FPS / 1000`, with `-maxrate` at 1.5x and `-bufsize` at 2x that.  
 Resize: videos wider than 1280px are scaled down to 1280px; narrower than 720px are scaled up to 720px.
 
 ### Examples
@@ -73,13 +80,34 @@ Resize: videos wider than 1280px are scaled down to 1280px; narrower than 720px
 
 ## vc1.sh — Single-File Converter
 
-Re-encodes a single video file. Auto-detects NVIDIA GPU and uses hardware acceleration when available.
+Re-encodes one video file for either optimal compression or optimal quality. Defaults to HEVC on NVENC; `-hwaccel cuda` lets ffmpeg pick NVDEC per stream and fall back to software for codecs the GPU cannot decode.
+
+    vc1.sh [options] <inFile>
 
-    vc1.sh <inFile>
+| Option | Meaning |
+|--------|---------|
+| `-m, --mode compression\|quality` | Quality intent (default: `compression`) |
+| `-c, --cpu` | Encode with libx265 instead of hevc_nvenc |
+| `--av1` | Encode AV1 with SVT-AV1 (implies `--cpu`) |
+| `-s, --original-size` | Do not cap width at 1280px |
+| `--max-width N` | Change the width cap |
+| `--fps N` | Frame rate cap (default `24000/1001`); `none` to disable |
+| `-q, --cq N` | Override the quality level |
+| `--mp4` | Write MP4 instead of MKV |
+| `--verify` | Score the result against the source with VMAF |
+| `-n, --dry-run` | Print the ffmpeg command and exit |
+| `-o, --output FILE` | Output path |
+
+Output defaults to `<basename>_recode.mkv`. Quality ladder:
 
-Output is written to `<basename>_recode.mp4` in the current directory.
+| Mode | hevc_nvenc | libx265 | libsvtav1 |
+|------|:----------:|:-------:|:---------:|
+| compression | cq 29 | crf 26 (slow) | crf 34 (preset 6) |
+| quality | cq 24 | crf 20 (slow) | crf 28 (preset 5) |
 
-Supported input codecs for hardware decoding: h264, hevc, av1, vp8, vp9, mpeg1, mpeg2, mpeg4, vc1, mjpeg. Falls back to software decode + NVENC encode for other codecs, or full software encode when no GPU is available.
+The frame rate is *capped*, never forced, so 23.976fps and slower sources pass through untouched. Nothing is ever upscaled. Subtitles, chapters and metadata are carried through.
+
+Measured on 30s of 720p, SSIM against a common reference: compression mode 1419 kb/s @ 0.98937, quality mode 2739 kb/s @ 0.99362. `--av1` gives the best quality-per-byte (1254 kb/s @ 0.99119) and is roughly 3x faster than `--cpu`, but is not widely direct-played — see the note at the top of this file.
 
 ---
 
@@ -199,3 +227,18 @@ Ensure `/usr/local/bin` is in your `$PATH`.
 \- Cheers,
 
 Rick
+
+---
+
+## tools/build-ffmpeg.sh — ffmpeg Builder
+
+Rebuilds the hand-compiled ffmpeg in `/usr/local` that these scripts call directly. Records the exact configure line, so the encoder features `vc1.sh` and `vc2.sh` rely on (SVT-AV1, libvmaf, `scale_cuda`, multilib x265) are reproducible rather than folklore.
+
+    tools/build-ffmpeg.sh
+
+Honours `BUILD_ROOT`, `PREFIX` and `X265_TAG` from the environment. Ends with a dozen self-checks that assert each feature by actually encoding with it.
+
+Two findings baked into it, both of which cost real time to diagnose:
+
+- **`--enable-cuda-nvcc` cannot be used** on a current Fedora. CUDA 12.9 refuses gcc 15 outright, and against gcc-14 it collides with glibc 2.42's C23 `sinpi`/`cospi`/`tanpi`. `--enable-cuda-llvm` builds the same filters via clang's NVPTX backend and needs no CUDA toolkit at all. `--enable-libnpp` is also gone — upstream removed it, so `scale_npp` no longer exists.
+- **x265 must be built from a real, reachable git tag.** Its CMakeLists guards the shared-library install with `# shared library is not installed if a tag is not found`, so a tagless *or shallow* checkout installs only `libx265.a` and silently leaves the previous `.so` in place. The symptom is an ffmpeg that still reports 8-bit-only x265 after an apparently successful multilib build, and an `HEVC encoder version unknown` banner.

cc_norm.sh

@@ -9,7 +9,10 @@ sample=${3:-'-ss 01:00 -t 06:00'}
 target_i="-24.0"
 target_tp="-2.0"
 target_lra="11.0"
-outFile="/tmp/sample.json"
+# Unique per invocation: a fixed path here is clobbered when several encodes
+# run concurrently, silently applying one file's loudnorm values to another.
+outFile=$(mktemp -t cc_norm.XXXXXXXXXX.json) || exit 1
+trap 'rm -f "$outFile"' EXIT
 
 ff_string="${ffmpeg_bin} -hide_banner -y"
 ff_string+=" ${sample}"
@@ -56,5 +59,4 @@ loudnorm_string+="measured_thresh=${measured_thresh}:"
 loudnorm_string+="offset=${measured_offset}"
 
 echo "$loudnorm_string"
-rm "$outFile"
 exit $STATUS

cc_probe.sh

@@ -99,8 +99,9 @@ if [[ -n $mainAStream ]]; then
   fi
 fi
 
-vQF=$(mediainfo "$inFile" | grep 'Bits' | awk '{ print $3 }')
-fSize=$(mediainfo "$inFile" | grep 'File size' | awk -F': ' '{ print $2 }')
+## Only take the first (main) video track; cover art/thumbnail tracks add extra lines.
+vQF=$(mediainfo "$inFile" | awk -F': ' '/Bits\/\(Pixel\*Frame\)/ { print $2; exit }')
+fSize=$(mediainfo "$inFile" | awk -F': ' '/^File size/ { print $2; exit }')
 ## Remove leading spaces
 #aMap=$(sed 's/^ *//' <<< "$aMap")
 aMap=${aMap# }
@@ -120,7 +121,7 @@ vFPS=$vFPS
 vLanguage="$vLanguage"
 vMap="$vMap"
 mainVideo=$mainVideo
-vQF=$vQF
+vQF="$vQF"
 ## AUDIO (Main)
 aBitrate=$aBitrate
 aChannels=$aChannels

tools/build-ffmpeg.sh

@@ -0,0 +1,199 @@
+#!/bin/bash
+#
+# Reproducible rebuild of the /usr/local ffmpeg used by Video_Convert_2.
+#
+# Adds over the previous build:
+#   libsvtav1   AV1 encoding           (vc1.sh --av1)
+#   libvmaf     quality metric         (vc1.sh --verify)
+#   cuda-llvm   scale_cuda et al       (vc1.sh assumed these existed; they did not)
+#   libzimg     colorspace-aware scaling (zscale)
+#   libsoxr     high quality audio resampling
+#   libplacebo  HDR -> SDR tonemapping
+#   x265        rebuilt MULTILIB (8/10/12-bit); the old local build was 8-bit only
+#
+# Two things this deliberately does NOT do, both verified on this host:
+#
+#   1. --enable-cuda-nvcc is not used. CUDA 12.9 cannot compile here:
+#        vs gcc 15.3.1  -> "unsupported GNU version! gcc versions later than 14
+#                          are not supported!"
+#        vs gcc-14      -> "mathcalls.h: exception specification is incompatible
+#                          with that of previous function 'cospi'"
+#      glibc 2.42 declares the C23 sinpi/cospi/tanpi that CUDA also declares, and
+#      cuda-nvcc-12-9 is the newest available. --enable-cuda-llvm builds the same
+#      filters with clang's NVPTX backend and needs no CUDA toolkit; configure
+#      accepts either (scale_cuda_filter_deps_any="cuda_nvcc cuda_llvm").
+#      NOTE: --enable-libnpp is NOT used either, and no CUDA toolkit is needed at
+#      all. Upstream ffmpeg has REMOVED libnpp ("WARNING: libnpp has been removed
+#      and enabling it does nothing"), so scale_npp no longer exists; scale_cuda
+#      via cuda-llvm covers it. This build therefore does not link anything out
+#      of /usr/local/cuda-*, so installing or removing a CUDA toolkit cannot
+#      break it.
+#
+#   2. The system (RPMFusion) x265 is not used. It shares soname libx265.so.215
+#      with the local build, so mixing them risks loading the wrong library at
+#      runtime. x265 is rebuilt multilib in place instead.
+#
+# Local x264 and fdk-aac builds in /usr/local are kept as-is (full fdk-aac is
+# preferable to Fedora's stripped fdk-aac-free). dav1d comes from the system.
+
+set -euo pipefail
+
+BUILD_ROOT="${BUILD_ROOT:-$HOME/source/ffmpeg_build}"
+PREFIX="${PREFIX:-/usr/local}"
+JOBS=$(nproc)
+
+# x265 release to build. Must be a real tag: see the x265 section below for why
+# a tagless or shallow checkout silently produces an 8-bit-only install.
+X265_TAG="${X265_TAG:-4.2}"
+
+say () { printf '\n\033[1;36m==> %s\033[0m\n' "$*"; }
+die () { printf '\033[1;31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
+
+[[ -d $BUILD_ROOT/ffmpeg ]] || die "no ffmpeg source tree at $BUILD_ROOT/ffmpeg"
+[[ -d $BUILD_ROOT/x265   ]] || die "no x265 source tree at $BUILD_ROOT/x265"
+
+# ---------------------------------------------------------------- dependencies
+say "Installing build dependencies"
+sudo dnf install -y \
+  clang \
+  libdav1d-devel svt-av1-devel libaom-devel \
+  libvmaf-devel zimg-devel soxr-devel \
+  libplacebo-devel vulkan-loader-devel \
+  numactl-devel cmake nasm yasm \
+  libass-devel libvpx-devel opus-devel lame-devel xvidcore-devel \
+  openssl-devel freetype-devel fontconfig-devel harfbuzz-devel fribidi-devel
+
+# clang must be able to target the GPU, or --enable-cuda-llvm silently disables
+clang -print-targets | grep -q nvptx64 \
+  || die "this clang has no nvptx64 target; --enable-cuda-llvm will not work"
+
+# ----------------------------------------------------------- x265 (multilib)
+say "Rebuilding x265 with 8/10/12-bit support"
+cd "$BUILD_ROOT/x265"
+
+# x265's CMakeLists guards the shared-library install with
+#   if(X265_LATEST_TAG OR NOT GIT_FOUND)   # shared library is not installed if a tag is not found
+# so a checkout that `git describe` cannot resolve installs ONLY libx265.a, the
+# headers and the CLI -- silently leaving the previous .so in place. That is how
+# a perfectly good multilib build still left ffmpeg linked against an 8-bit
+# libx265, and why x265 printed "HEVC encoder version unknown" on every run.
+#
+# Fetching tags is not sufficient on its own: this tree was a SHALLOW clone, so
+# the tag commits were not connected ancestors and `git describe` still failed
+# with "No tags can describe ...". Unshallow first, then build from a real
+# release tag so X265_LATEST_TAG resolves.
+[[ $(git rev-parse --is-shallow-repository) == 'false' ]] \
+  || git fetch --unshallow || die "could not unshallow the x265 checkout"
+git fetch --tags --force origin || die "could not fetch x265 tags"
+git checkout "$X265_TAG" || die "no such x265 tag: $X265_TAG"
+git describe --abbrev=0 --tags >/dev/null 2>&1 \
+  || die "git describe cannot resolve a tag at $X265_TAG; the shared library would not install"
+
+cd "$BUILD_ROOT/x265/build/linux"
+# The 2025 build left artifacts directly in build/linux/; clear them too or a
+# stale CMakeCache here can shadow the new out-of-tree builds.
+rm -rf 8bit 10bit 12bit CMakeCache.txt CMakeFiles libx265.a libx265.so libx265.so.*
+mkdir -p 8bit 10bit 12bit
+
+( cd 12bit && cmake ../../../source \
+    -DCMAKE_INSTALL_PREFIX="$PREFIX" -DHIGH_BIT_DEPTH=ON -DMAIN12=ON \
+    -DEXPORT_C_API=OFF -DENABLE_SHARED=OFF -DENABLE_CLI=OFF >/dev/null
+  make -j"$JOBS" )
+
+( cd 10bit && cmake ../../../source \
+    -DCMAKE_INSTALL_PREFIX="$PREFIX" -DHIGH_BIT_DEPTH=ON \
+    -DEXPORT_C_API=OFF -DENABLE_SHARED=OFF -DENABLE_CLI=OFF >/dev/null
+  make -j"$JOBS" )
+
+cd 8bit
+ln -sf ../10bit/libx265.a libx265_main10.a
+ln -sf ../12bit/libx265.a libx265_main12.a
+cmake ../../../source \
+  -DCMAKE_INSTALL_PREFIX="$PREFIX" \
+  -DEXTRA_LIB="x265_main10.a;x265_main12.a" -DEXTRA_LINK_FLAGS=-L. \
+  -DLINKED_10BIT=ON -DLINKED_12BIT=ON \
+  -DENABLE_SHARED=ON -DENABLE_CLI=ON >/dev/null
+make -j"$JOBS"
+sudo make install
+sudo ldconfig
+
+# Fail loudly rather than letting ffmpeg quietly link the old 8-bit library.
+grep -q 'libx265\.so' install_manifest.txt \
+  || die "x265 install did not include the shared library -- check that tags were fetched"
+
+# The 8-bit-only build reported "8bit"; multilib reports "8bit+10bit+12bit".
+x265ver=$("$PREFIX/bin/x265" --version 2>&1 | grep -o '\[64 bit\].*' || true)
+say "x265 now reports: $x265ver"
+[[ $x265ver == *10bit* ]] || die "installed x265 is still not multilib: $x265ver"
+
+# --------------------------------------------------------------------- ffmpeg
+say "Updating ffmpeg source"
+cd "$BUILD_ROOT/ffmpeg"
+git fetch --all --tags
+git checkout master
+git pull --ff-only
+say "ffmpeg now at: $(git log --oneline -1)"
+
+say "Configuring ffmpeg"
+make distclean >/dev/null 2>&1 || true
+
+PKG_CONFIG_PATH="$PREFIX/lib/pkgconfig:${PKG_CONFIG_PATH:-}" \
+./configure \
+  --prefix="$PREFIX" \
+  --enable-shared \
+  --enable-gpl --enable-nonfree --enable-version3 \
+  --enable-libx264 --enable-libx265 --enable-libsvtav1 --enable-libaom \
+  --enable-libvpx --enable-libxvid --enable-libdav1d \
+  --enable-libfdk-aac --enable-libopus --enable-libmp3lame --enable-libsoxr \
+  --enable-libass --enable-libfreetype --enable-libfontconfig \
+  --enable-libharfbuzz --enable-libfribidi \
+  --enable-libzimg --enable-libvmaf \
+  --enable-nvenc --enable-nvdec --enable-cuda-llvm \
+  --enable-libplacebo --enable-vulkan \
+  --enable-openssl
+
+say "Building with $JOBS jobs"
+make -j"$JOBS"
+sudo make install
+sudo ldconfig
+
+# --------------------------------------------------------------- verification
+say "Verifying"
+fail=0
+# NOTE: never pipe a long listing straight into `grep -q` here. grep exits on the
+# first match, ffmpeg takes SIGPIPE, and under `set -o pipefail` the whole check
+# reports failure -- which made a perfectly good build look like 8 failures.
+# Helpers below capture first, then match.
+ff_filters=$("$PREFIX/bin/ffmpeg" -hide_banner -filters 2>/dev/null)
+ff_encoders=$("$PREFIX/bin/ffmpeg" -hide_banner -encoders 2>/dev/null)
+has_filter ()  { grep -qw -- "$1" <<< "$ff_filters"; }
+has_encoder () { grep -qw -- "$1" <<< "$ff_encoders"; }
+
+check () {
+  if eval "$2" >/dev/null 2>&1; then
+    printf '  \033[1;32mok\033[0m   %s\n' "$1"
+  else
+    printf '  \033[1;31mFAIL\033[0m %s\n' "$1"; fail=1
+  fi
+}
+
+check "scale_cuda present"      "has_filter scale_cuda"
+check "libvmaf filter present"  "has_filter libvmaf"
+check "zscale filter"           "has_filter zscale"
+check "libplacebo filter"       "has_filter libplacebo"
+check "libsvtav1 encoder"       "has_encoder libsvtav1"
+check "hevc_nvenc encoder"      "has_encoder hevc_nvenc"
+# Asserted by a real encode, not by the pix_fmt listing: the listing is built at
+# runtime from whatever libx265.so is actually loaded, so this is what proves the
+# multilib library got installed rather than the old 8-bit one.
+check "x265 10-bit encodes"     "$PREFIX/bin/ffmpeg -hide_banner -f lavfi -i testsrc2=s=320x180:r=24 -t 1 -c:v libx265 -pix_fmt yuv420p10le -f null -"
+check "x265 12-bit encodes"     "$PREFIX/bin/ffmpeg -hide_banner -f lavfi -i testsrc2=s=320x180:r=24 -t 1 -c:v libx265 -pix_fmt yuv420p12le -f null -"
+check "SVT-AV1 encodes"         "$PREFIX/bin/ffmpeg -hide_banner -f lavfi -i testsrc2=s=640x360:r=24 -t 1 -c:v libsvtav1 -f null -"
+check "hevc_nvenc encodes"      "$PREFIX/bin/ffmpeg -hide_banner -f lavfi -i testsrc2=s=640x360:r=24 -t 1 -c:v hevc_nvenc -f null -"
+check "scale_cuda runs"         "$PREFIX/bin/ffmpeg -hide_banner -f lavfi -i testsrc2=s=640x360:r=24 -t 1 -vf hwupload_cuda,scale_cuda=w=320:h=180 -c:v hevc_nvenc -f null -"
+
+if (( fail )); then
+  die "one or more checks failed"
+fi
+
+say "Done. ffmpeg $("$PREFIX/bin/ffmpeg" -hide_banner -version | head -1)"