Repository navigation
media: apple: avd: hardware video decode in Chromium - #436
hungbenjamin402 wants to merge 8 commits into
Conversation
…trol Chromium's V4L2 stateless decoder sets the decoded (CAPTURE) format and allocates its buffers before it sets any stream control. The first stream control (H.264 or HEVC SPS, VP9 frame, AV1 sequence) then makes get_image_fmt() resolve the image format from ANY, which avd_s_ctrl() treats as a format change on a busy CAPTURE queue and rejects with -EBUSY. v4l2_ctrl_request_setup() fails, the job runs with zeroed parameters, validate_dec_params() rejects them with -EINVAL, and every CAPTURE buffer comes back with V4L2_BUF_FLAG_ERROR. Chromium still displays those buffers, so VP9 and AV1 video is solid green. When the image format is still ANY, the CAPTURE queue is busy and the chosen decoded format suits the resolved image format, recompute the decoded layout for it and keep the buffers if they still fit. The layout was sized for ANY, whose compressed planes are those of 8-bit 4:2:0, so a 10-bit or 4:2:2 stream usually needs more: refuse it with -EBUSY when the allocated buffers are smaller than its layout, so the hardware never writes past them. Tested on a MacBook Pro 16" M3 Max (J516M, T6034): Chromium 153 plays VP9 and AV1 on the AVD and its frames match software decode (VP9 mean |RGB| diff 1.6, was 115.8 with solid green frames). Fixes: d9bc216 ("media: apple: add avd driver") Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
Chromium's V4L2 stateless HEVC decoder queues each picture as Annex B
slice NAL units (3-byte start codes) and sets only the SPS, PPS,
scaling matrix and decode parameters. It never sets
V4L2_CID_STATELESS_HEVC_SLICE_PARAMS or the entry point offsets. The
driver then programs the AVD from stale or zeroed slice parameters,
every job times out ("H0 error") and the video is solid green.
When a picture's request carries no slice parameters and the picture
starts with a start code, parse the slice segment headers from the
bitstream instead (7.3.6.1): the slice parameters, the weighted
prediction table, the entry point offsets and the reference picture
lists (8.3.4), built from the decode parameters' RPS index lists. The
short and long term RPS in the slice header are skipped by the sizes
the decode parameters give, which are counted in RBSP bits as the bit
reader reads them. Offsets and sizes are raw byte counts from each
slice's start code, so the existing walk over consecutive slices is
unchanged. Requests that carry slice parameters (FFmpeg, GStreamer,
libva-v4l2-request), and pictures without start codes from a client
that set its slice parameters once, take the old path.
The bitstream now comes straight from the client, so the parser
refuses what would make the driver index past its tables or the
hardware read past a slice: entry point counts beyond the picture's
CTBs or the table, entry points past the slice data, a slice address
past the picture, a collocated reference past the active list, weight
denominators outside 0..7 (7.4.7.3), and decode parameter reference
counts or DPB indices past the active DPB, which the V4L2 core does not
bound. It parses Main and Main 10 only: Range Extensions slice headers
can carry fields gated on PPS flags that the V4L2 PPS does not have.
Tested on a MacBook Pro 16" M3 Max (J516M, T6034) with Chromium 153:
640x360, 1920x1080, 4 slices per picture with WPP entry points, and
weighted P/B prediction all match software decode on I, P and B
frames (mean |RGB| diff < 0.2). VA-API decode through
libva-v4l2-request and GStreamer's v4l2codecs stay bit-exact.
Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
Build Annex B pictures bit by bit from the slice segment header syntax and check the parser's output against values worked out by hand: a two-slice I picture and a P picture with reference lists, a collocated picture and weights. Check that it refuses what would make the driver index past its tables: a wrapping entry point count, entry points past the slice data, a slice address past the picture, a collocated reference past the list, out-of-range weight denominators, reference counts and DPB indices past the DPB, and Range Extensions formats. Add a .kunitconfig, so that the tests run with KASAN under tools/testing/kunit/kunit.py --arch=arm64. Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
Chromium's V4L2 stateless decoder sets a placeholder H.264 SPS before it has parsed the stream, to learn which decoded formats the driver offers for the bit depth: only the bit depths and chroma_format_idc are filled in, every flag is clear. The driver refused any SPS without frame_mbs_only_flag in try_ctrl, so that VIDIOC_S_EXT_CTRLS failed with -EINVAL and Chromium gave up on the hardware decoder and fell back to software decoding. Interlaced coding is a property of the stream being decoded, not of the control value set before streaming starts. Accept the SPS and refuse to decode a picture whose SPS lacks frame_mbs_only_flag instead; the CAPTURE buffer comes back with V4L2_BUF_FLAG_ERROR, as for other unsupported input. Fixes: d9bc216 ("media: apple: add avd driver") Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
stream_refs() pushes a reference header for every VALID DPB entry and the slice's reference list commands address those headers by DPB index, while the header's reference count is taken from the number of ACTIVE entries. The V4L2 interface lets a client list pictures that are only waiting for output as VALID but not ACTIVE, and Chromium and GStreamer do. The count in the header then falls short of the headers pushed and the VP stalls on some P pictures until the watchdog resets it (seen with Chromium on an x264 stream that uses weighted prediction with duplicated references). Number a VP slot for each entry that is both VALID and ACTIVE, push only those, give the header their count and translate DPB indices to slots in the reference list commands. A reference to an entry without a slot is refused like a reference to an invalid entry. For clients that list only reference pictures, packed from index 0 (FFmpeg and the VA-API v4l2-request driver), the commands are unchanged. Fixes: d9bc216 ("media: apple: add avd driver") Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
…e run stream_slice() derived the address and length of the coded slice data from the copy of the slice it was queued in. Store both in the run when the slice is copied, so that a later change can point the VP at slice data found elsewhere. No functional change. Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
Offer V4L2_STATELESS_H264_START_CODE_ANNEX_B. When it is selected, skip to the slice's NAL unit before looking for the slice data. The NAL unit runs up to the next start code, without the trailing zero bytes in front of a four-byte start code: they are not part of the NAL unit, and the AVD never completes a CAVLC picture whose slice data carries them. Clients may send start codes but leave the start code control at its default (Chromium does), so data that begins with a start code is taken as Annex B as well: a NAL unit never begins with a zero byte. The control default stays without start codes, which FFmpeg and the VA-API driver use, and those streams decode as before. Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
Chromium's V4L2 stateless H.264 decoder only works in frame-based mode: it queues all slices of a picture in one OUTPUT buffer, each behind an Annex B start code, with the SPS, PPS, scaling matrix and decode parameters but without slice parameters or prediction weights. The driver offered only slice-based mode, so Chromium fell back to software decoding. The VP itself is driven per slice: every slice needs its position, QP, deblocking parameters, reference lists, weights and the bit position where its data starts. In frame-based mode, find the slice NAL units in the buffer and parse their headers in the kernel (H.264 7.3.3): build each reference list from the initial list that the v4l2-h264 helpers derive from the DPB, apply ref_pic_list_modification(), read pred_weight_table() with the inferred default factors, and skip dec_ref_pic_marking(). The parsed values fill the same slice parameter and weight structures a slice-based client sets, so the instruction stream is built by the existing code, one job per picture. A slice may make more reference list entries active than the stream has references (8.2.4.2); conforming slice data never uses the extra ones, so they point at the first reference rather than at an unset DPB slot. The VP reads the slices straight from the OUTPUT buffer, which stays queued until the picture completes, so nothing is copied. Redundant slices are dropped; SP/SI slices, field pictures and pictures that do not start at macroblock 0 are refused. Frame-based decoding needs Annex B start codes: without them nothing marks where one slice ends and the next begins. Tested on a MacBook Pro 16" M3 Max (J516M, T6034) with Chromium 153: 640x360, 854x480 Baseline, 1920x1080 and 4 slices per picture match software decode on I, P and B frames. Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
|
Claude Code wrote this up from my tests. I ran this series on a J615 (15-inch M3 Air, YouTubeBig Buck Bunny in 4K60 (
The software run came after an earlier run and started at 83.8 C. The AVD runs started at about 75 C, so read the die row as rough. With hardware decode on, YouTube switched to AV1 by itself. To get the VP9 run I hid AV1 from the page ( CallsWebRTC loopback in the
The kernel log had no AVD errors during any of it. VM_BIND errors in the first YouTube tabChromium logged lines like these six times during the YouTube tests, each followed by All six came in the first two minutes of the first YouTube tab. One was at playback start, four right after the full screen switch and one after another page resize. It never came back in two later tabs that went through the same steps. Steady playback and the WebRTC runs never logged it either. None of the measured 60 s windows dropped a frame. At startup the first tab dropped 10 frames and the later tabs 7 and 2, so I can't tie the drops to it. It looks like a problem with importing the decoded buffers on the GPU side, but my kernel also carries #410 and an earlier #500, so I can't rule those out. Did you see it on the J516M? Not tested
|
Summary
Chromium's built-in V4L2 stateless decoder (
--enable-features=AcceleratedVideoDecoder) now decodes H.264, HEVC, VP9 and AV1 on the AVD. On the current driver every codec fails in Chromium, each for a different reason:-EBUSY, the job runs on zeroed parameters, every buffer returnsV4L2_BUF_FLAG_ERRORH0 errortimeoutsSLICE_PARAMSor entry points; the driver programs stale onestry_ctrlrefuses Chromium's placeholder SPS; frame-based mode and Annex B start codes are not offered; VALID-but-not-ACTIVE DPB entries stall the VPThe HEVC parser reads untrusted web content, so it refuses anything that would index past the driver's tables: entry point counts and offsets, slice addresses, collocated references, weight denominators, and decode-parameter DPB indices the V4L2 core leaves unbounded. Clients that send slice parameters (FFmpeg, GStreamer, libva-v4l2-request) keep the existing paths, and the H.264 control defaults are unchanged.
Refs #44: this is the Chromium side; the Firefox RDD sandbox is separate.
Overlap: #433 carries aurora-silicon's AVD rewrite into the same files, and #378 changes
avd_h264_max_slices(), which commit 8 calls. This series is againstintegration/auroraat 8eddf5a; I will rebase onto whichever lands first.Evidence
J516M (MacBook Pro 16" M3 Max,
apple,t6034, AVD hw 40000), OS firmware 14.7, m1n1 1.6.1.aurora22, kernel 7.2.11-15 (linux-aurora-j516m, AVD directory identical tointegration/aurora8eddf5a) with this series' AVD directory built out of tree. Chromium 153.0.8010.36, mpv 0.41.0, FFmpeg 9.0.2, GStreamer 1.28.8, libva-v4l2_request-avd 1.3.r11.r169, Hyprland 0.56.2.Before: Chromium VP9 frames differ from software decode by mean |RGB| 115.8 (green). HEVC jobs time out. H.264 decodes in software.
After: Chromium hardware vs software decode, frames 0, 7, 13 and 24 of 14 clips (640x360 to 1080p, 4 slices per picture, weighted P/B, H.264 Baseline): 14/14 match (mean |RGB| diff < 3, AVD interrupts > 0).
Before: a V4L2 test on
/dev/video0sets P010 and allocates CAPTURE buffers, then sends a 10-bit VP9 frame control. The driver accepts it into 4,259,840-byte buffers; the stream needs 5,437,440.After:
-EBUSY. 4:2:2 into NV16 likewise.Before:
avd_hevc_parse_rejects_wrapping_entry_point_countagainst the parser without the bound fix:After:
kunit.py run --arch=arm64 --kunitconfig=drivers/media/platform/apple/avd(LLVM, 16K pages, KASAN): 32/32, the 22 existing H.264 cases and 10 new HEVC parser cases.Other clients, same build: mpv
hwdec=auto-safevsnoongpuandgpu-next, 28/28 match. FFmpeg VA-API vs software, 16/16 bit-exact including 10-bit VP9 and AV1. GStreamerv4l2sl{h264,h265,vp9,av1}dec, bit-exact including 10-bit. No AVD errors in the kernel log.YouTube, the same 4K60 VP9 video: Chromium CPU 290% → about 63% of one core, whole machine 22.6% → about 8.5%. A 1080p30 AV1 video played on the AVD with 2 of 833 frames dropped.
check-commits.py: 8/8 signed off.compile-check.py --per-commit(LLVM, W=1): pass, 0 new warnings.checkpatch.pl --strict: clean apart from "does MAINTAINERS need updating?" for the two new files.Not tested: other SoCs (M1, M2, M4, M5), Firefox, interlaced H.264. 10-bit VP9 and AV1 do not play in Chromium 153, which cannot derive a P010 layout from V4L2 (
Cannot derive stride); the driver now refuses that stream rather than overrunning its buffers.Merge Danger
Door: two-way
Driver-only. No uAPI structures change; the H.264 start code and decode mode menus gain values, with the same defaults.
Blast Radius: AVD
Every AVD client on every SoC. HEVC pictures queued without slice parameters but with start codes are now parsed in the kernel instead of decoded with stale parameters. A client that allocates CAPTURE buffers before its first stream control and then sends a 10-bit or 4:2:2 stream now gets
-EBUSYinstead of buffers the hardware overruns.