Skip to content

media: apple: avd: hardware video decode in Chromium - #436

Open
hungbenjamin402 wants to merge 8 commits into
omacom:integration/aurorafrom
hungbenjamin402:avd-chromium-hwdec
Open

hungbenjamin402 wants to merge 8 commits into
omacom:integration/aurorafrom
hungbenjamin402:avd-chromium-hwdec

Conversation

@hungbenjamin402

Copy link
Copy Markdown

Summary

Chromium's built-in V4L2 stateless decoder (--enable-features=AcceleratedVideoDecoder) now decodes H.264, HEVC, VP9 and AV1 on the AVD. On the current driver every codec fails in Chromium, each for a different reason:

Codec Chromium 153, current driver Cause
VP9, AV1 solid green The first stream control resolves the image format while CAPTURE buffers exist: -EBUSY, the job runs on zeroed parameters, every buffer returns V4L2_BUF_FLAG_ERROR
HEVC solid green, H0 error timeouts Chromium sets no SLICE_PARAMS or entry points; the driver programs stale ones
H.264 silent software fallback try_ctrl refuses Chromium's placeholder SPS; frame-based mode and Annex B start codes are not offered; VALID-but-not-ACTIVE DPB entries stall the VP
 avd_s_ctrl()                                   # 1
-  ANY -> stream format, CAPTURE busy: -EBUSY
+  ANY -> stream format, CAPTURE busy: recompute the layout,
+    keep the buffers if they fit, else -EBUSY
 avd_hevc_run_preamble()                        # 2, 3
+  no SLICE_PARAMS in the request && picture starts with a start code
+    avd_hevc_parse_slices()                    # 7.3.6.1, 8.3.4, bounds-checked
 avd_h264_run()                                 # 4-8
-  try_ctrl: refuse SPS without frame_mbs_only
+  refuse interlaced per picture
+  VP slots for VALID && ACTIVE entries only
+  START_CODE_ANNEX_B (+ leading start code detected)
+  DECODE_MODE_FRAME_BASED
+    avd_h264_run_frame()                       # in-kernel slice headers, 7.3.3

The HEVC parser reads untrusted web content, so it refuses anything that would index past the driver's tables: entry point counts and offsets, slice addresses, collocated references, weight denominators, and decode-parameter DPB indices the V4L2 core leaves unbounded. Clients that send slice parameters (FFmpeg, GStreamer, libva-v4l2-request) keep the existing paths, and the H.264 control defaults are unchanged.

Refs #44: this is the Chromium side; the Firefox RDD sandbox is separate.

Overlap: #433 carries aurora-silicon's AVD rewrite into the same files, and #378 changes avd_h264_max_slices(), which commit 8 calls. This series is against integration/aurora at 8eddf5a; I will rebase onto whichever lands first.

Evidence

J516M (MacBook Pro 16" M3 Max, apple,t6034, AVD hw 40000), OS firmware 14.7, m1n1 1.6.1.aurora22, kernel 7.2.11-15 (linux-aurora-j516m, AVD directory identical to integration/aurora 8eddf5a) with this series' AVD directory built out of tree. Chromium 153.0.8010.36, mpv 0.41.0, FFmpeg 9.0.2, GStreamer 1.28.8, libva-v4l2_request-avd 1.3.r11.r169, Hyprland 0.56.2.

  • Before: Chromium VP9 frames differ from software decode by mean |RGB| 115.8 (green). HEVC jobs time out. H.264 decodes in software.
    After: Chromium hardware vs software decode, frames 0, 7, 13 and 24 of 14 clips (640x360 to 1080p, 4 slices per picture, weighted P/B, H.264 Baseline): 14/14 match (mean |RGB| diff < 3, AVD interrupts > 0).

  • Before: a V4L2 test on /dev/video0 sets P010 and allocates CAPTURE buffers, then sends a 10-bit VP9 frame control. The driver accepts it into 4,259,840-byte buffers; the stream needs 5,437,440.
    After: -EBUSY. 4:2:2 into NV16 likewise.

  • Before: avd_hevc_parse_rejects_wrapping_entry_point_count against the parser without the bound fix:

    BUG: KASAN: use-after-free in parse_slice.constprop.0+0x4098/0x4c5c
    

    After: kunit.py run --arch=arm64 --kunitconfig=drivers/media/platform/apple/avd (LLVM, 16K pages, KASAN): 32/32, the 22 existing H.264 cases and 10 new HEVC parser cases.

  • Other clients, same build: mpv hwdec=auto-safe vs no on gpu and gpu-next, 28/28 match. FFmpeg VA-API vs software, 16/16 bit-exact including 10-bit VP9 and AV1. GStreamer v4l2sl{h264,h265,vp9,av1}dec, bit-exact including 10-bit. No AVD errors in the kernel log.

  • YouTube, the same 4K60 VP9 video: Chromium CPU 290% → about 63% of one core, whole machine 22.6% → about 8.5%. A 1080p30 AV1 video played on the AVD with 2 of 833 frames dropped.

  • check-commits.py: 8/8 signed off. compile-check.py --per-commit (LLVM, W=1): pass, 0 new warnings. checkpatch.pl --strict: clean apart from "does MAINTAINERS need updating?" for the two new files.

Not tested: other SoCs (M1, M2, M4, M5), Firefox, interlaced H.264. 10-bit VP9 and AV1 do not play in Chromium 153, which cannot derive a P010 layout from V4L2 (Cannot derive stride); the driver now refuses that stream rather than overrunning its buffers.

Merge Danger

Door: two-way

Driver-only. No uAPI structures change; the H.264 start code and decode mode menus gain values, with the same defaults.

Blast Radius: AVD

Every AVD client on every SoC. HEVC pictures queued without slice parameters but with start codes are now parsed in the kernel instead of decoded with stale parameters. A client that allocates CAPTURE buffers before its first stream control and then sends a 10-bit or 4:2:2 stream now gets -EBUSY instead of buffers the hardware overruns.

…trol

Chromium's V4L2 stateless decoder sets the decoded (CAPTURE) format and
allocates its buffers before it sets any stream control. The first
stream control (H.264 or HEVC SPS, VP9 frame, AV1 sequence) then makes
get_image_fmt() resolve the image format from ANY, which avd_s_ctrl()
treats as a format change on a busy CAPTURE queue and rejects with
-EBUSY. v4l2_ctrl_request_setup() fails, the job runs with zeroed
parameters, validate_dec_params() rejects them with -EINVAL, and every
CAPTURE buffer comes back with V4L2_BUF_FLAG_ERROR. Chromium still
displays those buffers, so VP9 and AV1 video is solid green.

When the image format is still ANY, the CAPTURE queue is busy and the
chosen decoded format suits the resolved image format, recompute the
decoded layout for it and keep the buffers if they still fit. The
layout was sized for ANY, whose compressed planes are those of 8-bit
4:2:0, so a 10-bit or 4:2:2 stream usually needs more: refuse it with
-EBUSY when the allocated buffers are smaller than its layout, so the
hardware never writes past them.

Tested on a MacBook Pro 16" M3 Max (J516M, T6034): Chromium 153 plays
VP9 and AV1 on the AVD and its frames match software decode (VP9 mean
|RGB| diff 1.6, was 115.8 with solid green frames).

Fixes: d9bc216 ("media: apple: add avd driver")
Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
Chromium's V4L2 stateless HEVC decoder queues each picture as Annex B
slice NAL units (3-byte start codes) and sets only the SPS, PPS,
scaling matrix and decode parameters. It never sets
V4L2_CID_STATELESS_HEVC_SLICE_PARAMS or the entry point offsets. The
driver then programs the AVD from stale or zeroed slice parameters,
every job times out ("H0 error") and the video is solid green.

When a picture's request carries no slice parameters and the picture
starts with a start code, parse the slice segment headers from the
bitstream instead (7.3.6.1): the slice parameters, the weighted
prediction table, the entry point offsets and the reference picture
lists (8.3.4), built from the decode parameters' RPS index lists. The
short and long term RPS in the slice header are skipped by the sizes
the decode parameters give, which are counted in RBSP bits as the bit
reader reads them. Offsets and sizes are raw byte counts from each
slice's start code, so the existing walk over consecutive slices is
unchanged. Requests that carry slice parameters (FFmpeg, GStreamer,
libva-v4l2-request), and pictures without start codes from a client
that set its slice parameters once, take the old path.

The bitstream now comes straight from the client, so the parser
refuses what would make the driver index past its tables or the
hardware read past a slice: entry point counts beyond the picture's
CTBs or the table, entry points past the slice data, a slice address
past the picture, a collocated reference past the active list, weight
denominators outside 0..7 (7.4.7.3), and decode parameter reference
counts or DPB indices past the active DPB, which the V4L2 core does not
bound. It parses Main and Main 10 only: Range Extensions slice headers
can carry fields gated on PPS flags that the V4L2 PPS does not have.

Tested on a MacBook Pro 16" M3 Max (J516M, T6034) with Chromium 153:
640x360, 1920x1080, 4 slices per picture with WPP entry points, and
weighted P/B prediction all match software decode on I, P and B
frames (mean |RGB| diff < 0.2). VA-API decode through
libva-v4l2-request and GStreamer's v4l2codecs stay bit-exact.

Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
Build Annex B pictures bit by bit from the slice segment header syntax
and check the parser's output against values worked out by hand: a
two-slice I picture and a P picture with reference lists, a collocated
picture and weights. Check that it refuses what would make the driver
index past its tables: a wrapping entry point count, entry points past
the slice data, a slice address past the picture, a collocated
reference past the list, out-of-range weight denominators, reference
counts and DPB indices past the DPB, and Range Extensions formats.

Add a .kunitconfig, so that the tests run with KASAN under
tools/testing/kunit/kunit.py --arch=arm64.

Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
Chromium's V4L2 stateless decoder sets a placeholder H.264 SPS before
it has parsed the stream, to learn which decoded formats the driver
offers for the bit depth: only the bit depths and chroma_format_idc are
filled in, every flag is clear. The driver refused any SPS without
frame_mbs_only_flag in try_ctrl, so that VIDIOC_S_EXT_CTRLS failed with
-EINVAL and Chromium gave up on the hardware decoder and fell back to
software decoding.

Interlaced coding is a property of the stream being decoded, not of the
control value set before streaming starts. Accept the SPS and refuse to
decode a picture whose SPS lacks frame_mbs_only_flag instead; the
CAPTURE buffer comes back with V4L2_BUF_FLAG_ERROR, as for other
unsupported input.

Fixes: d9bc216 ("media: apple: add avd driver")
Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
stream_refs() pushes a reference header for every VALID DPB entry and
the slice's reference list commands address those headers by DPB
index, while the header's reference count is taken from the number of
ACTIVE entries. The V4L2 interface lets a client list pictures that
are only waiting for output as VALID but not ACTIVE, and Chromium and
GStreamer do. The count in the header then falls short of the headers
pushed and the VP stalls on some P pictures until the watchdog resets
it (seen with Chromium on an x264 stream that uses weighted prediction
with duplicated references).

Number a VP slot for each entry that is both VALID and ACTIVE, push
only those, give the header their count and translate DPB indices to
slots in the reference list commands. A reference to an entry without
a slot is refused like a reference to an invalid entry. For clients
that list only reference pictures, packed from index 0 (FFmpeg and the
VA-API v4l2-request driver), the commands are unchanged.

Fixes: d9bc216 ("media: apple: add avd driver")
Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
…e run

stream_slice() derived the address and length of the coded slice data
from the copy of the slice it was queued in. Store both in the run when
the slice is copied, so that a later change can point the VP at slice
data found elsewhere.

No functional change.

Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
Offer V4L2_STATELESS_H264_START_CODE_ANNEX_B. When it is selected, skip
to the slice's NAL unit before looking for the slice data. The NAL unit
runs up to the next start code, without the trailing zero bytes in
front of a four-byte start code: they are not part of the NAL unit, and
the AVD never completes a CAVLC picture whose slice data carries them.

Clients may send start codes but leave the start code control at its
default (Chromium does), so data that begins with a start code is taken
as Annex B as well: a NAL unit never begins with a zero byte. The
control default stays without start codes, which FFmpeg and the VA-API
driver use, and those streams decode as before.

Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
Chromium's V4L2 stateless H.264 decoder only works in frame-based mode:
it queues all slices of a picture in one OUTPUT buffer, each behind an
Annex B start code, with the SPS, PPS, scaling matrix and decode
parameters but without slice parameters or prediction weights. The
driver offered only slice-based mode, so Chromium fell back to software
decoding.

The VP itself is driven per slice: every slice needs its position, QP,
deblocking parameters, reference lists, weights and the bit position
where its data starts. In frame-based mode, find the slice NAL units
in the buffer and parse their headers in the kernel (H.264 7.3.3):
build each reference list from the initial list that the v4l2-h264
helpers derive from the DPB, apply ref_pic_list_modification(), read
pred_weight_table() with the inferred default factors, and skip
dec_ref_pic_marking(). The parsed values fill the same slice parameter
and weight structures a slice-based client sets, so the instruction
stream is built by the existing code, one job per picture.

A slice may make more reference list entries active than the stream
has references (8.2.4.2); conforming slice data never uses the extra
ones, so they point at the first reference rather than at an unset DPB
slot.

The VP reads the slices straight from the OUTPUT buffer, which stays
queued until the picture completes, so nothing is copied. Redundant
slices are dropped; SP/SI slices, field pictures and pictures that do
not start at macroblock 0 are refused. Frame-based decoding needs
Annex B start codes: without them nothing marks where one slice ends
and the next begins.

Tested on a MacBook Pro 16" M3 Max (J516M, T6034) with Chromium 153:
640x360, 854x480 Baseline, 1920x1080 and 4 slices per picture match
software decode on I, P and B frames.

Signed-off-by: Benjamin Nguyen <hungbenjamin402@gmail.com>
@freber

freber commented Nov 17, 2026

Copy link
Copy Markdown

Claude Code wrote this up from my tests.

I ran this series on a J615 (15-inch M3 Air, apple,t8122, AVD hw 40000) to get a T8122 data point next to your J516M. The module was the 8 commits at ab11ae516004 plus #260, cherry-picked onto my J615 tree and built out of tree against the running kernel (7.2.11-16). Only the sources in drivers/media/platform/apple/avd/ differ from the stock module's. The kernel also carries #410, #491 and an earlier version of #500. Chromium 156.0.8078.79 on Hyprland with --use-angle=gles-egl and --enable-features=AcceleratedVideoDecoder, mesa-aurora 26.3.0_devel.20261107, on AC.

YouTube

Big Buck Bunny in 4K60 (aqz-KE-bpKQ), full screen on an external 4K monitor (3840x2160 at 60 Hz over USB-C, lid closed), 60 s per run. The software run is the same browser with --disable-features=AcceleratedVideoDecoder on the stock module. Power is the SMC's Total System Power, which covers the whole Mac but not the external monitor (the built-in panel was off). The die figure is the hottest of the 12 SMC die sensors (the macsmc_soc_die zone from #410), averaged over the run.

Software VP9 AVD VP9 AVD AV1
Format (itag) vp09.00.51.08 (315) vp09.00.51.08 (315) av01.0.13M.08 (401)
Chromium CPU, 100 % is one core 194 % 13 % 13 %
Whole machine busy 29.6 % 6.7 % 6.8 %
Dropped frames 195 of 3684 0 of 3670 0 of 3671
AVD interrupts 0 7926 7492
Total system power 16.9 W 14.2 W 14.3 W
Hottest die sensor, mean 83.2 C 74.5 C 75.5 C

The software run came after an earlier run and started at 83.8 C. The AVD runs started at about 75 C, so read the die row as rough.

With hardware decode on, YouTube switched to AV1 by itself. To get the VP9 run I hid AV1 from the page (MediaSource.isTypeSupported, canPlayType and mediaCapabilities.decodingInfo answering no for av01).

Calls

WebRTC loopback in the change-codecs sample, since pc1 no longer has a codec list. The camera was Chromium's fake device at 640x480 and 20 fps, 60 s per codec. The sender encodes in software in the same tab.

VP9 profile 0 AV1 profile 0 H.264 42001f
decoderImplementation ExternalDecoder (V4L2VideoDecoder) same same
powerEfficientDecoder true true true
Frames decoded, dropped, freezes 1222, 0, 0 1223, 0, 0 1222, 0, 0
Decode time per frame 0.51 ms 0.47 ms 0.35 ms
Chromium CPU, encoder included 9 % 10 % 7 %

The kernel log had no AVD errors during any of it.

VM_BIND errors in the first YouTube tab

Chromium logged lines like these six times during the YouTube tests, each followed by eglCreateImage failed with 0x00003003 and OzoneImageBacking::ProduceSkiaGanesh failed to create GL representation. The address, size and handle vary.

DRM_IOCTL_ASAHI_VM_BIND failed: Operation already in progress (vm=1, ops=1): {addr=0x4df4604000 range=0x1874000 offset=0 handle=24 flags=0x6}
import failed: Could not bind BO at 0x4df4604000

All six came in the first two minutes of the first YouTube tab. One was at playback start, four right after the full screen switch and one after another page resize. It never came back in two later tabs that went through the same steps. Steady playback and the WebRTC runs never logged it either. None of the measured 60 s windows dropped a frame. At startup the first tab dropped 10 frames and the later tabs 7 and 2, so I can't tie the drops to it. It looks like a problem with importing the decoded buffers on the GPU side, but my kernel also carries #410 and an earlier #500, so I can't rule those out. Did you see it on the J516M?

Not tested

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants