Platform View Implementation for Video Player#254
Open
jwinarske wants to merge 19 commits into
Open
Conversation
shared GL context, test corpus harness Critical bug fixes - nv12.h: move the YUV→RGB conversion off the hardcoded BT.601 math in the fragment shader into a mat3 + vec3 uniform pair and add six presets (BT.601 / BT.709 / BT.2020 × limited / full). Host picks the right preset per stream from GstVideoColorimetry — modern HD content (BT.709 limited) and UHD (BT.2020) no longer render with subtly wrong hue / gamma. Sampler bindings (glUniform1i texY/texUV) moved from load_pixels into draw_core so uniform state travels with the draw. - nv12.h: drop the dead `load_rgb_pixels` path. The upstream capsfilter pins the fakesink to NV12 so `n_planes != 2` never legitimately happens. - video_player.cc handoff_handler: fail-loud format + geometry check replaces the silent RGB fallback. Unexpected format / plane count / dimensions log an error, bump `frames_dropped`, and return. Also bumps `frames_dropped` on `gst_video_frame_map` failure. - video_player.cc prepare(): read source colorimetry from caps before `gst_video_info_set_format` overwrites it with the NV12 default; push the matching preset into the shader. Instrumentation - New stats.h with VideoPlayerStats: frames_received / rendered / dropped / late (atomic uint64), upload_ns_total / render_ns_total, last_frame_pts_ns / last_wallclock_ns, negotiated_format / colorspace / decoder_name (mutex-guarded strings), and three boolean flags (uses_dmabuf, uses_hw_decoder, uses_shared_gl_context) surfaced for later use. - handoff_handler times load_pixels and draw_core with steady_clock into upload/render_ns_total. frames_late increments when wallclock advance between consecutive frames outruns PTS delta by > 33ms. - `deep-element-added` on playbin auto-discovers the decoder factory name; prefixes (v4l2/omx/mpp/vpu/vaapi/nvh264/msdk) set `uses_hw_decoder` as a heuristic until later hardware backends report it authoritatively. - prepare() stores `negotiated_format` (GstVideoFormatInfo name) and `negotiated_colorspace` (gst_video_colorimetry_to_string — canonical strings like `bt709:16-235`, `bt2020-10:16-235`). - New EventChannel at `video_player_linux/stats/<texture_id>`: 1Hz push while a listener is subscribed, silent and zero-cost otherwise. Emits the full snapshot as an EncodableMap and sends an immediate first value on subscribe so consumers don't wait a full second for the initial reading. Fetched test-corpus harness - `test/.gitignore` excludes everything under `media/` except the manifest + .gitkeep, keeping binary samples out of the repo. - `test/media/manifest.json` lists all ten streams with expected properties (codec / container / w×h / duration / colorimetry / interlaced / has_audio / rotation / license). One entry has a real URL + pinned sha256 (Flutter's CC-BY-3.0 bee.mp4); nine are `url: null` placeholders signalling "source URL TBD". - `test/scripts/fetch_media.py` downloads each entry, verifies SHA-256, and has a `--learn` mode that records first-seen hashes back into the manifest. `--only` limits to specific streams. Cached files are revalidated on every run; hash mismatch deletes the local file and exits non-zero. - `test/CMakeLists.txt` registers two custom targets (`fetch_video_player_test_media` and its `_learn` variant) plus two harness-level ctests (manifest parses, fetcher --help) that run without the corpus. Gated behind new plugin option `BUILD_VIDEO_PLAYER_LINUX_TESTS` (OFF by default). Dedicated shared EGL context per player - video_player.h / video_player.cc: VideoPlayer gains EGLDisplay / EGLContext / EGLSurface fields and a `use_legacy_context_` fallback flag. `CreateSharedGlContext` calls FlutterDesktopPluginRegistrarGetEglContext to obtain the embedder's display + config + share_context and creates an ES 3 context with a 1×1 pbuffer surface. Non-EGL backends (Vulkan, headless) or older embedders fail the call and the ctor falls back to the legacy global-mutex + TextureMakeCurrent path. - Plumbed FlutterDesktopPluginRegistrarRef through the C API entry point and plugin / player constructors. Needed because the client-wrapper `PluginRegistrarDesktop` keeps the raw C ref behind a protected accessor. - handoff_handler refactored into a `gl_work` lambda with two dispatch paths. Legacy: lock g_texture_context_mutex, TextureMakeCurrent, VAO rebind, run gl_work, TextureClearCurrent. Shared: `MakeContextCurrent` (fast-path skip when already current), run gl_work, glFlush, `eglMakeCurrent(NO_CONTEXT)` to release. The release is deliberate — the plan called for keeping the context current across frames, but GStreamer recycles streaming threads on state transitions and `eglMakeCurrent(ctx)` on thread B while still current on thread A returns EGL_BAD_ACCESS. Releasing per-frame trades one extra eglMakeCurrent for correctness. - With shared context, the VAO is bound once in the ctor and the per-frame defensive rebind is dropped (VAO state is context-level and survives current/release cycles). - Dispose branches on the same flag: shared path makes-current, resets the shader, releases, then destroys context + surface. Legacy path keeps the original TextureMakeCurrent/Clear dance. - stats_.uses_shared_gl_context records which path was chosen so the stats event channel reflects it. Init-flow fix — send `initialized` from GstDiscoverer metadata - Dart's MiniController.initialize() awaits the `initialized` EventChannel event and only then calls play(). The previous handler gated `initialized` on reaching PLAYING, which required play() having already been called — a deadlock. Frames never rendered. - Init()'s EventChannel listen callback now fires SendInitialized immediately using the GstDiscoverer metadata already captured at construction (width / height / duration / has_video). `sent_initialized_` still dedups against the handoff path and the audio-only PLAYING case so the event is sent at most once per player. - Matches how upstream Flutter video_player emits `initialized` on Android (Player.Listener.onPlayerReady) and iOS (AVPlayerItem.status == .readyToPlay): off source metadata, not first rendered frame. Build / CI hygiene - New `BUILD_VIDEO_PLAYER_LINUX_TESTS` CMake option and optional `test/` subdirectory. Known follow-up (tracked separately): first play of an HTTP video stalls at one frame due to playbin queue2 `use-buffering=TRUE` underrun-after-EOF thrash. Unmasked by this change — the init-flow fix lets Dart actually trigger play() on a cold HTTP pipeline for the first time. Larger VIDEO_PLAYER_BUFFER_SIZE helps but doesn't fully eliminate it; proper fix is `use-buffering=FALSE` or ignoring buffering<100% events after the first 100% has landed. Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Two fixes land together because their symptoms masked each other during earlier smoke testing. Previously: first rendered frame stuck on screen, no further frames, both for file-backed asset playback and for HTTP sources. After: local asset and warm-cache HTTP render continuously at 30fps; cold-cache HTTP still stalls at frame #1 (separate issue, tracked as a follow-up). fakesink can-activate-pull -> FALSE - Was TRUE in the original code. With pull mode enabled, basesink may activate its pull scheduling after the preroll buffer lands and stop pulling further buffers from upstream. `basesink:5` GST_DEBUG showed only ~10 `rendering object` events total across the video + audio sinks and position frozen at ~166ms for the whole session. fakesink's handoff signal fired 2× (preroll + first PLAYING buffer) and never fired again. - Push mode explicitly selected via `can-activate-pull=FALSE`. Upstream pushes every decoded buffer into the chain function unconditionally, handoff fires per buffer, rendering proceeds. GST_MESSAGE_BUFFERING handler — gate forced state transitions on a new ever_played_ latch - Added `std::atomic<bool> ever_played_{false}` on VideoPlayer. Latches true in OnMediaStateChange the first time the pipeline reaches GST_STATE_PLAYING. - Handler still forwards bufferingStart/End edges to Dart (via std::atomic::exchange for single-shot edge detection under concurrent bus dispatch) so UI can reflect buffering. - Forced pipeline state transitions (`set_state(PAUSED)` on <100%, `set_state(PLAYING)` on 100%) now only run BEFORE `ever_played_` is set — i.e. during the cold-start buffering ramp. After first-play those transitions actively fight playbin's own buffering state machine on short HTTP streams: queue2 drops below low-percent immediately after EOS as the decoder drains it, which previously thrashed us back to PAUSED every frame and prevented the 100% message from ever meaning "done". GST_DEBUG confirmed the 0-3% buffering oscillation disappears with the gate in place. - The PAUSED branch also picks up a defensive `gst_element_get_state` check so we don't redundantly pause a pipeline that's still in PAUSED preroll. Rejected (reverted) approach — preroll to PAUSED in the ctor - Tried adding `gst_element_set_state(playbin_, GST_STATE_PAUSED)` after pipeline wiring to warm up the HTTP source during Dart's initialize(). Caused a white-screen regression on the second load of an HTTP URI in the same process (sequence: remote -> asset -> back to remote produced all-white frame on the new instance). Likely interaction with shared-EGL texture-namespace reuse when glGenTextures on the new player picks up a name that was just freed. Removed from this commit; documented on the follow-up ticket so it isn't retried. Test matrix after the fix, verified against tcna-packages/.../example/basic bundle via ivi-homescreen with cmake-debug-vp (Wayland EGL + Compositor, no DRM): - Local asset (Butterfly-209.mp4): plays continuously, handoff fires 1/30/60/90/120/150... - Second-load HTTP (bee.mp4, after tab-switch dispose+recreate): plays continuously, same pattern. - First-load HTTP (cold process): stalls at frame #1 (task #7). Known follow-ups: - First-load HTTP cold-start stall (task #7). Not a regression from this commit; was previously masked by the frame-#1 stall on all paths. Hypothesis: cold TCP+TLS handshake + first Range request timing collides with Play()'s clock start so the first decoded frames arrive "late" against pipeline clock and the queue2 -> decoder -> videoconvert chain wedges. Standard fix to try next: `GST_PLAY_FLAG_DOWNLOAD` on playbin flags, which spools small HTTP streams to a temp file before playback. Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
unblock cold HTTP playback ApplyPlaybackSpeed() in OnMediaStateChange(PLAYING) unconditionally sends a FLUSH + ACCURATE seek on the first PLAYING transition because rate_ starts at a -2.0 sentinel that never matches the default pending rate of 1.0. On cold HTTP sources that flush resets souphttpsrc: it closes the in-flight TCP + TLS connection and re-issues the Range request from byte 0, and the pipeline sits idle waiting for re-fetch while Dart's play() has already driven PLAYING. Result: first HTTP load of a URI plays exactly one rendered frame then stalls for the lifetime of the player. Second and subsequent loads hit souphttpsrc's warm connection / OS page cache and work, which is why the symptom previously looked like "first click frozen, come back and it plays". Root cause verified with GST_DEBUG=multiqueue:5,decodebin:5,\ videoconvert:4,souphttpsrc:5,qtdemux:5: multiqueue queue_0 pushes buffers with PTS 0 / 33ms / 66ms, then an additional PTS-0 buffer appears — the second cycle of PTS-0 buffers is the flush-seek restarting the stream from scratch. Fix: when rate_ is still at the -2.0 sentinel, use GST_SEEK_FLAG_NONE for the flags instead of GST_SEEK_FLAG_FLUSH | GST_SEEK_FLAG_ACCURATE. The rate event propagates without flushing the source queue, so souphttpsrc doesn't tear down and reopen. Mid-stream rate changes after the first call still use FLUSH + ACCURATE so user-driven playback-speed changes remain snappy. Verified against tcna-packages/.../example/basic bundle: - local asset: plays end-to-end - cold HTTP first load: plays (was: frozen at frame #1) - warm HTTP second+ load: plays Known imperfection that's out of scope for this commit: the first ~1s of cold HTTP playback runs faster than 1.0× because fakesink with sync=TRUE + default max-lateness=-1 renders the post-preroll backlog of decoded frames (all arriving late because cold preroll takes ~600ms) back-to-back without dropping. Playback self-corrects when the buffer queue catches up with pipeline clock (typically at the first loop restart for short streams). A proper fix requires splitting Dart's "initialize" state into "metadata available" vs "ready to play" so play() doesn't drive PLAYING until preroll is actually complete; ticketed separately. Rejected approaches from the same investigation (do NOT retry; each produced a distinct regression): - Preroll to PAUSED in the ctor → white-screen regression on the second HTTP load after an intervening asset tab visit. Texture lifecycle interaction on the shared-EGL namespace. - GST_PLAY_FLAG_DOWNLOAD on playbin → changed the cold-HTTP failure mode from frozen frame to full white screen, same user-visible "no video" outcome. - fakesink max-lateness=20ms + qos=TRUE → drops every cold-start frame because they're all >20ms past their PTS after the preroll delay; "no video" until loop restart. - Skip the first-time seek entirely when pending=1.0 → defeats the sentinel's original purpose; playback stays fast for the whole session (never self-corrects). Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Sets up the runtime selection and sink-element plumbing for the
dmabuf → EGLImage zero-copy path. This is
scaffolding: both paths exist, the probe picks one at ctor, but the
dmabuf branch still falls through to the existing PBO upload until the
EGLImage import lands in a follow-up. No playback regression
intended — on systems without hw decoders the appsink simply
receives raw NV12 samples and the CPU path runs as before.
Render-path probe
- New RenderPath enum {PboShaderUpload, DmabufZeroCopy} on
VideoPlayer.
- `ProbeRenderPath()` runs after CreateSharedGlContext in the ctor.
Returns PboShaderUpload when any of the following is true:
* VIDEO_PLAYER_DISABLE_DMABUF env var is set (kill switch for
field debugging and for drivers that advertise dmabuf import
but choke on it).
* `use_legacy_context_` is true (the dmabuf path needs our own
shared EGL context to own the EGLImage → texture binding).
* The EGL display doesn't advertise EGL_EXT_image_dma_buf_import.
- `has_egl_extension()` helper does whole-token matching against the
eglQueryString(EXTENSIONS) list so "EGL_KHR_image" doesn't
false-match "EGL_KHR_image_base".
- Result stored in `render_path_` and reflected into
`stats_.uses_dmabuf` so the stats event channel surfaces it.
- GL_OES_EGL_image is NOT probed here — doing so requires making
our context current just for the query, which we haven't done at
ctor time. It's checked at bind time.
Appsink replaces fakesink on the zero-copy path
- CMakeLists: add gstreamer-app-1.0 and gstreamer-allocators-1.0 to
the pkg-config list.
- video_player.cc: include <gst/app/gstappsink.h> and
<gst/allocators/gstdmabuf.h>.
- Ctor sink construction splits on `render_path_`:
* DmabufZeroCopy → `appsink` with
caps="video/x-raw(memory:DMABuf), format=NV12;
video/x-raw, format=NV12",
emit-signals=TRUE, max-buffers=2, drop=TRUE, sync=TRUE.
Connect the `new-sample` signal to OnNewSample.
* PboShaderUpload → existing fakesink + signal-handoffs +
can-activate-pull=FALSE path (untouched).
- The caps advertise dmabuf AND raw NV12 so the pipeline negotiates
whatever the upstream decoder can produce. On software decoders
(openh264dec, avdec_*) the raw side wins; on v4l2/vaapi/vpu the
dmabuf side wins.
OnNewSample handler
- Pulls the GstSample, inspects the first GstMemory to detect
`gst_is_dmabuf_memory()`.
- On first dmabuf sample: flips `stats_.uses_dmabuf` to true and
logs once (observed-capability vs advertised-capability).
- On first raw sample: flips `stats_.uses_dmabuf` to false and
logs once (common case on software decoders).
- For now every sample — dmabuf or raw — is routed through
the existing handoff_handler via a synthetic call, so the CPU
render path handles both. A follow-up will branch on is_dmabuf
and import the FDs into an EGLImage instead of mapping to CPU.
- appsink's first argument to the new-sample callback is already a
GstElement (appsink is a subclass); handoff_handler's element
parameter is unused, so passing the appsink through is safe.
Smoke-tested on AMD Ryzen 9 9950X (no VA-API for H.264) with the
tcna-packages/.../example/basic bundle:
- Probe returns dmabuf-zerocopy (EGL extension present)
- Decoder: openh264dec (software)
- First sample arrives → "Appsink delivering raw (non-dmabuf) NV12;
staying on CPU upload path"
- `start playing` fires, playback continues, no errors
- VIDEO_PLAYER_DISABLE_DMABUF=1 forces back to fakesink path.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Builds on the appsink scaffolding commit with a real, reusable
dmabuf frame descriptor. Still no EGLImage import yet — every sample,
dmabuf or raw, falls through to the existing CPU upload path. What's
new is that OnNewSample can now observe the dmabuf shape of each
buffer and stats_.uses_dmabuf tracks what's actually being delivered
instead of what was advertised at ctor time.
New header — dmabuf_frame.h
- `DmabufPlane { fd, offset, stride }` and `DmabufFrame { w, h,
planes[2], n_planes, drm_fourcc, drm_modifier }` POD-ish structs.
Copied cheaply between the GStreamer streaming thread and the GL
upload path; the GstSample must be kept alive until the FDs are
consumed because `fd` is borrowed from
`gst_dmabuf_memory_get_fd()` (do not close).
- `Fourcc(a,b,c,d)` constexpr helper so we don't need to pull in
<drm_fourcc.h> just to get DRM_FORMAT_NV12.
- `kDrmFormatModLinear = 0`, `kDrmFormatModInvalid = 0x00ffffffffffffff`.
video_player.cc
- Include dmabuf_frame.h and <optional>.
- New anonymous-namespace helper `GstVideoFormatToDrmFourcc()` maps
the gst formats we can plausibly import via EGL_LINUX_DMA_BUF_EXT
(NV12, NV21, I420, YV12, BGRA, BGRx) and returns 0 for anything
else (forces CPU fallback).
- New anonymous-namespace helper `ExtractDmabufFrame(GstSample*)`:
* Reads GstVideoInfo from the sample caps for width / height /
n_planes / plane strides / plane offsets.
* Detects single-FD-many-plane layout (typical for VA-API and
driver-side allocations, `gst_buffer_n_memory` == 1, offsets
from GstVideoInfo) vs multi-FD-one-plane-each layout (typical
for v4l2 stateful/stateless decoders, each plane in its own
GstMemory) and populates DmabufPlane correctly either way.
* Parses drm modifier from the `drm-format` caps string format
"NV12:0x200000000b" introduced in GStreamer 1.24; defaults to
DRM_FORMAT_MOD_LINEAR when absent.
* Returns nullopt on any inconsistency (non-dmabuf memory mixed
in, unsupported format, fewer mem chunks than declared planes).
OnNewSample now
- Peeks memory 0, short-circuits the extraction call when it's not
dmabuf. This keeps the hot path cheap for the common
non-hw-decoder case.
- When the sample IS dmabuf, calls ExtractDmabufFrame. On the first
transition (raw→dmabuf or dmabuf→raw) logs the shape of the
first dmabuf frame once, or the fallback reason:
* "dmabuf sample: 1920x1080 fourcc=NV12 modifier=0x0 planes=2 fd0=41
off0=0 stride0=1920" — what the next commit will feed into
eglCreateImageKHR.
* "Appsink sample is dmabuf but extraction failed …"— a decoder
that advertised dmabuf caps but produced a format/layout we
don't know how to import; a follow-up may extend the format map.
* "Appsink delivering raw (non-dmabuf) NV12; staying on CPU upload
path" — software decoder, expected fallback.
- stats_.uses_dmabuf now reflects reality (what the decoder is
producing) rather than capability (what the embedder advertised).
- Frame still routed to handoff_handler for CPU upload — a follow-up
replaces that call with the EGLImage bind on the dmabuf branch.
Smoke-tested on AMD Ryzen 9 9950X (no VA-API H.264) with
tcna-packages/.../example/basic:
- Probe: render path: dmabuf-zerocopy.
- Negotiated decoder: openh264dec (software).
- First sample: "Appsink delivering raw (non-dmabuf) NV12; staying
on CPU upload path" logged once; subsequent samples don't re-log.
- start playing, no errors.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Completes the core of the zero-copy path set up by the preceding
scaffolding commits.
OnNewSample on the DmabufZeroCopy render path now builds an EGLImage
from the extracted DmabufFrame via eglCreateImageKHR with
EGL_LINUX_DMA_BUF_EXT, binds it to shader_->textureId via
glEGLImageTargetTexture2DOES, and calls MarkTextureFrameAvailable —
no CPU map, no NV12 shader, no PBO upload. The fragment shader kept
from the PBO path is bypassed entirely; the EGL driver performs the
YUV→RGB conversion in hardware using the color-space + range hints
we attach.
On software decoders (openh264dec, avdec_*) the sink still negotiates
plain raw NV12 and the handler falls through to handoff_handler, so
this change is a no-op on the current test platform (AMD Ryzen 9 CPU,
no VA-API H.264).
video_player.h
- Include <EGL/eglext.h> and "dmabuf_frame.h".
- New VideoPlayer members:
* EGLImageKHR egl_image_current_ — at most one live image at a
time for now (a follow-up replaces this with a bounded in-flight
deque so the compositor can still sample the previous frame
while we import the next one).
* bool egl_dmabuf_modifiers_ok_ — cached once at ctor from
EGL_EXT_image_dma_buf_import_modifiers; required for tiled /
compressed formats (AFBC, UBWC, Amphion) but optional for
linear buffers.
* ImportDmabufFrame(const DmabufFrame&) / DestroyDmabufImage()
instance methods.
video_player.cc
- Include <GLES2/gl2ext.h> for GLeglImageOES.
- Anonymous-namespace lazy-resolver for eglCreateImageKHR,
eglDestroyImageKHR, glEGLImageTargetTexture2DOES via
eglGetProcAddress. Resolved once and cached in static function
pointers. The streaming thread is the sole caller so there's no
thread-safety ceremony.
- ColorspaceHintsForCaps() parses the GstVideoColorimetry string
we already store in stats_.negotiated_colorspace
(e.g. "bt709:16-235", "bt2020-10:16-235") into the
EGL_YUV_COLOR_SPACE_HINT_EXT / EGL_SAMPLE_RANGE_HINT_EXT pair.
bt2020 → REC2020, bt601 → REC601, anything else → REC709 default.
Range: "0-255" or ":full" → FULL, otherwise NARROW.
- ImportDmabufFrame builds the attr list in a fixed-size EGLint
array (no allocation on the hot path):
WIDTH, HEIGHT, EGL_LINUX_DRM_FOURCC_EXT, and per-plane
FD/OFFSET/PITCH for planes 0..n_planes-1. Optional per-plane
MODIFIER_LO/HI pairs when frame.drm_modifier is non-linear and
the modifier extension is available. Optional YUV hints at the
end. eglCreateImageKHR is called with EGL_NO_CONTEXT so the
image is display-wide and safe to bind from any context in the
share group.
- Destroys the previous EGLImage only after the new one succeeds,
so a creation failure leaves the last-good frame on screen.
- glBindTexture + glEGLImageTargetTexture2DOES re-specifies the
texture's storage. shader_->textureId keeps its GL name (Flutter
is already told to sample that name) — the underlying dmabuf
replaces the RGBA/FBO attachment the Shader class set up. The
FBO becomes stale but we never draw to it on this path.
- Ctor caches egl_dmabuf_modifiers_ok_ right after ProbeRenderPath
so the hot path doesn't repeatedly query eglQueryString.
- OnNewSample: when a dmabuf sample successfully extracts AND
render_path_ is DmabufZeroCopy, MakeContextCurrent, import, bind,
glFlush, release context, MarkTextureFrameAvailable, increment
frames_rendered, return. On import failure the frame is marked
dropped (stats_.frames_dropped++) and falls through to the CPU
handoff_handler — next frame's Import retry is the recovery
mechanism rather than a runtime sink swap.
- Dispose: DestroyDmabufImage() inside the context-current block,
before shader_.reset(), before eglDestroyContext.
Runtime behavior matrix:
| Decoder path | Result |
|---------------------------------------|---------------------------------|
| Software (openh264dec, avdec_*) | CPU NV12 upload (unchanged) |
| dmabuf linear (v4l2h264dec) | Zero-copy, no modifier attrs |
| dmabuf tiled w/ mod ext available | Zero-copy with modifier attrs |
| dmabuf tiled w/o mod ext | ImportDmabufFrame returns false,|
| | CPU fallback per frame |
| VA-API (vavp9dec, vaav1dec on AMD) | Zero-copy when caps negotiate |
Smoke-tested against tcna-packages/.../example/basic:
- Probe: render path: dmabuf-zerocopy.
- Decoder: openh264dec (software).
- Appsink delivering raw (non-dmabuf) NV12 → staying on CPU upload
path; start playing fires; no errors.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
dmabuf/EGLImage/GstSample lifetime
Replaces the preceding single-slot EGLImage model
with a bounded in-flight deque keyed on {EGLImage, GstSample*}.
Each new import pushes a fresh entry and evicts the oldest when the
deque exceeds kMaxInFlight=2, guaranteeing the frame the compositor
is actively sampling stays live for at least one additional frame
cycle. Eliminates the tearing risk from the preceding commit's eager
destroy-on-replace on real hardware-decoded content.
video_player.h
- Add <deque> to the include block.
- Replace the `EGLImageKHR egl_image_current_` member with:
* Nested `struct InFlightFrame { EGLImageKHR image; GstSample*
sample; }`.
* `static constexpr size_t kMaxInFlight = 2;` — current frame +
one backstop so the compositor's last-sampled frame survives
until the next import completes.
* `std::deque<InFlightFrame> in_flight_frames_` +
`std::mutex in_flight_mutex_`.
- Extend `ImportDmabufFrame` signature to take the originating
`GstSample*`. On success the deque owns the ref; on failure the
caller retains ownership and can fall back / unref.
- Rename `DestroyDmabufImage() → DrainInFlightFrames()`; signature
and ownership rules documented inline.
video_player.cc
- ImportDmabufFrame:
* Builds + binds the EGLImage as before (same attr list etc.).
* If GL bind fails, destroys the just-created image and returns
false without touching the deque — caller retains sample ref.
* On bind success, acquires in_flight_mutex_, push_back's
{new_image, gst_sample_ref(sample)}, pops the front if size >
kMaxInFlight, and releases the lock. Destroys the popped
InFlightFrame's image + unrefs its sample outside the lock so
GL calls and gst_sample_unref don't happen with the mutex
held.
* Returns true. Caller must not unref the sample.
- DrainInFlightFrames:
* Swaps in_flight_frames_ under the mutex, destroys every
EGLImage + unrefs every sample in the local copy. Caller
contract: the player's EGL context is current.
- OnNewSample:
* Dmabuf-success branch drops the old `gst_sample_unref(sample)`
call — the deque owns the ref now.
* On ImportDmabufFrame failure the existing
stats_.frames_dropped++ and fall-through to handoff_handler
stay intact; the caller still owns the sample and unref's it
via the common tail-call path.
- Dispose: replace `DestroyDmabufImage()` with
`DrainInFlightFrames()` under the `MakeContextCurrent() …
shader_.reset() … eglMakeCurrent(NO_CONTEXT)` block so every
image is released from the thread the context is bound to.
Threading contract
- Pushes happen on the GStreamer streaming thread (OnNewSample).
- Drain happens on the Dart platform thread (Dispose), with
`gst_element_set_state(NULL)` having already joined the
streaming thread so no concurrent push is possible.
- Pop-on-evict happens on the streaming thread under the deque
mutex.
- `gst_sample_unref` is documented thread-safe so unref'ing the
popped sample outside the lock is fine regardless of which
thread last touched it.
Smoke-tested on AMD Ryzen 9 9950X (software decoder — the CPU
fallback path is what actually runs in this environment; dmabuf
branch is dead code here but compiles cleanly and the types line
up): render path: dmabuf-zerocopy, Appsink delivering raw
(non-dmabuf) NV12 → staying on CPU upload path, start playing
fires, no EGL errors, no regression.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Header-only scaffolding for the hardware-backend abstraction.
No implementation, no callers yet — follow-up commits add a
self-registering BackendRegistry, platform detection that populates
PlatformProfile, the baseline GenericV4L2Backend, and VideoPlayer ctor
wiring that selects a backend for the stream's codec and builds the
decoder fragment of the pipeline explicitly (instead of the current
playbin-auto-plug-everything approach).
New file plugins/video_player_linux/backend_interface.h contains:
- PlatformProfile enum: 18 SoC families we plan to target (NXP
i.MX 8M / 8M+ / 8QM / 8QXP / 95, Rockchip RK3568 / RK3588,
Renesas R-Car H3 / M3 / V3H / V4H / H4, Qualcomm SA8155 / SA8295,
MediaTek MT8195 / CT-X1) plus GenericV4L2 and Software fallbacks,
plus Auto sentinel. Kept in this header rather than a detection
header so backend implementations can branch on profile without
pulling in detection code.
- DecoderConfig struct: low_latency flag (disable frame reordering,
emit in decode order — good for ADAS/camera, bad for B-frame
content), buffer_count (0 = backend default, tune up for smoother
playback under load, tune down on memory-constrained targets).
- BackendCapabilities struct: supported_codecs (short names:
"h264", "h265", "vp9", "av1", "mjpeg"), supported_modifiers
(DRM_FORMAT_MOD_LINEAR must always be present; non-linear
modifiers are platform-specific AFBC / UBWC / Amphion), optional
supported_formats constraint, max_width/max_height (0 = unknown),
supports_10bit / supports_hdr / supports_afbc booleans.
- VideoDecoderBackend pure-virtual interface:
* name() — stable identifier matching config.toml strings
* priority() — selection order (software ~0, generic v4l2 ~50,
platform-specific 80..100)
* is_available() — cheap startup probe, must be side-effect-free
(registry calls it during construction)
* query_capabilities() — deferred full probe, may be expensive
(opens /dev/videoN etc.), registry caches the result
* build_decoder_bin(codec, cfg) — returns a floating-ref'd
GstElement that accepts encoded video on its sink pad and
produces dmabuf-backed NV12 (ideally modifier-tagged) on its
src pad; nullptr for unsupported codecs
* build_converter_bin(src_mod, dst_mod) — optional hardware
format converter between decoder and appsink (Rockchip RGA,
MediaTek MDP, Renesas FCP); nullptr when no conversion needed
Documentation-heavy. Ownership, lifetime, and threading contracts
for each method are inline so backends land consistently.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Process-wide registry of hardware decoder backends. Scaffolding only
— no backends register yet (first concrete implementation lands in a
follow-up (GenericV4L2Backend)). VideoPlayer doesn't call into the
registry yet either (wiring lands in a later commit). This commit is the
API and the selection / caching infrastructure.
New files
- plugins/video_player_linux/backend_registry.h
- plugins/video_player_linux/backend_registry.cc
Self-registration pattern documented in the header:
namespace {
[[maybe_unused]] auto reg_foo = [] {
BackendRegistry::Instance().RegisterBackend(
std::make_unique<FooBackend>());
return 0;
}();
}
API surface
- Instance() — Meyers singleton, guaranteed thread-safe construction
and late enough in the destructor ordering that backends which
touch the registry from their own cleanup (discouraged, but) still
see a live instance.
- RegisterBackend(std::unique_ptr<VideoDecoderBackend>) — called
from static initializers. Backend objects are retained for the
process lifetime; the registry owns them. Unavailable backends
(is_available() == false at registration time) are still kept in
the list so Select() doesn't re-probe on every stream creation.
- Select(codec, profile) — highest-priority available backend whose
cached query_capabilities() lists the codec. Returns nullptr when
nothing fits (caller falls back to playbin auto-plug). profile is
accepted but currently unused as a selector — backends encode
platform fit via is_available() + query_capabilities() so the
registry doesn't need a branch per platform.
- FindByName(name) — explicit lookup for config-driven overrides
(TOML "h264_backend = 'foo'" → FindByName("foo")).
- All() — debug / logging accessor.
Capability caching
query_capabilities() is allowed to be expensive (open /dev/video*,
enum formats, stat sysfs). The registry caches the result in an
unordered_map keyed by backend pointer, populated lazily under the
registry mutex. Process lifetime cache is fine for the <10 backends
we expect; no eviction logic needed.
Thread safety
- Registration runs at static-init, single-threaded.
- Selection runs from the Dart platform thread during VideoPlayer
construction — one thread per active player create, potentially
concurrent with other players.
- One mutex guards both the backends_ vector and the caps_cache_.
Each Select() takes it once for the whole selection pass so the
cache insert during the walk stays consistent.
CMake
backend_registry.cc added to plugin_video_player_linux sources.
Build is a no-op beyond registering the file; no caller exists yet.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Runtime configuration for everything we plan to expose as a user /
integrator knob: platform profile override, decoder selection per
codec, pipeline tuning, texture format preferences, HDR tone-mapping
parameters, and logging verbosity. No caller yet — wires into
VideoPlayer ctor in a follow-up after platform detection and the first
concrete backend land.
New files
- plugins/video_player_linux/config.h
- plugins/video_player_linux/config.cc
Typed Config struct carries
conservative defaults so a system with no TOML files gets sensible
behavior (auto-detect platform, auto-select backend, AFBC enabled
where supported, info-level logging, tone-map HDR with Reinhard at
500 nit target).
Merge order (later wins) — Config::Load iterates these paths and
calls ApplyDocument(doc, &cfg) for each file that parses:
1. Built-in defaults (struct initializers).
2. /etc/video_player_linux.toml (system).
3. $XDG_CONFIG_HOME/video_player_linux.toml (user), with
$HOME/.config/video_player_linux.toml as the XDG fallback.
4. $VIDEO_PLAYER_LINUX_CONFIG (single-file env override).
5. [platform.<profile>] section of every loaded document, in the
same merge order — applied last so platform overrides still win
over user-specified base-section values.
Platform resolution
- If any config file sets [platform].profile to something other
than "auto", that wins.
- Otherwise the detect_platform_fn callback is invoked — populated
by a devicetree / sysfs detector added in a follow-up. Today callers
pass nullptr and the profile stays "auto".
- Empty / "auto" profile skips the overlay pass entirely.
[platform.<name>] structure
TOML examples nest subtables under the platform section:
[platform.rockchip_rk3588]
decoder.h264_backend = "rockchip_mpp"
texture.enable_afbc = true
toml++ parses that as platforms → rockchip_rk3588 → { decoder: {...},
texture: {...} }. ApplyPlatformOverlay unwraps the profile subtree
and hands it to the same field-by-field ApplyDocument() used for
root sections — avoids a second parser.
Error handling
- Missing files: silently skipped. Typical deployment ships no
/etc file and no user override, so the default case is "all
paths missing, use built-ins".
- Malformed TOML: caught (toml::parse_error) and logged at warn
level. Continue with whatever was already merged. Users with a
typo see a clear error pointing at the line + file.
- Final config echoed at SPDLOG_DEBUG after Load() so field
overrides are auditable.
CMake
config.cc added to plugin_video_player_linux's sources. toml++ is
header-only and already on the include path via ivi-homescreen's
third_party/tomlplusplus — no pkg-config or CMake target changes
needed.
Smoke test deferred — Config::Load has no caller in this commit.
Manual verification against a test file under
$VIDEO_PLAYER_LINUX_CONFIG will happen when a follow-up integrates the
loader into VideoPlayer's ctor.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Probes /sys/firmware/devicetree/base/compatible first, falls back to
vendor-specific sysfs indicators and finally a V4L2-codec-device
presence check, resolving the running SoC to the PlatformProfile
enum defined in backend_interface.h. Feeds Config::Load's platform
callback and backend selection.
No caller yet in this commit — the Config loader currently runs
with a null detect_platform_fn so everything stays on "auto".
A follow-up wires DetectPlatform() in during VideoPlayer ctor setup.
New files
- plugins/video_player_linux/platform_detection.h
- plugins/video_player_linux/platform_detection.cc
API
- DetectPlatform() — returns PlatformProfile. Process-lifetime
stable result; safe to call from any thread (pure reads from
/sys).
- PlatformProfileName(PlatformProfile) — enum → lowercase
snake_case string ("imx8m_plus", "rockchip_rk3588",
"qualcomm_sa8295", "generic_v4l2", "software", …). Keys match
[platform.<key>] sections so the same string round-
trips through Config.
- PlatformProfileFromName(string_view) — inverse; unknown → Auto
(so config typos don't silently pin the wrong platform).
Probe order (first match wins)
1. Devicetree compatible — null-separated blob at
/sys/firmware/devicetree/base/compatible. 24-entry match table
covering every platform enumerated in backend_interface.h:
* NXP i.MX 8M / 8M+ / 8QM / 8QXP / 95 (fsl,imx8m[pqmn],
fsl,imx8qm, fsl,imx8qxp, fsl,imx95)
* Rockchip RK3568 / RK3588 / RK3588s (rockchip,rk356[68],
rockchip,rk3588, rockchip,rk3588s — subtle ordering: -s must
precede non-s so substring matches land on the right entry)
* Renesas R-Car H3 / M3 / V3H / V4H / H4 (renesas,r8a77950,
r8a77951, r8a77960, r8a77965, r8a77980, r8a779g0, r8a779f0)
* Qualcomm SA8155 / SA8155p / SA8295 / SA8295p (qcom,sa8155p
before sa8155, sa8295p before sa8295 for the same reason)
* MediaTek MT8195 / MT8678 (internal name for CT-X1)
2. Vendor sysfs probes for thin-DT systems:
* /sys/devices/soc0/family — Qualcomm qcom_socinfo. "Snapdragon"
or "MSM8" coarsely picks QualcommSA8155 (caller can override
via config.toml if they know it's actually SA8295).
* /sys/devices/platform/*vcodec* — MediaTek's mtk-vcodec kernel
module registers here; presence → MediatekMT8195.
3. Generic V4L2 — scan /sys/class/video4linux/*/name for known
substrings ("codec", "vpu", "venus", "hantro", "rkvdec",
"amphion", "decoder"). Matches any V4L2 hardware codec
regardless of family, so boards we haven't explicitly enrolled
still light up the GenericV4L2 backend.
4. Software — nothing detected. This is what happens on x86 dev
hosts (no DT, no vendor sysfs, no V4L2 codec device).
Directory walks are defensive: skip_permission_denied option on
directory_iterator so we don't throw inside a sandboxed rootfs.
Build-only change in this commit. No runtime test because nothing
calls DetectPlatform() yet; the AMD test host would correctly
return Software. clang-format-18 + clang-tidy-18 clean.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Concrete VideoDecoderBackend for the V4L2 M2M baseline. Priority 50 — above pure software (~0) and below every platform-specific backend (80..100), so it naturally takes the slot when no vendor backend matches on platforms that expose generic v4l2codecs factories. New files - backend_generic_v4l2.h — GenericV4L2Backend final class declaration plus RegisterGenericV4L2Backend() free function for explicit registration from plugin init (works around static-library initializer stripping — a [[maybe_unused]] lambda in an unreferenced TU gets dropped at link time). - backend_generic_v4l2.cc — implementation. Codec → factory table h264 → v4l2h264dec → v4l2slh264dec h265 → v4l2h265dec → v4l2slh265dec vp8 → v4l2vp8dec → v4l2slvp8dec vp9 → v4l2vp9dec → v4l2slvp9dec av1 → v4l2av1dec → v4l2slav1dec Stateful factory is preferred (faster on existing vendor BSPs); stateless (media request API) is the fallback. FactoryChoice struct keeps the primary/fallback pair together so the list is trivially extensible when new codecs land. Behavior - is_available(): probes all eight known factory names via gst_element_factory_find; returns true if any registered. GStreamer only registers v4l2codecs factories when it finds a matching kernel device at init, so factory presence is a reasonable proxy for "has V4L2 codec hardware". - query_capabilities(): reports DRM_FORMAT_MOD_LINEAR only; supports_10bit / supports_hdr / supports_afbc all false at this tier (platform backends will override). supported_codecs populated from the factory probe. - build_decoder_bin(codec, cfg): instantiates the decoder and sets output-io-mode = dmabuf-export (5) when the property exists, so decoded frames land as dmabuf FDs for the EGLImage import. Applies num-output-buffers from DecoderConfig.buffer_count when the decoder exposes it. Returns nullptr if no factory matches. - build_converter_bin(): returns nullptr — generic tier has no hardware converter; caller falls back to videoconvert. Rockchip RGA / MediaTek MDP / Renesas FCP backends will override. Registration RegisterGenericV4L2Backend() is idempotent (std::atomic<bool> guard) and registers a fresh unique_ptr<GenericV4L2Backend> with BackendRegistry::Instance(). To be wired into the plugin init path by a later commit. Build + lint - CMake: backend_generic_v4l2.cc added to plugin_video_player_linux sources. - clang-format-18: clean. - clang-tidy-18: clean (is_available() uses std::any_of to satisfy readability-use-anyofallof; std::array over C-array). Known limitations - No HW decoder on the x86 dev box — is_available() returns false locally, so this backend is a no-op in smoke testing until we run on a v4l2-equipped target (i.MX8M, RK3588, …). - low_latency flag is accepted but ignored; there is no generic v4l2 property for it. Platform backends will honor it per-SoC. Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Plumbs the backend abstraction (registry, config, platform detection,
generic V4L2 backend) into the running plugin so every VideoPlayer
instance asks the registry which backend owns its codec and the
selected backend gets to tune the decoder that playbin auto-plugs.
No pipeline re-architecture — playbin stays the owner of pipeline
construction. Observe-and-tune for now; follow-up platform backends will expand the surface.
Plugin init (video_player_plugin.{h,cc})
- New Config + PlatformProfile members on VideoPlayerPlugin. Loaded
once in the ctor via DetectPlatform() + Config::Load(), then passed
to every VideoPlayer. A non-"auto" platform_profile in the TOML
overrides the detected value.
- Explicit RegisterGenericV4L2Backend() call at ctor time — static
library initializers can be dropped by the linker when nothing in
the TU is externally referenced, so the backend needs a visible
entry point called from the init path.
- Logs the detected platform and every registered backend's
priority/availability once at startup.
Codec extraction (video_player_plugin.cc)
- discover_media_info() now pulls the video stream's caps and turns
"video/x-h264" / "video/x-h265" / "video/x-vp9" / "video/x-vp8" /
"video/x-av1" / "image/jpeg" into the short codec key that
MediaInfo.video_codec carries forward. Anything unrecognized
leaves the field empty and Select() returns nullptr, letting
playbin auto-plug work exactly as before.
Per-player selection (video_player.{h,cc})
- VideoPlayer constructor now takes (Config, PlatformProfile) and
copies them onto the player. Config is passed by value for
modernize-pass-by-value — the struct is short-lived in the plugin
and moved in.
- Picks a backend in ctor:
1. Per-codec override from Config (h264_backend, h265_backend, …).
2. Global decoder_backend override.
3. BackendRegistry::Select(codec, profile) by priority.
Bad override names fall through with a warning; empty codec
(audio-only or unknown) skips selection entirely.
- selected_backend_ is a non-owning pointer into the process-wide
registry (lifetime is safe — registry is a Meyers singleton).
Stored alongside DecoderConfig built from Config fields.
Observe-and-tune hook (backend_interface.h, backend_generic_v4l2.*)
- New virtual VideoDecoderBackend::ConfigureAutoPluggedDecoder(dec,
codec, cfg) with a default no-op. Called from
OnDeepElementAdded — playbin has just auto-plugged a video decoder
and the element is still in GST_STATE_NULL (safe moment to set
v4l2 io-mode properties).
- GenericV4L2Backend overrides it to apply output-io-mode =
dmabuf-export (5) and num-output-buffers, but only when the
factory it sees matches its own primary/fallback table — foreign
decoders (avdec_*, vaapi*, openh264dec) are left untouched.
- Extracted ApplyDecoderConfig() helper so build_decoder_bin() and
ConfigureAutoPluggedDecoder() share the same tuning path.
Stats surface (stats.h, video_player.cc EmitStats)
- Adds selected_backend to VideoPlayerStats (under meta_mutex) and
to the stats channel's EncodableMap. Decoupled from decoder_name
because a backend may be "selected" while playbin still auto-plugs
a foreign factory (and the field then flags the mismatch).
Smoke tested on an x86 dev box (no V4L2 codec kernel device):
- Platform detected as "software".
- GenericV4L2Backend registers with priority=50, available=false.
- MediaInfo.video_codec extracted from caps as "h264" for the
standard bee.mp4 remote asset.
- Select() returns nullptr → "No backend claims codec 'h264' on
software; using playbin auto-plug" — playbin's existing
openh264dec path runs unchanged, no regression vs. the preceding commit.
Build + lint
- CMake: no changes (all new code lives in existing TUs plus the
already-added backend_generic_v4l2.cc from the preceding commit).
- clang-format-18 clean.
- clang-tidy-18 clean (Config passed by-value to satisfy
modernize-pass-by-value; member init order fixed for -Wreorder).
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
First platform-specific decoder backend built on the backend
abstraction. Targets NXP's gstreamer-imx port of the Hantro VPU
(`imxvpudec_h264`, `imxvpudec_h265`, `imxvpudec_vp8`, `imxvpudec_vp9`,
plus the older generic `vpudec`). Priority 100 so it wins over
GenericV4L2Backend (priority 50) when the NXP BSP is present.
Mainline-kernel i.MX boards that expose the same VPU through
v4l2codecs are intentionally handled by GenericV4L2Backend — this
backend's is_available() returns false when no vendor factory is
registered, so the two never race.
New files
- backend_imx8m.h — Imx8mVpuBackend final class + RegisterImx8mVpuBackend()
- backend_imx8m.cc — implementation
Factory probe
h264 → imxvpudec_h264 → vpudec
h265 → imxvpudec_h265 → vpudec
vp8 → imxvpudec_vp8 → vpudec
vp9 → imxvpudec_vp9 → vpudec
Platform-aware capabilities (keyed off DetectPlatform() cached in a
local function-static, since profile is stable for the process):
- i.MX 8M base (Hantro G1): H.264 + VP8 up to 1080p, 8-bit, LINEAR only.
- i.MX 8M Plus (Hantro G1 + G2): adds H.265 and VP9 up to 4K, 10-bit,
plus the DRM_FORMAT_MOD_ARM_NV12_4L4 4x4-tiled modifier the G2 emits
for H.265. Declared in supported_modifiers but not forced —
downstream caps negotiation picks it when the display path (Mali
with the right extensions, KMS plane) can sample it; other paths
fall back through build_converter_bin().
Low-latency path
- ApplyLowLatency() sets `low-latency=TRUE, frame-plus=0,
frame-drop=FALSE` when each property exists on the element (the
knobs vary between gstreamer-imx releases).
- Shared between build_decoder_bin() and ConfigureAutoPluggedDecoder()
so both construction paths apply identical tuning.
Buffer pool knob
- ApplyBufferCount() sets `num-capture-buffers` when exposed, falling
back to `output-buffers`. Gated on DecoderConfig.buffer_count > 0.
Converter bin
- build_converter_bin() returns nullptr. The 8M Plus
tiled→linear detile via imxvideoconvert_g2d lands with the follow-up Amphion Malone backend,
where the same G2D mechanism also detiles Amphion Malone output on
i.MX 8QM — factoring them together.
Auto-plug guard
- ConfigureAutoPluggedDecoder() tunes only elements whose factory
name is `imxvpudec*` or exactly `vpudec`. Foreign decoders
(avdec_*, v4l2*dec, vaapi*) are ignored so property-name mismatches
can't corrupt their state.
Registration
- Explicit RegisterImx8mVpuBackend() called from VideoPlayerPlugin's
ctor right after RegisterGenericV4L2Backend(). Same
static-lib-stripping rationale as the earlier generic-V4L2 registration.
Build + lint
- CMake: backend_imx8m.cc added to plugin sources.
- clang-format-18 clean.
- clang-tidy-18 clean.
Smoke tested on x86:
Registered backends:
- v4l2_stateless (priority=50, available=false)
- imx_vpu (priority=100, available=false)
h264 -> <none: playbin auto-plug>
(No regression — both backends correctly report
unavailable on hardware that has neither v4l2codecs factories nor
gstreamer-imx.)
Untestable locally
- All decoder behaviour requires an actual i.MX 8M / 8M Plus board
with gstreamer-imx installed. Commit is intentionally scoped small
so hardware validation can happen platform-by-platform.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Second platform backend. Targets the Amphion Malone video codec on
NXP's i.MX 8QM / 8QXP. Priority 100. Platform-gated so it only wins
on Amphion boards — on any other i.MX or non-i.MX system it reports
is_available=false and GenericV4L2Backend drives.
Why not just let GenericV4L2Backend handle it
NXP exposes Amphion through the standard V4L2 M2M interface, so the
decoder factory is the same `v4l2h264dec` / `v4l2slh264dec` set the
generic backend already knows. The reason for a dedicated backend is
the OUTPUT side: Amphion emits NV12 with DRM_FORMAT_MOD_AMPHION_TILED,
which current GL drivers on i.MX 8QM cannot sample. Shader output is
corrupt unless the pipeline detiles to LINEAR first — via
imxvideoconvert_g2d, zero-copy, G2D-backed.
New files
- backend_imx8qm.h — Imx8QmBackend final class + RegisterImx8QmBackend()
- backend_imx8qm.cc — implementation
Capabilities
- Codecs: h264, h265, vp8, vp9 (whatever v4l2*dec factories are live).
- Modifiers: LINEAR (always) + AMPHION_TILED (declared so a
modifier-aware caller can keep zero-copy; callers that can only
sample LINEAR ignore it and negotiate the detile path).
- max 3840x2160, 10-bit supported (Amphion V4), no HDR, no AFBC.
Decoder construction
- build_decoder_bin() tries primary/fallback pairs for each codec
(e.g. v4l2h264dec → v4l2slh264dec), applies `output-io-mode =
dmabuf-export (5)` and `num-output-buffers` when exposed. Same
tuning surface as GenericV4L2Backend — low_latency is a no-op at
this tier because Amphion low-latency is kernel-side.
Detile converter (the heart of 3.2)
- build_converter_bin() returns a real bin when src=AMPHION_TILED and
dst=LINEAR:
(sink ghost pad) imxvideoconvert_g2d → capsfilter(NV12,
drm-modifier=0) (src ghost pad)
Zero-copy preserved across the G2D hop (dmabuf in, dmabuf out —
G2D just does the detile). Other modifier pairs or same-modifier
requests return nullptr so the caller falls back to software
videoconvert.
- Missing imxvideoconvert_g2d (no gstreamer-imx installed) also
returns nullptr with a DEBUG log so the caller can diagnose.
Auto-plug guard + warning
- ConfigureAutoPluggedDecoder() applies dmabuf-export only to
`v4l2*` factories — foreign decoders are ignored.
- Emits a spdlog::warn() to flag that playbin won't insert G2D on its
own. On 8QM the playbin path will produce corrupt video until
VideoPlayer takes over explicit pipeline construction (a follow-up). Warning beats silent corruption at first play.
Registration
- RegisterImx8QmBackend() called from VideoPlayerPlugin ctor next to
the other backend registrations.
Build + lint
- CMake: backend_imx8qm.cc added to plugin sources.
- clang-format-18 clean.
- clang-tidy-18 clean.
Smoke tested on x86:
Registered backends:
- v4l2_stateless (priority=50, available=false)
- imx_vpu (priority=100, available=false)
- imx_amphion (priority=100, available=false)
No regression — both imx backends correctly gate off
on non-i.MX hardware via the platform-profile check.
Untestable locally
- All decoder behaviour + G2D detile bin linking requires an actual
i.MX 8QM/8QXP board with gstreamer-imx installed. Commit scope is
intentionally the backend only; VideoPlayer pipeline construction
(which would drive build_converter_bin) stays on playbin for now.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Third platform backend in the i.MX family. Targets i.MX 95's VPU,
which is the first NXP part to decode AV1 in hardware and the first
to preserve HDR10 / HLG side data through the decode pipeline.
Priority 105 — above the 8M (Hantro) and 8QM (Amphion) backends at
100 so it wins on i.MX 95 boards regardless of which vendor-element
set the BSP exposes.
New files
- backend_imx95.h — Imx95Backend final class + RegisterImx95Backend()
- backend_imx95.cc — implementation
Availability signal
- PlatformProfile::Imx95 (from DT compatible "fsl,imx95") OR the
presence of the imxvpudec_av1 factory. AV1 in an NXP VPU element
is a tight fingerprint — no other i.MX variant has an AV1 decoder,
so the factory check reliably identifies i.MX 95 even when DT is
thin or behind ACPI.
Per-codec factory table (ordered)
h264 → imxvpudec_h264 → v4l2h264dec → v4l2slh264dec
h265 → imxvpudec_h265 → v4l2h265dec → v4l2slh265dec
vp9 → imxvpudec_vp9 → v4l2vp9dec → v4l2slvp9dec
av1 → imxvpudec_av1 (no mainline v4l2 AV1 decoder yet)
Reflects the current split — NXP BSP ships vendor elements for all
four; mainline kernel exposes h264/h265/vp9 via v4l2codecs without
NXP patches but has no AV1 path yet.
Capabilities
- Codecs advertised only when at least one candidate factory is
registered — no lies about coverage the device doesn't have.
- Modifiers: LINEAR only. Unlike the 8M Plus Hantro G2 or 8QM
Amphion, the i.MX 95 VPU emits linear directly, so no tiled
modifier or detile bin is needed.
- max 3840x2160, 10-bit, supports_hdr=true (VPU preserves HDR side
data), supports_afbc=false.
Factory-aware tuning
- IsV4L2Factory() gate: ApplyDmabufExport() only runs on mainline
v4l2* factories. NXP's imxvpudec_* elements manage dmabuf
internally and don't expose output-io-mode, so setting it there
would silently fail the property-find check anyway — gating is
style, not correctness.
- ApplyLowLatency() shared with the 8M backend (low-latency,
frame-plus, frame-drop), gated on property presence for version
drift across gstreamer-imx releases.
- ApplyBufferCount() tries num-output-buffers (mainline v4l2),
num-capture-buffers (recent gstreamer-imx), and output-buffers
(older gstreamer-imx) in that order.
ConfigureAutoPluggedDecoder()
- Accepts both NXP vendor factories and mainline v4l2 — IsSupportedFactory
prefix-matches on both "imxvpudec" and "v4l2". Applies the same
tuning the build path does so playbin auto-plug gets parity.
build_converter_bin()
- Returns nullptr — no hardware detile needed at this tier.
Registration
- RegisterImx95Backend() called from VideoPlayerPlugin ctor next to
the other backend registrations. Same static-lib-stripping
rationale as earlier backends.
Low-latency wiring
- ApplyLowLatency() in imx8m, imx8qm, and imx95 honors
DecoderConfig.low_latency in both build and auto-plug paths.
Build + lint
- CMake: backend_imx95.cc added to plugin sources.
- clang-format-18 clean.
- clang-tidy-18 clean.
Smoke tested on x86:
Registered backends:
- v4l2_stateless (priority=50, available=false)
- imx_vpu (priority=100, available=false)
- imx_amphion (priority=100, available=false)
- imx95_vpu (priority=105, available=false)
No regression — all four backends report unavailable on hardware
that has neither the NXP vendor factories nor v4l2codecs.
Untestable locally
- All decoder behaviour requires an actual i.MX 95 board with the
vendor BSP installed. Commit scope intentionally backend-only;
VideoPlayer still uses the playbin auto-plug path.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Overrides ICompositorSurface::TextureIsTopFirst() → true. The plugin's render path produces a texture whose row 0 is the visual top, so the embedder's V-invert compensation must engage to land the image right-side-up on both window FBO 0 (Wayland) and the KMS-inverted comp.fbo (DRM framed mode). Pairs with the embedder-side contract added in ivi-homescreen. Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Platform-view entry point
-------------------------
Adds VideoPlayerView — multi-inherits PlatformView + ICompositorSurface
to bridge a VideoPlayer to the embedder's platform-view pipeline under
viewType "@views/video-player". The player continues to live in
VideoPlayerPlugin::videoPlayers and is driven through the existing
Pigeon surface; the view publishes the player's GL texture each frame
for compositor-side blending.
New C-API symbol VideoPlayerLinuxPluginCApiPlatformViewCreate wires
into the generated plugin registrant's platform-view dispatch.
VideoPlayerPlugin gains:
* Instance() — process-wide most-recent-plugin pointer so the view
path can reach the plugin-owned config and registrar without
plumbing them through the platform-views dispatcher.
* BuildPlayer() — asset/uri construction factored out of Create() so
both paths share URI whitelisting, header-injection rejection,
media discovery, and player construction.
* FindPlayer() / AdoptPlayer() — lookup and insertion for the view
path to attach to a Pigeon-created player or adopt a view-built
one.
VideoPlayer gains GetGlTextureName/Width/Height on the compositor-
surface contract; width/height come from the decoded frame and may
differ from the widget's layout size.
Rockchip backend
----------------
RockchipMppBackend (priority 100) targets gst-rockchip's MPP stack
(mppvideodec / mpph264dec / mpph265dec / mppvp9dec / mppav1dec) and
exposes ARM AFBC as an output modifier — the RK3588's 4K60 bandwidth
win. RockchipV4L2Backend (priority 90) is a thin platform-gated
subclass of the generic V4L2 path so the community V4L2 stack wins
over GenericV4L2 on Rockchip while still yielding to MPP when present.
DecoderConfig gains enable_afbc, driven from Config::texture_enable_afbc;
other backends gate on property presence and silently ignore it.
Audio-sink recovery
-------------------
OnMediaError walks GST_OBJECT_PARENT to detect errors originating
inside the audio-sink bin (e.g. pipewiresink "no target node available"
on a seat with no PipeWire session). Swallows the event, kicks
OnAudioRecovery via an idle source, and lets video keep playing.
OnAudioRecovery swaps the failed audio-sink for a sync=true fakesink
rather than leaving the broken sink attached (which risked re-firing
on the next state change), and nulls cached members that pointed into
the old bin so SetEqualizer and siblings no-op instead of dereferencing
freed memory. Adds m_valid / playbin_ null guards at entry.
EGL surfaceless context
-----------------------
CreateSharedGlContext prefers EGL_KHR_surfaceless_context over the 1×1
pbuffer. The embedder's EGLConfig is a window config; the ARM Mali
binary blob rejects eglCreatePbufferSurface on window-only configs
with EGL_BAD_MATCH. Falls back to pbuffer when the extension isn't
advertised.
Also: OnMediaError null-init of err/debug_info + null-guard err->message;
nv12.h vertex/fragment source pointers switched to auto for consistency.
Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
BuildPlayer's internal `fail` lambda: take `code` and `msg` by const reference instead of by value. The enclosed FlutterError constructor takes const refs, so the earlier std::move() calls did nothing (performance-move-const-arg); taking by value and copying on every failure was also needless (performance-unnecessary-value-param). Remaining hunks are clang-format reflows — a handful of log-statement line breaks in CreateSharedGlContext, OnMediaError, and VideoPlayerView's constructor trace. Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
jwinarske
force-pushed
the
jw/vp-updates2
branch
from
April 20, 2026 22:42
bc7faf9 to
eda6e10
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.