RK3588: RgaBlit fails (Invalid argument / I/O error) on NV12 DMABUF with padded stride (2304×1080 / 2816×1944) + dmesg: “RGA_MMU unsupported memory larger than 4G”

[chrisbeeze]

Environment

• SoC/Board: RK3588 (NanoPi-R6C)
• OS/Distro: Debian 11
• Kernel: 6.1
• librga: 1.10.5
• rga driver: 1.3.4

What I’m doing
I’m running "H.265 RTSP transcoding" using Rockchip MPP decode+encode (GStreamer-based). In “zero-copy” mode (DMABUF), the pipeline triggers "RGA blit" internally (likely for stride/align conversion). Under these resolutions, RgaBlit fails and video drops.

Case 1 (1920×1080):

• Buffer size: 3732480 bytes

• Source looks like padded NV12: stride_w=2304, h=1080 (because 2304×1080×3/2 = 3732480)

• Encoder log shows it wants aligned height: stride [1920:1088]

• librga print:

  • RgaBlit(...) RGA_BLIT fail: Invalid argument
  • rect[0, 0, 1920, 1080, 2304, 1080, ...]
  • rect[0, 0, 1920, 1080, 1920, 1088, ...]

• kernel dmesg around failure:

  • rga_mm: RGA_MMU unsupported memory larger than 4G!
  • ... job buffer map failed ... submit failed

Case 2 (2592×1944):

• Buffer size: 8211456 bytes
• Padded NV12: stride_w=2816, h=1944 (2816×1944×3/2 = 8211456)
• Similar RgaBlit failures and dmesg.

Note: system RAM usage is ~1GB during test, but dmesg still prints “memory larger than 4G”. I suspect this refers to DMA/IOMMU addressability rather than actual RAM usage.

What I tested

• If I fall back to CPU copy/repack (strip padding into tightly packed NV12), encoding works.
• Only the DMABUF “zero-copy” path triggers RGA and fails.

Questions:

1. On RK3588 (multi-core RGA), how should users handle buffers where width stride is padded but height is not 16-aligned (1080 vs 1088, 1944 vs 1952)?

2. Is this expected to be rejected by RGA core selection?

3. Your FAQ says “RGA_MMU unsupported memory larger than 4G” happens when memory does not satisfy the selected core requirements; what is the recommended way to:

   • force RGA3 when possible, or
   • allocate <4G-addressable memory (DMA32), or
   • avoid selecting RGA2-only paths? ([GitHub][2])

4. Is importbuffer_xx required/recommended when using external dmabuf FDs on RK3588 to avoid this under load? (FAQ mentions wrapbuffer vs importbuffer behavior.) ([GitHub][2])

#73

[chrisbeeze]

Latest finding from my tests on RK3588 / GStreamer + MPP zero-copy path:

I do not think this is simply “system RAM > 4 GB”.
In my tests, total RAM usage is around ~1 GB, but RGA still fails with:

Example failing case:

• H.265 RTSP transcoding
• visible resolution: 2560x1440
• MPP frame size: 6082560 bytes
• that corresponds to NV12 with padded source stride 2816x1440
  (2816 * 1440 * 3 / 2 = 6082560)

RGA failure log shows:

Important observation:

• with the same stream count / system load, switching this case from H.265 to H.264 avoids the failure
• CPU repack fallback also works
• only the DMABUF zero-copy path that triggers RGA repack fails

So my current hypothesis is:
this is not actual RAM exhaustion, but a failing combination of:

1. padded-stride imported DMABUF,
2. RGA core selection / hardware policy,
3. memory addressability requirements for the selected core.

The librga FAQ says this error pattern can happen when no hardware core can access the memory configured by the current request, and that some cores only support under-4G-addressable memory.

Could you please confirm:

1. Should NV12 DMABUF copy from src stride 2816x1440 to dst stride 2560x1440 be supported on RK3588?
2. For imported external DMA FDs, is the correct usage:
   • importbuffer_fd(fd, src_wstride, src_hstride, format)
   • wrapbuffer_handle(... visible_width, visible_height, format, wstride, hstride)
     ?
3. Is the expected workaround for this case:
   • allocate all RGA-touched buffers from DMA32 / under-4G-addressable memory, or
   • force scheduling to an RGA core that can access this memory, or
   • avoid this RGA repack path entirely?

At the moment, the practical workaround on my side is CPU repack for these problematic padded-stride cases, but I would like to keep zero-copy if there is a correct allocator/import/core-selection solution.