Skip to content

fix: register runtime actors for depth and object masks - #95

Open
chaiyuntian wants to merge 1 commit into
SimWorld-AI:mainfrom
chaiyuntian:fix/runtime-spawn-sensors
Open

chaiyuntian wants to merge 1 commit into
SimWorld-AI:mainfrom
chaiyuntian:fix/runtime-spawn-sensors

Conversation

@chaiyuntian

@chaiyuntian chaiyuntian commented Sep 11, 2026

Copy link
Copy Markdown

Runtime actors spawned through Communicator appear in RGB but are absent from depth and object masks in the official Base20260201 Windows backend. Calling its existing color annotation command registers the render components for both sensors. The spawn methods now perform that registration after setting the initial transform and mobility, so the annotation also follows subsequent movement.

Fixes #88.

  • Register objects, humanoids, scooters, vehicles, pedestrians, traffic signals, and waypoint marks after placement. City generation keeps its configured palette and annotates at the final initial pose.
  • Derive a stable nonblack default RGB label from the actor name. Explicit set_color(name, rgb) palettes remain supported; automatic colors do not guarantee global uniqueness.
  • Raise on failed/unacknowledged blueprint spawns and color registration instead of continuing to configure an actor that may not exist. Accept both ok and the actor-name response returned by the official package.
  • Add 19 regression tests, a four-platform/version CI matrix, sensor documentation, and examples/verify_runtime_sensors.py for reproducible live validation.

Verification

The same live script was run against upstream 1f588ec and this fix on fresh servers using the complete official Base20260201 Windows package, UE 5.3.2, RTX 4090, 640×480 capture. The ZIP SHA-256 and exact commands are recorded in the evidence report.

Actor Mask pixels before → after Median actor / background depth after Mask center movement
Box 0 → 4,802 11.23 m / 24.38 m 39.37 px
Humanoid 0 → 410 11.92 m / 15.09 m 44.96 px
Vehicle 0 → 3,765 10.64 m / 14.44 m 54.03 px

Before the fix, the original depth NPY and mask PNG responses remain byte-for-byte identical to the empty-scene responses despite the actors appearing in RGB. After the fix, all three actors have foreground depth in over 99% of their mask pixels; all three retain foreground depth and move their masks when repositioned. The live script exits 1 on baseline and 0 with the fix.

Before, fixed, and moved: RGB, metric depth, and object mask

Full evidence and reproduction instructions include unmodified PNGs, losslessly compressed original NPY payloads, SHA-256 request transcripts, per-actor measurements, engine annotation logs, and JUnit reports. Evidence is committed separately on the fork so maintainers can inspect/download it without adding recorded scenes to the code PR.

This live validation covers the bundled box, humanoid, and vehicle assets on Windows. Other spawner paths are covered at the protocol boundary by the automated tests; custom Blueprint components still need to be renderable by the backend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runtime spawned actors are visible in RGB but missing from depth and object_mask observations

1 participant