Scale a stacked video context on several threads - #723
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 38b0b1d216
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
A video-conditioned policy sends a temporal stack — 25 frames per camera for a 1.6 s window at 15 Hz — and `RestrictImageSize` scaled all 50 of them one after another, on the thread that drives the arm. The frames are independent and Pillow drops the GIL for a resize, so they do not have to queue. Measured on one round's real 1280x720 frames, scaling a 25-frame two-camera stack to 512x288: 267 ms serial, 83 ms threaded. The output is byte-identical either way — a test asserts a stack over the bar equals the same frames scaled one at a time, so the model is shown exactly what it was before. A stack shorter than four frames stays serial: below that the pool costs more to raise than it saves. Ticket: Positronic-Robotics/internal#1168 #refs
38b0b1d to
3d4c303
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3d4c3037af
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
A pool sized only by frame count raises workers on a rig whose process has one usable CPU, where threading wins nothing. `_workers` reads `os.process_cpu_count()`, so a cgroup quota or an affinity mask bounds it rather than the machine's core count, and one usable CPU takes the serial path. Ticket: Positronic-Robotics/internal#1168 #refs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 26b238bd00
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
`os.cpu_count()` reports the host, so a process pinned to one core of many still read as many and took the threaded path. `_usable_cpus` reads the affinity mask where the platform publishes one. A cgroup CPU quota is still invisible to both, and the docstring says so. Ticket: Positronic-Robotics/internal#1168 #refs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0da1ee47f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| usable CPU, and below ``_PARALLEL_FROM`` it costs more to raise than the frames take.""" | ||
| if frames < self._PARALLEL_FROM: | ||
| return 1 | ||
| return max(1, min(frames, self._MAX_WORKERS, _usable_cpus())) |
There was a problem hiding this comment.
Rule overspecific violated:
_workers chooses multiple threads solely from frame count and CPU capacity, so a stack of at least four frames whose dimensions already fit the bound—for example, 640×480 frames with the default 640×640 codec—creates a thread pool even though _scaled immediately returns every frame without performing a Pillow resize. This adds thread creation and scheduling to the synchronous inference path without parallelizable work; include whether the stack actually needs resizing in the decision and retain the serial path when it does not.
AGENTS.md reference: AGENTS.md:L7-L8
Useful? React with 👍 / 👎.
1 participant
What
RestrictImageSizescales the frames of a stacked observation on a small thread pool instead of one after another. Output is byte-identical; a stack shorter than four frames stays serial, below which the pool costs more to raise than it saves.Why
A video-conditioned policy sends a temporal stack — for the current curie deployment, 25 frames per camera covering a 1.6 s window at 15 Hz, so 50 frames per inference. The codec scaled all 50 in a loop, on the thread that drives the arm, and that loop sits between the moment a chunk finishes playing and the moment the next chunk's first setpoint goes out. The frames are independent and Pillow drops the GIL for the resize, so nothing required them to queue.
Measured on one round's real 1280x720 frames, scaling a 25-frame two-camera stack to 512x288: 267 ms serial, 83 ms threaded on 8 cores. The win scales with the rig's core count and disappears on a single-core host, where the serial path is what runs anyway.
The threading is an optimisation, so equivalence is the property that matters:
test_a_threaded_stack_scales_to_the_same_pixels_as_one_threadasserts a stack over the bar comes back equal to the same frames scaled individually. If that ever fails, the model is being shown something different and the change is wrong.Not attempted here: a faster resampler. cv2 is up to 20x quicker than Pillow's antialiased bilinear, but it produces different pixels, which changes what an evaluated policy sees — a separate decision, with an eval behind it.
Verification
uv run pytest positronic --ignore=positronic/simulator— 1246 passed, 8 skipped.ruff checkandruff format --checkclean.The A/B above ran the two code paths against the same frames in one process and compared with
np.array_equal.Refs
Positronic-Robotics/internal#1168.