Skip to content

Start CPU workers safely and pass them state per call - #72

Merged
joeljose merged 1 commit into
mainfrom
fix/69-worker-safety
Sep 28, 2026
Merged

joeljose merged 1 commit into
mainfrom
fix/69-worker-safety

Conversation

@joeljose

Copy link
Copy Markdown
Owner

Closes #69

Changes

  • Start method: workers use forkserver (spawn where it doesn't exist), never plain fork. The fork server preloads numpy, scipy.fft/ndimage, dtcwt and cv2, so workers share those pages. The CPU path now also parallelises on platforms without fork; the old code fell back to serial there.

  • No module state: _POOL is gone. Each magnify_motions call allocates its input, lowpass, per-level highpass and result arrays with _SharedArrays:

    • Linux: anonymous in-memory files (memfd_create), which workers open via /proc/<pid>/fd/<n>.
    • Elsewhere: temp files.

    Workers receive only paths and memory-map them. jobs=1 uses plain arrays and no pool.

  • Why memfd and not /dev/shm or multiprocessing.shared_memory: Docker's default /dev/shm is 64 MB. My first attempt with files there crashed workers (BrokenProcessPool) on real clips. memfd isn't limited by that mount.

  • Pool lifetime: one pool per call, used for both the forward and inverse passes. The result is returned as an ndarray view of its mapping, with no copy. File descriptors are released when the call ends, and the result's mapping when it is freed.

  • _transform uses functools.lru_cache (per process).

  • README notes that scripts calling magnify_motions with jobs > 1 need an if __name__ == "__main__": guard (standard for forkserver/spawn). Performance numbers and CHANGELOG updated.

Measurements (face.mp4, default settings)

before (fork) after (forkserver)
--jobs 2 time 55.5–56.2 s 59.5–60.2 s (+7%)
--jobs 2 peak (cgroup) 2.32 GiB 2.49 GiB (+7%)
--jobs 1 time 87.9 s 84.4 s
output, --jobs 1 vs --jobs 2 identical identical (0 of 282M values differ)

The cost comes from each worker's own Python heap and the fork server process. estimate_memory still matches (2.48 GiB estimated).

Tests

  • New: two concurrent calls from threads each return their own correct result (vs serial); the start method is not fork; memfds are released (only the returned result's mapping remains, and it goes when the result is freed).
  • CPU image: 113 passed, 3 skipped. GPU image on an RTX 4050: 137 passed, 1 skipped.

Workers were forked (unsafe while OpenCV, BLAS or FFT threads run: a
child can inherit a held lock) and read their arrays from a module-level
dict, so two magnify_motions calls at once in one process would clash.

Workers now start with forkserver (spawn where unavailable), with numpy,
scipy, dtcwt and cv2 preloaded in the fork server. Each call allocates
its arrays as anonymous in-memory files (memfd_create; temp files on
other platforms) and passes only their paths; workers memory-map them.
memfd avoids Docker's 64 MB /dev/shm limit. The result is returned as a
view of its mapping (no copy). No module-level state remains.

face.mp4, --jobs 2: ~60 s (was ~56 s), peak 2.5 GiB (was 2.3 GiB);
output is bit-identical across --jobs values.

Closes #69
@joeljose
joeljose merged commit d2687ca into main Sep 28, 2026
2 checks passed
@joeljose
joeljose deleted the fix/69-worker-safety branch September 28, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CPU workers: fork with threads alive and shared module-level state

1 participant