Speed up process startup time by avoiding importing packages on version check

[fjetter]

importlib.metadata can check the version of a package without importing it. That makes our nanny and worker processes come up much faster, especially considering that the Nanny doesn't even have any usage for this (i.e. this also saves a bit on memory but that should be marginal in most cases).

Launching a LocalCluster with 10 workers on my machine is 2-3x faster with this change.

# Main
▶ time python -c "import distributed; distributed.LocalCluster(n_workers=10)"
10.98s user 3.07s system 147% cpu 9.516 total
▶ time python -c "import distributed; distributed.LocalCluster(n_workers=10)"
10.31s user 2.67s system 158% cpu 8.166 total

# This PR
▶ time python -c "import distributed; distributed.LocalCluster(n_workers=10)"
6.73s user 1.89s system 216% cpu 3.976 total
▶ time python -c "import distributed; distributed.LocalCluster(n_workers=10)"
6.52s user 1.76s system 252% cpu 3.275 total
▶ time python -c "import distributed; distributed.LocalCluster(n_workers=10)"
6.61s user 1.79s system 268% cpu 3.125 total

I hope this helps speed up our CI infrastructure and helps us avoid a couple of flaky tests which are related to Nanny processes startup failures

[mrocklin]

That's awesome.

…

On Wed, Apr 16, 2025 at 8:15 AM Florian Jetter ***@***.***> wrote: importlib.metadata can check the version of a package without importing it. That makes our nanny and worker processes come up *much* faster, especially considering that the Nanny doesn't even have any usage for this (i.e. this also saves a bit on memory but that should be marginal in *most* cases). Launching a LocalCluster with 10 workers on my machine is 2-3x faster with this change. # Main ▶ time python -c "import distributed; distributed.LocalCluster(n_workers=10)" 10.98s user 3.07s system 147% cpu 9.516 total ▶ time python -c "import distributed; distributed.LocalCluster(n_workers=10)" 10.31s user 2.67s system 158% cpu 8.166 total # This PR ▶ time python -c "import distributed; distributed.LocalCluster(n_workers=10)" 6.73s user 1.89s system 216% cpu 3.976 total ▶ time python -c "import distributed; distributed.LocalCluster(n_workers=10)" 6.52s user 1.76s system 252% cpu 3.275 total ▶ time python -c "import distributed; distributed.LocalCluster(n_workers=10)" 6.61s user 1.79s system 268% cpu 3.125 total I hope this helps speed up our CI infrastructure and helps us avoid a couple of flaky tests which are related to Nanny processes startup failures ------------------------------ You can view, comment on, or merge this pull request online at: #9048 Commit Summary - a1267e6 <a1267e6> Avoid importing critical packages on process startup - ff1a53d <ff1a53d> Fix test File Changes (2 files <https://github.com/dask/distributed/pull/9048/files>) - *M* distributed/tests/test_versions.py <https://github.com/dask/distributed/pull/9048/files#diff-4553904db2be9e5ecf54a3405fea89ada4466eabd642a4e9fa893a66f9ae495c> (11) - *M* distributed/versions.py <https://github.com/dask/distributed/pull/9048/files#diff-86e7e03bee50d99539a98a0acdf97d2ea0c177442d9d5e174630d8a6c5349a3f> (48) Patch Links: - https://github.com/dask/distributed/pull/9048.patch - https://github.com/dask/distributed/pull/9048.diff — Reply to this email directly, view it on GitHub <#9048>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTFGIN6YZFFACJBPZTT2ZZJVTAVCNFSM6AAAAAB3IIP2HOVHI2DSMVQWIX3LMV43ASLTON2WKOZSHE4TSNRSGY3DIMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jrbourbeau]

Nice find -- thanks @fjetter. This LGTM if tests are happy enough

[jrbourbeau]

Can get rid of these comments since we're not using a custom function for getting the version

[fjetter]

curiously, CI performance is much worse 🤔

[fjetter]

Well, importlib.metadata.version is much faster than an initial import but it is much slower compared to an import that already happened. Therefore, all of our test that do not use dedicated processes got slower... that should be fixed with a simple cache

Test runtime on my machine for test_scheduler.py

main: ~60s
PR w/out cache: ~120s
PR w/ cache: ~45s

Let's see if this also translate to CI

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    27 files  ±0      27 suites  ±0   11h 10m 36s ⏱️ + 1m 9s
 4 100 tests ±0   3 983 ✅ + 2    112 💤 ± 0  4 ❌  - 1  1 🔥  - 1 
51 401 runs  ±0  49 095 ✅ +11  2 297 💤  - 11  8 ❌ +1  1 🔥  - 1 

For more details on these failures and errors, see this check.

Results for commit 5e62cb5. ± Comparison against base commit 48fcf48.

♻️ This comment has been updated with latest results.

[fjetter]

Ok, there doesn't seem to be a noticeable difference for CI runtime. At least nothing I could pick out easily considering that the runtimes vary between 20 and 45min per job.