core: harden ImpressManager lifecycle, add GPU discovery, filterable logging

[drawadiagram]

Splits the impress core framework changes out of #59 so they can be reviewed on their own. First of four; the other three depend on this one (both workflows import find_gpus from impress, and both set IMPRESS_SESSION_DIR).

ImpressManager

• Session dirs move off scratch. asyncflow's session dir is now created under IMPRESS_SESSION_DIR (default: the system temp dir) rather than cwd, which exhausted the scratch inode quota over many HPC runs.
• Engine shutdown moved into a finally. A pipeline raising mid-run used to leak the WorkflowEngine and its backend. This is also where the per-example await manager.flow.shutdown() went — answering @AymenFJA's question on Impress fixes #59 about its removal from run_protein_binding.py; the examples no longer have to remember it.
• submit_new_pipelines() before start() now raises a clear RuntimeError instead of an AttributeError on a missing self.flow.
• Failed pipelines are reported as failures. Each pipeline's future is carried through cleanup, so a pipeline that raised is logged via the new logger.pipeline_failed() rather than silently logged as completed.
• activity_summary() stops reporting 0. The buffered count is captured before the buffer is cleared.

ImpressLogger

• min_level filtering on every level method and on pipeline_log(); defaults to DEBUG, so nothing is filtered unless opted in.
• New pipeline_failed(name, exc).
• ⚠️ error/critical now go to output_stream instead of stderr. Deliberate — under Dragon the job's stdout is the log anyone actually reads, and errors were being split into a stream nobody checks. Flagging it because it is a behaviour change for anything parsing stderr.

Other

• New impress.gpu.find_gpus(), re-exported from the package root: CUDA_VISIBLE_DEVICES, else nvidia-smi, else [].
• ImpressBasePipeline.finalize() is no longer abstract — pipelines needing no cleanup shouldn't have to define an empty override.
• Callable imported from collections.abc.
• radical-asyncflow>=0.4.0, the only pyproject.toml change. WorkflowEngine.create(work_dir=...) does not exist in 0.3.1 or earlier, so the session-dir change above needs a floor. Verified by unpacking both sdists.

Python version support

Python 3.9/3.10 support is retained. #59 dropped py39/py310 from the CI matrix; @AymenFJA objected and @mgoliyad agreed to put them back, so that change is reverted here and this PR does not touch .github/workflows/tests.yml or tox.ini at all.

Keeping 3.9 means keeping except asyncio.TimeoutError in test_pipeline_management.py rather than the shorter except TimeoutError: asyncio.TimeoutError only became an alias of the builtin in 3.11, so on 3.9/3.10 the short spelling silently fails to catch the timeout. Nothing else in src/ needs 3.10+ — the generics are PEP 585, which 3.9 evaluates fine — and radical-asyncflow itself declares requires-python >=3.9.

Tests

Adds test_logger.py, test_pipeline_base.py, test_pipeline_setup.py, plus regression tests for the two fixes above: test_start_exception_still_shuts_down_engine and test_submit_before_start_raises. 94 unit tests pass; ruff check and ruff format --check clean.

Checked against the existing examples

examples/discontinuous_scaffolds/, dummy.py and dummy_adaptive.py aren't touched by this PR, so I verified it doesn't break them. The one real risk was double shutdown — 8 entry points on main call await manager.flow.shutdown() after start(), which now runs twice. Exercised end-to-end with a real ImpressManager + LocalExecutionBackend: the second call returns cleanly and the process exits 0. Not exercisable for rhapsody's DragonExecutionBackend locally; PRs 2 and 3 delete those now-redundant calls anyway.

Unrelated pre-existing breakage noticed while checking: examples/dummy.py and dummy_adaptive.py both from radical.asyncflow import ConcurrentExecutionBackend, which moved to rhapsody.backends — they fail at import on main today. Not touched here; worth its own issue.

Replaces part of #59.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Lo8DwSbyvdWZRkkkka6gA2

[AymenFJA]

What about a multi-node setup? How is find_gpus being spawned?

GPU devices should be detected by the backend and then made visible to the application (the IMPRESS pipeline) through backend tasks.

We can keep find_gpus as a helper utility, but in general, having IMPRESS discover GPUs itself is a layer violation. IMPRESS should not need to know how or where GPUs are discovered; that responsibility belongs to the backend/infrastructure layer i.e: asyncflow-->rhapsody.

[AymenFJA]

What is the reason behind this architectural change of making WorkflowEngine an internal component?

[AymenFJA]

@drawadiagram I added 2 comments above. Thanks