feat!: introduce scoped dependencies

[project-defiant]

Rationale

Full gentropy stack of dependencies is not needed to run most of the steps except L2G. The same idea is behind fine-mapping & LD access that requires hail. Current gentropy build on the cluster takes ~5minutes due to excessive amount of dependencies.

This PR attempts to isolate steps that require L2G dependencies and Hail dependencies into 2 separate installation scopes.

The benefit of this addition should be in simplified dependency resolution (hence startup time) for variant, coloc, credible_set cluster, while retaining L2G dependencies for L2G steps on a separate cluster.

Summary

Carves gentropy's heavy ML and hail stacks behind opt-in extras. After this PR pip install gentropy ships a Spark-only core; users opt into gentropy[hail], gentropy[l2g],
or gentropy[all] based on what steps they run. Docker, Dataproc, and CI continue to install the fat image ([all]), so production behaviour is unchanged.

BREAKING CHANGE for end users: plain pip install gentropy no longer pulls hail, xgboost, scikit-learn, skops, shap, matplotlib, wandb, or huggingface-hub.
A step that needs one of those raises ImportError with the exact pip command to fix it.

Install matrix (now in README + docs/installation.md)

Use case	Command
Spark-only steps	pip install gentropy
+ hail-backed datasources (gnomAD LD, FinnGen finemapping, PanUKBB LD, susie)	pip install gentropy[hail]
+ L2G training, prediction, HF Hub publishing	pip install gentropy[l2g]
Full pipeline	pip install gentropy[all]

What changed

Packaging (pyproject.toml):

• Move hail, xgboost{,-cpu}, scikit-learn, skops, shap, matplotlib, wandb, huggingface-hub from [dependencies] to [project.optional-dependencies] under
• [dependency-groups].test self-references gentropy[all] so the test env stays fat.
• Register hail and l2g pytest markers; partition by them.

Source-side guards:

• New shared helper src/gentropy/common/imports.py with install_hint(extra) and optional_imports(extra) context manager — every guard funnels missing-import errors through
  one voice ("pip install gentropy[<extra>]").
• Module-level guards at the top of intrinsically extra-bound files: datasource/{gnomad/ld,gnomad/variants,finngen/finemapping,pan_ukbb_ld/ld}.py (hail) and
  method/l2g/{model,trainer}.py, dataset/l2g_prediction.py (l2g).
• Lazy in-function guards where heavy imports were already deferred: common/session.py, common/genomic_region.py, plus gentropy.l2g (top-level step module). The four
  non-training L2G step classes — LocusToGeneFeatureMatrixStep, LocusToGeneTrainTestSplitStep, LocusToGeneEvidenceStep, LocusToGeneAssociationsStep — remain importable and
  runnable on a core-only install; only LocusToGeneStep.run_train/run_predict pull the heavy stack.
• SessionConfig.hail_home flips from eagerly-resolved os.path.dirname(hail.__file__) to str | None = None. Session._setup_hail_config already lazy-resolved it, so the
  eager default was just leaking import hail into the CLI boot path.
• utils/spark.py::get_spark_testing_conf gains a with_hail: bool = False flag; the testing Spark conf is hail-free by default and only mixes in the Kryo registrator / hail
  jar paths when explicitly opted in.

Test infrastructure:

• conftest.py spark fixture introspects collected items: builds a hail-enabled session iff at least one collected test carries the hail marker. Keeps pytest -m "not hail" honestly hail-free.
• Existing hail- and L2G-using tests gain file-scope pytestmark = pytest.mark.hail / .l2g and pytest.importorskip so they're safe to collect on a core-only install.
• New tests/gentropy/no_spark/test_hail_optional.py and test_l2g_optional.py pin the public contract — install hint wording, the specific pip command substring, that the
  lazy step module imports cleanly with the L2G stack stubbed out via a sys.meta_path finder. New test_config.py cases pin hail_home default + import-without-hail.

Build & CI:

• Dockerfile: both uv sync invocations gain --all-extras so the production image still installs everything.
• utils/install_dependencies_on_cluster.sh: Dataproc init switches from gentropy[hail] to gentropy[all].
• .github/workflows/pr.yaml: install step now uv sync --all-groups --all-extras.
• Makefile: make test partitions runs into hail vs non-hail and no_shared_spark vs shared — required because the spark fixture needs to know the partition's
  hail-marker state at fixture-build time. Also (small fix) the uv sync step uses --all-groups so the docs test can collect (mkdocs lives in [dependency-groups].docs).
• tool.deptry: xgboost-cpu keeps the DEP002 ignore (no module of its own — platform-conditional drop-in); the xgboost ignore is dropped since it's now correctly declared
  via the extra.

Test plan

• make test passes locally with --all-extras --all-groups.
• tests/gentropy/no_spark/test_hail_optional.py simulates a hail-free env via a custom sys.meta_path finder and asserts pip install gentropy[hail] appears in the
  raised ImportError.
• tests/gentropy/no_spark/test_l2g_optional.py does the same for the L2G stack, and additionally asserts gentropy.l2g imports cleanly without the heavy deps
  (lazy-imports work).
• tests/gentropy/test_config.py::TestSessionConfig asserts hail_home defaults to None, is typed str | None, and is importable with hail blocked.

[ireneisdoomed]

Thank you for the initiative!!! My summary:

1. The PR creates different submodule for Gentropy: l2g, hail, all. pip install gentropy will only install the bare spark modules
2. Every module that imports hail or l2g dependencies adds some guardrails to check whether the dependency is in the environment. It raises a readable message otherwise
3. All tests that use the optional dependencies are skipped if the dependency is not found
4. Dataproc/Docker/CI install gentropy[all] so the behaviour in our pipelines shouldn't change

I have one comment that is not a deal-breaker: IMO, L2G prediction should be available in the bare Gentropy. This means that xgboost is installed as part of the main package. This is not currently straightforward to accomplish because L2GPrediction is bound to L2GModel, which imports wandb and huggingface. What do you think?

[ireneisdoomed]

Interesting pattern 😵‍💫

[ireneisdoomed]

Could we import thin function in gentropy/common/__init__.py?
Looks a bit too repetitive to me. We'd go from from gentropy.common.imports import optional_imports to from gentropy.common import optional_imports

[project-defiant]

totally agree!

[ireneisdoomed]

Could we use the context manager here?

[ireneisdoomed]

This could be moved inside explain, it's the only function that uses it.

[ireneisdoomed]

LocusToGeneModel is only used for type checking. I think we should be able to move it under a TYPE_CHECKING block

If that's the case, we could import L2GPrediction without the optional dependencies

[project-defiant]

@ireneisdoomed, I can see usecases that people actually do not need to run L2G training, but they actually still want to run the feature matrix, predictions and evidence with our model. So maybe the l2g scope is too wide. The only thing that I do not like with this approach is that we still need to install the xgboost with default gentropy, as the package is quiet large, but we might not be able to avoid it.

[ireneisdoomed]

@ireneisdoomed, I can see usecases that people actually do not need to run L2G training, but they actually still want to run the feature matrix, predictions and evidence with our model. So maybe the l2g scope is too wide. The only thing that I do not like with this approach is that we still need to install the xgboost with default gentropy, as the package is quiet large, but we might not be able to avoid it.

Yes. It makes sense to me, but I agree it is a product decision