"new" data sets are missing in the leader board

[gschramm]

Seems that the leaderboard only contains the initial 3 data sets.

If possible, they (e.g. the mMR ACR data sets) should be added here:

PETRIC/petric.py

Line 249 in 50f28f0

(SRCDIR / "Siemens_Vision600_thorax", OUTDIR / "Vision600_thorax",

[KrisThielemans]

As we're getting new data ready, we will likely only automatically run 1 dataset from each scanner, to avoid overloading the STFC computing once people start to submit a lot of jobs (hopefully!).

You can run them locally by adjusting your own petric. Or maybe there's a smarter option...

[gschramm]

makes sense. I guess in principle there could be a decision based on the hostname in petric.py which data sets to run.
(if the hostnames on STFC follow a pattern and it is safe to make that pattern public).

otherwise the decision could be made based on a env. variable that users can set to run all data sets locally.

[KrisThielemans]

could be doable I guess. @casperdcl, What do you think? 5 min job?

[casperdcl]

you mean truncating the dataset list length in #91 using int(os.getenv("PETRIC_MAX_DATASETS", "3")) or similar?

^{P.S. my last "5 min job" shaved days off my life}

[KrisThielemans]

let's leave it then. Participants can make this change themselves.