ensure `ci/run_*` scripts are actually used

[jameslamb]

Description

In many RAPIDS repos, you'll see a layout similar to this:

$ tree -L 2 ci/
├── build_cpp.sh
├── build_docs.sh
├── build_python.sh
├── build_wheel_cuml.sh
├── build_wheel_libcuml.sh
├── build_wheel.sh
├── check_style.sh
├── release
│   └── update-version.sh
├── run_clang_tidy.sh
├── run_ctests.sh
├── run_cuml_dask_pytests.sh
├── run_cuml_integration_pytests.sh
├── run_cuml_singlegpu_accel_pytests.sh
├── run_cuml_singlegpu_pytests.sh
├── test_cpp.sh
├── test_notebooks.sh
├── test_python_common.sh
├── test_python_cuml_accel_upstream.sh
├── test_python_dask.sh
├── test_python_integration.sh
├── test_python_scikit_learn_tests.sh
├── test_python_singlegpu.sh
├── test_wheel_dask.sh
├── test_wheel_integrations.sh
├── test_wheel.sh
└── validate_wheel.sh

Those run_* scripts are intended to be limited to selecting and running tests.

The test-* scripts are intended to do other work like installing dependencies, downloading datasets, uploading coverage artifacts, etc. and should re-use the run_* scripts for actual test-running.

We've been inconsistent about this across RAPIDS... this issue tracks the work of auditing the repos and enforcing this convention.

Benefits of this work

Ensures the run_* scripts stay up to date with changes to the repos.

Those scripts are useful for other development using RAPIDS repos, like:

• testing in devcontainers (e.g. Add test commands to rapids-build-utils devcontainers#630)
• alternative builds of RAPIDS (like DLFW builds)

Acceptance Criteria

• every RAPIDS repos follows this run_* vs. test_* convention, and test_* scripts invoke run_* scripts
• run_* scripts do not use anything from https://github.com/rapidsai/gha-tools (not guaranteed to be present in all test environments)

Approach

Look through the repos one-by-one and enforce this.

Notes

Would be a good candidate to pair with #249

[vyasr]

IMHO while this is a worthy goal, unless it is enforced we'll be back in this situation very soon because library devs modify these scripts as well as build engineers and they won't be as attuned to the conventions. If we want to write some small CI verification of this (e.g. a custom pre-commit hook to grep for ci/test_(.*).sh and match against ci/run_\1.sh if the file exists) then I would be supportive, but without adding verification I don't think this work is worth the work.

[jameslamb]

Yeah fair, it would be great if we could come up with some way to automatically prevent these things from creeping back in. Automated checks are better than convention.