Improve timeout handling in CI

[vyasr]

Currently we use the bash timeout command extensively in our scripts to avoid jobs from eating up too much time. This choice is useful, but since timeout provides diagnostics in the form of an exit code, unless we handle that exit code there is no useful information presented in CI regarding why the commands failed. Since we typically run with set -e the command simply fails outright with no information.

I propose that we replace all timeout commands with a new rapids_timeout command that provides a useful diagnostic here. The implementation would be fairly simple I think:

rapids_timeout() {
  local old_opts="$-"
  set +e

  timeout $@
  timeout_exit=$?
  if [[ $? == 124 ]]; do
    echo "The command timed out"
  done

  # Restore options
  case $old_opts in
    *e*) set -e ;;
    *)   set +e ;;
  esac
}

[TomAugspurger]

rapidsai/cudf#22543 has an approach I'd recommend for all pytest-based tests:

1. pytest-timeout is the first level. It offers per-test timeouts, which I think are generally preferable to global job timeouts (we don't want to burn a bunch of CI time waiting for a 1-hour timeout for an job hanging on a single test). When tests do timeout, pytest-timeout will ensure we get nice output including the name of the hanging test and a backtrace
2. A bash timeout (or the proposed rapids_timeout) provides a timeout on an individual command. This is a nice fallback for pytest-timeout, in case its unable to interrupt the test
3. A job-level timeout, in the YAML, adds a third layer of protection against hanging jobs.

Finally, rapidsmpf has some tooling to print the stacktrace of all threads at the time the timeout fires: https://github.com/rapidsai/rapidsmpf/blob/a52b6cec65be6f4efbba46ceaf442cee2c7d2b13/ci/timeout_with_stack.py. But that requires some adjustments to the test environment (at a minimum, python and gdb. maybe other things) that might not work for all cases.

[vyasr]

Those are more extensive changes than what I was proposing, but I definitely agree with that idea.