updating conda env on tallgrass

[gzt5142]

Hello,

The Open On Demand (OOD) service lets us spin up a jupyter server on a tallgrass node. It's an easy way to get hub-like interface and HPC computational power without any user training or configuration. There are some details to be aware of on this system. I offer them here to be sure that they are captured somewhere before I migrate off this project:

• The OOD system relies on a loadable 'module' on tallgrass.
  • This module is not limited to OOD function (you can use it any time you log into tallgrass or denali if you want.
  • Use module avail to see all the modules you have access to
  • hytest should be one of the options
  • module load hytest should configure your shell to use the right conda environment.
  • I almost never do it this way (see below for what I recommend). But I mention it here because this is how the OOD server sets the conda environment.
  • The module is defined/configured here: /caldera/projects/usgs/water/impd/hytest/modules/hytest
• The hytest module is built on a conda installation in /caldera/projects/usgs/water/impd/hytest/
• If you want to update the hytest conda environment, you need to activate that specific environment and update/install there.
• Here's how:
  • Log into tallgrass --
    • Note that if you have your own jupyter/conda config for your account, it may conflict with what we're going to do here.
    • Recommend that you disable that and re-login.
  • export PATH=/caldera/projects/usgs/water/impd/hytest/conda/bin:$PATH
  • source /caldera/projects/usgs/water/impd/hytest/conda/etc/profile.d/conda.sh
  • conda activate hytest
  • conda env list
    • This shows all the conda envs installed under the hytest space on caldera
  • conda update --dry-run --update-all
    • The --dry-run switch causes conda to just report what it would do.
    • This is so you can double-check that it is going to do something you agree with.
  • conda update --update-all
    • or conda update packagename if you only want to update a specific package.
• You can also create new environments if you want. At the time of this writing, a gdp environment is installed there as well. Once you've got this conda active, you can create and modify these environments per normal conda usage.

Note that you'll be modifying the common environment and its packages. This will affect all users of this environment. Also, updating a conda environment while processes are actively using it can yield odd results. Recommend that you make such upgrades at periods of low usage (evenings, early-mornings, weekends).

[gzt5142]

A follow-up note on that "extra" environment, gdp.

To get that environment as an option in your jupyter server, you need to add it as a kernel specification. There are a few ways to do that. The least hacky is to do this:

• export PATH=/caldera/projects/usgs/water/impd/hytest/conda/bin:$PATH
• source /caldera/projects/usgs/water/impd/hytest/conda/etc/profile.d/conda.sh
• conda activate gdp
• python -m ipykernel install --user --name gdp --display-name "GDP"

Tweak that last command to reflect the name and display name you like.

This will install a kernelspec file in your home folder, which the OOD server will consult when it decides which environments its jupyter server should include. This is a per-user configuration (because of the --user switch).

After you do this, the next time you run OOD, you should have access to this additional environment.

This mechanism will let you create a 'beta' or 'pre-release' HyTEST environment (substitute 'hytest-beta' for 'gdp' in all of the above). So doing will let you test-drive updated environment to satisfy yourself that it doesn't break anything. Then you can update the 'production' hytest environment.