Rapid deployment to Github, PyPI and anaconda.org

[tovrstra]

I've made an example project of how to build and deploy packages with Travis (Linux and OSX) and AppVeyor (Windows).

https://github.com/theochem/python-cython-ci-example

To make a new release for this package (source tar.gz on Github, PyPI and compiled package on anaconda.org), I only need to push a version tag (like 1.4.1) to the github repo, thanks to the deployment features of Travis-CI and some PowerShell scripting on AppVeyor. Also the documentation is rebuilt on Travis-CI when such a tag is pushed. This makes it super easy to publish new releases.

This rapid deployment could help us when splitting up HORTON in smaller packages without ending up in dependency hell. I'm mainly thinking of two relevant use cases:

(1) End users can simply install Conda as a standardized Python environment, which works on all platforms. Installing miniconda or anaconda is trivial. E.g. for Miniconda with Python 3, these are the steps:

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod +x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh -b   # -b is for non-interactive install
source miniconda3/bin/activate           # can be put in .bashrc

Once you have conda installed, installing the demo project just takes:

conda install -c tovrstra pycydemo

The -c tovrstra option selects my channel: https://anaconda.org/tovrstra. This command will install all dependencies as well. We can even put our compilations of LibXC and LibInt in such a channel, to avoid issues like #262.

When a user is interested in a package that only has Python dependencies, then pip can be used as well. For example, if your computer has Python and pip installed, one just has to run:

pip install numpy Cython
pip install pycydemo --user

Pip has some limitations, which make it not suitable for all purposes:

1. It can only install Python packages, which is a problem when you have non-python dependencies.
2. It does not make a distinction between stable, alpha and beta releases, which is possible on Conda and Github
3. Uploading something to PyPI is an irreversible operation, no mistakes allowed.
4. If you have import numpy or import Cython in setup.py, pip cannot install your package from scratch, e.g. in a new venv. There is an ugly workaround for numpy: https://stackoverflow.com/questions/19919905/how-to-bootstrap-numpy-installation-in-setup-py
5. Wheels (pip's binary packages) for a broad class of Linux systems are difficult to make. The solution can be found here: https://github.com/pypa/manylinux

(2) Developers can easily install a development environment, making use of the conda environments. My first idea was to work with so-called environment.yml files, but these do not allow easy upgrading when the development version moves on to newer versions of dependencies. It would be better to add a script that takes the dependencies out of conda.recipe/meta.yaml and that installs/upgrades/downgrades these in the current Conda environment. On top of these, one may clone git repos for a few dependencies, in case one wants to develop against a development version of the dependency. Installing these into a conda environment (without root) is relatively easy.

[tovrstra]

These are the steps to implement all of the rapid deployment goodies in a Python project:

• Fix Remove context.py #257 if needed, includes switching to setuptools and clean handling of data files.

• Derive the package version from the git tag in a PEP 440 compatible way. See https://github.com/theochem/python-cython-ci-example/blob/master/setup.py#L16-L49:

  • Get rid of scripts to upgrade the version of the package.
  • In your package __init__.py add a line from .version import __version__.
  • Add package/version.py to your .gitignore.
  • Read version string in doc/conf.py
  • Remove detailed URLs with version numbers from the docs. We're going to use conda (and pip) anyways.

• Remove the Development Status classifier from setup.py because this information can be inferred from the version tag.

• Specify the correct Python version dependency in setup.py. E.g. use the 'Programming Language :: Python :: 2.7' classifier and add python_requires='>=2.7' to the setup call.

• Add runtime dependencies to the setup call, e.g. install_requires=['numpy', 'nose', 'cython']

• Add the correct license to the classifiers, e.g. ''License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)'

• Add zip_safe=False to your setup() call. This avoids installation into eggs, which prevents nosetests from finding the tests of an installed package. See Nosetests cannot find tests inside eggs nose-devs/nose#1057.

• Rename COPYING to LICENSE.txt

• Add a tools/conda.recipe/meta.yml file. https://github.com/theochem/python-cython-ci-example/blob/master/tools/conda.recipe/meta.yaml

• Use the following .travis.yml file and update it: https://github.com/theochem/python-cython-ci-example/blob/master/.travis.yml

  • Update env vars.
  • Update encrypted tokens and passwords (see below).
  • Enable Travis-CI for your repo if not done yet.

• Use the following .appveyor.yml file and update it: https://github.com/theochem/python-cython-ci-example/blob/master/.appveyor.yml (only needed to compile extensions, could be useful just for testing pure python projects to)

  • Update env vars.
  • Update anaconda token.
  • Enable AppVeyor for your repo.
  • Also add the file tools/appveyor/run_with_env.cmd to your repo.

• Some links

  • Travis-CI: https://travis-ci.org/
  • AppVeyor: https://www.appveyor.com/
  • http://anaconda.org
  • http://pypi.python.org
  • General deployment docs on Travis: https://docs.travis-ci.com/user/deployment/
  • Github releases with Travis: https://docs.travis-ci.com/user/deployment/releases/
  • Documentation updates to gh-pages with Travis: https://docs.travis-ci.com/user/deployment/pages/
  • PyPI releases with Travis: https://docs.travis-ci.com/user/deployment/pypi/
  • Custom deployment scripts Travis-CI: https://docs.travis-ci.com/user/deployment/script/
  • Encrypting stuff in .travis.yml: https://docs.travis-ci.com/user/encryption-keys/
  • Creating github personal access tokens: https://github.com/settings/tokens/new
  • Anaconda client (creating and using tokens): https://docs.continuum.io/anaconda-cloud/commandreference
  • AppVeyor encryption tool: https://ci.appveyor.com/tools/encrypt

  Make sure you always encrypt passwords and tokens in .travis.yml and .appveyor.yml!!!!

[tovrstra]

With the above changes to your project, making a new release takes the following steps:

1. Update the release history. Build documentation locally and make sure it looks good.

2. Commit the final changes to master and push to github (or make a PR).

3. Wait for the CI tests to pass. Check if the README looks OK on Github, etc. If needed, fix things and repeat step 2.

4. Make a git version tag: git tag <some_new_version> Follow the semantic versioning guidelines: http://semver.org. Always increase the version. Do not prefix the tag with a v: use 1.0.0 instead of v1.0.0. Add an a<counter> suffix to the version for alpha releases (features and API still changing). Add an b<counter> suffix to the version for beta releases (features and API stable, bug fixes only). Some examples in chronological order:

   1.0.0a1
   1.0.0a2
   1.0.0a3
   1.0.0b1
   1.0.0b2
   1.0.0
   1.0.1a1
   1.0.1
   1.1.0
   1.1.1
   2.0.0a1
   2.0.0a2
   

5. Push the tag to github: git push origin master --tags and wait for all the magic to happen.

[loriab]

Hi @matt-chan, there was some talk at PyQC this year that we should shut down the PyQC channel on anaconda.org (https://anaconda.org/pyqc/repo). In particular, PySCF has its own channel, but the pyqc pyscf channel version was showing up in google instead (who googles anaconda.org?). How does that interact with your Horton distribution scheme? Paul Ayers suggested to consult you. No rush, just saw the reminder tab I've kept open.

[matt-chan]

Hi @loriab, yep, we've moved almost everything over to the "theochem" channel. We used to get libint and libxc from PyQC, but we simplified things for our users so they could do a 1-step install. I'm not sure if we should keep it or not, since technically we should have a common deps. Maybe push for inclusion of common dependencies to conda-forge?

And it turns out I'm guilty of googling for those anaconda packages =S, although I've never searched Psi4.