Merge dev into main

.gitignore

@@ -1,7 +1,9 @@
 qemu/
 virtualbox-iso/
 vmware-iso/
-
+packer_plugins/
+ansible/collections
+ansible/roles
 # Created by https://www.gitignore.io/api/packer
 
 ### Packer ###
@@ -12,8 +14,10 @@ packer_cache/
 *.box
 
 # End of https://www.gitignore.io/api/packer
-centos-7.3-x86_64
+
 .vault_pass
+.vault_password
 *.retry
 .venv
-centos-7.x-x86_64/
+.vscode
+.idea

CONTRIBUTING.md

@@ -0,0 +1,5 @@
+# Contributor information
+
+Images are built from each tag and published as
+[GitHub releases](https://github.com/usegalaxy-eu/vgcn/releases). All releases (including old ones) are also available
+[on this website](https://usegalaxy.eu/static/vgcn/).

README.md

@@ -1,102 +1,90 @@
-# Virtual Galaxy Compute Nodes (VGCN)
-
-This repo contains all of the components required to build the "Virtual Galaxy
-Compute Nodes" (VGCN) that make up the HTCondor cloud used in UseGalaxy.eu
-
-Pre-built images are available here: [https://usegalaxy.eu/static/vgcn/](https://usegalaxy.eu/static/vgcn/)
-
-## Features
-
-All in one image, a single image gets you:
-
-- HTCondor
-- Pulsar
-- NFS
-- CVMFS
-- Singularity
-- Docker
-
-Everything you need to run Galaxy jobs. You can structure this in various ways that meet your needs:
-
-- Like useGalaxy.eu: one VM as the condor master, another N as job runners
-- Like EU's remote clusters: one VM as the condor master + pulsar, another as NFS, and the rest as job runners
-- For BYOC: A single node that does condor master + pulsar + job execution
-
-## Development
-
-Development happens into the __dev__ branch, images are built from __main__ branch.
-
-## Changelog
-
-- 60:
-    - MODIFIED: moved to Rocky Linux 8
-- 50:
-    - MODIFIED: updated Pulsar to version 0.14.11
-    - MODIFIED: moved to kernel 5.x
-- 40:
-    - ADDED: Nvidia driver
-    - ADDED: Cuda toolkit
-    - ADDED: GPU accelerated Docker containers support
-    - MODIFIED: moved to CENTOS8
-    - MODIFIED: updated Pulsar to version 0.14.0
-    - MODIFIED: updated HTCondor to version 8.8
-- 32:
-    - MODIFIED: Pulsar toward Py3
-- 31:
-    - ADDED: Pulsar
-    - ADDED: Fonts for some jobs
-    - ADDED: `at` daemon
-    - MODIFIED: Additional 'internal' and 'external' targets allowing us to include signed SSH host keys.
-    - REMOVED: vault
-    - REMOVED: roced
-- 21: Initial Release
-
-## Makefile
-
-We include a makefile that should build the images, running `make` will inform you of the available targets:
-
-```console
-$ make
-General syntax: <template>/<flavor>[/boot]
-Detected builders:
-        qemu
-Base images:
-        centos-8.x-x86_64/base
-        rockylinux-8.x-x86_64/base
-        centos-7.x-x86_64/base
-
-Provisioning:
-        centos-8.x-x86_64/vgcn-bwcloud
-        centos-8.x-x86_64/jenkins
-        centos-8.x-x86_64/generic
-        rockylinux-8.x-x86_64/vgcn-bwcloud
-        rockylinux-8.x-x86_64/jenkins
-        rockylinux-8.x-x86_64/generic
-        centos-7.x-x86_64/vgcn-bwcloud
-        centos-7.x-x86_64/jenkins
-        centos-7.x-x86_64/generic
+# Virtual Galaxy Compute Nodes
+
+[Galaxy Europe](https://usegalaxy.eu/) runs jobs on compute nodes belonging
+to the
+[bwCloud](https://www.bw-cloud.org/)/[de.NBI-cloud](https://www.denbi.de/).
+These compute nodes are known as "Virtual Galaxy Compute Nodes" (VGCN).
+
+Virtual Galaxy Compute Nodes boot from VGCN images, which are built off this
+repository and made available as
+[GitHub releases](https://github.com/usegalaxy-eu/vgcn/releases) and on
+[this static site](https://usegalaxy.eu/static/vgcn/).
+
+## Virtual Galaxy Compute Node images
+
+Virtual Galaxy Compute Node images contain everything needed to run
+Galaxy jobs.
+
+- [Apptainer](https://apptainer.org/)
+- [CVMFS](https://cernvm.cern.ch/fs/)
+- [Docker](https://www.docker.com/)
+- [NFS](https://nfs.sourceforge.net/)
+- [Pulsar](https://github.com/galaxyproject/pulsar)
+- [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/)
+
+Each service can be further configured via
+[cloud-init](https://cloudinit.readthedocs.io/en/23.2.2/) scripts to meet 
+specific needs. A few setup ideas:
+
+- Like on usegalaxy.eu: one machine as the HTCondor master, the rest as job 
+  runners.
+- Like usegalaxy.eu's remote clusters: one machine as the HTCondor master and
+  pulsar server, another as NFS file server, the rest as job runners.
+- For "Bring Your own Compute" (BYOC) use cases: A single node that acts as
+  HTCondor master, Pulsar server, and job runner.
+
+**Important:** HTCondor has a new authentication mechanism, which uses
+passwords and tokens, generated automatically during startup. The easiest and
+least error-prone way to set this up is during
+[installation](https://htcondor.readthedocs.io/en/latest/getting-htcondor/admin-quick-start.html#administrative-quick-start-guide).
+
+```diff
+! This is why we removed HTCondor from public images.
 ```
 
-## Dependencies
+You can easily install it and set up your password by adding the `execute`
+installation line from the link above to your cloud-init script. There you can
+set your own password and host. Both should be identical on all machines you
+use, the host is the hostname or IP address of your central manager - usually
+the node that accepts Pulsar requests. An implementation in
+[Pulsar-Infrastructure-Playbook](https://github.com/usegalaxy-eu/pulsar-infrastructure-playbook)
+is in progress.
 
-We have listed the versions we use, but other versions may work.
+## Build instructions
 
-| Component                                      | Version    |
-|------------------------------------------------|------------|
-| [Packer](https://www.packer.io/downloads.html) | 1.0.4      |
-| Ansible                                        | >= 2.4.3.0 |
-| qemu                                           | 2.5.0      |
+Make sure the following packages are installed on your system.
 
-## Building This Yourself
+- [Packer](https://www.packer.io/downloads.html) >= 1.9.1, < 2
+- [QEMU](https://www.packer.io/downloads.html) >= 6.2, < 9
+- [Ansible](https://www.ansible.com/), see [requirements.txt](./requirements.txt)
 
-All of the images are designed to be as generic as possible so you can use them
-as-is. We will provide built images, but if you wish to build them yourself,
-you'll simply want to do:
+Run the `build.py` script to automatically template, provision, and assemble the image. 
 
+```shell
+python build.py <template> <provisioning>... <delivery> [options]
 ```
-make rockylinux-8.x-x86_64/vgcn-bwcloud
+
+**Example:**
+```shell
+python build.py rockylinux-10-latest-x86_64 workers internal cloud -q
 ```
 
-## Running It
+### Arguments
+
+- **`template`**: Selects the underlying operating system on which the image will be based. Supported templates can be found as `source.qemu.*` blocks in `templates/build.pkr.hcl` (e.g., `rockylinux-10-latest-x86_64`, `rockylinux-9-latest-x86_64`).
+- **`provisioning`**: One or more Ansible playbooks to run (e.g., `workers`, `internal`, `jenkins`). Note that the `generic` playbook is implicit and applied globally to all builds.
+- **`delivery`**: The delivery method or destination environment, which dictates the cloud-init and boot structure. This argument expects exactly one of the mutually exclusive choices: `cloud`, `kvm`, or `pxe` (use `no` to omit all). For local development this argument should default to `no`.
+
+### Options
+
+- `--openstack`: Automatically convert and upload the built image to your OpenStack tenant. Ensure your OpenStack RC file credentials are sourced.
+- `--conda-env <path>`: Specifies a conda environment path containing `packer` and `qemu-img` binaries.
+- `--publish <key_path>`: Secure copy the resulting raw image to the static hosting site and adjust permissions.
+- `--dry-run`: Output the generated Packer and shell commands without executing them.
+- `-q`, `--quiet`: Supress the spinner output during build.
+
+Once the images are built and converted, a `.raw` output file will be available in the root directory.
+
+## Running VGCN images
 
-Please see https://github.com/usegalaxy-eu/terraform/ for examples of how to launch and configure this.
+Please see [https://github.com/usegalaxy-eu/terraform/](https://github.com/usegalaxy-eu/terraform/) for examples of how to launch and configure this.