Setting up a Development Machine Virtual Machine

I've noticed these containers don't tend to work on many types of machines. For instance, my M3 Macbook doesn't play well, in part due to the newer ARM architecture, and in part due to the lack of OS support for certain dependencies in the Dockerfiles.

Besides the points made in other documentation pages on how to set up tooling, I wanted to go through the steps of setting up a VM to run the E2E tests and models in general.

In any case, I'm struggling to build these files locally.

For this reason I personally prefer working on a virtual machine. The big 3 cloud providers—AWS, GCP and Azure—offer offer pretty nice free tiers with plenty of credit. Remember to get in the habit of shutting down your VM after using it.

Once you have a new VM provisioned. Let's say it's a Ubuntu/Debian machine. Start by adding docker and docker compose to your VM.

sudo apt update
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

Now we want to clone the repository. Let's clone into your home directory since it's just a bare VM

cd
sudo apt install -y git

It appears that sometimes git is already added by default (e.g., on GCP).

Before we can clone you need to set up SSH authentication. I won't go into the details, but they can be found here

git clone git@github.com:LAAC-LSCP/analysis-service.git

And now you should be ready for development.

Connect VSCode to your Development Machine

Things are a little more streamlined if you connect your IDE as well. I use VSCode together with the Remote - SSH extension.

Add this to your local ~/.ssh/config, remember to add

Host my-elsi-dev-vm
    HostName <your-vm-external-ip>
    User <your-remote-username (use `whoami` to see)>
    IdentityFile <~/.ssh/your-gcp-private-key (e.g., ~/.ssh/id_ed25519)>

Note that this certain rules can overshadow this rule, e.g., if you have a Host * block further up above this one.

You also need to add the ssh public key to the VM itself under the authorized_keys file.

mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "MY PUBLIC KEY" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

In the cloud it can also typically be configured with the VM via a user interface.

Back in VSCode, press the remote explorer icon on the sidebar (left sidebar by default). Click the arrow next to your vm to connect. Now you can cook.

Setting up Logging and Monitoring

We could open port 3000 and 19999 to ingress traffic, but it's not secure. Grafana has password protection, but NetData has nothing by default.

We will use an SSH tunnel to connect to port 19999 (NetData for monitoring) and 3000 (Grafana for logging). In the background or a separate terminal run

ssh -L 19999:localhost:19999 my-elsi-dev-vm

And likewise

ssh -L 3000:localhost:3000 my-elsi-dev-vm

Now go to http://localhost:19999/ and click on (skip and use the dashboard anonymously) for NetData. Go to http://localhost:3000/ and use username "admin" and the GRAFANA_PASSWORD in your .env.test file to login for Grafana.

Install the Rest

Install pipx

sudo apt install -y pipx
pipx ensurepath

Python as well. The easiest way turned out to just use uv's Python installer.

curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.13

And the CI tools

pipx install tox
pipx install poetry
pipx install black
pipx install isort
pipx install mypy
pipx install flake8
pipx install autoflake

pipx ensurepath

source ~/.bashrc

Systemd to Auto-Start the Containers on Boot

You might want to test out the auto-start behavior of the production machine. You can mimick it easily.

Save the below to /etc/systemd/system/echoservice.service. It can be useful if you don't want to have to start it up each time.

[Unit]
Description=Echoservice Docker Compose
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/path/to/elsi/repo (where the dockerfile is)
ExecStart=/usr/bin/docker compose --env-file .env.test -f docker-compose.dev.yml up -d
ExecStop=/usr/bin/docker compose --env-file .env.test -f docker-compose.dev.yml down

[Install]
WantedBy=multi-user.target

You could even configure it to wipe the outputs folder on startup

ExecStartPre=/bin/rm -rf /path/to/outputs/ (outputs are found in .../analysis-service/tests/dummy_datasets/*/outputs)

Future -- Watching Changes

At present I run docker compose with the --build flag when I make changes, because we don't actually volume map our code into it. For development containers this would be ideal. If this is a real pain point, consider making a PR to add some development-only DockerFiles that do this volume mapping.

Future -- Automating This

This is the kind of approach that guarantees what's called a "snowflake server"—not good. preseed + cloud-init config script might be a good mix to automate this stuff for when you want to provision a new machine with all this configuration.