Archlinux on Azure — Squashfs + vTPM Architecture

A hardened Arch Linux image for Azure Trusted Launch VMs with:

• Squashfs fallback root — minimal boot image, no Packer required
• vTPM disk encryption — LUKS2 sealed to vTPM (PCR 11), replaces luks_unlocker
• Auto-provisioning — raw data disk is encrypted and installed on first boot
• systemd-boot + UKI + Secure Boot
• BTRFS with subvolumes + zstd compression
• cloud-init, AppArmor, linux-hardened
• No walinuxagent

Architecture

Boot VHD (~1.5GB)                     Data Disk (attached in Azure)
┌─────────────────────┐               ┌──────────────────────────┐
│ ESP: UKI + loader   │               │ LUKS2 (vTPM-sealed)      │
│ Root: squashfs.sfs  │  ──detects──▶ │  └─ BTRFS                │
│       /var/log      │               │     ├─ @      (root)     │
│       /var/cache    │               │     ├─ @home             │
└─────────────────────┘               │     ├─ @log              │
                                      │     ├─ @cache            │
                                      │     └─ @srv              │
                                      └──────────────────────────┘

Boot Flow

1. UKI boots → initramfs runs arch-root-setup.service (setup-root)
2. Data disk with LUKS found → vTPM unseals key → mount BTRFS subvols → boot into data disk root
3. Raw/unformatted data disk found → boot into squashfs + overlayfs → provision-data-disk.service runs:
   • LUKS2 format + systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=11
   • BTRFS with subvolumes
   • System install from squashfs + pacman -Syu
   • Regenerate UKI for data-disk root
   • Reboot
4. No data disk → boot into squashfs maintenance environment (SSH available via cloud-init)

Build (replaces Packer)

Build directly on an Arch Linux host — no QEMU/Packer needed:

# Install build dependencies
pacman -S arch-install-scripts squashfs-tools dosfstools btrfs-progs \
    qemu-img mkinitcpio systemd-ukify sbsigntools azure-cli

# Build the VHD image
sudo KEY_VAULT_NAME=<your-key-vault-name> ./build.sh

# Build and upload to Azure
sudo AZURE_STORAGE_ACCOUNT=mystorageacct \
     AZURE_STORAGE_CONTAINER=vhds \
  KEY_VAULT_NAME=<your-key-vault-name> \
     ./build.sh --upload

Output: out/azure-archlinux.vhd

Then create the Azure managed image:

az image create --resource-group <rg> --name azure-archlinux \
  --os-type Linux --hyper-v-generation V2 \
  --source https://<storage>.blob.core.windows.net/vhds/azure-archlinux.vhd

Deployment

Deploy with the existing Bicep templates:

az deployment group create \
  --resource-group <rg> \
  --template-file bicep/main.bicep \
  --parameters baseName=myvm pubkeydata="$(cat ~/.ssh/id_rsa.pub)" \
               vm_admin_name=$(whoami) local_public_ip=$(curl -s ifconfig.me)

Post-Deployment

1. Set VM tag and Key Vault permissions before first provisioning reboot:

• VM tag: KeyVaultName=<your-key-vault-name>
• Grant VM managed identity secrets/get on that Key Vault

2. Attach a data disk (Premium SSD v2 recommended) to the VM
3. Reboot — the first-boot service will automatically:
   • Encrypt the data disk with LUKS2 + vTPM
   • Install the full system

• Fetch Secure Boot private key from Key Vault into encrypted root
• Reboot into the data disk root

4. Secure Boot — after first successful boot, enable Secure Boot and run:

   /usr/local/sbin/setup-secureboot.sh

Secure Boot Private Key Flow

• Build time: key is generated, used for signing UKI/systemd-boot, uploaded to Key Vault, then shredded from the build workspace
• Build time: a random 64-character root password is generated and uploaded to Key Vault as root-login-password (override with ROOT_PASSWORD_SECRET_NAME)
• Boot disk image: only the public certificate is included
• Provisioning: private key is fetched via managed identity and stored on encrypted data disk only
• Provisioning: root password is fetched from Key Vault and applied to the data-disk root
• Runtime updates: /usr/local/bin/secure-boot-resign uses local encrypted key and can recover it from Key Vault if missing

Recovery / Maintenance

If the data disk is removed or fails, the VM boots into the squashfs maintenance environment with password-based root login on console only (sshd is masked).

Public and NetBird-only HTTPS

The caddy role now owns /etc/caddy/Caddyfile (with a timestamped backup on replacement). It replaces the legacy mixed public/private routes rather than continuing to patch repeated IP allowlists. Site fragments live under /etc/caddy/sites/public/ and /etc/caddy/sites/internal/; NetBird and Open WebUI manage their own validated fragments. Keep these directories free of obsolete or unreviewed site files.

Only these application endpoints are public, with Let's Encrypt certificates:

• https://netbird.example.com — NetBird dashboard/API/relay.
• https://example.com/vault/ — Vaultwarden, excluding /vault/admin*.

All other public main-domain paths return 404; Vaultwarden admin returns 403. On the WAN address, HTTP port 80 redirects only the two public hostnames. NetBird's STUN/WireGuard transport ports remain separate from the HTTPS application policy.

All sites use standard HTTP 80 / HTTPS 443. Public sites bind only to caddy_public_bind_addresses (normally the WAN NIC's local addresses behind Azure NAT, optionally also an existing management-VPN address), and private sites bind only to caddy_internal_bind_addresses (NetBird IPv4/IPv6 addresses). No wildcard web sockets mix the two groups. Internal HTTP redirects to HTTPS on the same hostname, with no explicit port needed.

nftables drops TCP 80/443 and UDP 443 to the NetBird addresses unless packets arrive on wt0, before conntrack/service accepts. Binding an IP alone is not an ingress-interface restriction on Linux; this guard also blocks access through LAN/WAN or the old wg0. Private DNS must resolve internal hostnames to the configured NetBird addresses, not the public IP.

The networking role enables IPv4/IPv6 ip_nonlocal_bind in /etc/sysctl.d/60-caddy-bind.conf, allowing bound sockets to start before wt0 exists. These are host-wide settings. They avoid a self-hosted bootstrap cycle where Caddy would wait for a VPN whose control plane itself needs public Caddy. The firewall still enforces ingress restrictions. If the server's addresses change, update the private bind-address lists and redeploy firewall and Caddy.

The following URLs are documentation examples, not deployment settings:

Internal URL	Backend
https://example.com/note/	Trilium; /note stripped; NetBird only
https://example.com/vault/	Vaultwarden; same URL internally and externally
https://example.com/vault/admin	Vaultwarden admin; NetBird only
https://home.example.com/audiobookshelf/	Audiobookshelf
https://home.example.com/admin/	Cockpit
https://home.example.com/vault/admin	Vaultwarden admin
https://home.example.com/	Existing home page
https://docs.example.com	Papra
https://pics.example.com	Immich
https://media.example.com	Jellyfin
https://aigw.example.com	Bifrost
https://chat.example.com	Open WebUI

Configure split DNS in NetBird (or client hosts entries) for these names pointing to the server's NetBird IPv4/IPv6 addresses. Public DNS records are not required for internal-only names. The primary hostname is served on both WAN and NetBird addresses with the same Let's Encrypt certificate. /vault/ works on both; /note/ and /vault/admin are routed only on NetBird. Keep Vaultwarden's public DOMAIN unchanged. Split DNS should resolve the primary name to the NetBird address for enrolled clients and to the public address for external clients. The existing home name remains an internal-CA alias for its private routes.

Additional internal hostnames use tls internal; the shared primary hostname uses its public certificate even when accessed internally. Caddy generates/renews leaf certificates and reuses its persistent CA. The role exports the public root certificate to /etc/caddy/internal-root.crt; install this certificate in each client/browser trust store. Never distribute the CA private key. On Arch the CA is under /var/lib/caddy/pki/authorities/local/. Both /etc/caddy and all of /var/lib/caddy, including the CA key, root certificate, intermediate and ACME state, are registered in /etc/restic/root.files. Restore these paths before starting Caddy to retain client trust. Existing deployments may already have a CA; ensure /var/lib/caddy is backed up and protect restic credentials accordingly.

Private host settings and migration

Repository hostnames and email addresses are reserved examples. Keep real values in a private file outside the repository, such as $HOME/.config/azure-archlinux-image/host.yml, with mode 0600. Either pass -e @/absolute/path/to/host.yml or export the following before any playbook run:

export AZURE_ARCHLINUX_HOST_VARS="$HOME/.config/azure-archlinux-image/host.yml"

The playbook loads that file even for tagged runs. Caddy refuses to deploy the example public hostname, preventing an unconfigured run from replacing a real site. Include caddy_public_domain, caddy_acme_email, caddy_internal_sites, netbird_domain, netbird_client_management_url, and open_webui_domain in your private vars. Also set application base URLs/trusted origins and deployment-specific bifrost_proxy_domains there when applicable. Never commit the private file. Review ansible/roles/caddy/defaults/main.yml for the full site structure. Set caddy_public_bind_addresses, caddy_internal_bind_addresses, and caddy_netbird_interface in the private vars file; networking-hardening and Caddy must use the same NetBird addresses. Use explicit, disjoint local IP lists, never wildcard addresses or the Azure NAT public IP. Do not add the shared primary hostname to caddy_internal_sites: its internal Vaultwarden/Trilium routes are already managed separately with the public certificate. Existing application data/secrets are not replaced. Caddy's ACME egress allowlist is now managed by its own role.

Apply firewall, backup registration, and both site-fragment roles together:

env LANG=C.UTF-8 LC_ALL=C.UTF-8 ansible-playbook \
  -i '192.0.2.10,' -u operator ansible/local.yml \
  --tags firewall,restic,caddy,netbird,open-webui

The firewall is applied first, and Caddy refuses to proceed without the live interface guard. A first-migration --check cannot satisfy this live prerequisite before the firewall has actually been applied. Caddy stages both app fragments before validating and activating the complete listener configuration, avoiding mixed wildcard/per-address listeners during migration. Allow a short interruption while Caddy and any changed application services restart. restic and caddy are enabled in the default playbook before the application roles.

Verify public Vaultwarden/NetBird from outside the VPN, and verify internal URLs with the trusted CA from a NetBird peer. Requests for private apps on the WAN listener return 404. Connections to the NetBird web addresses via another interface must fail. Only standard web ports need NSG rules; backend ports stay off the public firewall. Enable any optional backend roles separately as needed.

Trilium on-demand activation

The trilium-next-server role enables a loopback systemd socket on port 8080 (the existing Caddy upstream). Requests start trilium-next-server-proxy.service, which starts Trilium on 127.0.0.1:8081, waits for /api/health-check, and forwards connections using systemd-socket-proxyd. Only the socket is boot-enabled. After 10 minutes without open connections, the proxy exits and StopWhenUnneeded=yes stops Trilium gracefully.

The role validates and updates the existing plain Caddy directive reverse_proxy * 127.0.0.1:8080 to disable upstream keep-alive for this route only. It preserves other routes and backs up Caddy's configuration before changing it. Apply the caddy role first; it creates the internal Trilium route. Persistent browser/WebSocket or sync connections can keep Trilium awake, and scheduled scripts do not run while it is stopped. This is connection-idle activation, not application-aware background-job management.

Defaults expose trilium_next_server_activation_port, trilium_next_server_backend_port, and trilium_next_server_idle_timeout. The backend port must be free. Changing the public activation port also requires updating the existing Caddy route. Apply only this role with:

env LANG=C.UTF-8 LC_ALL=C.UTF-8 ansible-playbook \
  -i '192.0.2.10,' -u operator ansible/local.yml --tags trilium-next-server

python -m unittest discover -s ansible/tests -p 'test_trilium_activation.py'

Inspect trilium-next-server-proxy.socket, trilium-next-server-proxy.service and trilium-next-server.service with systemctl. The socket should remain active (listening) when both services are inactive. A request through https://example.com/note/api/health-check over NetBird should wake Trilium.

Vaultwarden on-demand activation

The vaultwarden role uses the same activation pattern as Trilium: Caddy keeps its upstream 127.0.0.1:8000, a systemd socket starts vaultwarden-proxy.service, and the proxy starts Vaultwarden on 127.0.0.1:8001. Only the socket is boot-enabled. After 10 minutes without open connections, the proxy exits and Vaultwarden stops gracefully. Notification WebSockets can keep it running while clients remain connected; this deliberately avoids interrupting those clients.

A separate, last-loaded environment file sets the backend address/port without overwriting /etc/vaultwarden.env, passwords, keys, or the admin configuration. Readiness uses /alive, including the URL prefix from DOMAIN or the persisted admin configuration (for example /vault/alive). The role preserves existing Caddy headers/access rules, disables upstream pooling for Vaultwarden, and removes the obsolete port-3012 notification directive. Current Vaultwarden serves HTTP and notification WebSockets through the same listener. Existing Caddy sites must already be present; this role does not reconstruct them. Stopping Vaultwarden also postpones scheduled maintenance until it next runs.

Defaults expose vaultwarden_activation_port, vaultwarden_backend_port, and vaultwarden_idle_timeout. Apply just this role with:

env LANG=C.UTF-8 LC_ALL=C.UTF-8 ansible-playbook \
  -i '192.0.2.10,' -u operator ansible/local.yml --tags vaultwarden

python -m unittest discover -s ansible/tests -p 'test_vaultwarden_activation.py'

Inspect vaultwarden-proxy.socket, vaultwarden-proxy.service, and vaultwarden.service using systemctl. A request through https://example.com/vault/alive wakes the application without needing an administrator login. Boot enablement is configured but does not require a reboot to apply.

Recreating Immich, Papra, and OneDrive mounts

The opt-in roles in ansible/local.yml reproduce the a rootful Podman and rclone setup:

Role	Services	Persistent storage
rclone-onedrive	rclone-onedrive, rclone-media	onedrive_media: at /mnt/onedrive; onedrive_media_crypt: at /mnt/media
immich	immich-compose	Originals at /mnt/media/Pictures; local thumbnails/previews at /var/lib/immich-thumbs; PostgreSQL at /etc/containers/systemd/immich/postgres; Podman model-cache volume
papra	papra (generated from a Quadlet)	Documents at /mnt/onedrive/Papra_Docs; SQLite at /var/lib/papra-db

Role defaults expose image versions, paths, ports, proxies, and mount flags. Immich retains v3, the inspected PostgreSQL/Valkey image digests, port 2283, and the 10.89.0.0/24 bridge with gateway 10.89.0.1 on podman1. The containers.podman.podman_network module manages that bridge; Compose treats it as external so stopping Immich does not remove it. Papra retains its latest image, port 1221, configured URLs, encrypted document storage, and English-only OCR. Its language data comes from Arch's tesseract-data-eng package rather than a manually downloaded file.

Credentials and restoration

1. Restore local application databases/configuration before starting the roles. Keep the Immich database password from the original .env.
2. Copy ansible/secrets/apps.example.yml to ansible/secrets/apps.local.yml (gitignored), restrict its permissions to 0600, and fill in the existing credentials. Alternatively supply an Ansible Vault vars file with -e @<vault-file> --ask-vault-pass. No production credentials are included in the repository.
3. For Papra's first migration from the existing inline Quadlet configuration, securely recover AUTH_SECRET and DOCUMENT_STORAGE_DOCUMENT_KEY_ENCRYPTION_KEYS from the original Quadlet into papra_auth_secret and papra_document_encryption_keys. The role moves these into a root-only environment file. Do not generate replacement encryption keys for existing documents: the old keys are needed to decrypt them. Neither secret is written to the generated unit or shown in Ansible diffs.
4. Restore /var/lib/rclone/.config/rclone/rclone.conf, or supply its complete contents in rclone_onedrive_config_content. It must contain both onedrive_media and onedrive_media_crypt, including the OneDrive OAuth token, drive settings, and original crypt passwords. The role only seeds a missing config and never overwrites refreshed tokens on subsequent runs. Update or reauthorize an existing config with rclone config separately.
5. Existing Immich .env and Papra environment files are preserved when their secret variables are empty. Supplying credentials opts into managing those files from the role variables; include any non-default settings as overrides.

When /etc/restic/root.files exists, the roles register the local database and configuration paths there, including the rclone credentials. Protect those backups. Back up PostgreSQL/SQLite consistently (stop the apps or use database-aware snapshots/dumps); listing the database directories alone does not guarantee a consistent live backup. Cloud documents/photos and the crypt keys must also be retained. The model cache can be downloaded again.

Local Immich thumbnails and previews

The Immich role keeps the existing ${UPLOAD_LOCATION}:/data mount and overlays immich_thumbs_location (default /var/lib/immich-thumbs) at /data/thumbs. Both timeline thumbnails and larger photo previews stay on the durable SSD holding /, not /mnt/arch-var on the small OS/cache disk. Originals and encoded videos remain on OneDrive. The role registers the local directory for restic backup and adds RequiresMountsFor plus a .immich marker check to the service.

Existing installation: stop Immich and copy the complete remote thumbs/ directory, including .immich and person thumbnails, to the local directory before applying the role. For example, after checking free space:

sudo systemctl stop immich-compose
sudo mkdir -p /var/lib/immich-thumbs
sudo rsync -a --checksum /mnt/media/Pictures/thumbs/ /var/lib/immich-thumbs/
# Apply the immich role to add the nested mount and start the service.

Do not delete the remote copy during migration. Verify the container mounts, startup health, and image reads before considering its later removal. Once the local directory is active it is authoritative: do not periodically overwrite it from the now-stale remote copy. The role refuses to hide existing remote thumbs with an uninitialized local bind mount; it does not copy a live library. Restore the local thumbnail directory alongside the database/configuration when rebuilding. A genuinely new library gets a fresh .immich marker without copying anything.

Jellyfin without artwork

The Jellyfin role defaults to jellyfin_disable_artwork: true. It updates only supported settings in existing jellyfin_library_options_paths (default the Music library): photos off, image providers/order empty, and all image-type limits zero. It sets scan fanout and metadata refresh concurrency to one. Other library paths, metadata providers, database contents, and music files are preserved. Missing XML files are skipped, so run the role again after creating a new Music library. Settings are checked read-only first; Jellyfin is stopped only when an update is needed, with backups under /var/backups/jellyfin-settings, and started afterward.

Existing artwork associations are intentionally retained. The Caddy role's caddy_jellyfin_disable_artwork: true blocks artwork GET/HEAD requests with empty 204 responses only on the internal site whose upstream matches caddy_jellyfin_upstream (default 127.0.0.1:8096). This also covers native apps using that HTTPS hostname; direct port-8096 connections bypass it. Immich's images, normal Jellyfin metadata/streaming routes, and image-management writes are not blocked. There are no web/client code changes or CSS overrides.

To re-enable artwork, disable both flags and restore image settings through Jellyfin's library UI (or restore the saved XML while Jellyfin is stopped). Disabling management does not automatically undo previously applied XML settings.

Applying the roles

Install Ansible's community.general and containers.podman collections on the controller. Apply networking-hardening before the app roles on a new host: it configures the host/container proxy ACL, registry/model/OneDrive allowlist, and the container-to-proxy firewall rule. Apply the caddy role for the managed public/internal application sites described above; configure private DNS separately.

Uncomment the three tagged role entries in ansible/local.yml, retaining the order rclone-onedrive, immich, papra. On an already provisioned host:

env LANG=C.UTF-8 LC_ALL=C.UTF-8 ansible-playbook \
  -i '192.0.2.10,' -u operator ansible/local.yml --tags apps --check

# Apply after reviewing the check-mode output:
env LANG=C.UTF-8 LC_ALL=C.UTF-8 ansible-playbook \
  -i '192.0.2.10,' -u operator ansible/local.yml --tags apps

apps.local.yml is loaded automatically even for tagged runs. Run without --tags apps for full host provisioning, reviewing the other enabled roles first. rclone runs as root, matching the existing system services, with --allow-other so containers can access the mounts. Type=notify waits for each FUSE mount to be ready before dependent applications start; both applications also check that their cloud mountpoint is actually mounted. The cache remains at /mnt/arch-var/rclone-cache-onedrive and must have sufficient local space. Papra's generated service is enabled via the Quadlet's [Install] section, not systemctl enable on a generated unit.

Run the focused template/configuration tests with:

python -m unittest discover -s ansible/tests -p 'test_app_roles.py'

NetBird server and dashboard

The netbird role, enabled in ansible/local.yml, installs netbird-server and netbird-dashboard from AUR using yay as the unprivileged AUR builder. The server package is the current combined management/signal/relay/STUN server; the separately packaged dashboard is generated with netbird-dashboard-generate and served directly by Caddy. No NetBird containers, Nginx, external identity provider, or client VPN service are installed.

Example URL: https://netbird.example.com; configure the real name privately.

Provisioning

1. Point the hostname's DNS record to the server's public IP. Allow TCP 80/443 and UDP 3478 through the Azure NSG/router/firewall. Caddy handles HTTPS; STUN uses UDP directly and cannot go through Caddy.

2. Set the three netbird_* secrets directly on the server in the root-only /etc/netbird-server/secrets.yml, or use apps.local.yml / an Ansible Vault vars file. Ansible-supplied nonempty values take precedence over server-local values. Generate three independent values using openssl rand -base64 32. Raw 32-byte ASCII encryption keys are also accepted and base64-encoded when rendering the server configuration; the server-local secrets file is not rewritten. The relay secret is used as supplied and must have at least 32 characters. Keep these keys alongside the database backups; do not rotate encryption keys when restoring existing data.

3. The NetBird role itself allows netbird_stun_port (3478 by default) in nftables. It installs /etc/nftables.d/netbird-input.nft, includes service rules before the input chain's terminal drop, and inserts the live rule at that same position. It does not flush the ruleset or reload nftables, preserving Podman's dynamically installed container/NAT rules. The networking-hardening template retains the include on subsequent runs. If overriding the STUN port, also update the external firewall.

4. Apply the existing yay-bin and caddy roles before netbird, after networking-hardening. The NetBird role checks these prerequisites and extends tinyproxy's allowlist for AUR/Go/npm builds, the initial geolocation database download from pkgs.netbird.io, and its own public issuer. Review AUR PKGBUILDs before use; the first source builds can take some time.

5. Once prerequisites are applied, preview/apply only NetBird with:

   env LANG=C.UTF-8 LC_ALL=C.UTF-8 ansible-playbook \
     -i '192.0.2.10,' -u operator ansible/local.yml --tags netbird --check
   
   env LANG=C.UTF-8 LC_ALL=C.UTF-8 ansible-playbook \
     -i '192.0.2.10,' -u operator ansible/local.yml --tags netbird

6. Open the HTTPS dashboard and immediately complete initial administrator setup before allowing other users access. Subsequent user management uses NetBird's embedded identity provider.

The role manages a validated fragment at /etc/caddy/sites/public/netbird.caddy, imported by the Caddy role, leaving the other applications' sites intact. Caddy serves the dashboard and callback pages, forwards gRPC using h2c, and forwards REST, OAuth2, relay, and WebSocket paths without stripping their prefixes. The server listens only on 127.0.0.1:8085; its health endpoint is on 127.0.0.1:9001. Metrics use port 9091, remain outside the Caddy routes, and should not be exposed by the host or external firewall. The server runs as a dedicated netbird user; the configuration is root-owned, group-readable, and omitted from Ansible diffs.

The role starts the service and checks embedded OIDC discovery against the configured public issuer. It registers /etc/netbird-server, /var/lib/netbird, the dashboard environment, and Caddyfile with restic when its root file list exists. Stop the service or take consistent database backups, and retain store.db, events.db, idp.db, and the encryption keys together.

Deploying before secrets are available

To install packages, publish the dashboard, and configure the firewall without starting an unconfigured backend:

env LANG=C.UTF-8 LC_ALL=C.UTF-8 ansible-playbook \
  -i '192.0.2.10,' -u operator ansible/local.yml --tags netbird \
  -e netbird_allow_unconfigured=true

This explicit staging mode only accepts all three secrets empty, not partial or malformed credentials. It seeds /etc/netbird-server/secrets.yml with empty values and mode 0600 without overwriting an existing file. The dashboard is served, but login/API calls are unavailable and netbird-server remains stopped and disabled.

On the server, edit the file with sudoedit /etc/netbird-server/secrets.yml and set:

netbird_relay_auth_secret: "<first openssl rand -base64 32 value>"
netbird_store_encryption_key: "<second independent value>"
netbird_session_cookie_encryption_key: "<third independent value>"

Then rerun the role without netbird_allow_unconfigured=true. It reads the secrets from the server, renders the private configuration, enables/starts the backend, and verifies OIDC discovery. No credentials need to be copied back to the controller. Do not start the staged service manually before this render.

Installed AUR packages are not rebuilt on each Ansible run. Perform upgrades explicitly with yay as the AUR builder. The dashboard package's upgrade hook regenerates its assets from the managed environment file. Check-mode does not build missing packages or run the generator, so a fresh-install check cannot verify the resulting binaries.

During initial AUR builds, the role temporarily stops an active pacman-auto-update.timer, waits for any running update to finish, and restores the timer in an always block. This prevents the updater's orphan cleanup from removing Go or other build dependencies during compilation. Apply the same precaution when performing manual AUR upgrades.

Package/configuration references: AUR server, AUR dashboard, combined server configuration, reverse-proxy requirements.

NetBird client

The separate netbird-client role installs netbird-bin from the AUR through yay as the unprivileged AUR builder. This package repackages the upstream prebuilt client; the role does not download/install an unmanaged binary or compile the Go client. It requires the existing yay-bin, host proxy and kernel TUN setup.

The role is enabled in the local playbook. From the repository root, deploy only this role with:

env LANG=C.UTF-8 LC_ALL=C.UTF-8 ANSIBLE_CONFIG=ansible/ansible.cfg \
  ansible-playbook -i '192.0.2.10,' -u operator \
  ansible/local.yml --tags netbird-client

It enables the AUR-provided netbird@client.service, with a managed drop-in isolating all five systemd directories from the backend. Client configuration is in /etc/netbird-client, state and profiles are under /var/lib/netbird-client through NB_STATE_DIR, and cache, logs and runtime files use their respective netbird-client directories. The daemon runs as the separate, non-login netbird-client account with only CAP_NET_ADMIN, CAP_NET_RAW, and CAP_NET_BIND_SERVICE. Configuration/state directories are 0700 and remain separate from the backend's netbird account. A narrowly scoped polkit rule authorizes this account's systemd-resolved DNS operations; no general sudo permission is granted. The role links /etc/resolv.conf to the resolved stub so Go programs do not fall back to a nonexistent localhost DNS server.

The daemon creates its RPC socket in /run/netbird-client; a root-owned symlink at /run/netbird.sock, persisted with tmpfiles, preserves the normal CLI:

sudo systemctl status netbird@client.service
sudo netbird status

Installation does not enroll the client. A fresh daemon reports NeedsLogin. For an already enrolled client, the role updates only WgPort to the managed netbird_client_wireguard_port (default 51821), preserving identity, keys, and other enrollment settings. It stops the daemon before a required port edit; unchanged runs do not restart it. Client configuration/state are registered with restic when its file list exists.

The role installs persistent firewall snippets with these exceptions:

• Inbound UDP 51821 for the client, distinct from existing wg0/51820.
• WAN TCP 443 and UDP only for the dedicated netbird-client UID. STUN/ICE and QUIC use ephemeral source ports, so restricting all userspace UDP to the WireGuard source port breaks discovery. The daemon explicitly unsets uppercase/lowercase HTTP, HTTPS and ALL proxy variables: management, signal, relay and tunnel transport never go through Tinyproxy. Package installation still uses Tinyproxy; the management hostname is no longer client-allowlisted.
• Kernel WireGuard UDP is UID 0 and is restricted to source port 51821.
• Direct host-originated traffic on the NetBird interface, subject to NetBird's own ACL table; this does not grant arbitrary WAN egress.
• Overlay input remains subject to NetBird's own ACL table. Forwarding from the overlay is limited to DNAT/published container services and established replies.

There is no all-user outbound UDP rule. Other applications still use allowlisted tinyproxy for web egress. The service is a trusted network administrator because CAP_NET_ADMIN is required; this is UID separation, not a sandbox against a compromised VPN daemon or root. PID-based nftables rules are not used.

Tinyproxy also accepts connections from NetBird peers at http://<proxy-vm-netbird-ip>:8888. The networking-hardening role allows 100.64.0.0/10 clients on wt0 by default, preserving localhost and Podman access. An early firewall guard blocks the proxy port on WAN and other NICs, even if later service rules broadly accept traffic. NetBird ACLs must also permit the peer connection. The destination-domain allowlist remains enforced. Use tinyproxy_netbird_interfaces and tinyproxy_netbird_allowed_clients in private host vars to select other interfaces or restrict access to specific peer IPs/CIDRs. The default listener is IPv4 (tinyproxy_listen_address: 0.0.0.0). Remote services can set their proxy URL to the proxy VM's NetBird address; keep internal service hostnames in their NO_PROXY settings.

networking-hardening sets both no_proxy and NO_PROXY in /etc/environment and the shared systemd proxy.env to localhost,127.0.0.1,::1,100.64.0.0/10,.netbird.selfhosted. Override networking_hardening_no_proxy in private host vars to add custom NetBird DNS suffixes or routed networks. Clients must support CIDR bypass entries; for clients that do not, use NetBird DNS names or add exact peer addresses. New login sessions pick up /etc/environment; the role restarts running Caddy, Vaultwarden, Grafana Alloy and Jellyfin consumers of the shared systemd file. One-shot backup/update services read it on their next start.

Apply only the proxy bypass and client migration on an existing host with --tags proxy,netbird-client. Restarting the client can briefly interrupt SSH over NetBird; reconnect after the daemon comes back.

Apply the firewall migration and both NetBird roles together:

env LANG=C.UTF-8 LC_ALL=C.UTF-8 ANSIBLE_CONFIG=ansible/ansible.cfg \
  ansible-playbook -i '192.0.2.10,' -u operator ansible/local.yml \
  --tags firewall,netbird,netbird-client

The base ruleset atomically replaces only inet filter using destroy table (nftables >= 1.0.9), preserving Podman/NetBird tables. Never use flush ruleset. The firewall tag applies just the base firewall tasks, without package/mirror refreshes. Run that migration before applying only netbird-client on an older installation. The installed snippets are included on future hardening runs.

For initial enrollment, use the same managed port:

sudo netbird up \
  --management-url https://netbird.example.com \
  --wireguard-port 51821

Complete the interactive login or supply a setup key when explicitly enrolling. The role neither requires nor stores a setup key in the repository. netbird_client_management_url controls the management hostname allowed through the proxy and shown in the enrollment instructions; enrollment itself must use that URL explicitly.

Caddy and direct-connection troubleshooting

The NetBird role removes a dedicated loopback /etc/hosts entry for its public hostname: that override also redirects STUN and prevents public NAT discovery. The Caddy role sets request-body idle and write-idle timeouts for long-lived gRPC streams (caddy_netbird_idle_timeout, default 24h). This requires the Caddy 2.11.6+ idle-timeout options. Merely changing reverse-proxy transport timeouts is insufficient. These are server-wide settings in the existing global servers block; other sites share the longer idle allowance. Header timeouts remain intact.

Internal applications use an interface-restricted listener instead of IP matchers. The base firewall allows public UDP 443 for Caddy HTTP/3 as well as TCP HTTPS; the separate NetBird-bound HTTP/HTTPS listeners accept ingress only on wt0.

A relayed peer is not necessarily a host-firewall error. Inspect the actual daemon: sudo netbird --daemon-addr unix:///var/run/netbird-client/netbird.sock status -d and /var/log/netbird-client/client.log. Check whether remote offers support ICE, then test STUN from the VM, not only from an external client. An initial timeout when contacting the VM's own public STUN address is not proof of a persistent cloud NAT hairpin problem: blocked cloud egress can produce the same symptom.

On the follow-up investigation, allowing UDP 3478 and UDP 443 in the Azure NSG's egress policy coincided with all three peers switching from relayed WebSocket to direct P2P, without a NetBird restart or configuration change. Public-IP STUN probes then succeeded. Keep host and NSG policies aligned in both directions: inbound fixed WireGuard port 51821 and STUN 3478, outbound configured STUN ports, and the required peer UDP flows (remote NAT-mapped ports are not necessarily 51820/51821). UDP 443 supports HTTP/3/QUIC where configured; it is not a substitute for STUN. Scope rules appropriately rather than allowing arbitrary UDP globally. The existing host policy restricts userspace discovery egress to the dedicated NetBird service UID and kernel WireGuard egress to its fixed source port.

The measured overlay MTU remains 1280 (IPv4 TCP MSS 1240). Full-size 1280-byte DF probes to the vnic peer succeeded; a 1281-byte packet was correctly rejected locally. A padded STUN request also verified a 1500-byte, nonfragmenting IPv4 outbound underlay path. That does not establish every peer's return-path MTU, especially behind mobile networks, so no MTU increase was applied.

Secure TCP transport defaults

The networking-hardening role manages /etc/sysctl.d/zz-tcp-transport.conf (tag tcp), which sorts after the legacy alphabetically named sysctl files:

net.ipv4.tcp_sack = 1
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_abort_on_overflow = 0

These are modern TCP defaults for patched kernels: selective loss recovery, randomized per-connection timestamp offsets (mode 1, not mode 2), and retries instead of resets on transient accept-queue pressure. Historical SACK fixes are present in the current hardened kernel; disabling SACK globally is not necessary. The first-boot defaults and legacy image playbooks use the same values and no longer write the removed tcp_fack setting. Only this small sysctl file is loaded; firewall policy, SYN cookies, source-route/redirect rejection, reverse-path filtering, and kernel hardening are not relaxed. Existing TCP connections retain the options negotiated at establishment; new connections get the updated options.

Bifrost AI gateway and Open WebUI

The bifrost role runs a pinned Podman image through a rootful Quadlet, like Papra. The open-webui role installs the AUR package through the unprivileged yay builder. Apply networking-hardening, yay-bin, and caddy first. Both backends bind only to loopback:

Application	Local endpoint	Persistent storage
Bifrost	127.0.0.1:8086	/var/lib/bifrost, /etc/bifrost
Open WebUI	127.0.0.1:3000	/var/lib/open-webui, /etc/open-webui

Deploy just these roles to the application server:

env LANG=C.UTF-8 LC_ALL=C.UTF-8 ANSIBLE_CONFIG=ansible/ansible.cfg \
  ansible-playbook -i '192.0.2.10,' -u operator \
  ansible/local.yml --tags bifrost,open-webui

The roles extend tinyproxy's allowlist without changing the host firewall or existing Caddy sites. AUR installation temporarily pauses automatic updates and restores the timer even on failure. Installed packages are not rebuilt on every run; upgrade them explicitly through yay.

Bifrost uses docker.io/maximhq/bifrost:v2.3.1, configurable with bifrost_image. Its Quadlet uses host networking to reach the existing loopback-only tinyproxy, explicitly binds the app to 127.0.0.1, and runs as the image's non-root UID 1000 with capabilities dropped. Its data mount uses Podman's :U ownership adjustment. No container bridge/firewall exceptions are needed. This avoids the AUR package's mutable README checksum failure. The AUR Open WebUI package uses uv to install its Python environment on first launch; this can take several minutes and several GB of disk space. The role uses the installed AUR package's application version and the CPU-only PyTorch index. It overrides the package's hard-coded all-interface port 8080 and removes the service's Ollama dependency, since Bifrost supplies the OpenAI-compatible API instead.

Access and initial configuration

Caddy serves https://chat.example.com with WebSocket support and unbuffered streaming, using tls internal and the NetBird-only firewall listener. The role manages /etc/caddy/sites/internal/open-webui.caddy, not the main Caddyfile. Configure private DNS and trust /etc/caddy/internal-root.crt on the client as described above. No public DNS record or ACME certificate is needed; neither backend port needs opening.

Open WebUI defaults to the loopback Bifrost API. To use the internal Caddy endpoint (on this VM or another NetBird VM), set private host variables:

open_webui_bifrost_url: https://aigw.example.net/openai/v1
# Path on the Open WebUI VM to the gateway's public CA certificate:
open_webui_gateway_ca_file: /etc/caddy/internal-root.crt
# Optional additional bypass domains; the gateway hostname is added automatically:
networking_hardening_no_proxy: localhost,127.0.0.1,::1,100.64.0.0/10,.netbird.selfhosted,.example.net

For separate VMs, securely provision the gateway's public root certificate on Open WebUI's VM and point open_webui_gateway_ca_file there. Never copy the CA private key. The role installs the CA into Arch's trust store and configures Python HTTP clients to use the system CA bundle; TLS verification stays enabled. Local connections to this VM's overlay Caddy address are allowed through lo; non-overlay external interfaces remain blocked. Remote connections still require NetBird routing and ACLs. Both proxy bypass environment variables inherit networking_hardening_no_proxy, plus the configured gateway hostname.

Existing installations: Open WebUI persists connection settings in its database. Changing the environment seeds new installs but does not overwrite saved admin settings. Update the connection URL in Admin Settings → Connections to match open_webui_bifrost_url, retaining its Bifrost virtual key as the API key.

Public signup is disabled. On first installation the role generates an administrator password and a session-signing key on the server, storing them in /etc/open-webui/secret.env with mode 0600. Log in as admin@chat.example.com (override open_webui_admin_email if needed). Retrieve the initial password privately on the server:

sudo grep '^WEBUI_ADMIN_PASSWORD=' /etc/open-webui/secret.env

Change the password after login. Rerunning the role does not replace the credentials or reset the account's password. Restore the database and /etc/open-webui/secret.env together to retain the session-signing key.

Bifrost's management UI is deliberately not public. Reach it over SSH:

ssh -L 8086:127.0.0.1:8086 operator@192.0.2.10

Open http://localhost:8086. On first setup, the dashboard requires the operator's setup token before an administrator account can be created. The role generates this token on the server, stores it in /etc/bifrost/secret.env (root-only, mode 0600), and passes it into the container as BIFROST_SETUP_TOKEN. Retrieve it privately on the server:

sudo grep '^BIFROST_SETUP_TOKEN=' /etc/bifrost/secret.env

Enter the value in the dashboard, then enable dashboard authentication and create your administrator account. The setup token stops working once dashboard authentication is enabled. The role preserves the token across reruns and registers /etc/bifrost for backup alongside the database. It verifies that setup is possible or dashboard authentication is already enabled; a basic health check alone does not confirm this.

Then configure providers and their API keys. Open WebUI initially connects to http://127.0.0.1:8086/openai/v1; provider credentials stay in Bifrost rather than this repository. No provider keys or paid inference requests are supplied by these roles. Add any additional provider hosts to bifrost_proxy_domains in your external private vars file. For Azure Foundry, explicitly allow your own resource's inference hostname. For each provider, explicitly set its proxy configuration in the Bifrost dashboard to HTTP, URL http://127.0.0.1:8888, or choose its environment proxy mode. Bifrost's provider HTTP clients do not use the container's proxy environment when the provider has no proxy configuration. Without this setting, the default-deny host firewall causes provider requests, including model discovery, to time out.

For Azure, use the resource's inference endpoint rather than a Foundry project management URL ending in /api/projects/<project>. Verify the endpoint against the deployment's connection details; Bifrost appends /openai/v1/models when discovering Azure models.

If enabling Bifrost virtual-key authentication, configure the matching key privately in Open WebUI's admin connection settings. These connection settings persist in Open WebUI's database after first launch; later environment changes do not overwrite admin-managed settings.

When /etc/restic/root.files exists, both roles register their persistent state. Protect these backups, which contain chats, credentials and provider keys; stop the services or use database-consistent backups.

Repository Structure

build.sh                          # Build script (replaces Packer)
packages.conf                     # Package lists for squashfs + full install
initcpio/
  install/squashfs-overlay         # Initramfs: install hook
  systemd/arch-root-setup.service  # Initramfs: orchestrates root selection
  systemd/setup-root               # Initramfs: data-disk or squashfs setup logic
first-boot/
  provision-data-disk.sh           # First-boot: encrypt, format, install
  provision-data-disk.service      # systemd unit for provisioning
bicep/                             # Azure infrastructure (unchanged)
playbooks/                         # Ansible playbooks (optional, for customization)

Key Differences from Previous Architecture

Aspect	Old (Packer + luks_unlocker)	New (squashfs + vTPM)
Build tool	Packer + QEMU/KVM	Native Arch (pacstrap + mksquashfs)
Image size	4GB (full LUKS root)	~1.5GB (squashfs + ESP)
Encryption unlock	luks_unlocker binary (Azure KeyVault)	systemd-cryptenroll (vTPM, PCR 7+11)
Root filesystem	LUKS on OS disk	LUKS on data disk, squashfs fallback
Recovery	Fallback initramfs with passphrase	Squashfs maintenance environment
First boot	Manual data disk migration	Automatic provisioning
External deps	GitHub binary download	Azure Key Vault (for Secure Boot private key storage)