Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/bright-telemetry-shines.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@wpelevator/tug': minor
---

Add authenticated control readiness and project-scoped Prometheus metrics, configurable bounded writer admission with retry hints, and structured control/agent logs with safe default failure classifications and opt-in sanitized debug causes.
5 changes: 5 additions & 0 deletions .changeset/green-foxes-check.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@wpelevator/tug': minor
---

Add signed best-effort webhook notifications with bounded retries, startup grace and delivery metrics. Optional notification configuration errors keep control available. Add an external HTTP probe with bounded same-host redirects.
5 changes: 5 additions & 0 deletions .changeset/quiet-cats-retire.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@wpelevator/tug': patch
---

Document operational reliability and recovery responsibilities, and cover interrupted activation, ENOSPC, damaged-repository recovery, rollout behavior and integration convergence with regression tests.
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,8 @@ Project commands sit at the top level and the rest read `tug <subject> <verb>`.

Control serves mutual TLS on one port (`8443`). Every operator, CI job and node authenticates with a certificate from a private CA created at install, so the control address can be a bare IP and can change later. Do not terminate TLS in front of it. [Architecture](docs/architecture.md) covers every flow, and the [security audit](docs/security-audit.md) covers the trust boundaries.

Control readiness monitoring uses an authenticated `GET /api/v1/readyz` with a `read` credential; it checks database reads, overdue writer work and whether control needs a restart after restore. Project-filtered Prometheus metrics are at `/api/v1/metrics`; optional signed webhooks report actionable state changes. See [Observability](docs/observability.md) for setup and limits.

## Documentation

The [documentation index](docs/index.md) lists every page by task. The ones you will reach for first:
Expand Down
2 changes: 2 additions & 0 deletions docs/access-control.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,8 @@ The project column says how a scoped credential's allowed projects are checked:
| `GET /api/v1/holds` | `admin` | Query; deployment move holds only |
| `GET /api/v1/status` | `read` | Response is filtered to allowed projects |
| `GET /api/v1/version` | `read` | None; reports control's own version |
| `GET /api/v1/metrics` | `read` | Project-scoped workload series; global node, writer and webhook metrics only for unscoped credentials |
| `GET /api/v1/readyz` | `read` | None; database read and writer age checks, without database error details |
| `GET /api/v1/whoami` | `read` | None; reports the caller's credential name, role and project scope |
| `GET /api/v1/events` | `read` | `project` query, or filtered to allowed projects; `after` follows the log by `seq`, and `cursor` is only ever from rows the caller can read |
| `GET /api/v1/watch` | `read` | WebSocket; hints and new events filtered to allowed projects |
Expand Down
4 changes: 2 additions & 2 deletions docs/backups.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ tug backup password

The password is derived from control's master key, so losing both the password and the master key makes the repository unreadable. Credentials are encrypted on control and released only to a node running a backup. Each project has its own repository under `<destination>/<project>`, and commands use the [resolved project](cli.md#arguments-options-and-configuration).

Control also backs itself up to the same destination every night, see [Control plane backup](#control-plane-backup). No backup notification is pushed, so monitor `tug backup list --check`.
Control also backs itself up to the same destination every night, see [Control plane backup](#control-plane-backup). Configure [signed webhooks](observability.md#alerting) for backup failures, missed protection and recovery notifications. Delivery is best effort; keep an independent scheduled `tug backup list --check`.

## Plan for recovery

Expand Down Expand Up @@ -220,7 +220,7 @@ A node runs backup and restore jobs in lanes: a restore and its safety backup fi

| Limit | Detail |
| --- | --- |
| No push notifications | A failed night is a `backup.failed` or `backup.stale` event, until [webhooks](roadmap.md#m8--central-api-advantages-partly-done) exist. Run `tug backup list --check` from cron or a monitor. It exits nonzero when a target is stale, failed or missed, and when nothing is protected (pass `--allow-empty` for a project without data) |
| Best-effort notification | A failed night produces a `backup.failed` or `backup.stale` event and a [configured webhook](observability.md#alerting). Delivery may be lost across control outages or shutdown. Run `tug backup list --check` from cron or a monitor. It exits nonzero when a target is stale, failed or missed, and when nothing is protected (pass `--allow-empty` for a project without data) |
| Control down | No backup starts. Running jobs finish and report when control returns, and a long gap yields one missed event and one catch-up run per target |
| Consistency | Live volumes are crash-consistent, not application-consistent. Use a command backup for anything with its own consistency |
| One node per volume | A volume is backed up from the node that holds it. A volume mounted from a file server with `driver_opts` is read through that node's mount, so back up the file server's own disk where you can |
Expand Down
13 changes: 13 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ The CLI config is a YAML or JSON file that is safe to commit. It holds the contr
|---|---|---|
| `PORT` | `8443` | The single mTLS port. |
| `TUG_CONTROL_URL` | required | Credential-free `https://` origin clients connect to by default, including the port. It is written into credentials and its host is added to the server certificate, but clients verify the generated control name, not this address. |
| `TUG_LOG_FORMAT` | `text` | Process log format: `text` or `json`; applies to control and agents. |
| `TUG_LOG_LEVEL` | `info` | Minimum process log level: `debug`, `info`, `warn`, `error`; applies to control and agents. |
| `TUG_WRITER_MAX_PENDING` | `100` | Positive maximum number of waiting control-writer operations. Excess admission returns 503 with `Retry-After: 5`; active work continues. |
| `TUG_WEBHOOK_URL` | unset | HTTPS notification receiver; requires `TUG_WEBHOOK_SECRET`. Invalid optional configuration disables notifications and logs an error. See [Alerting](observability.md#alerting). |
| `TUG_WEBHOOK_SECRET` | unset | HMAC-SHA256 signing secret; requires `TUG_WEBHOOK_URL`. Never place it in a URL. |
| `TUG_WATCH_TICK_MS` | `1000` | How often the [watch channel](architecture.md#operator-watch-channel) looks for changes nothing announced (deployment, backup and restore states, stale nodes), while at least one operator is watching. Requests that change something trigger a look at once. |
| `TUG_EXTRA_SANS` | unset | Comma-separated extra DNS names or IPs for the server certificate. |
| `TUG_RENEW_MIN_INTERVAL_MS` | `3600000` | Minimum time between certificate renewals per node. |
Expand All @@ -48,8 +53,16 @@ The CLI config is a YAML or JSON file that is safe to commit. It holds the contr
| `TUG_ACME_EMAIL` | unset | Email for ACME registration in rendered Caddy configs. |
| `TUG_INGRESS_ACCESS_LOG` | unset (off) | Set `true` and each ingress site writes a JSON access log line per request to the `tug-caddy` container's output (`docker logs tug-caddy` on the ingress node). Caddy redacts cookies and `Authorization` headers, but URLs (including query-string tokens) and client IPs are logged. Docker's log cap applies only to a newly created container; see [Networking](ingress.md#edge-features). |

The global writer limit applies to background work too: automatic rollback can be refused with a `deployment.auto_rollback_skipped` event; image maintenance logs its refusal and is retried after later deploys. Control backup failures remain visible in backup status and notifications. A full queue does not cancel active work. Retry operator work after the five-second hint; an automatically skipped rollback may require an explicit `tug rollback` once pressure clears. Readiness reports overdue active writer work after 35 minutes.

## External reachability probe

The source-checkout tool `node tools/scripts/check-reachability.mjs` takes URLs as arguments or from `TUG_PROBE_URLS` (comma/newline-separated). `TUG_PROBE_TIMEOUT_MS` defaults to `5000`. It requires a final 2xx response after at most five same-host redirects and exits nonzero on failure. Schedule it outside workload hosts; see [Alerting](observability.md#alerting).

## Agent

Agents also honor `TUG_LOG_FORMAT` and `TUG_LOG_LEVEL` with the defaults above.

| Variable | Default | Purpose |
|---|---|---|
| `TUG_CONTROL_URL` | unset | Connect-only override of the control address (for example a private endpoint); the server is verified by its generated control name. |
Expand Down
2 changes: 2 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,3 +159,5 @@ Run the agent in a container, as the installers do (`--network host`, the Docker
The interface follows the conventions of `wp-release`: `tug <subject> <verb>` operations with the everyday project commands at the top level, kebab-case command and flag names, primary targets as positional arguments, named qualifiers with consistent meanings, and positive camelCase internal options (`--no-wait` becomes `wait: false`, never `noWait`). New structured CLI results use camelCase, and API responses printed unchanged may keep their database field names. Help, docs and tests change together when the interface changes.

CLI adapters resolve configuration, parse arguments and format results. Build, archive, installation and API primitives receive explicit values and injectable runners, and they do not discover CLI config or read unrelated environment variables. Keep domain logic in ordinary module functions, and add stateful classes only when a multi-step workflow needs shared state. Only operations that use Docker or SSH require those tools. Validation completes before side effects, and stdout meant as a runnable artifact or JSON stays directly usable.

The QEMU first-boot harness accepts `--verify-reliability` to require authenticated `/api/v1/readyz` and node-report metrics after enrollment. `TUG_LOG_FORMAT=json` exercises structured control logs during that run. Use a disposable work directory; the default cleans up the guest and source control process, and `--keep` deliberately leaves them running.
2 changes: 2 additions & 0 deletions docs/networking.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,5 @@ sequenceDiagram
## Verifying

`npm run test:integration:network` starts three privileged fake VPSes with plain numeric node `address` values (`vps-c` as ingress node), deploys `tests/integration/compose/docker-compose.network-sample.yml`, and checks a container on `vps-b` fetching `http://web:8080` by its cross-node service name, Caddy on `vps-c` serving `app.test` from `vps-a`, and published ports bound to the intended private node addresses. See [Development](development.md#integration-test).

Cross-node host entry ordering is preserved across this release, including the first address a resolver may select. Endpoint appearance, removal or address change updates the hosts file by replacing the affected container. Excluding those changes from identity would preserve stale name resolution; tug does not provide an independent DNS service or a stable service VIP.
Loading
Loading