diff --git a/.changeset/bright-telemetry-shines.md b/.changeset/bright-telemetry-shines.md new file mode 100644 index 0000000..d9d9ed2 --- /dev/null +++ b/.changeset/bright-telemetry-shines.md @@ -0,0 +1,5 @@ +--- +'@wpelevator/tug': minor +--- + +Add authenticated control readiness and project-scoped Prometheus metrics, configurable bounded writer admission with retry hints, and structured control/agent logs with safe default failure classifications and opt-in sanitized debug causes. diff --git a/.changeset/green-foxes-check.md b/.changeset/green-foxes-check.md new file mode 100644 index 0000000..1baed6f --- /dev/null +++ b/.changeset/green-foxes-check.md @@ -0,0 +1,5 @@ +--- +'@wpelevator/tug': minor +--- + +Add signed best-effort webhook notifications with bounded retries, startup grace and delivery metrics. Optional notification configuration errors keep control available. Add an external HTTP probe with bounded same-host redirects. diff --git a/.changeset/quiet-cats-retire.md b/.changeset/quiet-cats-retire.md new file mode 100644 index 0000000..11c796a --- /dev/null +++ b/.changeset/quiet-cats-retire.md @@ -0,0 +1,5 @@ +--- +'@wpelevator/tug': patch +--- + +Document operational reliability and recovery responsibilities, and cover interrupted activation, ENOSPC, damaged-repository recovery, rollout behavior and integration convergence with regression tests. diff --git a/README.md b/README.md index 4b93ea0..ed1409f 100644 --- a/README.md +++ b/README.md @@ -166,6 +166,8 @@ Project commands sit at the top level and the rest read `tug `. Control serves mutual TLS on one port (`8443`). Every operator, CI job and node authenticates with a certificate from a private CA created at install, so the control address can be a bare IP and can change later. Do not terminate TLS in front of it. [Architecture](docs/architecture.md) covers every flow, and the [security audit](docs/security-audit.md) covers the trust boundaries. +Control readiness monitoring uses an authenticated `GET /api/v1/readyz` with a `read` credential; it checks database reads, overdue writer work and whether control needs a restart after restore. Project-filtered Prometheus metrics are at `/api/v1/metrics`; optional signed webhooks report actionable state changes. See [Observability](docs/observability.md) for setup and limits. + ## Documentation The [documentation index](docs/index.md) lists every page by task. The ones you will reach for first: diff --git a/docs/access-control.md b/docs/access-control.md index e225098..8b61660 100644 --- a/docs/access-control.md +++ b/docs/access-control.md @@ -129,6 +129,8 @@ The project column says how a scoped credential's allowed projects are checked: | `GET /api/v1/holds` | `admin` | Query; deployment move holds only | | `GET /api/v1/status` | `read` | Response is filtered to allowed projects | | `GET /api/v1/version` | `read` | None; reports control's own version | +| `GET /api/v1/metrics` | `read` | Project-scoped workload series; global node, writer and webhook metrics only for unscoped credentials | +| `GET /api/v1/readyz` | `read` | None; database read and writer age checks, without database error details | | `GET /api/v1/whoami` | `read` | None; reports the caller's credential name, role and project scope | | `GET /api/v1/events` | `read` | `project` query, or filtered to allowed projects; `after` follows the log by `seq`, and `cursor` is only ever from rows the caller can read | | `GET /api/v1/watch` | `read` | WebSocket; hints and new events filtered to allowed projects | diff --git a/docs/backups.md b/docs/backups.md index 103012f..ce13243 100644 --- a/docs/backups.md +++ b/docs/backups.md @@ -25,7 +25,7 @@ tug backup password The password is derived from control's master key, so losing both the password and the master key makes the repository unreadable. Credentials are encrypted on control and released only to a node running a backup. Each project has its own repository under `/`, and commands use the [resolved project](cli.md#arguments-options-and-configuration). -Control also backs itself up to the same destination every night, see [Control plane backup](#control-plane-backup). No backup notification is pushed, so monitor `tug backup list --check`. +Control also backs itself up to the same destination every night, see [Control plane backup](#control-plane-backup). Configure [signed webhooks](observability.md#alerting) for backup failures, missed protection and recovery notifications. Delivery is best effort; keep an independent scheduled `tug backup list --check`. ## Plan for recovery @@ -220,7 +220,7 @@ A node runs backup and restore jobs in lanes: a restore and its safety backup fi | Limit | Detail | | --- | --- | -| No push notifications | A failed night is a `backup.failed` or `backup.stale` event, until [webhooks](roadmap.md#m8--central-api-advantages-partly-done) exist. Run `tug backup list --check` from cron or a monitor. It exits nonzero when a target is stale, failed or missed, and when nothing is protected (pass `--allow-empty` for a project without data) | +| Best-effort notification | A failed night produces a `backup.failed` or `backup.stale` event and a [configured webhook](observability.md#alerting). Delivery may be lost across control outages or shutdown. Run `tug backup list --check` from cron or a monitor. It exits nonzero when a target is stale, failed or missed, and when nothing is protected (pass `--allow-empty` for a project without data) | | Control down | No backup starts. Running jobs finish and report when control returns, and a long gap yields one missed event and one catch-up run per target | | Consistency | Live volumes are crash-consistent, not application-consistent. Use a command backup for anything with its own consistency | | One node per volume | A volume is backed up from the node that holds it. A volume mounted from a file server with `driver_opts` is read through that node's mount, so back up the file server's own disk where you can | diff --git a/docs/configuration.md b/docs/configuration.md index a7e9ef2..8c0ef3a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -26,6 +26,11 @@ The CLI config is a YAML or JSON file that is safe to commit. It holds the contr |---|---|---| | `PORT` | `8443` | The single mTLS port. | | `TUG_CONTROL_URL` | required | Credential-free `https://` origin clients connect to by default, including the port. It is written into credentials and its host is added to the server certificate, but clients verify the generated control name, not this address. | +| `TUG_LOG_FORMAT` | `text` | Process log format: `text` or `json`; applies to control and agents. | +| `TUG_LOG_LEVEL` | `info` | Minimum process log level: `debug`, `info`, `warn`, `error`; applies to control and agents. | +| `TUG_WRITER_MAX_PENDING` | `100` | Positive maximum number of waiting control-writer operations. Excess admission returns 503 with `Retry-After: 5`; active work continues. | +| `TUG_WEBHOOK_URL` | unset | HTTPS notification receiver; requires `TUG_WEBHOOK_SECRET`. Invalid optional configuration disables notifications and logs an error. See [Alerting](observability.md#alerting). | +| `TUG_WEBHOOK_SECRET` | unset | HMAC-SHA256 signing secret; requires `TUG_WEBHOOK_URL`. Never place it in a URL. | | `TUG_WATCH_TICK_MS` | `1000` | How often the [watch channel](architecture.md#operator-watch-channel) looks for changes nothing announced (deployment, backup and restore states, stale nodes), while at least one operator is watching. Requests that change something trigger a look at once. | | `TUG_EXTRA_SANS` | unset | Comma-separated extra DNS names or IPs for the server certificate. | | `TUG_RENEW_MIN_INTERVAL_MS` | `3600000` | Minimum time between certificate renewals per node. | @@ -48,8 +53,16 @@ The CLI config is a YAML or JSON file that is safe to commit. It holds the contr | `TUG_ACME_EMAIL` | unset | Email for ACME registration in rendered Caddy configs. | | `TUG_INGRESS_ACCESS_LOG` | unset (off) | Set `true` and each ingress site writes a JSON access log line per request to the `tug-caddy` container's output (`docker logs tug-caddy` on the ingress node). Caddy redacts cookies and `Authorization` headers, but URLs (including query-string tokens) and client IPs are logged. Docker's log cap applies only to a newly created container; see [Networking](ingress.md#edge-features). | +The global writer limit applies to background work too: automatic rollback can be refused with a `deployment.auto_rollback_skipped` event; image maintenance logs its refusal and is retried after later deploys. Control backup failures remain visible in backup status and notifications. A full queue does not cancel active work. Retry operator work after the five-second hint; an automatically skipped rollback may require an explicit `tug rollback` once pressure clears. Readiness reports overdue active writer work after 35 minutes. + +## External reachability probe + +The source-checkout tool `node tools/scripts/check-reachability.mjs` takes URLs as arguments or from `TUG_PROBE_URLS` (comma/newline-separated). `TUG_PROBE_TIMEOUT_MS` defaults to `5000`. It requires a final 2xx response after at most five same-host redirects and exits nonzero on failure. Schedule it outside workload hosts; see [Alerting](observability.md#alerting). + ## Agent +Agents also honor `TUG_LOG_FORMAT` and `TUG_LOG_LEVEL` with the defaults above. + | Variable | Default | Purpose | |---|---|---| | `TUG_CONTROL_URL` | unset | Connect-only override of the control address (for example a private endpoint); the server is verified by its generated control name. | diff --git a/docs/development.md b/docs/development.md index 7a27d91..e34a02a 100644 --- a/docs/development.md +++ b/docs/development.md @@ -159,3 +159,5 @@ Run the agent in a container, as the installers do (`--network host`, the Docker The interface follows the conventions of `wp-release`: `tug ` operations with the everyday project commands at the top level, kebab-case command and flag names, primary targets as positional arguments, named qualifiers with consistent meanings, and positive camelCase internal options (`--no-wait` becomes `wait: false`, never `noWait`). New structured CLI results use camelCase, and API responses printed unchanged may keep their database field names. Help, docs and tests change together when the interface changes. CLI adapters resolve configuration, parse arguments and format results. Build, archive, installation and API primitives receive explicit values and injectable runners, and they do not discover CLI config or read unrelated environment variables. Keep domain logic in ordinary module functions, and add stateful classes only when a multi-step workflow needs shared state. Only operations that use Docker or SSH require those tools. Validation completes before side effects, and stdout meant as a runnable artifact or JSON stays directly usable. + +The QEMU first-boot harness accepts `--verify-reliability` to require authenticated `/api/v1/readyz` and node-report metrics after enrollment. `TUG_LOG_FORMAT=json` exercises structured control logs during that run. Use a disposable work directory; the default cleans up the guest and source control process, and `--keep` deliberately leaves them running. diff --git a/docs/networking.md b/docs/networking.md index 27d7bbd..05e6503 100644 --- a/docs/networking.md +++ b/docs/networking.md @@ -121,3 +121,5 @@ sequenceDiagram ## Verifying `npm run test:integration:network` starts three privileged fake VPSes with plain numeric node `address` values (`vps-c` as ingress node), deploys `tests/integration/compose/docker-compose.network-sample.yml`, and checks a container on `vps-b` fetching `http://web:8080` by its cross-node service name, Caddy on `vps-c` serving `app.test` from `vps-a`, and published ports bound to the intended private node addresses. See [Development](development.md#integration-test). + +Cross-node host entry ordering is preserved across this release, including the first address a resolver may select. Endpoint appearance, removal or address change updates the hosts file by replacing the affected container. Excluding those changes from identity would preserve stale name resolution; tug does not provide an independent DNS service or a stable service VIP. diff --git a/docs/observability.md b/docs/observability.md index 6b90c19..0f7185f 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1,6 +1,6 @@ # Observability -**Status: mostly proposed.** tug today has an audit trail, a few read commands and a certificate and connectivity check, and nothing that measures CPU, memory, requests or latency. Sections marked **Proposed** describe a setup that is not built; the milestone is [M9](roadmap.md#m9--observability). Nothing marked proposed works yet, and none of the names, flags or environment variables in those sections exist in the code. +**Status: minimal operational telemetry implemented.** Metrics, readiness, structured logs and best-effort webhooks cover control operations. Continuous resource and ingress request telemetry remain external. The aim is the smallest setup that answers the day-to-day questions: is every node reporting, did the last deployment succeed, did an automatic rollback happen, will a certificate expire, is the control database growing. It follows the shape of the rest of tug: one central control plane, agents that only dial out, and no required extra services. @@ -18,10 +18,10 @@ Verified against the code on `main`. | `tug doctor` | Local checks (Node.js and Docker), control checks (SQLite, CA and server certificate expiry, each node's certificate expiry and connectivity, each online node's observed clock skew, master key, Docker, ingress state, ports 80 and 443 on ingress nodes, and control's load, memory, disk and container CPU and memory) and, with `--nodes` or `--node`, host diagnostics through each agent, including the node's load, memory, free disk and the CPU and memory of the agent and Caddy containers. A local Docker daemon is checked only for `--build-local` and prints `skip` when absent. See [`tug doctor`](cli.md#tug-doctor). It waits for those results, prints one line per check (the full document with `--json`) and exits nonzero when a check fails. Clock skew is unknown, not zero, for a node that is offline. | A point-in-time check, not a time series. Resource values are one sample (CPU is measured over about a second) and have no thresholds | | `tug logs ` | Prints a service log tail; `--follow` streams logs through the live channel. | Following logs requires an active WebSocket session. | | `tug ingress logs `, `tug ingress tls` | The ingress node's Caddy output, and per-host certificate state from a TLS handshake against that Caddy, shown also in `domain list`, `status` and `doctor`. See [Certificate status](ingress.md#certificate-status). | `admin` only. State is a probe every twenty seconds to an hour, not a stream | -| Reachability | Control has no unauthenticated route, not even a health check: every request needs a client certificate. Monitor TCP reachability of the mTLS port for liveness, and use an authenticated read (`tug doctor`, `tug status`) for readiness. | A TCP check proves the listener accepts connections, not that the database works | -| Process logs | Control and agent write plain `console.log` and `console.error` text. | No timestamps, levels, structure or request ids | +| Reachability | Control has no unauthenticated route, not even a health check: every request needs a client certificate. Monitor TCP reachability of the mTLS port for liveness, and use authenticated `GET /api/v1/readyz` for control readiness ([Health endpoints](#health-endpoints)). | A TCP check proves the listener accepts connections, not that the database works | +| Process logs | Text or JSON records with levels and available operation context. | No integrated log retention or search | -What does not exist: a `/metrics` endpoint, any history of CPU, memory or disk (`tug doctor` shows the current values only), restart data, request, build or poll metrics, structured or leveled logs, alerting, outbound webhooks (M8, unchecked) and a dashboard (M8, unchecked). The managed Caddy ingress enables Caddy's own metrics on the ingress node's loopback admin endpoint, but nothing collects them yet ([Ingress metrics](#ingress-metrics)). +Continuous resource history, application request metrics and an integrated dashboard are not collected by tug. Managed Caddy enables its own loopback metrics; see [Ingress metrics](#ingress-metrics). ### Events @@ -31,186 +31,74 @@ Before [PR #13](https://github.com/wpelevator/tug/pull/13) agents emitted those Containers that crash, are killed for memory or report unhealthy produce `container.exited`, `container.oom`, `container.unhealthy` and, for repeated crashes, one `container.crashloop` event, from the Docker daemon's event stream rather than from the next poll ([Architecture](architecture.md#container-events-from-the-docker-daemon)). Restart counts and a restart-rate metric are still proposed under [node and service stats](#node-and-service-stats-reported-by-agents). -## Proposed +## Structured logs -Everything below is **Proposed** unless a status line says otherwise. +Control and agents write text logs by default at level `info`. Set `TUG_LOG_FORMAT=json` for one JSON object per line, or `TUG_LOG_LEVEL=debug|info|warn|error` to select the minimum level. JSON records include timestamp, level, component and a fixed message, with available project, deployment and node context. Control responses include `x-request-id`; the matching completion record includes that ID, method, route pattern, status and duration. Request bodies and query strings are excluded. Default logs exclude raw error details; `TUG_LOG_LEVEL=debug` adds a bounded cause with URLs, common credential assignments, private-key blocks and control characters sanitized. Pattern sanitization cannot identify every application secret, so enable debug deliberately and restrict access to those logs. Routine successful agent requests use debug; operator requests, failures and aborted requests remain visible at info/warn. Allowlisted failure classes and codes, HTTP status and numeric SQLite codes retain diagnostic categories without arbitrary exception messages. Sensitive context keys are redacted, but a secret deliberately included in an arbitrary message or resource name is outside that protection. Logs are process output for your existing collector; tug does not store another log database. -| Piece | Section | Status | -| --- | --- | --- | -| Structured logs | [Structured logs](#structured-logs) | Proposed | -| Metrics on control | [Metrics on control](#metrics-on-control) | Proposed | -| Node and service stats from agents | [Node and service stats reported by agents](#node-and-service-stats-reported-by-agents) | Proposed | -| Readiness endpoint | [Health endpoints](#health-endpoints) | Proposed | -| Ingress metrics | [Ingress metrics](#ingress-metrics) | Proposed, with open questions | -| Sample alerts and webhooks | [Alerting](#alerting) | Proposed | -| Live commands | [Live commands](#live-commands) | Partly built in M5 | -| Metrics test gate | [Test gate](#test-gate) | Proposed | - -### Structured logs - -Status: proposed. - -Control and agent would write one JSON object per line with `time`, `level`, `component`, `msg` and context fields where they apply: `requestId`, `node`, `project`, `service`, `deploymentId`. Two environment variables would select the output: `TUG_LOG_FORMAT=json|text` and `TUG_LOG_LEVEL` (`debug`, `info`, `warn`, `error`). Text would stay the default so existing installs read the same, and JSON is meant for `docker logs` piped to whatever the operator already uses. - -Credential private keys, join keys and secret values are never logged, and the logger would redact known field names as a second line of defence. The CLI keeps its human-readable output; only the long-running processes change. Every existing `console.log` and `console.error` in `src/control/control.ts` and `src/agent/agent.ts` would move to the logger. - -### Metrics on control - -Status: proposed. - -Control would serve `GET /metrics` in the Prometheus text format. It needs the `read` role from [Access control](access-control.md#roles), so Prometheus would present the client certificate and key of a `read` operator credential (`tls_config` with `cert_file`, `key_file` and `ca_file` extracted from the credential file); a credential limited to projects would see only those projects' series. Adding the route means adding it to `ROUTE_ROLES` in `src/control/credentials.ts` and to the endpoint table, which the existing test already requires. - -Naming follows the Prometheus conventions: a `tug_` prefix, base units (`_seconds`, `_bytes`), `_total` for counters. Labels are bounded: project, node and service names are operator-defined and few; route labels use the route pattern (`/api/v1/deployments/:id`), never the raw path; deployment IDs, command IDs, versions and error messages are never labels. - -| Metric | Type | Labels | Meaning | -| --- | --- | --- | --- | -| `tug_build_info` | gauge | `version` | Constant 1; the running control version | -| `tug_deployments` | gauge | `project`, `state` | Deployments currently in each state (`building`, `rolling_out`, `succeeded`, `failed`, `build_failed`, `cancelled`) | -| `tug_deployment_duration_seconds` | histogram | `project`, `outcome` | Time from creation to a terminal state | -| `tug_rollout_outcomes_total` | counter | `project`, `outcome` | Rollout results: `rolled_out`, `rejected` (a candidate failed its health check while the old revision kept serving), `auto_rollback` | -| `tug_nodes` | gauge | `status` | Registered nodes by `online` or `stale`, computed from `last_seen_at` at scrape time rather than stored | -| `tug_node_poll_age_seconds` | gauge | `node` | Seconds since the node last reported | -| `tug_api_requests_total` | counter | `route`, `method`, `status_class` | API requests by route pattern and status class (`2xx`, `4xx`, `5xx`) | -| `tug_api_request_duration_seconds` | histogram | `route` | API request latency | -| `tug_builds_total` | counter | `outcome` | Builds by result (`complete`, `failed`) | -| `tug_build_duration_seconds` | histogram | none | Remote build duration | -| `tug_image_archive_bytes` | gauge | none | Total size of retained image archives under the data directory | -| `tug_db_size_bytes` | gauge | none | Size of the control SQLite database file | -| `tug_events_rows` | gauge | none | Rows in the `events` table | -| `tug_project_locks` | gauge | none | Project operation locks currently held | -| `tug_writer_queue_depth` | gauge | none | Operations waiting for the single writer | - -Gauges that describe state (`tug_deployments`, `tug_nodes`, `tug_events_rows`, `tug_project_locks`) would be computed from the database at scrape time, so they stay correct across control restarts. Counters and histograms live in process memory and reset on restart, which Prometheus handles as a counter reset. - -### Node and service stats reported by agents - -Status: proposed. - -Agents already send a status report on every poll. The proposal adds a `stats` block to that report and has control expose it on the same `/metrics` endpoint, so one scrape target covers everything: - -| Metric | Type | Labels | Meaning | -| --- | --- | --- | --- | -| `tug_service_cpu_seconds_total` | counter | `node`, `project`, `service` | CPU time used by the service's containers | -| `tug_service_memory_bytes` | gauge | `node`, `project`, `service` | Memory in use | -| `tug_service_up` | gauge | `node`, `project`, `service` | 1 when the container is running, 0 otherwise | -| `tug_service_restarts_total` | counter | `node`, `project`, `service` | Container restarts since the container was created | -| `tug_node_disk_free_bytes` | gauge | `node`, `path` | Free space on the Docker data and tug state paths | -| `tug_node_load1` | gauge | `node` | One-minute load average | - -The agent would read these from `docker stats --no-stream` and `docker inspect`, and control would drop series for services that no longer exist. A node that stops reporting keeps its last values, so alerts use `tug_node_poll_age_seconds` to tell fresh data from stale data. - -Why report through control: agents sit behind NAT and only dial out, so Prometheus often cannot reach them. Reporting through the existing authenticated channel needs no inbound port, no extra credential and no firewall change, and adds one scrape target regardless of node count. The cost is resolution: values are as fresh as the poll interval (`TUG_POLL_INTERVAL_MS`), and the status report grows with the number of services. +## Metrics on control -The alternative is scraping an exporter on each agent directly over the operator's private network from [Networking](networking.md#bring-your-own-network). It gives finer resolution and keeps control out of the data path, but it requires a listener on every node, firewall rules that admit only Prometheus and a service-discovery list. It does not work where Prometheus cannot route to the node addresses, including nodes behind NAT. If both are ever wanted, the metric names above can be reused so dashboards do not change. +Scrape authenticated `GET /api/v1/metrics` using a `read` credential over mTLS. Scoped credentials see only their projects' deployment and backup series; unscoped credentials also see nodes, control writer and webhook delivery. No deployment IDs, command IDs, error messages or raw URLs are metric labels. State counts come from SQLite and survive restarts; webhook delivery counters reset on restart; pending delivery is a gauge. -### Health endpoints - -Status: proposed. - -Control has no unauthenticated route (there is no `/healthz`), so liveness is TCP reachability of the mTLS port and the container restart policy. An authenticated `GET /api/v1/readyz` with the `read` role would return 200 only when control can do its job: it can read the database, and the writer queue is not stuck (the operation at the head of the queue has not been running longer than a limit). Otherwise it returns 503 with the names of the failing checks and nothing more. An external uptime check would present a `read` credential. - -### Ingress metrics - -Status: Caddy's metrics are enabled on the ingress node's loopback admin endpoint; the status-report and control re-export below are proposed, with open questions. - -The managed Caddy enables its metrics with the global option `metrics { per_host }` in the generated Caddyfile (see `renderCaddyfile` in `src/ingress.ts`). They are served at `/metrics` on Caddy's admin endpoint, which listens on `localhost:2019` and, because the container uses host networking, is reachable on the ingress node's loopback only (`curl http://127.0.0.1:2019/metrics` there, or a local exporter or agent). A request-count series carries a `host` label, so cardinality is bounded by the number of routes. That gives request counts by status class and request duration histograms. To reach the same scrape target, the agent on the ingress node would read the admin metrics and include a filtered subset (request counts, status classes, latency buckets, certificate expiry per host) in its status report, and control would re-export it as `tug_ingress_*` series labelled by `node` and `host`. - -Open questions: - -- Whether re-exporting parsed Caddy metrics through the status report is worth the complexity, compared with publishing Caddy's admin metrics port on a private node address, firewalling it to Prometheus and scraping it directly. -- Which certificate expiry signal Caddy exposes reliably across versions; the alert below assumes a `tug_ingress_certificate_expiry_timestamp_seconds` gauge that may have to be derived from the certificate files. -- Label cardinality: one `host` label per route is bounded by the number of routes, but per-upstream labels are not planned. - -### Alerting +| Metric | Meaning | +|---|---| +| `tug_node_reported` | Whether the node has ever reported; check this before interpreting poll age | +| `tug_node_poll_age_seconds` | Age of its last report, `NaN` when absent | +| `tug_deployments` | Recorded deployments grouped by project and state, including retained history | +| `tug_backup_targets` | Enabled backup targets grouped by project and latest run state | +| `tug_backup_has_success` | Whether a configured target has a successful snapshot; zero means protection has not been established | +| `tug_backup_success_age_seconds` | Age of the target's last success, `NaN` when absent; use with `has_success` | +| `tug_writer_queue_depth` | Operations waiting for the single control writer | +| `tug_writer_active_age_seconds` | Age of active writer work, zero when idle | +| `tug_webhook_enabled` | Whether optional notification delivery is configured and enabled | +| `tug_webhook_delivered_total`, `tug_webhook_failed_total`, `tug_webhook_dropped_total`, `tug_webhook_pending` | Delivery outcomes, overflow and current backlog | -Status: proposed. - -tug would not bundle Prometheus or Alertmanager. A sample scrape configuration: +Prometheus needs the CA, certificate and key from a read credential: ```yaml scrape_configs: - job_name: tug - metrics_path: /metrics + metrics_path: /api/v1/metrics scheme: https tls_config: ca_file: /etc/prometheus/tug-ca.pem cert_file: /etc/prometheus/tug-read.crt key_file: /etc/prometheus/tug-read.key static_configs: - - targets: ['control.example.com'] -``` - -Sample alert rules, written against the proposed metric names: - -```yaml -groups: - - name: tug - rules: - - alert: TugNodeStale - expr: tug_node_poll_age_seconds > 120 - for: 2m - annotations: { summary: "Node {{ $labels.node }} has not reported for over two minutes" } - - alert: TugDeploymentFailed - expr: increase(tug_deployment_duration_seconds_count{outcome=~"failed|build_failed"}[10m]) > 0 - annotations: { summary: "A deployment of {{ $labels.project }} failed" } - - alert: TugAutomaticRollback - expr: increase(tug_rollout_outcomes_total{outcome="auto_rollback"}[10m]) > 0 - annotations: { summary: "{{ $labels.project }} was rolled back automatically" } - - alert: TugCertificateExpiresSoon - expr: tug_ingress_certificate_expiry_timestamp_seconds - time() < 14 * 86400 - annotations: { summary: "Certificate for {{ $labels.host }} expires in under 14 days" } - - alert: TugDatabaseGrowing - expr: predict_linear(tug_db_size_bytes[6h], 7 * 86400) > 5 * 1024 ^ 3 - annotations: { summary: "Control database is on track to pass 5 GiB within a week" } - - alert: TugAgentPollAgeHigh - expr: tug_node_poll_age_seconds > 30 - for: 10m - annotations: { summary: "Node {{ $labels.node }} polls much slower than its interval" } + - targets: ['control.example.com:8443'] ``` -Thresholds are starting points. The last rule assumes the default poll interval and should be set from `TUG_POLL_INTERVAL_MS`. - -Outbound webhooks (M8) are the lightweight alternative for operators without Prometheus: control would POST a signed JSON body to configured URLs. The events worth sending are a deployment reaching `succeeded`, `failed`, `build_failed` or `cancelled`, a rollout candidate rejected, an automatic rollback, a node going stale and recovering, and an ingress node reporting `failed`. Webhooks send state changes, never metrics, and depend on event hygiene so that an event always means something happened. +### Health endpoints -### Live commands +Authenticated `GET /api/v1/readyz` returns 200 when the persisted projects table can be read and active writer work has not exceeded 35 minutes. This gives default 30-minute operation deadlines a five-minute margin. It returns 503 with sanitized check states when those checks fail, without waiting behind the writer. The read probe does not establish database writeability. After control restore closes the database, the existing generic restart-required 503 takes precedence until restart. Monitor this endpoint externally: a dead control cannot notify you of its own failure. A TCP connection to the mTLS port establishes listener liveness only. -Status: non-TTY container commands built in M5; agent/daemon logs and `top` remain proposed. +## Alerting -`tug logs ` keeps its durable tail through the command queue, while `-f` follows the current container over the live channel. `tug exec`, `tug run`, `tug ps` and `tug inspect ` use that channel too; `exec`, `run` and `logs -f` data is flow-controlled end to end, so piping large input or output through it works ([Architecture](architecture.md#agent-wake-channel)). Agent and daemon log streams are still proposed for troubleshooting. These commands answer what is happening right now on a node and complement metrics, which answer what has been happening. An optional `tug top` would render the agent-reported stats as a refreshing table of CPU, memory and restarts per service, reading `GET /metrics` or a read-only JSON view rather than a new data path. +Set both `TUG_WEBHOOK_URL` (an HTTPS receiver) and `TUG_WEBHOOK_SECRET` on control to enable notifications. Leaving both unset disables delivery. An incomplete or invalid configuration logs a fixed error and disables notifications while control remains available; `tug_webhook_enabled` reports zero. Control checks committed events and deployment/node transitions every five seconds, independently of operator views. Never-seen nodes do not generate stale/recovered alerts. After startup, node notifications wait for the greater of 60 seconds and the configured stale threshold so agents can reconnect without an upgrade producing a fleet-wide page. It sends actionable generic descriptions and resource identifiers for supported failures, backup results and stale/recovered nodes; raw event messages, commands and metadata are excluded. The observer processes up to 100 new events each tick and the latest 1000 deployment records; this is sized for tug’s small-server scope and is not a durable incident ledger. -### Test gate +Delivery runs outside the writer, with at most 100 pending/in-flight events, three attempts, a five-second timeout per attempt and exponential delays starting at 250 milliseconds capped at two seconds. Redirects are not followed. Shutdown aborts active delivery and discards queued work. Retries reuse the event ID; receivers should deduplicate it. An exhausted attempt or queue overflow is visible in metrics and delivery exhaustion is logged. The queue and event cursor are in memory: restart drops queued work and does not replay historical events. This is best-effort notification, so retain an independent availability check and schedule `tug backup list --check` as a protection backstop. -Status: proposed. +Each POST body is JSON with `timestamp` and `event`. Verify `x-tug-webhook-signature` against `sha256=` followed by the hex HMAC-SHA256 of `.`, using the configured secret. Reject timestamps outside your replay window and compare signatures in constant time. `x-tug-webhook-id` identifies the event. Treat this as a control-wide integration: the receiver sees identifiers across projects. -Modelled on madplay's `@magplay/observability` package: metrics sit behind a small injectable interface whose default is a no-op, so unit tests of control and agent logic need no registry; a small closed outcome taxonomy (`success`, `failure`, `timeout`, `rejected`, `cancelled`) with one helper that maps an error to an outcome; and a CI check (`test:observability`) that fails when the code and the documentation disagree. +For an external reachability check from a source checkout: -For tug the equivalent check would assert: +```sh +node tools/scripts/check-reachability.mjs https://blog.example.com/health +``` -- every metric registered in code appears in this page's metric tables with the same name, type and label set, and every row in the tables is registered in code; -- every label name is on an allow-list, and none is `id`, `version`, `error`, `path` or `message`, so cardinality stays bounded; -- `GET /metrics` with a certificate for an unknown key returns 401 and with a `read` credential returns 200, and `ROUTE_ROLES` lists the route; -- a scrape after a credential is created and used contains no credential private key, join key or secret value; -- with the no-op default, the existing control and agent tests behave unchanged. +Run it outside the workload host on a schedule and route a nonzero exit to your notification system. The script accepts URLs as arguments or comma/newline-separated `TUG_PROBE_URLS`; `TUG_PROBE_TIMEOUT_MS` defaults to 5000. It requires a final 2xx response, follows up to five same-host redirects (including HTTP-to-HTTPS), rejects HTTPS downgrades and redirects to another host or port, removes query strings from output, rejects credentialed URLs, and exits 1 for a failed probe or 2 when no URLs are configured. Use the canonical URL when a site redirects to another hostname. Choose an application endpoint that exercises important dependencies; an HTTP success alone cannot establish business correctness. -## Rollout order +## Node and service stats reported by agents -1. Event hygiene (done, [PR #13](https://github.com/wpelevator/tug/pull/13)). Independent of everything else. -2. Structured logs and the `/metrics` endpoint on control with the control-side metrics and the readiness endpoint, plus the test gate so later metrics cannot ship undocumented. -3. Agent-reported node and service stats, then ingress metrics once the open questions are settled. -4. Sample alert rules (they need steps 2 and 3) and outbound webhooks. +Continuous CPU, memory, disk and restart series remain outside this minimal slice. `tug doctor --nodes` supplies point-in-time resource diagnostics. Use node_exporter or cAdvisor when you need resource history. Agent-reported aggregation through control remains a possible extension for nodes behind NAT. -Dependencies: the remaining agent/daemon live log streams and TTY backend extend M5, while webhooks and the dashboard are M8 items. Steps 1 to 3 need neither. +## Ingress metrics -## Open questions and non-goals +Caddy exposes per-host metrics on its loopback admin endpoint; tug does not collect or re-export them. Use a collector on the ingress host or private network. Keep Caddy's admin endpoint private. Request latency, traffic and error-rate objectives belong at this application boundary, separately from tug's deployment state. -Open questions: +## Live commands -- Where the first dashboard JSON would live: in this repository (for example a `dashboards/` directory) or published separately, and whether the M8 read-only web dashboard makes it unnecessary. -- Whether agents should send stats on every poll or on a slower interval of their own. -- Whether `/metrics` should accept the `root` admin credential as well as named `read` credentials (an admin passes every role check today). -- How the ingress metrics reach control (see [Ingress metrics](#ingress-metrics)). -- Whether control should keep a short history of stats for `tug top` and a dashboard without Prometheus. +`tug logs`, `logs -f`, `exec`, `run`, `ps` and `inspect` supply current evidence. Streaming stdin/stdout has end-to-end flow control. Agent/daemon log streaming through the control API and a `top` view remain proposed; process logs are available through Docker today. -Non-goals: a log aggregation system (JSON on stdout is the interface), distributed tracing, and a bundled Prometheus, Alertmanager or Grafana. tug would expose metrics and document how to scrape them; running the monitoring stack is the operator's choice. +## Test gate -See also [Architecture](architecture.md), [Control plane](control-plane.md), [Networking](networking.md), [Testing](testing.md) and the [Roadmap](roadmap.md). +Renderer tests cover finite states, label escaping, missing observations and success history. HTTP authorization tests verify project filtering and exclusion of global node/writer state. Notification tests cover signing, sanitization, queue bounds, deduplication and retries. Readiness is tested with a blocked real deployment, an injected clock, database failure and recovery. Fault-test evidence and remaining runtime coverage are in [Testing](testing.md). diff --git a/docs/production-checklist.md b/docs/production-checklist.md index 984a079..49e8922 100644 --- a/docs/production-checklist.md +++ b/docs/production-checklist.md @@ -23,11 +23,25 @@ tug backup list --check ## After launch -- Look at `tug status`, `tug events` and `tug backup list --check` on a schedule. No backup notification is pushed automatically. +- Schedule `tug status`, `tug events` and `tug backup list --check`; route nonzero backup-check exits to an operator. Configure [signed notifications](observability.md#alerting) and monitor delivery failures; delivery is best effort, so retain the scheduled backup check. +- Treat workload health and user impact as separate checks. `tug status` reports the last node observation and service health; verify the public ingress URL from outside the node when customer reachability matters. A healthy container does not prove DNS, TLS, ingress routing or an upstream dependency works. +- Keep an operator path to the control host, node access, backup destination credentials and repository passwords. Use the matrix below to decide what to check and what evidence closes an incident. - Watch expiry. Operator credentials last at most 365 days, and `tug doctor` warns before the CA, the server certificate and node certificates expire. Agents renew node certificates themselves. - Upgrade control and nodes together with [`tug upgrade`](upgrades.md), and run `tug doctor` afterwards. - Know the limits. `tug rollback` cannot undo database writes or schema changes, and it does not roll back the host. One control plane is one point of failure for operations, though running containers keep serving while it is down. See [Not for you if](https://github.com/wpelevator/tug/blob/main/README.md#not-for-you-if). +## Failure response and recovery evidence + +| Failure | Expected impact and responsibility | Operator action | Evidence to close the incident | +|---|---|---|---| +| Control host or process is unavailable | Existing containers can keep serving, but tug operations, new deployment work, agent coordination and control-hosted recovery material are unavailable. Restore the control host and its state; tug does not provide a second control plane. | Check the host and disk, restore control from an off-host archive using [control recovery](control-plane.md#recover-control-on-another-host), then let agents reconnect. | `tug status` reaches control, expected projects and nodes appear, agents report recently, and an external request to each important ingress host succeeds. | +| Node is permanently lost | Services on that node stop. tug does not reschedule a node-pinned service or copy its local volume to another node. The operator owns host replacement and workload-data recovery. | Replace or repair the node as appropriate, restore workload data from its backup or recover the host, and follow [migrating to a new server](migration.md). | The replacement node reports online, the service and its data are restored, and an external request to the service succeeds. | +| Control-to-node network path is partitioned | The last node report may be stale; workloads already running may still serve, while reconciliation, deploys and node-local backup work cannot complete. The operator owns host and network repair. | Check `tug status`, `tug doctor --nodes`, host reachability and the provider firewall; restore the path and let the agent reconcile. | The node reports again, unchanged desired services are healthy, and an external request to affected services succeeds. | +| Node or control data disk is full | Writes, image transfers, logs or backup jobs may fail; existing workload responses can continue until their own storage is affected. tug does not free operator data or guarantee a safe cleanup. | Check free space with `tug doctor --nodes` and host tools; identify the consuming data before deleting anything. Restore capacity, then retry the failed operation or backup. | Free space is available, the failed operation completes, and `tug backup list --check` passes for protected targets. | +| Backup destination is unavailable | Running workloads are not stopped by a destination outage, but scheduled workload and control backups cannot become recoverable copies. Configured notifications report backup failures and stale protection; a dead control still needs an external monitor. | Alert on scheduled `tug backup list --check` failures; verify destination reachability and credentials, then run `tug backup run` and `tug control backup list`. | New workload snapshots pass `backup list --check`; a control archive is present off-host in `tug control backup list`. A restore drill is separate evidence that contents are usable. | + +These checks define observable recovery evidence, not recovery-time or data-loss guarantees. Set such targets only after repeated exercises on representative hosts and data. + ## Next steps When something fails, start with [Troubleshooting](troubleshooting.md). diff --git a/docs/real-host-evaluation.md b/docs/real-host-evaluation.md index ad1a161..a10378d 100644 --- a/docs/real-host-evaluation.md +++ b/docs/real-host-evaluation.md @@ -521,3 +521,26 @@ Disposable Ubuntu 24.04.5 arm64 guests ran under QEMU 11.1.3 with HVF on Apple S The empty `env_file: []` fixture exposed a diagnostic defect: the rejection suggests `tug secret import undefined`. The fixture was removed from the upgrade runner because v3 refuses `env_file`; the product diagnostic was not changed. Interrupted control activation, corrupted repositories, workload recovery on a replacement node, database content recovery, network partitions, migration into staging, monitoring-system integration and public DNS/ACME were not exercised in this run. The recovery drivers were scratch scripts; only first boot and the upgrade rehearsal have committed QEMU runners. Keep these results separate from automated coverage of those missing scenarios. + +## Reliability control smoke on QEMU, 16 November 2026 + +A disposable Ubuntu 24.04 arm64 cloud guest enrolled through cloud-init against the current source control process with JSON logging. The official image matched its published SHA256SUMS; the checksum file was not independently signature-verified. The harness reported enrollment and successful authenticated readiness and node-report metrics over mTLS at 115 seconds. The guest used the published agent image, so this run establishes current control behavior and enrollment compatibility, not validation of the modified agent logger or a workload restore. The harness cleans up the guest and control processes by default. + +```sh +TUG_LOG_FORMAT=json node tools/scripts/qemu-first-boot.js --image /tmp/tug-sre-vm/ubuntu-24.04-server-cloudimg-arm64.img --work /tmp/tug-sre-vm/final --port 18444 --ssh-port 12223 --timeout 600 --verify-reliability +``` + +```text +[ 115s] SUCCESS: first-boot cloud-init enrolled the node +[ 115s] SUCCESS: authenticated readiness and node-report metrics verified over mTLS +``` + +The new damaged-repository backup drill was not executed: this host has a Docker client but no daemon socket. Unit-level injected disk exhaustion and interrupted activation tests supply narrower failure evidence; they do not substitute for a Docker or physically full-host exercise. + +## Source-built agent logging on QEMU, 16 November 2026 + +A second disposable Ubuntu 24.04.5 arm64 guest ran Docker 29.10.0. After published-agent enrollment and readiness/metrics checks, the current source was packaged from explicit Dockerfile inputs, built inside the guest, and used to replace only its agent while preserving the state volume, socket bind, networking and restart policy. The source-built agent emitted valid JSON startup records and remained online and connected to the throwaway control. Debug failure checks inside that image confirmed that synthetic bearer/basic credentials, cookie/token values, URL credentials/query strings and control characters were excluded from the diagnostic cause. These checks exercise known patterns, not arbitrary application-secret recognition. + +The built image was `sha256:863017764465b231d38e770ac53ed9fef625d711da4f9d86f3f4f190ce6b3b29`. Its logger SHA-256, `5f6be459cf42d4dcfd6d760e57d1e3ef7d3daae603a5d905792cc4e0abfd38a5`, matched the pushed logger source. The source archive SHA-256 was `e4e8f8f3afc1cdabc10ce83e44c750928e52d6894a7b639ff7032ccdaea212e8`. The guest, throwaway control, rollback container, temporary credentials and build artifacts were removed after verification. + +The damaged-repository backup drill also [passed in CI](https://github.com/wpelevator/tug/actions/runs/43955052052) after its root-owned config fixture was corrected. That run passed the complete Node and integration matrix; it supplies Docker evidence in addition to the local injected-failure tests. diff --git a/docs/roadmap.md b/docs/roadmap.md index b4c74b9..74834db 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -95,11 +95,11 @@ An `x-tug.type: nfs` volume type was built and verified on real hosts, then remo ### M8 — Central API advantages (partly done) -Shipped: scoped operator credentials (`read`, `deploy`, `admin`, per project), a GitHub Actions deploy example and credential names in the audit trail ([Access control](access-control.md)). Remaining: outbound webhooks for deployment state changes, failed health checks and backup results (shared with M9 alerting). The read-only web dashboard is dropped to [Later](#later--only-if-needed): `tug status`, `/metrics` and webhooks cover the need. +Shipped: scoped operator credentials (`read`, `deploy`, `admin`, per project), a GitHub Actions deploy example and credential names in the audit trail ([Access control](access-control.md)). Shipped: best-effort signed outbound webhooks for actionable state transitions, with bounded retries and delivery metrics (shared with M9 alerting). The read-only web dashboard is dropped to [Later](#later--only-if-needed): `tug status`, `/metrics` and webhooks cover the need. ### M9 — Observability -Make behavior visible without adding required services: event hygiene (done), structured logs, a Prometheus `/metrics` endpoint on control that also carries node and service stats reported by agents (Prometheus cannot scrape nodes behind NAT, which is why agents report through control), a readiness endpoint, sample alert rules and webhooks. Design and status are in [Observability](observability.md). Keep agent-reported stats small; point operators who want depth at cAdvisor or node_exporter deployed as ordinary services. +Make behavior visible without adding required services: event hygiene (done), actionable notifications, structured logs, a small Prometheus `/metrics` endpoint on control, a readiness endpoint, sample alert rules and optional user-facing probes. Start with bounded signals for node report age, deployment outcomes, backup outcomes and control readiness; add resource statistics only when a recovery or diagnosis exercise demonstrates a need. Logs should identify the operation and affected resource while excluding credentials and secret values. The broader metric catalog in [Observability](observability.md) is a candidate inventory, not a commitment to ship every metric in the first slice. Point operators who want deeper resource detail at cAdvisor or node_exporter deployed as ordinary services. ### M10 — Private-by-default port exposure (done) @@ -170,13 +170,13 @@ Proposed, after 1.0, and additive, so every step is a minor. The `root` credenti tug serves a trusted operator or small trusted team running Compose applications on one or a few ordinary VPSes. The primary topology is an application and its state on one node, including several independent applications on one node; additional nodes first add capacity and placement for independent applications. A scaled single application is a later, demand-driven use case. Judge work by whether it makes installing, operating and recovering those applications easier and safer, not by feature parity or the number of orchestration milestones completed. -The control plane earns its cost through understandable state, reliable deployments, backup visibility and recovery. Prefer a proven, repeatable recovery journey over additional scheduling, networking or infrastructure features. Restic is the backup repository mechanism; tug's responsibility is to make backup status, credentials, off-host copies and actual restore outcomes understandable. Do not set recovery time or data-loss targets until a representative exercise measures them. +The control plane earns its cost through understandable state, reliable deployments, backup visibility and recovery. Prefer a proven, repeatable recovery journey over additional scheduling, networking or infrastructure features. Restic is the backup repository mechanism; tug's responsibility is to make backup status, credentials, off-host copies and actual restore outcomes understandable. Historical QEMU recovery exercises have recovered a lost workload node and control plane; they do not cover every failure mode. Do not set recovery time or data-loss targets until repeated representative exercises measure them. Deployment authors are trusted infrastructure operators. Project-scoped credentials are operational least-privilege controls that reduce accidental access and limit a compromised credential's API authority; they do not isolate mutually untrusted code, shared kernels, application networks, builds or workload secrets. An operator who can deploy or execute code must be trusted with the workload's secrets and the nodes they can select. Use separate hosts or control planes for mutually untrusted authors. Continue fixing concrete authorization, confidentiality and data-safety defects, but do not add a complex tenant-isolation policy without an explicit user need and a coherent end-to-end threat model. An earlier project node/hostname grants and host-feature restrictions experiment remains separate from the recovery-first branch; evaluate it against this contract before adopting it. It is not a product decision to support hostile tenants. Every subsystem tug owns carries maintenance and failure costs. Keep the outbound agent while its NAT and unattended reconciliation benefits justify it, but a bounded SSH-controller experiment is legitimate if it preserves the operational guarantees users need. Compare control-hosted image archives with an existing registry on deployment, recovery, storage, security and operational cost before expanding either path. Keep the Compose subset explicit and predictable, use the operator's private network for cross-node workloads, and prefer simple placement over a general scheduler. Remove or narrow an existing feature if its cost exceeds its demonstrated value. -Claims must match verified behavior. Image rollback does not undo data, schema or external side effects; private port binding depends on the configured address and network; scoped credentials are not tenant isolation. Deployment success needs a documented definition that distinguishes node health from ingress applying the intended revision and, if relevant, an external probe. Each claim needs tests at the cheapest useful layer and real-host evidence for the end-to-end operating journey. +Claims must match verified behavior. Image rollback does not undo data, schema or external side effects; private port binding depends on the configured address and network; scoped credentials are not tenant isolation. Workload health on a node and user impact through DNS, TLS, ingress and dependencies are separate observations. Each claim needs tests at the cheapest useful layer and real-host evidence for the end-to-end operating journey. ## Sequencing @@ -185,12 +185,12 @@ Claims must match verified behavior. Image rollback does not undo data, schema o 3. **Done:** 1.0.0 tagged once the [gate below](#road-to-100) passed; 1.x minors since. 4. **Now, ordered by operator value** (working file `plan-operator-value.md`; a hands-on VM run and an outside product review on 9 November 2026 independently reached the same order): 1. **Move bulk data in and out.** End-to-end backpressure for `exec`/`run` stdin and stdout (every live-channel hop still caps at 1 MiB, and a slow reader cuts a download far sooner), tested at 100 MB in both directions with a hash check; a documented way to move a database or a directory, and a "move an existing site in" guide. - 2. **Hear first, and say what is true.** Outbound webhooks (the M8 slice: failed, missed or stale backups, crash loops, failed or rolled-back deployments, stale nodes) and an optional external HTTP probe per ingress host. - 3. **Backups you can trust.** `backup list --check` fails when nothing is protected, a protection level per target (local, off-host, verified, restore-tested), `restic check` and a destination test, cleanup of scratch volumes and safety snapshots, and a control-recovery drill covering a damaged repository, a crash during activation and agents reconnecting after an older snapshot. + 2. **Hear first, and say what is true.** Actionable notifications for failed, missed or stale backups, crash loops, failed or rolled-back deployments and stale nodes; an optional external HTTP probe per ingress host distinguishes service health from user reachability. Keep notifications actionable and include the affected resource, observed failure, and next operator check. + 3. **Backups you can trust.** `backup list --check` fails when nothing is protected, a protection level per target (local, off-host, verified, restore-tested), `restic check` and a destination test, cleanup of scratch volumes and safety snapshots, and fault exercises for destination outage, damaged repository, crash during control activation and agents reconnecting after an older snapshot. The historical QEMU recovery drill covers lost workload and control hosts; it is evidence for those paths, not these additional failure scenarios. 4. **Staging that is more than a pattern.** `tug project delete` and a cross-project data copy. 5. **Show the operator's scope.** Label control-wide views, show the resolved control and project before high-impact commands, give `ps`, `events` and `doctor` a compact human default. 6. **Done: control is the only source of deployment values.** Operators could not tell whether `${NAME}` came from control, their shell or `.env`. Now every value comes from the project store (secrets through `tug secret`, visible values such as image tags, hosts and ports through `tug var`, one name space), an explicit `deploy --var`, or the default in the file; the shell, `.env`, `env_file` and `COMPOSE_PROFILES` are ignored, `deploy --dry-run` and the deployment record show each value's source, and `secret import` / `var import` move an existing `.env` in ([Where values come from](compose.md#where-values-come-from), [upgrade note](releases.md#upgrading-to-30-values-come-only-from-control)). A `major`. - 7. **Then:** `x-tug.release` hooks, structured logs and `/metrics` (demoted: webhooks and the probe come first), SEC-09 and SEC-06 as minors, and the `dnf` and two-node verification debt. + 7. **Then:** `x-tug.release` hooks, additional telemetry only when operating evidence demands it (the minimal structured logs, `/api/v1/metrics`, notifications and external probe are implemented), SEC-09 and SEC-06 as minors, and the `dnf` and two-node verification debt. Keep logs useful for diagnosing one failed operation and avoid per-deployment or per-request identifiers as metric labels. 5. Reconsider flexible node roles, scaled WordPress, additional networking, dashboards and stronger isolation only for a demonstrated need. Co-locating control is attractive for small installations but requires an independently tested off-host recovery path and an explicit host trust model. ## Road to 1.0.0 @@ -231,12 +231,15 @@ The test layers, the metrics milestones are held to and the backlog of mocked te - [x] `tug tui` terminal UI phases 1 to 3: overview and attention list, live logs, inspect and shell, confirmed restart, rollback, cancel and backup, and `GET /api/v1/whoami` ([Terminal UI](tui.md)) - [ ] `tug tui` phase 4 (admin actions typed-name confirmation, screen lock, preferences, credential contexts) and phase 5 (metrics columns with M9) - [x] M8 scoped operator credentials, CI deploy example, credential names in the audit trail ([Access control](access-control.md)) -- [ ] M8 outbound webhooks +- [x] M8 best-effort signed outbound webhooks with bounded retries and delivery metrics - [x] M9 event hygiene ([Architecture](architecture.md#events)) and observability design ([Observability](observability.md)) - [x] Streaming: Docker daemon events in the agent (crashes, OOM kills, crash loops), agent events over the stream socket, the operator watch channel (`GET /api/v1/watch`, `events?after=`) and node presence (`connected`) ([Architecture](architecture.md#operator-watch-channel)) -- [ ] M9 structured logs (`TUG_LOG_FORMAT`, `TUG_LOG_LEVEL`) -- [ ] M9 `GET /metrics` on control (Prometheus text, `read` credential), with agent-reported node and service stats -- [ ] M9 an authenticated `/api/v1/readyz`, sample alert rules, and the metrics documentation test gate +- [x] M9 structured logs (`TUG_LOG_FORMAT`, `TUG_LOG_LEVEL`), request IDs and bounded context +- [x] M9 `GET /api/v1/metrics` on control (Prometheus text, `read` credential): project-filtered deployment and backup state, node report age, bounded writer queue and webhook delivery +- [ ] M9 continuous agent-reported resource statistics, only when demonstrated necessary +- [x] M9 authenticated `/api/v1/readyz`: database read, overdue writer and restart-required checks ([Observability](observability.md#health-endpoints)) +- [x] M9 metrics authorization, missing-data, escaping and documentation checks; external reachability probe tool +- [ ] M9 application-specific alert thresholds selected from measured operating objectives - [x] M10 published ports bound to the node address or loopback by default; `doctor` reports public listeners - [x] M11 spec written - [x] M11 `x-tug.backup` (generic: a command in a service, or a volume's files), scheduled backups to S3-compatible storage with restic, catalog, `backup run`, `backup list`, `restore` (verified against real Docker and on two real hosts with an S3-compatible server and a scheduled run: [Real-host evaluation](real-host-evaluation.md#generic-backups-restore-and-tug-run-on-fresh-hosts-1-november-2026); not yet against AWS S3) diff --git a/docs/testing.md b/docs/testing.md index df2ab7d..45d1773 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -56,9 +56,22 @@ assert.deepEqual(metrics(fake), { revisionsStarted: 1, peakRevisions: 2, unchang Inject the clock and sleep into the rollout code so drain and poll timing is tested without real waits. +### Reliability acceptance and regression coverage + +The acceptance criteria below are checks for the reliability work tracked in [Roadmap](roadmap.md#sequencing); they are not uptime, recovery-time or data-loss guarantees. A workload check answers whether tug sees a service as healthy on a node. It does not establish that users can reach the service through DNS, TLS, ingress and its dependencies, so real-host evidence must include an external request when user impact is in scope. Keep the result and failure context from each exercise, and mark an item complete only after the implementation and evidence are reviewed. + +| Scenario | Acceptance evidence | Coverage/status | +|---|---|---| +| Control loss | During a controlled control outage, an external probe records zero failed requests for workloads expected to remain available; after restoring control from an off-host archive, expected projects and nodes return and agents reconnect. | Historical QEMU recovery evidence exists in [Real-host evaluation](real-host-evaluation.md#recovery-drill); new damage-during-activation and older-snapshot reconnection cases remain gaps. | +| Permanent node loss | A node-pinned workload does not silently move with its local volume; record the outage, restore the data to a replacement node, then verify the service and an external request. | Existing QEMU recovery evidence covers loss of a workload node; the checklist spells out the local-volume boundary. Add a specific regression only if the implementation can be reproduced at a cheaper layer. | +| Network partition | Status exposes stale observations; when connectivity returns, a service with unchanged resolved configuration does not restart. Record external request results separately. Endpoint changes may change the resolved service definition and restart a client; that behavior must remain visible in tests until stable endpoint handling is implemented. | The first rolling regression tests now guard stable inputs and explicitly show endpoint appearance causing a second revision with `ExtraHosts`; a real partition/recovery exercise remains proposed. | +| Full disk | A bounded disk-pressure exercise captures the failing operation and available space; after capacity is restored, deploy and backup operations succeed without losing the last known-good workload. | No fault-injection acceptance test is claimed. Add a reproducible test at the cheapest layer that can model the actual write failure. | +| Backup destination outage | During the outage, `backup list --check` exits nonzero and no success is recorded for the failed target; after access returns, a new backup passes the check. Prove contents separately with a restore. | Backup executor tests cover cut-short streams and local repositories; best-effort notifications are tested; the damaged-repository Docker drill passed in CI; it was not run locally on the host without a daemon. | +| Operator diagnosis | Each failure produces an actionable event or notification naming the affected resource, observed state and next check; logs retain enough operation context to diagnose it without exposing credentials or secret values. Metrics use bounded labels and expose counts/ages needed by these checks, not request-specific IDs. | Existing events cover several container and deployment changes; best-effort notifications, structured logs and `/api/v1/metrics` are implemented; application-level monitoring remains external. | + ### Backlog of mocked tests (proposed) -1. **One revision per deployment as endpoints settle.** A rolling service on one node gets an `extra_hosts` entry in its container definition for every service on another node that is serving. Those entries are part of the revision's identity, so when the other node's service becomes ready between two polls, the agent starts a second revision of the same image. On the real hosts the first deploy ended with two revisions of the same image, and the surviving one carries such an entry, which points to this cause. The old revision drained and no request failed, but the app was started twice. Decide whether those entries should be excluded from the identity, and pin the decision with a test that counts the revisions started. +1. **Endpoint settling can start a second revision.** A rolling service on one node gets an `extra_hosts` entry in its container definition for every service on another node that is serving. Those entries are part of the revision's identity, so when the other node's service becomes ready between two polls, the agent starts a second revision of the same image. On the real hosts the first deploy ended with two revisions of the same image, and the surviving one carries such an entry, which points to this cause. The old revision drained and no request failed, but the app was started twice. Tests now guard stable inputs and reproduce endpoint appearance starting revision two with `ExtraHosts`; this documents the current behavior, not a fix. Decide whether endpoint changes should restart clients, and consider stable endpoint handling only if the observed restart cost warrants it. 2. **Rollout metrics from the fakes**: at most two revisions of a service at once, host ports released after the drain, unchanged services never stopped, a failed candidate removed while the previous revision keeps serving, and exactly one automatic rollback. ### Adding a tier 1 test @@ -280,3 +293,9 @@ These results are useful for choosing operational defaults and follow-up measure | Cached lists, S3 downloads and WordPress HTML have very different CPU, latency and memory profiles; whole-stack memory also includes storage caches. | Use per-service resource evidence for VPS sizing and troubleshooting. These isolated sequential runs cannot establish safe node packing, tenant isolation or the number of sites a VPS can host. | Run mixed workloads through one managed edge with the control plane and agent present, including backups and overlapping rolling revisions, on an otherwise quiet host. | The most valuable next benchmark for tug itself is an end-to-end deployment of these same workloads, comparing the standalone baseline with tug-managed ingress under the same server settings and resource limits. Include mixed traffic and a rolling deployment to measure latency, errors and resource headroom during a change. That would separate application tuning from platform overhead and provide evidence for ingress, placement and rollout design decisions. No tug runtime behavior is changed by this benchmark PR. + +## Reliability failure coverage + +The control readiness HTTP test holds image resolution inside an actual deployment, advances an injected writer clock beyond 35 minutes, observes a prompt 503 and verifies recovery after the deploy completes. The writer admission test fills its bounded waiting queue, expects a 503 for excess work and proves accepted work still completes. A blocked webhook receiver leaves readiness responsive and receives no raw private event text or metadata. Scoped metrics tests exclude other projects and global node/writer state. + +Restore activation tests interrupt all three replacement boundaries and inject `ENOSPC` before database replacement, then check recovery of the complete prior generation. The existing control-backup test restores an older snapshot, reconnects an agent and proves retired commands are not delivered again. The Docker backup drill damages a disposable repository configuration, requires a failed job without reinitialization, restores that configuration and checks a subsequent backup retains history. Run the Docker drill with `npm run test:integration:backups`; its result is a skip when Docker is unavailable, which is not runtime validation. These controlled faults do not establish safe operation on a physically full host or a production availability objective. diff --git a/package.json b/package.json index 7455d79..ad95196 100644 --- a/package.json +++ b/package.json @@ -28,7 +28,7 @@ "typecheck": "tsc -p tsconfig.json --noEmit", "schema": "node tools/scripts/write-x-tug-schema.js", "prepack": "npm run build", - "test": "node --test 'src/**/*.test.ts'", + "test": "node --test 'src/**/*.test.ts' tools/scripts/check-reachability.test.mjs", "lint": "eslint . && prettier --check .", "format": "prettier --write . && eslint . --fix", "test:integration": "bash tools/scripts/integration-test.sh", @@ -44,7 +44,8 @@ "test:integration:no-registry": "node tools/scripts/integration-no-registry.js", "test:smoke:image": "node tools/scripts/smoke-runtime-image.js", "test:benchmarks": "node --test 'benchmarks/**/*.test.mjs'", - "benchmark:examples": "node benchmarks/examples/run.mjs" + "benchmark:examples": "node benchmarks/examples/run.mjs", + "check:reachability": "node tools/scripts/check-reachability.mjs" }, "dependencies": { "@peculiar/x509": "^3.1.0", diff --git a/src/agent/agent.ts b/src/agent/agent.ts index 2ef6e7c..a03538c 100755 --- a/src/agent/agent.ts +++ b/src/agent/agent.ts @@ -76,6 +76,7 @@ import type { HttpClient } from './runners.ts'; import { CADDY_CONTAINER, reconcileIngress, removeIngress } from '../ingress.ts'; import { collectResources, ownContainer } from '../host-resources.ts'; import { createTlsMonitor } from '../ingress-tls.ts'; +import { createLogger } from '../logger.ts'; import type { TlsMonitor } from '../ingress-tls.ts'; import type { Command, @@ -105,6 +106,7 @@ let NODE_NAME = process.env.TUG_NODE_NAME || process.env.HOSTNAME || 'node1'; const STATE_DIR = process.env.TUG_STATE_DIR || '/var/lib/tug'; const APPLY = process.env.TUG_APPLY !== 'false'; const INTERVAL = Number(process.env.TUG_POLL_INTERVAL_MS || 5000); +const log = createLogger({ component: 'agent' }); // Everything the reconcile/command logic needs from the outside world. Production wiring is // `defaultContext()`; tests pass a fake `docker` runner / `http` client and temp dirs. @@ -292,7 +294,7 @@ export class AgentWakeChannel { if (typeof frame.type === 'string' && frame.type.startsWith('live.')) { if (frame.v === 1) void Promise.resolve(this.handlers.onLive?.(message as AgentLiveControlFrame)).catch( - (error) => console.error('agent live frame failed:', errMessage(error)), + (error) => log.failure('agent live frame failed', error, {}, 'warn'), ); return; } @@ -1453,7 +1455,12 @@ export async function endpointAddressesForAgent( try { return { ...ep, address: await resolve(ep.address) }; } catch (error) { - console.error(errMessage(error)); + log.failure( + 'could not resolve a cross-node service address', + error, + { service }, + 'warn', + ); return null; } }), @@ -1511,7 +1518,7 @@ export async function pollOnce(ctx: AgentContext = defaultContext()): Promise false); @@ -1593,7 +1600,7 @@ async function enroll(joinKey: string): Promise { throw new Error('invalid join key (expected a base64url-encoded v2 join key)', { cause: e }); } const transport = createTransport(decoded.credential, CONTROL_URL_OVERRIDE || decoded.connectUrl); - console.log(`enrolling with ${decoded.credential.url} as ${decoded.name}`); + log.info('enrolling with control', { node: decoded.name }); const csr = await agentIdentityCsr(STATE_DIR); const body: EnrollRequest = { csr, @@ -1607,7 +1614,7 @@ async function enroll(joinKey: string): Promise { }); if (!res.ok) throw new Error(`enroll failed: ${res.status} ${await res.text()}`); const doc = await installIdentity(STATE_DIR, (await res.json()) as EnrollResponse); - console.log(`enrolled as node ${doc.nodeName}`); + log.info('enrolled', { node: doc.nodeName }); return doc; } finally { void transport.close(); @@ -1631,7 +1638,7 @@ const RENEW_RETRY_MS = 15 * 60_000; export async function main(): Promise { const doc = await loadOrEnroll(); const ctx = defaultContext(doc); - console.log(`tug-agent node=${NODE_NAME} control=${ctx.controlUrl} apply=${APPLY}`); + log.info('agent started', { node: NODE_NAME, apply: APPLY }); let renewAt = renewalTime(doc.cert); /** Renews with the same key, saves, then swaps the client and reconnects. Failure keeps the old identity until it expires. */ const renewIfDue = async (): Promise => { @@ -1652,10 +1659,12 @@ export async function main(): Promise { void previous; renewAt = renewalTime(next.cert); wake.close(); - console.log('node certificate renewed'); + log.info('node certificate renewed', { node: NODE_NAME }); } catch (e) { renewAt = Date.now() + RENEW_RETRY_MS; - console.error(`certificate renewal failed (keeping the current one): ${errMessage(e)}`); + log.failure('certificate renewal failed; keeping the current certificate', e, { + node: NODE_NAME, + }); } }; const wake = new AgentWakeChannel(ctx, undefined, { @@ -1672,7 +1681,8 @@ export async function main(): Promise { nodeName: ctx.nodeName, post: (event) => postEvent(ctx, event), wake: () => wake.poke(), - log: (message) => console.error(message), + log: (message) => + log.failure('docker event watcher reported a problem', new Error(message), {}, 'warn'), }); if (ctx.apply) watcher.start(); for (;;) { @@ -1680,12 +1690,12 @@ export async function main(): Promise { try { await pollOnce(ctx); } catch (e) { - console.error(errMessage(e)); + log.failure('agent poll failed', e, { node: NODE_NAME }); } try { await wake.wait(INTERVAL); } catch (e) { - console.error(`agent wake wait failed: ${errMessage(e)}`); + log.failure('agent wake wait failed', e, { node: NODE_NAME }, 'warn'); await new Promise((r) => setTimeout(r, INTERVAL)); } } diff --git a/src/agent/reconciler.test.ts b/src/agent/reconciler.test.ts index fe7d1e6..414fc0f 100644 --- a/src/agent/reconciler.test.ts +++ b/src/agent/reconciler.test.ts @@ -336,16 +336,26 @@ test('a revision that fails its health check is removed and the old revision kee ); }); -test('repeated reconciliation of the same desired state creates nothing new', async () => { +test('the first rolling deployment starts one revision and unchanged reconciliation starts none', async () => { const host = fakeHost(); const ctx = await makeCtx(host); const first = await reconcile(desired(rolling('web:1'), 'v1'), ctx); + assert.equal( + host.docker.calls.filter((call) => call.startsWith('create tug-proj-web-')).length, + 1, + 'the first rolling deployment starts exactly one revision', + ); const ids = [...host.docker.containers.keys()]; host.docker.calls.length = 0; await reconcile(desired(rolling('web:1'), 'v1'), ctx); const third = await reconcile(desired(rolling('web:1'), 'v1'), ctx); assert.deepEqual([...host.docker.containers.keys()], ids, 'no container is created or recreated'); + assert.deepEqual( + host.docker.calls.filter((call) => call.startsWith('create tug-proj-web-')), + [], + 'unchanged reconciliation starts no additional rolling revision', + ); assert.deepEqual([...host.docker.calls], [], 'steady state issues no mutating call'); assert.equal(third.serviceStates?.web.port, 20000, 'the port is stable'); assert.equal( @@ -355,6 +365,36 @@ test('repeated reconciliation of the same desired state creates nothing new', as ); }); +test('a cross-node endpoint appearing after the first poll changes the revision definition', async () => { + const host = fakeHost(); + const ctx = await makeCtx(host); + const compose = rolling('web:1'); + compose.services.db['x-tug'] = { placement: { nodes: ['n2'] } }; + const state = desired(compose, 'v1'); + + await reconcile(state, ctx); + const initial = host.docker.service('proj', 'web')[0]; + assert.deepEqual( + initial.body.HostConfig?.ExtraHosts, + undefined, + 'the first revision has no cross-node host entry before the remote service is serving', + ); + + await reconcile(state, ctx, { db: [{ node: 'n2', address: '10.0.0.2' }] }); + const revisions = host.docker.service('proj', 'web'); + assert.equal( + host.docker.calls.filter((call) => call.startsWith('create tug-proj-web-')).length, + 2, + 'an endpoint appearing later changes the container definition and starts another revision', + ); + assert.ok( + revisions.some((container) => + (container.body.HostConfig?.ExtraHosts as string[] | undefined)?.includes('db:10.0.0.2'), + ), + 'the replacement revision receives the newly available cross-node service name', + ); +}); + test('a service whose definition did not change is not restarted when a sibling rolls', async () => { const host = fakeHost(); const ctx = await makeCtx(host); diff --git a/src/control/control-backup.test.ts b/src/control/control-backup.test.ts index aa904f5..52895a8 100644 --- a/src/control/control-backup.test.ts +++ b/src/control/control-backup.test.ts @@ -189,6 +189,10 @@ test('with a destination control backs itself up into it without images and keep ); const create = docker.calls.filter((c) => c.startsWith('create tug-control-backup')); assert.equal(create.length, 1, 'one helper container runs the backup'); + await until( + async () => (await fs.readdir(path.join(s.dir, 'control-backups'))).length === 0, + 'the completed backup archive to be removed from control', + ); assert.deepEqual( await fs.readdir(path.join(s.dir, 'control-backups')), [], @@ -985,7 +989,7 @@ test('after a restore the commands it carried are retired, not delivered to agen artifactRef: artifact, confirm: true, }); - assert.equal(res.status, 200, 'the restore succeeds'); + assert.equal(res.status, 200, 'the restore succeeds: ' + (await res.text())); } finally { await target.close(true); } diff --git a/src/control/control.test.ts b/src/control/control.test.ts index 7aab3e3..7d8c69a 100644 --- a/src/control/control.test.ts +++ b/src/control/control.test.ts @@ -5,28 +5,108 @@ import os from 'node:os'; import path from 'node:path'; import { createApp } from './control.ts'; import { testImageResolver, testControlUrl, testIdentity } from '../../tests/helpers.ts'; +import { until } from '../../tests/wait.ts'; +import { recordEvent } from './credentials.ts'; +import { WRITER_OVERDUE_AFTER_MS } from './writer-queue.ts'; +import { openDb } from '../db/db.ts'; import type { Server } from 'node:http'; import type { AddressInfo } from 'node:net'; import { json } from '../../tests/helpers.ts'; -async function server() { +async function server(options: Parameters[0] = {}) { const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'tug-')); process.env.TUG_DATA_DIR = dir; const app = await createApp({ - identity: testIdentity, + identity: (req, db) => { + for (const role of ['read', 'deploy'] as const) { + if (req.headers[`x-tug-test-${role}`]) { + const id = `cred_test_${role}`; + db.run( + 'INSERT OR IGNORE INTO operator_credentials(id,name,key_hash,role,created_at,created_by) VALUES(?,?,?,?,?,?)', + id, + role, + `test-${role}`, + role, + new Date().toISOString(), + 'test', + ); + return { uri: `tug:operator:${id}`, keyHash: `test-${role}` }; + } + } + return testIdentity(req, db); + }, controlUrl: testControlUrl, images: { resolve: testImageResolver }, + ...options, }); const srv = await new Promise((resolve) => { const s = app.listen(0, () => resolve(s)); }); const base = `http://127.0.0.1:${(srv.address() as AddressInfo).port}`; - return { base, close: () => new Promise((resolve) => srv.close(resolve)) }; + return { base, dir, close: () => new Promise((resolve) => srv.close(resolve)) }; } function op() { return { 'x-tug-test-operator': 'operator', 'content-type': 'application/json' }; } +test('readiness is protected by the read role and reports database failures without details', async () => { + let failDatabaseProbe = false; + const s = await server({ + openDatabase: async (filename) => { + const db = await openDb(filename); + return { + get: (sql, ...values) => { + if (failDatabaseProbe && sql === 'SELECT id FROM projects LIMIT 1') + throw new Error('private sqlite path and details'); + return db.get(sql, ...values); + }, + all: db.all.bind(db), + run: db.run.bind(db), + exec: db.exec.bind(db), + close: db.close.bind(db), + }; + }, + }); + try { + let res = await fetch(`${s.base}/api/v1/readyz`); + assert.equal(res.status, 401, 'readiness requires an authenticated operator credential'); + + res = await fetch(`${s.base}/api/v1/readyz`, { headers: { 'x-tug-test-read': 'read' } }); + assert.equal( + res.status, + 200, + 'a read credential gets readiness when the database probe succeeds', + ); + assert.deepEqual(await json(res), { + ready: true, + checks: { database: 'ok', writer: 'ok' }, + }); + + res = await fetch(`${s.base}/api/v1/readyz`, { headers: { 'x-tug-test-deploy': 'deploy' } }); + assert.equal( + res.status, + 200, + 'higher-privilege operator credentials also satisfy the read role', + ); + + failDatabaseProbe = true; + res = await fetch(`${s.base}/api/v1/readyz`, { headers: { 'x-tug-test-read': 'read' } }); + assert.equal(res.status, 503, 'a failed database probe makes readiness unavailable'); + const body = await json(res); + assert.deepEqual(body, { + ready: false, + checks: { database: 'failed', writer: 'ok' }, + }); + assert.equal( + JSON.stringify(body).includes('private sqlite'), + false, + 'database errors stay private', + ); + } finally { + await s.close(); + } +}); + test('registers node, stores secret, creates desired state, serves agent work', async () => { const s = await server(); try { @@ -177,3 +257,260 @@ test('failed node status fails deployment and unblocks future deploys', async () await s.close(); } }); + +test('readiness stays responsive behind an overdue deploy and recovers when the writer completes', async () => { + let time = 1000; + let entered = false; + let release!: () => void; + const blocked = new Promise((resolve) => { + release = resolve; + }); + const s = await server({ + writerClock: () => time, + images: { + resolve: async (...args) => { + entered = true; + await blocked; + return testImageResolver(args[0]); + }, + }, + }); + let deployment: Promise | undefined; + try { + await fetch(s.base + '/api/v1/nodes/register', { + method: 'POST', + headers: op(), + body: JSON.stringify({ name: 'n1' }), + }); + deployment = fetch(s.base + '/api/v1/deployments', { + method: 'POST', + headers: op(), + body: JSON.stringify({ + project: 'site', + version: 'v1', + compose: { + services: { web: { image: 'nginx', 'x-tug': { placement: { nodes: ['n1'] } } } }, + }, + }), + }); + await until(() => entered, 'deployment to hold the writer during image resolution'); + let response = await fetch(s.base + '/api/v1/readyz', { + headers: op(), + signal: AbortSignal.timeout(5000), + }); + assert.equal(response.status, 200, 'a normal in-progress deploy stays ready'); + time += WRITER_OVERDUE_AFTER_MS + 1; + response = await fetch(s.base + '/api/v1/readyz', { + headers: op(), + signal: AbortSignal.timeout(5000), + }); + assert.equal( + response.status, + 503, + 'an overdue active writer makes readiness fail without waiting', + ); + assert.equal( + (await json(response)).checks.writer, + 'overdue', + 'readiness identifies the overdue writer', + ); + release(); + assert.equal( + (await deployment).status, + 200, + 'the deployment still completes after readiness failed', + ); + response = await fetch(s.base + '/api/v1/readyz', { headers: op() }); + assert.equal(response.status, 200, 'readiness recovers after writer completion'); + } finally { + release(); + await deployment; + await s.close(); + } +}); + +test('webhook delivery cannot block readiness and omits private event details', async () => { + let release!: () => void; + const blocked = new Promise((resolve) => { + release = resolve; + }); + const bodies: string[] = []; + const s = await server({ + notificationTickMs: 10, + webhook: { + url: 'https://receiver.example.test/hooks', + secret: 'test-signing-key', + fetch: async (_url, options) => { + bodies.push(String(options?.body)); + await blocked; + return new Response('', { status: 200 }); + }, + }, + }); + const db = await openDb(path.join(s.dir, 'tug.db')); + try { + recordEvent(db, { + actor: 'test', + type: 'backup.failed', + message: 'private-bucket-secret', + metadata: { command: 'private-password' }, + }); + await until( + () => bodies.length > 0, + 'a committed backup failure to be delivered without a watcher', + ); + const ready = await fetch(s.base + '/api/v1/readyz', { + headers: op(), + signal: AbortSignal.timeout(5000), + }); + assert.equal(ready.status, 200, 'blocked receiver leaves readiness responsive'); + const metrics = await (await fetch(s.base + '/api/v1/metrics', { headers: op() })).text(); + assert.match( + metrics, + /# TYPE tug_webhook_delivered_total counter/, + 'delivery totals use Prometheus counter semantics', + ); + assert.match( + metrics, + /# TYPE tug_webhook_pending gauge/, + 'pending delivery is a current-value gauge', + ); + assert.ok( + !bodies[0].includes('private-'), + 'outbound notification omits raw message and metadata', + ); + release(); + } finally { + release(); + db.close(); + await s.close(); + } +}); + +test('configured writer admission returns Retry-After and preserves accepted deployments', async () => { + const previous = process.env.TUG_WRITER_MAX_PENDING; + process.env.TUG_WRITER_MAX_PENDING = '1'; + let release!: () => void; + const blocked = new Promise((resolve) => { + release = resolve; + }); + let entered = false; + let s: Awaited>; + try { + s = await server({ + images: { + resolve: async (image) => { + entered = true; + await blocked; + return testImageResolver(image); + }, + }, + }); + } finally { + if (previous === undefined) delete process.env.TUG_WRITER_MAX_PENDING; + else process.env.TUG_WRITER_MAX_PENDING = previous; + } + let first: Promise | undefined; + let second: Promise | undefined; + try { + await fetch(s.base + '/api/v1/nodes/register', { + method: 'POST', + headers: op(), + body: JSON.stringify({ name: 'n1' }), + }); + const deploy = (project: string) => + fetch(s.base + '/api/v1/deployments', { + method: 'POST', + headers: op(), + body: JSON.stringify({ + project, + version: 'v1', + compose: { services: { web: { image: 'nginx' } } }, + }), + }); + first = deploy('first'); + await until(() => entered, 'the first deployment to acquire the writer'); + second = deploy('second'); + await until( + async () => + /tug_writer_queue_depth 1\b/.test( + await (await fetch(s.base + '/api/v1/metrics', { headers: op() })).text(), + ), + 'the second deployment to enter the configured queue', + ); + const overflow = await deploy('overflow'); + assert.equal( + overflow.status, + 503, + 'excess operator work is refused before changing desired state', + ); + assert.equal( + overflow.headers.get('retry-after'), + '5', + 'overloaded clients get a bounded retry hint', + ); + release(); + assert.equal( + (await first).status, + 200, + 'active work completes despite refused excess admission', + ); + assert.equal((await second).status, 200, 'accepted queued work completes in order'); + } finally { + release(); + await first; + await second; + await s.close(); + } +}); + +test('invalid optional webhook configuration disables delivery without disabling control', async () => { + const s = await server({ webhook: { url: 'http://not-https.example.test', secret: 'private' } }); + try { + const response = await fetch(s.base + '/api/v1/readyz', { headers: op() }); + assert.equal( + response.status, + 200, + 'optional notification misconfiguration cannot take control offline', + ); + } finally { + await s.close(); + } +}); + +test('aborted requests are logged without their incomplete body', async (t) => { + const { request } = await import('node:http'); + let entered = false; + const s = await server({ + identity: (req, db) => { + entered = true; + return testIdentity(req, db); + }, + }); + const lines: string[] = []; + const original = process.stderr.write; + t.mock.method(process.stderr, 'write', (chunk: string | Uint8Array) => { + lines.push(String(chunk)); + return true; + }); + const pending = request(s.base + '/api/v1/deployments', { method: 'POST', headers: op() }); + pending.on('error', () => {}); + try { + pending.write('{"private":"never-log-this-body"'); + await until(() => entered, 'the server to accept request headers before an aborted upload'); + pending.destroy(); + await until( + () => lines.some((line) => line.includes('request aborted')), + 'an aborted request to be recorded without a finish event', + ); + assert.doesNotMatch( + lines.join('\n'), + /never-log-this-body/, + 'request diagnostics never serialize incomplete request bodies', + ); + } finally { + pending.destroy(); + process.stderr.write = original; + await s.close(); + } +}); diff --git a/src/control/control.ts b/src/control/control.ts index d7d8259..9ced8bd 100755 --- a/src/control/control.ts +++ b/src/control/control.ts @@ -27,7 +27,13 @@ import https from 'node:https'; import http from 'node:http'; import path from 'node:path'; import { WebSocketServer, WebSocket } from 'ws'; +import { createLogger } from '../logger.ts'; +import { renderMetricsSnapshot } from './metrics.ts'; +import { createWebhookNotifier, webhookOptionsFromEnv } from './notifications.ts'; +import type { WebhookOptions } from './notifications.ts'; +import { createReliabilityMonitor } from './reliability-monitor.ts'; import { WatchHub } from './watch.ts'; +import { createWriterQueue, WRITER_OVERDUE_AFTER_MS } from './writer-queue.ts'; import type tls from 'node:tls'; import * as tar from 'tar'; import { openDb, openSnapshotDb } from '../db/db.ts'; @@ -454,7 +460,7 @@ export class LiveHub { try { this.onAgentEvents(nodeId, frame.events); } catch (error) { - console.error(`agent events from node ${nodeId} were not stored: ${errMessage(error)}`); + log.failure('agent event persistence failed', error, { node: nodeId }, 'error'); } return; } @@ -690,6 +696,7 @@ declare module 'express-serve-static-core' { interface HttpError extends Error { status?: number; + retryAfter?: number; lock?: LockRow; } const httpError = (status: number, message: string): HttpError => @@ -730,6 +737,7 @@ const ROLLBACK_VERSION_PREFIX = 'rollback-'; const ACTIVE_STATES_SQL = statesSql(ACTIVE_DEPLOYMENT_STATES); const PORT = Number(process.env.PORT || 8443); +const log = createLogger({ component: 'control' }); export interface ControlImageOperations { resolve: typeof resolveImage; @@ -747,6 +755,12 @@ export async function createApp( /** Test hook: replaces the TCP connect `doctor` uses to check ports 80 and 443 on ingress nodes. */ probePort?: PortProbe; openDatabase?: typeof openDb; + /** Monotonic time source for writer diagnostics; injectable for deterministic tests. */ + writerClock?: () => number; + /** Test-only webhook transport/config override; production reads environment. */ + webhook?: WebhookOptions; + /** Test-only notification observation interval; production uses five seconds. */ + notificationTickMs?: number; /** Test hook: runs after each step of putting a restored backup in place and may throw to simulate a crash. */ restoreAfterStep?: (step: ActivationStep) => void | Promise; /** Minimum time between event prunes triggered by agent reports; default one hour. */ eventPruneIntervalMs?: number; @@ -811,7 +825,7 @@ export async function createApp( if (raw === undefined || raw === '') return fallback; const value = Number(raw); if (Number.isInteger(value) && value >= 0) return value; - console.warn(`${name}=${JSON.stringify(raw)} is not a whole number; using ${fallback}`); + log.warn('invalid nonnegative integer setting; using default', { setting: name, fallback }); return fallback; }; const EVENT_RETENTION_DAYS = envCount('TUG_EVENT_RETENTION_DAYS', 30); @@ -834,15 +848,17 @@ export async function createApp( EVENT_MAX_ROWS, ).changes ?? 0; if (removed) - console.log( - `removed ${removed} old events (kept ${EVENT_RETENTION_DAYS || 'any'} days, at most ${EVENT_MAX_ROWS || 'unlimited'} rows)`, - ); + log.info('old events pruned', { + removed, + retentionDays: EVENT_RETENTION_DAYS, + maxRows: EVENT_MAX_ROWS, + }); return removed; } try { pruneEvents(); } catch (e) { - console.error(`event pruning failed: ${errMessage(e)}`); + log.failure('event pruning failed', e, {}, 'error'); } const masterKey = await loadOrCreateMasterKey(masterKeyFile); // A CA is created only for a fresh installation; existing state with missing or mismatched CA material is a startup error. @@ -858,6 +874,67 @@ export async function createApp( }; const app = express(); + let notifier: ReturnType | undefined; + try { + const webhookConfig = options.webhook ?? webhookOptionsFromEnv(); + if (webhookConfig) + notifier = createWebhookNotifier({ + ...webhookConfig, + onError: () => log.warn('webhook delivery exhausted retries'), + }); + } catch { + log.error( + 'notifications disabled: configure both TUG_WEBHOOK_URL and TUG_WEBHOOK_SECRET with an HTTPS receiver', + ); + } + const monitor = notifier + ? createReliabilityMonitor(db, (event) => { + notifier?.enqueue(event); + }) + : undefined; + const notificationTimer = monitor + ? setInterval(() => { + if (restartRequired) return; + try { + monitor.tick(); + } catch { + log.error('notification observation failed'); + } + }, options.notificationTickMs ?? 5000) + : undefined; + notificationTimer?.unref(); + const stopNotifications = () => { + clearInterval(notificationTimer); + notifier?.stop(); + }; + app.locals.stopNotifications = stopNotifications; + app.use((req, res, next) => { + const requestId = crypto.randomUUID(); + res.setHeader('x-request-id', requestId); + const started = performance.now(); + let recorded = false; + const recordRequest = (aborted: boolean) => { + if (recorded) return; + recorded = true; + const context = { + requestId, + method: req.method, + route: req.route?.path ?? 'unmatched', + status: res.statusCode, + durationMs: Math.round(performance.now() - started), + aborted, + }; + if (aborted || res.statusCode >= 400) + log.warn(aborted ? 'request aborted' : 'request failed', context); + else if (req.path.startsWith('/agent/')) log.debug('request completed', context); + else log.info('request completed', context); + }; + res.once('finish', () => recordRequest(false)); + res.once('close', () => { + if (!res.writableFinished) recordRequest(true); + }); + next(); + }); app.locals.pki = pki; app.locals.controlUrl = configuredControlUrl; app.locals.dataDir = DATA_DIR; @@ -969,21 +1046,19 @@ export async function createApp( next(); }); // Hold the writer for the operation lifetime, including after an HTTP disconnect. - let writer = Promise.resolve(); - async function withWriter(operation: () => T | Promise): Promise { - const previous = writer; - let release!: () => void; - writer = new Promise((resolve) => { - release = resolve; - }); - await previous; - try { + const configuredWriterLimit = Number(process.env.TUG_WRITER_MAX_PENDING ?? 100); + const writerLimit = + Number.isSafeInteger(configuredWriterLimit) && configuredWriterLimit >= 1 + ? configuredWriterLimit + : 100; + if (writerLimit !== configuredWriterLimit) + log.warn('invalid writer queue limit; using default', { limit: 100 }); + const writerQueue = createWriterQueue(options.writerClock, writerLimit); + const withWriter = (operation: () => T | Promise): Promise => + writerQueue.run(() => { if (restartRequired) throw httpError(503, 'control restored; restart required'); - return await operation(); - } finally { - release(); - } - } + return operation(); + }); let publication = Promise.resolve(); async function withPublication(operation: () => T): Promise { @@ -1047,8 +1122,14 @@ export async function createApp( const audit = (actor: Actor, event: Omit[1], 'actor'>) => { try { recordEvent(db, { ...event, actor: actor.name }); + log.info('control action', { + actor: actor.name, + eventType: event.type, + project: event.project, + deploymentId: event.deploymentId, + }); } catch (e) { - console.error(`audit event failed: ${errMessage(e)}`); + log.failure('audit event failed', e, {}, 'error'); } }; const projectScope = (actor: Actor, column: string) => @@ -1488,7 +1569,7 @@ export async function createApp( /* Best-effort audit event. */ } } - })().catch((e) => console.error('auto rollback failed:', errMessage(e))); + })().catch((e) => log.failure('auto rollback failed', e, {}, 'error')); } function currentDesired(projectId: string): DesiredStateRow | undefined { @@ -1644,6 +1725,7 @@ export async function createApp( } function sendError(res: Response, e: unknown): void { const err = e as HttpError; + if (err.retryAfter) res.setHeader('Retry-After', String(err.retryAfter)); res .status(restartRequired ? 503 : err.status || 500) .json({ error: err.message, lock: err.lock || undefined }); @@ -2199,7 +2281,7 @@ export async function createApp( controlDestinationTick(); restores.advance(); } catch (e) { - console.error(`backup scheduler failed: ${errMessage(e)}`); + log.failure('backup scheduler failed', e, {}, 'error'); } }, backupTickMs); backupTimer.unref(); @@ -2208,14 +2290,17 @@ export async function createApp( const listenOnApp = app.listen.bind(app) as (...args: unknown[]) => import('node:http').Server; app.listen = ((...args: unknown[]) => { const server = listenOnApp(...args); - server.once('close', () => clearInterval(backupTimer)); + server.once('close', () => { + clearInterval(backupTimer); + stopNotifications(); + }); return server; }) as typeof app.listen; try { backups.tick(); // catch up on slots missed while control was down restores.advance(); // re-adopt unfinished restores and keep their projects locked } catch (e) { - console.error(`backup scheduler failed: ${errMessage(e)}`); + log.failure('backup scheduler failed', e, {}, 'error'); } api('post', '/api/v1/backups/run', bodyProject, async (req, res) => { const project = String(req.body.project || 'site'); @@ -2884,7 +2969,7 @@ export async function createApp( '--keep-storage', BUILD_CACHE_KEEP.replace(/\s/g, ''), ]); - })().catch((e) => console.error('image maintenance failed:', errMessage(e))); + })().catch((e) => log.failure('image maintenance failed', e, {}, 'error')); } await collectArtifacts(); await markMissingArtifacts(); @@ -3485,7 +3570,7 @@ export async function createApp( releaseLock(lock); await Promise.all(files.map((file) => fs.rm(file.path, { force: true }).catch(() => {}))); if (contextDir) await fs.rm(contextDir, { recursive: true, force: true }); - await collectArtifacts().catch((e) => console.error('image cleanup failed:', errMessage(e))); + await collectArtifacts().catch((e) => log.failure('image cleanup failed', e, {}, 'error')); } if (responseError) sendError(res, responseError); else res.json(responseBody); @@ -4008,7 +4093,7 @@ export async function createApp( const last = lastControlDestination(key, false); if (last?.state === 'failed' && at - Date.parse(last.created_at) < controlRetryMs()) return; void runControlDestination('schedule').catch((e) => - console.error(`control backup failed: ${errMessage(e)}`), + log.failure('control backup failed', e, {}, 'error'), ); }; api('get', '/api/v1/control/backups', null, async (_req, res) => { @@ -4059,9 +4144,7 @@ export async function createApp( remote, }); }); - void sweepControlLeftovers().catch((e) => - console.error(`startup cleanup failed: ${errMessage(e)}`), - ); + void sweepControlLeftovers().catch((e) => log.failure('startup cleanup failed', e, {}, 'error')); api('get', '/api/v1/control/backups/:id/download', null, async (req, res) => { const row = db.get<{ artifact_ref: string | null }>( "SELECT artifact_ref FROM backups WHERE id=? AND project_id='_control' AND target='control-plane'", @@ -4497,6 +4580,67 @@ export async function createApp( api('get', '/api/v1/version', null, async (req, res) => { res.json({ v: ownVersion }); }); + api('get', '/api/v1/metrics', null, (req, res) => { + const scope = projectScope(req.actor!, 'project_id'); + const global = !req.actor!.projects; + const queueAge = writerQueue.activeForMs(); + const metrics = renderMetricsSnapshot({ + webhook: global + ? { + enabled: Boolean(notifier), + ...(notifier?.stats ?? { delivered: 0, failed: 0, dropped: 0, pending: 0 }), + } + : undefined, + queue: global + ? { + depth: writerQueue.pending, + activeSince: queueAge === null ? null : Date.now() - queueAge, + } + : undefined, + nodes: global + ? db + .all<{ name: string; last_seen_at: string | null }>( + 'SELECT name,last_seen_at FROM nodes WHERE disabled_at IS NULL', + ) + .map((row) => ({ name: row.name, lastSeenAt: row.last_seen_at })) + : [], + deployments: db + .all<{ project_id: string; state: string; count: number }>( + `SELECT project_id,state,COUNT(*) AS count FROM deployments WHERE ${scope.where} GROUP BY project_id,state`, + ...scope.args, + ) + .map((row) => ({ project: row.project_id, state: row.state, count: row.count })), + backups: db + .all<{ + project_id: string; + target: string; + last_status: string | null; + last_success_at: string | null; + }>( + `SELECT project_id,target,last_status,last_success_at FROM backup_schedules WHERE enabled=1 AND ${scope.where}`, + ...scope.args, + ) + .map((row) => ({ + project: row.project_id, + target: row.target, + state: row.last_status ?? 'unknown', + succeededAt: row.last_success_at, + })), + }); + res.type('text/plain; version=0.0.4').send(metrics); + }); + api('get', '/api/v1/readyz', null, (_req, res) => { + let database = 'ok'; + try { + db.get('SELECT id FROM projects LIMIT 1'); + } catch { + database = 'failed'; + } + const activeForMs = writerQueue.activeForMs(); + const writer = activeForMs !== null && activeForMs > WRITER_OVERDUE_AFTER_MS ? 'overdue' : 'ok'; + const ready = database === 'ok' && writer === 'ok'; + res.status(ready ? 200 : 503).json({ ready, checks: { database, writer } }); + }); // Lets a client such as `tug tui` offer only the actions its credential may use; nothing here is secret. api('get', '/api/v1/whoami', null, async (req, res) => { const { name, role, projects } = req.actor!; @@ -5018,7 +5162,7 @@ export async function createApp( try { restores.advance(); } catch (e) { - console.error(`restore failed to advance: ${errMessage(e)}`); + log.failure('restore failed to advance', e, {}, 'error'); } const nextEndpointSnapshot = JSON.stringify(endpointsFor(currentDesiredStates())); if (nextEndpointSnapshot !== endpointSnapshot) { @@ -5084,7 +5228,7 @@ export async function createApp( try { pruneEvents(); } catch (e) { - console.error(`event pruning failed: ${errMessage(e)}`); + log.failure('event pruning failed', e, {}, 'error'); } } post('/agent/v1/events', authAgent, async (req, res) => { @@ -5314,7 +5458,7 @@ export async function listen(app: express.Express, port = PORT): Promise; server.on('tlsClientError', (error) => { - console.warn(`tls handshake refused: ${errMessage(error)}`.slice(0, 200)); + log.failure('tls handshake refused', error, {}, 'warn'); }); // Connection authorization gate: after a full handshake, refuse reused sessions and identities the database does not authorize. server.on('secureConnection', (socket) => { @@ -5340,7 +5484,7 @@ export async function listen(app: express.Express, port = PORT): Promise void)(); } catch (e) { - console.error(`session sweep failed: ${errMessage(e)}`); + log.failure('session sweep failed', e, {}, 'error'); } }, 1000); const renew = setInterval( @@ -5351,13 +5495,13 @@ export async function listen(app: express.Express, port = PORT): Promise console.error(`server certificate renewal failed: ${errMessage(e)}`)); + .catch((e) => log.failure('server certificate renewal failed', e, {}, 'error')); }, Math.min(SERVER_LIFETIME_MS / 90, 3_600_000), ); @@ -5365,12 +5509,13 @@ export async function listen(app: express.Express, port = PORT): Promise { (app.locals.stopBackupScheduler as () => void)(); + (app.locals.stopNotifications as () => void)(); clearInterval(sweep); clearInterval(renew); }); return new Promise((resolve) => server.listen(port, () => { - console.log(`tug-control listening on ${port} (mTLS)`); + log.info('tug-control listening (mTLS)', { port }); resolve(server); }), ); diff --git a/src/control/credentials.test.ts b/src/control/credentials.test.ts index 83681e3..0c4dac0 100644 --- a/src/control/credentials.test.ts +++ b/src/control/credentials.test.ts @@ -222,6 +222,8 @@ const SAMPLES: Record = { 'GET /api/v1/holds': { path: '/api/v1/holds' }, 'GET /api/v1/status': { path: '/api/v1/status' }, 'GET /api/v1/version': { path: '/api/v1/version' }, + 'GET /api/v1/readyz': { path: '/api/v1/readyz' }, + 'GET /api/v1/metrics': { path: '/api/v1/metrics' }, 'GET /api/v1/whoami': { path: '/api/v1/whoami' }, 'GET /api/v1/events': { path: '/api/v1/events' }, 'GET /api/v1/projects/:project': { path: '/api/v1/projects/site' }, @@ -530,6 +532,15 @@ test('project-scoped credentials only reach their own projects and learn nothing 'a scoped credential cannot create a project outside its scope', ); + const metrics = await s.call(viewer, 'GET', '/api/v1/metrics'); + assert.equal(metrics.status, 200, 'a scoped read credential can scrape its metrics'); + assert.ok(metrics.text.includes('project="a"'), 'metrics include the authorized project'); + assert.ok(!metrics.text.includes('project="b"'), 'metrics exclude other projects'); + assert.ok( + !metrics.text.includes('tug_writer_'), + 'global writer state is hidden from scoped metrics', + ); + assert.ok(!metrics.text.includes('node="'), 'global node names are hidden from scoped metrics'); const status = (await s.call(viewer, 'GET', '/api/v1/status')).body; assert.deepEqual( status.projects.map((p: { id: string }) => p.id), diff --git a/src/control/credentials.ts b/src/control/credentials.ts index 1ef7fce..ca3360d 100644 --- a/src/control/credentials.ts +++ b/src/control/credentials.ts @@ -31,6 +31,8 @@ export const ROUTE_ROLES: Record = { 'GET /api/v1/holds': 'admin', 'GET /api/v1/status': 'read', 'GET /api/v1/version': 'read', + 'GET /api/v1/readyz': 'read', + 'GET /api/v1/metrics': 'read', 'GET /api/v1/whoami': 'read', 'GET /api/v1/events': 'read', 'GET /api/v1/projects/:project': 'read', diff --git a/src/control/images.test.ts b/src/control/images.test.ts index 3a0bed6..8f15e12 100644 --- a/src/control/images.test.ts +++ b/src/control/images.test.ts @@ -1006,14 +1006,17 @@ test('image prune counts distinct builds, so redeploying an unchanged image keep } }); -test('a failing build-cache prune is logged without breaking later deploys', async () => { +test('a failing build-cache prune is logged without breaking later deploys', async (t) => { const s = await server(); try { const n1 = await s.node('n1'); s.failMaintenance(); - const originalError = console.error; + const originalWrite = process.stderr.write; const logged: string[] = []; - console.error = (...args: unknown[]) => void logged.push(args.join(' ')); + t.mock.method(process.stderr, 'write', (chunk: string | Uint8Array) => { + logged.push(String(chunk)); + return true; + }); try { const first = await json(await s.deploy('v1')); await until( @@ -1022,11 +1025,16 @@ test('a failing build-cache prune is logged without breaking later deploys', asy ); await s.report(n1.nodeId, first.desiredState); } finally { - console.error = originalError; + process.stderr.write = originalWrite; } assert.ok( - logged.some((line) => line.includes('image maintenance failed: prune broke')), - 'a failing maintenance step must be logged with its reason', + logged.some( + (line) => + line.includes('image maintenance failed') && + line.includes('failureCode=') && + !line.includes('prune broke'), + ), + 'a failing maintenance step is logged without arbitrary private error details', ); assert.equal( (await s.deploy('v2')).status, diff --git a/src/control/metrics.test.ts b/src/control/metrics.test.ts new file mode 100644 index 0000000..acadf4f --- /dev/null +++ b/src/control/metrics.test.ts @@ -0,0 +1,171 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import fs from 'node:fs/promises'; +import { renderMetricsSnapshot } from './metrics.ts'; + +test('metrics render ages, report presence, deployment outcomes, backup state and queue values', () => { + const now = Date.parse('2026-01-01T00:00:10.000Z'); + const output = renderMetricsSnapshot({ + now, + nodes: [ + { name: 'edge-a', lastSeenAt: '2026-01-01T00:00:05.000Z' }, + { name: 'edge-b', lastSeenAt: null }, + ], + deployments: [ + { project: 'shop', state: 'succeeded', outcome: 'success', count: 2 }, + { project: 'shop', state: 'failed', outcome: 'failure' }, + { project: 'shop', state: 'unexpected-secret-state', outcome: 'unexpected' }, + ], + backups: [ + { + project: 'shop', + target: 'db', + state: 'succeeded', + succeededAt: '2026-01-01T00:00:02.000Z', + }, + { project: 'shop', target: 'uploads', state: 'failed' }, + ], + queue: { depth: 2.7, activeSince: '2026-01-01T00:00:08.000Z' }, + }); + + assert.match( + output, + /tug_node_poll_age_seconds\{node="edge-a"\} 5/, + 'node poll age is in seconds', + ); + assert.match( + output, + /tug_node_poll_age_seconds\{node="edge-b"\} NaN/, + 'unknown node poll age is represented as missing rather than fresh', + ); + assert.match( + output, + /tug_node_reported\{node="edge-b"\} 0/, + 'never-reported nodes are not presented as fresh', + ); + assert.match( + output, + /tug_deployments\{project="shop",state="succeeded",outcome="success"\} 2/, + 'deployment row counts are aggregated', + ); + assert.match( + output, + /state="other",outcome="other"/, + 'deployment labels are constrained to finite values', + ); + assert.match( + output, + /tug_backup_success_age_seconds\{project="shop",target="db"\} 8/, + 'backup success age is in seconds', + ); + assert.match( + output, + /tug_backup_targets\{project="shop",target="uploads",state="failed"\} 1/, + 'backup target states are reported from the supplied schedule snapshot', + ); + assert.match( + output, + /tug_backup_has_success\{project="shop",target="uploads"\} 0/, + 'configured targets without a success are explicit', + ); + assert.match( + output, + /tug_backup_success_age_seconds\{project="shop",target="uploads"\} NaN/, + 'unknown backup success age is represented as missing rather than fresh', + ); + assert.match( + output, + /tug_writer_queue_depth 2/, + 'global queue depth is non-negative and integral', + ); + assert.match(output, /tug_writer_active_age_seconds 2/, 'global active queue age is in seconds'); + assert.ok(output.endsWith('\n'), 'Prometheus output ends in a newline'); +}); + +test('metrics escape label values and never include resource IDs or error text', () => { + const output = renderMetricsSnapshot({ + now: 1000, + nodes: [{ name: 'edge"\\\nname', lastSeenAt: 0 }], + deployments: [{ project: 'shop', state: 'failed' }], + }); + + assert.match( + output, + /node="edge\\"\\\\\\nname"/, + 'Prometheus labels escape quotes, backslashes and newlines', + ); + assert.doesNotMatch( + output, + /deployment_id|error_message|failure detail/, + 'resource IDs and error details are never exported', + ); +}); + +test('global queue values are rendered without project or resource labels', () => { + const output = renderMetricsSnapshot({ queue: { depth: 0 } }); + assert.match( + output, + /tug_writer_queue_depth 0/, + 'global queue depth has no project or resource labels', + ); + assert.match(output, /tug_writer_active_age_seconds 0/, 'idle global queue age is zero seconds'); +}); + +test('optional webhook metrics use counters for totals and a gauge for pending work', () => { + const values = { delivered: 4, failed: 2, dropped: 1 }; + const output = renderMetricsSnapshot({ + webhook: { ...values, enabled: true, pending: 3 }, + }); + assert.match(output, /# HELP tug_webhook_enabled /, 'enabled metric has Prometheus HELP text'); + assert.match(output, /# TYPE tug_webhook_enabled gauge/, 'enabled status is a gauge'); + assert.match(output, /tug_webhook_enabled 1/, 'enabled webhook reports one'); + for (const [key, value] of Object.entries(values)) { + assert.match( + output, + new RegExp(`# TYPE tug_webhook_${key}_total counter`), + `webhook ${key} total is a counter`, + ); + assert.match( + output, + new RegExp(`tug_webhook_${key}_total ${value}`), + `webhook ${key} total is rendered`, + ); + } + assert.match(output, /# HELP tug_webhook_pending /, 'pending metric has Prometheus HELP text'); + assert.match(output, /# TYPE tug_webhook_pending gauge/, 'pending work is a gauge'); + assert.match(output, /tug_webhook_pending 3/, 'pending work reports its current value'); + assert.doesNotMatch( + renderMetricsSnapshot({ webhook: { ...values, enabled: false, pending: 0 } }), + /tug_webhook_enabled 1/, + 'a disabled webhook reports zero enabled state', + ); + assert.doesNotMatch( + renderMetricsSnapshot({}), + /tug_webhook_(?:enabled|delivered|failed|dropped|pending)/, + 'webhook metrics are omitted when the snapshot has no global webhook stats', + ); +}); + +test('omitting global queue values omits samples from project-scoped snapshots', () => { + const output = renderMetricsSnapshot({ deployments: [{ project: 'shop', state: 'succeeded' }] }); + assert.doesNotMatch( + output, + /tug_writer_queue_depth\s+\d/, + 'project metrics omit global writer queue depth', + ); + assert.doesNotMatch( + output, + /tug_writer_active_age_seconds\s+\d/, + 'project metrics omit global writer age', + ); +}); + +test('every exported metric family is documented for operators', async () => { + const docs = await fs.readFile(new URL('../../docs/observability.md', import.meta.url), 'utf8'); + const output = renderMetricsSnapshot({ + queue: { depth: 0 }, + webhook: { enabled: false, delivered: 0, failed: 0, dropped: 0, pending: 0 }, + }); + for (const match of output.matchAll(/^# HELP (\w+)/gm)) + assert.ok(docs.includes(match[1]), 'observability docs must explain ' + match[1]); +}); diff --git a/src/control/metrics.ts b/src/control/metrics.ts new file mode 100644 index 0000000..4241a2e --- /dev/null +++ b/src/control/metrics.ts @@ -0,0 +1,188 @@ +/** Input to the Prometheus snapshot renderer. Callers pass only rows visible to the requester. */ +export interface MetricsSnapshot { + now?: number; + nodes?: Array<{ name: string; lastSeenAt: string | number | null }>; + deployments?: Array<{ project: string; state: string; count?: number; outcome?: string | null }>; + backups?: Array<{ + project: string; + target: string; + state: string; + succeededAt?: string | number | null; + }>; + queue?: { depth: number; activeSince?: string | number | null }; + webhook?: { + enabled?: boolean; + delivered: number; + failed: number; + dropped: number; + pending: number; + }; +} + +const label = (value: string): string => + value.replaceAll('\\', '\\\\').replaceAll('\n', '\\n').replaceAll('"', '\\"'); + +const labels = (values: Record): string => + `{${Object.entries(values) + .map(([key, value]) => `${key}="${label(value)}"`) + .join(',')}}`; + +const timestamp = (value: string | number | null | undefined): number | undefined => { + if (value === null || value === undefined) return undefined; + const result = typeof value === 'number' ? value : Date.parse(value); + return Number.isFinite(result) ? result : undefined; +}; + +const age = ( + now: number, + since: string | number | null | undefined, + unknown: number = Number.NaN, +): number => { + const time = timestamp(since); + return time === undefined ? unknown : Math.max(0, (now - time) / 1000); +}; + +const finiteState = (value: string, allowed: readonly string[]): string => + allowed.includes(value) ? value : 'other'; + +const countLines = ( + name: string, + help: string, + samples: Map; count: number }>, +): string[] => [ + `# HELP ${name} ${help}`, + `# TYPE ${name} gauge`, + ...[...samples.values()].map((sample) => `${name}${labels(sample.labels)} ${sample.count}`), +]; + +/** Renders a stable, low-cardinality Prometheus snapshot without resource IDs or error text. */ +export function renderMetricsSnapshot(snapshot: MetricsSnapshot): string { + const now = snapshot.now ?? Date.now(); + const lines = [ + '# HELP tug_node_poll_age_seconds Seconds since a node last reported status; NaN means never reported.', + '# TYPE tug_node_poll_age_seconds gauge', + ...(snapshot.nodes ?? []).map( + (node) => + `tug_node_poll_age_seconds${labels({ node: node.name })} ${age(now, node.lastSeenAt)}`, + ), + '# HELP tug_node_reported Whether a node has ever reported status (1) or has never reported (0).', + '# TYPE tug_node_reported gauge', + ...(snapshot.nodes ?? []).map( + (node) => + `tug_node_reported${labels({ node: node.name })} ${timestamp(node.lastSeenAt) === undefined ? 0 : 1}`, + ), + ]; + + const deploymentCounts = new Map; count: number }>(); + for (const deployment of snapshot.deployments ?? []) { + const state = finiteState(deployment.state, [ + 'building', + 'build_failed', + 'rolling_out', + 'succeeded', + 'failed', + 'cancelled', + ]); + const outcome = deployment.outcome + ? finiteState(deployment.outcome, ['success', 'failure', 'cancelled']) + : ''; + const sampleLabels = { project: deployment.project, state, ...(outcome ? { outcome } : {}) }; + const key = JSON.stringify(sampleLabels); + const sample = deploymentCounts.get(key) ?? { labels: sampleLabels, count: 0 }; + sample.count += Math.max(0, Math.floor(deployment.count ?? 1)); + deploymentCounts.set(key, sample); + } + lines.push( + ...countLines( + 'tug_deployments', + 'Current deployment count by project and finite deployment state.', + deploymentCounts, + ), + ); + + const backupCounts = new Map; count: number }>(); + const backupSuccesses = new Map< + string, + { labels: Record; age: number; hasSuccess: boolean } + >(); + for (const backup of snapshot.backups ?? []) { + const state = finiteState(backup.state, [ + 'never', + 'disabled', + 'queued', + 'running', + 'succeeded', + 'failed', + 'cancelled', + 'missed', + 'unknown', + ]); + const stateLabels = { project: backup.project, target: backup.target, state }; + const stateKey = JSON.stringify(stateLabels); + const stateSample = backupCounts.get(stateKey) ?? { labels: stateLabels, count: 0 }; + stateSample.count++; + backupCounts.set(stateKey, stateSample); + const successLabels = { project: backup.project, target: backup.target }; + const successKey = JSON.stringify(successLabels); + const successAge = age(now, backup.succeededAt); + const hasSuccess = timestamp(backup.succeededAt) !== undefined; + const existing = backupSuccesses.get(successKey); + if (!existing || (hasSuccess && (!existing.hasSuccess || successAge < existing.age))) + backupSuccesses.set(successKey, { labels: successLabels, age: successAge, hasSuccess }); + } + lines.push( + ...countLines( + 'tug_backup_targets', + 'Configured backup target count by project and finite latest backup state in the supplied snapshot.', + backupCounts, + ), + '# HELP tug_backup_success_age_seconds Seconds since the most recent successful backup per project and target; NaN means no success is known.', + '# TYPE tug_backup_success_age_seconds gauge', + ...[...backupSuccesses.values()].map( + (sample) => `tug_backup_success_age_seconds${labels(sample.labels)} ${sample.age}`, + ), + '# HELP tug_backup_has_success Whether a successful backup exists for a configured project target.', + '# TYPE tug_backup_has_success gauge', + ...[...backupSuccesses.values()].map( + (sample) => `tug_backup_has_success${labels(sample.labels)} ${sample.hasSuccess ? 1 : 0}`, + ), + ); + + if (snapshot.queue) + lines.push( + '# HELP tug_writer_queue_depth Number of pending items in the global control writer queue.', + '# TYPE tug_writer_queue_depth gauge', + ...(snapshot.queue + ? [`tug_writer_queue_depth ${Math.max(0, Math.floor(snapshot.queue.depth))}`] + : []), + '# HELP tug_writer_active_age_seconds Seconds since the global control writer queue item started; zero means idle or unknown.', + '# TYPE tug_writer_active_age_seconds gauge', + ...(snapshot.queue + ? [`tug_writer_active_age_seconds ${age(now, snapshot.queue.activeSince, 0)}`] + : []), + ); + + if (snapshot.webhook) { + if (snapshot.webhook.enabled !== undefined) + lines.push( + '# HELP tug_webhook_enabled Whether webhook delivery is enabled.', + '# TYPE tug_webhook_enabled gauge', + `tug_webhook_enabled ${snapshot.webhook.enabled ? 1 : 0}`, + ); + for (const key of ['delivered', 'failed', 'dropped'] as const) { + const name = `tug_webhook_${key}_total`; + lines.push( + `# HELP ${name} Total webhook ${key} events since control started.`, + `# TYPE ${name} counter`, + `${name} ${Math.max(0, Math.floor(snapshot.webhook[key]))}`, + ); + } + lines.push( + '# HELP tug_webhook_pending Current number of webhook deliveries waiting or in progress.', + '# TYPE tug_webhook_pending gauge', + `tug_webhook_pending ${Math.max(0, Math.floor(snapshot.webhook.pending))}`, + ); + } + + return `${lines.join('\n')}\n`; +} diff --git a/src/control/notifications.test.ts b/src/control/notifications.test.ts new file mode 100644 index 0000000..5885dfd --- /dev/null +++ b/src/control/notifications.test.ts @@ -0,0 +1,274 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { createHmac } from 'node:crypto'; +import { + createWebhookNotifier, + validateWebhookUrl, + webhookOptionsFromEnv, +} from './notifications.ts'; + +const baseEvent = { + id: 'ev-1', + type: 'node.stale', + level: 'warning', + message: 'node n1 has stopped reporting', + metadata_json: JSON.stringify({ project: 'alpha', access_token: 'secret-token' }), + project_id: 'alpha', + created_at: '2026-11-15T00:00:00.000Z', +}; + +test('webhook sends a signed minimal payload and omits arbitrary metadata', async () => { + let request: RequestInit | undefined; + const notifier = createWebhookNotifier({ + url: 'https://hooks.example.test/ingest', + secret: 'unit-test-secret', + now: () => 1_800_000_123_000, + fetch: async (_url, init) => { + request = init; + return new Response(null, { status: 204 }); + }, + }); + assert.equal(notifier.enqueue(baseEvent), true, 'new state transition enters the queue'); + await notifier.close(); + const body = String(request?.body); + const timestamp = '1800000123'; + const expectedSignature = createHmac('sha256', 'unit-test-secret') + .update(`${timestamp}.${body}`) + .digest('hex'); + assert.equal( + (request?.headers as Record)['x-tug-webhook-signature'], + `sha256=${expectedSignature}`, + 'signature authenticates the exact timestamp and body', + ); + const wire = JSON.parse(body) as { event: Record }; + assert.equal(wire.event.id, 'ev-1', 'delivery identifies the transition'); + assert.match( + String(wire.event.message), + /Check node connectivity/, + 'actionable guidance is included', + ); + assert.equal('metadata_json' in wire.event, false, 'arbitrary metadata is never sent'); + assert.equal( + 'created_at' in wire.event, + false, + 'only resource identifiers and guidance are sent', + ); + assert.doesNotMatch( + body, + /secret-token|node n1 has stopped reporting/, + 'stored metadata and raw messages cannot leak', + ); + assert.equal(request?.redirect, 'manual', 'destination redirects are not followed'); +}); + +test('only whitelisted transitions send generic guidance and raw event messages are ignored', async () => { + const sent: Record[] = []; + const notifier = createWebhookNotifier({ + url: 'https://hooks.example.test', + secret: 'secret', + fetch: async (_url, init) => { + sent.push(JSON.parse(String(init?.body)).event); + return new Response(null, { status: 204 }); + }, + }); + assert.equal( + notifier.enqueue({ + ...baseEvent, + id: 'job-1', + type: 'job.run', + message: 'ci ran a one-off web container on n1: tool --password=hunter2', + metadata_json: JSON.stringify({ + node: 'n1', + service: 'web', + command: ['--password=hunter2'], + }), + }), + false, + 'unapproved event types are rejected', + ); + assert.equal( + notifier.stats.dropped, + 0, + 'unsupported informational events are filtered without counting delivery loss', + ); + notifier.enqueue({ + ...baseEvent, + id: 'other-1', + type: 'node.stale', + message: 'ci: password=hunter2', + }); + await notifier.close(); + assert.equal(sent.length, 1, 'one approved transition is sent'); + assert.match(String(sent[0].message), /Check node connectivity/, 'fixed guidance is sent'); + assert.doesNotMatch( + JSON.stringify(sent), + /hunter2|ci:/, + 'raw event messages and command secrets are absent', + ); +}); + +test('backup, restore, and ingress alerts are whitelisted with generic safe guidance', async () => { + const sent: Record[] = []; + const notifier = createWebhookNotifier({ + url: 'https://hooks.example.test', + secret: 'secret', + fetch: async (_url, init) => { + sent.push(JSON.parse(String(init?.body)).event); + return new Response(null, { status: 204 }); + }, + }); + for (const [index, type] of [ + 'backup.failed', + 'restore.failed', + 'restore.completed', + 'ingress.failed', + ].entries()) { + notifier.enqueue({ + ...baseEvent, + id: `safe-${index}`, + type, + message: 'database password=hunter2', + metadata_json: '{"error":"token=private"}', + }); + } + await notifier.close(); + assert.equal(sent.length, 4, 'supported operational outcomes are delivered'); + assert.doesNotMatch( + JSON.stringify(sent), + /hunter2|private/, + 'error messages and metadata are never sent', + ); + assert.ok( + sent.every((event) => typeof event.message === 'string' && event.message.length > 0), + 'each alert has generic guidance', + ); +}); + +test('retries are bounded with capped exponential delays and duplicate transition IDs are ignored', async () => { + let attempts = 0; + const delays: number[] = []; + const notifier = createWebhookNotifier({ + url: 'https://hooks.example.test', + secret: 'secret', + maxAttempts: 4, + retryBaseMs: 10, + retryMaxMs: 15, + sleep: async (ms) => void delays.push(ms), + fetch: async () => { + attempts++; + return new Response(null, { status: attempts < 4 ? 503 : 204 }); + }, + }); + assert.equal(notifier.enqueue(baseEvent), true, 'first transition is accepted'); + assert.equal(notifier.enqueue(baseEvent), false, 'same transition is deduplicated'); + await notifier.close(); + assert.equal(attempts, 4, 'delivery stops after eventual success'); + assert.deepEqual(delays, [10, 15, 15], 'backoff grows and stays within configured cap'); + assert.deepEqual( + notifier.stats, + { delivered: 1, failed: 0, dropped: 0, pending: 0 }, + 'delivery counters reflect completion', + ); +}); + +test('request timeout aborts delivery, retries to the configured limit, and counts exhausted failures', async () => { + let attempts = 0; + const errors: string[] = []; + const notifier = createWebhookNotifier({ + url: 'https://hooks.example.test', + secret: 'secret', + timeoutMs: 2, + maxAttempts: 2, + retryBaseMs: 1, + sleep: async () => {}, + onError: (eventId) => errors.push(eventId), + fetch: async (_url, init) => { + attempts++; + return new Promise((_resolve, reject) => { + init?.signal?.addEventListener('abort', () => + reject(new DOMException('aborted', 'AbortError')), + ); + }); + }, + }); + notifier.enqueue(baseEvent); + await notifier.close(); + assert.equal(attempts, 2, 'timeout retries stop at the configured attempt limit'); + assert.deepEqual(errors, ['ev-1'], 'failure hook receives only the event ID'); + assert.deepEqual( + notifier.stats, + { delivered: 0, failed: 1, dropped: 0, pending: 0 }, + 'exhaustion is counted', + ); +}); + +test('queue overflow rejects immediately without waiting for an in-flight request', async () => { + let releaseFetch!: () => void; + const blockedFetch = new Promise((resolve) => { + releaseFetch = () => resolve(new Response(null, { status: 204 })); + }); + const notifier = createWebhookNotifier({ + url: 'https://hooks.example.test', + secret: 'secret', + maxQueue: 1, + fetch: async () => blockedFetch, + }); + assert.equal(notifier.enqueue(baseEvent), true, 'first transition is accepted synchronously'); + await new Promise((resolve) => queueMicrotask(resolve)); + const startedAt = Date.now(); + assert.equal( + notifier.enqueue({ ...baseEvent, id: 'ev-2' }), + false, + 'full queue rejects new work', + ); + assert.equal(notifier.stats.dropped, 1, 'queue overflow increments dropped count'); + assert.ok(Date.now() - startedAt < 100, 'enqueue does not wait for the network'); + releaseFetch(); + await notifier.close(); +}); + +test('destination validation requires HTTPS except explicitly allowed loopback test URLs', () => { + assert.throws( + () => validateWebhookUrl('http://hooks.example.test'), + /HTTPS/, + 'remote HTTP is rejected', + ); + assert.throws( + () => validateWebhookUrl('https://user:pass@example.test'), + /credentials/, + 'URL credentials are rejected', + ); + assert.equal( + validateWebhookUrl('http://127.0.0.1:8080/hook', true).hostname, + '127.0.0.1', + 'test loopback HTTP is injectable', + ); + assert.throws( + () => webhookOptionsFromEnv({ TUG_WEBHOOK_URL: 'https://hooks.test' }), + /both/, + 'partial environment config fails clearly', + ); +}); + +test('shutdown aborts active delivery and discards queued work', async () => { + let requests = 0; + const notifier = createWebhookNotifier({ + url: 'https://hooks.example.test', + secret: 'secret', + fetch: async (_url, init) => { + requests++; + return new Promise((_resolve, reject) => + init?.signal?.addEventListener('abort', () => reject(new Error('aborted')), { once: true }), + ); + }, + }); + notifier.enqueue({ ...baseEvent, id: 'shutdown-one' }); + notifier.enqueue({ ...baseEvent, id: 'shutdown-two' }); + await new Promise((resolve) => setImmediate(resolve)); + notifier.stop(); + notifier.stop(); // Server and process shutdown paths may both call it. + await notifier.close(); + assert.equal(requests, 1, 'shutdown does not attempt waiting notifications'); + assert.equal(notifier.pending, 0, 'shutdown clears active and waiting work'); + assert.equal(notifier.stats.dropped, 2, 'shutdown reports discarded delivery work'); +}); diff --git a/src/control/notifications.ts b/src/control/notifications.ts new file mode 100644 index 0000000..2f982f9 --- /dev/null +++ b/src/control/notifications.ts @@ -0,0 +1,275 @@ +import { createHmac } from 'node:crypto'; +import type { RedactableEvent } from './event-redaction.ts'; + +export interface NotificationEvent extends RedactableEvent { + id: string; + level?: string | null; + project_id?: string | null; + deployment_id?: string | null; + node_id?: string | null; +} + +export interface WebhookOptions { + url: string; + secret: string; + fetch?: typeof fetch; + timeoutMs?: number; + maxAttempts?: number; + retryBaseMs?: number; + retryMaxMs?: number; + maxQueue?: number; + maxRememberedIds?: number; + now?: () => number; + sleep?: (ms: number) => Promise; + /** Permit HTTP loopback endpoints for tests that use a local receiver. */ + allowHttpLoopback?: boolean; + /** Called after retries are exhausted; receives only the transition ID, never request/error details. */ + onError?: (eventId: string) => void; +} + +export interface WebhookNotifier { + /** Enqueues without waiting for the network. Returns false when disabled or full. */ + enqueue(event: NotificationEvent): boolean; + /** Stops new work and waits for the current queue to drain. */ + close(): Promise; + /** Discards waiting work and aborts delivery during process shutdown. */ + stop(): void; + readonly pending: number; + readonly stats: Readonly<{ delivered: number; failed: number; dropped: number; pending: number }>; +} + +interface WireEvent { + id: string; + type: string; + message: string; + project_id?: string | null; + deployment_id?: string | null; + node_id?: string | null; +} + +const defaultSleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +const TRANSITION_MESSAGES: Record = { + 'control.backup_failed': + 'Control backup failed. Check control backup status and the off-host destination.', + 'node.stale': 'Node stopped reporting. Check node connectivity and the tug agent service.', + 'node.recovered': 'Node is reporting again. Confirm its services and deployments are healthy.', + 'service.failed': 'A service failed. Check the service status and recent deployment.', + 'container.crashloop': 'A container keeps restarting. Check its health and recent logs.', + 'container.oom': 'A container ran out of memory. Check memory use and workload limits.', + 'container.unhealthy': + 'A container failed its health check. Check its status and health endpoint.', + 'deploy.failed': + 'A deployment failed. Inspect deployment status and retry after resolving the cause.', + 'deployment.failed': + 'A deployment failed. Inspect deployment status and retry after resolving the cause.', + 'deployment.build_failed': 'A deployment build failed. Check build inputs and retry the build.', + 'deploy.succeeded': 'A deployment completed. Confirm the application is healthy.', + 'deployment.succeeded': 'A deployment completed. Confirm the application is healthy.', + 'build.failed': 'An image build failed. Check build inputs and retry the build.', + build_failed: 'An image build failed. Check build inputs and retry the build.', + 'rollback.auto': 'An automatic rollback ran. Confirm the previous deployment is healthy.', + 'deployment.auto_rollback': + 'An automatic rollback ran. Confirm the previous deployment is healthy.', + 'deployment.auto_rollback_skipped': + 'Automatic rollback was skipped. Review deployment health and rollback settings.', + 'backup.failed': 'A backup failed. Check the target and backup destination, then retry it.', + 'backup.missed': 'A scheduled backup was missed. Check the schedule and backup destination.', + 'backup.stale': 'A backup is taking longer than expected. Check the target node and backup job.', + 'backup.completed': 'A backup completed. Confirm the latest recovery point is available.', + 'backup.no_destination': 'A backup was skipped because no destination is configured.', + 'restore.failed': + 'A restore failed. Review the restore status before releasing any held writers.', + 'restore.interrupted': + 'A restore was interrupted. Review its status before releasing held writers.', + 'restore.succeeded': 'A restore completed. Confirm the restored service is healthy.', + 'restore.completed': 'A restore completed. Confirm the restored service is healthy.', + 'ingress.failed': 'Ingress health failed. Check the ingress node and its recent logs.', +}; + +/** Whitelist the node state transitions and build a safe, generic outbound message. */ +export function sanitizeNotificationEvent(event: NotificationEvent): WireEvent | undefined { + const message = TRANSITION_MESSAGES[event.type]; + if (!message) return undefined; + return { + id: event.id, + type: event.type, + message, + ...(event.project_id !== undefined ? { project_id: event.project_id } : {}), + ...(event.deployment_id !== undefined ? { deployment_id: event.deployment_id } : {}), + ...(event.node_id !== undefined ? { node_id: event.node_id } : {}), + }; +} + +function isLoopback(hostname: string): boolean { + const host = hostname.toLowerCase().replace(/^\[|\]$/g, ''); + return host === 'localhost' || host === '::1' || host === '127.0.0.1' || host.startsWith('127.'); +} + +export function validateWebhookUrl(rawUrl: string, allowHttpLoopback = false): URL { + let url: URL; + try { + url = new URL(rawUrl); + } catch { + throw new Error('Webhook URL must be an absolute HTTPS URL'); + } + if (url.username || url.password) throw new Error('Webhook URL must not contain credentials'); + if ( + url.protocol !== 'https:' && + !(allowHttpLoopback && url.protocol === 'http:' && isLoopback(url.hostname)) + ) + throw new Error('Webhook destination must use HTTPS'); + return url; +} + +/** Build a best-effort webhook sender. Network work runs in a bounded background queue. */ +export function createWebhookNotifier(options: WebhookOptions): WebhookNotifier { + const url = validateWebhookUrl(options.url, options.allowHttpLoopback); + if (!options.secret) throw new Error('Webhook signing secret must not be empty'); + const fetcher = options.fetch ?? fetch; + const now = options.now ?? Date.now; + const sleep = options.sleep ?? defaultSleep; + const timeoutMs = options.timeoutMs ?? 5_000; + const maxAttempts = options.maxAttempts ?? 3; + const retryBaseMs = options.retryBaseMs ?? 250; + const retryMaxMs = options.retryMaxMs ?? 2_000; + const maxQueue = options.maxQueue ?? 100; + const maxRememberedIds = options.maxRememberedIds ?? 2_000; + const queue: WireEvent[] = []; + const seen = new Set(); + let running = false; + let closed = false; + let stopped = false; + let activeController: AbortController | undefined; + let idleWaiters: (() => void)[] = []; + const counts = { delivered: 0, failed: 0, dropped: 0 }; + + function remember(id: string) { + seen.add(id); + if (seen.size > maxRememberedIds) seen.delete(seen.values().next().value as string); + } + + async function deliver(event: WireEvent) { + const safeEvent = event; + const timestamp = Math.floor(now() / 1000).toString(); + const body = JSON.stringify({ timestamp: Number(timestamp), event: safeEvent }); + const signature = createHmac('sha256', options.secret) + .update(`${timestamp}.${body}`) + .digest('hex'); + for (let attempt = 0; attempt < maxAttempts; attempt++) { + if (stopped) return; + const controller = new AbortController(); + activeController = controller; + const timeout = setTimeout(() => controller.abort(), timeoutMs); + try { + const response = await fetcher(url, { + method: 'POST', + redirect: 'manual', + signal: controller.signal, + headers: { + 'content-type': 'application/json', + 'x-tug-webhook-id': event.id, + 'x-tug-webhook-timestamp': timestamp, + 'x-tug-webhook-signature': `sha256=${signature}`, + }, + body, + }); + await response.body?.cancel(); + if (response.ok) { + counts.delivered++; + return; + } + if ( + response.status >= 400 && + response.status < 500 && + ![408, 429].includes(response.status) + ) + break; + } catch { + // Failure details may include destination URLs, so retain only the outcome counter. + } finally { + clearTimeout(timeout); + activeController = undefined; + } + if (attempt + 1 < maxAttempts) { + await sleep(Math.min(retryBaseMs * 2 ** attempt, retryMaxMs)); + } + } + counts.failed++; + try { + options.onError?.(event.id); + } catch { + // Metrics hooks must not stop delivery of later transitions. + } + } + + function notifyIdle() { + if (!running && queue.length === 0) { + for (const resolve of idleWaiters) resolve(); + idleWaiters = []; + } + } + + async function drain() { + if (running) return; + running = true; + try { + while (queue.length) { + const event = queue.shift()!; + await deliver(event); + } + } finally { + running = false; + notifyIdle(); + } + } + + return { + enqueue(event) { + if (closed || seen.has(event.id)) return false; + const safe = sanitizeNotificationEvent(event); + if (!safe) return false; + if (queue.length + (running ? 1 : 0) >= maxQueue) { + counts.dropped++; + return false; + } + remember(event.id); + queue.push(safe); + queueMicrotask(() => void drain()); + return true; + }, + stop() { + if (stopped) return; + closed = true; + stopped = true; + counts.dropped += queue.length + (running ? 1 : 0); + queue.length = 0; + activeController?.abort(); + notifyIdle(); + }, + async close() { + closed = true; + if (!running && queue.length === 0) return; + await new Promise((resolve) => idleWaiters.push(resolve)); + }, + get pending() { + return queue.length + (running ? 1 : 0); + }, + get stats() { + return { ...counts, pending: queue.length + (running ? 1 : 0) }; + }, + }; +} + +/** Read the minimal environment configuration; no webhook means notifications stay disabled. */ +export function webhookOptionsFromEnv( + env: NodeJS.ProcessEnv = process.env, +): Pick | undefined { + const url = env.TUG_WEBHOOK_URL?.trim(); + const secret = env.TUG_WEBHOOK_SECRET; + if (!url && !secret) return undefined; + if (!url || !secret) + throw new Error('Set both TUG_WEBHOOK_URL and TUG_WEBHOOK_SECRET to enable webhooks'); + validateWebhookUrl(url); + return { url, secret }; +} diff --git a/src/control/reliability-monitor.test.ts b/src/control/reliability-monitor.test.ts new file mode 100644 index 0000000..ff7663a --- /dev/null +++ b/src/control/reliability-monitor.test.ts @@ -0,0 +1,149 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs/promises'; +import os from 'node:os'; +import path from 'node:path'; +import { openDb } from '../db/db.ts'; +import { recordEvent } from './credentials.ts'; +import { createReliabilityMonitor } from './reliability-monitor.ts'; +import type { NotificationEvent } from './notifications.ts'; + +test('committed failures and stale/recovered nodes are observed once without an operator watching', async () => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'tug-monitor-')); + const db = await openDb(path.join(dir, 'tug.db')); + let time = Date.now(); + const seen: NotificationEvent[] = []; + try { + recordEvent(db, { actor: 'test', type: 'service.failed', message: 'historical' }); + const monitor = createReliabilityMonitor( + db, + (e) => { + seen.push(e); + }, + () => time, + ); + db.run( + 'INSERT INTO nodes(id,name,last_seen_at) VALUES(?,?,?)', + 'node1', + 'app1', + new Date(time - 3600000).toISOString(), + ); + db.run( + 'INSERT INTO deployments(id,project_id,state,created_at) VALUES(?,?,?,?)', + 'deploy1', + 'site', + 'rolling_out', + new Date(time).toISOString(), + ); + time += 60_001; + monitor.tick(); + monitor.tick(); + assert.deepEqual( + seen.map((e) => e.type), + ['node.stale'], + 'staleness is detected once and historical events are not replayed', + ); + recordEvent(db, { + actor: 'test', + type: 'backup.failed', + message: 'private details', + project: 'site', + }); + db.run('UPDATE deployments SET state=? WHERE id=?', 'failed', 'deploy1'); + db.run('UPDATE nodes SET last_seen_at=? WHERE id=?', new Date(time).toISOString(), 'node1'); + time++; + monitor.tick(); + monitor.tick(); + assert.deepEqual( + seen.map((e) => e.type), + ['node.stale', 'backup.failed', 'deployment.failed', 'node.recovered'], + 'committed events and state transitions are detected once', + ); + db.run( + 'UPDATE nodes SET ingress_status_json=? WHERE id=?', + JSON.stringify({ state: 'failed', error: 'private-ingress-detail' }), + 'node1', + ); + db.run( + 'INSERT INTO backups(id,project_id,state,created_at) VALUES(?,?,?,?)', + 'backup-failed', + '_control', + 'failed', + new Date(time).toISOString(), + ); + recordEvent(db, { + actor: 'test', + type: 'backup.failed', + project: '_control', + message: 'private control backup cause', + }); + monitor.tick(); + monitor.tick(); + assert.deepEqual( + seen.slice(-2).map((e) => e.type), + ['backup.failed', 'ingress.failed'], + 'control protection and ingress failures notify once', + ); + } finally { + db.close(); + await fs.rm(dir, { recursive: true, force: true }); + } +}); + +test('never-seen nodes and reconnecting nodes during startup grace do not page operators', async () => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'tug-grace-')); + const db = await openDb(path.join(dir, 'tug.db')); + let time = Date.now(); + const seen: NotificationEvent[] = []; + try { + db.run( + 'INSERT INTO nodes(id,name,last_seen_at) VALUES(?,?,?)', + 'registered', + 'registered', + null, + ); + db.run( + 'INSERT INTO nodes(id,name,last_seen_at) VALUES(?,?,?)', + 'returning', + 'returning', + new Date(time - 3600000).toISOString(), + ); + const monitor = createReliabilityMonitor( + db, + (e) => { + seen.push(e); + }, + () => time, + ); + monitor.tick(); + assert.equal( + seen.length, + 0, + 'startup grace suppresses old observations and registration is not a stale report', + ); + time += 30_000; + db.run('UPDATE nodes SET last_seen_at=? WHERE id=?', new Date(time).toISOString(), 'returning'); + monitor.tick(); + time += 31_000; + monitor.tick(); + assert.equal( + seen.length, + 0, + 'normal reconnect within grace produces no stale/recovered notification pair', + ); + time += 31_000; + monitor.tick(); + monitor.tick(); + assert.deepEqual( + seen.map((e) => [e.type, e.node_id]), + [['node.stale', 'returning']], + 'only a previously seen node that remains silent after grace becomes stale', + ); + db.run('UPDATE nodes SET last_seen_at=? WHERE id=?', new Date(time).toISOString(), 'returning'); + monitor.tick(); + assert.equal(seen[1].type, 'node.recovered', 'a genuinely stale node still reports recovery'); + } finally { + db.close(); + await fs.rm(dir, { recursive: true, force: true }); + } +}); diff --git a/src/control/reliability-monitor.ts b/src/control/reliability-monitor.ts new file mode 100644 index 0000000..829dfd9 --- /dev/null +++ b/src/control/reliability-monitor.ts @@ -0,0 +1,98 @@ +import type { Db } from '../db/db.ts'; +import type { NotificationEvent } from './notifications.ts'; + +/** Observes committed state without running network work or joining the control writer. */ +export function createReliabilityMonitor( + db: Db, + emit: (event: NotificationEvent) => void, + clock = Date.now, +) { + let cursor = db.get<{ seq: number }>('SELECT MAX(rowid) AS seq FROM events')?.seq ?? 0; + let deployments = new Map( + db + .all<{ id: string; state: string }>( + 'SELECT id,state FROM deployments ORDER BY created_at DESC LIMIT 1000', + ) + .map((row) => [row.id, row.state]), + ); + const nodes = new Map(); + const ingressStates = new Map(); + const staleMs = Number(process.env.TUG_STALE_SECONDS || 60) * 1000; + const graceUntil = clock() + Math.max(60_000, staleMs); + const event = ( + id: string, + type: string, + project: string | null, + node: string | null = null, + ): NotificationEvent => ({ + id, + type, + project_id: project, + node_id: node, + message: '', + metadata_json: null, + }); + return { + tick() { + const latest = db.get<{ seq: number }>('SELECT MAX(rowid) AS seq FROM events')?.seq ?? 0; + if (latest < cursor) cursor = 0; // Retention may empty the table and SQLite can reuse rowids. + for (const row of db.all( + 'SELECT rowid AS seq,* FROM events WHERE rowid>? ORDER BY rowid LIMIT 100', + cursor, + )) { + emit(row); + cursor = row.seq; + } + const current = db.all<{ id: string; project_id: string; state: string }>( + 'SELECT id,project_id,state FROM deployments ORDER BY created_at DESC LIMIT 1000', + ); + for (const row of current) { + if ( + deployments.get(row.id) !== row.state && + ['succeeded', 'failed', 'build_failed', 'cancelled'].includes(row.state) + ) + emit( + event(`deployment:${row.id}:${row.state}`, `deployment.${row.state}`, row.project_id), + ); + } + deployments = new Map(current.map((row) => [row.id, row.state])); + const live = new Set(); + for (const row of db.all<{ + id: string; + last_seen_at: string | null; + ingress_status_json: string | null; + }>('SELECT id,last_seen_at,ingress_status_json FROM nodes WHERE disabled_at IS NULL')) { + live.add(row.id); + const stale = + Boolean(row.last_seen_at) && + clock() >= graceUntil && + clock() - Date.parse(row.last_seen_at!) > staleMs; + const before = nodes.get(row.id); + if ((before === undefined && stale) || (before !== undefined && before !== stale)) + emit( + event( + `node:${row.id}:${stale ? 'stale' : 'recovered'}:${clock()}`, + stale ? 'node.stale' : 'node.recovered', + null, + row.id, + ), + ); + nodes.set(row.id, stale); + let ingressState = ''; + try { + ingressState = JSON.parse(row.ingress_status_json ?? '{}').state ?? ''; + } catch { + /* malformed state supplies no healthy evidence */ + } + if (ingressState === 'failed' && ingressStates.get(row.id) !== 'failed') + emit(event('ingress:' + row.id + ':' + clock(), 'ingress.failed', null, row.id)); + ingressStates.set(row.id, ingressState); + } + for (const id of nodes.keys()) + if (!live.has(id)) { + nodes.delete(id); + ingressStates.delete(id); + } + }, + }; +} diff --git a/src/control/restore-activation.test.ts b/src/control/restore-activation.test.ts index e5794db..1b78ea0 100644 --- a/src/control/restore-activation.test.ts +++ b/src/control/restore-activation.test.ts @@ -94,6 +94,41 @@ test('a failure at any step puts the whole previous generation back before the e } }); +test('an ENOSPC error while replacing the database restores the complete previous generation', async (t) => { + const { root, dataDir, restoreDir } = await setup(); + try { + const copyFile = fs.copyFile.bind(fs); + t.mock.method(fs, 'copyFile', async (...args: Parameters) => { + if (args[0] === path.join(restoreDir, 'tug.db')) { + throw Object.assign(new Error('no space left on device'), { code: 'ENOSPC' }); + } + return copyFile(...args); + }); + + await assert.rejects( + activateRestore({ dataDir, restoreDir }), + (error: NodeJS.ErrnoException) => error.code === 'ENOSPC', + 'the storage exhaustion error reaches the caller', + ); + assert.deepEqual( + [ + await read(dataDir, 'tug.db'), + await read(dataDir, 'master.key'), + await read(dataDir, 'pki/ca.crt'), + ], + ['db-old', 'key-old', 'ca-old'], + 'storage exhaustion leaves the previous generation complete', + ); + assert.deepEqual( + (await names(dataDir)).filter((name) => name.startsWith('restore-')), + [], + 'storage exhaustion leaves no journal or partial generation behind', + ); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } +}); + test('a start that finds an interrupted activation rolls it back before opening anything', async () => { const { root, dataDir } = await setup(); try { @@ -125,6 +160,62 @@ test('a start that finds an interrupted activation rolls it back before opening } }); +test('a restart rolls back a process crash after each activation boundary', async () => { + for (const boundary of ['database', 'master-key', 'pki'] as const) { + const { root, dataDir, restoreDir } = await setup(); + const previous = path.join(dataDir, `restore-previous-${boundary}`); + try { + await fs.mkdir(previous); + await fs.copyFile(path.join(dataDir, 'tug.db'), path.join(previous, 'tug.db')); + await fs.copyFile(path.join(dataDir, 'master.key'), path.join(previous, 'master.key')); + await fs.cp(path.join(dataDir, 'pki'), path.join(previous, 'pki'), { recursive: true }); + const journal = { + state: 'applying', + previous, + startedAt: new Date().toISOString(), + }; + await fs.writeFile(path.join(dataDir, 'restore-activation.json'), JSON.stringify(journal)); + + // Model the on-disk state left when the process exits immediately after this durable step. + await fs.copyFile(path.join(restoreDir, 'tug.db'), path.join(dataDir, 'tug.db')); + if (boundary !== 'database') + await fs.copyFile(path.join(restoreDir, 'master.key'), path.join(dataDir, 'master.key')); + if (boundary === 'pki') { + await fs.rm(path.join(dataDir, 'pki'), { recursive: true, force: true }); + await fs.cp(path.join(restoreDir, 'pki'), path.join(dataDir, 'pki'), { recursive: true }); + } + await fs.writeFile(path.join(dataDir, 'tug.db-wal'), 'partial wal'); + await fs.writeFile(path.join(dataDir, 'tug.db-shm'), 'partial shm'); + + assert.equal(await recoverActivation(dataDir), null, `${boundary} crash is rolled back`); + assert.deepEqual( + [ + await read(dataDir, 'tug.db'), + await read(dataDir, 'master.key'), + await read(dataDir, 'pki/ca.crt'), + ], + ['db-old', 'key-old', 'ca-old'], + `a crash after ${boundary} restores one complete previous generation`, + ); + assert.deepEqual( + (await names(dataDir)).filter((name) => name.startsWith('restore-')), + [], + `a crash after ${boundary} leaves no activation journal or previous generation`, + ); + await assert.rejects( + fs.stat(path.join(dataDir, 'tug.db-wal')), + `${boundary} rollback removes stale WAL`, + ); + await assert.rejects( + fs.stat(path.join(dataDir, 'tug.db-shm')), + `${boundary} rollback removes stale SHM`, + ); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } + } +}); + test('an interrupted activation that cannot be rolled back stops the start instead of guessing', async () => { const { root, dataDir } = await setup(); try { diff --git a/src/control/writer-queue.test.ts b/src/control/writer-queue.test.ts new file mode 100644 index 0000000..2400cbe --- /dev/null +++ b/src/control/writer-queue.test.ts @@ -0,0 +1,69 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { createWriterQueue, WRITER_OVERDUE_AFTER_MS } from './writer-queue.ts'; + +test('writer queue marks work overdue only after the deadline and clears it after recovery', async () => { + let time = 1_000; + const queue = createWriterQueue(() => time); + let finish!: () => void; + const operation = queue.run(() => new Promise((resolve) => (finish = resolve))); + + await new Promise((resolve) => setImmediate(resolve)); + assert.equal(queue.activeForMs(), 0, 'active work starts with zero elapsed time'); + time += WRITER_OVERDUE_AFTER_MS; + assert.equal( + queue.activeForMs(), + WRITER_OVERDUE_AFTER_MS, + 'work at the deadline remains within its grace period', + ); + time += 1; + assert.equal( + queue.activeForMs(), + WRITER_OVERDUE_AFTER_MS + 1, + 'overdue work remains observable without waiting for the queue', + ); + + finish(); + await operation; + assert.equal(queue.activeForMs(), null, 'a completed writer clears the overdue signal'); +}); + +test('a failed writer releases the queue and clears its active age', async () => { + const queue = createWriterQueue(() => 1_000); + await assert.rejects( + queue.run(() => { + throw new Error('writer failed'); + }), + /writer failed/, + 'writer failures should reach their caller', + ); + assert.equal(queue.activeForMs(), null, 'failed work must clear its active age'); + assert.equal( + await queue.run(() => 'recovered'), + 'recovered', + 'later work should proceed after failure', + ); +}); + +test('writer admission bounds waiting work and preserves accepted work order', async () => { + const queue = createWriterQueue(() => 1000, 1); + let release!: () => void; + const first = queue.run( + () => + new Promise((resolve) => { + release = resolve; + }), + ); + await new Promise((resolve) => setImmediate(resolve)); + const second = queue.run(() => 'second'); + assert.equal(queue.pending, 1, 'one waiting operation is admitted'); + await assert.rejects( + queue.run(() => 'overflow'), + { status: 503 }, + 'excess waiting work is rejected before it runs', + ); + release(); + await first; + assert.equal(await second, 'second', 'accepted work survives overload rejection'); + assert.equal(queue.pending, 0, 'the queue drains after accepted work completes'); +}); diff --git a/src/control/writer-queue.ts b/src/control/writer-queue.ts new file mode 100644 index 0000000..9fe3405 --- /dev/null +++ b/src/control/writer-queue.ts @@ -0,0 +1,38 @@ +export const WRITER_OVERDUE_AFTER_MS = 35 * 60 * 1000; + +export function createWriterQueue(now: () => number = Date.now, maxPending = 100) { + let tail = Promise.resolve(); + let activeSince: number | null = null; + let pending = 0; + + return { + async run(operation: () => T | Promise): Promise { + if (pending >= maxPending) + throw Object.assign(new Error('control writer queue is full; retry later'), { + status: 503, + retryAfter: 5, + }); + pending++; + const previous = tail; + let release!: () => void; + tail = new Promise((resolve) => { + release = resolve; + }); + await previous; + pending--; + activeSince = now(); + try { + return await operation(); + } finally { + activeSince = null; + release(); + } + }, + get pending(): number { + return pending; + }, + activeForMs(): number | null { + return activeSince === null ? null : now() - activeSince; + }, + }; +} diff --git a/src/logger.test.ts b/src/logger.test.ts new file mode 100644 index 0000000..344aeb1 --- /dev/null +++ b/src/logger.test.ts @@ -0,0 +1,198 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { createLogger } from './logger.ts'; + +test('JSON logs include component, level and context while redacting sensitive keys recursively', () => { + const lines: string[] = []; + const logger = createLogger({ + component: 'agent', + format: 'json', + level: 'debug', + write: (line) => lines.push(line), + }); + logger.info('poll completed', { + node: 'node-a', + credential: { token: 'secret-token', nested: { password: 'secret-password' } }, + services: [{ name: 'web', authorization: 'Bearer abc' }], + }); + + const record = JSON.parse(lines[0]) as Record; + assert.equal(record.component, 'agent', 'JSON logs identify their component'); + assert.equal(record.level, 'info', 'JSON logs include the selected severity'); + assert.equal(record.message, 'poll completed', 'JSON logs retain the fixed event message'); + assert.equal(record.node, 'node-a', 'JSON logs retain safe correlation fields'); + assert.deepEqual(record.credential, '[REDACTED]', 'credential values are redacted at any depth'); + assert.deepEqual( + record.services, + [{ name: 'web', authorization: '[REDACTED]' }], + 'sensitive keys inside arrays are redacted recursively', + ); + assert.ok(typeof record.timestamp === 'string', 'JSON logs include an ISO timestamp'); +}); + +test('logger suppresses messages below the configured level and emits readable text by default', () => { + const lines: string[] = []; + const logger = createLogger({ + component: 'control', + level: 'warn', + write: (line) => lines.push(line), + }); + logger.info('startup complete'); + logger.warn('queue is busy', { depth: 3 }); + + assert.equal(lines.length, 1, 'messages below the configured threshold are suppressed'); + assert.match( + lines[0], + /^WARN control: queue is busy depth=3$/, + 'text logs include level, component, message and context', + ); +}); + +test('logger redacts error context instead of serializing arbitrary error details', () => { + const lines: string[] = []; + const logger = createLogger({ + component: 'agent', + format: 'json', + write: (line) => lines.push(line), + }); + logger.error('poll failed', { error: 'request included a one-time secret value' }); + + assert.equal( + (JSON.parse(lines[0]) as Record).error, + '[REDACTED]', + 'error details are redacted because they may contain credentials or request data', + ); +}); + +test('failure classifications retain safe diagnosis while excluding arbitrary exception details', async () => { + const { failureContext } = await import('./logger.ts'); + const classified = failureContext( + Object.assign(new Error('secret destination password'), { code: 'ENOSPC' }), + ); + assert.deepEqual( + classified, + { failureClass: 'Error', failureCode: 'ENOSPC' }, + 'disk exhaustion remains actionable without arbitrary text', + ); + const unknown = failureContext({ + name: 'private-class', + code: 'private-code', + message: 'secret', + status: 503, + }); + assert.deepEqual( + unknown, + { failureClass: 'Error', failureCode: 'UNKNOWN', httpStatus: 503 }, + 'unknown categories cannot leak private details', + ); +}); + +test('failure logs keep exception text hidden unless debug logging is enabled', () => { + const normalLines: string[] = []; + createLogger({ + component: 'agent', + format: 'json', + level: 'info', + write: (line) => normalLines.push(line), + }).failure( + 'poll failed', + Object.assign(new Error('ECONNREFUSED at https://host/path'), { code: 'ECONNREFUSED' }), + { node: 'node-a' }, + ); + + assert.equal(normalLines.length, 1, 'normal logging emits only the safe failure classification'); + const normal = JSON.parse(normalLines[0]) as Record; + assert.equal(normal.failureCode, 'ECONNREFUSED', 'normal logs retain a useful safe error code'); + assert.equal(normal.node, 'node-a', 'normal logs retain supplied safe context'); + assert.equal(normal.cause, undefined, 'normal logs do not include exception text'); + + const debugLines: string[] = []; + const previousLevel = process.env.TUG_LOG_LEVEL; + process.env.TUG_LOG_LEVEL = 'debug'; + try { + createLogger({ + component: 'agent', + format: 'json', + write: (line) => debugLines.push(line), + }).failure( + 'poll failed', + new Error('connect ECONNREFUSED at https://user:pass@example.test/path'), + ); + } finally { + if (previousLevel === undefined) delete process.env.TUG_LOG_LEVEL; + else process.env.TUG_LOG_LEVEL = previousLevel; + } + assert.equal( + debugLines.length, + 2, + 'debug logging adds one cause record after safe classification', + ); + const debug = JSON.parse(debugLines[1]) as Record; + assert.equal(debug.message, 'failure cause', 'debug cause has a fixed event message'); + assert.equal( + debug.cause, + 'connect ECONNREFUSED at [URL REDACTED]', + 'debug causes remove URL contents', + ); +}); + +test('debug failure messages redact common credentials, PEM, controls, and excessive text', () => { + const lines: string[] = []; + createLogger({ + component: 'agent', + format: 'json', + level: 'debug', + write: (line) => lines.push(line), + }).failure( + 'operation failed', + new Error( + 'authorization: Bearer abc cookie=choc-chip password="p a s s" client_secret=hidden\n' + + '-----BEGIN PRIVATE KEY-----\nprivate material\n-----END PRIVATE KEY-----\u0000' + + 'x'.repeat(600), + ), + {}, + 'warn', + ); + + const detail = JSON.parse(lines[1]) as Record; + const cause = String(detail.cause); + assert.doesNotMatch( + cause, + /Bearer abc|choc-chip|p a s s|hidden|private material/, + 'debug cause removes common credentials and PEM contents', + ); + assert.match(cause, /authorization=\[REDACTED\]/, 'authorization assignments are redacted'); + assert.match(cause, /cookie=\[REDACTED\]/, 'cookie assignments are redacted'); + assert.match(cause, /password=\[REDACTED\]/, 'password assignments are redacted'); + assert.match(cause, /client_secret=\[REDACTED\]/, 'client secret assignments are redacted'); + assert.match(cause, /\[PEM REDACTED\]/, 'PEM blocks are redacted'); + assert.ok( + [...cause].every((character) => { + const code = character.charCodeAt(0); + return code >= 32 && (code < 127 || code > 159); + }), + 'debug causes contain no control characters', + ); + assert.ok(cause.length <= 501, 'debug causes are bounded to 500 characters plus an ellipsis'); +}); + +test('debug causes redact entire bearer credentials and cookie headers', () => { + const output: string[] = []; + const log = createLogger({ + component: 'test', + format: 'json', + level: 'debug', + write: (line) => output.push(line), + }); + log.failure( + 'request failed', + new Error( + 'Authorization: Bearer very-private-value\nCookie: session=private-session; token=private-cookie', + ), + ); + assert.doesNotMatch( + output.join('\n'), + /very-private-value|private-session|private-cookie/, + 'known header formats must not expose trailing credential values', + ); +}); diff --git a/src/logger.ts b/src/logger.ts new file mode 100644 index 0000000..2756745 --- /dev/null +++ b/src/logger.ts @@ -0,0 +1,177 @@ +export type LogLevel = 'debug' | 'info' | 'warn' | 'error'; +export type LogContext = Record; + +const levelValue: Record = { debug: 10, info: 20, warn: 30, error: 40 }; +const sensitiveKey = + /(?:secret|token|password|passwd|credential|authorization|cookie|private.?key|api.?key|join.?key|error|\.key$)/i; +const validLevel = (value: string | undefined): LogLevel => + value && value in levelValue ? (value as LogLevel) : 'info'; + +function clean(value: unknown, depth = 0): unknown { + if (depth > 5) return '[truncated]'; + if (value === null || typeof value === 'boolean' || typeof value === 'number') return value; + if (typeof value === 'string') return value.length > 500 ? `${value.slice(0, 500)}…` : value; + if (Array.isArray(value)) return value.slice(0, 30).map((item) => clean(item, depth + 1)); + if (typeof value === 'object') { + return Object.fromEntries( + Object.entries(value as Record) + .slice(0, 50) + .map(([key, item]) => [ + key, + sensitiveKey.test(key) ? '[REDACTED]' : clean(item, depth + 1), + ]), + ); + } + return String(value); +} + +export interface LoggerOptions { + component: string; + format?: 'text' | 'json'; + level?: LogLevel; + write?: (line: string, level: LogLevel) => void; +} + +/** Remove common accidental credential disclosures from debug-only exception text. */ +function sanitizeFailureMessage(message: string): string { + const sanitized = message + .replace( + /-----BEGIN [^-]*PRIVATE KEY-----[\s\S]*?-----END [^-]*PRIVATE KEY-----/gi, + '[PEM REDACTED]', + ) + .replace( + /\b(?:authorization|proxy-authorization)\s*[:=]\s*(?:Bearer|Basic)\s+[^\s,;]+/gi, + 'authorization=[REDACTED]', + ) + .replace(/\b(?:cookie|set-cookie)\s*:[^\r\n]*/gi, 'cookie=[REDACTED]') + .replace(/\b(?:https?|ftp):\/\/[^\s"'<>]+/gi, '[URL REDACTED]') + .replace( + /\b(?:authorization|proxy-authorization|cookie|set-cookie|credentials?|password|passwd|secret|token|api[_-]?key|access[_-]?key|client[_-]?secret)\b\s*[:=]\s*(?:"[^"]*"|'[^']*'|[^\s,;]+)/gi, + (assignment) => `${assignment.slice(0, assignment.search(/[:=]/))}=[REDACTED]`, + ) + // eslint-disable-next-line no-control-regex + .replace(/[\u0000-\u001f\u007f-\u009f]+/g, ' ') + .replace(/\s+/g, ' ') + .trim(); + return sanitized.length > 500 ? `${sanitized.slice(0, 500)}…` : sanitized; +} + +/** Create a logger whose messages are fixed descriptions and whose context is sanitized before output. */ +export function createLogger(options: LoggerOptions) { + const format = options.format ?? (process.env.TUG_LOG_FORMAT === 'json' ? 'json' : 'text'); + const threshold = levelValue[options.level ?? validLevel(process.env.TUG_LOG_LEVEL)]; + const write = + options.write ?? + ((line: string, level: LogLevel) => + (level === 'warn' || level === 'error' ? process.stderr : process.stdout).write(`${line}\n`)); + + const log = (level: LogLevel, message: string, context: LogContext = {}): void => { + if (levelValue[level] < threshold) return; + const safeContext = clean(context) as LogContext; + const record = { + timestamp: new Date().toISOString(), + level, + component: options.component, + message, + ...Object.fromEntries( + Object.entries(safeContext).filter( + ([key]) => !['timestamp', 'level', 'component', 'message'].includes(key), + ), + ), + }; + if (format === 'json') write(JSON.stringify(record), level); + else { + const details = Object.entries(record) + .filter(([key]) => !['timestamp', 'level', 'component', 'message'].includes(key)) + .map( + ([key, value]) => + `${key}=${typeof value === 'string' ? JSON.stringify(value) : JSON.stringify(value)}`, + ) + .join(' '); + write( + `${level.toUpperCase()} ${options.component}: ${message}${details ? ` ${details}` : ''}`, + level, + ); + } + }; + + return { + debug: (message: string, context?: LogContext) => log('debug', message, context), + info: (message: string, context?: LogContext) => log('info', message, context), + warn: (message: string, context?: LogContext) => log('warn', message, context), + error: (message: string, context?: LogContext) => log('error', message, context), + /** + * Log safe classification at `level` and the sanitized cause only when debug logging is enabled. + * Set `TUG_LOG_LEVEL=debug` to opt in. Sanitization covers common patterns; it cannot detect every secret. + */ + failure: ( + message: string, + error: unknown, + context: LogContext = {}, + level: 'warn' | 'error' = 'error', + ) => { + log(level, message, { ...context, ...failureContext(error) }); + if (threshold <= levelValue.debug) { + const value = error && typeof error === 'object' ? (error as Record) : {}; + const detail = typeof value.message === 'string' ? value.message : String(error); + log('debug', 'failure cause', { + ...context, + failureEvent: message, + cause: sanitizeFailureMessage(detail), + }); + } + }, + }; +} + +export const logger = createLogger({ component: 'tug' }); + +/** Preserve actionable failure categories without serializing arbitrary exception messages. */ +export function failureContext(error: unknown): LogContext { + const value = error && typeof error === 'object' ? (error as Record) : {}; + const classes = new Set([ + 'Error', + 'TypeError', + 'RangeError', + 'SyntaxError', + 'AbortError', + 'TimeoutError', + 'DbError', + ]); + const codes = new Set([ + 'ENOSPC', + 'EACCES', + 'EPERM', + 'EIO', + 'ENOENT', + 'ECONNREFUSED', + 'ECONNRESET', + 'ETIMEDOUT', + 'ENOTFOUND', + 'EHOSTUNREACH', + 'ENETUNREACH', + 'EPIPE', + 'ERR_SQLITE_ERROR', + 'CERT_HAS_EXPIRED', + 'CERT_NOT_YET_VALID', + 'SELF_SIGNED_CERT_IN_CHAIN', + 'UNABLE_TO_VERIFY_LEAF_SIGNATURE', + 'ERR_TLS_CERT_ALTNAME_INVALID', + 'DEPTH_ZERO_SELF_SIGNED_CERT', + 'ABORT_ERR', + 'UND_ERR_CONNECT_TIMEOUT', + ]); + return { + failureClass: typeof value.name === 'string' && classes.has(value.name) ? value.name : 'Error', + failureCode: typeof value.code === 'string' && codes.has(value.code) ? value.code : 'UNKNOWN', + ...(typeof value.status === 'number' && + Number.isInteger(value.status) && + value.status >= 400 && + value.status < 600 + ? { httpStatus: value.status } + : {}), + ...(typeof value.errcode === 'number' && Number.isSafeInteger(value.errcode) + ? { sqliteCode: value.errcode } + : {}), + }; +} diff --git a/src/real-host-tools.test.ts b/src/real-host-tools.test.ts index 3f2c6ff..9fc40b5 100644 --- a/src/real-host-tools.test.ts +++ b/src/real-host-tools.test.ts @@ -374,3 +374,39 @@ test('versionFrom extracts the first capture group and marks other bodies as unm 'a body without a match should be labelled unmatched', ); }); + +test('integration inspection retries only the known transient replacement gap', async () => { + const { inspectPublishedServices } = await import( + new URL('../tools/scripts/integration-verify-lib.mjs', import.meta.url).href + ); + let attempts = 0; + const inspect = async () => { + if (++attempts === 1) + throw Object.assign(new Error('missing'), { + stderr: 'Service has no container on this node\n', + }); + return [{ Config: { Image: 'nginx@sha256:' + 'a'.repeat(64) } }]; + }; + assert.equal( + await inspectPublishedServices(['cache'], inspect), + false, + 'a replacement gap must retry within the existing convergence deadline', + ); + assert.equal( + await inspectPublishedServices(['cache'], inspect), + true, + 'a later inspection must still prove a pinned running container', + ); + await assert.rejects( + inspectPublishedServices(['cache'], async () => [{ Config: { Image: 'nginx:latest' } }]), + /digest pinned/, + 'an unpinned image must fail instead of being retried', + ); + await assert.rejects( + inspectPublishedServices(['cache'], async () => { + throw Object.assign(new Error('denied'), { stderr: 'forbidden' }); + }), + /denied/, + 'unrelated inspection failures must fail immediately', + ); +}); diff --git a/tests/integration/backup-job.test.ts b/tests/integration/backup-job.test.ts index e38b29d..85f459b 100644 --- a/tests/integration/backup-job.test.ts +++ b/tests/integration/backup-job.test.ts @@ -182,6 +182,64 @@ test( 'a wrong repository password fails and does not re-initialize', ); + // A damaged repository must fail closed; restoring its real config should make a later run recover. + const repositoryFile = (...args: string[]) => + docker('run', '--rm', '-v', `${dir}:/work`, 'alpine', ...args); + const snapshotsBeforeDamage = snapshots(repo).length; + const damage = repositoryFile( + 'sh', + '-c', + 'cp /work/repo/config /work/config.saved && printf %s "damaged repository config" > /work/repo/config', + ); + assert.equal( + damage.status, + 0, + 'the disposable root helper can save and damage the root-owned repository config: ' + + damage.stderr, + ); + try { + await assert.rejects( + runBackupJob(base(repo, { command: service }), 'cmd_damaged', { + docker: engine, + credentials: credentials(), + }), + /repository check failed/, + 'a damaged repository is reported instead of initialized as a replacement', + ); + const damaged = repositoryFile('cat', '/work/repo/config'); + assert.equal( + damaged.status, + 0, + 'the root helper can read the damaged config: ' + damaged.stderr, + ); + assert.equal( + damaged.stdout, + 'damaged repository config', + 'a failed check does not replace the damaged repository with a new one', + ); + } finally { + const restored = repositoryFile('mv', '/work/config.saved', '/work/repo/config'); + assert.equal( + restored.status, + 0, + 'the saved repository config is restored with its original permissions: ' + + restored.stderr, + ); + } + const recovered = await runBackupJob(base(repo, { command: service }), 'cmd_recovered', { + docker: engine, + credentials: credentials(), + }); + assert.ok( + recovered.snapshotId, + 'restoring the repository config allows a later backup to succeed', + ); + assert.equal( + snapshots(repo).length, + snapshotsBeforeDamage + 1, + 'the recovered repository retains its history and adds the new snapshot', + ); + docker( 'volume', 'create', diff --git a/tools/scripts/check-reachability.mjs b/tools/scripts/check-reachability.mjs new file mode 100755 index 0000000..916ebdc --- /dev/null +++ b/tools/scripts/check-reachability.mjs @@ -0,0 +1,114 @@ +#!/usr/bin/env node + +function displayUrl(value) { + try { + const url = new URL(value); + url.username = ''; + url.password = ''; + url.search = ''; + url.hash = ''; + return url.toString(); + } catch { + return ''; + } +} + +export async function checkReachability(urls, { fetchImpl = fetch, timeoutMs = 5_000 } = {}) { + const maxRedirects = 5; + return Promise.all( + urls.map(async (rawUrl) => { + let url; + try { + url = new URL(rawUrl); + if (!['http:', 'https:'].includes(url.protocol)) throw new Error('unsupported protocol'); + if (url.username || url.password) throw new Error('credentials are not allowed'); + } catch { + return { url: displayUrl(rawUrl), ok: false, detail: 'invalid HTTP URL' }; + } + + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), timeoutMs); + try { + let redirects = 0; + for (;;) { + const response = await fetchImpl(url, { + method: 'GET', + redirect: 'manual', + signal: controller.signal, + }); + const location = response.headers.get('location'); + const isRedirect = [301, 302, 303, 307, 308].includes(response.status); + if (!isRedirect || !location) { + await response.body?.cancel(); + return { + url: displayUrl(rawUrl), + ok: response.status >= 200 && response.status < 300, + detail: `HTTP ${response.status}`, + }; + } + + await response.body?.cancel(); + if (redirects >= maxRedirects) + return { url: displayUrl(rawUrl), ok: false, detail: 'too many redirects' }; + let next; + try { + next = new URL(location, url); + if (!['http:', 'https:'].includes(next.protocol) || next.username || next.password) + throw new Error('invalid redirect'); + } catch { + return { url: displayUrl(rawUrl), ok: false, detail: 'invalid redirect' }; + } + if (url.protocol === 'https:' && next.protocol === 'http:') + return { url: displayUrl(rawUrl), ok: false, detail: 'HTTPS downgrade redirect' }; + const sameDefaultHttpHost = + !next.port && + !url.port && + ['http:', 'https:'].includes(next.protocol) && + ['http:', 'https:'].includes(url.protocol); + const sameExplicitPort = next.port === url.port; + if (next.hostname !== url.hostname || (!sameDefaultHttpHost && !sameExplicitPort)) + return { url: displayUrl(rawUrl), ok: false, detail: 'redirect to another host' }; + url = next; + redirects++; + } + } catch (error) { + return { + url: displayUrl(rawUrl), + ok: false, + detail: error?.name === 'AbortError' ? `timeout after ${timeoutMs}ms` : 'request failed', + }; + } finally { + clearTimeout(timeout); + } + }), + ); +} + +function configuredUrls(env) { + const positional = process.argv.slice(2); + const urls = positional.length ? positional : (env.TUG_PROBE_URLS ?? '').split(/[\n,]/); + return urls.map((value) => value.trim()).filter(Boolean); +} + +async function main() { + const urls = configuredUrls(process.env); + if (!urls.length) { + console.error('Usage: node tools/scripts/check-reachability.mjs [url ...]'); + console.error('Or set TUG_PROBE_URLS to comma or newline separated URLs.'); + process.exitCode = 2; + return; + } + const parsedTimeout = Number(process.env.TUG_PROBE_TIMEOUT_MS ?? 5_000); + const timeoutMs = Number.isFinite(parsedTimeout) && parsedTimeout > 0 ? parsedTimeout : 5_000; + const results = await checkReachability(urls, { timeoutMs }); + for (const result of results) + console.log(`${result.ok ? 'OK' : 'FAIL'} ${result.url} ${result.detail}`); + if (results.some((result) => !result.ok)) process.exitCode = 1; +} + +if ( + process.argv[1] && + new URL(import.meta.url).pathname === new URL(`file://${process.argv[1]}`).pathname +) { + await main(); +} diff --git a/tools/scripts/check-reachability.test.mjs b/tools/scripts/check-reachability.test.mjs new file mode 100644 index 0000000..d1bc044 --- /dev/null +++ b/tools/scripts/check-reachability.test.mjs @@ -0,0 +1,190 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { checkReachability } from './check-reachability.mjs'; + +test('probe reports HTTP success and failure when a redirect has no Location header', async () => { + const calls = []; + const results = await checkReachability( + ['https://service.test/health?token=hidden', 'https://service.test/redirect'], + { + fetchImpl: async (url, options) => { + calls.push({ url: String(url), options }); + return new Response(null, { status: url.pathname.endsWith('health') ? 204 : 302 }); + }, + }, + ); + assert.deepEqual( + results.map(({ ok, detail }) => ({ ok, detail })), + [ + { ok: true, detail: 'HTTP 204' }, + { ok: false, detail: 'HTTP 302' }, + ], + 'only successful 2xx responses pass when no redirect target is supplied', + ); + assert.equal(calls[0].options.redirect, 'manual', 'probe handles redirect policy itself'); + assert.doesNotMatch(results[0].url, /hidden/, 'printed URL omits query strings'); +}); + +test('probe follows bounded same-host redirects, including HTTP to HTTPS', async () => { + const calls = []; + const bodiesCancelled = []; + const results = await checkReachability(['http://service.test/start?secret=hidden'], { + fetchImpl: async (url, options) => { + calls.push({ url: String(url), options }); + const body = + calls.length < 3 + ? new ReadableStream({ + cancel() { + bodiesCancelled.push(calls.length); + }, + }) + : null; + return new Response(body, { + status: calls.length < 3 ? 302 : 204, + headers: + calls.length === 1 + ? { location: '/secure?secret=hidden' } + : calls.length === 2 + ? { location: 'https://service.test/ready' } + : undefined, + }); + }, + }); + assert.equal(results[0].ok, true, 'a same-host redirect chain ending in 2xx is reachable'); + assert.deepEqual( + calls.map(({ url }) => url), + [ + 'http://service.test/start?secret=hidden', + 'http://service.test/secure?secret=hidden', + 'https://service.test/ready', + ], + 'relative and HTTP-to-HTTPS same-host redirects are followed', + ); + assert.ok( + calls.every(({ options }) => options.redirect === 'manual'), + 'each redirect response is handled explicitly', + ); + assert.ok( + calls.every(({ options }) => options.signal === calls[0].options.signal), + 'the redirect chain shares one timeout signal', + ); + assert.deepEqual( + bodiesCancelled, + [1, 2], + 'each redirect response body is canceled before continuing', + ); + assert.doesNotMatch(results[0].url, /hidden/, 'reported URL omits query strings'); +}); + +test('probe rejects cross-host redirects without requesting or printing the target', async () => { + const calls = []; + const results = await checkReachability(['https://service.test/health?token=hidden'], { + fetchImpl: async (url) => { + calls.push(String(url)); + return new Response(null, { + status: 302, + headers: { location: 'https://outside.test/collect?token=hidden' }, + }); + }, + }); + assert.deepEqual( + calls, + ['https://service.test/health?token=hidden'], + 'external host is not fetched', + ); + assert.deepEqual( + results[0], + { + url: 'https://service.test/health', + ok: false, + detail: 'redirect to another host', + }, + 'cross-host redirect failure does not expose either query string', + ); +}); + +test('probe rejects a redirect to a different port on the same hostname', async () => { + const calls = []; + const results = await checkReachability(['https://service.test/health'], { + fetchImpl: async (url) => { + calls.push(String(url)); + return new Response(null, { + status: 302, + headers: { location: 'https://service.test:8443/health' }, + }); + }, + }); + assert.equal(calls.length, 1, 'probe does not request a different port'); + assert.equal(results[0].detail, 'redirect to another host', 'port change is rejected safely'); +}); + +test('probe stops redirect loops after five hops', async () => { + let calls = 0; + const results = await checkReachability(['https://service.test/loop'], { + fetchImpl: async () => { + calls++; + return new Response(null, { + status: 302, + headers: { location: '/loop' }, + }); + }, + }); + assert.equal(calls, 6, 'probe makes the initial request plus at most five redirect hops'); + assert.equal(results[0].ok, false, 'a redirect loop is not considered reachable'); + assert.equal(results[0].detail, 'too many redirects', 'redirect limit has a safe detail'); +}); + +test('probe rejects embedded credentials before making a request', async () => { + let calls = 0; + const results = await checkReachability(['https://user:secret@service.test/health'], { + fetchImpl: async () => { + calls++; + return new Response(null, { status: 204 }); + }, + }); + assert.equal(calls, 0, 'credentialed URL is not requested'); + assert.deepEqual( + results[0], + { url: 'https://service.test/health', ok: false, detail: 'invalid HTTP URL' }, + 'credential rejection does not reveal embedded credentials', + ); +}); + +test('probe bounds time and redacts request failures', async () => { + const results = await checkReachability(['https://service.test/health'], { + timeoutMs: 4, + fetchImpl: async (_url, options) => { + assert.ok(options.signal instanceof AbortSignal, 'timeout signal is supplied'); + return new Promise((_resolve, reject) => + options.signal.addEventListener('abort', () => + reject(new DOMException('aborted', 'AbortError')), + ), + ); + }, + }); + assert.deepEqual( + results[0], + { + url: 'https://service.test/health', + ok: false, + detail: 'timeout after 4ms', + }, + 'probe returns bounded status without leaking error details', + ); +}); + +test('probe rejects HTTPS downgrade redirects', async () => { + let calls = 0; + const [result] = await checkReachability(['https://site.test/health'], { + fetchImpl: async () => { + calls++; + return new Response(null, { status: 302, headers: { location: 'http://site.test/health' } }); + }, + }); + assert.equal( + result.ok, + false, + 'a successful TLS connection cannot authorize an insecure redirect', + ); + assert.equal(calls, 1, 'the downgraded URL is never requested'); +}); diff --git a/tools/scripts/integration-verify-lib.mjs b/tools/scripts/integration-verify-lib.mjs new file mode 100644 index 0000000..0881bec --- /dev/null +++ b/tools/scripts/integration-verify-lib.mjs @@ -0,0 +1,22 @@ +import assert from 'node:assert/strict'; + +/** Discovery may replace a client after its first serving report; retry only that known gap. */ +export async function inspectPublishedServices(services, inspect) { + for (const service of services) { + let containers; + try { + containers = await inspect(service); + } catch (error) { + if (error?.stderr?.trim() === 'Service has no container on this node') return false; + throw error; + } + assert.ok(containers.length > 0, 'each published service must have a running container'); + for (const container of containers) + assert.match( + container.Config.Image, + /@sha256:[a-f0-9]{64}$/, + 'running containers must use the digest pinned by control', + ); + } + return true; +} diff --git a/tools/scripts/integration-verify.js b/tools/scripts/integration-verify.js index 57441c2..65313b7 100755 --- a/tools/scripts/integration-verify.js +++ b/tools/scripts/integration-verify.js @@ -1,5 +1,5 @@ #!/usr/bin/env node -import assert from 'node:assert/strict'; +import { inspectPublishedServices } from './integration-verify-lib.mjs'; import { execFile } from 'node:child_process'; import { promisify } from 'node:util'; import { loadCredential, mtlsFetch } from './mtls-fetch.js'; @@ -37,22 +37,22 @@ while (Date.now() - started < timeoutMs) { .map((s) => s.node), ); if (deployment && nodes.every((n) => seenNodes.has(n))) { - for (const service of ['web', 'worker', 'cache']) { - const { stdout } = await exec(process.execPath, [ - 'src/cli.ts', - 'inspect', - service, - '--project', - project, - ]); - const containers = JSON.parse(stdout); - assert.ok(containers.length > 0, 'each published service must have a running container'); - for (const container of containers) - assert.match( - container.Config.Image, - /@sha256:[a-f0-9]{64}$/, - 'running containers must use the digest pinned by control', - ); + const inspected = await inspectPublishedServices( + ['web', 'worker', 'cache'], + async (service) => { + const { stdout } = await exec(process.execPath, [ + 'src/cli.ts', + 'inspect', + service, + '--project', + project, + ]); + return JSON.parse(stdout); + }, + ); + if (!inspected) { + await new Promise((resolve) => setTimeout(resolve, 2000)); + continue; } console.log( JSON.stringify({ ok: true, version: deployment.version, nodes: [...seenNodes] }, null, 2), diff --git a/tools/scripts/qemu-first-boot.js b/tools/scripts/qemu-first-boot.js index 95aff32..4cd26c2 100644 --- a/tools/scripts/qemu-first-boot.js +++ b/tools/scripts/qemu-first-boot.js @@ -2,7 +2,7 @@ // Prototype: prove the cloud-init enrollment path on a real first boot, locally, with QEMU. // // node tools/scripts/qemu-first-boot.js --image [--work ] [--name app1] [--keep] -// [--ssh-user ] [--agent-image ] [--block-docker-seconds N] [--timeout ] +// [--ssh-user ] [--agent-image ] [--block-docker-seconds N] [--verify-reliability] [--timeout ] // // It runs a throwaway control on this machine, mints user-data with `tug node enroll --install-docker`, boots a cloud // image with that user-data as a NoCloud seed (a real first boot, not a `cloud-init clean` replay) and watches until the @@ -349,6 +349,30 @@ try { timeoutSeconds, ); log('SUCCESS: first-boot cloud-init enrolled the node'); + if (flag('verify-reliability')) { + const { decodeCredential } = await import('../../src/pki.ts'); + const { createTransport } = await import('../../src/transport.ts'); + const client = createTransport( + decodeCredential(credential.stdout), + 'https://127.0.0.1:' + port, + ); + try { + const ready = await client.http(client.base + '/api/v1/readyz', { + signal: AbortSignal.timeout(5000), + }); + if (ready.status !== 200 || !(await ready.json()).ready) + throw new Error('control readiness failed after guest enrollment'); + const metrics = await client.http(client.base + '/api/v1/metrics', { + signal: AbortSignal.timeout(5000), + }); + const body = await metrics.text(); + if (metrics.status !== 200 || !body.includes('tug_node_reported{node="' + nodeName + '"} 1')) + throw new Error('metrics did not observe guest report'); + log('SUCCESS: authenticated readiness and node-report metrics verified over mTLS'); + } finally { + await client.close(); + } + } } catch (error) { log(`FAILED: ${error.message}`); console.log('--- serial console (last 40 lines) ---');