diff --git a/.env.example b/.env.example index 1692918..d336f71 100644 --- a/.env.example +++ b/.env.example @@ -4,13 +4,14 @@ FIBER_DATABASE_URL=postgres://fiber:fiber@127.0.0.1:15432/fiber FIBER_REDIS_URL=redis://:fiber@127.0.0.1:16379 # password = FIBER_REDIS_PASSWORD; `make infra` uses `fiber` on a dev box -FIBER_LISTEN=0.0.0.0:18080 +FIBER_LISTEN=127.0.0.1:18080 # loopback; set 0.0.0.0:18080 to serve other machines (needs a non-default admin password) FIBER_ARTIFACTS_DIR=./data/artifacts FIBER_ADMIN_USER=admin FIBER_ADMIN_PASSWORD=fiber +# FIBER_ALLOW_DEFAULT_ADMIN=1 # start on a non-loopback FIBER_LISTEN even while admin/fiber still works # FIBER_MAX_STEPS=500 # most expanded steps (matrix cells included) one pipeline may compile to # FIBER_SEED_SHOWCASE=auto # auto = seed the demo project only on a fresh instance; 1 = this boot; 0 = never -# FIBER_SECRETS_KEY= # openssl rand -hex 32 +# FIBER_SECRETS_KEY= # openssl rand -hex 32; set but not 64 hex chars = fiber-api refuses to start # FIBER_CORS_ORIGINS=http://localhost:3100,http://127.0.0.1:3100 # browser origins; * = any # FIBER_STEP_TIMEOUT_DEFAULT_MINUTES=60 # steps without timeout_minutes # FIBER_STEP_TIMEOUT_GRACE_MINUTES=5 # server fails a step this long after its timeout diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d1da45..a131348 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ minor versions may carry breaking changes. ## [Unreleased] +**Three things to do before upgrading** (details under *Changed*): put a +`FIBER_SECRETS_KEY` in `deploy/.env` — Compose now requires it; change the admin password +if it is still `fiber` — the API no longer starts on a non-loopback address with it; and +set `FIBER_LISTEN` for a host-run API that other machines connect to — the default is now +loopback. + ### Security - **Seven RustSec advisories in the lockfile, all fixed by upgrades.** `rustls` 0.23.46 @@ -13,9 +19,121 @@ minor versions may carry breaking changes. verification could accept a distrusted intermediate), `tokio` 1.54.1 (RUSTSEC-2026-0386) and `ctutils` 0.4.8 (RUSTSEC-2026-0385). TLS is on every path that carries an agent token or a session, so the 0.6.7 agent, CLI and images are affected; upgrade all three. +- **The login lockout could be outrun by sending the guesses in parallel.** The handler + checked the lockout, verified the password, and only then recorded the failure, so every + request in a burst passed the check before any of them counted. The guess rate against a + known username was bounded by the server's CPU, not by "10 failures, then 60 s". An + attempt is now reserved before the verify and settled after it, and attempts still in + flight count against the same ten; a request that errors or is dropped gives its slot + back. Three things next to it: + - argon2id ran inline on the async workers, so the same burst stalled agent heartbeats + and lease renewals. Hashing and verifying now run on the blocking pool, at most four + at a time across the whole process (login, password change, user creation). A slot + is held until the hash finishes, not until the request that asked for it goes away, + so hanging up mid-login does not free it for more. A request that finds all four + busy waits its turn for up to 5 s and then gets a `503` that says nothing was + checked; it is not a lockout and does not count as a failed login. + - An unknown username returned before any hashing, a wrong password after it, so + response time said which usernames exist. Both now cost one argon2id verify, and so + does an account still on a legacy SHA-256 hash, which is also now compared in + constant time. + - Once the guard held 10 000 usernames it dropped every entry that was not locked out, + so a flood of made-up usernames reset a real account's failure count on demand. It + now evicts expired entries, then the fewest failures first. + + The `429` body's `error` text changed from "too many failed logins" to "too many login + attempts", since it now also covers attempts refused while others for the same + username are being verified. + [authz](docs/authz.md#authentication). +- **An agent token could claim any labels and any concurrency.** `Hello` carries the + agent's labels and concurrency, and the server registered the agent with them as sent — + it reloaded the agent's row for pool scope and ignored the labels and concurrency stored + beside it. Anyone holding one agent token could reconnect claiming every label and + `u32::MAX` slots, and be offered steps, with their secrets, meant for other agents in + the same pool. The row is now the ceiling: an agent is registered with the row's labels + that it also claims, and the smaller of the two concurrencies — at connect, and again + whenever the row is edited while the agent is connected. A row with no labels grants + none. Asking for more is logged with the agent id. See *Changed* for what this does to + an existing agent. + [agents](docs/agents.md#labels-and-concurrency-the-registration-is-the-ceiling). +- **The instance's GitHub token could be aimed at any repository by a project admin.** + Listing a pull request's files (for `paths` filters) fell back from the project's + `GITHUB_TOKEN` secret to `FIBER_GITHUB_TOKEN` / `GITHUB_TOKEN` in `fiber-api`'s + environment, and the repository came from the webhook payload. The payload's signature + proves only that the sender holds the project's webhook secret, which a project admin + sets — and any user can create a project. So a user could sign a `pull_request` payload + naming a private repository and read from the response (`files_source`, + `changed_files`, which path-filtered pipelines started) what the instance's token can + see there. The fallback is gone: both GitHub calls now use the project's secret only, + as commit statuses already did. Checking the payload's repository against the + project's pipelines would not have closed it, since the same user writes the pipeline. + See *Changed*. [triggers](docs/triggers.md#pull-request). +- **A `FIBER_SECRETS_KEY` that was set but invalid meant plaintext secrets, silently.** A + key that was not hex, or not 64 characters — a truncated paste, a passphrase, a value + with quotes round it — was treated as no key: the boot log carried the ordinary "unset + (dev only)" line and every secret written afterwards was stored in plaintext, for an + operator who had configured encryption. It is now a boot error that says what is wrong + with the value (never the value), and encrypting or decrypting with such a key fails + rather than passing plaintext through. Unset is still allowed, and still warned about. +- **A bare `fiber-api` listened on every interface with `admin` / `fiber`.** The default + `FIBER_LISTEN` was `0.0.0.0:18080`, while `SECURITY.md` said the defaults were all + loopback. It is now `127.0.0.1:18080`, and on any other address the API refuses to + start while the bootstrap admin can still log in with `fiber`. The check asks the + database, not the environment, so an instance whose password was changed in the UI is + not refused, and one where only the variable was changed (it applies to an empty + database only) is. It checks `FIBER_ADMIN_USER` and the literal `admin`. ### Changed +- **Compose: set `FIBER_SECRETS_KEY` in `deploy/.env` before upgrading.** + `docker compose` now refuses to read the file without it, as it does for the four + credentials. Generate one with `openssl rand -hex 32` and keep a backup. If you ran + without a key until now, existing secrets stay readable — a stored value without the + `enc:v1:` prefix is read as plaintext — and each is encrypted the next time it is + written. If your key was set but not 64 hex characters, your secrets are in plaintext + today and the API will not start until the key is valid. `make infra` and + `scripts/smoke_compose.sh` supply a throwaway key themselves. +- **Change the admin password if it is still `fiber`, or the API will not start in a + container.** Any `FIBER_LISTEN` other than loopback — which is every container, Compose + included — now refuses to boot while the bootstrap admin can log in with the default + password. `fiber-api reset-password admin` changes it without a running server + (`docker compose -f deploy/docker-compose.yml run --rm fiber-api fiber-api reset-password admin`). + `FIBER_ALLOW_DEFAULT_ADMIN=1` starts anyway; the Compose smoke uses it. +- **A host-run `fiber-api` listens on `127.0.0.1:18080` by default, not `0.0.0.0`.** If + agents on other machines, or a reverse proxy in a container, connect to it, set + `FIBER_LISTEN=0.0.0.0:18080` (and a real admin password, per the item above). Compose + is unaffected: it sets `FIBER_LISTEN` itself. The `fiber-api` image now sets + `FIBER_LISTEN=0.0.0.0:8080`, matching its `EXPOSE` and `HEALTHCHECK`; a bare + `docker run` of it previously listened on 18080 and reported unhealthy. +- **Check each agent's registered labels and concurrency against its flags before + upgrading.** Until now the agent's own `--labels` / `--concurrency` won at connect and + the values on its row (the UI's Agents page, `fiber agents list`) were only applied + after an edit. Now the row caps both, so an agent whose flags ask for more than its row + gets less than it did: + - *Concurrency.* `fiber agents create` and the UI default to **1**; `install-agent.sh` + and the Compose agent default to **2**. An agent registered and installed with those + defaults ran two steps at once and will now run one. Raise the row: + `fiber agents update --concurrency 2`. + - *Labels.* A label passed to the agent but missing from its row no longer matches + steps. Add it to the row: `fiber agents update --labels os=linux,docker=true`. + + - *No labels on the row.* An agent whose row has no labels at all now receives only + steps that require no labels, whatever its own flags say. Find such rows with + `SELECT id, name FROM agents WHERE labels = '[]'::jsonb;` and give each the labels it + should have. + + The API logs `agent hello asked for more than its registration allows` with the agent + id for every agent this affects, each time it connects. Editing a connected agent's row + no longer needs a reconnect to be applied correctly: narrowing takes effect at once, + widening on the agent's next heartbeat, and never beyond the agent's own flags. +- **If you rely on `FIBER_GITHUB_TOKEN` in `fiber-api`'s environment, add a project secret + before upgrading.** It is no longer read. A project with `pull_request` `paths` / + `paths_ignore` filters and no `GITHUB_TOKEN` secret of its own will stop starting those + pipelines on pull requests (they skip, as they always have without a token). Add the + token to each such project: `fiber secrets set GITHUB_TOKEN --value-stdin`. + `fiber-api` logs one warning the first time a project needed the environment's token. + The docs also said the environment variable was a fallback for commit statuses; it has + not been since statuses moved to the project's secret, and they now say so. - **The advisory gate can no longer go red unseen.** `deny` only ran on a push or pull request, was not a required check, and was not run on a release tag — so with no pushes to `main` since early October, the advisories above went unreported. A daily diff --git a/Makefile b/Makefile index 642f6ec..3957e9f 100644 --- a/Makefile +++ b/Makefile @@ -2,16 +2,19 @@ smoke smoke-authz smoke-pools smoke-artifacts smoke-concurrency smoke-runs smoke-s3 smoke-compose \ login validate ready -# deploy/docker-compose.yml has no working default for the four credentials — a -# deployment must put them in deploy/.env, and `docker compose config` fails until it -# does. A dev box is not a deployment: these services publish on 127.0.0.1 only, and -# scripts/dev-env.sh already assumes these throwaway values, so the make targets supply -# them here rather than making every contributor write a deploy/.env. +# deploy/docker-compose.yml has no working default for the four credentials or the +# secrets key — a deployment must put them in deploy/.env, and `docker compose config` +# fails until it does. A dev box is not a deployment: these services publish on +# 127.0.0.1 only, and scripts/dev-env.sh already assumes these throwaway values, so the +# make targets supply them here rather than making every contributor write a deploy/.env. +# The secrets key is here only so Compose can interpolate the file: these targets start +# Postgres, Redis and the S3 store, never the containerised fiber-api that would use it. DEV_CREDS := FIBER_POSTGRES_PASSWORD=$${FIBER_POSTGRES_PASSWORD:-fiber} \ FIBER_REDIS_PASSWORD=$${FIBER_REDIS_PASSWORD:-fiber} \ FIBER_S3_ACCESS_KEY=$${FIBER_S3_ACCESS_KEY:-fiber} \ FIBER_S3_SECRET_KEY=$${FIBER_S3_SECRET_KEY:-fiberfiber} \ - FIBER_ADMIN_PASSWORD=$${FIBER_ADMIN_PASSWORD:-fiber} + FIBER_ADMIN_PASSWORD=$${FIBER_ADMIN_PASSWORD:-fiber} \ + FIBER_SECRETS_KEY=$${FIBER_SECRETS_KEY:-00000000000000000000000000000000000000000000000000000000000000ff} # ...but a deploy/.env, if there is one, is the operator's own and wins outright: # environment beats an env-file in Compose, so injecting these would silently override # it and Redis would come back up on a password their host API does not have. Anchored diff --git a/SECURITY.md b/SECURITY.md index a1aeeab..dd676d1 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -43,7 +43,8 @@ boundaries are. - **Steps run without a container unless the step sets `image:`.** A host-shell step runs as the agent's user. - **Defaults are for a local trial, not the internet.** `admin` / `fiber`, no secrets - encryption key, everything on loopback. [docs/operations.md](docs/operations.md#deployment) + encryption key, everything on loopback — `fiber-api` included, which also refuses to + start on any other address while `admin` / `fiber` still works. [docs/operations.md](docs/operations.md#deployment) describes what to change before exposing an instance. ## Hardening a deployment diff --git a/crates/fiber-api/src/github.rs b/crates/fiber-api/src/github.rs index c82f2e9..562106e 100644 --- a/crates/fiber-api/src/github.rs +++ b/crates/fiber-api/src/github.rs @@ -6,50 +6,86 @@ use tracing::{info, warn}; const DEFAULT_API: &str = "https://api.github.com"; -/// A token for **writing** a commit status: project secrets only. +/// Project secrets that may hold a GitHub token, in the order they are tried. +const PROJECT_TOKEN_SECRETS: [&str; 2] = ["GITHUB_TOKEN", "FIBER_GITHUB_TOKEN"]; +/// Environment variables that used to be an instance-wide fallback. Read now only to +/// tell the operator they are no longer used. +const INSTANCE_TOKEN_ENV: [&str; 2] = ["FIBER_GITHUB_TOKEN", "GITHUB_TOKEN"]; + +/// Which token a GitHub call made for a project uses. +#[derive(Debug, PartialEq, Eq)] +enum TokenChoice { + /// The project's own secret. + Project(String), + /// The project has none, so the call is not made. `instance_token_ignored` says the + /// process environment holds one that was deliberately not used. + None { instance_token_ignored: bool }, +} + +/// The project's own token, or nothing — never the instance's. /// -/// Deliberately not `resolve_token`, whose environment fallback belongs to whoever runs -/// `fiber-api`. Writing with that would let any project admin point a pipeline at a -/// repository they do not own and post statuses to it under the instance's identity. -pub async fn resolve_status_token( - store: &fiber_core::Store, - project_id: uuid::Uuid, -) -> Result> { - for key in ["GITHUB_TOKEN", "FIBER_GITHUB_TOKEN"] { - if let Some(v) = store.get_secret_plain(project_id, key).await? { - let v = v.trim().to_string(); - if !v.is_empty() { - return Ok(Some(v)); - } - } +/// The token is spent on a repository named by a webhook body (`repository.full_name`). +/// The signature on that body proves only that the sender holds the project's webhook +/// secret, and a project admin sets that secret themselves; any logged-in user can create +/// a project and be its admin. With an instance-wide fallback, such a user could sign a +/// `pull_request` payload naming a repository they cannot see and read from the response +/// (`files_source`, `changed_files`, which path-filtered pipelines started) what the +/// instance's token can. Requiring the payload's repository to match a pipeline's +/// `workspace.repo` would not help: the same user writes the pipeline. +/// +/// A project's secret is set by the same people who set its webhook secret, so spending +/// it on a repository they name gives them nothing they did not have. +fn select_token( + project_secrets: &[Option], + instance_env: &[Option], +) -> TokenChoice { + fn first(values: &[Option]) -> Option { + values + .iter() + .flatten() + .map(|v| v.trim()) + .find(|v| !v.is_empty()) + .map(str::to_string) + } + match first(project_secrets) { + Some(token) => TokenChoice::Project(token), + None => TokenChoice::None { + instance_token_ignored: first(instance_env).is_some(), + }, } - Ok(None) } -/// Resolve a token for read-only GitHub API calls (the PR file list). -/// Order: project secrets `GITHUB_TOKEN` / `FIBER_GITHUB_TOKEN`, then env -/// `FIBER_GITHUB_TOKEN` / `GITHUB_TOKEN`. -pub async fn resolve_token( +/// A token for GitHub API calls made on a project's behalf — the PR file list and commit +/// statuses: the project secret `GITHUB_TOKEN` / `FIBER_GITHUB_TOKEN`, and nothing else. +/// See [`select_token`] for why the process environment is not a fallback. +pub async fn resolve_project_token( store: &fiber_core::Store, project_id: uuid::Uuid, ) -> Result> { - for key in ["GITHUB_TOKEN", "FIBER_GITHUB_TOKEN"] { - if let Some(v) = store.get_secret_plain(project_id, key).await? { - let v = v.trim().to_string(); - if !v.is_empty() { - return Ok(Some(v)); - } - } + let mut project_secrets = Vec::with_capacity(PROJECT_TOKEN_SECRETS.len()); + for key in PROJECT_TOKEN_SECRETS { + project_secrets.push(store.get_secret_plain(project_id, key).await?); } - for key in ["FIBER_GITHUB_TOKEN", "GITHUB_TOKEN"] { - if let Ok(v) = std::env::var(key) { - let v = v.trim().to_string(); - if !v.is_empty() { - return Ok(Some(v)); + let instance_env = INSTANCE_TOKEN_ENV.map(|key| std::env::var(key).ok()); + match select_token(&project_secrets, &instance_env) { + TokenChoice::Project(token) => Ok(Some(token)), + TokenChoice::None { + instance_token_ignored, + } => { + if instance_token_ignored { + // Once: this is an upgrade note, not an event. + static WARNED: std::sync::Once = std::sync::Once::new(); + WARNED.call_once(|| { + warn!( + %project_id, + "FIBER_GITHUB_TOKEN / GITHUB_TOKEN in the environment is no longer used as a fallback; \ + set a project secret GITHUB_TOKEN for PR path filters and commit statuses" + ); + }); } + Ok(None) } } - Ok(None) } pub fn api_base() -> String { @@ -273,7 +309,7 @@ pub async fn report_run_status(store: &fiber_core::Store, run: &fiber_core::Run) warn!(%full_name, "refusing to post a status for an implausible repo or sha"); return; } - let token = match resolve_status_token(store, run.project_id).await { + let token = match resolve_project_token(store, run.project_id).await { Ok(Some(t)) => t, Ok(None) => return, Err(e) => { @@ -339,6 +375,47 @@ pub async fn report_run_status(store: &fiber_core::Store, run: &fiber_core::Run) mod tests { use super::*; + fn some(s: &str) -> Option { + Some(s.to_string()) + } + + /// The hole: a project with no token of its own borrowed the instance's, for a + /// repository its own webhook payload named. + #[test] + fn the_instance_token_is_never_chosen() { + assert_eq!( + select_token(&[None, None], &[some("ghp_instance"), None]), + TokenChoice::None { + instance_token_ignored: true + } + ); + assert_eq!( + select_token(&[some(" "), None], &[None, some("ghp_instance")]), + TokenChoice::None { + instance_token_ignored: true + }, + "a blank project secret is not a token, and still does not fall through" + ); + assert_eq!( + select_token(&[None, None], &[None, some(" ")]), + TokenChoice::None { + instance_token_ignored: false + } + ); + } + + #[test] + fn a_project_secret_is_used_trimmed_and_in_order() { + assert_eq!( + select_token(&[some(" ghp_a\n"), some("ghp_b")], &[some("ghp_instance")]), + TokenChoice::Project("ghp_a".into()) + ); + assert_eq!( + select_token(&[some(""), some("ghp_b")], &[]), + TokenChoice::Project("ghp_b".into()) + ); + } + #[test] fn repo_from_remote_handles_common_forms() { for url in [ diff --git a/crates/fiber-api/src/login_guard.rs b/crates/fiber-api/src/login_guard.rs index 3149f22..fdeef0c 100644 --- a/crates/fiber-api/src/login_guard.rs +++ b/crates/fiber-api/src/login_guard.rs @@ -6,19 +6,33 @@ //! attacker-controlled without extra configuration. State is per process; a //! multi-instance deployment gets `MAX_FAILURES` per instance, which is still a //! hard ceiling on brute force against argon2id. +//! +//! An attempt is **reserved before** the password is verified and settled afterwards: +//! [`LoginGuard::begin_attempt`] counts recorded failures plus attempts still in flight, +//! so a burst of parallel requests cannot all pass a read-only check and then each get a +//! guess. The reservation is an [`Attempt`]; dropping it without [`Attempt::finish`] +//! releases the slot, so an early return in the handler cannot hold one forever. use std::collections::HashMap; -use std::sync::Mutex; +use std::sync::{Arc, Mutex}; use std::time::{Duration, Instant}; -/// Failures allowed inside `WINDOW` before a lockout. +/// Failures allowed inside `WINDOW` before a lockout. Also the most attempts for one +/// username that may be unsettled at once, since failures and in-flight attempts share +/// the budget. pub const MAX_FAILURES: u32 = 10; /// Failure counting window. pub const WINDOW: Duration = Duration::from_secs(10 * 60); /// Lockout applied once `MAX_FAILURES` is reached. pub const LOCKOUT: Duration = Duration::from_secs(60); -/// Prune the table when it grows past this many usernames. -const PRUNE_ABOVE: usize = 10_000; +/// `Retry-After` for an attempt refused because the budget is held by attempts that have +/// not settled yet (rather than by a lockout, which knows its own remaining time). +pub const IN_FLIGHT_RETRY: Duration = Duration::from_secs(1); +/// The table never holds more than this many usernames. +const MAX_ENTRIES: usize = 10_000; +/// Eviction frees down to this many, so a flood pays for one sort per +/// `MAX_ENTRIES - EVICT_TO` new usernames rather than one per request. +const EVICT_TO: usize = MAX_ENTRIES - MAX_ENTRIES / 10; /// Longest username retained as a key. const MAX_KEY_CHARS: usize = 128; @@ -29,10 +43,62 @@ pub struct LoginGuard { struct Entry { failures: u32, + /// Attempts admitted by `begin_attempt` and not yet settled. + in_flight: u32, window_start: Instant, locked_until: Option, } +impl Entry { + fn locked_at(&self, now: Instant) -> bool { + self.locked_until.is_some_and(|t| t > now) + } +} + +enum Outcome { + Success, + Failure, + /// Dropped without a verdict (store error, cancelled request): releases the slot + /// and counts as neither. + Abandoned, +} + +/// One admitted login attempt. Settle it with [`Attempt::finish`]; dropping it instead +/// releases the slot without recording a failure or a success. +#[must_use = "an unsettled attempt is released on drop and records nothing"] +pub struct Attempt { + guard: Arc, + key: String, + open: bool, +} + +impl Attempt { + /// Release the slot and record the verdict; returns the lockout duration if this + /// failure triggered one. + pub fn finish(self, success: bool) -> Option { + self.finish_at(success, Instant::now()) + } + + fn finish_at(mut self, success: bool, now: Instant) -> Option { + self.open = false; + let outcome = if success { + Outcome::Success + } else { + Outcome::Failure + }; + self.guard.settle(&self.key, outcome, now) + } +} + +impl Drop for Attempt { + fn drop(&mut self) { + if self.open { + self.guard + .settle(&self.key, Outcome::Abandoned, Instant::now()); + } + } +} + impl LoginGuard { pub fn new() -> Self { Self::default() @@ -49,63 +115,99 @@ impl LoginGuard { .collect() } - /// `Err(retry_after)` when the username is currently locked out. - pub fn check(&self, key: &str) -> Result<(), Duration> { - self.check_at(key, Instant::now()) + /// Reserve one attempt for `key`. `Err(retry_after)` when the username is locked + /// out, or when its recorded failures plus the attempts still in flight already + /// fill the budget. + pub fn begin_attempt(self: &Arc, key: &str) -> Result { + self.begin_attempt_at(key, Instant::now()) } - /// Record a failed attempt; returns the lockout duration if this attempt triggered one. - pub fn record_failure(&self, key: &str) -> Option { - self.record_failure_at(key, Instant::now()) - } - - pub fn record_success(&self, key: &str) { - let mut map = self.inner.lock().unwrap_or_else(|e| e.into_inner()); - map.remove(key); - } - - fn check_at(&self, key: &str, now: Instant) -> Result<(), Duration> { + fn begin_attempt_at(self: &Arc, key: &str, now: Instant) -> Result { let mut map = self.inner.lock().unwrap_or_else(|e| e.into_inner()); - if map.len() > PRUNE_ABOVE { - map.retain(|_, e| { - e.locked_until.is_some_and(|t| t > now) || now - e.window_start < WINDOW - }); - // A flood of distinct usernames keeps every entry inside WINDOW; under that - // pressure keep only active lockouts — losing partial counts is the cheap side. - if map.len() > PRUNE_ABOVE { - map.retain(|_, e| e.locked_until.is_some_and(|t| t > now)); - } - } - let Some(entry) = map.get(key) else { - return Ok(()); - }; - match entry.locked_until { - Some(until) if until > now => Err(until - now), - _ => Ok(()), + if !map.contains_key(key) && map.len() >= MAX_ENTRIES { + evict(&mut map, now); } - } - - fn record_failure_at(&self, key: &str, now: Instant) -> Option { - let mut map = self.inner.lock().unwrap_or_else(|e| e.into_inner()); let entry = map.entry(key.to_string()).or_insert(Entry { failures: 0, + in_flight: 0, window_start: now, locked_until: None, }); + if let Some(until) = entry.locked_until + && until > now + { + return Err(until - now); + } // A lockout that has expired, or a stale window, starts a fresh count. - let lock_expired = entry.locked_until.is_some_and(|t| t <= now); - if lock_expired || now - entry.window_start >= WINDOW { + if entry.locked_until.is_some() || now - entry.window_start >= WINDOW { entry.failures = 0; entry.window_start = now; entry.locked_until = None; } - entry.failures += 1; - if entry.failures >= MAX_FAILURES { - entry.locked_until = Some(now + LOCKOUT); - Some(LOCKOUT) - } else { - None + if entry.failures + entry.in_flight >= MAX_FAILURES { + return Err(IN_FLIGHT_RETRY); + } + entry.in_flight += 1; + Ok(Attempt { + guard: Arc::clone(self), + key: key.to_string(), + open: true, + }) + } + + fn settle(&self, key: &str, outcome: Outcome, now: Instant) -> Option { + let mut map = self.inner.lock().unwrap_or_else(|e| e.into_inner()); + // Present by construction: eviction never takes an entry with attempts in flight. + let entry = map.get_mut(key)?; + entry.in_flight = entry.in_flight.saturating_sub(1); + let lock = match outcome { + Outcome::Success => { + entry.failures = 0; + entry.locked_until = None; + None + } + Outcome::Failure => { + entry.failures += 1; + if entry.failures >= MAX_FAILURES { + entry.locked_until = Some(now + LOCKOUT); + Some(LOCKOUT) + } else { + None + } + } + Outcome::Abandoned => None, + }; + if entry.in_flight == 0 && entry.failures == 0 && entry.locked_until.is_none() { + map.remove(key); } + lock + } +} + +/// Make room in a full table. +/// +/// Expired entries go first. If that is not enough — a flood of distinct usernames keeps +/// every entry inside `WINDOW` — the cheapest state to lose goes next: unlocked before +/// locked, fewest failures first, oldest window first. A flood of junk usernames holds one +/// failure each, so it evicts itself before it reaches a real account's partial count; the +/// old behaviour (drop everything not locked out) let that flood reset any count on demand. +/// An entry with attempts in flight is never evicted: its `Attempt` still has to settle. +fn evict(map: &mut HashMap, now: Instant) { + map.retain(|_, e| { + e.in_flight > 0 || e.locked_at(now) || (e.failures > 0 && now - e.window_start < WINDOW) + }); + if map.len() <= EVICT_TO { + return; + } + let mut victims: Vec<(bool, u32, Instant, String)> = map + .iter() + .filter(|(_, e)| e.in_flight == 0) + .map(|(k, e)| (e.locked_at(now), e.failures, e.window_start, k.clone())) + .collect(); + victims.sort_unstable(); + let excess = map.len() - EVICT_TO; + for (_, _, _, key) in victims.into_iter().take(excess) { + map.remove(&key); } } @@ -113,37 +215,54 @@ impl LoginGuard { mod tests { use super::*; + fn guard() -> Arc { + Arc::new(LoginGuard::new()) + } + + /// One whole failed login at `now`: reserve, then settle as a failure. + fn fail(g: &Arc, key: &str, now: Instant) -> Option { + g.begin_attempt_at(key, now) + .unwrap_or_else(|_| panic!("attempt for {key} should be admitted")) + .finish_at(false, now) + } + + fn failures(g: &LoginGuard, key: &str) -> Option { + g.inner.lock().unwrap().get(key).map(|e| e.failures) + } + #[test] fn locks_after_max_failures_and_expires() { - let g = LoginGuard::new(); + let g = guard(); let t0 = Instant::now(); for i in 1..MAX_FAILURES { - assert_eq!(g.record_failure_at("admin", t0), None, "failure {i}"); - assert!(g.check_at("admin", t0).is_ok()); + assert_eq!(fail(&g, "admin", t0), None, "failure {i}"); } - assert_eq!(g.record_failure_at("admin", t0), Some(LOCKOUT)); - let retry = g.check_at("admin", t0).unwrap_err(); + assert_eq!(fail(&g, "admin", t0), Some(LOCKOUT)); + let retry = g.begin_attempt_at("admin", t0).err().expect("locked"); assert!(retry <= LOCKOUT && retry > Duration::ZERO); - assert!(g.check_at("admin", t0 + LOCKOUT).is_ok()); // After the lockout a fresh window starts. - assert_eq!(g.record_failure_at("admin", t0 + LOCKOUT), None); + assert_eq!(fail(&g, "admin", t0 + LOCKOUT), None); + assert_eq!(failures(&g, "admin"), Some(1)); } #[test] fn success_clears_and_window_resets() { - let g = LoginGuard::new(); + let g = guard(); let t0 = Instant::now(); for _ in 0..(MAX_FAILURES - 1) { - g.record_failure_at("bob", t0); + fail(&g, "bob", t0); } - g.record_success("bob"); - assert_eq!(g.record_failure_at("bob", t0), None); + g.begin_attempt_at("bob", t0) + .expect("admitted") + .finish_at(true, t0); + assert_eq!(failures(&g, "bob"), None, "a clean entry is removed"); + assert_eq!(fail(&g, "bob", t0), None); // Failures older than WINDOW do not count. - let g = LoginGuard::new(); + let g = guard(); for _ in 0..(MAX_FAILURES - 1) { - g.record_failure_at("eve", t0); + fail(&g, "eve", t0); } - assert_eq!(g.record_failure_at("eve", t0 + WINDOW), None); + assert_eq!(fail(&g, "eve", t0 + WINDOW), None); } #[test] @@ -155,15 +274,115 @@ mod tests { ); } + /// The bypass this module exists to close: every request in a parallel burst used to + /// pass a read-only check before any of them recorded a failure. + #[test] + fn concurrent_attempts_admit_at_most_the_limit() { + let g = guard(); + let burst = 64; + let barrier = Arc::new(std::sync::Barrier::new(burst)); + let handles: Vec<_> = (0..burst) + .map(|_| { + let (g, barrier) = (Arc::clone(&g), Arc::clone(&barrier)); + std::thread::spawn(move || { + barrier.wait(); + g.begin_attempt("admin").ok() + }) + }) + .collect(); + // Hold every admitted attempt open, as the handler does across the verify. + let admitted: Vec = handles + .into_iter() + .filter_map(|h| h.join().expect("thread")) + .collect(); + assert_eq!(admitted.len(), MAX_FAILURES as usize); + assert_eq!( + g.begin_attempt("admin").err(), + Some(IN_FLIGHT_RETRY), + "the budget is held by attempts in flight" + ); + // All of them fail: exactly MAX_FAILURES guesses were bought, and the lockout holds. + let locks: Vec<_> = admitted.into_iter().map(|a| a.finish(false)).collect(); + assert_eq!(locks.iter().flatten().count(), 1); + assert!(g.begin_attempt("admin").is_err()); + } + + #[test] + fn failures_and_in_flight_attempts_share_the_budget() { + let g = guard(); + let t0 = Instant::now(); + for _ in 0..(MAX_FAILURES - 2) { + fail(&g, "admin", t0); + } + let a = g.begin_attempt_at("admin", t0).expect("ninth"); + let b = g.begin_attempt_at("admin", t0).expect("tenth"); + assert_eq!(g.begin_attempt_at("admin", t0).err(), Some(IN_FLIGHT_RETRY)); + drop((a, b)); + } + + #[test] + fn a_dropped_attempt_releases_its_slot_and_records_nothing() { + let g = guard(); + let held: Vec = (0..MAX_FAILURES) + .map(|_| g.begin_attempt("admin").expect("admitted")) + .collect(); + assert!(g.begin_attempt("admin").is_err()); + // What an early `?` in the handler does to the reservation. + drop(held); + assert!( + g.inner.lock().unwrap().is_empty(), + "no slot held, no failure recorded, no entry left behind" + ); + let again: Vec = (0..MAX_FAILURES) + .map(|_| g.begin_attempt("admin").expect("slots came back")) + .collect(); + assert_eq!(again.len(), MAX_FAILURES as usize); + } + #[test] fn flood_of_usernames_does_not_grow_without_bound() { - let g = LoginGuard::new(); + let g = guard(); let t0 = Instant::now(); - for i in 0..(PRUNE_ABOVE + 500) { - g.record_failure_at(&format!("user{i}"), t0); + for i in 0..(MAX_ENTRIES * 2) { + fail(&g, &format!("user{i}"), t0); + assert!(g.inner.lock().unwrap().len() <= MAX_ENTRIES); } - // The next check prunes: nothing is locked, so the table collapses. - assert!(g.check_at("anyone", t0).is_ok()); - assert!(g.inner.lock().unwrap().len() <= PRUNE_ABOVE); + } + + /// The old pruning dropped every entry that was not locked out, so nine guesses, a + /// flood of junk usernames, nine more guesses — repeated — never reached a lockout. + #[test] + fn a_junk_username_flood_does_not_erase_a_targets_count() { + let g = guard(); + let t0 = Instant::now(); + for _ in 0..(MAX_FAILURES - 1) { + fail(&g, "admin", t0); + } + for i in 0..(MAX_ENTRIES * 2) { + fail(&g, &format!("junk{i}"), t0); + } + assert_eq!(failures(&g, "admin"), Some(MAX_FAILURES - 1)); + assert_eq!(fail(&g, "admin", t0), Some(LOCKOUT)); + } + + #[test] + fn eviction_spares_attempts_in_flight_and_takes_expired_entries_first() { + let g = guard(); + let t0 = Instant::now(); + let held = g.begin_attempt_at("in-flight", t0).expect("admitted"); + fail(&g, "stale", t0); + let later = t0 + WINDOW; + for i in 0..MAX_ENTRIES { + fail(&g, &format!("junk{i}"), later); + } + let map = g.inner.lock().unwrap(); + assert!( + map.contains_key("in-flight"), + "zero failures, but unsettled" + ); + assert!(!map.contains_key("stale")); + drop(map); + assert_eq!(held.finish_at(false, later), None); + assert_eq!(failures(&g, "in-flight"), Some(1)); } } diff --git a/crates/fiber-api/src/main.rs b/crates/fiber-api/src/main.rs index a5265c8..7178c59 100644 --- a/crates/fiber-api/src/main.rs +++ b/crates/fiber-api/src/main.rs @@ -43,6 +43,15 @@ const REDIS_RESPONSE_TIMEOUT: Duration = Duration::from_secs(2); /// How long the boot-time Redis probe waits before the process starts without it. const REDIS_BOOT_PROBE: Duration = Duration::from_secs(5); +/// Loopback: a bare `fiber-api` comes up with a bootstrap admin whose password is in the +/// README, so the address it answers on by default must not be one a network can reach. +/// A deployment sets `FIBER_LISTEN` (Compose and the image do). +const DEFAULT_LISTEN: &str = "127.0.0.1:18080"; +/// The bootstrap admin's name when `FIBER_ADMIN_USER` is not set. +const DEFAULT_ADMIN_USER: &str = "admin"; +/// The bootstrap admin's password when `FIBER_ADMIN_PASSWORD` is not set. +const DEFAULT_ADMIN_PASSWORD: &str = "fiber"; + #[derive(Parser, Debug)] #[command(name = "fiber-api", version)] struct Args { @@ -60,18 +69,23 @@ struct Args { )] redis_url: String, - #[arg(long, env = "FIBER_LISTEN", default_value = "0.0.0.0:18080")] + #[arg(long, env = "FIBER_LISTEN", default_value = DEFAULT_LISTEN)] listen: SocketAddr, #[arg(long, env = "FIBER_ARTIFACTS_DIR", default_value = "./data/artifacts")] artifacts_dir: String, - #[arg(long, env = "FIBER_ADMIN_USER", default_value = "admin")] + #[arg(long, env = "FIBER_ADMIN_USER", default_value = DEFAULT_ADMIN_USER)] admin_user: String, - #[arg(long, env = "FIBER_ADMIN_PASSWORD", default_value = "fiber")] + #[arg(long, env = "FIBER_ADMIN_PASSWORD", default_value = DEFAULT_ADMIN_PASSWORD)] admin_password: String, + /// `1` to start on a non-loopback address even though the bootstrap admin can still + /// log in with the default password. Without it that combination refuses to boot. + #[arg(long, env = "FIBER_ALLOW_DEFAULT_ADMIN")] + allow_default_admin: Option, + /// Seed the Showcase demo project: `auto` (only when the instance has no users yet), /// `1` to seed on this boot, `0` never. #[arg(long, env = "FIBER_SEED_SHOWCASE", default_value = "auto")] @@ -99,6 +113,49 @@ enum Command { }, } +/// What to do about a bootstrap admin that can log in with the default password. +#[derive(Debug, PartialEq, Eq)] +enum DefaultAdmin { + /// Nobody can log in with it. + Absent, + /// Usable, but only from this machine, or the operator said to start anyway. + Warn, + /// Usable from the network. `admin` / `fiber` is in the README. + Refuse, +} + +fn default_admin_exposure( + listen: std::net::IpAddr, + default_login_works: bool, + allowed: bool, +) -> DefaultAdmin { + if !default_login_works { + DefaultAdmin::Absent + } else if listen.is_loopback() || allowed { + DefaultAdmin::Warn + } else { + DefaultAdmin::Refuse + } +} + +/// The accounts that may still hold the published login: the configured bootstrap admin, +/// and the literal `admin` when the configuration names someone else. An instance +/// bootstrapped as `admin` / `fiber` whose `FIBER_ADMIN_USER` was later pointed at another +/// account still has `admin` / `fiber` in its users table. +fn default_admin_candidates(configured: &str) -> Vec<&str> { + let mut names = vec![configured]; + if configured != DEFAULT_ADMIN_USER { + names.push(DEFAULT_ADMIN_USER); + } + names +} + +/// `FIBER_ALLOW_DEFAULT_ADMIN`: on for `1` or `true`. Compose passes an empty string for +/// an unset variable, so anything else — including empty — is off. +fn flag_is_on(value: Option<&str>) -> bool { + matches!(value.map(str::trim), Some("1" | "true")) +} + /// `fiber-api reset-password`: no migrations, no server — just the one update. async fn reset_password(database_url: &str, username: &str, from_stdin: bool) -> Result<()> { let (password, generated) = if from_stdin { @@ -152,7 +209,9 @@ async fn main() -> Result<()> { tracing::info!(max_steps, "step cap overridden"); } std::fs::create_dir_all(&args.artifacts_dir)?; - fiber_core::secrets::init_from_env(); + // A key that is set but unusable stops the boot: carrying on would store secrets in + // plaintext for an operator who believes they are encrypted. + fiber_core::secrets::init_from_env()?; let pool = db::connect(&args.database_url).await?; db::migrate(&pool).await?; @@ -186,9 +245,51 @@ async fn main() -> Result<()> { // Read before `ensure_admin_user`, which creates one: this is what "fresh instance" // means for the seeder too, so a deleted demo project stays deleted across restarts. let users_existed = store.count_users().await? > 0; + // Decided before the admin is created, not after: `FIBER_ADMIN_PASSWORD` is applied + // only to an empty database, so refusing *after* creating `admin` / `fiber` would + // leave an instance that setting the variable could no longer fix. + // Which account, if any, the published login opens. + let mut default_login: Option<&str> = None; + if users_existed { + for name in default_admin_candidates(&args.admin_user) { + if store.password_is(name, DEFAULT_ADMIN_PASSWORD).await? { + default_login = Some(name); + break; + } + } + } else if args.admin_password == DEFAULT_ADMIN_PASSWORD { + default_login = Some(&args.admin_user); + } + match default_admin_exposure( + args.listen.ip(), + default_login.is_some(), + flag_is_on(args.allow_default_admin.as_deref()), + ) { + DefaultAdmin::Absent => {} + DefaultAdmin::Warn => tracing::warn!( + user = default_login.unwrap_or_default(), + "this account can log in with the default password (`fiber`) — change it before exposing this instance" + ), + DefaultAdmin::Refuse if users_existed => anyhow::bail!( + "refusing to listen on {}: `{user}` can still log in with the default password `fiber`. \ + Change it (`fiber-api reset-password {user}` works without the server), \ + or listen on a loopback address, or set FIBER_ALLOW_DEFAULT_ADMIN=1 to start anyway", + args.listen, + user = default_login.unwrap_or_default(), + ), + DefaultAdmin::Refuse => anyhow::bail!( + "refusing to listen on {} with the default admin password `fiber`. \ + Set FIBER_ADMIN_PASSWORD, or listen on a loopback address, \ + or set FIBER_ALLOW_DEFAULT_ADMIN=1 to start anyway", + args.listen, + ), + } let admin = store .ensure_admin_user(&args.admin_user, &args.admin_password) .await?; + // Make the dummy hash now, so the first login for an unknown username does not pay + // for it and stand out against the ones after. + tokio::task::spawn_blocking(fiber_core::tokens::dummy_password_hash); if fiber_core::seed::should_seed_showcase(Some(&args.seed_showcase), users_existed) .map_err(anyhow::Error::msg)? { @@ -307,12 +408,6 @@ async fn main() -> Result<()> { sessions: sessions.clone(), }; - if args.admin_password == "fiber" { - tracing::warn!( - "FIBER_ADMIN_PASSWORD is the default (`fiber`) — change it before exposing this instance" - ); - } - let app = routes::router(state) .layer(cors_layer()) .layer(TraceLayer::new_for_http()); @@ -450,6 +545,78 @@ fn cors_layer() -> CorsLayer { mod tests { use super::*; + #[test] + fn the_default_listen_address_is_loopback() { + let addr: SocketAddr = DEFAULT_LISTEN.parse().unwrap(); + assert!(addr.ip().is_loopback(), "{addr}"); + assert_eq!(addr.port(), 18080); + } + + #[test] + fn a_default_admin_login_is_refused_on_any_address_a_network_can_reach() { + use std::net::IpAddr; + let ip = |s: &str| s.parse::().unwrap(); + for public in [ + "0.0.0.0", + "::", + "192.168.1.10", + "10.0.0.5", + "::ffff:127.0.0.1", + ] { + assert_eq!( + default_admin_exposure(ip(public), true, false), + DefaultAdmin::Refuse, + "{public}" + ); + // The operator's explicit override, and a changed password, both start. + assert_eq!( + default_admin_exposure(ip(public), true, true), + DefaultAdmin::Warn + ); + assert_eq!( + default_admin_exposure(ip(public), false, false), + DefaultAdmin::Absent + ); + } + for local in ["127.0.0.1", "127.8.8.8", "::1"] { + assert_eq!( + default_admin_exposure(ip(local), true, false), + DefaultAdmin::Warn, + "{local}" + ); + assert_eq!( + default_admin_exposure(ip(local), false, false), + DefaultAdmin::Absent + ); + } + } + + /// Checking only `FIBER_ADMIN_USER` let an instance bootstrapped as `admin` / `fiber` + /// boot publicly once the variable named another account. + #[test] + fn the_literal_admin_is_checked_even_when_another_name_is_configured() { + assert_eq!(default_admin_candidates("admin"), ["admin"]); + assert_eq!(default_admin_candidates("root"), ["root", "admin"]); + assert_eq!(default_admin_candidates("Admin"), ["Admin", "admin"]); + } + + #[test] + fn the_override_is_on_only_when_it_says_so() { + for on in ["1", "true", " 1 "] { + assert!(flag_is_on(Some(on)), "{on:?}"); + } + for off in [ + None, + Some(""), + Some("0"), + Some("false"), + Some("no"), + Some("yes"), + ] { + assert!(!flag_is_on(off), "{off:?}"); + } + } + #[test] fn with_no_subcommand_the_binary_still_serves() { let args = Args::try_parse_from(["fiber-api"]).unwrap(); diff --git a/crates/fiber-api/src/routes.rs b/crates/fiber-api/src/routes.rs index 7354efe..91d9e5a 100644 --- a/crates/fiber-api/src/routes.rs +++ b/crates/fiber-api/src/routes.rs @@ -8,6 +8,7 @@ use axum::response::IntoResponse; use axum::routing::{MethodRouter, delete, get, post, put}; use axum::{Json, Router}; use chrono::{DateTime, Utc}; +use fiber_core::tokens::constant_time_eq; use fiber_core::{ AddMemberRequest, CreateAgentRequest, CreatePipelineRequest, CreateProjectRequest, CreateUserRequest, LoginRequest, ProjectRole, StartRunRequest, StepRun, UpdateAgentRequest, @@ -424,9 +425,16 @@ async fn login( Json(req): Json, ) -> Result { let key = crate::login_guard::LoginGuard::key(&req.username); - if let Err(retry_after) = state.login_guard.check(&key) { - return Ok(too_many_logins(retry_after)); - } + // Reserve the attempt before verifying, so parallel requests share one budget. The + // reservation releases itself if this handler returns early or is dropped. + let attempt = match state.login_guard.begin_attempt(&key) { + Ok(attempt) => attempt, + Err(retry_after) => return Ok(too_many_logins(retry_after)), + }; + // The verify waits its turn for one of a few process-wide hashing slots + // (`fiber_core::tokens::MAX_CONCURRENT_HASHES`). If none comes free in time that is + // `ApiError::Busy`, a 503 — and the `?` drops the reservation, so being turned away + // by a busy server never counts as a failed login. let resp = state .store .login(&req.username, &req.password) @@ -434,11 +442,11 @@ async fn login( .map_err(ApiError::from)?; match resp { Some(resp) => { - state.login_guard.record_success(&key); + attempt.finish(true); Ok(Json(resp).into_response()) } None => { - if let Some(lock) = state.login_guard.record_failure(&key) { + if let Some(lock) = attempt.finish(false) { tracing::warn!(username = %key, lock_secs = lock.as_secs(), "login locked out"); } else { tracing::warn!(username = %key, "login failed"); @@ -454,7 +462,7 @@ fn too_many_logins(retry_after: std::time::Duration) -> axum::response::Response StatusCode::TOO_MANY_REQUESTS, [(header::RETRY_AFTER, secs.to_string())], Json(json!({ - "error": "too many failed logins; try again later", + "error": "too many login attempts; try again later", "retry_after_secs": secs, })), ) @@ -1778,9 +1786,14 @@ async fn update_agent( .collect::>() }) .unwrap_or_default(); + // Narrow a live agent at once; never widen it here. What the agent may be offered + // is `row ∩ Hello`, and only its WebSocket session knows what `Hello` claimed — + // installing the row verbatim gave a live agent labels it never said it could run. + // The session recomputes the full registration from the row on its next heartbeat, + // which is when a widening takes effect. state .scheduler - .update_agent_presence(id, labels, agent.concurrency.max(1) as u32) + .narrow_agent_presence(id, &labels, agent.concurrency.max(1) as u32) .await; agent.token_hash = "***".into(); Ok(Json(agent)) @@ -2023,7 +2036,7 @@ async fn github_webhook( .await .unwrap_or(true) { - match crate::github::resolve_token(&state.store, id).await { + match crate::github::resolve_project_token(&state.store, id).await { Ok(Some(token)) => { if let Some((owner, repo)) = crate::github::repo_full_name(&payload) { match crate::github::list_pull_request_files( @@ -2046,7 +2059,7 @@ async fn github_webhook( } Ok(None) => { tracing::debug!( - "no GITHUB_TOKEN; PR path filters need API file list or payload.changed_files" + "no GITHUB_TOKEN project secret; PR path filters need API file list or payload.changed_files" ); } Err(e) => { @@ -2300,16 +2313,6 @@ fn verify_github_sig(secret: &str, body: &str, signature: &str) -> bool { constant_time_eq(expected.as_bytes(), hex_sig.as_bytes()) } -fn constant_time_eq(a: &[u8], b: &[u8]) -> bool { - if a.len() != b.len() { - return false; - } - a.iter() - .zip(b.iter()) - .fold(0u8, |acc, (x, y)| acc | (x ^ y)) - == 0 -} - #[derive(Debug)] pub enum ApiError { NotFound, @@ -2317,8 +2320,14 @@ pub enum ApiError { Forbidden, BadRequest(String), Internal(String), + /// The server could not take the request on right now (password hashing is + /// saturated). Nothing was decided about the caller: not a `401`, not a lockout. + Busy, } +/// `Retry-After` for [`ApiError::Busy`]: a hashing slot turns over in well under this. +const BUSY_RETRY_AFTER_SECS: u64 = 2; + impl From for ApiError { fn from(e: fiber_core::StoreError) -> Self { use fiber_core::StoreError as S; @@ -2348,6 +2357,9 @@ impl From for ApiError { S::Other(_) => ApiError::Internal(format!("{e:#}")), }; } + if e.downcast_ref::().is_some() { + return ApiError::Busy; + } // A definition the caller wrote wrongly is the caller's fault. if e.downcast_ref::().is_some() { return ApiError::BadRequest(format!("{e:#}")); @@ -2369,6 +2381,18 @@ impl IntoResponse for ApiError { tracing::error!(error = %m, "internal error"); (StatusCode::INTERNAL_SERVER_ERROR, "internal error".into()) } + ApiError::Busy => { + tracing::warn!("password hashing saturated; request turned away"); + return ( + StatusCode::SERVICE_UNAVAILABLE, + [(header::RETRY_AFTER, BUSY_RETRY_AFTER_SECS.to_string())], + Json(json!({ + "error": "server busy; nothing was checked — try again shortly", + "retry_after_secs": BUSY_RETRY_AFTER_SECS, + })), + ) + .into_response(); + } }; (status, Json(json!({ "error": msg }))).into_response() } @@ -2991,6 +3015,21 @@ mod tests { ); } + /// A saturated hasher is the server's condition, not a verdict on the caller: it must + /// not come back as the 401 of a bad password, the 429 of a lockout, or a 500. + #[test] + fn a_busy_hasher_is_a_503_with_retry_after_through_a_context_layer() { + let e = anyhow::Error::new(fiber_core::tokens::HashBusy).context("login"); + let api = ApiError::from(e); + assert!(matches!(api, ApiError::Busy), "{api:?}"); + let resp = api.into_response(); + assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE); + assert_eq!( + resp.headers().get(header::RETRY_AFTER).unwrap(), + BUSY_RETRY_AFTER_SECS.to_string().as_str() + ); + } + #[test] fn api_error_maps_the_store_error_type_not_its_message() { use fiber_core::StoreError as S; diff --git a/crates/fiber-api/src/ws.rs b/crates/fiber-api/src/ws.rs index c59301d..927c516 100644 --- a/crates/fiber-api/src/ws.rs +++ b/crates/fiber-api/src/ws.rs @@ -917,6 +917,9 @@ async fn handle_agent( // is exiting on purpose. Not persisted — `agents` has no column for the revision // yet; the Hello log line carries it. let mut agent_protocol: Option = None; + // What this session's `Hello` claimed, kept so the registration can be recomputed + // against the agent's row whenever the row may have changed (every heartbeat). + let mut hello_claim: Option<(Vec, u32)> = None; let mut goodbye = false; let mut shutdown = state.shutdown.clone(); let mut liveness = tokio::time::Instant::now() + AGENT_LIVENESS_TIMEOUT; @@ -974,8 +977,9 @@ async fn handle_agent( break; } agent_protocol = Some(protocol_version); - info!(%agent_id, %name, ?labels, protocol_version, "agent hello"); - // Reload from DB so pool scope is authoritative (not client-supplied). + info!(%agent_id, %name, ?labels, concurrency, protocol_version, "agent hello"); + // Reload from DB so pool scope, labels and concurrency are authoritative + // (not client-supplied). // A missing row ends the session rather than defaulting: `.flatten()` // into `and_then(project_id)` made "agent deleted" indistinguishable from // "global agent", so a deleted project-scoped agent would re-register @@ -988,9 +992,35 @@ async fn handle_agent( break; }; let project_id = agent_row.project_id; + // `Hello` says what the agent can run; the row says what it may. Taking + // labels and concurrency from `Hello` let anyone holding an agent token + // reconnect claiming every label and `u32::MAX` slots, and lease steps — + // with their secrets — meant for other agents in the pool. + let (effective_labels, effective_concurrency, refused) = effective_registration( + &agent_row.labels_vec(), + agent_row.concurrency, + &labels, + concurrency, + ); + hello_claim = Some((labels.clone(), concurrency)); + if refused { + warn!( + %agent_id, + hello_labels = ?labels, + hello_concurrency = concurrency, + ?effective_labels, + effective_concurrency, + "agent hello asked for more than its registration allows; registered with the allowed subset" + ); + } state .scheduler - .register_agent(agent_id, labels, concurrency, project_id) + .register_agent( + agent_id, + effective_labels, + effective_concurrency, + project_id, + ) .await; // Re-bind connection under confirmed agent_id (same as token) state @@ -1005,18 +1035,17 @@ async fn handle_agent( warn_if_spoofed(agent_id, claimed, &mut spoof_logged); // Revocation must not depend on Redis delivery: a rotated or deleted token // ends the session on its next heartbeat, whichever replica holds it. - let still_valid = matches!( - state.store.get_agent(agent_id).await, - Ok(Some(a)) if a.token_hash == token_hash - ); - if !still_valid { - warn!(%agent_id, "agent token no longer valid; ending session"); - revoked = true; - let _ = tx.send(ServerMessage::Error { - message: "agent token revoked — reconnect with a valid token".into(), - }); - break; - } + let agent_row = match state.store.get_agent(agent_id).await { + Ok(Some(a)) if a.token_hash == token_hash => a, + _ => { + warn!(%agent_id, "agent token no longer valid; ending session"); + revoked = true; + let _ = tx.send(ServerMessage::Error { + message: "agent token revoked — reconnect with a valid token".into(), + }); + break; + } + }; // force_disconnect_agent drops the connection; the next heartbeat ends the session. if !state.scheduler.has_connection(agent_id).await { let _ = tx.send(ServerMessage::Error { @@ -1024,6 +1053,24 @@ async fn handle_agent( }); break; } + // The row was just read, so apply it: an edit to the agent's labels or + // concurrency — made on this replica or another — reaches the live + // registration within one heartbeat, in both directions, and always as + // `row ∩ Hello`. (`PUT /api/agents/{id}` narrows at once on its own + // replica; widening waits for this, because only the session knows what + // `Hello` claimed.) Before any offer below is made. + if let Some((hello_labels, hello_concurrency)) = &hello_claim { + let (labels, concurrency, _) = effective_registration( + &agent_row.labels_vec(), + agent_row.concurrency, + hello_labels, + *hello_concurrency, + ); + state + .scheduler + .update_agent_presence(agent_id, labels, concurrency) + .await; + } let _ = state.store.touch_agent(agent_id).await; let _ = state.scheduler.renew_leases(agent_id).await; fill_agent(&state, agent_id, &tx).await; @@ -1507,9 +1554,125 @@ async fn handle_run_events( } } +/// What an agent is registered with, given its database row and what its `Hello` claimed. +/// +/// The row is a ceiling and `Hello` a declaration of capability, so the result is never +/// more than either: +/// +/// - **labels**: the row's labels that `Hello` also sent. An agent cannot add a label by +/// claiming it, and is not offered work for a label it did not claim (a host started +/// with `os=linux` under a row that says `os=linux,docker=true` cannot run `docker=true` +/// steps, whatever the row says). +/// - **concurrency**: the smaller of the two, at least 1. More than the row is not +/// allowed; more than `Hello` would only park offers on the agent's local semaphore. +/// +/// No exception for a row with **no** labels: it grants none, and the agent is offered +/// only steps that require none (the queue match is `step.labels <@ agent.labels`). +/// Treating an empty row as "unrestricted" would leave the original hole open for exactly +/// those rows, and would turn an admin clearing an agent's labels to restrict it into +/// the opposite. +/// +/// The third value is `true` when `Hello` asked for something it did not get — a label +/// outside the row, or more slots than the row has. That is a misconfiguration or a +/// probe, and the caller logs it; an agent that claims *less* than its row is normal. +fn effective_registration( + db_labels: &[String], + db_concurrency: i32, + hello_labels: &[String], + hello_concurrency: u32, +) -> (Vec, u32, bool) { + let allowed_slots = u32::try_from(db_concurrency).unwrap_or(0).max(1); + let wanted_slots = hello_concurrency.max(1); + let concurrency = wanted_slots.min(allowed_slots); + let mut refused = wanted_slots > allowed_slots; + + refused |= hello_labels.iter().any(|l| !db_labels.contains(l)); + let labels = db_labels + .iter() + .filter(|l| hello_labels.contains(l)) + .cloned() + .collect(); + (labels, concurrency, refused) +} + #[cfg(test)] mod tests { use super::*; + + fn labels(csv: &str) -> Vec { + csv.split(',') + .filter(|s| !s.is_empty()) + .map(str::to_string) + .collect() + } + + #[test] + fn a_hello_cannot_claim_labels_or_slots_its_row_does_not_have() { + // The attack: a token for a plain linux agent reconnects as everything, unbounded. + let (l, c, refused) = effective_registration( + &labels("os=linux"), + 2, + &labels("os=linux,docker=true,pool=release,gpu=true"), + u32::MAX, + ); + assert_eq!(l, labels("os=linux")); + assert_eq!(c, 2); + assert!(refused); + // Nothing in common leaves an agent that matches only label-less steps. + let (l, c, refused) = + effective_registration(&labels("os=linux"), 1, &labels("os=macos"), 1); + assert!(l.is_empty()); + assert_eq!(c, 1); + assert!(refused); + } + + #[test] + fn a_matching_hello_registers_unchanged_and_quietly() { + let (l, c, refused) = effective_registration( + &labels("os=linux,docker=true"), + 4, + &labels("docker=true,os=linux"), + 4, + ); + assert_eq!(l, labels("os=linux,docker=true")); + assert_eq!(c, 4); + assert!(!refused); + } + + #[test] + fn an_agent_may_claim_less_than_its_row_and_is_registered_with_what_it_claimed() { + // The UI's default labels against the installer's: this host cannot run docker + // steps and must not be offered them because its row could. + let (l, c, refused) = + effective_registration(&labels("os=linux,docker=true"), 4, &labels("os=linux"), 1); + assert_eq!(l, labels("os=linux")); + assert_eq!(c, 1); + assert!(!refused); + } + + /// An empty row used to mean "whatever the agent says", which kept the hole open + /// for those rows and inverted an admin clearing labels to restrict an agent. + #[test] + fn a_row_without_labels_grants_none() { + let (l, c, refused) = effective_registration(&[], 2, &labels("os=linux,arch=arm64"), 9); + assert!(l.is_empty(), "{l:?}"); + assert_eq!(c, 2); + assert!(refused); + // An agent that claims none either is simply a label-less agent. + let (l, _, refused) = effective_registration(&[], 2, &[], 2); + assert!(l.is_empty()); + assert!(!refused); + } + + #[test] + fn concurrency_is_never_below_one() { + for (db, hello) in [(0, 0), (-3, 5), (5, 0)] { + let (_, c, _) = effective_registration(&[], db, &[], hello); + assert_eq!(c, 1, "db {db}, hello {hello}"); + } + // A zero in Hello is clamped, not reported: older agents send it. + assert!(!effective_registration(&[], 1, &[], 0).2); + } use serde_json::json; #[test] diff --git a/crates/fiber-core/Cargo.toml b/crates/fiber-core/Cargo.toml index 3743ca2..c5f4b8e 100644 --- a/crates/fiber-core/Cargo.toml +++ b/crates/fiber-core/Cargo.toml @@ -29,8 +29,6 @@ argon2 = { workspace = true } aes-gcm = { workspace = true } zeroize = { workspace = true } getrandom = { workspace = true } - -[dev-dependencies] tokio = { workspace = true } [lints] diff --git a/crates/fiber-core/src/secrets.rs b/crates/fiber-core/src/secrets.rs index c222c99..680aa4a 100644 --- a/crates/fiber-core/src/secrets.rs +++ b/crates/fiber-core/src/secrets.rs @@ -2,6 +2,10 @@ //! //! Set `FIBER_SECRETS_KEY` to a 64-char hex string (32 bytes). When unset, values //! are stored plaintext (local/dev). Ciphertext is prefixed with `enc:v1:`. +//! +//! Unset and unusable are different things. A key that is set but is not 64 hex +//! characters is an error — at boot, and on every encrypt or decrypt — never a quiet +//! fall back to plaintext: an operator who set a key believes their secrets are encrypted. use aes_gcm::aead::{Aead, KeyInit}; use aes_gcm::{Aes256Gcm, Nonce}; @@ -13,43 +17,73 @@ use zeroize::Zeroizing; const PREFIX: &str = "enc:v1:"; -static KEY: OnceLock> = OnceLock::new(); +static KEY: OnceLock, KeyError>> = OnceLock::new(); + +/// Why a `FIBER_SECRETS_KEY` that is set cannot be used. Carries a length, never the +/// value: this ends up in a boot log. +#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] +pub enum KeyError { + #[error( + "FIBER_SECRETS_KEY is set but is not hexadecimal ({chars} characters; it must be exactly 64 hex characters — generate one with `openssl rand -hex 32`, and check for quotes or stray characters)" + )] + NotHex { chars: usize }, + #[error( + "FIBER_SECRETS_KEY is set but is {chars} hex characters long; it must be exactly 64 (32 bytes) — generate one with `openssl rand -hex 32`" + )] + WrongLength { chars: usize }, +} -fn load_key() -> Option<[u8; 32]> { - let raw = std::env::var("FIBER_SECRETS_KEY").ok()?; - let raw = raw.trim(); +/// The key in `raw`, the value of `FIBER_SECRETS_KEY`. +/// +/// `Ok(None)` only when there is no key at all — unset, empty, or whitespace (Compose +/// passes an empty string for an unset variable). Anything else that is not 64 hex +/// characters is an error. It used to be `None` as well, which logged the same "unset +/// (dev only)" line as a missing key and stored every secret written afterwards in +/// plaintext. +pub fn parse_key(raw: Option<&str>) -> Result, KeyError> { + let raw = raw.unwrap_or_default().trim(); if raw.is_empty() { - return None; + return Ok(None); } - let bytes = hex::decode(raw).ok()?; - if bytes.len() != 32 { - tracing::warn!( - len = bytes.len(), - "FIBER_SECRETS_KEY must be 32 bytes (64 hex chars); ignoring" - ); - return None; + if !raw.chars().all(|c| c.is_ascii_hexdigit()) { + return Err(KeyError::NotHex { + chars: raw.chars().count(), + }); } let mut key = [0u8; 32]; - key.copy_from_slice(&bytes); - Some(key) + hex::decode_to_slice(raw, &mut key).map_err(|_| KeyError::WrongLength { chars: raw.len() })?; + Ok(Some(key)) +} + +fn key_from_env() -> Result, KeyError> { + match std::env::var("FIBER_SECRETS_KEY") { + Ok(raw) => parse_key(Some(&raw)), + Err(std::env::VarError::NotPresent) => Ok(None), + Err(std::env::VarError::NotUnicode(raw)) => Err(KeyError::NotHex { chars: raw.len() }), + } } -fn key() -> Option<&'static [u8; 32]> { - KEY.get_or_init(load_key).as_ref() +fn key() -> Result, KeyError> { + match KEY.get_or_init(key_from_env) { + Ok(key) => Ok(key.as_ref()), + Err(e) => Err(e.clone()), + } } -/// Call once at process start so missing-key warning is visible. -pub fn init_from_env() { - match key() { +/// Call once at process start: an unusable key stops the boot, and a missing one is +/// warned about where it will be seen. +pub fn init_from_env() -> Result<(), KeyError> { + match key()? { Some(_) => tracing::info!("project secrets encryption enabled (FIBER_SECRETS_KEY)"), None => tracing::warn!( "FIBER_SECRETS_KEY unset — project secrets stored in plaintext (dev only)" ), } + Ok(()) } pub fn encrypt_secret(plaintext: &str) -> Result { - let Some(k) = key() else { + let Some(k) = key()? else { return Ok(plaintext.to_string()); }; encrypt_with(k, plaintext) @@ -60,7 +94,7 @@ pub fn decrypt_secret(stored: &str) -> Result { // Plaintext (legacy / no key). return Ok(stored.to_string()); } - let Some(k) = key() else { + let Some(k) = key()? else { return Err(anyhow!( "encrypted secret found but FIBER_SECRETS_KEY is not set" )); @@ -120,6 +154,56 @@ mod tests { const KEY_A: [u8; 32] = [7u8; 32]; const KEY_B: [u8; 32] = [9u8; 32]; + #[test] + fn no_key_is_allowed_and_means_plaintext() { + for raw in [None, Some(""), Some(" "), Some("\n")] { + assert_eq!(parse_key(raw), Ok(None), "{raw:?}"); + } + } + + #[test] + fn a_64_hex_key_is_accepted_in_either_case_and_trimmed() { + let lower = "07".repeat(32); + assert_eq!(parse_key(Some(&lower)), Ok(Some(KEY_A))); + assert_eq!(parse_key(Some(&format!(" {lower}\n"))), Ok(Some(KEY_A))); + assert_eq!(parse_key(Some(&"AB".repeat(32))), Ok(Some([0xab; 32]))); + } + + /// The fail-open this replaces: each of these was treated as "unset", and secrets + /// were then written in plaintext under a key the operator believed was in force. + #[test] + fn a_key_that_is_set_but_unusable_is_an_error_not_no_key() { + let hex64 = "07".repeat(32); + for (raw, want) in [ + (hex64[..63].to_string(), KeyError::WrongLength { chars: 63 }), + (hex64[..62].to_string(), KeyError::WrongLength { chars: 62 }), + (format!("{hex64}07"), KeyError::WrongLength { chars: 66 }), + ("abcd".to_string(), KeyError::WrongLength { chars: 4 }), + (format!("\"{hex64}\""), KeyError::NotHex { chars: 66 }), + ( + "correct horse battery staple".to_string(), + KeyError::NotHex { chars: 28 }, + ), + ( + format!("{}zz", &hex64[..62]), + KeyError::NotHex { chars: 64 }, + ), + ] { + assert_eq!(parse_key(Some(&raw)), Err(want), "{raw}"); + } + } + + #[test] + fn a_key_error_names_the_problem_and_never_the_value() { + let secretish = "s3cr3t-value-that-is-not-hex"; + for raw in [secretish.to_string(), "deadbeefcafe".to_string()] { + let msg = parse_key(Some(&raw)).unwrap_err().to_string(); + assert!(msg.contains("FIBER_SECRETS_KEY"), "{msg}"); + assert!(msg.contains("openssl rand -hex 32"), "{msg}"); + assert!(!msg.contains(&raw), "{msg}"); + } + } + /// Written by aes-gcm 0.10 with `KEY_A`. Secrets already in the database were /// encrypted by whatever version was current when they were written, so a crate /// upgrade that changed the nonce layout or the tag position has to fail here @@ -227,7 +311,7 @@ mod tests { #[test] fn an_encrypted_value_without_a_configured_key_is_an_error_not_a_leak() { // Never hand the raw ciphertext back to a step as if it were the secret. - if key().is_none() { + if matches!(key(), Ok(None)) { let sealed = encrypt_with(&KEY_A, "hunter2").unwrap(); let err = decrypt_secret(&sealed).unwrap_err().to_string(); assert!(err.contains("FIBER_SECRETS_KEY is not set"), "{err}"); @@ -236,18 +320,9 @@ mod tests { #[test] fn without_a_key_the_env_facing_api_stores_plaintext() { - if key().is_none() { + if matches!(key(), Ok(None)) { assert_eq!(encrypt_secret("hello").unwrap(), "hello"); assert_eq!(decrypt_secret("hello").unwrap(), "hello"); } } - - #[test] - fn load_key_accepts_exactly_thirty_two_bytes_of_hex() { - // `load_key` reads the env directly; exercise the parsing rules it applies by - // driving the same hex decode, since the OnceLock can only be set once. - assert_eq!(hex::decode("ab".repeat(32)).unwrap().len(), 32); - assert_ne!(hex::decode("ab".repeat(16)).unwrap().len(), 32); - assert!(hex::decode("zz".repeat(32)).is_err()); - } } diff --git a/crates/fiber-core/src/store.rs b/crates/fiber-core/src/store.rs index f61d94a..cd955c6 100644 --- a/crates/fiber-core/src/store.rs +++ b/crates/fiber-core/src/store.rs @@ -288,7 +288,7 @@ impl Store { pub async fn create_user(&self, username: &str, password: &str) -> Result { let id = Uuid::new_v4(); - let hash = crate::tokens::hash_password(password); + let hash = crate::tokens::hash_password_blocking(password).await?; Ok(sqlx::query_as::<_, PublicUser>( "INSERT INTO users (id, username, password_hash) VALUES ($1, $2, $3) RETURNING id, username, is_admin", @@ -2587,15 +2587,18 @@ impl Store { .bind(username) .fetch_optional(&self.pool) .await?; + // An unknown username still pays for a verify (against a dummy hash), so the + // response time does not say whether the account exists. + let stored = user.as_ref().map(|u| u.password_hash.as_str()); + if !crate::tokens::verify_password_blocking(password, stored).await? { + return Ok(None); + } let Some(user) = user else { return Ok(None); }; - if !crate::tokens::verify_password(password, &user.password_hash) { - return Ok(None); - } // Upgrade legacy SHA-256 password hashes to argon2id on successful login. if crate::tokens::password_needs_rehash(&user.password_hash) { - let new_hash = crate::tokens::hash_password(password); + let new_hash = crate::tokens::hash_password_blocking(password).await?; let _ = sqlx::query("UPDATE users SET password_hash = $2 WHERE id = $1") .bind(user.id) .bind(&new_hash) @@ -2627,6 +2630,20 @@ impl Store { })) } + /// Whether `username` exists and `password` is its password. Creates no session. + /// + /// For the boot check that refuses to serve a known login on a public address; it + /// asks the database rather than the configuration, because the configured admin + /// password is applied only on the first boot of an empty database. + pub async fn password_is(&self, username: &str, password: &str) -> Result { + let stored: Option = + sqlx::query_scalar("SELECT password_hash FROM users WHERE username = $1") + .bind(username) + .fetch_optional(&self.pool) + .await?; + crate::tokens::verify_password_blocking(password, stored.as_deref()).await + } + pub async fn user_by_session_token(&self, token: &str) -> Result> { let hash = crate::tokens::hash_token(token); Ok(sqlx::query_as::<_, PublicUser>( @@ -2661,10 +2678,10 @@ impl Store { let Some(stored) = stored else { return Ok(false); }; - if !crate::tokens::verify_password(current, &stored) { + if !crate::tokens::verify_password_blocking(current, Some(&stored)).await? { return Ok(false); } - let hash = crate::tokens::hash_password(new_password); + let hash = crate::tokens::hash_password_blocking(new_password).await?; let mut tx = self.pool.begin().await?; sqlx::query("UPDATE users SET password_hash = $2 WHERE id = $1") .bind(user_id) diff --git a/crates/fiber-core/src/tokens.rs b/crates/fiber-core/src/tokens.rs index 9e89ff7..3360805 100644 --- a/crates/fiber-core/src/tokens.rs +++ b/crates/fiber-core/src/tokens.rs @@ -54,14 +54,180 @@ pub fn verify_password(password: &str, stored: &str) -> bool { .verify_password(password.as_bytes(), &parsed) .is_ok(); } - // Legacy: `{salt_hex}${sha256(salt||password)}` + // Legacy: `{salt_hex}${sha256(salt||password)}`. One SHA-256 is microseconds where + // every other outcome is an argon2id verify, so an account still on a legacy hash + // answered measurably faster than a current one or an unknown name. Pay for the + // verify here too, and compare without stopping at the first differing byte. + burn_one_verify(password); let Some((salt_hex, expected)) = stored.split_once('$') else { return false; }; let mut hasher = Sha256::new(); hasher.update(salt_hex.as_bytes()); hasher.update(password.as_bytes()); - hex::encode(hasher.finalize()) == expected + constant_time_eq( + hex::encode(hasher.finalize()).as_bytes(), + expected.as_bytes(), + ) +} + +/// One argon2id verify whose answer is thrown away, so a path that has no argon2 hash to +/// check costs what one that has does. +fn burn_one_verify(password: &str) { + #[cfg(test)] + BURNED_VERIFIES.with(|n| n.set(n.get() + 1)); + if let Ok(parsed) = PasswordHash::new(dummy_password_hash()) { + let _ = Argon2::default().verify_password(password.as_bytes(), &parsed); + } +} + +#[cfg(test)] +thread_local! { + /// How many times this thread has paid for a throwaway verify. + static BURNED_VERIFIES: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + +/// Byte equality that takes the same time wherever the first difference is. Length is +/// not hidden: the values compared here (hex digests, HMAC signatures) have public lengths. +pub fn constant_time_eq(a: &[u8], b: &[u8]) -> bool { + if a.len() != b.len() { + return false; + } + a.iter() + .zip(b.iter()) + .fold(0u8, |acc, (x, y)| acc | (x ^ y)) + == 0 +} + +/// argon2id runs at once, process-wide, across every caller of the `*_blocking` helpers. +/// +/// Each is tens of milliseconds of CPU and 19 MiB by design. `spawn_blocking` alone +/// bounds nothing: tokio's blocking pool grows to 512 threads, so a flood of logins — +/// which need no credentials — was 512 × 19 MiB and every core. Four keeps the worst +/// case under 80 MiB and leaves cores for the scheduler. +pub const MAX_CONCURRENT_HASHES: usize = 4; +/// How long a caller waits for one of those slots before giving up with [`HashBusy`]. +/// Slots are handed out in arrival order, so a real login behind a flood is served in +/// turn rather than refused; this bounds how long "in turn" may be. +pub const HASH_SLOT_WAIT: std::time::Duration = std::time::Duration::from_secs(5); +/// The most callers that may be waiting for a slot. A backstop on memory, set so that +/// it refuses nobody who could have been served: with `MAX_CONCURRENT_HASHES` slots and +/// a verify that takes 25 ms or more, at most 800 waiters can reach a slot inside +/// `HASH_SLOT_WAIT`. Anyone further back would have timed out anyway. +pub const MAX_WAITING_HASHES: usize = 1024; + +/// No password-hashing slot came free in time. The server is busy, not the caller wrong: +/// handlers answer `503`, never the `401`/`429` that a bad password or a lockout get. +#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)] +#[error("password hashing is saturated; no slot came free in time")] +pub struct HashBusy; + +/// Bounds how many blocking jobs run at once, and how many callers wait for a turn. +struct HashGate { + slots: std::sync::Arc, + waiting: std::sync::atomic::AtomicUsize, + max_waiting: usize, + wait: std::time::Duration, +} + +impl HashGate { + fn new(slots: usize, max_waiting: usize, wait: std::time::Duration) -> Self { + Self { + slots: std::sync::Arc::new(tokio::sync::Semaphore::new(slots)), + waiting: std::sync::atomic::AtomicUsize::new(0), + max_waiting, + wait, + } + } + + /// Run `f` on the blocking pool once a slot is free. + /// + /// The permit moves **into** the blocking task and is released when `f` returns — + /// not when this future is dropped. A request that is cancelled (client gone, the + /// 30 s request timeout) stops waiting for the answer, but the work it started is + /// still running, and a permit released at cancellation would let the next request + /// start more on top of it. + async fn run(&self, f: F) -> anyhow::Result + where + F: FnOnce() -> T + Send + 'static, + T: Send + 'static, + { + use std::sync::atomic::Ordering; + /// Leaves the queue however the wait ends, cancellation included. + struct Waiting<'a>(&'a std::sync::atomic::AtomicUsize); + impl Drop for Waiting<'_> { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::SeqCst); + } + } + + if self.waiting.fetch_add(1, Ordering::SeqCst) >= self.max_waiting { + self.waiting.fetch_sub(1, Ordering::SeqCst); + return Err(HashBusy.into()); + } + let waiting = Waiting(&self.waiting); + let permit = tokio::time::timeout(self.wait, self.slots.clone().acquire_owned()) + .await + .map_err(|_| HashBusy)? + .map_err(|_| HashBusy)?; + drop(waiting); + Ok(tokio::task::spawn_blocking(move || { + let _permit = permit; + f() + }) + .await?) + } +} + +fn hash_gate() -> &'static HashGate { + static GATE: std::sync::OnceLock = std::sync::OnceLock::new(); + GATE.get_or_init(|| HashGate::new(MAX_CONCURRENT_HASHES, MAX_WAITING_HASHES, HASH_SLOT_WAIT)) +} + +/// [`hash_password`] off the async workers, and inside [`MAX_CONCURRENT_HASHES`]. +/// +/// argon2id is tens of milliseconds of CPU and 19 MiB by design. Run inline in a handler +/// it parks a tokio worker for that long, so a burst of logins stalls everything else the +/// runtime is driving — agent heartbeats and lease renewals included. +/// +/// Fails with [`HashBusy`] (downcast it) when no slot came free within [`HASH_SLOT_WAIT`]. +pub async fn hash_password_blocking(password: &str) -> anyhow::Result { + let password = password.to_owned(); + hash_gate().run(move || hash_password(&password)).await +} + +/// [`verify_password`] off the async workers and inside [`MAX_CONCURRENT_HASHES`], at +/// the same cost whether or not there is a stored hash to verify against. +/// +/// `stored` is `None` when no such user exists. Answering that case without doing the +/// work made "unknown username" return in microseconds and "wrong password" in tens of +/// milliseconds — an oracle for which usernames exist. The password is checked against +/// [`dummy_password_hash`] instead and the answer is always `false`. +/// +/// Fails with [`HashBusy`] (downcast it) when no slot came free within [`HASH_SLOT_WAIT`]. +pub async fn verify_password_blocking( + password: &str, + stored: Option<&str>, +) -> anyhow::Result { + let password = password.to_owned(); + let stored = stored.map(str::to_owned); + hash_gate() + .run(move || match stored { + Some(stored) => verify_password(&password, &stored), + None => { + burn_one_verify(&password); + false + } + }) + .await +} + +/// An argon2id hash of a random password nobody knows, made once per process with the +/// same parameters as [`hash_password`], so verifying against it costs what verifying a +/// real one does. +pub fn dummy_password_hash() -> &'static str { + static DUMMY: std::sync::OnceLock = std::sync::OnceLock::new(); + DUMMY.get_or_init(|| hash_password(&generate_password())) } pub fn password_needs_rehash(stored: &str) -> bool { @@ -135,6 +301,191 @@ mod tests { assert!(!verify_password("wrong", &h)); } + /// Everything in a PHC string before the salt: algorithm, version, cost parameters. + fn phc_params(hash: &str) -> &str { + let hash_at = hash.rfind('$').unwrap(); + &hash[..hash[..hash_at].rfind('$').unwrap()] + } + + #[test] + fn the_dummy_hash_costs_what_a_real_one_does() { + let dummy = dummy_password_hash(); + assert!(phc_params(dummy).starts_with("$argon2id$")); + assert_eq!(phc_params(dummy), phc_params(&hash_password("fiber"))); + assert_eq!(dummy, dummy_password_hash(), "computed once"); + } + + #[tokio::test] + async fn blocking_verify_matches_the_inline_one_and_never_accepts_a_missing_user() { + let h = hash_password_blocking("fiber").await.unwrap(); + assert!(verify_password_blocking("fiber", Some(&h)).await.unwrap()); + assert!(!verify_password_blocking("wrong", Some(&h)).await.unwrap()); + assert!(!verify_password_blocking("fiber", None).await.unwrap()); + assert!(!verify_password_blocking("", None).await.unwrap()); + } + + fn legacy_hash(password: &str) -> String { + let salt_hex = hex::encode([1u8; 16]); + let mut hasher = Sha256::new(); + hasher.update(salt_hex.as_bytes()); + hasher.update(password.as_bytes()); + format!("{salt_hex}${}", hex::encode(hasher.finalize())) + } + + /// A legacy hash is one SHA-256. Unless it also pays for an argon2id verify, an + /// account still on one answers in microseconds and stands out from every other. + #[test] + fn a_legacy_hash_pays_for_an_argon2_verify_too() { + let burned = || BURNED_VERIFIES.with(|n| n.get()); + let stored = legacy_hash("fiber"); + let before = burned(); + assert!(verify_password("fiber", &stored)); + assert!(!verify_password("wrong", &stored)); + assert_eq!( + burned() - before, + 2, + "one per legacy verify, right or wrong" + ); + // A current hash is already an argon2id verify and pays once, not twice. + let before = burned(); + assert!(verify_password("fiber", &hash_password("fiber"))); + assert_eq!(burned(), before); + } + + #[test] + fn constant_time_eq_is_plain_equality() { + assert!(constant_time_eq(b"abc", b"abc")); + assert!(!constant_time_eq(b"abc", b"abd")); + assert!(!constant_time_eq(b"abc", b"ab")); + assert!(!constant_time_eq(b"xbc", b"abc")); + assert!(constant_time_eq(b"", b"")); + } + + fn is_busy(e: &anyhow::Error) -> bool { + e.downcast_ref::().is_some() + } + + async fn until(what: &str, mut done: impl FnMut() -> bool) { + for _ in 0..400 { + if done() { + return; + } + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + } + panic!("timed out waiting for {what}"); + } + + /// The hole in a permit held by the caller: a client that sends a login and hangs up + /// frees the slot while argon2 is still running, so the next request starts another. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn a_slot_is_held_until_the_work_ends_even_if_the_caller_is_dropped() { + use std::sync::{Arc, mpsc}; + let gate = Arc::new(HashGate::new(1, 8, std::time::Duration::from_millis(30))); + let (started_tx, started_rx) = mpsc::channel::<()>(); + let (release_tx, release_rx) = mpsc::channel::<()>(); + let caller = tokio::spawn({ + let gate = Arc::clone(&gate); + async move { + gate.run(move || { + started_tx.send(()).unwrap(); + // Stands in for argon2: runs until told to stop. + let _ = release_rx.recv(); + }) + .await + } + }); + tokio::task::spawn_blocking(move || started_rx.recv().unwrap()) + .await + .unwrap(); + // The request goes away mid-verify. + caller.abort(); + assert!(caller.await.unwrap_err().is_cancelled()); + + assert_eq!( + gate.slots.available_permits(), + 0, + "the work is still running" + ); + let err = gate.run(|| ()).await.unwrap_err(); + assert!(is_busy(&err), "{err:#}"); + + release_tx.send(()).unwrap(); + until("the slot to come back", || { + gate.slots.available_permits() == 1 + }) + .await; + gate.run(|| ()).await.unwrap(); + assert_eq!(gate.waiting.load(std::sync::atomic::Ordering::SeqCst), 0); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + async fn no_more_than_the_limit_run_at_once() { + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + const LIMIT: usize = 3; + let gate = Arc::new(HashGate::new( + LIMIT, + 100, + std::time::Duration::from_secs(30), + )); + let running = Arc::new(AtomicUsize::new(0)); + let peak = Arc::new(AtomicUsize::new(0)); + let jobs: Vec<_> = (0..24) + .map(|_| { + let (gate, running, peak) = (gate.clone(), running.clone(), peak.clone()); + tokio::spawn(async move { + gate.run(move || { + let now = running.fetch_add(1, Ordering::SeqCst) + 1; + peak.fetch_max(now, Ordering::SeqCst); + std::thread::sleep(std::time::Duration::from_millis(5)); + running.fetch_sub(1, Ordering::SeqCst); + }) + .await + }) + }) + .collect(); + for job in jobs { + job.await.unwrap().unwrap(); + } + assert_eq!(peak.load(Ordering::SeqCst), LIMIT, "reached, never passed"); + assert_eq!(gate.slots.available_permits(), LIMIT); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn a_waiter_gives_up_after_the_wait_and_the_queue_itself_is_capped() { + use std::sync::atomic::Ordering; + use std::sync::{Arc, mpsc}; + let gate = Arc::new(HashGate::new(1, 1, std::time::Duration::from_millis(100))); + let (release_tx, release_rx) = mpsc::channel::<()>(); + let holder = tokio::spawn({ + let gate = Arc::clone(&gate); + async move { gate.run(move || release_rx.recv().unwrap()).await } + }); + until("the slot to be taken", || { + gate.slots.available_permits() == 0 + }) + .await; + + let waiter = tokio::spawn({ + let gate = Arc::clone(&gate); + async move { gate.run(|| ()).await } + }); + until("the waiter to queue", || { + gate.waiting.load(Ordering::SeqCst) == 1 + }) + .await; + // The queue is full: refused at once, not after the wait. + let started = std::time::Instant::now(); + assert!(is_busy(&gate.run(|| ()).await.unwrap_err())); + assert!(started.elapsed() < std::time::Duration::from_millis(100)); + // The one in the queue waits its turn out, then gives up and leaves the queue. + assert!(is_busy(&waiter.await.unwrap().unwrap_err())); + assert_eq!(gate.waiting.load(Ordering::SeqCst), 0); + + release_tx.send(()).unwrap(); + holder.await.unwrap().unwrap(); + } + #[test] fn legacy_sha_still_verifies() { let mut salt = [0u8; 16]; diff --git a/crates/fiber-scheduler/src/lib.rs b/crates/fiber-scheduler/src/lib.rs index 7fd8d01..3b4ee83 100644 --- a/crates/fiber-scheduler/src/lib.rs +++ b/crates/fiber-scheduler/src/lib.rs @@ -127,6 +127,15 @@ impl AgentPresence { pub fn release_slot(&mut self) { self.reserved = self.reserved.saturating_sub(1); } + + /// Cut the registration down to a new ceiling: keep only the labels in + /// `allowed_labels`, and no more than `max_concurrency` slots (at least 1). Never + /// adds a label or a slot — what is registered is what the agent claimed, capped by + /// its row, and this only knows the row. + pub fn narrow_to(&mut self, allowed_labels: &[String], max_concurrency: u32) { + self.labels.retain(|l| allowed_labels.contains(l)); + self.concurrency = self.concurrency.min(max_concurrency.max(1)); + } } /// Envelope on `fiber:agent_cmds`. @@ -394,6 +403,19 @@ impl Scheduler { } } + /// Apply an edited agent row to a live agent without widening it; see + /// [`AgentPresence::narrow_to`]. A no-op for an agent that is not connected here. + pub async fn narrow_agent_presence( + &self, + agent_id: Uuid, + allowed_labels: &[String], + max_concurrency: u32, + ) { + if let Some(a) = self.agents.write().await.get_mut(&agent_id) { + a.narrow_to(allowed_labels, max_concurrency); + } + } + pub async fn register_connection( &self, agent_id: Uuid, @@ -1423,6 +1445,38 @@ mod tests { } } + // --- editing a live agent ----------------------------------------------------------- + + /// `PUT /api/agents/{id}` used to install the row's labels and concurrency verbatim, + /// so widening a row gave a live agent labels it never claimed. + #[test] + fn an_edited_row_narrows_a_live_agent_and_never_widens_it() { + let l = |csv: &str| -> Vec { + csv.split(',') + .filter(|s| !s.is_empty()) + .map(str::to_string) + .collect() + }; + let mut a = presence(2, 1); + a.labels = l("os=linux,docker=true"); + + // Widened row: more labels, more slots. Nothing the agent did not claim appears. + a.narrow_to(&l("os=linux,docker=true,gpu=true,pool=release"), 16); + assert_eq!(a.labels, l("os=linux,docker=true")); + assert_eq!(a.concurrency, 2); + + // Narrowed row: takes effect at once. + a.narrow_to(&l("os=linux"), 1); + assert_eq!(a.labels, l("os=linux")); + assert_eq!(a.concurrency, 1); + assert_eq!(a.reserved, 1, "reservations in flight are not dropped"); + + // Cleared labels restrict to label-less steps; a zero is still one slot. + a.narrow_to(&[], 0); + assert!(a.labels.is_empty()); + assert_eq!(a.concurrency, 1); + } + // --- concurrency slots ------------------------------------------------------------ #[test] diff --git a/deploy/.env.example b/deploy/.env.example index 7413fe9..c66c54b 100644 --- a/deploy/.env.example +++ b/deploy/.env.example @@ -24,14 +24,17 @@ FIBER_S3_ACCESS_KEY= FIBER_S3_SECRET_KEY= # Bootstrap admin password. Applied only on the first boot of an empty database; after -# that, change it in the UI or through the API. The API warns at boot while it is the -# old default `fiber`. +# that, change it in the UI or through the API. fiber-api refuses to start while the +# admin can still log in with the old default `fiber` — change that password +# (`fiber-api reset-password admin`), or set FIBER_ALLOW_DEFAULT_ADMIN=1 to start anyway. FIBER_ADMIN_PASSWORD= FIBER_ADMIN_USER=admin +# FIBER_ALLOW_DEFAULT_ADMIN=1 # ------------------------------------------------------------------- secrets at rest -# 64 hex chars: openssl rand -hex 32 — keep a backup, there is no recovery path. -# Empty = project and webhook secrets are stored in plaintext and the API warns at boot. +# Required. 64 hex chars: openssl rand -hex 32 — keep a backup, there is no recovery +# path. Compose refuses to start without it, and fiber-api refuses a value that is not +# 64 hex characters rather than storing secrets in plaintext. FIBER_SECRETS_KEY= # ---------------------------------------------------------------------------- network @@ -60,12 +63,10 @@ FIBER_S3_REGION=us-east-1 FIBER_S3_PUBLIC_ENDPOINT=http://127.0.0.1:19000 # ---------------------------------------------------------------------------- GitHub -# Instance-wide fallback for the PR file API and commit statuses. A project secret of -# the same name wins over it. It is instance-wide in the literal sense: any project's -# webhook can reach it, so a project admin who controls their own webhook secret can -# have Fiber spend this token on their behalf. Scope it as narrowly as the two uses -# allow — `repo:status` for statuses, read-only repository contents for the PR file -# list — and prefer a per-project secret for anything private. +# No longer used. This was an instance-wide fallback for the PR file API, which any +# project's webhook could reach. GitHub tokens are per project now: add a project secret +# named GITHUB_TOKEN (Settings -> Secrets, or `fiber secrets set`). fiber-api logs one +# warning if this is set and a project needed it. Leave empty. FIBER_GITHUB_TOKEN= # GitHub Enterprise API base; empty = https://api.github.com FIBER_GITHUB_API_URL= diff --git a/deploy/Dockerfile b/deploy/Dockerfile index bec6617..c8af9b6 100644 --- a/deploy/Dockerfile +++ b/deploy/Dockerfile @@ -40,7 +40,12 @@ RUN useradd --create-home --uid 10001 fiber \ && chown -R 10001:10001 /data/artifacts USER 10001 WORKDIR /home/fiber -ENV FIBER_ARTIFACTS_DIR=/data/artifacts +# The binary's own default is loopback, which inside a container reaches nobody; this +# matches EXPOSE and the HEALTHCHECK below. On this address fiber-api refuses to start +# while the admin password is the default, so a bare `docker run` needs +# -e FIBER_ADMIN_PASSWORD=... +ENV FIBER_ARTIFACTS_DIR=/data/artifacts \ + FIBER_LISTEN=0.0.0.0:8080 EXPOSE 8080 # /ready, not /health: it is the probe that means "Postgres answers and the background # loops are running". Compose overrides the timings; this is for a bare `docker run`. diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index 4b44849..6670e08 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -2,11 +2,12 @@ name: fiber # Backing services bind to loopback only; put fiber-api / fiber-ui behind TLS # (docs/operations.md#deployment). Settings come from deploy/.env — see deploy/.env.example. # -# Four credentials are required, not defaulted: FIBER_POSTGRES_PASSWORD, -# FIBER_REDIS_PASSWORD, FIBER_S3_ACCESS_KEY/FIBER_S3_SECRET_KEY and -# FIBER_ADMIN_PASSWORD. `docker compose config` fails until deploy/.env sets them. A -# default that works is a default nobody changes, and FIBER_API_BIND=0.0.0.0 plus a -# published admin password is a public instance with a known login. +# Four credentials and the secrets key are required, not defaulted: +# FIBER_POSTGRES_PASSWORD, FIBER_REDIS_PASSWORD, FIBER_S3_ACCESS_KEY/FIBER_S3_SECRET_KEY, +# FIBER_ADMIN_PASSWORD and FIBER_SECRETS_KEY. `docker compose config` fails until +# deploy/.env sets them. A default that works is a default nobody changes, and +# FIBER_API_BIND=0.0.0.0 plus a published admin password is a public instance with a +# known login; an empty secrets key is every project secret in plaintext. # # Networks: everything shares `backend`, except the agent, which sits alone with the API # on `agentnet`. Pipeline steps are repo-supplied shell, so an agent that could reach @@ -183,19 +184,23 @@ services: # Applied only on the first boot of an empty database; change it later through the # UI or the API. FIBER_ADMIN_PASSWORD: ${FIBER_ADMIN_PASSWORD:?set it in deploy/.env (cp deploy/.env.example deploy/.env)} + # fiber-api listens on 0.0.0.0 in here, so it refuses to start while the admin can + # still log in with the old default password `fiber`. 1 = start anyway. + FIBER_ALLOW_DEFAULT_ADMIN: ${FIBER_ALLOW_DEFAULT_ADMIN:-} # Showcase demo project: auto = only on the first boot of an empty database, so a # deleted demo project stays deleted. 1 = seed on this boot, 0 = never. FIBER_SEED_SHOWCASE: ${FIBER_SEED_SHOWCASE:-auto} # Most expanded steps (matrix cells included) one pipeline may compile to. FIBER_MAX_STEPS: ${FIBER_MAX_STEPS:-} - # Required for encrypted project/webhook secrets: openssl rand -hex 32 → deploy/.env. - # Empty = plaintext at rest (fiber-api warns at boot). - FIBER_SECRETS_KEY: ${FIBER_SECRETS_KEY:-} + # Encrypts project and webhook secrets at rest. Required: a deployment does not get + # plaintext secrets by leaving a line empty. 64 hex characters; anything else set + # here stops fiber-api at boot. + FIBER_SECRETS_KEY: ${FIBER_SECRETS_KEY:?generate one with `openssl rand -hex 32` and set it in deploy/.env (keep a backup)} FIBER_CORS_ORIGINS: ${FIBER_CORS_ORIGINS:-http://localhost:3100,http://127.0.0.1:3100} # Public base URL of this instance; commit statuses link back to it. FIBER_PUBLIC_URL: ${FIBER_PUBLIC_URL:-} - # Instance-wide fallback for the PR file API and commit statuses. A project secret - # of the same name wins over it. + # No longer used: GitHub tokens are per-project secrets now. Still passed through so + # fiber-api can warn when one is set here and a project needed it. FIBER_GITHUB_TOKEN: ${FIBER_GITHUB_TOKEN:-} FIBER_GITHUB_API_URL: ${FIBER_GITHUB_API_URL:-} # Retention. Empty falls back to the documented default (30 / 20 / 100 / 3600). diff --git a/docs/agents.md b/docs/agents.md index b68f584..f471078 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -2,7 +2,7 @@ Agents are outbound WebSocket workers that execute CI steps. -An agent's identity is the token it connected with. The server binds `agent_id` from that token and ignores the `agent_id` carried in messages (a mismatch is logged). Log lines are accepted only for a step **last leased to that agent** (so the tail of a cancelled or reclaimed step is still recorded); artifacts and completions additionally require the lease to be **live** — anything after a reclaim or cancel is dropped, which is the at-least-once contract: the re-leased attempt reports its own result. Artifact restore downloads are limited to runs in which the agent holds a running step. A socket that never sends `Hello` is offered nothing, and pool scope always comes from the agent's database row, never from the connection. +An agent's identity is the token it connected with. The server binds `agent_id` from that token and ignores the `agent_id` carried in messages (a mismatch is logged). Log lines are accepted only for a step **last leased to that agent** (so the tail of a cancelled or reclaimed step is still recorded); artifacts and completions additionally require the lease to be **live** — anything after a reclaim or cancel is dropped, which is the at-least-once contract: the re-leased attempt reports its own result. Artifact restore downloads are limited to runs in which the agent holds a running step. A socket that never sends `Hello` is offered nothing, and pool scope always comes from the agent's database row, never from the connection. Labels and concurrency are capped by the row too — see [Labels and concurrency](#labels-and-concurrency-the-registration-is-the-ceiling). A lease belongs to the agent, not to the socket. A step keeps running through a lost connection — an API restart, a proxy timeout, a network blip — and the agent renews its leases on its first heartbeat back; the server judges every late message by the row (is the step still `running` under this agent, on the attempt the message names?), never by which session it arrived on. Every `Offer` carries the attempt number and the agent echoes it on each log batch, artifact, and completion — and on the HTTP artifact routes, as `X-Fiber-Attempt` on the upload and an `attempt` field in the presign and complete bodies ([api](./api.md#agent-endpoints)) — so output an agent held through a reclaim can never be filed under, overwrite, or close the attempt that replaced it; a message whose attempt is not the row's is dropped with a warning, and an upload for one is refused with `401`. A log line the agent re-sends after an aborted write can appear twice: `log_lines` is append-only with no dedupe. See [Lifecycle](#lifecycle) for the limits. @@ -318,6 +318,35 @@ cargo run -p fiber-agent Connects to `/ws/agent?token=…`, sends `Hello`, then heartbeats every **10s**. Pool scope comes from the token's agent row (not from the client). A `concurrency` below 1 is treated as 1 — an agent that is online but can never be offered anything is a misconfiguration nobody would notice. +### Labels and concurrency: the registration is the ceiling + +The agent's row (what you set at **Register**, or later with `PUT /api/agents/{id}` / +`fiber agents update`) says what the agent **may** do. `--labels` / `--concurrency` on the +agent (`FIBER_AGENT_LABELS`, `FIBER_AGENT_CONCURRENCY`) say what this host **can** do. The +agent is registered with what both agree on: + +| | Registered with | +|---|---| +| Labels | The row's labels that the agent also sent. A label the agent claims but the row lacks is dropped | +| Concurrency | The smaller of the row's and the agent's (at least 1) | + +So to give an agent a new label or more slots, change **both**: the row, and the agent's +flags. Changing only the agent's flags does nothing, and the API logs a warning naming the +agent id each time it connects asking for more than its row has +(`agent hello asked for more than its registration allows`). An agent that claims *less* +than its row is fine and is offered only what it claimed. + +A row with **no labels** (`"labels": []`, `--labels ""`) grants none: whatever the agent +claims, it is offered only steps that require no labels. Clearing an agent's labels is +therefore a way to restrict it, not to open it up. + +Editing the row of a connected agent takes effect without a reconnect. Removing a label +or lowering concurrency applies at once on the API replica that handled the edit, and +within one heartbeat (10 s) on any other. Adding a label or raising concurrency applies +on the agent's next heartbeat, and only as far as the agent's own flags go. + +The UI and `GET /api/agents` show the row, not the effective registration. + ## Label matching A step is offered only if: @@ -355,7 +384,7 @@ The server counts an agent's in-flight steps **from the database** (`step_runs` | Server shutdown | On SIGTERM the API sends Close 1012 ("server shutting down") to every agent and waits for the sessions to end (up to 20 s) before exiting; the agent reconnects with its usual backoff and its in-flight steps follow the Disconnect and Reconnect rows — they keep running | | Older agent | An agent whose `Hello` has no `protocol_version` cancels its steps on any close, so for it a close is still the end of the attempt: the server requeues its steps at once on disconnect, exactly as before | | Token rotate | `POST /api/agents/{id}/rotate-token` — new token once; force-disconnect; old session cannot keep leasing | -| Update | `PUT /api/agents/{id}` — name / labels / concurrency (pool unchanged) | +| Update | `PUT /api/agents/{id}` — name / labels / concurrency (pool unchanged). A connected agent is re-registered as *row ∩ its own flags*: narrowing at once, widening on its next heartbeat | | Delete | `DELETE /api/agents/{id}` — disconnect cleanup then delete | ## Log path diff --git a/docs/authz.md b/docs/authz.md index 76f02b5..eff6ea2 100644 --- a/docs/authz.md +++ b/docs/authz.md @@ -6,12 +6,21 @@ - `Authorization: Bearer ` on API calls - `POST /api/auth/logout` invalidates the session - Passwords: **argon2id** (legacy SHA-256 hashes verify and upgrade on login) -- Brute-force guard: 10 failed logins for a username within 10 minutes → that username is locked for 60 s (`429` + `Retry-After`; ~10 guesses/minute sustained). Keyed by username, so a known username can be held locked by an attacker — accepted over IP keying, which is spoofable behind a proxy -- Default bootstrap user: `FIBER_ADMIN_USER` / `FIBER_ADMIN_PASSWORD` (admin / fiber) — also the first **instance admin** +- Brute-force guard: 10 failed logins for a username within 10 minutes → that username is locked for 60 s (`429` + `Retry-After`). An attempt is reserved **before** the password is verified, and attempts still being verified count against the same 10, so parallel requests cannot outrun the lockout: at most 10 guesses per username before it locks, then at most 10 per 60 s. Keyed by username, so a known username can be held locked by an attacker — accepted over IP keying, which is spoofable behind a proxy +- Password hashing is bounded process-wide: at most **4** argon2id hashes or verifies run at once, across login, password change, and user creation. They run off the async workers, so a login flood cannot stall agent heartbeats, and a slot is held until the hash finishes even if the request that started it has gone. A request that finds all four busy waits its turn (first come, first served) for up to 5 s, then gets `503` with `Retry-After` and "server busy; nothing was checked" — never the `429` of a lockout, and never counted as a failed login. So a flood of made-up usernames slows logins down rather than refusing them: a real login waits roughly *(requests queued ahead ÷ 4) × one verify* (tens of milliseconds each). A flood that sustains more verifies than four slots can clear will push that past 5 s; rate-limit `/api/auth/login` per client at the reverse proxy to close that, since the API does not trust a forwarded client address +- Every wrong login costs one argon2id verify — unknown username, wrong password, or an account still on a legacy SHA-256 hash — so response time does not reveal which usernames exist +- The guard is in memory and per process: a restart clears it, and each `fiber-api` replica counts separately. It tracks up to 10,000 usernames; past that it drops expired entries, then those with the fewest failures — a flood of made-up usernames evicts itself before a real account's count +- Default bootstrap user: `FIBER_ADMIN_USER` / `FIBER_ADMIN_PASSWORD` (admin / fiber) — also the first **instance admin**. `fiber-api` listens on loopback by default and refuses to start on any other address while that default password still works, unless `FIBER_ALLOW_DEFAULT_ADMIN=1` WebSocket `/ws/runs/{id}` takes the session token in the `Sec-WebSocket-Protocol` header as `fiber.token.` (never the query string) and requires **reader** on the run’s project. Agent auth uses a separate agent token (`/ws/agent?token=` or Bearer on agent HTTP routes). +The token decides **which** agent a connection is; the agent's database row decides what +that agent may be offered. Pool scope comes from the row alone. Labels and concurrency are +capped by the row: a connecting agent is registered with the row's labels that it also +claims and the smaller of the two concurrencies, so holding a token does not let its +holder claim labels or slots the agent was not registered with. A row with no labels +grants none. See [agents](./agents.md#labels-and-concurrency-the-registration-is-the-ceiling). ## Roles @@ -68,7 +77,7 @@ CLI: `fiber members …` — [cli.md](./cli.md). Project secrets (`POST/GET/DELETE /api/projects/{id}/secrets`) inject as env vars, except into runs built from a fork's pull request, which get none — see [triggers](./triggers.md). By default every step of the project receives all of them; a step can narrow that with `secrets:` in the pipeline (see [pipeline-yaml](./pipeline-yaml.md)). Agents mask secret values in log lines and pass them to containers through a private env-file rather than the command line. -- At rest: AES-GCM when `FIBER_SECRETS_KEY` is 64 hex chars; otherwise plaintext (dev warning) +- At rest: AES-GCM when `FIBER_SECRETS_KEY` is 64 hex chars; plaintext when it is unset (dev warning). A key that is set but invalid stops the API at boot — it is never treated as unset - Admins+ only to list/mutate - Special: `GITHUB_TOKEN` / `FIBER_GITHUB_TOKEN` used for PR file listing — see [Triggers](./triggers.md) diff --git a/docs/configuration.md b/docs/configuration.md index 301e184..9f9706d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -8,14 +8,15 @@ All product env vars use the `FIBER_*` prefix (plus standard OTEL names). |---|---|---| | `FIBER_DATABASE_URL` | `postgres://fiber:fiber@localhost:15432/fiber` | Postgres | | `FIBER_REDIS_URL` | `redis://localhost:16379` | Redis (events + coordination). The Compose Redis requires a password (`FIBER_REDIS_PASSWORD`, no default — `make infra` passes `fiber` on a dev box): `redis://:fiber@127.0.0.1:16379`, which is what `scripts/dev-env.sh` builds | -| `FIBER_LISTEN` | `0.0.0.0:18080` | Bind address | +| `FIBER_LISTEN` | `127.0.0.1:18080` | Bind address. Loopback by default, so a bare `fiber-api` with its bootstrap login is reachable only from the same machine; the image sets `0.0.0.0:8080`. Set it explicitly to serve other hosts (an agent on another machine, a reverse proxy in another container) | | `FIBER_ARTIFACTS_DIR` | `./data/artifacts` | Local artifact root | | `FIBER_ADMIN_USER` | `admin` | Bootstrap admin username | -| `FIBER_ADMIN_PASSWORD` | `fiber` | Bootstrap admin password (API warns at boot while it is the default) | +| `FIBER_ADMIN_PASSWORD` | `fiber` | Bootstrap admin password, applied only on the first boot of an empty database. While the bootstrap admin can log in with `fiber`, the API warns at boot on a loopback `FIBER_LISTEN` and **refuses to start** on any other address. Both `FIBER_ADMIN_USER` and the literal `admin` are checked, so renaming the variable does not hide an old `admin` / `fiber` account | +| `FIBER_ALLOW_DEFAULT_ADMIN` | unset | `1` (or `true`) starts on a non-loopback `FIBER_LISTEN` even though the bootstrap admin still has the default password. For a throwaway stack such as the Compose smoke; the fix for a real one is to change the password (`fiber-api reset-password admin`) | | `FIBER_MAX_STEPS` | `500` | Most expanded steps (matrix cells included) one pipeline may compile to; past it the definition is rejected with `TooManySteps`. Raise it for a generated fan-out. It bounds the **number** of cells only — a cell is not a fixed size, so a separate, non-configurable 8 MiB budget bounds the bytes a definition expands to, and the three routes that accept a definition cap the request body at 256 KiB. `0` or a non-number fails the boot. **`fiber validate` reads it too**, from the developer's shell: a server raised to 2000 and a developer left unset means a pipeline that fails locally and is accepted on push | | `FIBER_SEED_SHOWCASE` | `auto` | Seed the **Showcase** demo project and its pipelines. `auto` = only when the users table was empty at boot (a fresh instance), so a demo project you delete stays deleted; `1` = seed on this boot (restores it, and picks up pipelines a newer version added); `0` = never. Any other value fails the boot | | `FIBER_CORS_ORIGINS` | `http://127.0.0.1:3100,http://localhost:3100` | Comma-separated browser origins allowed to call the API; `*` = any origin | -| `FIBER_SECRETS_KEY` | unset | 64 hex chars → encrypt project **and webhook** secrets at rest | +| `FIBER_SECRETS_KEY` | unset | 64 hex chars → encrypt project **and webhook** secrets at rest. Unset (or empty) = plaintext, with a warning at boot. Set to anything that is not 64 hex characters = **the API refuses to start** and says why — it does not fall back to plaintext | | `FIBER_S3_BUCKET` | unset | If set, use the S3 backend | | `FIBER_S3_ENDPOINT` | `http://127.0.0.1:19000` | S3 API endpoint | | `FIBER_S3_PUBLIC_ENDPOINT` | = endpoint | Presign host agents can reach | @@ -29,7 +30,7 @@ All product env vars use the `FIBER_*` prefix (plus standard OTEL names). | `FIBER_AGENT_STALE_SECS` | `45` | Offline threshold for heartbeats | | `FIBER_STEP_TIMEOUT_DEFAULT_MINUTES` | `60` | Per-attempt limit for steps without `timeout_minutes` | | `FIBER_STEP_TIMEOUT_GRACE_MINUTES` | `5` | Extra minutes the server waits past a step's limit before failing it itself (backstop for hung/old agents) | -| `FIBER_GITHUB_TOKEN` | unset | Fallback for the PR file API and commit statuses (needs `repo:status`) | +| `FIBER_GITHUB_TOKEN` | unset | **No longer used.** Was an instance-wide fallback for the PR file API; set a project secret `GITHUB_TOKEN` instead ([triggers](./triggers.md)). A warning is logged once if it is set and a project needed it | | `FIBER_PUBLIC_URL` | unset | Public base URL of this Fiber; used to link commit statuses to the run page | | `FIBER_GITHUB_API_URL` | `https://api.github.com` | GitHub / GHE API base | | `OTEL_EXPORTER_OTLP_ENDPOINT` | unset | Collector **base** URL; `/v1/traces` and `/v1/metrics` are appended | @@ -61,14 +62,15 @@ file passes the variable through as an empty string and each reader falls back. | `FIBER_POSTGRES_PASSWORD` | Postgres password (loopback-bound port `15432`); applied when the volume is first initialised, so changing it later also needs `ALTER ROLE` | | `FIBER_REDIS_PASSWORD` | Redis `requirepass` (loopback-bound port `16379`), delivered through a Compose `configs:` file, not argv | | `FIBER_S3_ACCESS_KEY` / `FIBER_S3_SECRET_KEY` | Root credentials of the bundled S3 store (RustFS) and what `fiber-api` signs S3 requests with. Required even with the `s3` profile off: Compose interpolates the whole file before it filters by profile. Any placeholder does when `FIBER_S3_BUCKET` is empty | -| `FIBER_ADMIN_PASSWORD` | Bootstrap admin password — applied only on the first boot of an empty database; rotate an existing admin through the API/UI | +| `FIBER_ADMIN_PASSWORD` | Bootstrap admin password — applied only on the first boot of an empty database; rotate an existing admin through the API/UI. `fiber` is refused at boot (see `FIBER_ALLOW_DEFAULT_ADMIN`) | +| `FIBER_SECRETS_KEY` | Encrypts project and webhook secrets at rest: `openssl rand -hex 32`. Keep a backup — there is no recovery path | **Optional.** | Variable | Default | Purpose | |---|---|---| | `FIBER_VERSION` | `latest` | Tag for the `ghcr.io/durablefibers/fiber-api` and `fiber-agent` images | -| `FIBER_SECRETS_KEY` | empty (plaintext, warned) | Passed through to `fiber-api` | +| `FIBER_ALLOW_DEFAULT_ADMIN` | empty | `1` lets `fiber-api` start while the admin can still log in with `fiber`. See the API table | | `FIBER_ADMIN_USER` | `admin` | Bootstrap admin username | | `FIBER_MAX_STEPS` | `500` | See above. Set it for the CLI as well as the server, or the two disagree | | `FIBER_SEED_SHOWCASE` | `auto` | See above; `auto` seeds the demo project only on the first boot of an empty database | @@ -80,7 +82,7 @@ file passes the variable through as an empty string and each reader falls back. | `FIBER_API_BIND` / `FIBER_UI_BIND` | `127.0.0.1` | Host interface for `18080` / `3100`; set `0.0.0.0` only without a reverse proxy | | `VITE_FIBER_API_URL` | `http://localhost:18080` | Baked into the UI bundle | | `FIBER_PUBLIC_URL` | empty | Public base URL; commit statuses link back to it | -| `FIBER_GITHUB_TOKEN` | empty | Instance-wide fallback for the PR file API and commit statuses | +| `FIBER_GITHUB_TOKEN` | empty | No longer used; set a project secret `GITHUB_TOKEN` instead | | `FIBER_GITHUB_API_URL` | empty (`https://api.github.com`) | GitHub Enterprise API base | | `FIBER_RETENTION_DAYS` / `_KEEP_RUNS` / `_BATCH` / `_INTERVAL_SECS` | empty (`30` / `20` / `100` / `3600`) | Run GC; see the API table | | `FIBER_RETENTION_FIBER_DAYS` | `7` | Durable-fiber GC | diff --git a/docs/development.md b/docs/development.md index 571e6a4..b516dae 100644 --- a/docs/development.md +++ b/docs/development.md @@ -11,6 +11,16 @@ docker compose -f deploy/docker-compose.yml up -d fiber-postgres fiber-redis make infra-s3 ``` +`make infra` (and `make down`) supply throwaway credentials themselves — **unless you +have a `deploy/.env`**, which then wins outright. Compose now requires +`FIBER_SECRETS_KEY` there, so a `deploy/.env` written before that fails every `make` +target that runs Compose with `required variable FIBER_SECRETS_KEY is missing a value`. +Add one (this replaces an empty line and leaves a set one alone): + +```bash +grep -q '^FIBER_SECRETS_KEY=.' deploy/.env || { sed -i.bak '/^FIBER_SECRETS_KEY=/d' deploy/.env; echo "FIBER_SECRETS_KEY=$(openssl rand -hex 32)" >> deploy/.env; } +``` + Copy [`.env.example`](../.env.example) into your shell, or always use: ```bash diff --git a/docs/getting-started.md b/docs/getting-started.md index e9e829b..0ed8a1a 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -83,7 +83,7 @@ Now **fill in the four credentials** in `deploy/.env` — `FIBER_POSTGRES_PASSWO `FIBER_ADMIN_PASSWORD`. They have no defaults on purpose; Compose refuses to read the file until each one has a value, naming the one it is missing. The S3 pair is only used by the optional S3 profile, but it still needs a placeholder. Then generate the -secrets key and start: +secrets key, which Compose also requires, and start: ```bash sed -i'' -e "s/^FIBER_SECRETS_KEY=.*/FIBER_SECRETS_KEY=$(openssl rand -hex 32)/" deploy/.env # keep a backup of this key diff --git a/docs/operations.md b/docs/operations.md index ac39357..cc4d1dd 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -4,7 +4,7 @@ `deploy/docker-compose.yml` is the reference deployment. Its defaults are chosen so that `docker compose up` on a shared host does not expose anything by accident: -- **Four credentials have no default and must be set in `deploy/.env` before anything starts**: `FIBER_POSTGRES_PASSWORD`, `FIBER_REDIS_PASSWORD`, `FIBER_S3_ACCESS_KEY` / `FIBER_S3_SECRET_KEY`, and `FIBER_ADMIN_PASSWORD`. `docker compose config` (and therefore `up`, `pull`, `logs`) fails with the name of the first one missing until they are. A default that works is a default nobody changes, and `FIBER_API_BIND=0.0.0.0` plus a published admin password is a public instance with a known login. The S3 pair is required even when the S3 store is off — Compose interpolates the whole file before it filters by profile — so give it any non-empty placeholder if you do not use the object store. +- **Four credentials and the secrets key have no default and must be set in `deploy/.env` before anything starts**: `FIBER_POSTGRES_PASSWORD`, `FIBER_REDIS_PASSWORD`, `FIBER_S3_ACCESS_KEY` / `FIBER_S3_SECRET_KEY`, `FIBER_ADMIN_PASSWORD`, and `FIBER_SECRETS_KEY`. `docker compose config` (and therefore `up`, `pull`, `logs`) fails with the name of the first one missing until they are. A default that works is a default nobody changes, and `FIBER_API_BIND=0.0.0.0` plus a published admin password is a public instance with a known login. The S3 pair is required even when the S3 store is off — Compose interpolates the whole file before it filters by profile — so give it any non-empty placeholder if you do not use the object store. - Postgres, Redis, and the S3 store publish only on **127.0.0.1**; Redis requires a password, which reaches it through a Compose `configs:` entry rather than `--requirepass` on the command line, where every process in the host's pid namespace could read it out of `ps`. Its healthcheck is unauthenticated on purpose (`NOAUTH` counts as an answer) so the password does not go back into the service's environment to satisfy a probe. This is not secrecy from the host: `fiber-api`'s own environment still carries both passwords inside `FIBER_DATABASE_URL` and `FIBER_REDIS_URL`, so anyone who can reach the Docker daemon — which is root-equivalent anyway — can read them from `docker inspect`. Treat `deploy/.env` and the daemon socket as the trust boundary. - **Compose 2.20 or newer**, and **2.23** for `docker compose up` to work at all: the Redis password uses an inline `configs:` `content:` block (2.23) and `depends_on.required` (2.20). An older Compose fails to parse the file rather than silently ignoring either, so the failure is loud — `docker compose version` tells you what you have. - `fiber-api` (`18080`) and `fiber-ui` (`3100`) also bind to `127.0.0.1` by default — terminate TLS with a reverse proxy and forward to them. Set `FIBER_API_BIND=0.0.0.0` / `FIBER_UI_BIND=0.0.0.0` only for a trusted network. @@ -14,8 +14,8 @@ - Every service has `restart: unless-stopped`; the API waits for Postgres and Redis health, and for the S3 store too when the `s3` profile is active. - Both images carry a `HEALTHCHECK` (API: `GET /ready`; UI: `GET /`), so `docker run` and non-Compose orchestrators get the same probe Compose uses. - `fiber-api` runs with `init: true` and `stop_grace_period: 30s`, so `docker stop` and a `compose up` of a new image deliver SIGTERM and the API drains (see [Shutdown and deploys](#shutdown-and-deploys)) instead of being killed after Docker's default 10 s. -- Settings live in `deploy/.env` (copy `deploy/.env.example`). Generate `FIBER_SECRETS_KEY` with `openssl rand -hex 32` before storing any real secret; without it, secrets are stored in plaintext and the API warns at boot. -- Set `FIBER_ADMIN_PASSWORD` **before the first boot**: it is applied only when the users table is empty. For an existing instance, change the admin password through the API/UI instead. The API warns at boot while the configured value is the default `fiber`. +- Settings live in `deploy/.env` (copy `deploy/.env.example`). Generate `FIBER_SECRETS_KEY` with `openssl rand -hex 32` and keep a backup; Compose requires it, and the API refuses to start on a value that is not 64 hex characters rather than storing secrets in plaintext. (A host-run `fiber-api` with the variable unset still starts, stores plaintext, and warns at boot — that is the development default.) +- Set `FIBER_ADMIN_PASSWORD` **before the first boot**: it is applied only when the users table is empty. For an existing instance, change the admin password through the API/UI instead. While the bootstrap admin can still log in with the old default `fiber`, the API refuses to start on a non-loopback `FIBER_LISTEN` — which is every container. Change the password with `fiber-api reset-password admin` (it needs only the database, not a running server), or set `FIBER_ALLOW_DEFAULT_ADMIN=1` to start anyway. Minimal Caddy front end (automatic TLS): @@ -621,6 +621,7 @@ Volume name may be prefixed by the Compose project (`fiber_fiber_pg` when using ``` Whether this affects a Compose install depends on which backend it used. Until this version Compose set `FIBER_S3_BUCKET` unconditionally, so blobs went to MinIO and the `fiber_artifacts` volume held nothing — but the same upgrade **makes the local filesystem the default**, so an install that says nothing in `deploy/.env` switches backends and starts writing to that root-owned volume. Either run the `chown` above, or keep the object store by setting `FIBER_S3_BUCKET=fiber-artifacts` in `deploy/.env` and bringing the stack up with `--profile s3` (`--profile minio` before 0.6.5, see below). Artifacts already in MinIO are not copied to the local disk (or the other way round); the rows keep pointing at the backend that stored them, so switching leaves older artifacts undownloadable until you switch back. +- **`FIBER_SECRETS_KEY` is required by Compose, and `admin` / `fiber` no longer boots there** (after 0.6.7). Before upgrading: (1) make sure `deploy/.env` has a `FIBER_SECRETS_KEY` — if yours is empty, generate one with `openssl rand -hex 32`; secrets already stored in plaintext stay readable and are encrypted as they are next written. If it is set but is not 64 hex characters, the API was silently storing plaintext and will now refuse to start: replace it with a real key. (2) If the admin password is still `fiber`, change it — `docker compose -f deploy/docker-compose.yml run --rm fiber-api fiber-api reset-password admin` — or set `FIBER_ALLOW_DEFAULT_ADMIN=1`. (3) A host-run `fiber-api` now listens on `127.0.0.1:18080` unless `FIBER_LISTEN` says otherwise; set `FIBER_LISTEN=0.0.0.0:18080` if agents or a proxy on other hosts connect to it. - **The four Compose credentials are now required.** `FIBER_POSTGRES_PASSWORD`, `FIBER_REDIS_PASSWORD`, `FIBER_S3_ACCESS_KEY`/`FIBER_S3_SECRET_KEY` and `FIBER_ADMIN_PASSWORD` no longer default to `fiber` / `fiberfiber`; `docker compose` refuses to read the file until `deploy/.env` sets them. An existing deployment that relied on the defaults must write those same values into `deploy/.env` before the upgrade — Postgres keeps the password its volume was initialised with, and Redis keeps whatever the new config file says, so putting the old values back is the no-downtime path. Changing the Postgres password needs `ALTER ROLE fiber PASSWORD …` inside the container as well as the `.env` edit; `FIBER_ADMIN_PASSWORD` is read only on the first boot of an empty database, so for an existing instance any placeholder is fine. - **Since 0.6.5 the Compose S3 store is RustFS, not MinIO** (`quay.io/minio/minio` stopped serving anonymous pulls). The service is `fiber-s3`, the profile `s3`, the volume `fiber_s3`, and the default `FIBER_S3_ENDPOINT` is `http://fiber-s3:9000`. An install on the local filesystem backend (`FIBER_S3_BUCKET` empty, the default) needs nothing. An install that stored artifacts in MinIO needs three things: - **Edit `deploy/.env`.** If it sets `FIBER_S3_ENDPOINT=http://fiber-minio:9000` (the old example did), change it to `http://fiber-s3:9000`. Start the stack with `--profile s3` instead of `--profile minio`. diff --git a/docs/triggers.md b/docs/triggers.md index fa2e6d3..0e2e75a 100644 --- a/docs/triggers.md +++ b/docs/triggers.md @@ -34,12 +34,18 @@ on: PR webhooks do **not** include file lists. When path filters are set, Fiber lists files via the GitHub API: -1. Project secret `GITHUB_TOKEN` or `FIBER_GITHUB_TOKEN`, else env `FIBER_GITHUB_TOKEN` / `GITHUB_TOKEN` +1. Project secret `GITHUB_TOKEN` or `FIBER_GITHUB_TOKEN` (read access to the repository's pull requests). **Only** the project's secret: an environment variable of that name on `fiber-api` is not a fallback 2. `GET /repos/{owner}/{repo}/pulls/{n}/files` (paginated) 3. Optional `FIBER_GITHUB_API_URL` for GitHub Enterprise (default `https://api.github.com`) Without a token (and without embedded `changed_files` in a test payload), path-filtered PR pipelines **skip**. +The token is per project because the repository it is spent on is named by the webhook +payload, and the payload is trusted on the strength of the project's webhook secret — +which the project's own admins set. An instance-wide token would be usable by any of +them against any repository it can read. Up to 0.6.7 the environment variable was such a +fallback; `fiber-api` now logs one warning if it is set and a project needed it. + ## Schedules ```yaml @@ -72,8 +78,9 @@ A run started by a webhook records the commit that triggered it (`head_sha`, `he - The status links to the run page when `FIBER_PUBLIC_URL` is set. Posting needs a token with `repo:status` — a project secret `GITHUB_TOKEN` / -`FIBER_GITHUB_TOKEN`, or the same environment variables on `fiber-api`. Without one, -runs still work and the status is simply not posted. A failure to post never fails a run. +`FIBER_GITHUB_TOKEN`. The environment variables of the same name on `fiber-api` are not +used. Without a token, runs still work and the status is simply not posted. A failure to +post never fails a run. ## What gets built diff --git a/examples/paths-filtered.yml b/examples/paths-filtered.yml index e34a060..cc3bd01 100644 --- a/examples/paths-filtered.yml +++ b/examples/paths-filtered.yml @@ -1,6 +1,6 @@ name: paths-filtered # Fires on push/PR to main only when Rust/Cargo files change; ignores docs. -# For PR path filters, set project secret GITHUB_TOKEN (or FIBER_GITHUB_TOKEN env) +# For PR path filters, set project secret GITHUB_TOKEN # so Fiber can list changed files via the GitHub API. workspace: repo: https://github.com/octocat/Hello-World.git diff --git a/scripts/dev-env.sh b/scripts/dev-env.sh index eaadd83..8361b61 100755 --- a/scripts/dev-env.sh +++ b/scripts/dev-env.sh @@ -8,7 +8,10 @@ export FIBER_ROOT="$ROOT" export FIBER_DATABASE_URL="${FIBER_DATABASE_URL:-postgres://fiber:fiber@127.0.0.1:15432/fiber}" export FIBER_REDIS_URL="${FIBER_REDIS_URL:-redis://:${FIBER_REDIS_PASSWORD:-fiber}@127.0.0.1:16379}" -export FIBER_LISTEN="${FIBER_LISTEN:-0.0.0.0:18080}" +# Loopback. To serve other machines set FIBER_LISTEN=0.0.0.0:18080 — fiber-api then +# refuses to start while admin/fiber still works, unless FIBER_ALLOW_DEFAULT_ADMIN=1. +export FIBER_LISTEN="${FIBER_LISTEN:-127.0.0.1:18080}" +# export FIBER_ALLOW_DEFAULT_ADMIN=1 export FIBER_ARTIFACTS_DIR="${FIBER_ARTIFACTS_DIR:-$ROOT/data/artifacts}" export FIBER_ADMIN_USER="${FIBER_ADMIN_USER:-admin}" export FIBER_ADMIN_PASSWORD="${FIBER_ADMIN_PASSWORD:-fiber}" @@ -26,7 +29,8 @@ export RUST_LOG="${RUST_LOG:-info,fiber_api=info,fiber_agent=info}" # export FIBER_STEP_TIMEOUT_DEFAULT_MINUTES=60 # export FIBER_STEP_TIMEOUT_GRACE_MINUTES=5 -# Optional secrets encryption (generate once: openssl rand -hex 32) +# Optional secrets encryption (generate once: openssl rand -hex 32). Set but not 64 hex +# characters is a boot error, not plaintext. # export FIBER_SECRETS_KEY=... # Optional S3 store (make infra-s3 first) diff --git a/scripts/smoke_compose.sh b/scripts/smoke_compose.sh index be8ec12..6998aac 100755 --- a/scripts/smoke_compose.sh +++ b/scripts/smoke_compose.sh @@ -9,9 +9,13 @@ ok() { echo "OK $*"; } fail() { echo "FAIL $*"; FAILS=$((FAILS + 1)); } # The deployment profile expects deploy/.env; supply throwaway values when absent. -# The four credentials are required by deploy/docker-compose.yml (no working defaults), -# so without these exports `docker compose` refuses to interpolate the file at all. -# FIBER_ADMIN_PASSWORD has to stay `fiber`: _compose_pipeline.py logs in with it. +# The four credentials and the secrets key are required by deploy/docker-compose.yml +# (no working defaults), so without these exports `docker compose` refuses to +# interpolate the file at all. +# FIBER_ADMIN_PASSWORD has to stay `fiber`: _compose_pipeline.py logs in with it. The +# containerised fiber-api listens on 0.0.0.0 and would refuse to start with that +# password, so say that it is meant here. +export FIBER_ALLOW_DEFAULT_ADMIN="${FIBER_ALLOW_DEFAULT_ADMIN:-1}" export FIBER_SECRETS_KEY="${FIBER_SECRETS_KEY:-$(openssl rand -hex 32)}" export FIBER_POSTGRES_PASSWORD="${FIBER_POSTGRES_PASSWORD:-fiber}" export FIBER_REDIS_PASSWORD="${FIBER_REDIS_PASSWORD:-fiber}"