diff --git a/.env.example b/.env.example index 1692918..f5c6287 100644 --- a/.env.example +++ b/.env.example @@ -25,6 +25,9 @@ FIBER_ADMIN_PASSWORD=fiber FIBER_API_URL=http://127.0.0.1:18080 FIBER_AGENT_USE_DOCKER=false +# Unset = true for a shell agent (USE_DOCKER=false), false for a Docker agent, which then +# refuses a step without `image:`. Fork pull request steps never run on the host shell. +# FIBER_AGENT_ALLOW_HOST_SHELL=true FIBER_AGENT_NAME=local FIBER_AGENT_LABELS=os=linux FIBER_AGENT_CONCURRENCY=1 diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d1da45..1dc7f18 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,8 +6,58 @@ minor versions may carry breaking changes. ## [Unreleased] +**Breaking: a Docker agent no longer runs a step without `image:`.** It used to run it on +the agent's host shell without being asked; now the step fails, saying so. Add `image:` to +the step, or put the old behaviour back with one line on the agent: +`FIBER_AGENT_ALLOW_HOST_SHELL=true`. `FIBER_AGENT_USE_DOCKER` defaults to `true` in the +binary, so an agent started without it is a Docker agent and is affected. Shell agents +(`FIBER_AGENT_USE_DOCKER=false` — the published image, the plain Compose `agent` profile, +`install-agent.sh` without `--docker`, `make agent`) are unchanged for the project's own +runs. **A fork's pull request now runs in containers or +fails** — on a shell agent it fails, every step, and no setting changes that. + +Wire: `Offer` gains `untrusted` (defaulted, so 0.6.7 agents and servers keep working with +the new ones) and `PROTOCOL_VERSION` is now `3`. The fork-PR control is carried out +by the agent, so the server offers a fork pull request's steps only to an agent that +declared revision 3: **until a project agent at this release is online, those steps wait +in `queued`.** Upgrade the agents, not only the API. + ### Security +- **A step without `image:` ran on the host of a Docker agent, as the agent's user.** An + agent with `FIBER_AGENT_USE_DOCKER=true` ran a step in a container only when the step + named an image, and dropped every other step to its own shell, with no way to refuse. + On the `agent-docker` Compose profile that shell is root with the engine's socket + mounted, so any project `writer` had root on the Docker host by leaving `image:` out — + and so did the author of a fork's pull request, through any imageless step that runs a + script from the repository. + - `FIBER_AGENT_ALLOW_HOST_SHELL` decides whether an agent runs steps on its host. Unset, + it is `false` on a Docker agent and `true` on a shell agent. It takes `true`, `false`, + `1` or `0`; anything else stops the agent at startup rather than meaning one of them. + - A refused step fails before anything is cloned or restored, with the remedy in its + log and its error: ``this agent runs steps in containers only: add `image:` to the + step, or set FIBER_AGENT_ALLOW_HOST_SHELL=true on the agent``. + - The `agent-docker` Compose profile pins it to `false`. The pin only takes effect when + the agent image is this release or later: the profile builds `FROM` the published + `fiber-agent:${FIBER_VERSION}`, and a 0.6.7 binary ignores the variable silently. + Rebuild the service, then confirm in its startup log a line beginning `steps run in + containers only`. + - An agent installed with `install-agent.sh --docker` gets the refusing default with no + extra flag. A `FIBER_AGENT_ALLOW_HOST_SHELL` line added to `/etc/fiber/agent.env` is + kept across re-runs of the installer. + + [agents](docs/agents.md#steps-that-would-run-on-the-host). +- **A fork's pull request never runs on an agent's host shell.** The offer now tells the + agent the run is untrusted, and the agent runs such a step in a container or fails it — + `FIBER_AGENT_ALLOW_HOST_SHELL=true` does not apply. A fork PR therefore needs an + `image:` on every step and a project agent with Docker on; on a shell agent it fails + with `fork pull request steps only run in containers, and this is a shell agent …`. + That is the intended outcome: a shell agent has nowhere to run the code but its own + account. An agent older than this release does not know the flag, so the server does + not offer it such a step: a project agent whose `Hello` declares a protocol revision + below 3 is passed over, the step stays `queued` rather than failing, and the server + warns once when that agent connects. The agent keeps taking the project's own runs. + [triggers](docs/triggers.md#pull-requests-from-forks). - **Seven RustSec advisories in the lockfile, all fixed by upgrades.** `rustls` 0.23.46 (RUSTSEC-2026-0346 to -0349), `rustls-webpki` 0.103.16 (RUSTSEC-2026-0382: certificate verification could accept a distrusted intermediate), `tokio` 1.54.1 (RUSTSEC-2026-0386) @@ -23,6 +73,15 @@ minor versions may carry breaking changes. workflow runs `cargo deny check` before it builds anything. - The release workflow passes the tag to its version check through the environment rather than interpolating it into the script. +- **The Compose agents carry a label each: `docker=true` on the `agent-docker` profile, + `executor=shell` on the plain `agent` profile,** both appended to `FIBER_AGENT_LABELS`. + An `image:` step asks for `docker=true` to reach the Docker agent. A step without + `image:` asks for `executor=shell` to stay off it — `docker=true` alone does not keep + it away, since a step matches any agent that has every label it asks for. + [agents](docs/agents.md#steps-that-would-run-on-the-host). +- **`smoke-runs` covers the refusal.** Its fork-PR check now expects a shell agent to + refuse the run, then — where a Docker engine is available — runs the same pull request + in a container to check it gets no secrets. ## [0.6.7] — 2026-10-01 diff --git a/SECURITY.md b/SECURITY.md index a1aeeab..8b70f1f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -23,8 +23,11 @@ boundaries are. reaching instance-level surfaces, a non-admin managing global agents or users. - Authentication or session flaws: forging a session or agent token, bypassing webhook signature verification, session fixation. -- A **fork's** pull request obtaining project secrets, or its run being scheduled onto an - agent outside that project's own pool. +- A **fork's** pull request obtaining project secrets, its run being scheduled onto an + agent outside that project's own pool, or a step of it running on an agent's host shell + instead of in a container. +- A Docker agent running a step on its host shell without + `FIBER_AGENT_ALLOW_HOST_SHELL=true`. - Secrets escaping where they should not go: into logs, into a process list, into an artifact, into an API response. - Anything unauthenticated that starts a run, mutates state, or discloses data. @@ -36,12 +39,15 @@ boundaries are. `fiber.yml`, and see [docs/agents.md](docs/agents.md) for how a step is confined. - **An instance admin is effectively root.** They manage the global agent pool, and an agent token receives the secrets of the projects it serves. -- **Fork pull requests still execute untrusted code.** Fiber withholds secrets and - restricts them to project-dedicated agents, which limits the blast radius; it is not a - sandbox. Run them on a disposable agent with containerised steps — - [docs/triggers.md](docs/triggers.md#pull-requests-from-forks). -- **Steps run without a container unless the step sets `image:`.** A host-shell step runs - as the agent's user. +- **Fork pull requests still execute untrusted code.** Fiber withholds secrets, restricts + them to project-dedicated agents, and runs their steps in containers or not at all, + which limits the blast radius; a container is not a sandbox. Run them on a disposable + agent — [docs/triggers.md](docs/triggers.md#pull-requests-from-forks). +- **A host-shell step runs as the agent's user.** That is every step on a shell agent + (`FIBER_AGENT_USE_DOCKER=false`), and a step without `image:` on a Docker agent whose + operator set `FIBER_AGENT_ALLOW_HOST_SHELL=true` — where the agent's user can reach the + Docker engine, so such a step is root on the host. A Docker agent refuses those steps + by default. - **Defaults are for a local trial, not the internet.** `admin` / `fiber`, no secrets encryption key, everything on loopback. [docs/operations.md](docs/operations.md#deployment) describes what to change before exposing an instance. diff --git a/crates/fiber-agent/src/main.rs b/crates/fiber-agent/src/main.rs index 964d11b..43f5678 100644 --- a/crates/fiber-agent/src/main.rs +++ b/crates/fiber-agent/src/main.rs @@ -53,6 +53,18 @@ struct Args { #[arg(long, env = "FIBER_AGENT_USE_DOCKER", default_value_t = true)] use_docker: bool, + /// Whether a step may run on this host's shell: `true`/`false`/`1`/`0`. Unset, it is + /// `false` for a Docker agent (a step without `image:` is refused) and `true` for a + /// shell agent. A fork pull request's steps never run on the host shell, whatever + /// this says. Kept as text and checked at startup (`host_shell_policy`), so a typo + /// stops the agent instead of quietly meaning one or the other. + #[arg(long, env = "FIBER_AGENT_ALLOW_HOST_SHELL")] + allow_host_shell: Option, + + /// `allow_host_shell` resolved against `use_docker`, once, in `main`. + #[arg(skip)] + host_shell_allowed: bool, + #[arg( long, env = "FIBER_AGENT_WORKSPACE_DIR", @@ -160,6 +172,8 @@ fn docker_client_reads(k: &str) -> bool { #[derive(Clone)] struct ExecConfig { use_docker: bool, + /// Whether a trusted step may run on the host shell (`host_shell_policy`). + allow_host_shell: bool, /// `--label`s on every step container: this agent, and this process of it. The /// startup sweep uses them to find its own orphans without touching another agent's /// running containers. @@ -176,6 +190,7 @@ impl ExecConfig { fn from_args(args: &Args, container_labels: ContainerLabels) -> Self { Self { use_docker: args.use_docker, + allow_host_shell: args.host_shell_allowed, container_labels, env_passthrough: args .env_passthrough @@ -192,6 +207,112 @@ impl ExecConfig { } } +/// Where one step runs. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum ExecutionMode<'a> { + /// `docker run` of this image (trimmed), with the step's workspace mounted. + Container(&'a str), + /// The agent's own shell, as the agent's user. + HostShell, +} + +/// Why a step that would have run on the host shell was refused instead. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum HostShellRefusal { + /// The run builds code from outside the project. Not the operator's to override. + Untrusted, + /// This agent does not run steps on its host (`FIBER_AGENT_ALLOW_HOST_SHELL`). + NotAllowed, +} + +impl HostShellRefusal { + /// What the step's log and `error` say. `has_image` and `use_docker` pick the remedy: + /// telling someone to add an `image:` the step already has, or that a shell agent + /// will never use, sends them the wrong way. + fn message(self, has_image: bool, use_docker: bool) -> String { + match (self, use_docker) { + (HostShellRefusal::Untrusted, true) => "fork pull request steps only run in \ + containers: this run builds code from outside the project and the step has \ + no `image:`, so it would have run on the agent's host. Add `image:` to the \ + step (FIBER_AGENT_ALLOW_HOST_SHELL does not apply to fork pull requests)" + .into(), + (HostShellRefusal::Untrusted, false) => format!( + "fork pull request steps only run in containers, and this is a shell agent \ + (FIBER_AGENT_USE_DOCKER=false) that runs every step on its host: run fork \ + pull requests on a project agent with FIBER_AGENT_USE_DOCKER=true{}", + if has_image { + "" + } else { + ", and add `image:` to the step" + } + ), + (HostShellRefusal::NotAllowed, true) => "this agent runs steps in containers \ + only: add `image:` to the step, or set FIBER_AGENT_ALLOW_HOST_SHELL=true on \ + the agent" + .into(), + // `host_shell_policy` refuses to start an agent configured like this, so no + // offer reaches here; worded anyway rather than left to mislead. + (HostShellRefusal::NotAllowed, false) => "this agent has host shell turned off \ + (FIBER_AGENT_ALLOW_HOST_SHELL=false) and Docker turned off \ + (FIBER_AGENT_USE_DOCKER=false), so it cannot run any step" + .into(), + } + } +} + +/// Decide where a step runs, or that it must not run here at all. +/// +/// A step runs in a container when it names an image and the agent uses Docker. +/// Everything else is the host shell — a step with no `image:`, and any step on a shell +/// agent, which ignores `image:` — and the host shell is the agent's own user, token and +/// (on a Docker agent) engine socket. So it is refused for a fork's pull request always, +/// and otherwise unless the agent allows it. +fn execution_mode( + image: Option<&str>, + use_docker: bool, + allow_host_shell: bool, + untrusted: bool, +) -> std::result::Result, HostShellRefusal> { + // Trimmed here as at compile time, so the value docker sees is the value that was + // checked. Empty means "no image". + let image = image.map(str::trim).filter(|i| !i.is_empty()); + match image { + Some(img) if use_docker => Ok(ExecutionMode::Container(img)), + // Before `allow_host_shell`: that flag is the operator vouching for the people + // who can write the pipeline, and a fork's author is not one of them. + _ if untrusted => Err(HostShellRefusal::Untrusted), + _ if !allow_host_shell => Err(HostShellRefusal::NotAllowed), + _ => Ok(ExecutionMode::HostShell), + } +} + +/// Resolve `FIBER_AGENT_ALLOW_HOST_SHELL` at startup. Unset (or empty, which is what +/// Compose passes for a variable nobody set) takes the default: allowed on a shell +/// agent, where the host is all there is, and refused on a Docker agent. Anything that is +/// not a boolean is an error rather than a guess, since guessing wrong in one direction +/// runs pipeline code on the host. +fn host_shell_policy(raw: Option<&str>, use_docker: bool) -> std::result::Result { + let explicit = match raw.map(str::trim).filter(|v| !v.is_empty()) { + None => None, + Some("true" | "1") => Some(true), + Some("false" | "0") => Some(false), + Some(other) => { + return Err(format!( + "FIBER_AGENT_ALLOW_HOST_SHELL must be true, false, 1 or 0 (got `{other}`)" + )); + } + }; + let allowed = explicit.unwrap_or(!use_docker); + if !allowed && !use_docker { + return Err( + "FIBER_AGENT_ALLOW_HOST_SHELL=false with FIBER_AGENT_USE_DOCKER=false leaves the \ + agent unable to run any step: enable Docker, or allow host shell" + .into(), + ); + } + Ok(allowed) +} + /// Per-run workspace refcount: the last step of a run to finish on this agent deletes /// the run's tree. Without it every run leaks a checkout for the agent's lifetime. #[derive(Default)] @@ -1428,6 +1549,28 @@ async fn main() -> Result<()> { } } info!(workspace_dir = %args.workspace_dir.display(), "workspace root"); + match host_shell_policy(args.allow_host_shell.as_deref(), args.use_docker) { + Ok(allowed) => args.host_shell_allowed = allowed, + Err(e) => { + error!("{e}"); + std::process::exit(2); + } + } + match (args.use_docker, args.host_shell_allowed) { + (true, false) => info!( + "steps run in containers only: a step without `image:` is refused \ + (FIBER_AGENT_ALLOW_HOST_SHELL=true allows it)" + ), + (true, true) => warn!( + "FIBER_AGENT_ALLOW_HOST_SHELL=true: a step without `image:` runs on this host as \ + the agent's user, with whatever that user can reach — including the Docker \ + engine. Fork pull request steps are still refused" + ), + (false, _) => info!( + "shell agent: every step runs on this host as the agent's user, `image:` is \ + ignored, and fork pull request steps are refused" + ), + } // A macOS workspace reaches step containers through Docker Desktop's file sharing, // which intermittently loses files a build has just written — `cargo` fails with // `can't find crate for …` naming a crate it compiled seconds earlier. Same image and @@ -1842,6 +1985,7 @@ async fn run_session( working_directory, shell, attempt, + untrusted, }) => { info!(%step_id, %step_name, %run_id, ?attempt, "offered step"); // Anything still queued about this step_run_id is from an @@ -1898,6 +2042,7 @@ async fn run_session( traceparent.as_deref(), working_directory.as_deref(), shell.as_deref(), + untrusted, ).await; if let Ok(mut g) = cancels.lock() { @@ -2051,6 +2196,7 @@ async fn execute_step( traceparent: Option<&str>, working_directory: Option<&str>, shell: Option<&str>, + untrusted: bool, ) -> Result { let run_dir = workspace_root.join(run_id.to_string()); // A fresh directory per execution, never a reused one. A previous attempt's container @@ -2096,6 +2242,7 @@ async fn execute_step( workspaces, working_directory, shell, + untrusted, ) // `.instrument`, not `span.enter()`: a guard held across an await point attributes // whatever else the runtime schedules on this thread to this step. @@ -2138,6 +2285,7 @@ async fn execute_step_inner( workspaces: &Workspaces, working_directory: Option<&str>, shell: Option<&str>, + untrusted: bool, ) -> Result { // One sequence for system/stdout/stderr so the server's `ORDER BY seq` interleaves // streams in emission order (the old per-stream bases collided after 1000 lines). @@ -2202,6 +2350,18 @@ async fn execute_step_inner( log("system", format!("step timeout: {m} min")); } + // Decided before anything touches the disk or the network: a step this agent will + // not run gets no checkout, no artifact restore, and no directory. + let mode = match execution_mode(image, exec.use_docker, exec.allow_host_shell, untrusted) { + Ok(mode) => mode, + Err(refusal) => { + let has_image = image.is_some_and(|i| !i.trim().is_empty()); + let msg = refusal.message(has_image, exec.use_docker); + log("system", format!("step refused: {msg}")); + bail!("{msg}"); + } + }; + tokio::fs::create_dir_all(work_dir).await?; // Workspace prep can be interrupted by cancel. @@ -2318,13 +2478,8 @@ async fn execute_step_inner( } let mut env_file: Option = None; - // Trimmed here as at compile time, so the value docker sees is the value that was - // checked. Empty means "no image". - let mut child = if let Some(img) = image - .map(str::trim) - .filter(|i| !i.is_empty()) - .filter(|_| exec.use_docker) - { + // `mode` was settled at the top (`execution_mode`), before the workspace existed. + let mut child = if let ExecutionMode::Container(img) = mode { let mount = format!("{}:/workspace", work_dir.display()); log( "system", @@ -3788,6 +3943,141 @@ mod tests { } } + #[test] + fn execution_mode_covers_every_combination() { + use ExecutionMode::{Container, HostShell}; + use HostShellRefusal::{NotAllowed, Untrusted}; + let img = Some("alpine:3"); + // (image, use_docker, allow_host_shell, untrusted) -> outcome. All sixteen rows, + // so a change to any arm of the decision moves at least one of them. + let table = [ + // An image on a Docker agent is a container, whoever asks and whatever the flag. + (img, true, true, false, Ok(Container("alpine:3"))), + (img, true, false, false, Ok(Container("alpine:3"))), + (img, true, true, true, Ok(Container("alpine:3"))), + (img, true, false, true, Ok(Container("alpine:3"))), + // No image on a Docker agent: the host, so only when the operator allows it, + // and never for a fork's pull request. + (None, true, true, false, Ok(HostShell)), + (None, true, false, false, Err(NotAllowed)), + (None, true, true, true, Err(Untrusted)), + (None, true, false, true, Err(Untrusted)), + // A shell agent ignores `image:` — the documented behaviour — so both of + // these are the host too. + (img, false, true, false, Ok(HostShell)), + (None, false, true, false, Ok(HostShell)), + (img, false, true, true, Err(Untrusted)), + (None, false, true, true, Err(Untrusted)), + // Not a configuration the agent starts with (`host_shell_policy`), but the + // decision still has to refuse rather than run. + (img, false, false, false, Err(NotAllowed)), + (None, false, false, false, Err(NotAllowed)), + (img, false, false, true, Err(Untrusted)), + (None, false, false, true, Err(Untrusted)), + ]; + for (image, use_docker, allow, untrusted, want) in table { + assert_eq!( + execution_mode(image, use_docker, allow, untrusted), + want, + "image={image:?} use_docker={use_docker} allow_host_shell={allow} untrusted={untrusted}" + ); + } + } + + #[test] + fn a_blank_image_is_no_image_and_a_padded_one_is_trimmed() { + // The server trims at compile time; a snapshot from before that can hold either. + assert_eq!( + execution_mode(Some(" "), true, false, false), + Err(HostShellRefusal::NotAllowed) + ); + assert_eq!( + execution_mode(Some(""), true, true, false), + Ok(ExecutionMode::HostShell) + ); + assert_eq!( + execution_mode(Some(" alpine:3 "), true, false, true), + Ok(ExecutionMode::Container("alpine:3")) + ); + } + + #[test] + fn a_refusal_names_the_remedy_that_applies() { + let not_allowed = HostShellRefusal::NotAllowed.message(false, true); + assert!(not_allowed.contains("add `image:` to the step")); + assert!(not_allowed.contains("FIBER_AGENT_ALLOW_HOST_SHELL=true")); + + // The flag is no way out for a fork, so the message must not offer it as one. + let fork = HostShellRefusal::Untrusted.message(false, true); + assert!(fork.starts_with("fork pull request steps only run in containers")); + assert!(fork.contains("add `image:`") || fork.contains("Add `image:`")); + assert!(!fork.contains("FIBER_AGENT_ALLOW_HOST_SHELL=true")); + + // On a shell agent an `image:` changes nothing; the agent is what has to change. + let shell = HostShellRefusal::Untrusted.message(true, false); + assert!(shell.starts_with("fork pull request steps only run in containers")); + assert!(shell.contains("FIBER_AGENT_USE_DOCKER=true")); + assert!(!shell.contains("`image:`")); + assert!( + HostShellRefusal::Untrusted + .message(false, false) + .contains("add `image:`") + ); + + // The completion handler reads these two prefixes as a cancel and a timeout. + for refusal in [HostShellRefusal::Untrusted, HostShellRefusal::NotAllowed] { + for (has_image, use_docker) in + [(false, false), (false, true), (true, false), (true, true)] + { + let m = refusal.message(has_image, use_docker); + assert!( + !m.contains("cancelled") && !m.starts_with("timed out"), + "{m}" + ); + } + } + } + + #[test] + fn host_shell_defaults_off_for_a_docker_agent_and_on_for_a_shell_agent() { + assert_eq!(host_shell_policy(None, true), Ok(false)); + assert_eq!(host_shell_policy(None, false), Ok(true)); + // Compose passes an unset variable as empty; that is unset, not a value. + assert_eq!(host_shell_policy(Some(""), true), Ok(false)); + assert_eq!(host_shell_policy(Some(" "), false), Ok(true)); + } + + #[test] + fn an_explicit_host_shell_setting_overrides_the_default() { + for yes in ["true", "1", " true "] { + assert_eq!(host_shell_policy(Some(yes), true), Ok(true), "{yes}"); + assert_eq!(host_shell_policy(Some(yes), false), Ok(true), "{yes}"); + } + for no in ["false", "0"] { + assert_eq!(host_shell_policy(Some(no), true), Ok(false), "{no}"); + } + } + + #[test] + fn a_host_shell_setting_that_is_not_a_boolean_stops_the_agent() { + for bad in ["yes", "no", "TRUE", "on", "2", "tru"] { + let err = host_shell_policy(Some(bad), true).unwrap_err(); + assert!( + err.contains("FIBER_AGENT_ALLOW_HOST_SHELL") && err.contains(bad), + "{err}" + ); + assert!(host_shell_policy(Some(bad), false).is_err(), "{bad}"); + } + } + + #[test] + fn an_agent_that_could_run_nothing_does_not_start() { + for no in ["false", "0"] { + let err = host_shell_policy(Some(no), false).unwrap_err(); + assert!(err.contains("FIBER_AGENT_USE_DOCKER=false"), "{err}"); + } + } + #[test] fn a_normal_line_loses_its_newline_and_nothing_else() { assert_eq!(truncate_line(b"hello world\n"), "hello world"); diff --git a/crates/fiber-api/src/ws.rs b/crates/fiber-api/src/ws.rs index c59301d..bf7d675 100644 --- a/crates/fiber-api/src/ws.rs +++ b/crates/fiber-api/src/ws.rs @@ -292,6 +292,9 @@ async fn offer_for_step(state: &AppState, step: StepRun) -> Result, agent_labels: &[String], + include_untrusted: bool, ) -> Result> { if let Some(pid) = agent_project_id { Ok(sqlx::query_as::<_, StepRun>(AssertSqlSafe(format!( @@ -2366,11 +2373,13 @@ impl Store { WHERE s.status = 'queued' AND r.project_id = $1 AND (s.not_before IS NULL OR s.not_before <= NOW()) AND s.labels <@ $2 + AND ($3 OR NOT r.untrusted) ORDER BY s.queued_at, s.id LIMIT {QUEUE_SCAN_LIMIT}" ))) .bind(pid) .bind(json!(agent_labels)) + .bind(include_untrusted) .fetch_all(&self.pool) .await?) } else { diff --git a/crates/fiber-proto/src/lib.rs b/crates/fiber-proto/src/lib.rs index 4ced800..084d8db 100644 --- a/crates/fiber-proto/src/lib.rs +++ b/crates/fiber-proto/src/lib.rs @@ -19,7 +19,18 @@ pub mod validate; /// accepts everything revision 1 did; a server that does not know `LogBatch` would /// answer `Error { "invalid message" }` and lose the output, which is why the revision /// is on the wire even though nothing gates on it yet. -pub const PROTOCOL_VERSION: u32 = 2; +/// - `3` — the agent reads `untrusted` on [`ServerMessage::Offer`] and never runs such a +/// step on its host shell: it runs in a container or fails the step. It also refuses a +/// step with no `image` when it is a Docker agent, unless its operator set +/// `FIBER_AGENT_ALLOW_HOST_SHELL=true`. The server gates on this one: see +/// [`UNTRUSTED_OFFER_PROTOCOL`]. +pub const PROTOCOL_VERSION: u32 = 3; + +/// The revision from which an agent honours `untrusted` on [`ServerMessage::Offer`]. The +/// server offers a step of an untrusted run only to an agent whose `Hello` declared this +/// or later: an older one would ignore the flag and run a fork's pull request on its +/// host, so for it the step stays queued until an agent that qualifies is online. +pub const UNTRUSTED_OFFER_PROTOCOL: u32 = 3; #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] @@ -404,6 +415,15 @@ pub enum ServerMessage { /// unless the server exports OpenTelemetry, and ignored by older agents. #[serde(default)] traceparent: Option, + /// The run builds code from outside the project (a fork's pull request). An + /// agent that reads this never runs such a step on its host shell, whatever + /// `FIBER_AGENT_ALLOW_HOST_SHELL` says: it runs in a container or fails. Absent + /// from older servers, which the agent reads as the old contract — a trusted + /// run. An agent below [`UNTRUSTED_OFFER_PROTOCOL`] would ignore it and run the + /// step as before, so the server never sends such an agent an offer with this + /// set: the step waits in the queue for one that declared that revision. + #[serde(default)] + untrusted: bool, }, Cancel { step_run_id: Uuid, @@ -716,8 +736,9 @@ mod tests { #[test] fn an_offer_from_an_older_server_omits_the_newer_fields() { - // timeout_minutes / secret_keys / traceparent / working_directory / shell are all - // `#[serde(default)]` precisely so a mixed-version deploy keeps working. + // timeout_minutes / secret_keys / traceparent / working_directory / shell / + // untrusted are all `#[serde(default)]` precisely so a mixed-version deploy keeps + // working. let minimal = json!({ "type": "offer", "step_run_id": Uuid::nil(), @@ -738,11 +759,15 @@ mod tests { shell, artifacts, restore, + untrusted, .. } = msg else { panic!("expected an offer"); }; + // An older server cannot say a run is a fork's: absent reads as trusted, which is + // what that server's agents did with it anyway. + assert!(!untrusted); assert_eq!(timeout_minutes, None); assert!(secret_keys.is_empty()); assert_eq!(traceparent, None); @@ -752,6 +777,41 @@ mod tests { assert!(restore.is_empty()); } + #[test] + fn an_untrusted_offer_says_so_on_the_wire() { + // The agent refuses host shell on this field alone, so it has to survive the trip + // under this exact name; an older agent skips a key it does not know. + let offer = ServerMessage::Offer { + step_run_id: Uuid::nil(), + run_id: Uuid::nil(), + step_id: "build".into(), + step_name: "Build".into(), + image: None, + run: "make".into(), + workspace: None, + env: vec![], + attempt: Some(1), + artifacts: vec![], + restore: vec![], + timeout_minutes: None, + working_directory: None, + shell: None, + secret_keys: vec![], + traceparent: None, + untrusted: true, + }; + let v = serde_json::to_value(&offer).unwrap(); + assert_eq!(v["untrusted"], json!(true)); + let back: ServerMessage = serde_json::from_value(v).unwrap(); + assert!(matches!( + back, + ServerMessage::Offer { + untrusted: true, + .. + } + )); + } + #[test] fn a_log_batch_round_trips_with_its_lines_in_order() { let batch = AgentMessage::LogBatch { diff --git a/crates/fiber-scheduler/src/lib.rs b/crates/fiber-scheduler/src/lib.rs index 7fd8d01..ad84b7b 100644 --- a/crates/fiber-scheduler/src/lib.rs +++ b/crates/fiber-scheduler/src/lib.rs @@ -109,6 +109,9 @@ pub struct AgentPresence { pub reserved: u32, /// `None` = global pool. pub project_id: Option, + /// `Hello.protocol_version`; `0` from an agent older than the field. Decides + /// whether the agent is offered untrusted steps ([`takes_untrusted_steps`]). + pub protocol_version: u32, } impl AgentPresence { @@ -358,6 +361,7 @@ impl Scheduler { labels: Vec, concurrency: u32, project_id: Option, + protocol_version: u32, ) { // Nothing about the agent's running steps is kept here: a lease outlives the // session that took it, so every offer counts the rows the database shows @@ -376,6 +380,7 @@ impl Scheduler { concurrency: concurrency.max(1), reserved, project_id, + protocol_version, }, ); } @@ -803,7 +808,7 @@ impl Scheduler { ) -> Result> { // Fail closed without presence: a socket that never sent Hello (or whose // presence was reclaimed) gets nothing rather than the global pool. - let (agent_labels, concurrency, others_reserved) = { + let (agent_labels, concurrency, others_reserved, protocol) = { let mut agents = self.agents.write().await; match agents.get_mut(&agent_id) { Some(a) => { @@ -811,7 +816,7 @@ impl Scheduler { if !a.try_reserve_slot() { return Ok(None); } - (a.labels.clone(), a.concurrency, others) + (a.labels.clone(), a.concurrency, others, a.protocol_version) } None => { debug!(%agent_id, "no presence for agent; not offering"); @@ -820,7 +825,14 @@ impl Scheduler { } }; let leased = self - .lease_within_slots(agent_id, agent_labels, concurrency, others_reserved, skip) + .lease_within_slots( + agent_id, + agent_labels, + concurrency, + others_reserved, + protocol, + skip, + ) .await; // Leased or not, the reservation is done: a leased step is now `running` in // the database and counted from there. @@ -837,19 +849,22 @@ impl Scheduler { agent_labels: Vec, concurrency: u32, others_reserved: u32, + protocol: u32, skip: &[Uuid], ) -> Result> { let db_running = self.store.count_running_steps_for_agent(agent_id).await?; if !slots_available(concurrency, db_running, others_reserved) { return Ok(None); } - self.try_lease_for(agent_id, agent_labels, skip).await + self.try_lease_for(agent_id, agent_labels, protocol, skip) + .await } async fn try_lease_for( &self, agent_id: Uuid, agent_labels: Vec, + protocol: u32, skip: &[Uuid], ) -> Result> { // Pool scope is authoritative from the DB row, never from in-memory state. @@ -858,9 +873,20 @@ impl Scheduler { None => return Ok(None), }; + // Decided before the candidates are read, so an untrusted step this agent may + // not take is never leased, and never occupies a place in the scan window. + let include_untrusted = takes_untrusted_steps(agent_project_id, protocol); + if agent_project_id.is_some() && !include_untrusted { + // Debug: this is every fill of every such agent. The line an operator sees + // is the warning the session logs once, at `Hello`. + debug!( + %agent_id, protocol, + "agent predates the untrusted-offer protocol; fork pull request steps stay queued for another agent" + ); + } let queued = self .store - .list_queued_steps_for_pool(agent_project_id, &agent_labels) + .list_queued_steps_for_pool(agent_project_id, &agent_labels, include_untrusted) .await?; for step in queued { if skip.contains(&step.id) { @@ -1335,6 +1361,20 @@ pub fn disconnect_policy(hello: Option, goodbye: bool) -> DisconnectPolicy } } +/// Whether an agent may be offered steps of an untrusted run (a fork's pull request). +/// +/// Two conditions, both required. The agent is bound to a project — the query scopes it +/// to that project's runs, and the global pool never sees an untrusted step. And its +/// `Hello` declared [`fiber_proto::UNTRUSTED_OFFER_PROTOCOL`] or later (`hello` is `0` +/// from an agent older than the field): an agent below it ignores `untrusted` on the +/// offer and would run the fork's step on its host shell. +/// +/// False means ineligible, not failed: the step stays queued for an agent that +/// qualifies, which matters when an old and a new agent serve the same project. +pub fn takes_untrusted_steps(agent_project_id: Option, hello: u32) -> bool { + agent_project_id.is_some() && hello >= fiber_proto::UNTRUSTED_OFFER_PROTOCOL +} + /// Whether a message's `attempt` (echoed from the offer; `None` from older agents) /// refers to the attempt the row is on. A `step_run_id` is stable across attempts, so /// this is what keeps a line, artifact or completion that an agent held through a @@ -1420,6 +1460,7 @@ mod tests { concurrency, reserved, project_id: None, + protocol_version: fiber_proto::PROTOCOL_VERSION, } } @@ -1683,6 +1724,27 @@ mod tests { ); } + #[test] + fn only_a_current_project_agent_is_offered_untrusted_steps() { + let project = Some(Uuid::nil()); + let gate = fiber_proto::UNTRUSTED_OFFER_PROTOCOL; + // A project agent that honours `untrusted` on the offer. + assert!(takes_untrusted_steps(project, gate)); + assert!(takes_untrusted_steps(project, gate + 1)); + assert!(takes_untrusted_steps( + project, + fiber_proto::PROTOCOL_VERSION + )); + // One revision short, and an agent older than `protocol_version` itself (0): + // both would run a fork's step on the host, so neither is offered one. + assert!(!takes_untrusted_steps(project, gate - 1)); + assert!(!takes_untrusted_steps(project, 0)); + // The global pool never is, however new the agent. + assert!(!takes_untrusted_steps(None, gate)); + assert!(!takes_untrusted_steps(None, u32::MAX)); + assert!(!takes_untrusted_steps(None, 0)); + } + #[test] fn a_session_that_never_said_hello_touches_no_leases() { // A second socket for the same token that closes before Hello: the agent's diff --git a/deploy/.env.example b/deploy/.env.example index 7413fe9..06e6b09 100644 --- a/deploy/.env.example +++ b/deploy/.env.example @@ -119,7 +119,15 @@ FIBER_AGENT_CONCURRENCY=2 # ------------------------------------------------- docker agent (profile: agent-docker) # docker compose -f deploy/docker-compose.yml --profile agent-docker up -d fiber-agent-docker -# For pipelines with `image:` steps; uses the token and labels above. Bound at the same +# For pipelines with `image:` steps; uses the token above, and the labels above plus +# `docker=true`. It runs steps in containers only (FIBER_AGENT_ALLOW_HOST_SHELL is pinned +# to false in the Compose file, not read from here): a step without `image:` is refused, +# because on this agent it would run as root beside the Docker socket. That needs +# FIBER_VERSION at this release or later: the image is built FROM the published agent, and +# a 0.6.7 binary ignores the variable silently. Check the agent's startup log for a line +# beginning "steps run in containers only". `docker=true` lets an `image:` step ask for +# this agent; a step without `image:` stays off it by asking for `executor=shell`, which +# the plain agent's labels end with. Bound at the same # path inside the agent and on the Docker host. On Docker Desktop use a path inside its # VM, e.g. `docker volume create fiber_docker_ws` then # /var/lib/docker/volumes/fiber_docker_ws/_data (docs/agents.md). diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index 4b44849..6799406 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -281,7 +281,9 @@ services: FIBER_API_URL: ws://fiber-api:8080 FIBER_AGENT_TOKEN: ${FIBER_AGENT_TOKEN:-} FIBER_AGENT_NAME: ${FIBER_AGENT_NAME:-compose} - FIBER_AGENT_LABELS: ${FIBER_AGENT_LABELS:-os=linux} + # `executor=shell` is what a step without `image:` asks for to stay off the + # agent-docker profile's agent, which has every other label this one does. + FIBER_AGENT_LABELS: ${FIBER_AGENT_LABELS:-os=linux},executor=shell FIBER_AGENT_CONCURRENCY: ${FIBER_AGENT_CONCURRENCY:-2} RUST_LOG: ${RUST_LOG:-info,fiber_agent=info} # `image:` steps need the docker CLI and the socket, which this image deliberately @@ -299,7 +301,20 @@ services: # Optional worker for pipelines with `image:` steps: # `docker compose --profile agent-docker up -d fiber-agent-docker`. Same token and labels - # as fiber-agent (run one or the other). It gets the Docker socket — root on the host. + # as fiber-agent (run one or the other), plus `docker=true`. It gets the Docker socket — + # root on the host. + # + # Which is why FIBER_AGENT_ALLOW_HOST_SHELL is pinned to false here rather than read + # from .env: a step without `image:` would run in THIS container, as root, next to that + # socket. Such a step is refused; give it an `image:` or run it on the plain agent. + # The pin needs an agent binary that knows the variable: the image is built FROM + # fiber-agent:${FIBER_VERSION}, and 0.6.7 or older ignores it silently. Confirm with + # the startup log line "steps run in containers only". + # + # `docker=true` lets an `image:` step ask for this agent. It does NOT keep a step + # without `image:` away: a step matches any agent holding every label it asks for, and + # this agent holds all of the plain agent's. An imageless step opts out by asking for + # `executor=shell`, which only the plain agent has. # # The workspace is bound at the SAME absolute path inside this container and on the # Docker host. Steps run as `docker run -v /:/workspace`, and the daemon @@ -321,9 +336,10 @@ services: FIBER_API_URL: ws://fiber-api:8080 FIBER_AGENT_TOKEN: ${FIBER_AGENT_TOKEN:-} FIBER_AGENT_NAME: ${FIBER_AGENT_NAME:-compose} - FIBER_AGENT_LABELS: ${FIBER_AGENT_LABELS:-os=linux} + FIBER_AGENT_LABELS: ${FIBER_AGENT_LABELS:-os=linux},docker=true FIBER_AGENT_CONCURRENCY: ${FIBER_AGENT_CONCURRENCY:-2} FIBER_AGENT_USE_DOCKER: "true" + FIBER_AGENT_ALLOW_HOST_SHELL: "false" FIBER_AGENT_WORKSPACE_DIR: ${FIBER_AGENT_DOCKER_WORKSPACE:-/var/lib/fiber/workspaces} RUST_LOG: ${RUST_LOG:-info,fiber_agent=info} volumes: diff --git a/docs/agents.md b/docs/agents.md index b68f584..34cc8e8 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -6,7 +6,7 @@ An agent's identity is the token it connected with. The server binds `agent_id` A lease belongs to the agent, not to the socket. A step keeps running through a lost connection — an API restart, a proxy timeout, a network blip — and the agent renews its leases on its first heartbeat back; the server judges every late message by the row (is the step still `running` under this agent, on the attempt the message names?), never by which session it arrived on. Every `Offer` carries the attempt number and the agent echoes it on each log batch, artifact, and completion — and on the HTTP artifact routes, as `X-Fiber-Attempt` on the upload and an `attempt` field in the presign and complete bodies ([api](./api.md#agent-endpoints)) — so output an agent held through a reclaim can never be filed under, overwrite, or close the attempt that replaced it; a message whose attempt is not the row's is dropped with a warning, and an upload for one is refused with `401`. A log line the agent re-sends after an aborted write can appear twice: `log_lines` is append-only with no dedupe. See [Lifecycle](#lifecycle) for the limits. -`Hello` carries `protocol_version` (`fiber_proto::PROTOCOL_VERSION`, currently **2**; absent from older agents, read as `0`). The server logs it and uses it to tell an agent that keeps its steps across sessions (1 and up) from one that cancels them on any close (0), and a later server may refuse a revision it no longer supports. It is not yet stored on the agent row. `Welcome` carries the **server's** revision the same way, and the agent reads its absence as "older than `log_batch`" and sends one `log_chunk` per line to that server — so a rolling deploy in either order keeps a build's log, though **upgrade the API before the agents** remains the supported direction. Revision 2 adds batched log messages; see [Log path](#log-path). +`Hello` carries `protocol_version` (`fiber_proto::PROTOCOL_VERSION`, currently **3**; absent from older agents, read as `0`). The server logs it and uses it to tell an agent that keeps its steps across sessions (1 and up) from one that cancels them on any close (0), and a later server may refuse a revision it no longer supports. It is not yet stored on the agent row. `Welcome` carries the **server's** revision the same way, and the agent reads its absence as "older than `log_batch`" and sends one `log_chunk` per line to that server — so a rolling deploy in either order keeps a build's log, though **upgrade the API before the agents** remains the supported direction. Revision 2 adds batched log messages; see [Log path](#log-path). Revision 3 means the agent reads `untrusted` on an `Offer` and never runs a fork pull request's step on its host shell, and that a Docker agent refuses a step without `image:` unless told otherwise; see [Executor](#executor). `untrusted` is absent from an older server, which the agent reads as a trusted run. An older agent would ignore it, so the server gates on the revision (`fiber_proto::UNTRUSTED_OFFER_PROTOCOL`): a step of an untrusted run is offered only to a project agent whose `Hello` declared 3 or later, and otherwise stays `queued` — it is not leased and does not fail — until such an agent (0.6.8+) is online in the project's pool. The server warns once per session when a project agent below that revision connects. ## Pools @@ -79,7 +79,10 @@ Two details that are not decoration: Downloads the release binary for the host architecture, creates the `fiber` system user, writes `/etc/fiber/agent.env` (mode `0640`, root-owned so the token is not world-readable), installs [`deploy/fiber-agent.service`](../deploy/fiber-agent.service), and enables it. Add -`--docker` to allow `image:` steps (adds `fiber` to the `docker` group). `--uninstall` +`--docker` to allow `image:` steps (adds `fiber` to the `docker` group); an agent installed +that way runs steps in containers only and refuses one without `image:` — see +[Executor](#executor). A `FIBER_AGENT_ALLOW_HOST_SHELL` line you add to `agent.env` is +kept by the next run of the installer. `--uninstall` removes the service, keeping `/etc/fiber` and `/var/lib/fiber`. #### What it verifies @@ -275,7 +278,22 @@ docker compose -f deploy/docker-compose.yml --profile agent up -d fiber-agent Pipelines with `image:` steps need the `agent-docker` profile instead. It builds the published agent plus the Docker CLI locally (`deploy/agent-docker.Dockerfile`), mounts the -engine's socket, and runs as root, because the socket is root on the host anyway: +engine's socket, and runs as root, because the socket is root on the host anyway. For +that reason it runs steps **in containers only**: `FIBER_AGENT_ALLOW_HOST_SHELL` is pinned +to `false` in the Compose file, so a step without `image:` is refused rather than run as +root inside the agent's container, next to the socket. + +**The pin only works on an agent that knows the variable.** The profile's image is built +`FROM` the published `fiber-agent:${FIBER_VERSION}`, and a 0.6.7 or older binary ignores +`FIBER_AGENT_ALLOW_HOST_SHELL` without a word and runs imageless steps on its host as +before. Set `FIBER_VERSION` to this release or later, rebuild the service, and confirm it +took effect: the agent's startup log (`docker compose … logs fiber-agent-docker`) has a +line beginning `steps run in containers only`. No such line means the old binary. + +It registers with `FIBER_AGENT_LABELS` plus `docker=true`, and the plain `agent` profile +with `FIBER_AGENT_LABELS` plus `executor=shell`. See +[Steps that would run on the host](#steps-that-would-run-on-the-host) for what those two +labels do and do not route. ```bash echo "FIBER_AGENT_DOCKER_WORKSPACE=/var/lib/fiber/workspaces" >> deploy/.env @@ -309,6 +327,8 @@ export FIBER_AGENT_NAME=local export FIBER_AGENT_LABELS=os=linux,docker=true export FIBER_AGENT_CONCURRENCY=1 export FIBER_AGENT_USE_DOCKER=true # false = host shell +# With Docker on, a step without `image:` is refused. To run those on this host instead: +# export FIBER_AGENT_ALLOW_HOST_SHELL=true export FIBER_AGENT_WORKSPACE_DIR=./data/workspaces cargo run -p fiber-agent # or: cargo run -p fiber-cli -- agent --token "$FIBER_AGENT_TOKEN" @@ -420,8 +440,75 @@ repeat is identifiable. ## Executor -- **Shell** (`FIBER_AGENT_USE_DOCKER=false`): runs `run` in the step's workspace directory. -- **Docker**: if `image` is set and docker mode is on, runs inside the image with the workspace mounted, as a named container (`fiber-step-`) so cancel and timeout `docker kill` it rather than only the client. The image is checked against the docker reference grammar and passed after `--`, so a pipeline cannot smuggle a `docker run` flag through it. +- **Shell** (`FIBER_AGENT_USE_DOCKER=false`): runs `run` in the step's workspace directory, on the host, as the agent's user. A step's `image:` is ignored. +- **Docker** (`FIBER_AGENT_USE_DOCKER=true`, the binary's default): if `image` is set, runs inside the image with the workspace mounted, as a named container (`fiber-step-`) so cancel and timeout `docker kill` it rather than only the client. The image is checked against the docker reference grammar and passed after `--`, so a pipeline cannot smuggle a `docker run` flag through it. + +### Steps that would run on the host + +A step runs on the host shell when it has no `image:`, or when the agent is a shell agent. +The host shell is the agent's own user, with whatever that user can reach — on a Docker +agent that includes the Docker engine, which is root on the host. So the agent decides, +before it clones anything, whether a step may run there: + +| Step | Agent | `FIBER_AGENT_ALLOW_HOST_SHELL` | Run from a fork's pull request? | Outcome | +|---|---|---|---|---| +| has `image:` | Docker | any | either | container | +| no `image:` | Docker | unset or `false` | no | **refused** | +| no `image:` | Docker | `true` | no | host shell | +| no `image:` | Docker | any | yes | **refused** | +| either | shell | unset or `true` | no | host shell | +| either | shell | any | yes | **refused** | + +`FIBER_AGENT_ALLOW_HOST_SHELL` defaults to `true` on a shell agent, where the host is all +there is, and to `false` on a Docker agent. Set it to `true` on a Docker agent to run +steps without `image:` on its host — that is the operator saying everyone who can write +this agent's pipelines may act as the agent's user. It accepts `true`, `false`, `1` and +`0`; anything else stops the agent at startup, and so does `false` on a shell agent, which +could then run nothing. + +A fork's pull request is never run on the host shell, and the setting does not change +that: its author is not someone the operator vouched for. Its steps need an `image:` and a +project agent with Docker on. On a shell agent they fail, by design. The server offers +them only to a project agent at 0.6.8 or later (protocol revision 3), since an older one +would not know to refuse; until one is online they wait in `queued` — see +[triggers](./triggers.md#pull-requests-from-forks). + +A refused step fails, and a retry on the same agent fails the same way. With `retries: N` +a refused step is leased again up to N times, each to whichever matching agent has a free +slot — so in a pool that mixes agents it may be refused N more times, or land on an agent +that accepts it and pass. Its log and its `error` say which case it was and what to +change; nothing is cloned or restored first: + +```text +this agent runs steps in containers only: add `image:` to the step, or set FIBER_AGENT_ALLOW_HOST_SHELL=true on the agent +fork pull request steps only run in containers: this run builds code from outside the project and the step has no `image:` … +fork pull request steps only run in containers, and this is a shell agent (FIBER_AGENT_USE_DOCKER=false) … +``` + +Labels decide which agent a step reaches, and a step matches an agent that has **every** +label the step asks for — extra labels on the agent do not exclude it. So a label that +only the Docker agent carries (`docker=true`, which the Compose `agent-docker` profile +adds) lets an `image:` step opt **in** to it; it does not keep a step without `image:` +away, because that step asks for nothing the Docker agent lacks. To keep it away, the +shell agent needs a label of its own that the Docker agent does not have, and the step +has to ask for it. The Compose `agent` profile adds `executor=shell` for this: + +```yaml +steps: + build: + image: rust:1-bookworm + labels: [os=linux, docker=true] # only a Docker agent + notify: + run: ./scripts/notify.sh + labels: [os=linux, executor=shell] # only a shell agent +``` + +A step that asks for neither can be leased by either kind of agent, and fails if the +Docker agent gets it first. Agents you start yourself carry only the labels you give them +in `FIBER_AGENT_LABELS`. + +Releases up to 0.6.7 ran a step without `image:` on a Docker agent's host without being +asked, and ran a fork's pull request on the host shell like any other run. Process groups: cancel kills the step's process group so grandchildren die too. @@ -471,7 +558,7 @@ Steps run repo-supplied shell, so the agent narrows what is reachable: | **Container limits** | `--security-opt no-new-privileges` and `--pids-limit 512` by default, plus `--user`, `--network`, `--memory`, `--cpus` from `FIBER_AGENT_DOCKER_*`. Memory and CPU limits are off unless set; whatever applies is printed as a `system` log line so an exit 137 is diagnosable | | **Workspace** | One directory per step, deleted when the step finishes — including on cancel, timeout, or failure. The run's tree goes when its last step on this agent finishes; anything older than `FIBER_AGENT_WORKSPACE_TTL_HOURS` (24) is swept at startup | | **Artifacts** | Only those produced by the steps this one transitively `needs` — enforced when the offer is built *and* again when the agent asks to download one | -| **Untrusted runs** | A fork's pull request is offered only to agents bound to that project, never to the global pool, and receives no secrets — see [triggers](./triggers.md#pull-requests-from-forks) | +| **Untrusted runs** | A fork's pull request is offered only to agents bound to that project, never to the global pool, receives no secrets, and runs in a container or not at all ([Executor](#steps-that-would-run-on-the-host)) — see [triggers](./triggers.md#pull-requests-from-forks) | A step with a git workspace is cloned from one reference clone per run, so a second step costs a local object copy rather than another fetch, and the checkout is self-contained diff --git a/docs/configuration.md b/docs/configuration.md index 301e184..336a469 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -94,7 +94,7 @@ file passes the variable through as an empty string and each reader falls back. | `FIBER_METRICS_TOKEN` | empty | Bearer token for `/metrics`; empty = off | | `OTEL_EXPORTER_OTLP_ENDPOINT` | empty | OTLP collector base URL. Use this name in Compose, not `FIBER_OTEL_ENDPOINT`: the API reads the alias only when this one is *absent*, and Compose always defines what it passes | | `RUST_LOG` | `info,fiber_api=info` | API log filter | -| `FIBER_AGENT_TOKEN` / `_NAME` / `_LABELS` / `_CONCURRENCY` | — / `compose` / `os=linux` / `2` | The optional `agent` profile | +| `FIBER_AGENT_TOKEN` / `_NAME` / `_LABELS` / `_CONCURRENCY` | — / `compose` / `os=linux` / `2` | The optional `agent` profile. Compose appends `executor=shell` to the labels there, and `docker=true` on the `agent-docker` profile | ## fiber-agent @@ -106,6 +106,7 @@ file passes the variable through as an empty string and each reader falls back. | `FIBER_AGENT_LABELS` | `os=linux,docker=true` | Comma-separated | | `FIBER_AGENT_CONCURRENCY` | `1` | Max parallel steps | | `FIBER_AGENT_USE_DOCKER` | `true` | Docker vs host shell | +| `FIBER_AGENT_ALLOW_HOST_SHELL` | `false` with Docker, `true` without | Whether a step may run on the agent's host shell. On a Docker agent the default refuses a step without `image:`; `true` runs it on the host as the agent's user, which can reach the Docker engine. `true`, `false`, `1` or `0` — anything else, or `false` on a shell agent, stops the agent at startup. A fork pull request's steps are never run on the host shell, whatever this is set to. [agents](./agents.md#steps-that-would-run-on-the-host) | | `FIBER_AGENT_WORKSPACE_DIR` | `./data/workspaces` | Root for per-step work dirs | | `FIBER_AGENT_ENV_PASSTHROUGH` | empty | Comma-separated names to pass from the agent's environment into steps, on top of the built-in allowlist (shell basics, proxy and CA settings). Everything else is cleared — use this for `SSH_AUTH_SOCK`, `CARGO_HOME`, `JAVA_HOME`, `NVM_DIR` and similar. **Under systemd, `SSH_AUTH_SOCK` needs a drop-in** — see below | | `FIBER_AGENT_WORKSPACE_TTL_HOURS` | `24` | Sweep run workspaces older than this at startup; `0` disables | @@ -118,7 +119,10 @@ file passes the variable through as an empty string and each reader falls back. Installed by `scripts/install-agent.sh` into `/etc/fiber/agent.env` (root-owned, mode `0640`); in Compose they come from `deploy/.env` (`FIBER_AGENT_TOKEN`, `FIBER_AGENT_NAME`, -`FIBER_AGENT_LABELS`, `FIBER_AGENT_CONCURRENCY`, `FIBER_AGENT_USE_DOCKER`). The installer also writes +`FIBER_AGENT_LABELS`, `FIBER_AGENT_CONCURRENCY`, `FIBER_AGENT_USE_DOCKER`). The Compose +`agent-docker` profile pins `FIBER_AGENT_ALLOW_HOST_SHELL=false` and appends `docker=true` +to the labels, and the `agent` profile appends `executor=shell`; the installer writes `FIBER_AGENT_ALLOW_HOST_SHELL` only to carry over a +line already in `agent.env`. The installer also writes `FIBER_AGENT_INSTALLED_VERSION` and `FIBER_AGENT_INSTALLED_SHA256` there so a re-run can say what it is replacing; no crate reads them, and they are not deployment configuration. diff --git a/docs/pipeline-yaml.md b/docs/pipeline-yaml.md index bf195c8..2b950a4 100644 --- a/docs/pipeline-yaml.md +++ b/docs/pipeline-yaml.md @@ -104,7 +104,7 @@ API replicas still leave exactly one run in flight. | `run` | — | Shell script (required for execution) | | `needs` | `[]` | Upstream step ids (DAG edges) | | `labels` | `[]` | Matched against agent labels (all required labels must be present) | -| `image` | — | Optional Docker image; agent runs in container when Docker enabled. A docker image reference only (`name[:tag][@digest]`, optionally registry-qualified): the value becomes a `docker run` argument, so anything docker could read as a flag (`-v/:/host`, `--privileged`) is rejected when the pipeline compiles and again by the agent, which also passes it after `--` | +| `image` | — | Docker image to run the step in, on an agent with Docker enabled; a shell agent ignores it and runs on its host. **Without it the step runs on the agent's host shell, which a Docker agent refuses by default and every agent refuses for a fork's pull request** — see [agents](./agents.md#steps-that-would-run-on-the-host). A docker image reference only (`name[:tag][@digest]`, optionally registry-qualified): the value becomes a `docker run` argument, so anything docker could read as a flag (`-v/:/host`, `--privileged`) is rejected when the pipeline compiles and again by the agent, which also passes it after `--` | | `retries` | `0` | Extra attempts after failure (exponential backoff `2^attempt` s, max 60 s, persisted on the row). An attempt lost to an agent disconnect or lease expiry also counts, and gets one extra try: a step is failed once it has lost more than `retries + 1` leases | | `timeout_minutes` | server default (60) | Per-attempt wall-clock limit incl. workspace prep and restores. The agent kills the step and fails the attempt (retries still apply); the server independently fails it `FIBER_STEP_TIMEOUT_GRACE_MINUTES` later if the agent did not | | `secrets` | all | Project secrets to inject, by name. Omit for every secret (the default); `secrets: []` for none. Naming them keeps credentials out of steps that have no use for them, which matters most for a step running a third-party `image:` | diff --git a/docs/triggers.md b/docs/triggers.md index fa2e6d3..432fe0a 100644 --- a/docs/triggers.md +++ b/docs/triggers.md @@ -86,21 +86,45 @@ access to the fork. A fork's pull request is code written by someone outside the project, while the pipeline and its secrets belong to the base repository. Fiber marks such a run **untrusted** -(`runs.untrusted`) and applies two controls: +(`runs.untrusted`) and applies three controls: 1. **No project secrets.** No step of the run receives any, whatever its `secrets:` says. Provenance that cannot be determined from the payload counts as untrusted, and a retry of an untrusted run stays untrusted. 2. **Project-dedicated agents only.** Untrusted steps are never offered to the global - pool — only to an agent bound to that project. If the project has no such agent, the - steps stay queued. - -**These reduce the blast radius; they do not sandbox the code.** A step still executes as -the agent's user, so on a shared agent it could read that agent's token out of `/proc` and -reach other projects' work. Run untrusted pull requests only on an agent you are willing -to treat as compromised: dedicated to the project, disposable, and ideally running every -step in a container (`image:`), which is what keeps the agent's own environment out of -reach. + pool — only to an agent bound to that project, and only to one new enough to apply + the third control (0.6.8 or later). If the project has no such agent, the steps stay + queued. +3. **Containers only.** The offer tells the agent the run is untrusted, and the agent + never runs such a step on its host shell: a step with an `image:`, on an agent with + Docker on, runs in its container; anything else **fails** before a line of the fork's + code is checked out. `FIBER_AGENT_ALLOW_HOST_SHELL=true` does not change this — that + setting is the operator vouching for the people who can write the pipeline, and a + fork's author is not one of them. + +So a pipeline that builds fork pull requests needs an `image:` on every step and a +project agent with `FIBER_AGENT_USE_DOCKER=true`. Two consequences worth knowing before +the first fork PR arrives: + +- **On a shell agent (`FIBER_AGENT_USE_DOCKER=false`) a fork's pull request fails**, + every step, with `fork pull request steps only run in containers, and this is a shell + agent …` in the step's log and error. That is intended: a shell agent has nowhere to put + the code except its own user account. Same-repository pull requests are unaffected. +- A step without `image:` fails the same way on a Docker agent, even one that allows host + shell for the project's own runs. + +The third control is carried out by the agent, and an agent older than 0.6.8 (protocol +revision below 3) does not know the flag — it would run the step on its host. So the +server does not offer it one: **a fork pull request's steps wait in `queued` until an +agent at 0.6.8 or later is online in the project's pool.** Nothing fails and nothing is +leased in the meantime; an older agent in the same pool keeps taking the project's own +runs, and the server logs one warning when such an agent connects. If a fork's pull +request sits queued with a project agent online, check that agent's version. + +**These reduce the blast radius; they do not sandbox the code.** A container shares the +host's kernel, has the network unless `FIBER_AGENT_DOCKER_NETWORK=none`, and writes to a +workspace the agent later reads. Run untrusted pull requests only on an agent you are +willing to treat as compromised: dedicated to the project and disposable. A step that needs a secret will fail on a fork's pull request. That is the intended outcome — do the privileged part on `push` to a branch you control, after review. diff --git a/scripts/dev-env.sh b/scripts/dev-env.sh index eaadd83..e193763 100755 --- a/scripts/dev-env.sh +++ b/scripts/dev-env.sh @@ -44,6 +44,10 @@ export FIBER_AGENT_WORKSPACE_DIR="${FIBER_AGENT_WORKSPACE_DIR:-$ROOT/data/worksp # export FIBER_AGENT_ENV_PASSTHROUGH=SSH_AUTH_SOCK # export FIBER_AGENT_DOCKER_NETWORK=none export FIBER_AGENT_USE_DOCKER="${FIBER_AGENT_USE_DOCKER:-false}" +# Left unset on purpose: the agent's default follows FIBER_AGENT_USE_DOCKER (host shell +# allowed for a shell agent, refused for a Docker one). Set it only to run steps without +# `image:` on a Docker agent. +# export FIBER_AGENT_ALLOW_HOST_SHELL=true export FIBER_AGENT_NAME="${FIBER_AGENT_NAME:-local}" export FIBER_AGENT_LABELS="${FIBER_AGENT_LABELS:-os=linux}" export FIBER_AGENT_CONCURRENCY="${FIBER_AGENT_CONCURRENCY:-1}" diff --git a/scripts/install-agent.sh b/scripts/install-agent.sh index 1cb2e57..1884014 100755 --- a/scripts/install-agent.sh +++ b/scripts/install-agent.sh @@ -57,6 +57,10 @@ NAME="$(hostname -s 2>/dev/null || hostname 2>/dev/null || echo fiber-agent)" LABELS="os=linux" CONCURRENCY="2" USE_DOCKER="false" +# Never set by a flag: the agent's own default is the safe one (--docker refuses a step +# without `image:`). Only carried over from an existing agent.env, so an upgrade does not +# undo a choice the operator made there. +ALLOW_HOST_SHELL="" BIN_DIR="/usr/local/bin" ETC_DIR="/etc/fiber" STATE_DIR="/var/lib/fiber" @@ -75,7 +79,10 @@ Usage: install-agent.sh --api-url --token [options] --name NAME Agent display name [default: hostname] --labels LIST Comma-separated labels [default: os=linux] --concurrency N Max parallel steps [default: 2] - --docker Enable Docker steps (needs docker + group membership) + --docker Enable Docker steps (needs docker + group membership). The agent + then runs steps in containers only: one without `image:` is + refused unless FIBER_AGENT_ALLOW_HOST_SHELL=true is added to + /etc/fiber/agent.env. --version VER Release tag to install [default: latest] --tarball PATH Install from a local release tarball instead of downloading --check-only Resolve, download and verify a release, then exit (no root). @@ -527,8 +534,11 @@ fi PREV_VERSION="unknown" # Re-running is the upgrade path: keep settings that were not passed this time. if [[ -f "$ETC_DIR/agent.env" ]]; then + # From the file only, never from whatever environment this script was started in. + unset FIBER_AGENT_ALLOW_HOST_SHELL # shellcheck disable=SC1091 . "$ETC_DIR/agent.env" + ALLOW_HOST_SHELL="${FIBER_AGENT_ALLOW_HOST_SHELL:-}" API_URL="${API_URL:-${FIBER_API_URL:-}}" TOKEN="${TOKEN:-${FIBER_AGENT_TOKEN:-}}" [[ "$NAME" == "$(hostname -s 2>/dev/null || echo "$NAME")" ]] && NAME="${FIBER_AGENT_NAME:-$NAME}" @@ -595,6 +605,10 @@ RUST_LOG=info,fiber_agent=info FIBER_AGENT_INSTALLED_VERSION=$INSTALLED_FROM FIBER_AGENT_INSTALLED_SHA256=$BIN_SHA256 ENV +if [[ -n "$ALLOW_HOST_SHELL" ]]; then + echo "FIBER_AGENT_ALLOW_HOST_SHELL=$ALLOW_HOST_SHELL" >> "$ETC_DIR/agent.env" + echo "keeping FIBER_AGENT_ALLOW_HOST_SHELL=$ALLOW_HOST_SHELL from the previous agent.env" +fi chown root:fiber "$ETC_DIR/agent.env" chmod 0640 "$ETC_DIR/agent.env" umask 022 diff --git a/scripts/smoke_runs.py b/scripts/smoke_runs.py index 51c87d5..96d62b4 100644 --- a/scripts/smoke_runs.py +++ b/scripts/smoke_runs.py @@ -5,7 +5,9 @@ Needs an agent already online with `os=linux` (CI's smoke-host starts one; locally see docs/development.md), and `psql` for the one check that has to break a stored definition the API would refuse to save. The fork-PR check starts its own project-scoped agent from -`target/debug/fiber-agent` and stops it by PID afterwards. +`target/debug/fiber-agent` and stops it by PID afterwards — once as a shell agent, which +must refuse the run, and once as a Docker agent (when `docker info` works; it pulls +`alpine:3`), which must run it in a container with no secrets. """ from __future__ import annotations @@ -224,10 +226,13 @@ def main() -> None: req("POST", f"/api/projects/{pid}/secrets", token, {"key": "SMOKE_SECRET", "value": "s3cret"}) pr_secret = "smoke-runs-pr" req("PUT", f"/api/projects/{pid}/webhooks/github", token, {"secret": pr_secret}) + # `image:` because a fork's pull request runs in a container or not at all. A shell + # agent ignores it, so the same-repository run below still runs on the host. create_pipeline(token, pid, "pr", { "name": "pr", "on": {"pull_request": {"branches": ["main"]}}, "steps": [{"id": "probe", "name": "probe", "needs": [], "labels": labels, + "image": "alpine:3", "run": '[ -n "$SMOKE_SECRET" ] && echo has-secret || echo no-secret'}], }) # Same repository: trusted, so any agent runs it and the secret is there. This is @@ -249,23 +254,59 @@ def main() -> None: code, scoped = req("POST", "/api/agents", token, {"name": "smoke-runs-scoped", "labels": labels, "concurrency": 1, "project_id": pid}) check("create a project-scoped agent", code == 201, scoped) - env = dict(os.environ, FIBER_AGENT_TOKEN=scoped["token"], FIBER_API_URL=WS, - FIBER_AGENT_USE_DOCKER="false", FIBER_AGENT_LABELS=",".join(labels), - FIBER_AGENT_NAME="smoke-runs-scoped", - FIBER_AGENT_WORKSPACE_DIR=os.path.join(tempfile.gettempdir(), "fiber-smoke-runs-ws")) - agent = subprocess.Popen([os.path.join(ROOT, "target/debug/fiber-agent")], env=env, - stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - try: - fork = wait_run(token, fork_run) - probe = steps_by_id(fork)["probe"] - check("the project's own agent runs it, without the project's secrets", - fork["run"]["status"] == "succeeded" - and any("no-secret" in l["data"] for l in log_lines(token, probe["id"])), - [l["data"] for l in log_lines(token, probe["id"])]) - finally: + + def scoped_agent(use_docker: bool) -> subprocess.Popen: + env = dict(os.environ, FIBER_AGENT_TOKEN=scoped["token"], FIBER_API_URL=WS, + FIBER_AGENT_USE_DOCKER="true" if use_docker else "false", + FIBER_AGENT_LABELS=",".join(labels), FIBER_AGENT_NAME="smoke-runs-scoped", + FIBER_AGENT_WORKSPACE_DIR=os.path.join(tempfile.gettempdir(), "fiber-smoke-runs-ws")) + # Whatever the caller's shell exports, this agent takes the default for its mode. + env.pop("FIBER_AGENT_ALLOW_HOST_SHELL", None) + return subprocess.Popen([os.path.join(ROOT, "target/debug/fiber-agent")], env=env, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + + def stop(agent: subprocess.Popen) -> None: # By PID, never by name: a pattern can match the shell that started this. agent.send_signal(signal.SIGTERM) agent.wait(timeout=30) + + try: + # A shell agent has nowhere to run a fork's code but its own account, so it + # refuses — even though the step names an image, which a shell agent ignores. + agent = scoped_agent(use_docker=False) + try: + fork = wait_run(token, fork_run) + finally: + stop(agent) + probe = steps_by_id(fork)["probe"] + check("the project's shell agent refuses to run a fork's PR on its host", + fork["run"]["status"] == "failed" + and "fork pull request steps only run in containers" in (probe.get("error") or ""), + (fork["run"]["status"], probe.get("error"))) + + # In a container it runs, and without the project's secrets. Needs an engine, + # which CI's smoke host has; skipped out loud where there is none. + try: + have_docker = subprocess.run(["docker", "info"], stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, timeout=30).returncode == 0 + except (OSError, subprocess.TimeoutExpired): + have_docker = False + if have_docker: + fork_run = pull_request(pid, pr_secret, "mallory/fork", 43)["started"][0] + agent = scoped_agent(use_docker=True) + try: + # Longer than the default: the first run pulls the image. + fork = wait_run(token, fork_run, timeout=240) + finally: + stop(agent) + probe = steps_by_id(fork)["probe"] + check("the project's docker agent runs it in a container, without the project's secrets", + fork["run"]["status"] == "succeeded" + and any("no-secret" in l["data"] for l in log_lines(token, probe["id"])), + [l["data"] for l in log_lines(token, probe["id"])]) + else: + print("SKIP the project's docker agent runs a fork's PR in a container: no docker engine here") + finally: req("DELETE", f"/api/agents/{scoped['agent']['id']}", token) # ---- one durable fiber runs to completion ------------------------------------