From 134460fde7118b271435b6ea9f9cdd9e24449801 Mon Sep 17 00:00:00 2001 From: Adnan Khan Date: Sat, 14 Nov 2026 06:14:51 +0000 Subject: [PATCH] [#32] Bound the size of repository checkouts for builds --- docs/SECURITY.md | 7 +- docs/entity-diagram.md | 2 + docs/manual/en/03-roles.md | 4 +- docs/manual/en/05-builds.md | 8 +- docs/manual/en/08-workflows.md | 6 +- docs/manual/en/72-architecture.md | 2 + .../java/ai/mnemosyne/cisys/BuildRunner.java | 32 ++- .../java/ai/mnemosyne/cisys/Checkout.java | 206 ++++++++++++++++++ .../java/ai/mnemosyne/cisys/Settings.java | 4 + .../ai/mnemosyne/cisys/SettingsResource.java | 23 +- .../java/ai/mnemosyne/cisys/Workflow.java | 43 ++-- .../main/resources/application.properties | 3 + .../main/resources/db/migration/R__schema.sql | 4 + .../java/ai/mnemosyne/cisys/CheckoutTest.java | 128 +++++++++++ .../mnemosyne/cisys/SettingsResourceTest.java | 34 +++ .../java/ai/mnemosyne/cisys/WorkflowTest.java | 30 +-- src/frontend/src/SettingsPanel.jsx | 7 + 17 files changed, 465 insertions(+), 78 deletions(-) create mode 100644 src/backend/main/java/ai/mnemosyne/cisys/Checkout.java create mode 100644 src/backend/test/java/ai/mnemosyne/cisys/CheckoutTest.java diff --git a/docs/SECURITY.md b/docs/SECURITY.md index bb436ec..2e720d7 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -32,6 +32,7 @@ The Keycloak admin console () uses `admin` / `admin | `PUT /api/branding` | no | no | yes | | `GET`/`PUT /api/settings/retention` | no | no | yes | | `GET`/`PUT /api/settings/queue` | no | no | yes | +| `GET`/`PUT /api/settings/checkout` | no | no | yes | | `GET /api/images` (allowed images) | yes | yes | yes | | `PUT /api/images` | no | no | yes | | `GET/POST/DELETE /api/images/local` | no | no | yes | @@ -70,7 +71,8 @@ Controls that are in place * Only HTTPS repositories on an allowed host are accepted for builds and branch discovery; redirects, embedded credentials, query strings, fragments and non-default ports are rejected. See [Repository addresses](#repository-addresses). * Branch discovery for private projects is not attempted because credentials are not stored on the project; the endpoint returns an empty list with reason `private_repository`. * Branch names are validated as Git references. Builds may only use images an admin has allowed; image names are validated against a restrictive pattern. -* The branch is fetched afresh with JGit at depth 1 to read its workflow file and commit ID. The build then builds an image from the workflow: it installs `git`, `make`, `podman` and `podman-compose` and the workflow's `dependencies` as root, with network access, clones the branch at that commit as `65534:65534`, and runs the steps as `65534:65534` without network. The installs and the clone have network access. +* The branch is fetched afresh with JGit at depth 1 and its commit ID is recorded. The build then builds an image from the workflow: it installs `git`, `make`, `podman` and `podman-compose` and the workflow's `dependencies` as root, with network access, copies that checkout into the image as `65534:65534`, and runs the steps as `65534:65534` without network. Only the installs have network access; the image build does not fetch the repository again. +* A checkout is limited to `BUILD_MAX_CHECKOUT_MIB` (default 1024 MiB) and `BUILD_MAX_CHECKOUT_FILES` (default 100000 files); admins can change both under Settings. The received pack is capped while it is fetched, the files are measured as they will be written, after line ending conversion, before checkout, and the written working tree is measured again after checkout. A checkout over a limit fails the build with `Build failed: Repository checkout exceeds the limit of …` and its directory is removed. * The image build runs with `no-new-privileges`, 1 GiB of memory and one CPU under Podman; Docker applies neither limit. The image is removed when the build ends. * Builds run up to the concurrency limit (default 1, at most 16 when set by an admin); further requests wait in a bounded queue. Each build times out after ten minutes. Output is stored in full, up to the `BUILD_LOG_MAX_BYTES` safety limit (default 1 GiB); output beyond it is discarded and the build is marked truncated. * Builds left in the `RUNNING` state by a server restart are marked `FAILED` at startup. Startup also removes leftover `cisys-build-*` containers and images and `cisys-*` checkout directories from an unclean stop; a cleanup failure is logged and does not stop startup. @@ -84,14 +86,13 @@ Controls that are in place Known gaps -* Repository checkout size is not bounded. * Runner and Podman operation on Windows/WSL has not been verified. * Credentials are not scoped to individual projects; only admins may reference one. * DNS rebinding between the address check and JGit's own connection is not fully prevented, because JGit resolves the host again when it connects. ### Git credentials -For private repositories set an environment variable named `CISYS_GIT_` to a read-only provider token in the backend environment before starting it, then enter the variable name in the build form as an admin. The value is never stored in the database. The image build receives it as a build secret mounted for the clone step only; it is not stored in the image and later steps cannot read it. Never place secrets in workflow files or build output; each build stores its workflow file. +For private repositories set an environment variable named `CISYS_GIT_` to a read-only provider token in the backend environment before starting it, then enter the variable name in the build form as an admin. The value is never stored in the database. Only the backend's JGit fetch uses it; the image build does not receive it, so it is not in the image and steps cannot read it. Never place secrets in workflow files or build output; each build stores its workflow file. ## Public access limits diff --git a/docs/entity-diagram.md b/docs/entity-diagram.md index 14c8321..b57c33e 100644 --- a/docs/entity-diagram.md +++ b/docs/entity-diagram.md @@ -46,6 +46,8 @@ erDiagram INTEGER retention_max_builds INTEGER max_concurrent_builds INTEGER max_queued_builds + INTEGER max_checkout_mib + INTEGER max_checkout_files } PROJECT ||--o{ BUILD : has diff --git a/docs/manual/en/03-roles.md b/docs/manual/en/03-roles.md index e88fb64..5e689a1 100644 --- a/docs/manual/en/03-roles.md +++ b/docs/manual/en/03-roles.md @@ -14,7 +14,7 @@ The **Lead** role is for people who run projects. Leads can do everything a memb ## Admin -The **Admin** role manages the installation. Admins can see every project including private projects of other users, start and pin builds, use credential references for private Git repositories, manage the build images, change the branding, and change the settings such as build retention and the build queue. +The **Admin** role manages the installation. Admins can see every project including private projects of other users, start and pin builds, use credential references for private Git repositories, manage the build images, change the branding, and change the settings such as build retention, the build queue and the checkout limits. ## Access summary @@ -30,7 +30,7 @@ The **Admin** role manages the installation. Admins can see every project includ | Use a Git credential reference | no | no | yes | | Manage build images | no | no | yes | | Change branding | no | no | yes | -| Change settings (retention, queue) | no | no | yes | +| Change settings | no | no | yes | "View" always means projects the user may access. The address of a project or build that does not exist or is not visible to the user opens the front page instead, so a private project is not revealed; the same applies to any unknown address. diff --git a/docs/manual/en/05-builds.md b/docs/manual/en/05-builds.md index 78daaca..246b772 100644 --- a/docs/manual/en/05-builds.md +++ b/docs/manual/en/05-builds.md @@ -17,13 +17,15 @@ Members can view build history and logs for the projects they can access. Privat ## How a build runs -1. The branch is fetched afresh with JGit at depth 1, so a force push is reflected in the next build, and its commit ID is recorded. +1. The branch is fetched afresh with JGit at depth 1, so a force push is reflected in the next build, and its commit ID is recorded. The checkout must stay within the size and file limits. 2. The branch's `.cisys/workflows/ci.yml` is read and stored with the build, then validated. A branch without the file, an invalid file, or a `container` image that is not on the build host or that an admin has not allowed on the **Images** page fails the build with a message in the log. -3. The workflow becomes a Containerfile. Podman builds it with `no-new-privileges`, 1 GiB of memory, one CPU and no layer cache. The image installs `git`, `make`, `podman` and `podman-compose` if they are missing and the workflow's `dependencies`, clones the branch at the recorded commit, and runs the steps as `65534:65534` without network. See **Workflows**. +3. The workflow becomes a Containerfile. Podman builds it with `no-new-privileges`, 1 GiB of memory, one CPU and no layer cache. The image installs `git`, `make`, `podman` and `podman-compose` if they are missing and the workflow's `dependencies`, copies the checkout from step 1, and runs the steps as `65534:65534` without network. See **Workflows**. 4. The status is `PASSED` when the image build succeeds, otherwise `FAILED` with the exit code of the failing step. A build that exceeds ten minutes is stopped and reported as timed out. The image is removed afterwards. Build requests are stored as `QUEUED` and claimed in request order. By default one build runs at a time. Set `BUILD_CONCURRENCY` (`cisys.build.concurrency`) to change the number of concurrent builds and `BUILD_QUEUE_LIMIT` (`cisys.build.queue-limit`, default 100) to limit waiting requests. Both limits must be positive. Admins can change both under **Settings** → **Queue** (1–16 concurrent builds, 1–1000 queued builds); saved values replace the configured ones. A full queue returns HTTP 429 with a clear error. Requests keep their repository URL, branch and credential reference; credentials themselves are read from the environment when execution begins. +A checkout is limited to `BUILD_MAX_CHECKOUT_MIB` (`cisys.build.max-checkout-mib`, default 1024) MiB and `BUILD_MAX_CHECKOUT_FILES` (`cisys.build.max-checkout-files`, default 100000) files. Both limits must be positive. Admins can change both under **Settings** → **Checkout** (1–102400 MiB, 1–1000000 files); saved values replace the configured ones. The fetched data is checked during the fetch, the files are measured as they will be written, after line ending conversion such as `eol=crlf` in `.gitattributes`, before checkout, and the working tree is measured again after checkout. The image copies this checkout and does not fetch the branch again. A repository over a limit fails the build with `Build failed: Repository checkout exceeds the limit of N MiB` (or `N files`) in the log, and the checkout directory is removed. + If saving a completed result fails, the runner retains it in memory and retries on each dispatcher tick without rerunning the build. Its slot stays reserved until the result is saved. On restart, interrupted `RUNNING` builds become `FAILED` and retain their partial logs. `QUEUED` builds remain queued and run when a slot is available. Startup also removes leftover `cisys-build-*` containers and images and `cisys-*` checkout directories from an unclean stop; a cleanup failure is logged and does not stop startup. Queued and running builds are protected from retention cleanup and prevent project deletion. This runner supports one backend instance; startup recovery assumes the previous instance has stopped. @@ -67,7 +69,7 @@ Only `github.com` and `gitlab.com` HTTPS URLs are accepted for builds. Redirects Set an environment variable whose name starts with `CISYS_GIT_`, for example `CISYS_GIT_PROJECT`, to a read-only provider token in the backend environment before starting it. Then, as an admin, enter that variable name in the build form. -* The value is never stored in the project database. The image build receives it as a build secret for the clone step only; it is not stored in the image and later steps cannot read it. +* The value is never stored in the project database. Only the backend's fetch uses it; the image build does not receive it, so it is not stored in the image and steps cannot read it. * Only admins can use credential references until credentials are scoped to individual projects. * Never include secrets in workflow files or build output. diff --git a/docs/manual/en/08-workflows.md b/docs/manual/en/08-workflows.md index ae1b0d2..3683bd8 100644 --- a/docs/manual/en/08-workflows.md +++ b/docs/manual/en/08-workflows.md @@ -131,7 +131,7 @@ The build finds out what the container is and installs with its package manager: * `value` is one package: letters, digits and `+ . _ : = ~ @ -`, starting with a letter or digit, at most 200 characters. Options such as `-y`, spaces and shell characters are rejected. * `name` is required, at most 200 characters, and appears once in the list. * Packages are installed as root, with network access. The steps themselves still run as `65534:65534` without network. -* All dependencies are installed together, in one step, before the branch is cloned, so every job and step can use them. `dependencies` on a job or a step is rejected. +* All dependencies are installed together, in one step, before the checkout is copied in, so every job and step can use them. `dependencies` on a job or a step is rejected. * A package that does not exist, or a container without a supported package manager, fails the build with `==> Could not install …` or `==> Cannot install dependencies …`. * Dependencies are always up to date. The package manager reads fresh package metadata on every build, never what the image cached, and installs the newest version. A package the image already has is upgraded. The log shows the version each one ended up at, for example `==> Installed: jq 1.7.1-6+deb13u4`. To stay on one version, give it in `value`. * Packages are downloaded on every build. A `container` image that already has them starts faster. @@ -173,7 +173,7 @@ A build turns the workflow into a Containerfile and builds an image from it with 1. **Read.** The backend fetches the branch and records its commit ID. It reads `.cisys/workflows/ci.yml` and stores the text with the build, then validates it. 2. **Base.** `container` is looked up among the local images, and the image starts `FROM` its ID. Nothing is pulled. 3. **Tools and dependencies.** `git`, `make`, `podman` and `podman-compose` are installed as root if the image lacks them, then the workflow's `dependencies`. This step has network access. -4. **Checkout.** As the unprivileged user `65534:65534`, the branch is cloned into `/workspace` at depth 1. The build then checks that the clone is at the recorded commit; if the branch moved in the meantime, the build fails. The working tree is **on the branch being tested**, with its `.git` directory, so `git status`, `git log -1` and `git rev-parse HEAD` work. `HOME` is `/tmp`. +4. **Checkout.** The checkout from step 1 is copied into `/workspace`, owned by the unprivileged user `65534:65534`, so the image builds exactly the recorded commit, within the checkout limits. The working tree is **on the branch being tested**, with its `.git` directory, so `git status`, `git log -1` and `git rev-parse HEAD` work. `HOME` is `/tmp`. 5. **Steps.** The jobs and steps run in order as `65534:65534`, in `/workspace`, **without network**, with the workflow's, the job's and the step's `environment` variables set. The log marks every job and step with `==> Job:` and `==> Step:` lines. 6. **Result.** The first failing step stops the build. The log shows `==> Step failed:` with the step name and exit code, and the build records that exit code and the status `FAILED`. When every step succeeds, the build is `PASSED`. 7. **Cleanup.** The image is removed when the build ends. The log, the workflow text, the commit ID and the base image digest are kept. @@ -194,7 +194,7 @@ Steps cannot reach the network. A step that downloads dependencies, such as a `m ## Private repositories -For a private repository, an admin starts the build with a credential reference. The image build reads the token from the backend's environment and hands it to the clone step only, as a build secret. It is not written into a layer, the Containerfile, the build arguments or the log, and later steps cannot read it. See **Private Git repositories** in the **Builds** chapter. +For a private repository, an admin starts the build with a credential reference. The backend reads the token from its environment for its own fetch only. The image build does not receive it, so it is not written into a layer, the Containerfile, the build arguments or the log, and steps cannot read it. See **Private Git repositories** in the **Builds** chapter. Never put secrets in a workflow file, because each build stores the file and shows it on the build page. diff --git a/docs/manual/en/72-architecture.md b/docs/manual/en/72-architecture.md index 263e4f0..b763085 100644 --- a/docs/manual/en/72-architecture.md +++ b/docs/manual/en/72-architecture.md @@ -71,6 +71,8 @@ erDiagram INTEGER retention_max_builds INTEGER max_concurrent_builds INTEGER max_queued_builds + INTEGER max_checkout_mib + INTEGER max_checkout_files } PROJECT ||--o{ BUILD : has diff --git a/src/backend/main/java/ai/mnemosyne/cisys/BuildRunner.java b/src/backend/main/java/ai/mnemosyne/cisys/BuildRunner.java index 0e05877..65c1796 100644 --- a/src/backend/main/java/ai/mnemosyne/cisys/BuildRunner.java +++ b/src/backend/main/java/ai/mnemosyne/cisys/BuildRunner.java @@ -17,8 +17,6 @@ import jakarta.annotation.PostConstruct; import io.quarkus.logging.Log; import jakarta.ws.rs.WebApplicationException; -import org.eclipse.jgit.api.Git; -import org.eclipse.jgit.transport.RefSpec; import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider; import java.io.*; import java.nio.charset.StandardCharsets; @@ -55,6 +53,10 @@ private static boolean onPath(String program){ /** An admin's setting replaces the configured limit. */ int concurrency(){Integer v=Settings.current().maxConcurrentBuilds;return v!=null?v:concurrency;} int queueLimit(){Integer v=Settings.current().maxQueuedBuilds;return v!=null?v:queueLimit;} + @ConfigProperty(name="cisys.build.max-checkout-mib",defaultValue="1024") int maxCheckoutMib=1024; + @ConfigProperty(name="cisys.build.max-checkout-files",defaultValue="100000") int maxCheckoutFiles=100000; + int maxCheckoutMib(){Integer v=Settings.current().maxCheckoutMib;return v!=null?v:maxCheckoutMib;} + int maxCheckoutFiles(){Integer v=Settings.current().maxCheckoutFiles;return v!=null?v:maxCheckoutFiles;} private final ExecutorService executor=Executors.newCachedThreadPool(); private final Map completions=new ConcurrentHashMap<>(); record Completion(Long id,Build.Status status,String revision,Integer exitCode,String imageDigest, @@ -87,6 +89,7 @@ static void lockQueue(){ } void recover(@Observes StartupEvent event){ if(concurrency<1 || queueLimit<1)throw new IllegalArgumentException("Build concurrency and queue limit must be positive"); + if(maxCheckoutMib<1 || maxCheckoutFiles<1)throw new IllegalArgumentException("Checkout size and file limits must be positive"); cleanupOrphans(); List running=QuarkusTransaction.requiringNew().call(()->Build.list("status",Build.Status.RUNNING).stream().map(b->b.id).toList()); for(Long id:running){ @@ -234,17 +237,16 @@ private void run(Long id,String url,BuildResource.RunRequest request){ log=logs.open(id); // Checked again here because DNS can change after the project was saved. repositoryUrls.validate(url); + UsernamePasswordCredentialsProvider credentials=null; + if(request.credentialRef()!=null){String token=System.getenv(request.credentialRef());if(token==null || token.isBlank())throw new IllegalStateException("Missing Git credential");credentials=new UsernamePasswordCredentialsProvider("oauth2",token);} + int[] limits=QuarkusTransaction.requiringNew().call(()->new int[]{maxCheckoutMib(),maxCheckoutFiles()}); dir=Files.createTempDirectory(checkoutRoot(),"cisys-"); - try(Git git=Git.init().setDirectory(dir.toFile()).call()){ - var cfg=git.getRepository().getConfig();cfg.setString("http",null,"followRedirects","false");cfg.save(); - var fetch=git.fetch().setRemote(url).setRefSpecs(new RefSpec("+refs/heads/"+request.branch()+":refs/remotes/origin/build")).setDepth(1).setTimeout(60); - if(request.credentialRef()!=null){String token=System.getenv(request.credentialRef());if(token==null || token.isBlank())throw new IllegalStateException("Missing Git credential");fetch.setCredentialsProvider(new UsernamePasswordCredentialsProvider("oauth2",token));} - fetch.call();git.checkout().setName("build").setCreateBranch(true).setStartPoint("refs/remotes/origin/build").call();revision=git.getRepository().resolve("HEAD").name(); - } - // The workflow on the fetched branch describes the image. This checkout is only read for the workflow file; - // the image clones the branch itself. - String text=Workflow.read(dir); - delete(dir);dir=null; + // The image copies this checkout, so what it gets is what was measured against the limits. + Path context=Files.createDirectory(dir.resolve("context")); + Path workspace=context.resolve(Workflow.WORKSPACE); + revision=Checkout.fetch(workspace,url,request.branch(),credentials,limits[0],limits[1]); + // The workflow on the fetched branch describes the image. + String text=Workflow.read(workspace); if(text==null)throw new IllegalArgumentException("The branch has no "+Workflow.PATH); // Stored before parsing, so a build keeps the file that it ran or that failed validation. QuarkusTransaction.requiringNew().run(()->Build.update("workflow = ?1 where id = ?2",text,id)); @@ -254,16 +256,12 @@ private void run(Long id,String url,BuildResource.RunRequest request){ QuarkusTransaction.requiringNew().run(()->Build.update("image = ?1 where id = ?2",from.name(),id)); // Allowed when the admin allowed any name of the image, whichever way the workflow names it. if(!QuarkusTransaction.requiringNew().call(()->from.names().stream().anyMatch(AllowedImage::isAllowed)))throw new IllegalArgumentException("Image "+base+" is not allowed"); - String containerfile=workflow.containerfile(from.id(),url,request.branch(),revision,request.credentialRef()!=null); + String containerfile=workflow.containerfile(from.id()); log.line("Workflow "+workflow.name()+" on "+from.name()+" ("+from.id().substring(0,12)+")"); - dir=Files.createTempDirectory(checkoutRoot(),"cisys-"); - Path context=Files.createDirectory(dir.resolve("context")); Path file=Files.writeString(dir.resolve("Containerfile"),containerfile); List build=new ArrayList<>(List.of(runtime,"build","--no-cache","--pull=never","-f",file.toString(),"-t",name)); if(runtime.equals("podman"))build.addAll(List.of("--layers=false","--security-opt=no-new-privileges","--memory=1g","--cpu-period=100000","--cpu-quota=100000")); else build.add("--progress=plain"); - // The runtime reads the token from its environment, so it never appears in an argument or in the image. - if(request.credentialRef()!=null)build.add("--secret=id="+Workflow.GIT_SECRET+",env="+request.credentialRef()); build.add(context.toString()); Streamed result=stream(build,600,log,logMaxBytes); captureError=result.readerError(); diff --git a/src/backend/main/java/ai/mnemosyne/cisys/Checkout.java b/src/backend/main/java/ai/mnemosyne/cisys/Checkout.java new file mode 100644 index 0000000..e177f10 --- /dev/null +++ b/src/backend/main/java/ai/mnemosyne/cisys/Checkout.java @@ -0,0 +1,206 @@ +/* + * Eclipse Public License - v 2.0 + * + * THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE + * PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION + * OF THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT. + */ + +package ai.mnemosyne.cisys; + +import org.eclipse.jgit.api.Git; +import org.eclipse.jgit.dircache.DirCacheCheckout; +import org.eclipse.jgit.internal.storage.file.FileRepository; +import org.eclipse.jgit.lib.Constants; +import org.eclipse.jgit.lib.CoreConfig; +import org.eclipse.jgit.lib.FileMode; +import org.eclipse.jgit.lib.ObjectInserter; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.revwalk.RevWalk; +import org.eclipse.jgit.transport.CredentialsProvider; +import org.eclipse.jgit.transport.PackParser; +import org.eclipse.jgit.transport.RefSpec; +import org.eclipse.jgit.treewalk.TreeWalk; +import org.eclipse.jgit.treewalk.WorkingTreeOptions; + +import java.io.File; +import java.io.FilterInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.nio.file.FileVisitResult; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.SimpleFileVisitor; +import java.nio.file.attribute.BasicFileAttributes; + +/** + * Fetches one branch at depth 1 into a new directory, on that branch, with {@code origin} set to the repository. The + * received pack is capped while it is read, the files the checkout will write are measured before anything is + * checked out, and the written working tree is measured again afterwards. + */ +final class Checkout { + private static final long MIB = 1024 * 1024; + + private Checkout() {} + + /** Returns the commit ID. {@code dir} must not exist; it is removed again when the checkout fails. */ + static String fetch(Path dir, String url, String branch, CredentialsProvider credentials, int maxMib, int maxFiles) + throws Exception { + String ref = Constants.R_REMOTES + "origin/" + branch; + Files.createDirectory(dir); + try (LimitedRepository repo = new LimitedRepository(dir.resolve(".git").toFile(), maxMib * MIB); Git git = Git.wrap(repo)) { + repo.create(); + var cfg = repo.getConfig(); + cfg.setString("http", null, "followRedirects", "false"); + cfg.setString("remote", "origin", "url", url); + cfg.setString("remote", "origin", "fetch", "+refs/heads/*:refs/remotes/origin/*"); + cfg.setString("branch", branch, "remote", "origin"); + cfg.setString("branch", branch, "merge", Constants.R_HEADS + branch); + cfg.save(); + var fetch = git.fetch().setRemote(url).setRefSpecs(new RefSpec("+refs/heads/" + branch + ":" + ref)).setDepth(1).setTimeout(60); + if (credentials != null) fetch.setCredentialsProvider(credentials); + try { + fetch.call(); + } catch (Exception e) { + if (repo.exceeded) throw tooLarge(maxMib); + throw e; + } + checkTree(repo, ref, maxMib, maxFiles); + git.checkout().setName(branch).setCreateBranch(true).setStartPoint(ref).call(); + // Smudge filters from the server's Git configuration can still change what is written. + checkWritten(dir, maxMib, maxFiles); + return repo.resolve(Constants.HEAD).name(); + } catch (Exception e) { + try { BuildRunner.delete(dir); } catch (IOException suppressed) { e.addSuppressed(suppressed); } + throw e; + } + } + + /** Sizes are those of the files as checked out, after line ending conversion. */ + private static void checkTree(Repository repo, String ref, int maxMib, int maxFiles) throws IOException { + long bytes = 0; + int files = 0; + WorkingTreeOptions options = repo.getConfig().get(WorkingTreeOptions.KEY); + try (RevWalk revs = new RevWalk(repo); TreeWalk tree = new TreeWalk(repo)) { + tree.setOperationType(TreeWalk.OperationType.CHECKOUT_OP); + tree.addTree(revs.parseCommit(repo.resolve(ref)).getTree()); + tree.setRecursive(true); + while (tree.next()) { + if (++files > maxFiles) throw tooMany(maxFiles); + // Submodule commits are not fetched. + if (tree.getFileMode(0) == FileMode.GITLINK) continue; + CoreConfig.EolStreamType eol = tree.getCheckoutEolStreamType(0); + if (eol == null || eol == CoreConfig.EolStreamType.DIRECT || tree.getFileMode(0) == FileMode.SYMLINK) { + bytes += tree.getObjectReader().getObjectSize(tree.getObjectId(0), Constants.OBJ_BLOB); + } else { + Counter written = new Counter(maxMib * MIB - bytes); + try { + DirCacheCheckout.getContent(repo, tree.getPathString(), new DirCacheCheckout.CheckoutMetadata(eol, null), + tree.getObjectReader().open(tree.getObjectId(0), Constants.OBJ_BLOB), options, written); + } catch (Counter.Exceeded e) { + throw tooLarge(maxMib); + } + bytes += written.count; + } + if (bytes > maxMib * MIB) throw tooLarge(maxMib); + } + } + } + + private static void checkWritten(Path dir, int maxMib, int maxFiles) throws IOException { + Path git = dir.resolve(".git"); + long[] total = new long[2]; + Files.walkFileTree(dir, new SimpleFileVisitor<>() { + @Override public FileVisitResult preVisitDirectory(Path d, BasicFileAttributes attrs) { + return d.equals(git) ? FileVisitResult.SKIP_SUBTREE : FileVisitResult.CONTINUE; + } + + @Override public FileVisitResult visitFile(Path f, BasicFileAttributes attrs) { + if (++total[0] > maxFiles) throw tooMany(maxFiles); + total[1] += attrs.size(); + if (total[1] > maxMib * MIB) throw tooLarge(maxMib); + return FileVisitResult.CONTINUE; + } + }); + } + + private static IllegalArgumentException tooLarge(int maxMib) { + return new IllegalArgumentException("Repository checkout exceeds the limit of " + maxMib + " MiB"); + } + + private static IllegalArgumentException tooMany(int maxFiles) { + return new IllegalArgumentException("Repository checkout exceeds the limit of " + maxFiles + " files"); + } + + /** Counts what is written and stops once it passes {@code max}. */ + private static final class Counter extends OutputStream { + static final class Exceeded extends IOException {} + + private final long max; + long count; + + Counter(long max) { this.max = max; } + + @Override public void write(int b) throws IOException { add(1); } + + @Override public void write(byte[] b, int off, int len) throws IOException { add(len); } + + private void add(long n) throws IOException { + count += n; + if (count > max) throw new Exceeded(); + } + } + + private static final class LimitedRepository extends FileRepository { + private final long maxBytes; + private long received; + boolean exceeded; + + LimitedRepository(File gitDir, long maxBytes) throws IOException { + super(gitDir); + this.maxBytes = maxBytes; + } + + @Override public ObjectInserter newObjectInserter() { + ObjectInserter inserter = super.newObjectInserter(); + return new ObjectInserter.Filter() { + @Override protected ObjectInserter delegate() { return inserter; } + + @Override public PackParser newPackParser(InputStream in) throws IOException { + return super.newPackParser(new Limited(in)); + } + }; + } + + private final class Limited extends FilterInputStream { + Limited(InputStream in) { super(in); } + + @Override public int read() throws IOException { + int b = super.read(); + if (b >= 0) count(1); + return b; + } + + @Override public int read(byte[] buf, int off, int len) throws IOException { + int n = super.read(buf, off, len); + if (n > 0) count(n); + return n; + } + + @Override public long skip(long n) throws IOException { + long skipped = super.skip(n); + if (skipped > 0) count(skipped); + return skipped; + } + + private void count(long n) throws IOException { + received += n; + if (received > maxBytes) { + exceeded = true; + throw new IOException("Pack exceeds the checkout size limit"); + } + } + } + } +} diff --git a/src/backend/main/java/ai/mnemosyne/cisys/Settings.java b/src/backend/main/java/ai/mnemosyne/cisys/Settings.java index 62e61d1..8801410 100644 --- a/src/backend/main/java/ai/mnemosyne/cisys/Settings.java +++ b/src/backend/main/java/ai/mnemosyne/cisys/Settings.java @@ -25,6 +25,10 @@ public class Settings extends PanacheEntityBase { public Integer maxConcurrentBuilds; /** Null uses cisys.build.queue-limit. */ public Integer maxQueuedBuilds; + /** Null uses cisys.build.max-checkout-mib. */ + public Integer maxCheckoutMib; + /** Null uses cisys.build.max-checkout-files. */ + public Integer maxCheckoutFiles; public static Settings current() { Settings s = findById(1L); diff --git a/src/backend/main/java/ai/mnemosyne/cisys/SettingsResource.java b/src/backend/main/java/ai/mnemosyne/cisys/SettingsResource.java index d082eac..153c495 100644 --- a/src/backend/main/java/ai/mnemosyne/cisys/SettingsResource.java +++ b/src/backend/main/java/ai/mnemosyne/cisys/SettingsResource.java @@ -27,7 +27,8 @@ @Produces(MediaType.APPLICATION_JSON) @Consumes(MediaType.APPLICATION_JSON) public class SettingsResource { - static final int MAX_DAYS = 3650, MAX_BUILDS = 10000, MAX_CONCURRENT = 16, MAX_QUEUED = 1000; + static final int MAX_DAYS = 3650, MAX_BUILDS = 10000, MAX_CONCURRENT = 16, MAX_QUEUED = 1000, + MAX_CHECKOUT_MIB = 102400, MAX_CHECKOUT_FILES = 1000000; @Inject SecurityIdentity identity; @Inject BuildRunner runner; @@ -37,6 +38,8 @@ public record Retention(Integer maxAgeDays, Integer maxBuilds) {} public record Queue(Integer maxConcurrent, Integer maxQueued) {} + public record Checkout(Integer maxSizeMib, Integer maxFiles) {} + @GET @Path("/retention") public Retention retention() { requireAdmin(); Settings s = Settings.current(); @@ -74,6 +77,24 @@ public record Queue(Integer maxConcurrent, Integer maxQueued) {} return new Queue(s.maxConcurrentBuilds, s.maxQueuedBuilds); } + @GET @Path("/checkout") public Checkout checkout() { + requireAdmin(); + return new Checkout(runner.maxCheckoutMib(), runner.maxCheckoutFiles()); + } + + @PUT @Path("/checkout") @Transactional public Checkout updateCheckout(Checkout input) { + requireAdmin(); + if (input == null || input.maxSizeMib() == null || input.maxSizeMib() < 1 || input.maxSizeMib() > MAX_CHECKOUT_MIB) + throw new BadRequestException("Checkout size must be between 1 and " + MAX_CHECKOUT_MIB + " MiB"); + if (input.maxFiles() == null || input.maxFiles() < 1 || input.maxFiles() > MAX_CHECKOUT_FILES) + throw new BadRequestException("Checkout files must be between 1 and " + MAX_CHECKOUT_FILES); + Settings s = Settings.findById(1L); + if (s == null) { s = new Settings(); s.persist(); } + s.maxCheckoutMib = input.maxSizeMib(); + s.maxCheckoutFiles = input.maxFiles(); + return new Checkout(s.maxCheckoutMib, s.maxCheckoutFiles); + } + private void requireAdmin() { if (!identity.hasRole("admin")) throw new ForbiddenException(); } diff --git a/src/backend/main/java/ai/mnemosyne/cisys/Workflow.java b/src/backend/main/java/ai/mnemosyne/cisys/Workflow.java index f887656..e0598ae 100644 --- a/src/backend/main/java/ai/mnemosyne/cisys/Workflow.java +++ b/src/backend/main/java/ai/mnemosyne/cisys/Workflow.java @@ -51,11 +51,11 @@ * * * A step either runs a shell command ({@code run}) or waits until a TCP port accepts connections ({@code wait}). - * A build turns the workflow into a Containerfile, see {@link #containerfile(String, String, String, String, boolean)}. + * A build turns the workflow into a Containerfile, see {@link #containerfile(String)}. */ public record Workflow(String name, String container, List environment, List dependencies, List jobs) { /** - * Where the steps read {@link #environment()} from. Root writes it before the clone and makes it read-only, so a + * Where the steps read {@link #environment()} from. Root writes it before the checkout and makes it read-only, so a * step cannot add variables for the steps after it. */ static final String ENVIRONMENT_FILE = "/etc/cisys-environment"; @@ -210,40 +210,38 @@ private static void validateDependencies(List dependencies, String w /** How long a {@code wait} step waits for its port, in seconds. */ static final int WAIT_SECONDS = 300; static final int MAX_CONTAINERFILE = 200_000; - /** The user the clone and the steps run as. */ + /** The user the checkout and the steps belong to. */ static final String BUILD_USER = "65534:65534"; - /** The id of the build secret that holds the Git token, see {@link #containerfile}. */ - static final String GIT_SECRET = "git"; + /** The directory in the build context that holds the checkout. */ + static final String WORKSPACE = "workspace"; /** * The Containerfile that builds this workflow's image from the local image {@code baseId}, which - * {@link ImageReference#resolve} found for {@code container}; see - * {@link #containerfile(String, String, String, String, boolean, int)}. + * {@link ImageReference#resolve} found for {@code container}; see {@link #containerfile(String, int)}. */ - public String containerfile(String baseId, String repositoryUrl, String branch, String revision, boolean credential) { - return containerfile(baseId, repositoryUrl, branch, revision, credential, WAIT_SECONDS); + public String containerfile(String baseId) { + return containerfile(baseId, WAIT_SECONDS); } /** * Builds an image from the base image, named by its ID so the build uses exactly the image that was resolved: *
    *
  1. installs {@link #TOOLS} (Git, Make, Podman and Podman Compose) as root when the image does not have them,
    2. - *
  2. clones the branch into {@code /workspace} as an unprivileged user and checks that it is at {@code revision}; - * the working tree stays on the branch, with its {@code .git},
    3. + *
  3. copies the build's checkout from {@value #WORKSPACE} in the build context into {@code /workspace}, owned by an + * unprivileged user; the working tree is on the branch, with its {@code .git},
    4. *
  4. runs the steps as that user, without network, with the workflow's, the job's and the step's * {@code environment} exported, the innermost winning. A variable is unset again when its scope ends: a step's * variables when the step ends, a job's when the job ends. A step cannot change the variables of the steps after * it, as each step starts a new shell from the read-only {@link #ENVIRONMENT_FILE}.
    5. *
* {@code dependencies} are installed as root, with network, with the package manager the image has, before the - * clone. + * checkout is copied. *

* Each {@code run} step is its own {@code RUN}. A job with a {@code wait} step runs as a single {@code RUN}, * because processes started in the background end with the {@code RUN} that started them. The first failing - * step stops the build with its exit code. The Git token, when there is one, is a build secret - * ({@value #GIT_SECRET}) that is mounted for the clone only and is not stored in the image. + * step stops the build with its exit code. The image build does not fetch the repository, so it needs no Git token. */ - String containerfile(String baseId, String repositoryUrl, String branch, String revision, boolean credential, int waitSeconds) { + String containerfile(String baseId, int waitSeconds) { if (!baseId.matches("[0-9a-f]{64}")) throw new IllegalArgumentException("Not an image ID: " + baseId); StringBuilder f = new StringBuilder(); f.append("FROM ").append(baseId).append('\n'); @@ -257,9 +255,7 @@ String containerfile(String baseId, String repositoryUrl, String branch, String f.append("USER ").append(BUILD_USER).append('\n'); f.append("WORKDIR /workspace\n"); f.append("ENV HOME=/tmp GIT_TERMINAL_PROMPT=0\n"); - f.append("RUN "); - if (credential) f.append("--mount=type=secret,id=").append(GIT_SECRET).append(' '); - f.append(exec("sh", "-c", clone(credential), "clone", repositoryUrl, branch, revision)).append('\n'); + f.append("COPY --chown=").append(BUILD_USER).append(' ').append(WORKSPACE).append(" /workspace\n"); for (String run : runs(waitSeconds)) { f.append("RUN --network=none ").append(exec("sh", "-c", ". " + ENVIRONMENT_FILE + " || exit 1\n" + run)).append('\n'); } @@ -335,17 +331,6 @@ private static void install(StringBuilder f, List dependencies) { f.append("RUN ").append(exec(args.toArray(String[]::new))).append('\n'); } - /** Arguments: $1 repository URL, $2 branch, $3 expected revision. */ - private static String clone(boolean credential) { - String helper = credential - ? "-c 'credential.helper=!f() { echo username=oauth2; printf \"password=%s\\n\" \"$(cat /run/secrets/" + GIT_SECRET + ")\"; }; f' " - : ""; - return "echo \"==> Checkout: $2\"\n" - + "git -c credential.helper= " + helper + "clone --quiet --depth 1 --single-branch --branch \"$2\" -- \"$1\" . || exit $?\n" - + "rev=$(git rev-parse HEAD) || exit $?\n" - + "if [ \"$rev\" != \"$3\" ]; then echo \"==> Checkout failed: $2 is at $rev, expected $3\"; exit 1; fi\n"; - } - private static final JsonMapper JSON = new JsonMapper(); /** The exec (JSON) form, so nothing in the arguments is interpreted by the Containerfile parser or a shell. */ diff --git a/src/backend/main/resources/application.properties b/src/backend/main/resources/application.properties index b6ea70d..3c04fd3 100644 --- a/src/backend/main/resources/application.properties +++ b/src/backend/main/resources/application.properties @@ -59,3 +59,6 @@ cisys.git.allowed-networks=${CISYS_GIT_ALLOWED_NETWORKS:} # Single backend runner. Limits must be positive; queue-limit counts waiting builds only. cisys.build.concurrency=${BUILD_CONCURRENCY:1} cisys.build.queue-limit=${BUILD_QUEUE_LIMIT:100} +# Limits for one repository checkout, in MiB and files. Both must be positive. +cisys.build.max-checkout-mib=${BUILD_MAX_CHECKOUT_MIB:1024} +cisys.build.max-checkout-files=${BUILD_MAX_CHECKOUT_FILES:100000} diff --git a/src/backend/main/resources/db/migration/R__schema.sql b/src/backend/main/resources/db/migration/R__schema.sql index 30c8531..c126e03 100644 --- a/src/backend/main/resources/db/migration/R__schema.sql +++ b/src/backend/main/resources/db/migration/R__schema.sql @@ -98,3 +98,7 @@ ALTER TABLE build ADD COLUMN IF NOT EXISTS workflow character varying(65536); -- Null uses the configured queue limits. ALTER TABLE settings ADD COLUMN IF NOT EXISTS maxconcurrentbuilds integer; ALTER TABLE settings ADD COLUMN IF NOT EXISTS maxqueuedbuilds integer; + +-- Null uses the configured checkout limits. +ALTER TABLE settings ADD COLUMN IF NOT EXISTS maxcheckoutmib integer; +ALTER TABLE settings ADD COLUMN IF NOT EXISTS maxcheckoutfiles integer; diff --git a/src/backend/test/java/ai/mnemosyne/cisys/CheckoutTest.java b/src/backend/test/java/ai/mnemosyne/cisys/CheckoutTest.java new file mode 100644 index 0000000..1323c9c --- /dev/null +++ b/src/backend/test/java/ai/mnemosyne/cisys/CheckoutTest.java @@ -0,0 +1,128 @@ +/* + * Eclipse Public License - v 2.0 + * + * THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE + * PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION + * OF THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT. + */ + +package ai.mnemosyne.cisys; + +import org.eclipse.jgit.api.Git; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Random; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class CheckoutTest { + private static final int MIB = 1024 * 1024; + + @TempDir Path source; + @TempDir Path parent; + + @BeforeEach + void initSource() throws Exception { + Git.init().setDirectory(source.toFile()).setInitialBranch("main").call().close(); + } + + private String commit() throws Exception { + try (Git git = Git.open(source.toFile())) { + git.add().addFilepattern(".").call(); + return git.commit().setMessage("files").setAuthor("Test", "test@example.com").setCommitter("Test", "test@example.com").call().name(); + } + } + + private Path dir() { + return parent.resolve("workspace"); + } + + private String fetch(int maxMib, int maxFiles) throws Exception { + return Checkout.fetch(dir(), source.toUri().toString(), "main", null, maxMib, maxFiles); + } + + private void assertRejected(String message, int maxMib, int maxFiles) throws Exception { + IllegalArgumentException e = assertThrows(IllegalArgumentException.class, () -> fetch(maxMib, maxFiles)); + assertEquals(message, e.getMessage()); + try (var left = Files.list(parent)) { + assertEquals(0, left.count()); + } + } + + @Test + void checksOutBelowTheLimits() throws Exception { + Files.writeString(source.resolve("README"), "hello\n"); + Files.createDirectories(source.resolve("src")); + Files.writeString(source.resolve("src/main.sh"), "echo hi\n"); + String revision = commit(); + assertEquals(revision, fetch(1, 2)); + assertEquals("hello\n", Files.readString(dir().resolve("README"))); + assertEquals("echo hi\n", Files.readString(dir().resolve("src/main.sh"))); + } + + @Test + void checksOutTheBranchTrackingOrigin() throws Exception { + Files.writeString(source.resolve("README"), "hello\n"); + String revision = commit(); + fetch(1, 1); + try (Git git = Git.open(dir().toFile())) { + assertEquals("main", git.getRepository().getBranch()); + assertEquals(source.toUri().toString(), git.getRepository().getConfig().getString("remote", "origin", "url")); + assertEquals("origin", git.getRepository().getConfig().getString("branch", "main", "remote")); + assertEquals(revision, git.getRepository().resolve("refs/remotes/origin/main").name()); + } + } + + @Test + void acceptsExactlyTheLimit() throws Exception { + Files.write(source.resolve("zeros"), new byte[MIB]); + String revision = commit(); + assertEquals(revision, fetch(1, 1)); + } + + @Test + void countsLineEndingConversion() throws Exception { + // 1,000,000 bytes in Git, 1,500,000 bytes once checked out with CRLF line endings. + Files.writeString(source.resolve(".gitattributes"), "*.txt text eol=crlf\n"); + Files.writeString(source.resolve("lines.txt"), "a\n".repeat(500_000)); + commit(); + assertRejected("Repository checkout exceeds the limit of 1 MiB", 1, 10); + } + + @Test + void acceptsLineEndingConversionWithinTheLimit() throws Exception { + Files.writeString(source.resolve(".gitattributes"), "*.txt text eol=crlf\n"); + Files.writeString(source.resolve("lines.txt"), "ab\n".repeat(1000)); + commit(); + fetch(1, 10); + assertEquals(4000, Files.size(dir().resolve("lines.txt"))); + } + + @Test + void rejectsCompressibleContentBeforeCheckout() throws Exception { + Files.write(source.resolve("zeros"), new byte[2 * MIB]); + commit(); + assertRejected("Repository checkout exceeds the limit of 1 MiB", 1, 10); + } + + @Test + void rejectsALargePackDuringFetch() throws Exception { + byte[] noise = new byte[2 * MIB]; + new Random(32).nextBytes(noise); + Files.write(source.resolve("noise"), noise); + commit(); + assertRejected("Repository checkout exceeds the limit of 1 MiB", 1, 10); + } + + @Test + void rejectsTooManyFiles() throws Exception { + for (int i = 0; i < 6; i++) Files.writeString(source.resolve("f" + i), "x"); + commit(); + assertRejected("Repository checkout exceeds the limit of 5 files", 1, 5); + } +} diff --git a/src/backend/test/java/ai/mnemosyne/cisys/SettingsResourceTest.java b/src/backend/test/java/ai/mnemosyne/cisys/SettingsResourceTest.java index a6bf0ab..85623a3 100644 --- a/src/backend/test/java/ai/mnemosyne/cisys/SettingsResourceTest.java +++ b/src/backend/test/java/ai/mnemosyne/cisys/SettingsResourceTest.java @@ -84,4 +84,38 @@ void rejectsInvalidQueue() { given().contentType("application/json").body(body).put(QUEUE).then().statusCode(400); } } + + private static final String CHECKOUT = "/api/settings/checkout"; + + static void clearCheckout() { + QuarkusTransaction.requiringNew().run(() -> Settings.update("maxCheckoutMib = null, maxCheckoutFiles = null where id = 1")); + } + + @Test void anonymousCannotReadCheckout() { given().get(CHECKOUT).then().statusCode(401); } + + @Test @TestSecurity(user = "lead", roles = "lead") + void leadCannotReadOrUpdateCheckout() { + given().get(CHECKOUT).then().statusCode(403); + given().contentType("application/json").body("{\"maxSizeMib\":10,\"maxFiles\":10}").put(CHECKOUT).then().statusCode(403); + } + + @Test @TestSecurity(user = "admin", roles = "admin") + void adminSavesCheckoutOverConfiguredDefaults() { + given().get(CHECKOUT).then().statusCode(200).body("maxSizeMib", equalTo(1024), "maxFiles", equalTo(100000)); + try { + given().contentType("application/json").body("{\"maxSizeMib\":64,\"maxFiles\":500}").put(CHECKOUT).then().statusCode(200) + .body("maxSizeMib", equalTo(64), "maxFiles", equalTo(500)); + given().get(CHECKOUT).then().statusCode(200).body("maxSizeMib", equalTo(64), "maxFiles", equalTo(500)); + } finally { + clearCheckout(); + } + } + + @Test @TestSecurity(user = "admin", roles = "admin") + void rejectsInvalidCheckout() { + for (String body : new String[] {"{\"maxSizeMib\":0,\"maxFiles\":10}", "{\"maxSizeMib\":102401,\"maxFiles\":10}", + "{\"maxSizeMib\":1,\"maxFiles\":0}", "{\"maxSizeMib\":1,\"maxFiles\":1000001}", "{\"maxSizeMib\":1}", "{}"}) { + given().contentType("application/json").body(body).put(CHECKOUT).then().statusCode(400); + } + } } diff --git a/src/backend/test/java/ai/mnemosyne/cisys/WorkflowTest.java b/src/backend/test/java/ai/mnemosyne/cisys/WorkflowTest.java index e43bef6..fe79690 100644 --- a/src/backend/test/java/ai/mnemosyne/cisys/WorkflowTest.java +++ b/src/backend/test/java/ai/mnemosyne/cisys/WorkflowTest.java @@ -287,17 +287,15 @@ void rejectsContainerfilesTooLargeToRun() { // Each quote grows to five characters in the Containerfile ('\'' in JSON), so 13 steps exceed the limit. String step = " - name: s\n run: x" + "'".repeat(3999) + "\n"; Workflow w = workflow(step.repeat(13)); - assertThrows(Workflow.InvalidWorkflowException.class, () -> w.containerfile(ID, URL, "main", REV, false)); + assertThrows(Workflow.InvalidWorkflowException.class, () -> w.containerfile(ID)); } - static final String URL = "https://github.com/example/app"; - static final String REV = "0123456789abcdef0123456789abcdef01234567"; static final String ID = "164b5c6a230f645862b6123e07cf0a67f2f03371696aa82a12ccf3589e87c016"; @Test - void containerfileClonesTheBranchThenRunsEachStepWithoutNetwork() { + void containerfileCopiesTheCheckoutThenRunsEachStepWithoutNetwork() { List lines = Workflow.parse(VALID.replace(" wait: 8180\n", " run: make wait\n")) - .containerfile(ID, URL, "feature/x", REV, false).lines().toList(); + .containerfile(ID).lines().toList(); assertEquals("FROM " + ID, lines.get(0)); assertEquals("USER 0", lines.get(1)); assertTrue(lines.get(2).startsWith("RUN [\"sh\",\"-c\",\". /etc/os-release"), lines.get(2)); @@ -306,10 +304,9 @@ void containerfileClonesTheBranchThenRunsEachStepWithoutNetwork() { assertEquals("RUN --network=none [\"sh\",\"-c\",\"printf '%s' \\\"$1\\\" > /etc/cisys-environment && chown 0:0 /etc/cisys-environment && chmod 444 /etc/cisys-environment\",\"environment\",\"export MODE='test'\\nexport PORT='8180'\\n\"]", lines.get(3)); assertEquals("USER 65534:65534", lines.get(5)); assertEquals("WORKDIR /workspace", lines.get(6)); - String clone = lines.get(8); - assertTrue(clone.startsWith("RUN [\"sh\",\"-c\","), clone); - assertTrue(clone.endsWith(",\"clone\",\"" + URL + "\",\"feature/x\",\"" + REV + "\"]"), clone); - assertFalse(clone.contains("secret"), clone); + // The image gets the checkout that was measured against the limits; it does not fetch the branch again. + assertEquals("COPY --chown=65534:65534 workspace /workspace", lines.get(8)); + assertTrue(lines.stream().noneMatch(l -> l.contains("git clone") || l.contains("secret")), String.join("\n", lines)); List steps = lines.subList(9, lines.size()); assertEquals(4, steps.size(), String.join("\n", lines)); assertTrue(steps.stream().allMatch(l -> l.startsWith("RUN --network=none [\"sh\",\"-c\",\". /etc/cisys-environment || exit 1\\n")), String.join("\n", steps)); @@ -344,8 +341,8 @@ private static String installs(String line) { } @Test - void dependenciesAreInstalledAsRootBeforeTheClone() { - List lines = Workflow.parse(DEPENDENCIES).containerfile(ID, URL, "main", REV, false).lines().toList(); + void dependenciesAreInstalledAsRootBeforeTheCheckout() { + List lines = Workflow.parse(DEPENDENCIES).containerfile(ID).lines().toList(); assertEquals("USER 0", lines.get(1)); assertTrue(lines.get(3).startsWith("RUN [\"sh\",\"-c\",\". /etc/os-release"), lines.get(3)); assertEquals("\"JSON processor\",\"jq\",\"Compiler\",\"gcc\"]", installs(lines.get(3))); @@ -356,7 +353,7 @@ void dependenciesAreInstalledAsRootBeforeTheClone() { } assertTrue(lines.get(4).contains("/etc/cisys-environment"), lines.get(4)); assertEquals("USER 65534:65534", lines.get(6)); - assertTrue(lines.get(9).contains("\"clone\""), lines.get(9)); + assertTrue(lines.get(9).startsWith("COPY "), lines.get(9)); assertEquals(1, lines.stream().filter(l -> l.equals("USER 0")).count(), String.join("\n", lines)); } @@ -388,17 +385,10 @@ void rejectsInvalidDependencies() { .dependencies().get(0).value()); } - @Test - void credentialIsMountedForTheCloneOnly() { - String file = Workflow.parse(VALID).containerfile(ID, URL, "main", REV, true); - assertEquals(1, file.split("--mount=type=secret,id=" + Workflow.GIT_SECRET + " ", -1).length - 1, file); - assertTrue(file.contains("/run/secrets/" + Workflow.GIT_SECRET), file); - } - @Test void argumentsCannotEscapeTheExecForm() { String file = workflow(" - name: \"a\\\"]\\nRUN evil\"\n run: echo \"\\\\\\n\"\n") - .containerfile(ID, URL, "main", REV, false); + .containerfile(ID); assertEquals(10, file.lines().count(), file); assertTrue(file.lines().noneMatch(l -> l.startsWith("RUN evil")), file); } diff --git a/src/frontend/src/SettingsPanel.jsx b/src/frontend/src/SettingsPanel.jsx index a2ce3b8..d9e1c30 100644 --- a/src/frontend/src/SettingsPanel.jsx +++ b/src/frontend/src/SettingsPanel.jsx @@ -31,6 +31,13 @@ const SECTIONS = { { key: 'maxQueued', id: 'queue-queued', label: 'Queued builds', min: 1, max: 1000 }, ], }, + checkout: { + title: 'Checkout', + fields: [ + { key: 'maxSizeMib', id: 'checkout-size', label: 'Max size (MiB)', min: 1, max: 102400 }, + { key: 'maxFiles', id: 'checkout-files', label: 'Max files', min: 1, max: 1000000 }, + ], + }, }; function SettingsTab({ api, section }) {