diff --git a/.agents/skills/compiling/SKILL.md b/.agents/skills/compiling/SKILL.md new file mode 100644 index 0000000000..7c5acebe61 --- /dev/null +++ b/.agents/skills/compiling/SKILL.md @@ -0,0 +1,100 @@ +--- +name: compiling +description: How to compile Gkeyll libraries, executables, and specific unit or regression test targets on CPU or CUDA. +--- + +# Instructions + +Use this skill whenever you need to compile +* Any of the Gkeyll libraries. +* A C unit or regression test. +* A C input file. + +# Compiling in Gkeyll + +We assume the dependencies have already been installed (e.g. via a machines/mkdeps +file) and configured (e.g. via a machines/configure file). + +Operate relative to the repository root (`git rev-parse --show-toplevel`). +Dependencies are typically already installed in the sibling `gkylsoft/` +directory. Check the existing configuration before building. If configuration +is missing or needs changing, consult `machines/configure..sh` for +required modules and library paths, or use `./configure` with the intended +installation prefix and solver. + +## Compiling the Gkeyll library + +We compile using a Makefile. The `make` command allows you to specify the number +of cores to use via `-j`. We will use no more than half the available processors for parallel builds. +So before invoking `make`, set the `NPROC` environment variable on a Mac using +``` +export NPROC=$(sysctl -n hw.physicalcpu) +NPROC=$((NPROC > 1 ? NPROC / 2 : 1)) +``` +or on Linux using + +```sh +NPROC=$(nproc) +NPROC=$((NPROC > 1 ? NPROC / 2 : 1)) +``` + +Then, you can build the Gkeyll library with + +```sh +make -j"$NPROC" install +``` + +## Compiling a C unit or regression test + +Unit tests are located in the `unit/` directory of each solver +(core/moments/vlasov/gyrokinetic/pkpm), while C regression tests are located in `creg/` folders and +have names beginning with `rt_`. + +Compile a specific test by giving make its executable target: + +```sh +make -j"$NPROC" build/core/unit/ctest_array +make -j"$NPROC" build/gyrokinetic/creg/rt_gk_sheath_2x2v_p1 +``` + +CPU targets are located in `build/`, and GPU targets are in `cuda-build/`. +Match test targets to the configured build directory. + +For executing the resulting tests, see +[testing_and_verification](../testing_and_verification/SKILL.md). + +### Compiling all or groups of C unit tests + +The Makefile supports compiling groups of unit tests all at once. For example, to compile all the +unit tests at once run + +```sh +make -j"$NPROC" unit +``` + +Alternatively, you can compile all the unit tests for a specific solver. For example, to compute all +the unit tests for the gyrokinetic solver use + +```sh +make -j"$NPROC" unit-run +``` + +## Compiling C input files + +Input files written in C work just like C regression tests. They can be compiled similarly if they +are located in the `creg/` folder. For example, if `my_sim.c` is a gyrokinetic input file, and located +in `gyrokinetic/creg/`, it can be compiled with + +```sh +make -j"$NPROC" build/gyrokinetic/creg/my_sim +``` + +If the input file is outside of the repository, one needs to use the shared makefile in +`/gkylsoft/gkeyll/share/`. For example, if `my_other_sim.c` is an input file in `$HOME/new_sim/`, +compile it with: + +```sh +cd ~ +cp /gkylsoft/gkeyll/share/Makefile ./ +make +``` diff --git a/.agents/skills/directory_structure/SKILL.md b/.agents/skills/directory_structure/SKILL.md new file mode 100644 index 0000000000..acbb97e8e5 --- /dev/null +++ b/.agents/skills/directory_structure/SKILL.md @@ -0,0 +1,71 @@ +--- +name: directory_structure +description: Explanation of the Gkeyll directories. +--- + +# Instructions + +* Use when exploring the repository, trying to find something, or deciding where new code belongs. +* Operate relative to the repository root (`git rev-parse --show-toplevel`). + +# Gkeyll file structure + +Gkeyll has four PDE solvers: +* Moments or fluid solver. +* Vlasov solver. +* Gyrokinetic solver. +* PKPM solver. + +Correspondingly, these solvers are organized into four separate folders, and they +share some common functionality in a fifth folder. Gkeyll is mostly organized in: +* core/: functionality common to all solvers. +* moments/: files for the moments solver. +* vlasov/: files for the Vlasov solver. +* gyrokinetic/: files for the gyrokinetic solver. +* pkpm/: files for the PKPM solver. + +Note that the four solvers are not independent. They have the following +dependencies: +* moments depends on core. +* vlasov depends on moments. +* gyrokinetic depends on vlasov. +* pkpm depends on gyrokinetic. + +Keep lower layers independent of higher layers. + +Each of the solvers' folders have the sub-folders: +* ker/: C kernels generated with Maxima stored in the gkylcas repository. Do + not hand-edit generated kernels; modify their source templates instead. +* zero/: C and CUDA functions or modules (sometimes we call them updaters), + some of which call kernels in ker/. +* data/: data needed for some simulations. +* unit/: unit tests of specific components in zero/. +* apps/: apps are called by input files or regression tests, and they organize + solver workflow or simulations and call modules in zero/. Communication + happens in the app level. +* creg/: C regression tests or short simulations to ensure solvers work (these + are also examples of C input files). +* luareg/: Lua regression tests or short simulations to ensure solvers work + (these are also examples of Lua input files). + +## Library dependencies + +The Gkeyll source code in gkeyll/ depends on other libraries installed in gkylsoft/. +Most of the time you shouldn't need to read files in gkylsoft/ unless prompted. + +## CPU - GPU organization + +Gkeyll runs on both CPUs and GPUs, the latter using CUDA. +- CPU-only implementations live in .c files. +- C files call host-side wrappers of CUDA kernels, both of which live in _cu.cu + files. + +## Search hierarchy + +Rather than looking into the entire codebase without prior knowledge, look into +folders in the following order: +- Look into a folder (ending in /) the user referenced. +- If the user referenced a specific solver (e.g. gyrokinetic), look into its + folder. +- Look into folders for solvers that the specified solver depends on, following + the dependency chain up through core/. diff --git a/.agents/skills/gyrokinetic_details/SKILL.md b/.agents/skills/gyrokinetic_details/SKILL.md new file mode 100644 index 0000000000..3f52b99afb --- /dev/null +++ b/.agents/skills/gyrokinetic_details/SKILL.md @@ -0,0 +1,36 @@ +--- +name: gyrokinetic_details +description: Details specific to the gyrokinetic solver. +--- + +# Instructions + +* Use when working on the gyrokinetic/ code, or with an input file that uses + the gyrokinetic solver. + +# Gyrokinetics + +## MPI decomposition + +Gyrokinetic regression tests parallelize along the last configuration-space +dimension. Choose the partition flag by dimensionality: + +| Dimensionality | Flag for N partitions | +|---|---| +| 1x2v | `-c N` | +| 2x2v | `-d N` | +| 3x2v | `-e N` | + +For example, run the compiled 2x2v sheath regression on four MPI ranks: + +```sh +/gkylsoft/openmpi/bin/mpirun -np 4 ./build/gyrokinetic/creg/rt_gk_sheath_2x2v_p1 -M -d 4 +``` + +Use the MPI installation matching the build. `-M` enables MPI in the +regression executable; `-d 4` partitions its second configuration dimension. +Add `-s1` for a one-step smoke test. + +See [compiling](../compiling/SKILL.md) when a build is needed, +and [testing_and_verification](../testing_and_verification/SKILL.md) for general test +execution and memory checks. diff --git a/.agents/skills/naming_conventions/SKILL.md b/.agents/skills/naming_conventions/SKILL.md new file mode 100644 index 0000000000..7e42704f33 --- /dev/null +++ b/.agents/skills/naming_conventions/SKILL.md @@ -0,0 +1,33 @@ +--- +name: naming_conventions +description: Determine the correct name for files, functions, modules and variables. +--- + +# Instructions + +* Apply these naming conventions when creating, editing, or reviewing C, CUDA, and Lua code. + +# Naming conventions + +### Files. + +- Public header files have names starting with gkyl_ and ending in .h. +- Private header files have names starting with gkyl_ and ending in _priv.h. +- CUDA files have names ending in _cu.cu. +- luareg/ folders have Lua input files whose names end with .lua. The Lua + wrappers are in the apps/ folders and have names ending in _lw.c + +### Functions + +- Public functions (defined in public header files) should have a name that starts + with gkyl_. +- Public functions in files in zero/ folders should have a name that starts + with the name of the file that contains it. +- Private functions (defined in private header files) or static (and not defined + in private headers) in files in zero/ folders should have a name that starts + with an abbreviated version of the name of the file that contains it. + +### Variables + +- Do not use single letter names for variables whose scope spans more than 15 lines. + diff --git a/.agents/skills/software_design/SKILL.md b/.agents/skills/software_design/SKILL.md new file mode 100644 index 0000000000..6c075ad9e2 --- /dev/null +++ b/.agents/skills/software_design/SKILL.md @@ -0,0 +1,34 @@ +--- +name: software_design +description: Some design practices in Gkeyll code to consider when implementing new modules (apps, updaters) or modifying existing ones. +--- + +# Instructions + +* Load this skill whenever new modules or updaters in zero/ or apps/ folders + are to be created, or when existing ones will be edited. + +## Software design elements + +### Module structure + +Most modules in zero/ or apps/ consist of 3 public functions: +1. A creation function, typically called gkyl__new or gkyl__init. +2. An execution function, often called gkyl__advance or gkyl__apply. +3. A deletion function, typically called gkyl__release. + +There may also be some additional auxiliary private or public functions. + +### Module best practices + +- All dynamic (heap) allocations should happen in the initialization function, + and freed in the release function, using the appropriate gkyl_ allocation/deallocation functions when possible. +- Don't place logic branching (e.g. if-statements) that depend on a + time-independent choice or parameter inside the methods called in the time loop of a simulation (e.g. _advance). Instead, use function pointers to set the appropriate method during the initialization of the module, and call that method inside the time loop. + +### Other principles to follow + +- Consider extensibility, maintainability, simplicity and how modular design. +- Avoid code duplication whenever possible (e.g. write functions called + multiple times) and without breaking layering. +- Write shorter code and refactor into a sub-module whenever possible. diff --git a/.agents/skills/testing_and_verification/SKILL.md b/.agents/skills/testing_and_verification/SKILL.md new file mode 100644 index 0000000000..76ba3ab755 --- /dev/null +++ b/.agents/skills/testing_and_verification/SKILL.md @@ -0,0 +1,147 @@ +--- +name: testing_and_verification +description: How to run Gkeyll unit and regression tests, as well as performing memory checks. Use when testing changes and verifying that changes didn't break the code or change simulation results. +--- + +# Instructions + +* Run unit tests after completing a project. +* Ask the user whether to run regression tests, or which regression test they + would like to run (running all of them takes a long time). + +# Running unit tests + +Operate relative to the repository root (`git rev-parse --show-toplevel`). + +In order to run all Gkeyll unit tests: +1. Compile all unit tests; see [compilation](../compiling/SKILL.md). +2. Run all unit tests with +```sh +make -j"$NPROC" unit-run +``` + +In order to run all unit tests for a specific solversolversolver, e.g. gyrokinetic: +1. Compile all unit tests for that solver; see [compilation](../compiling/SKILL.md). +2. Run all unit tests for that solver, e.g. for gyrokinetic use +```sh +make -j"$NPROC" gyrokinetic-unit-run +``` + +One can also compile and run a single unit test of interest. Simply: +1. Compile the unit; see [compilation](../compiling/SKILL.md). +2. Run the executable from the repository root: +For example, in order to run `core/unit/ctest_array.c` do + +```sh +make -j"$NPROC" ./build/core/unit/ctest_array +./build/core/unit/ctest_array +``` + +If the Gkeyll was build on a CPU-only machine or without GPU support, unit tests will simply ignore +GPU tests (via pre-processor if-statements). When running on a machine with a supported GPU, and if +Gkeyll is configured to build GPU code, running these tests will automatically run the corresponding +GPU tests. If a GPU is not available but Gkeyll was built to use GPUs, the GPU tests may simply fail +or exit (e.g. due to GPU memory allocation errors). + +### Parallel unit tests + +There are a few parallel unit tests that use MPI, whose names begin with `mctest`. These tests are +compiled the same way as serial tests, but must be run with the appropriate MPI execution command. + +For example if Gkeyll was built and configured with the MPI in `gkylsoft/openmpi/`, then a parallel +unit test (for example `mctest_mpi_comm.c`) may be run with + +```sh +/gkylsoft/openmpi/bin/mpirun -np 4 build/core/unit/mctest_mpi_comm +``` + +# Running regression tests + +## C regression tests + +Regression tests written in C and located in `creg/` directories need to be compiled first, +see [compilation](../compiling/SKILL.md). + +Once compiled the regression test may be run serially with, for +`gyrokinetic/creg/rt_gk_sheath_2x2v_p1.c` for example: +```sh +./build/gyrokinetic/creg/rt_gk_sheath_2x2v_p1 +``` + +See +```sh +./build/gyrokinetic/creg/rt_gk_sheath_2x2v_p1 -h +``` +for command line arguments that may be available (not all are actually supported). For +example, the following command +```sh +./build/gyrokinetic/creg/rt_gk_sheath_2x2v_p1 -s1 +``` +limits the regression to a one-step smoke test. + +CUDA builds place the executable in `cuda-build/` instead of `build/`, and regression tests use `-g` +to indicate GPU execution. + +## Parallel regression tests + +Regression executables take the `-M` command line argument to indicate a parallel run (requiring +MPI). The flags `-c X -d Y -e Z` specify that the first, second and third configuration-space +dimensions are to be subdivided amongs X, Y and Z MPI processes, respecitively. The product `X*Y*Z` +must match the number of cores available for this run. If one of `-c`, `-d` or `-e` is not given, +it is assumed to be 1. + +For example, we may run `gyrokinetic/creg/rt_gk_sheath_2x2v_p1` using 2 cores in the second +dimension with +```sh +/openmpi/bin/mpirun -np 2 ./build/gyrokinetic/creg/rt_gk_sheath_2x2v_p1 -M -d 2 +``` + +For gyrokinetic partition restrictions and an example, read +[gyrokinetic_details](../gyrokinetic_details/SKILL.md). + +It is also possible to run with multiple GPUs. Gkeyll's model is to match each MPI process to a +single GPU. The procedure is similar as for multiple GPUs, but the additional `-g` flag is needed. +For example, to run `gyrokinetic/creg/rt_gk_sheath_2x2v_p1` using 2 GPUs in the second dimension +with +```sh +/openmpi/bin/mpirun -np 2 ./build/gyrokinetic/creg/rt_gk_sheath_2x2v_p1 -g -M -d 2 +``` + +## Lua regression tests + +Run Lua inputs using the installed executable: + +```sh +/gkylsoft/gkeyll/bin/gkeyll +``` + +## The runregression tool + +After installing the Gkeyll library and executable one may use the `runregression` tool to run +regression tests, see + +```sh +/gkylsoft/gkeyll/bin/gkeyll runregression -h +``` + +After configuring runregression, and having generated accepted results with +`runregression run create`, one may check the test using `runregression run check`. Note that the +flag `-r` allows check a specific test or set of tests. + +# Memory checks + +All code must be free of memory errors and leaks. Run the relevant CPU test +under Valgrind before committing: + +```sh +valgrind --leak-check=full ./build/core/unit/ctest_array +``` + +For GPU changes, run the relevant executable under Compute Sanitizer: + +```sh +compute-sanitizer --tool memcheck --leak-check full +``` + +Supply the executable's GPU options where required. Report unavailable tools +or hardware and distinguish completed checks from checks that could not run. diff --git a/.claude b/.claude new file mode 120000 index 0000000000..c0ca468566 --- /dev/null +++ b/.claude @@ -0,0 +1 @@ +.agents \ No newline at end of file diff --git a/.codex b/.codex new file mode 120000 index 0000000000..c0ca468566 --- /dev/null +++ b/.codex @@ -0,0 +1 @@ +.agents \ No newline at end of file diff --git a/.gitignore b/.gitignore index 4dca1b4f20..8204ec4c20 100644 --- a/.gitignore +++ b/.gitignore @@ -69,3 +69,6 @@ gyrokinetic/data/adas/*.npy # Adas data files data/adas/*.npy install-deps/*.npy + +# Agent related stuff +.entire/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000..415ce9ae6c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,40 @@ +# Agent Profile + +## Role + +You are a computational plasma physicist working on Gkeyll (Gkeyll solves partial differential +equations). You are: +- a critical thinker, +- analytical and precise, +- an assistant that communicates concisely, +- well versed the the previous and current, cutting edge literature on plasma physics and numerical +methods (both traditional and modern data-driven). + +Your responsibilities are to increase the capabilities of the Gkeyll codebase, elevate the quality +of its software, identify bugs and other issues, troubleshoot and make suggestions to your colleagues. + +## Core directives + +- Stick to the task colleagues pointed you to, but stay vigilant for bugs and + issues you indentify along the way that may not be related to your task. +- New and edited code should prioritize correctness, performance, + maintainability and simplicity, in that order (from most to least important). +- Test and verify new and edited code. +- Take into consideration the ideas and guidelines colleagues give you, but be + creative and suggest alternative approaches. + +## Skills and detailed instructions + +Consider the following skills when working on relevant tasks: + +* **Navigating Gkeyll directories:** `skills/directory_structure/SKILL.md`. +* **Creating or editing code:** `skills/software_design/SKILL.md`. +* **Naming files or code elements:** `skills/naming_conventions/SKILL.md` +* **Compiling code:** `skills/compiling/SKILL.md` +* **Running tests and input files:** `skills/testing_and_verification/SKILL.md` +* **Working on the gyrokinetic solver:** `skills/gyrokinetic_details/SKILL.md` + +## Execution protocol + +When a user query triggers one of the specific skill areas above, silently ingest the rules from the +corresponding referenced `.md` file. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/Makefile b/Makefile index c9b53f9481..69c95d824a 100644 --- a/Makefile +++ b/Makefile @@ -268,10 +268,10 @@ everything: regression unit gkeyll ## Build everything, including unit, regressi core: ## Build core infrastructure code cd core && $(MAKE) -f Makefile-core -core-unit: ## Build core unit tests +core-unit: core ## Build core unit tests cd core && $(MAKE) -f Makefile-core unit -core-regression: ## Build core regression tests +core-regression: core ## Build core regression tests cd core && $(MAKE) -f Makefile-core regression core-install: ## Install core infrastructure code @@ -402,7 +402,7 @@ pkpm-valcheck: pkpm ## Run valgrind on unit tests in PKPM gkeyll: ${BUILD_APP} ## Build Gkeyll executable cd gkeyll && ${MAKE} -f Makefile-gkeyll gkeyll -gkeyll-install: ${BUILD_APP}-install gkeyll ## Install Gkeyll executable +gkeyll-install: ${BUILD_APP}-install ## Install Gkeyll executable cd gkeyll && ${MAKE} -f Makefile-gkeyll install ## Targets to build things all parts of the code @@ -418,7 +418,9 @@ clean: rm -rf ${BUILD_DIR} # Check everything -check: unit unit-run ## Build (if needed) and run all unit tests +check: ## Build (if needed) and run all unit tests + $(MAKE) unit + $(MAKE) unit-run # Run all unit tests unit-run: ## Run all unit tests without (re)building them diff --git a/ci/jenkins/Jenkinsfile b/ci/jenkins/Jenkinsfile new file mode 100644 index 0000000000..e3c859e8c4 --- /dev/null +++ b/ci/jenkins/Jenkinsfile @@ -0,0 +1,197 @@ +// Jenkins pipeline for Gkeyll, built/run on our own persistent machines +// (see ci/jenkins/README.md for how the Jenkins job driving this is set up). +// +// Unlike a shared, long-lived gkylsoft/, each build clones the repo fresh +// and builds its own dependencies from scratch into its own workspace +// (mirroring .github/workflows/mac_build.yml), then actually runs the unit +// tests rather than just building them. +// +// Scripted (not declarative) pipeline, since we run the same steps in +// parallel across a set of named nodes that will grow over time and may mix +// operating systems (hence the per-node mkdeps/configure script choice). + +def buildAndTest(String mkdepsScript, String configureScript) { + stage('Environment') { + sh 'printenv' + } + stage('Clean') { + // The workspace (and its build/ and gkylsoft/ dirs) persists across + // builds for a given branch/PR, so a build interrupted or racing + // mid-compile can leave stale/corrupted files behind that a later + // incremental step wrongly treats as up to date (e.g. missing + // symbols at link time, without ever recompiling/reinstalling). + // Wipe both every run so each build is truly from scratch: the + // install-deps scripts already re-download/rebuild everything + // unconditionally, but don't clear their install prefix first, so a + // stale gkylsoft/ could otherwise mix old and new files. + sh 'make clean' + sh 'rm -rf "$WORKSPACE/gkylsoft"' + // Side workspace used to build a same-session `main` baseline for + // the regression-diff stage below (PR builds only). + sh 'rm -rf "$WORKSPACE/_main_baseline"' + } + stage('Dependencies') { + // Build gkylsoft/ from scratch inside this build's own workspace + // (rather than a shared, fixed path) so that concurrent builds for + // different branches/PRs on the same node never share or clobber + // each other's dependency trees. + sh "PREFIX=\"\$WORKSPACE/gkylsoft\" ./machines/${mkdepsScript}" + } + stage('Configure') { + sh "PREFIX=\"\$WORKSPACE/gkylsoft\" ./machines/${configureScript}" + } + stage('Unit tests') { + sh 'make -j3 check' + } + stage('Regression build') { + sh 'make -j3 regression' + } + stage('Install') { + // Needed by 'gkeyll runregression': it compiles C regression tests + // on the fly using the installed share/Makefile, and 'gkeyll' itself + // is the interpreter runregression runs as. + // + // Use 'install' (from the generated alltargets.mak), not + // 'gkeyll-install' (defined directly in the top-level Makefile): + // the latter has two independent prerequisite chains that both + // rebuild the same build/**/libg0*.so files, which can race under + // -j and fail with e.g. "cp: .../libg0vlasov.so: No such file or + // directory" when nothing has pre-built them yet. + sh 'make -j3 install' + } + + if (env.CHANGE_ID) { + // Only PR builds run the regression-diff stages below: they compare + // against a baseline built from the PR's target branch, which is + // meaningless for a build *of* that branch itself. + stage('Regression baseline (main)') { + buildRegressionBaseline(mkdepsScript, configureScript, env.CHANGE_TARGET ?: 'main') + } + stage('Regression tests (MOAT)') { + def gkeyll = "\$WORKSPACE/gkylsoft/gkeyll/bin/gkeyll" + def resultsDir = "\$WORKSPACE/gkylsoft/gkeyll-results" + def baselineResultsDir = "\$WORKSPACE/_main_baseline/gkylsoft/gkeyll-results" + sh "\"${gkeyll}\" runregression configure --source-dir \"\$WORKSPACE\" --prefix \"\$WORKSPACE/gkylsoft\"" + // Move (not copy) the main-branch baseline's accepted outputs + // into the PR's own results tree: _main_baseline is scratch + // space nothing else reads afterward and gets wiped by the next + // build's Clean stage anyway, so moving avoids briefly doubling + // disk usage on what can be a large number of regression files. + def moveCmd = "set -e\n" + // Layers with a runregression MOAT (Mother Of All Tests) subset. + def REGRESSION_LAYERS = ['moments', 'vlasov', 'gyrokinetic', 'pkpm'] + for (layer in REGRESSION_LAYERS) { + for (kind in ['luareg-accepted', 'creg-accepted']) { + moveCmd += "if [ -d \"${baselineResultsDir}/${layer}/${kind}\" ]; then rm -rf \"${resultsDir}/${layer}/${kind}\"; mv \"${baselineResultsDir}/${layer}/${kind}\" \"${resultsDir}/${layer}/${kind}\"; fi\n" + } + } + sh moveCmd + sh "\"${gkeyll}\" runregression run --moat check --timeout 300 --jobs 0" + sh "\"${gkeyll}\" \"\$WORKSPACE/ci/jenkins/check_regression_results.lua\" \"${resultsDir}\" \"\$WORKSPACE/ci/jenkins/expected_regression_diffs.txt\"" + } + } else { + // Cheap hygiene check for `main`-branch builds: a leftover + // acknowledgment entry from a just-merged PR would otherwise + // silently mask a real future regression on the same test name. + stage('Check for stale regression acknowledgments') { + sh ''' + if grep -qE '^[^#[:space:]]' ci/jenkins/expected_regression_diffs.txt 2>/dev/null; then + echo "WARNING: ci/jenkins/expected_regression_diffs.txt still has active entries on main -- clear it out now that the PR that added them is merged (see ci/jenkins/README.md)." + fi + ''' + } + } +} + +// Builds `baseBranch` (normally the PR's target branch) from scratch in a +// side workspace and runs the MOAT regression suite's 'create' step there, +// producing accepted baselines generated on this exact node/build -- so the +// PR's own MOAT 'check' run (see buildAndTest) has something to diff against +// without relying on a stale or cross-machine-generated baseline. +def buildRegressionBaseline(String mkdepsScript, String configureScript, String baseBranch) { + dir('_main_baseline') { + // A plain shallow, single-branch clone rather than a GitSCM + // checkout: the latter defaults to fetching every branch/tag in + // the whole repo (no refspec given), which timed out after 10 + // minutes enumerating 100k+ objects. We only need this branch's tip. + sh "git clone --depth 1 --branch ${baseBranch} https://github.com/gkeyllorg/gkeyll ." + sh "PREFIX=\"\$WORKSPACE/_main_baseline/gkylsoft\" ./machines/${mkdepsScript}" + sh "PREFIX=\"\$WORKSPACE/_main_baseline/gkylsoft\" ./machines/${configureScript}" + // See the comment on the 'Install' stage above re: install vs gkeyll-install. + sh 'make -j3 install' + def gkeyll = "\$WORKSPACE/_main_baseline/gkylsoft/gkeyll/bin/gkeyll" + sh "\"${gkeyll}\" runregression configure --source-dir \"\$WORKSPACE/_main_baseline\" --prefix \"\$WORKSPACE/_main_baseline/gkylsoft\"" + sh "\"${gkeyll}\" runregression run --moat create --timeout 300" + } +} + +properties([ + disableConcurrentBuilds(abortPrevious: true), + parameters([ + booleanParam( + name: 'FORCE_BUILD', + defaultValue: false, + description: 'Override this build\'s allowedPrAuthors/branch restrictions on every node (use "Build with Parameters" to set it for one manual run).' + ) + ]) +]) + +// Jenkins node label -> config to use on that machine (machines/ has a +// mkdeps/configure script variant per OS/site). This map is shared across +// every independent Jenkins controller building this repo (see +// ci/jenkins/README.md) — add a node here once it's registered as a Jenkins +// agent under whichever controller owns it, using whichever scripts match +// its OS, e.g.: +// 'workstation-node': [mkdeps: 'mkdeps.linux.sh', configure: 'configure.linux.cpu.sh'], +// +// allowedPrAuthors (optional): restricts this node to only building PRs +// authored by one of the listed GitHub usernames — for personal machines +// that shouldn't run arbitrary contributors' PR code. Omit it (as for a +// shared/team node) to build PRs from anyone. Builds of `main` itself are +// never restricted, since there's no CHANGE_AUTHOR for those. +// +// Every node also only builds `main` and PRs, never a plain branch pushed +// directly (e.g. one still awaiting a PR, or one Jenkins indexed before its +// PR was opened) — see below. Both restrictions can be bypassed for a single +// manual run via the FORCE_BUILD parameter (Build with Parameters), e.g. to +// deliberately run a specific contributor's PR on a personal machine. +def nodes = [ + 'manauref_lt1': [mkdeps: 'mkdeps.macos.sh', configure: 'configure.macos.sh', allowedPrAuthors: ['manauref']], + 'antoine_mac': [mkdeps: 'mkdeps.macos.sh', configure: 'configure.macos.sh', allowedPrAuthors: ['Antoinehoff']], +] + +// Every controller running this Jenkinsfile shares the `nodes` map above, +// but only owns a subset of those labels as actual agents. Each controller +// declares its own subset via a global environment variable (Manage Jenkins +// -> System -> Global properties -> Environment variables), set once locally +// and NOT committed to the repo -- see ci/jenkins/README.md. Without this, a +// controller would try node(label) for a label it has no agent for and hang +// forever waiting for an executor that will never exist. +def ownedLabels = (env.CI_OWNED_NODE_LABELS ?: '').split(',').collect { it.trim() }.findAll { it } +if (!ownedLabels) { + error("CI_OWNED_NODE_LABELS is not set on this controller (Manage Jenkins -> System -> Global properties -> Environment variables). Set it to a comma-separated list of the node label(s) this controller owns, e.g. 'manauref_lt1'. See ci/jenkins/README.md.") +} +def myNodes = nodes.findAll { label, cfg -> ownedLabels.contains(label) } + +def branches = myNodes.collectEntries { label, cfg -> + [(label): { + if (!params.FORCE_BUILD && env.CHANGE_AUTHOR && cfg.allowedPrAuthors && !cfg.allowedPrAuthors.contains(env.CHANGE_AUTHOR)) { + echo "Skipping ${label}: PR author '${env.CHANGE_AUTHOR}' is not in this node's allowedPrAuthors ${cfg.allowedPrAuthors}. Rebuild with FORCE_BUILD checked to override." + return + } + if (!params.FORCE_BUILD && !env.CHANGE_ID && env.BRANCH_NAME != 'main') { + echo "Skipping ${label}: plain branch build of '${env.BRANCH_NAME}' -- only 'main' and PRs are built here. Rebuild with FORCE_BUILD checked to override." + return + } + node(label) { + checkout scm + try { + buildAndTest(cfg.mkdeps, cfg.configure) + } finally { + archiveArtifacts allowEmptyArchive: true, artifacts: 'build/**/*.log,gkylsoft/gkeyll-results/**/*.txt,gkylsoft/gkeyll-results/**/regressiondb' + } + } + }] +} + +parallel(branches) diff --git a/ci/jenkins/README.md b/ci/jenkins/README.md new file mode 100644 index 0000000000..81fee2540b --- /dev/null +++ b/ci/jenkins/README.md @@ -0,0 +1,345 @@ +# Jenkins CI for Gkeyll + +This sets up Jenkins (following the pattern used by +[SUNDIALS](https://github.com/llnl/sundials/tree/main/test/jenkins), adapted +for GitHub instead of Bitbucket) to build Gkeyll and run its unit tests on our +own persistent machines any time a pull request into `main` is opened or +updated. + +Like `.github/workflows/mac_build.yml`, every build starts from a clean git +checkout and builds its dependencies (`gkylsoft/`) from scratch into its own +Jenkins workspace via `machines/mkdeps.macos.sh` — but on our own persistent +hardware instead of an ephemeral GitHub-hosted runner, and it actually +executes the unit tests rather than just building them. + +Each Jenkins controller runs on whichever machine you install it on — +laptop, desktop, or workstation. `manauref_lt1` (manauref's laptop) was the +first one set up this way. Build nodes — a controller's own machine, plus any +others registered later, personal or shared — are added and configured the +same way (see step 4). + +This doc covers both adding a node under an *existing* controller (e.g. +`manauref_lt1`'s), and setting up your *own*, fully independent controller on +your own machine (step 3a) if you'd rather not grant that existing +controller's admin SSH access to your machine. + +## 1. Install Jenkins + +**macOS:** + +``` +brew install jenkins-lts +brew services start jenkins-lts +``` + +This pulls in `openjdk@21` as a dependency. `JENKINS_HOME` is +`~/.jenkins` (Homebrew's default on macOS — not +`/opt/homebrew/var/lib/jenkins`, despite what some older docs say). Get the +initial admin password with: + +``` +cat ~/.jenkins/secrets/initialAdminPassword +``` + +**Linux:** + +Follow the [official Jenkins Linux install +instructions](https://www.jenkins.io/doc/book/installing/linux/) for your +distro (e.g. the `apt`/`yum` package repo), then start it via `systemctl +start jenkins` (most distros enable/start it automatically on install). +`JENKINS_HOME` is `/var/lib/jenkins` by default. Get the initial admin +password with: + +``` +sudo cat /var/lib/jenkins/secrets/initialAdminPassword +``` + +**Both:** + +Jenkins listens on `http://localhost:8080`. Open it, paste the +password, and install the "suggested plugins" set when prompted. + +## 2. Install additional plugins + +Manage Jenkins → Plugins → Available plugins, install: + +- **GitHub Branch Source** — lets Jenkins discover branches/PRs on a GitHub + repo and is the GitHub equivalent of the Bitbucket Branch Source plugin + SUNDIALS uses. +- **Collapsing Console Sections** (optional) — collapses long build log + sections for readability, same as SUNDIALS' setup. + +## 3. Add a GitHub credential + +Jenkins needs read access to `gkeyllorg/gkeyll` to poll for branches and PRs: + +1. Create a GitHub Personal Access Token with `repo` scope (Settings → + Developer settings → Personal access tokens on GitHub). +2. In Jenkins: Manage Jenkins → Credentials → System → Global credentials → + Add Credentials → kind "GitHub personal access token" (or "Username with + password", username = your GitHub username, password = the token). + +## 3a. Scope your controller to its own nodes + +`ci/jenkins/Jenkinsfile` has a single `nodes` map listing every known build +node across every independent controller that builds this repo (yours and +anyone else's). This keeps that map a shared, PR-reviewable source of truth +— but it also means your controller must be told which of those labels it +actually owns agents for, or it will try to start a `node(label)` block for +someone else's node and hang forever waiting for an executor that will never +exist under your controller. + +Manage Jenkins → System → Global properties → check "Environment variables" +→ Add: + +- Name: `CI_OWNED_NODE_LABELS` +- Value: a comma-separated list of the node label(s) *this controller* owns, + e.g. `manauref_lt1`, or `alice_laptop,alice_workstation` if you register + more than one node under your own controller. + +This is a controller-wide setting — set it once, regardless of how many +nodes your controller owns. If it's unset (or blank), the pipeline fails +fast with an error telling you to set it, rather than hanging; that's +expected the first time you set up a new controller. + +## 4. Add a build node + +Every machine that runs builds — including the controller's own machine — is +a Jenkins "node" with a label matching one used in `ci/jenkins/Jenkinsfile`'s +`nodes` map. This procedure is the same whether you're setting up +`manauref_lt1` (the first node) or adding another machine later. + +### 4.1. Register the node and label it + +- **This machine** (the controller itself, e.g. `manauref_lt1`): Manage + Jenkins → Nodes → "Built-In Node" → Configure → Labels: add the label. +- **A separate machine** (e.g. a workstation): Manage Jenkins → Nodes → New + Node → "Permanent Agent" → Launch method "Launch agents via SSH", + host/credentials for that machine, and give it a label. Nothing to install + there yourself — Jenkins pushes its agent jar over SSH. Make sure Java, a + C/C++/Fortran toolchain, and `cmake` are present; `mkdeps` builds everything + else from scratch per build. + +This label is what the Jenkinsfile's `node('