From dd69ed37168f1fe9580ea69da269bf3ee81dce60 Mon Sep 17 00:00:00 2001 From: lmoresi Date: Thu, 10 Sep 2026 16:38:20 -0700 Subject: [PATCH] CLAUDE.md is a router, not a manual: 22.9 KB to 8.3 KB MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every session loads this file verbatim before it loads anything else, and it had grown to more than twice the size of the Style Charter it defers to. That is the wrong way round: the file whose job is to point at the governing documents was the bulkiest thing in the context, and the 2026-09 audit found it competing for attention with the rules it exists to route to. Most of the bulk was duplication of documents that already govern their topic. `docs/developer/guides/branching-strategy.md` already carries the branch roles, the API-versus-implementation discipline, the worktree lifecycle, the branch policy and a section addressed to AI assistants; CLAUDE.md restated 4.3 KB of it. The on-demand documentation list restated the authority map in `docs/developer/index.md`. Both are now one pointer each. The build constraints move to `guides/development-setup.md`, which was a stub that still told developers to run `pixi run underworld-build` — a command the tooling replaced with `./uw build`. It now carries the rebuild rule, the editable-install prohibition and its recovery recipe, the PETSc non-relocatability warning, and how worktree environments work. CLAUDE.md keeps the four hard rules as one line each. Nothing normative was dropped. What remains is what a session needs before it knows where to look: the Charter mandate, session bootstrap and the external planning protocol, the hard constraints, where files go, the rulings that are non-obvious and easy to get wrong (rotated free-slip over Nitsche, the data access contract, unwrap-before-atoms, the ambiguous `model`), the test markers, and the pointer to the authority map. Every path this file names was checked to resolve. Underworld development team with AI support from Claude Code --- CLAUDE.md | 611 ++++----------------- docs/developer/guides/development-setup.md | 105 +++- 2 files changed, 198 insertions(+), 518 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 0f7868a1d..b49f666e7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,566 +1,197 @@ # Underworld3 AI Assistant Context > **MANDATORY**: Read `docs/developer/UW3_STYLE_CHARTER.md` before writing any code. -> It is the normative style contract for every session (human or AI), it is two pages, -> and it WINS over the surrounding code and over any other style document on conflict. +> It is the normative style contract for every session, it is two pages, and it WINS +> over the surrounding code and over any other style document on conflict. > Core clause: match the Charter, not the code next door — and flag deviations you find. -> **Note**: Human-readable developer documentation is in `docs/developer/` (Sphinx/MyST format). -> For development history and completed migrations, see `docs/developer/ai-notes/historical-notes.md` +This file is a router, not a manual. It carries what a session needs *before* it +knows where to look; everything else is one hop away through the authority map in +`docs/developer/index.md`. --- -## Extended AI Context (Optional) +## Session bootstrap -You can configure additional AI instruction files that live outside the Underworld repository. This is useful for: -- Personal coding style preferences -- Project planning and coordination systems -- Private instructions not shared with other contributors +**External planning.** If `UW_AI_TOOLS_PATH` is set (colon-separated directories), +check each for `.md` files — `underworld.md` especially — and report relevant +Active/Bugs items briefly. If it is unset, proceed with repo-local context. Set it +via `./uw setup`. -**Setup**: If the environment variable `UW_AI_TOOLS_PATH` is set (colon-separated directories, like `PATH`), check each directory for `.md` files containing additional context. Configure this via `./uw setup` or manually in your shell profile. - -**At conversation start**: If `UW_AI_TOOLS_PATH` is set, look for `underworld.md` (or other relevant files) in those directories. If found, report relevant Active/Bugs items briefly. If the variable is unset or files aren't found, proceed normally with repo-local context only. - -### Responding to External Planning Items - -When completing a task from an external planning file, add an annotation directly below the item: +When you complete a task from an external planning file, annotate it in place: ```markdown ``` -### Adding New Items to External Planning - -If you discover bugs, identify new tasks, or have items that should be tracked in the external planning file: -- **Don't create local TODO files or add to CLAUDE.md** -- **Do add to the external planning file** under the appropriate section (Bugs, Active, Nice to Have) -- Use the project tag: `` - -### Inline TODO Comments in Code - -Use inline `TODO` comments to mark **problem locations** in the source code. These provide self-documentation and help future developers (human or AI) find the relevant code quickly. - -**Format:** -```python -# TODO(BUG): Brief description of the issue -# More context if needed -# See planning file: underworld.md (section, date) -``` +Add newly discovered work to the external planning file under the appropriate +section with a `` tag — not to a local TODO +file, and not here. Don't rewrite strategic paragraphs, move items between +sections, or restructure the document; those carry cross-project context and are +handled by the planning tools. -**When to use:** -- Mark the exact location of a known bug -- Flag code that needs enhancement or refactoring -- Note incomplete implementations +**Inline TODOs** mark the *place*; the planning file tracks the *work*: -**Example:** ```python # TODO(BUG): add_natural_bc() causes PETSc error 73 # The Stokes solver works; issue is specific to scalar Poisson setup. # See planning file: underworld.md (Bugs section, 2026-01-19) -self.natural_bcs = [] ``` -This complements the planning file — the planning file tracks *what* needs doing, inline TODOs mark *where* in the code. - -### What Not to Do - -- Don't rewrite strategic paragraphs — they contain cross-project context -- Don't move items between sections (Active → Done) — planning-claude handles that -- Don't restructure the document - -If something needs more than an annotation or simple addition, mention it in conversation for the user to handle. - --- -## Documentation Requests - -**⚠️ MANDATORY - READ BEFORE WRITING ANY DOCUMENTATION ⚠️** - -- **ALL documentation MUST go in `docs/` directory** - NO exceptions -- **NEVER create .md files in the repository root, src/, tests/, or anywhere else** -- **NEVER create planning/design documents outside `docs/developer/design/`** -- If you're tempted to create a file like `SOME-FEATURE-NOTES.md` in the repo root - **DON'T**. Put it in `docs/developer/` instead. -- This applies to: design docs, how-to guides, technical notes, implementation plans, reviews, audits - EVERYTHING goes in `docs/` - -**Where to put documentation:** - -| Content Type | Location | -|--------------|----------| -| System documentation (meshing, solvers, swarms) | `docs/developer/subsystems/` | -| Architecture and design decisions | `docs/developer/design/` | -| How-to guides and best practices | `docs/developer/guides/` | -| User tutorials | `docs/beginner/tutorials/` | -| Advanced user guides | `docs/advanced/` | - -**Format** - Use MyST Markdown (`.md` files) compatible with Sphinx: -- Standard markdown with MyST extensions -- Use ```` ```python ```` for code blocks (not `{python}`) -- Use `{note}`, `{warning}`, `{tip}` for admonitions -- Math: `$inline$` and `$$display$$` - -**Style** - Concise, helpful, standalone: -- Self-contained explanations (don't assume reader has context) -- Include practical code examples -- Link to related documentation where appropriate -- Focus on "why" and "how to use", not just "what" -- Follow the notebook style guide for tutorials - -**Integration** - Link into the documentation system: -- Add to appropriate toctree in parent `index.md` -- Cross-reference related docs with `:doc:` or relative links -- Build and verify: `pixi run docs-build` - -**Style references**: -- Notebook writing: `docs/developer/guides/notebook-style-guide.md` -- Code patterns: `docs/developer/UW3_Style_and_Patterns_Guide.md` - ---- - -## Git and Branching Strategy - -**Full guide**: `docs/developer/guides/branching-strategy.md` - -### Branch Roles -- **`main`** — stable releases (tagged quarterly). No direct pushes. -- **`development`** — integration branch. Bug fixes land here. Features merge here via PR. -- **`feature/*`** — long-lived feature work. Branch from and PR back to `development`. - -### Key Discipline: Separate API from Implementation -Feature branches must not introduce API changes (new methods, changed signatures) that other branches can't access. When a feature needs an API change: -1. Extract the interface (stub or minimal implementation) into a separate commit. -2. Merge that to `development` first (or extract after the fact). -3. The feature PR should only contain implementation behind already-merged interfaces. - -This keeps feature branches independent and makes cross-pollination of fixes straightforward. +## Hard constraints -### Bug Fix Flow -- Fix on `development` (commit or small PR) -- Cherry-pick to `main` if critical → tag patch release -- Cherry-pick to active feature branches (underworld-claude handles this) +**Rebuild after every source change.** `./uw build`. Underworld3 is installed into +the environment, not imported from `src/`; verify with `uw.__file__`. -### Git Worktrees for Session Isolation -**Use a worktree for any multi-file change** (docs cleanup, refactoring, features). -Multiple Claude sessions sharing one working directory will overwrite each other's work. +**Never use an editable install.** `pip install -e .` contaminates every pixi +environment sharing the source tree and persists after uninstall. No exceptions. -Each worktree gets its **own pixi environment** (isolated site-packages, own -compiled extensions). Only PETSc is shared via symlink (non-relocatable, -expensive to rebuild). `./uw build` from inside a worktree installs that -worktree's source into the worktree's own environment. +**Never move `/Users/lmoresi/+Underworld/underworld-pixi-2/petsc/`.** PETSc +hardcodes paths at configure time; moving it costs a full rebuild. -**Full documentation**: `docs/developer/guides/branching-strategy.md` (Git Worktrees section) +**Worktrees are never on `development` or `main`.** All work happens on a side +branch (`feature/…`, `bugfix/…`, `docs/…`) and merges via PR. Use a worktree for +any multi-file change — concurrent sessions sharing one checkout overwrite each +other. Each worktree builds into its own environment, so enter it *first*, then +build. -#### Worktree branch policy +Detail: [`guides/development-setup.md`](docs/developer/guides/development-setup.md), +[`guides/branching-strategy.md`](docs/developer/guides/branching-strategy.md). -**Worktrees must NEVER be on `development` or `main` directly.** +**Attribution.** End commit messages and PR bodies with: -In this repo, `main` is the *release* branch (tagged quarterly, essentially -read-only history) and `development` is the integration trunk where active -work converges. The default repository checkout (`~/+Underworld/underworld3-pixi`) -should usually sit on `development` — that's where you read the current -working state and pull updates. All work — even work intended to land -on `development` — happens on a side branch (`feature/...`, `bugfix/...`, -`docs/...`) in a worktree, then merges to `development` via PR. - -`./uw worktree create` enforces this for new worktrees (always on a -side branch, reset to `origin/development`). It's on you not to break -it manually: - -- Never `git checkout development` (or `main`) inside a worktree -- Never `git worktree add ... development` to put a worktree on - `development` directly -- If you find a worktree on `development` (e.g. from older tooling), - branch off immediately (`git switch -c bugfix/whatever`) before - committing - -#### Creating and using a worktree - -```bash -# Create — own .pixi env, shared PETSc, names the branch -./uw worktree create # → feature/ -./uw worktree create bugfix # → bugfix/ - -# Work — drops you into a shell cd'd to the worktree -./uw worktree shell -./uw build # builds from THIS source into THIS worktree's env -./uw test # runs tests -exit # leave - -# List worktrees with branch and status -./uw worktree list - -# Bring files from other branches without switching: -git checkout origin/ -- path/to/file ``` - -#### Cleanup - -```bash -# Removes worktree directory and deletes the branch -./uw worktree remove -``` - -#### Important: always build and run from inside the worktree - -Each worktree has its own pixi environment. `./uw build` installs into the -environment of whichever worktree (or main repo) you run it from. Always: - -1. `./uw worktree shell ` (or `cd` into the worktree) -2. `./uw build` -3. Run your code / tests from there - -### AI-Assisted Attribution (Commits and PRs) -When committing code or creating pull requests with AI assistance, end the -message/body with: - -``` -Underworld development team with AI support from [Claude Code](https://claude.com/claude-code) -``` - -(In commit messages, use the plain-text form without the markdown link.) - -**Do NOT use**: -- `Co-Authored-By:` with a noreply email (useless for soliciting responses) -- Generic AI attribution without team context -- Emoji in PR descriptions - ---- - -## CRITICAL BUILD CONSTRAINTS - -### PETSc Directory (DO NOT MOVE) -**WARNING**: `/Users/lmoresi/+Underworld/underworld-pixi-2/petsc/` MUST NOT be moved. -- PETSc is NOT relocatable after compilation (hardcoded paths) -- Moving breaks petsc4py bindings and all pixi tasks -- Requires complete rebuild (~1 hour) if relocated - -### Rebuild After Source Changes -**After modifying source files, always run `./uw build`!** -- Underworld3 is installed as a package in the pixi environment -- Changes go to `.pixi/envs//lib/python3.12/site-packages/underworld3/` -- Verify with `uw.__file__` (should show site-packages path, NOT `src/`) - -**Note**: `./uw build` uses `--no-cache-dir` to prevent pip from reusing stale -wheels (UW3 is always version `0.0.0`). If you still suspect stale code, clean -the build directory: `rm -rf build/lib.* build/bdist.*` then rebuild. - -### NEVER Use Editable Installs -**DO NOT use `pip install -e .` (editable/development mode)!** -This is a hard rule — there are no exceptions. - -Editable installs create `.pth` files and `.so` symlinks in the source tree that: -- **Contaminate all pixi environments** sharing the same source directory -- **Break worktree isolation** (worktrees share pixi envs via symlinks) -- **Persist after uninstall** — stale `.pth` files redirect Python imports to `src/` - even after a proper `./uw build`, causing import errors or wrong library loading -- **Mix debug/release builds** — `.so` compiled against one PETSc arch get loaded - by environments expecting another, causing `dlopen` symbol errors - -Always use `./uw build` which runs `pip install .` (non-editable). If `./uw build` -is not available, use `pixi run -e pip install . --no-build-isolation --no-cache-dir`. - -**Recovery from editable install contamination:** -```bash -# Remove stale .pth files from ALL environments -find .pixi/envs -name "__editable__*underworld*" -delete -# Remove .so from source tree (they belong in site-packages) -find src/underworld3 -name "*.so" -delete -# Clean build cache -rm -rf build/ -# Rebuild properly -./uw build +Underworld development team with AI support from Claude Code ``` -### Test Quality Principles -**New tests must be validated before making code changes to fix them!** -- Validate test correctness before changing main code -- If core tests (0000-0599) pass, the system is working correctly -- Disable problematic new tests, validate core functionality, then fix test structure - -### JOSS Paper (FROZEN) -**Location**: `publications/joss-paper/` - Publication of record, DO NOT modify. - ---- - -## Units System Principles - -**Authoritative design doc**: `docs/developer/design/UNITS_SIMPLIFIED_DESIGN_2025-11.md` - -- Accept strings for convenience, store/return Pint objects: `uw.quantity(1e21, "Pa*s")` -- `.units` returns a Pint **Unit** (not string) — call `.to("m")` on the **Quantity**, not on `.units` -- UWexpression derives `.units` from contents (atomic: stored value; composite: `get_units(self._sym)`) - ---- - -## Parallel Computing Patterns +(PR bodies may link it.) Do not use `Co-Authored-By:` with a noreply address, and +no emoji in PR descriptions. -PETSc handles all parallel synchronization — avoid direct mpi4py unless necessary. -Use `uw.pprint()` and `uw.selective_ranks()` for rank-safe output and code blocks. +**Solver stability is paramount.** The PETSc-based solvers in +`petsc_generic_snes_solvers` are carefully tuned and validated. No changes +without extensive benchmarking. -**Implementation**: `src/underworld3/mpi.py` -**Documentation**: `docs/advanced/parallel-computing.md` +**The JOSS paper (`publications/joss-paper/`) is frozen.** It is the publication of +record; do not modify it. --- -## Architecture Priorities - -### Solver Stability is Paramount -The PETSc-based solvers are carefully optimized and validated. **NO CHANGES without extensive benchmarking.** - -### Module Boundaries - -| Module | Purpose | Access Pattern | -|--------|---------|----------------| -| **Solvers** (`petsc_generic_snes_solvers`) | High-performance PETSc solving | Direct `vec` property | -| **Mesh Variables** | User-facing field data | `array` property (new) | -| **Swarm Variables** | Particle data with mesh proxies | `data` property | - -### Conservative Migration Strategy -- **User-facing code**: Use `array` property with automatic sync -- **Solver internals**: Keep using `vec` property with direct PETSc access -- **Gradual transition**: Only change when driven by actual needs - ---- +## Where things go -## Boundary Conditions: Free-slip - -**Prefer rotated strong free-slip** (`solver.add_rotated_freeslip_bc(conds, boundary, normal=None)`, value-first: `conds=0` is free-slip; a non-zero scalar/expression `conds` prescribes the wall-normal datum `u·n̂ = ũ_n` strongly) -to impose `v·n̂ = 0`: - -- Enforces zero wall-normal flow to **machine precision** (Nitsche / penalty leak ~1e-3). -- Correct on **curved / tilted / deformed** boundaries — the normal is taken per node, - measure-weighted so it matches the straight-facet integral the assembler evaluates - (#560). Leave `normal=None` unless the constraint must follow the TRUE surface rather - than the mesh; an analytic `normal` (e.g. `X/|X|`) is exact for the geometry but keeps - a consistency error against the faceted assembly. See - `docs/developer/subsystems/rotated-freeslip.md` ("Which normal to use"). -- Works **inside the nonlinear SNES** and with **geometric FMG**. It honours - `solver.consistent_jacobian`: use `True` (consistent Newton) for smooth nonlinear - rheologies; `"continuation"` (staged Picard→Newton) for robustness far from the - solution. The rotated constraint is transparent to the tangent. -- The constraint **reaction** is the boundary normal traction σ_nn - (`solver.boundary_normal_traction(boundary)` / `solver.dynamic_topography(...)`) — no - augmented-Lagrangian splitting. - -**Reserve Nitsche / penalty** (`add_nitsche_bc`) for BCs that must **evolve in time** -(e.g. a Dirichlet→Neumann / traction ramp) — a hard rotated constraint cannot morph. - -**Implementation**: `src/underworld3/utilities/rotated_bc.py` (per-node rotation `Q`, -strong `v_n=0`, reaction = σ_nn); the solve dispatch is in -`src/underworld3/cython/petsc_generic_snes_solvers.pyx`. - ---- - -## Data Access Patterns - -**Authoritative Reference**: `docs/developer/subsystems/data-access.md` -(governing document per the Style Charter §10 authority map; see also the -master authority index in `docs/developer/index.md`) -**Pattern Checker**: Use `/check-patterns` to scan for deprecated patterns - -### Quick Summary -| Pattern | Status | Use Instead | -|---------|--------|-------------| -| `with mesh.access(var):` | **Deprecated** | Direct: `var.data[...]` | -| `with swarm.access(var):` | **Deprecated** | Direct: `var.data[...]` | -| `mesh.data` (coordinates) | **Deprecated** | `mesh.X.coords` | - -See `docs/developer/subsystems/data-access.md` for full patterns, array shapes, -and cache safety details (`docs/developer/UW3_Style_and_Patterns_Guide.md` is the -broader style reference; where the two disagree, `data-access.md` governs). - ---- - -## Expression Processing - -### Unwrap Before Extracting Atoms -When extracting `.atoms()` or `.free_symbols` from expressions before compilation: +| Content | Location | +|---|---| +| Subsystem documentation | `docs/developer/subsystems/` | +| Architecture and design decisions | `docs/developer/design/` | +| How-to guides | `docs/developer/guides/` | +| User tutorials | `docs/beginner/tutorials/` | +| Advanced user guides | `docs/advanced/` | +| Session scripts, benchmarks, profilers | `scripts/sessions/` | +| Simulation output worth keeping | `~/+Simulations/` | -```python -# CORRECT ORDER: -# 1. First unwrap UWexpressions to reveal hidden coordinates -if any_uwexpressions_in_expression: - expr = _unwrap_for_compilation(expr, keep_constants=False, return_self=False) -# 2. Then extract atoms/symbols from the FULLY PROCESSED expression -symbols = expr.atoms(...) -``` +**All documentation goes under `docs/`** — never the repository root, `src/`, or +`tests/`. MyST Markdown for Sphinx: ` ```python ` blocks, `{note}`/`{warning}` +admonitions, `$inline$` and `$$display$$` math. Verify with `pixi run docs-build`. -**Safe locations**: JIT Compiler (`utilities/_jitextension.py`), `extract_expressions()` -**Check if issues**: `is_pure_sympy_expression()` in `function/pure_sympy_evaluator.py`, `utilities/nondimensional.py` +Plan files in `~/.claude/plans/` take descriptive kebab-case names that say what +they are about (`mesh-adaptation-architecture.md`), never whimsical ones. --- -## Swarm Concepts +## Rulings a session needs in hand -### Migration -Migration moves particles between processors based on spatial location. -- Happens automatically when particles move -- Use `migration_disabled()` context for batch operations -- Essential for parallel correctness +**Free-slip: prefer rotated strong free-slip.** +`solver.add_rotated_freeslip_bc(conds, boundary, normal=None)`, value first +(`conds=0` is free-slip). It enforces `v·n̂ = 0` to machine precision where +Nitsche/penalty leaks ~1e-3, is correct on curved and deformed boundaries, works +inside the nonlinear SNES and with geometric FMG, and its reaction is the boundary +normal traction. Leave `normal=None` unless the constraint must follow the true +surface rather than the mesh. Reserve `add_nitsche_bc` for BCs that must *evolve in +time* — a hard rotated constraint cannot morph. +Governing doc: [`subsystems/rotated-freeslip.md`](docs/developer/subsystems/rotated-freeslip.md). -### Proxy Mesh Variables -Swarm variables with `proxy_degree > 0` create proxy mesh variables using RBF interpolation. -- Used for integration and derivative calculations -- Must be updated when swarm data/positions change -- Update happens automatically via `swarmVar._update()` +**Data access.** New code uses `.array` with three-index shapes — scalars +`(N,1,1)`, vectors `(N,1,dim)`, tensors `(N,dim,dim)` — and `mesh.X.coords` for +coordinates. `with mesh.access(...)`, `with swarm.access(...)`, `mesh.data` and +`mesh.points` exist only so old code keeps running. The flat `.data` has exactly +one sanctioned use: a raw variable-to-variable copy inside the +non-dimensionalisation boundary. Solver internals use `vec`. +Governing doc: [`subsystems/data-access.md`](docs/developer/subsystems/data-access.md). ---- - -## Mathematical Objects - -Variables support natural mathematical syntax: +**Unwrap before extracting atoms.** UWexpressions hide coordinates, so order +matters: ```python -# Direct arithmetic (no .sym needed) -momentum = density * velocity -strain_rate = velocity[0].diff(x) + velocity[1].diff(y) - -# Full SymPy Matrix API available -velocity.T # Transpose -velocity.dot(other) # Dot product -velocity.norm() # Magnitude +expr = _unwrap_for_compilation(expr, keep_constants=False, return_self=False) +symbols = expr.atoms(...) # only now ``` -**Implementation**: `MathematicalMixin` in `utilities/mathematical_mixin.py` - ---- - -## Coding Conventions - -### Prefer Glob and Grep Over find -**Use Glob and Grep tools instead of `find` or `grep` in Bash.** They are safer (no `-exec`), faster, and don't require user approval. Only fall back to `find` via Bash if Glob/Grep genuinely cannot express the query. - -### Desktop Notifications for Background Monitoring -When using CronCreate for background monitoring (CI status, issues, etc.), use -platform-appropriate notification commands. Both are in the allowed tools list: -- **macOS**: `osascript -e 'display notification "message" with title "title" sound name "Glass"'` -- **Linux**: `notify-send "title" "message"` +**Never name a variable `model`.** Use `constitutive_model` (material behaviour) or +`orchestration_model` / `uw.get_default_model()` (serialization). Two unrelated +concepts share the word. -Be quiet when everything is fine — only notify when something needs attention. +**Units.** Accept strings, store and return Pint objects: `uw.quantity(1e21, "Pa*s")`. +`.units` returns a Pint *Unit*; call `.to("m")` on the Quantity, not on `.units`. +Governing doc: [`design/UNITS_SIMPLIFIED_DESIGN_2025-11.md`](docs/developer/design/UNITS_SIMPLIFIED_DESIGN_2025-11.md). -### Plan File Naming Policy -**Plan files must have descriptive names that indicate their content.** +**Parallelism.** PETSc handles synchronisation — avoid direct mpi4py. Use +`uw.pprint()` and `uw.selective_ranks()`; a user should never see an MPI call. -``` -# GOOD - Descriptive names -mesh-adaptation-architecture.md -gradient-evaluation-p2-fix.md -units-system-refactor-plan.md - -# BAD - Random/whimsical names -proud-petting-pretzel.md -happy-dancing-dolphin.md -``` +**Swarms.** Migration moves particles between processors automatically; batch with +`migration_disabled()`. Swarm variables with `proxy_degree > 0` carry a proxy mesh +variable that must be refreshed (`swarmVar._update()`) when data or positions move. -When creating plan files in `~/.claude/plans/`, use kebab-case names that describe: -- The feature or subsystem being worked on -- The type of work (architecture, fix, refactor, feature) +**Prefer Glob and Grep over `find`/`grep` in Bash** — safer, faster, no approval +prompt. -### Avoid Ambiguous 'model' -Two different "model" concepts exist: -- `uw.Model`: Serialization/orchestration system -- Constitutive models: Material behavior (ViscousFlowModel, etc.) - -```python -# GOOD - Clear and unambiguous -constitutive_model = stokes.constitutive_model -orchestration_model = uw.get_default_model() +**Desktop notification** from a background monitor — be quiet unless something +needs attention: -# AVOID - Ambiguous -model = stokes.constitutive_model +```bash +osascript -e 'display notification "msg" with title "title" sound name "Glass"' # macOS +notify-send "title" "msg" # Linux ``` --- -## Test Classification - -### By Complexity Level (pytest markers) -- `@pytest.mark.level_1`: Quick core tests (seconds) -- `@pytest.mark.level_2`: Intermediate tests (minutes) -- `@pytest.mark.level_3`: Physics/solver tests (minutes to hours) +## Tests -### By Reliability Tier -- `@pytest.mark.tier_a`: Production-ready (TDD-safe) -- `@pytest.mark.tier_b`: Validated (use with caution) -- `@pytest.mark.tier_c`: Experimental (development only) +Files are `tests/test_NNNN_description.py` and carry both a level +(`level_1`/`level_2`/`level_3`) and a tier (`tier_a`/`tier_b`/`tier_c`). ```bash -# Quick validation -pytest -m "level_1 and tier_a" - -# Full validation -pytest -m "tier_a or tier_b" +pytest -m "level_1 and tier_a" # quick +pytest -m "tier_a or tier_b" # full validation ``` -**Details**: `docs/developer/TESTING-RELIABILITY-SYSTEM.md` +Tier A is hardened and reviewed — safe to build code around. Tier C is not mature +enough to drive coding. Every bug fix ships the regression test that would have +caught it, written first and shown to fail. Validate a new test's own correctness +before changing library code to satisfy it. ---- - -## On-Demand Documentation References - -When working on specific subsystems, these documents provide detailed guidance. -**Read them on demand using the Read tool** — do NOT load them all at conversation start. - -> **AI Assistant Protocol**: When you need deeper context, explicitly tell the user -> what you're reading and why. Use the Read tool to load the specific file. -> Example: "Let me check the units design doc for this..." - -### Units & Scaling -- `docs/developer/design/UNITS_SIMPLIFIED_DESIGN_2025-11.md` - **Authoritative** units architecture -- `docs/developer/ai-notes/COORDINATE-UNITS-TECHNICAL-NOTE.md` - Coordinate unit handling -- `docs/developer/design/WHY_UNITS_NOT_DIMENSIONALITY.md` - Design rationale - -### Testing -- `docs/developer/TESTING-RELIABILITY-SYSTEM.md` - Test tier classification (A/B/C) -- `docs/developer/ai-notes/TEST-CLASSIFICATION-2025-11-15.md` - Current test status - -### Code Style, Workflow & Patterns -- `docs/developer/guides/branching-strategy.md` - Branching, releases, API change discipline -- `docs/developer/UW3_Style_and_Patterns_Guide.md` - Development standards - -### Data Access & Variables -- `docs/developer/subsystems/data-access.md` - Data access patterns, self-validating cache -- `docs/developer/UW3_Developers_NDArrays.md` - NDArray_With_Callback internals - -### Architecture & Design -- `docs/developer/design/ARCHITECTURE_ANALYSIS.md` - System structure analysis -- `docs/developer/design/MATHEMATICAL_MIXIN_DESIGN.md` - Mathematical objects internals -- `docs/developer/design/GEOGRAPHIC_COORDINATE_SYSTEM_DESIGN.md` - Spherical/planetary meshes -- `docs/developer/design/SYMBOL_DISAMBIGUATION_2025-12.md` - Multi-mesh symbol identity -- `docs/developer/TEMPLATE_EXPRESSION_PATTERN.md` - Solver template expressions - -### Coordinates & Mesh -- `docs/developer/design/COORDINATE_MIGRATION_GUIDE.md` - Coordinate system changes -- `docs/developer/design/mesh-geometry-audit.md` - Mesh geometry patterns - -### Development History -- `docs/developer/ai-notes/historical-notes.md` - Completed migrations, fixed bugs +Every test file must be reachable from a glob in `scripts/test.sh`; +`scripts/check_test_coverage.py` enforces it. --- -## Quick Reference +## Finding the rest -### Build & Test Commands -```bash -./uw build # Rebuild after source changes (preferred) -./uw test # Run test suite -pixi run -e default python # Run Python in environment -``` +`docs/developer/index.md` carries the **authority map** — one governing document +per topic. Read it rather than guessing which of several documents on a topic is +current. Development history and completed migrations are in +`docs/developer/ai-notes/historical-notes.md`. -### Key Files -- `src/underworld3/mpi.py` - Parallel safety implementation -- `src/underworld3/scaling/` - Units system -- `src/underworld3/utilities/mathematical_mixin.py` - Mathematical objects -- `src/underworld3/function/expressions.py` - UWexpression (lazy evaluation, symbol disambiguation) -- `src/underworld3/function/_function.pyx` - UnderworldFunction (mesh variable symbols) -- `src/underworld3/discretisation/enhanced_variables.py` - EnhancedMeshVariable (units, math ops, persistence) - -### Historical Notes -For development history, completed migrations, and fixed bugs: -See `docs/developer/ai-notes/historical-notes.md` - ---- +Key implementation files: -*Reorganized 2025-12-13: Historical content moved to docs/developer/ai-notes/historical-notes.md* +| | | +|---|---| +| `src/underworld3/mpi.py` | parallel-safe output | +| `src/underworld3/scaling/` | units system | +| `src/underworld3/function/expressions.py` | UWexpression, lazy evaluation | +| `src/underworld3/function/_function.pyx` | mesh-variable symbols | +| `src/underworld3/utilities/mathematical_mixin.py` | mathematical objects | +| `src/underworld3/utilities/rotated_bc.py` | rotated free-slip | +| `src/underworld3/discretisation/enhanced_variables.py` | units, math ops, persistence | diff --git a/docs/developer/guides/development-setup.md b/docs/developer/guides/development-setup.md index acb18c9bc..cbef13f33 100644 --- a/docs/developer/guides/development-setup.md +++ b/docs/developer/guides/development-setup.md @@ -2,43 +2,92 @@ title: "Development Environment Setup" --- -# Setting Up Underworld3 Development Environment - -```{note} Coming Soon -This section will cover: -- Pixi environment setup and management -- Required dependencies and versions -- IDE configuration and tools -- Building from source -- Docker development containers - -**Priority**: High - needed for new developer onboarding -**Current Status**: Placeholder - needs development +# Setting up an Underworld3 development environment + +Underworld3 builds Cython extensions against a specific PETSc, so the build is +not as forgiving as a pure-Python package. The constraints below are the ones +that cost real time when they are broken. + +## Everyday commands + +```bash +./uw build # rebuild after ANY source change +./uw test # run the test suite +pixi run -e default python # a Python in the environment ``` -## Quick Commands +## Rebuild after every source change -For now, the essential commands are: +Underworld3 is *installed* into the pixi environment, not imported from `src/`. +Editing a file under `src/underworld3/` changes nothing until you rebuild: ```bash -# Build Underworld3 -pixi run underworld-build +./uw build +``` + +Check what you are actually running with `uw.__file__` — it must point into +`.pixi/envs//lib/python3.12/site-packages/underworld3/`, never `src/`. -# Run tests -pixi run underworld-test +`./uw build` passes `--no-cache-dir`, because the version is always `0.0.0` and +pip will otherwise reuse a stale wheel. If you still suspect stale code, clear +the intermediate build tree as well: -# Python environment -pixi run -e default python script.py +```bash +rm -rf build/lib.* build/bdist.* +./uw build ``` -## Environment Information +A `.pyx` change that appears not to take effect is almost always this: see also +[the JIT cache notes](../subsystems/jit-cache.md). -From CLAUDE.md context: -- Uses pixi for environment management -- Key dependencies: underworld3, petsc4py, numpy, scipy -- Must use `pixi run underworld-build` after ANY source code changes -- No in-place builds due to Cython/PETSc complexity +## Never use an editable install ---- +`pip install -e .` is prohibited. There are no exceptions, and the reason is +that the damage outlives the install: + +- the `.pth` files it writes **contaminate every pixi environment** sharing the + source directory, and worktrees share environments by symlink; +- they **persist after uninstall**, silently redirecting imports back to `src/` + even once a proper `./uw build` has run; +- `.so` files left in the source tree get loaded by an environment expecting a + different PETSc arch, which surfaces as a `dlopen` symbol error rather than + as anything that names the real cause. + +Always `./uw build`, which runs a non-editable `pip install .`. Where `./uw` is +unavailable: + +```bash +pixi run -e pip install . --no-build-isolation --no-cache-dir +``` + +### Recovering from editable-install contamination + +```bash +find .pixi/envs -name "__editable__*underworld*" -delete # stale .pth, all envs +find src/underworld3 -name "*.so" -delete # .so belong in site-packages +rm -rf build/ +./uw build +``` + +## PETSc is not relocatable + +``` +/Users/lmoresi/+Underworld/underworld-pixi-2/petsc/ +``` + +**Do not move this directory.** PETSc hardcodes paths at configure time, so +moving it breaks petsc4py and every pixi task that depends on it, and the only +repair is a full rebuild of roughly an hour. Worktrees share this one PETSc by +symlink for exactly that reason; everything else in a worktree environment is +its own. + +## Worktrees have their own environments + +Each worktree carries its own pixi environment — its own site-packages and its +own compiled extensions — with only PETSc shared. `./uw build` installs the +source of whichever worktree you run it from, into that worktree's environment. +So: enter the worktree first, then build, then run. Building from the main +checkout does nothing for a worktree. -*This document will be expanded to include complete setup instructions.* \ No newline at end of file +See [Branching and Release Strategy](branching-strategy.md) for the worktree +lifecycle and the branch policy.