diff --git a/CLAUDE.md b/CLAUDE.md index 0f7868a1..b49f666e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,566 +1,197 @@ # Underworld3 AI Assistant Context > **MANDATORY**: Read `docs/developer/UW3_STYLE_CHARTER.md` before writing any code. -> It is the normative style contract for every session (human or AI), it is two pages, -> and it WINS over the surrounding code and over any other style document on conflict. +> It is the normative style contract for every session, it is two pages, and it WINS +> over the surrounding code and over any other style document on conflict. > Core clause: match the Charter, not the code next door — and flag deviations you find. -> **Note**: Human-readable developer documentation is in `docs/developer/` (Sphinx/MyST format). -> For development history and completed migrations, see `docs/developer/ai-notes/historical-notes.md` +This file is a router, not a manual. It carries what a session needs *before* it +knows where to look; everything else is one hop away through the authority map in +`docs/developer/index.md`. --- -## Extended AI Context (Optional) +## Session bootstrap -You can configure additional AI instruction files that live outside the Underworld repository. This is useful for: -- Personal coding style preferences -- Project planning and coordination systems -- Private instructions not shared with other contributors +**External planning.** If `UW_AI_TOOLS_PATH` is set (colon-separated directories), +check each for `.md` files — `underworld.md` especially — and report relevant +Active/Bugs items briefly. If it is unset, proceed with repo-local context. Set it +via `./uw setup`. -**Setup**: If the environment variable `UW_AI_TOOLS_PATH` is set (colon-separated directories, like `PATH`), check each directory for `.md` files containing additional context. Configure this via `./uw setup` or manually in your shell profile. - -**At conversation start**: If `UW_AI_TOOLS_PATH` is set, look for `underworld.md` (or other relevant files) in those directories. If found, report relevant Active/Bugs items briefly. If the variable is unset or files aren't found, proceed normally with repo-local context only. - -### Responding to External Planning Items - -When completing a task from an external planning file, add an annotation directly below the item: +When you complete a task from an external planning file, annotate it in place: ```markdown ``` -### Adding New Items to External Planning - -If you discover bugs, identify new tasks, or have items that should be tracked in the external planning file: -- **Don't create local TODO files or add to CLAUDE.md** -- **Do add to the external planning file** under the appropriate section (Bugs, Active, Nice to Have) -- Use the project tag: `` - -### Inline TODO Comments in Code - -Use inline `TODO` comments to mark **problem locations** in the source code. These provide self-documentation and help future developers (human or AI) find the relevant code quickly. - -**Format:** -```python -# TODO(BUG): Brief description of the issue -# More context if needed -# See planning file: underworld.md (section, date) -``` +Add newly discovered work to the external planning file under the appropriate +section with a `` tag — not to a local TODO +file, and not here. Don't rewrite strategic paragraphs, move items between +sections, or restructure the document; those carry cross-project context and are +handled by the planning tools. -**When to use:** -- Mark the exact location of a known bug -- Flag code that needs enhancement or refactoring -- Note incomplete implementations +**Inline TODOs** mark the *place*; the planning file tracks the *work*: -**Example:** ```python # TODO(BUG): add_natural_bc() causes PETSc error 73 # The Stokes solver works; issue is specific to scalar Poisson setup. # See planning file: underworld.md (Bugs section, 2026-01-19) -self.natural_bcs = [] ``` -This complements the planning file — the planning file tracks *what* needs doing, inline TODOs mark *where* in the code. - -### What Not to Do - -- Don't rewrite strategic paragraphs — they contain cross-project context -- Don't move items between sections (Active → Done) — planning-claude handles that -- Don't restructure the document - -If something needs more than an annotation or simple addition, mention it in conversation for the user to handle. - --- -## Documentation Requests - -**⚠️ MANDATORY - READ BEFORE WRITING ANY DOCUMENTATION ⚠️** - -- **ALL documentation MUST go in `docs/` directory** - NO exceptions -- **NEVER create .md files in the repository root, src/, tests/, or anywhere else** -- **NEVER create planning/design documents outside `docs/developer/design/`** -- If you're tempted to create a file like `SOME-FEATURE-NOTES.md` in the repo root - **DON'T**. Put it in `docs/developer/` instead. -- This applies to: design docs, how-to guides, technical notes, implementation plans, reviews, audits - EVERYTHING goes in `docs/` - -**Where to put documentation:** - -| Content Type | Location | -|--------------|----------| -| System documentation (meshing, solvers, swarms) | `docs/developer/subsystems/` | -| Architecture and design decisions | `docs/developer/design/` | -| How-to guides and best practices | `docs/developer/guides/` | -| User tutorials | `docs/beginner/tutorials/` | -| Advanced user guides | `docs/advanced/` | - -**Format** - Use MyST Markdown (`.md` files) compatible with Sphinx: -- Standard markdown with MyST extensions -- Use ```` ```python ```` for code blocks (not `{python}`) -- Use `{note}`, `{warning}`, `{tip}` for admonitions -- Math: `$inline$` and `$$display$$` - -**Style** - Concise, helpful, standalone: -- Self-contained explanations (don't assume reader has context) -- Include practical code examples -- Link to related documentation where appropriate -- Focus on "why" and "how to use", not just "what" -- Follow the notebook style guide for tutorials - -**Integration** - Link into the documentation system: -- Add to appropriate toctree in parent `index.md` -- Cross-reference related docs with `:doc:` or relative links -- Build and verify: `pixi run docs-build` - -**Style references**: -- Notebook writing: `docs/developer/guides/notebook-style-guide.md` -- Code patterns: `docs/developer/UW3_Style_and_Patterns_Guide.md` - ---- - -## Git and Branching Strategy - -**Full guide**: `docs/developer/guides/branching-strategy.md` - -### Branch Roles -- **`main`** — stable releases (tagged quarterly). No direct pushes. -- **`development`** — integration branch. Bug fixes land here. Features merge here via PR. -- **`feature/*`** — long-lived feature work. Branch from and PR back to `development`. - -### Key Discipline: Separate API from Implementation -Feature branches must not introduce API changes (new methods, changed signatures) that other branches can't access. When a feature needs an API change: -1. Extract the interface (stub or minimal implementation) into a separate commit. -2. Merge that to `development` first (or extract after the fact). -3. The feature PR should only contain implementation behind already-merged interfaces. - -This keeps feature branches independent and makes cross-pollination of fixes straightforward. +## Hard constraints -### Bug Fix Flow -- Fix on `development` (commit or small PR) -- Cherry-pick to `main` if critical → tag patch release -- Cherry-pick to active feature branches (underworld-claude handles this) +**Rebuild after every source change.** `./uw build`. Underworld3 is installed into +the environment, not imported from `src/`; verify with `uw.__file__`. -### Git Worktrees for Session Isolation -**Use a worktree for any multi-file change** (docs cleanup, refactoring, features). -Multiple Claude sessions sharing one working directory will overwrite each other's work. +**Never use an editable install.** `pip install -e .` contaminates every pixi +environment sharing the source tree and persists after uninstall. No exceptions. -Each worktree gets its **own pixi environment** (isolated site-packages, own -compiled extensions). Only PETSc is shared via symlink (non-relocatable, -expensive to rebuild). `./uw build` from inside a worktree installs that -worktree's source into the worktree's own environment. +**Never move `/Users/lmoresi/+Underworld/underworld-pixi-2/petsc/`.** PETSc +hardcodes paths at configure time; moving it costs a full rebuild. -**Full documentation**: `docs/developer/guides/branching-strategy.md` (Git Worktrees section) +**Worktrees are never on `development` or `main`.** All work happens on a side +branch (`feature/…`, `bugfix/…`, `docs/…`) and merges via PR. Use a worktree for +any multi-file change — concurrent sessions sharing one checkout overwrite each +other. Each worktree builds into its own environment, so enter it *first*, then +build. -#### Worktree branch policy +Detail: [`guides/development-setup.md`](docs/developer/guides/development-setup.md), +[`guides/branching-strategy.md`](docs/developer/guides/branching-strategy.md). -**Worktrees must NEVER be on `development` or `main` directly.** +**Attribution.** End commit messages and PR bodies with: -In this repo, `main` is the *release* branch (tagged quarterly, essentially -read-only history) and `development` is the integration trunk where active -work converges. The default repository checkout (`~/+Underworld/underworld3-pixi`) -should usually sit on `development` — that's where you read the current -working state and pull updates. All work — even work intended to land -on `development` — happens on a side branch (`feature/...`, `bugfix/...`, -`docs/...`) in a worktree, then merges to `development` via PR. - -`./uw worktree create` enforces this for new worktrees (always on a -side branch, reset to `origin/development`). It's on you not to break -it manually: - -- Never `git checkout development` (or `main`) inside a worktree -- Never `git worktree add ... development` to put a worktree on - `development` directly -- If you find a worktree on `development` (e.g. from older tooling), - branch off immediately (`git switch -c bugfix/whatever`) before - committing - -#### Creating and using a worktree - -```bash -# Create — own .pixi env, shared PETSc, names the branch -./uw worktree create # → feature/ -./uw worktree create bugfix # → bugfix/ - -# Work — drops you into a shell cd'd to the worktree -./uw worktree shell -./uw build # builds from THIS source into THIS worktree's env -./uw test # runs tests -exit # leave - -# List worktrees with branch and status -./uw worktree list - -# Bring files from other branches without switching: -git checkout origin/ -- path/to/file ``` - -#### Cleanup - -```bash -# Removes worktree directory and deletes the branch -./uw worktree remove -``` - -#### Important: always build and run from inside the worktree - -Each worktree has its own pixi environment. `./uw build` installs into the -environment of whichever worktree (or main repo) you run it from. Always: - -1. `./uw worktree shell ` (or `cd` into the worktree) -2. `./uw build` -3. Run your code / tests from there - -### AI-Assisted Attribution (Commits and PRs) -When committing code or creating pull requests with AI assistance, end the -message/body with: - -``` -Underworld development team with AI support from [Claude Code](https://claude.com/claude-code) -``` - -(In commit messages, use the plain-text form without the markdown link.) - -**Do NOT use**: -- `Co-Authored-By:` with a noreply email (useless for soliciting responses) -- Generic AI attribution without team context -- Emoji in PR descriptions - ---- - -## CRITICAL BUILD CONSTRAINTS - -### PETSc Directory (DO NOT MOVE) -**WARNING**: `/Users/lmoresi/+Underworld/underworld-pixi-2/petsc/` MUST NOT be moved. -- PETSc is NOT relocatable after compilation (hardcoded paths) -- Moving breaks petsc4py bindings and all pixi tasks -- Requires complete rebuild (~1 hour) if relocated - -### Rebuild After Source Changes -**After modifying source files, always run `./uw build`!** -- Underworld3 is installed as a package in the pixi environment -- Changes go to `.pixi/envs//lib/python3.12/site-packages/underworld3/` -- Verify with `uw.__file__` (should show site-packages path, NOT `src/`) - -**Note**: `./uw build` uses `--no-cache-dir` to prevent pip from reusing stale -wheels (UW3 is always version `0.0.0`). If you still suspect stale code, clean -the build directory: `rm -rf build/lib.* build/bdist.*` then rebuild. - -### NEVER Use Editable Installs -**DO NOT use `pip install -e .` (editable/development mode)!** -This is a hard rule — there are no exceptions. - -Editable installs create `.pth` files and `.so` symlinks in the source tree that: -- **Contaminate all pixi environments** sharing the same source directory -- **Break worktree isolation** (worktrees share pixi envs via symlinks) -- **Persist after uninstall** — stale `.pth` files redirect Python imports to `src/` - even after a proper `./uw build`, causing import errors or wrong library loading -- **Mix debug/release builds** — `.so` compiled against one PETSc arch get loaded - by environments expecting another, causing `dlopen` symbol errors - -Always use `./uw build` which runs `pip install .` (non-editable). If `./uw build` -is not available, use `pixi run -e pip install . --no-build-isolation --no-cache-dir`. - -**Recovery from editable install contamination:** -```bash -# Remove stale .pth files from ALL environments -find .pixi/envs -name "__editable__*underworld*" -delete -# Remove .so from source tree (they belong in site-packages) -find src/underworld3 -name "*.so" -delete -# Clean build cache -rm -rf build/ -# Rebuild properly -./uw build +Underworld development team with AI support from Claude Code ``` -### Test Quality Principles -**New tests must be validated before making code changes to fix them!** -- Validate test correctness before changing main code -- If core tests (0000-0599) pass, the system is working correctly -- Disable problematic new tests, validate core functionality, then fix test structure - -### JOSS Paper (FROZEN) -**Location**: `publications/joss-paper/` - Publication of record, DO NOT modify. - ---- - -## Units System Principles - -**Authoritative design doc**: `docs/developer/design/UNITS_SIMPLIFIED_DESIGN_2025-11.md` - -- Accept strings for convenience, store/return Pint objects: `uw.quantity(1e21, "Pa*s")` -- `.units` returns a Pint **Unit** (not string) — call `.to("m")` on the **Quantity**, not on `.units` -- UWexpression derives `.units` from contents (atomic: stored value; composite: `get_units(self._sym)`) - ---- - -## Parallel Computing Patterns +(PR bodies may link it.) Do not use `Co-Authored-By:` with a noreply address, and +no emoji in PR descriptions. -PETSc handles all parallel synchronization — avoid direct mpi4py unless necessary. -Use `uw.pprint()` and `uw.selective_ranks()` for rank-safe output and code blocks. +**Solver stability is paramount.** The PETSc-based solvers in +`petsc_generic_snes_solvers` are carefully tuned and validated. No changes +without extensive benchmarking. -**Implementation**: `src/underworld3/mpi.py` -**Documentation**: `docs/advanced/parallel-computing.md` +**The JOSS paper (`publications/joss-paper/`) is frozen.** It is the publication of +record; do not modify it. --- -## Architecture Priorities - -### Solver Stability is Paramount -The PETSc-based solvers are carefully optimized and validated. **NO CHANGES without extensive benchmarking.** - -### Module Boundaries - -| Module | Purpose | Access Pattern | -|--------|---------|----------------| -| **Solvers** (`petsc_generic_snes_solvers`) | High-performance PETSc solving | Direct `vec` property | -| **Mesh Variables** | User-facing field data | `array` property (new) | -| **Swarm Variables** | Particle data with mesh proxies | `data` property | - -### Conservative Migration Strategy -- **User-facing code**: Use `array` property with automatic sync -- **Solver internals**: Keep using `vec` property with direct PETSc access -- **Gradual transition**: Only change when driven by actual needs - ---- +## Where things go -## Boundary Conditions: Free-slip - -**Prefer rotated strong free-slip** (`solver.add_rotated_freeslip_bc(conds, boundary, normal=None)`, value-first: `conds=0` is free-slip; a non-zero scalar/expression `conds` prescribes the wall-normal datum `u·n̂ = ũ_n` strongly) -to impose `v·n̂ = 0`: - -- Enforces zero wall-normal flow to **machine precision** (Nitsche / penalty leak ~1e-3). -- Correct on **curved / tilted / deformed** boundaries — the normal is taken per node, - measure-weighted so it matches the straight-facet integral the assembler evaluates - (#560). Leave `normal=None` unless the constraint must follow the TRUE surface rather - than the mesh; an analytic `normal` (e.g. `X/|X|`) is exact for the geometry but keeps - a consistency error against the faceted assembly. See - `docs/developer/subsystems/rotated-freeslip.md` ("Which normal to use"). -- Works **inside the nonlinear SNES** and with **geometric FMG**. It honours - `solver.consistent_jacobian`: use `True` (consistent Newton) for smooth nonlinear - rheologies; `"continuation"` (staged Picard→Newton) for robustness far from the - solution. The rotated constraint is transparent to the tangent. -- The constraint **reaction** is the boundary normal traction σ_nn - (`solver.boundary_normal_traction(boundary)` / `solver.dynamic_topography(...)`) — no - augmented-Lagrangian splitting. - -**Reserve Nitsche / penalty** (`add_nitsche_bc`) for BCs that must **evolve in time** -(e.g. a Dirichlet→Neumann / traction ramp) — a hard rotated constraint cannot morph. - -**Implementation**: `src/underworld3/utilities/rotated_bc.py` (per-node rotation `Q`, -strong `v_n=0`, reaction = σ_nn); the solve dispatch is in -`src/underworld3/cython/petsc_generic_snes_solvers.pyx`. - ---- - -## Data Access Patterns - -**Authoritative Reference**: `docs/developer/subsystems/data-access.md` -(governing document per the Style Charter §10 authority map; see also the -master authority index in `docs/developer/index.md`) -**Pattern Checker**: Use `/check-patterns` to scan for deprecated patterns - -### Quick Summary -| Pattern | Status | Use Instead | -|---------|--------|-------------| -| `with mesh.access(var):` | **Deprecated** | Direct: `var.data[...]` | -| `with swarm.access(var):` | **Deprecated** | Direct: `var.data[...]` | -| `mesh.data` (coordinates) | **Deprecated** | `mesh.X.coords` | - -See `docs/developer/subsystems/data-access.md` for full patterns, array shapes, -and cache safety details (`docs/developer/UW3_Style_and_Patterns_Guide.md` is the -broader style reference; where the two disagree, `data-access.md` governs). - ---- - -## Expression Processing - -### Unwrap Before Extracting Atoms -When extracting `.atoms()` or `.free_symbols` from expressions before compilation: +| Content | Location | +|---|---| +| Subsystem documentation | `docs/developer/subsystems/` | +| Architecture and design decisions | `docs/developer/design/` | +| How-to guides | `docs/developer/guides/` | +| User tutorials | `docs/beginner/tutorials/` | +| Advanced user guides | `docs/advanced/` | +| Session scripts, benchmarks, profilers | `scripts/sessions/` | +| Simulation output worth keeping | `~/+Simulations/` | -```python -# CORRECT ORDER: -# 1. First unwrap UWexpressions to reveal hidden coordinates -if any_uwexpressions_in_expression: - expr = _unwrap_for_compilation(expr, keep_constants=False, return_self=False) -# 2. Then extract atoms/symbols from the FULLY PROCESSED expression -symbols = expr.atoms(...) -``` +**All documentation goes under `docs/`** — never the repository root, `src/`, or +`tests/`. MyST Markdown for Sphinx: ` ```python ` blocks, `{note}`/`{warning}` +admonitions, `$inline$` and `$$display$$` math. Verify with `pixi run docs-build`. -**Safe locations**: JIT Compiler (`utilities/_jitextension.py`), `extract_expressions()` -**Check if issues**: `is_pure_sympy_expression()` in `function/pure_sympy_evaluator.py`, `utilities/nondimensional.py` +Plan files in `~/.claude/plans/` take descriptive kebab-case names that say what +they are about (`mesh-adaptation-architecture.md`), never whimsical ones. --- -## Swarm Concepts +## Rulings a session needs in hand -### Migration -Migration moves particles between processors based on spatial location. -- Happens automatically when particles move -- Use `migration_disabled()` context for batch operations -- Essential for parallel correctness +**Free-slip: prefer rotated strong free-slip.** +`solver.add_rotated_freeslip_bc(conds, boundary, normal=None)`, value first +(`conds=0` is free-slip). It enforces `v·n̂ = 0` to machine precision where +Nitsche/penalty leaks ~1e-3, is correct on curved and deformed boundaries, works +inside the nonlinear SNES and with geometric FMG, and its reaction is the boundary +normal traction. Leave `normal=None` unless the constraint must follow the true +surface rather than the mesh. Reserve `add_nitsche_bc` for BCs that must *evolve in +time* — a hard rotated constraint cannot morph. +Governing doc: [`subsystems/rotated-freeslip.md`](docs/developer/subsystems/rotated-freeslip.md). -### Proxy Mesh Variables -Swarm variables with `proxy_degree > 0` create proxy mesh variables using RBF interpolation. -- Used for integration and derivative calculations -- Must be updated when swarm data/positions change -- Update happens automatically via `swarmVar._update()` +**Data access.** New code uses `.array` with three-index shapes — scalars +`(N,1,1)`, vectors `(N,1,dim)`, tensors `(N,dim,dim)` — and `mesh.X.coords` for +coordinates. `with mesh.access(...)`, `with swarm.access(...)`, `mesh.data` and +`mesh.points` exist only so old code keeps running. The flat `.data` has exactly +one sanctioned use: a raw variable-to-variable copy inside the +non-dimensionalisation boundary. Solver internals use `vec`. +Governing doc: [`subsystems/data-access.md`](docs/developer/subsystems/data-access.md). ---- - -## Mathematical Objects - -Variables support natural mathematical syntax: +**Unwrap before extracting atoms.** UWexpressions hide coordinates, so order +matters: ```python -# Direct arithmetic (no .sym needed) -momentum = density * velocity -strain_rate = velocity[0].diff(x) + velocity[1].diff(y) - -# Full SymPy Matrix API available -velocity.T # Transpose -velocity.dot(other) # Dot product -velocity.norm() # Magnitude +expr = _unwrap_for_compilation(expr, keep_constants=False, return_self=False) +symbols = expr.atoms(...) # only now ``` -**Implementation**: `MathematicalMixin` in `utilities/mathematical_mixin.py` - ---- - -## Coding Conventions - -### Prefer Glob and Grep Over find -**Use Glob and Grep tools instead of `find` or `grep` in Bash.** They are safer (no `-exec`), faster, and don't require user approval. Only fall back to `find` via Bash if Glob/Grep genuinely cannot express the query. - -### Desktop Notifications for Background Monitoring -When using CronCreate for background monitoring (CI status, issues, etc.), use -platform-appropriate notification commands. Both are in the allowed tools list: -- **macOS**: `osascript -e 'display notification "message" with title "title" sound name "Glass"'` -- **Linux**: `notify-send "title" "message"` +**Never name a variable `model`.** Use `constitutive_model` (material behaviour) or +`orchestration_model` / `uw.get_default_model()` (serialization). Two unrelated +concepts share the word. -Be quiet when everything is fine — only notify when something needs attention. +**Units.** Accept strings, store and return Pint objects: `uw.quantity(1e21, "Pa*s")`. +`.units` returns a Pint *Unit*; call `.to("m")` on the Quantity, not on `.units`. +Governing doc: [`design/UNITS_SIMPLIFIED_DESIGN_2025-11.md`](docs/developer/design/UNITS_SIMPLIFIED_DESIGN_2025-11.md). -### Plan File Naming Policy -**Plan files must have descriptive names that indicate their content.** +**Parallelism.** PETSc handles synchronisation — avoid direct mpi4py. Use +`uw.pprint()` and `uw.selective_ranks()`; a user should never see an MPI call. -``` -# GOOD - Descriptive names -mesh-adaptation-architecture.md -gradient-evaluation-p2-fix.md -units-system-refactor-plan.md - -# BAD - Random/whimsical names -proud-petting-pretzel.md -happy-dancing-dolphin.md -``` +**Swarms.** Migration moves particles between processors automatically; batch with +`migration_disabled()`. Swarm variables with `proxy_degree > 0` carry a proxy mesh +variable that must be refreshed (`swarmVar._update()`) when data or positions move. -When creating plan files in `~/.claude/plans/`, use kebab-case names that describe: -- The feature or subsystem being worked on -- The type of work (architecture, fix, refactor, feature) +**Prefer Glob and Grep over `find`/`grep` in Bash** — safer, faster, no approval +prompt. -### Avoid Ambiguous 'model' -Two different "model" concepts exist: -- `uw.Model`: Serialization/orchestration system -- Constitutive models: Material behavior (ViscousFlowModel, etc.) - -```python -# GOOD - Clear and unambiguous -constitutive_model = stokes.constitutive_model -orchestration_model = uw.get_default_model() +**Desktop notification** from a background monitor — be quiet unless something +needs attention: -# AVOID - Ambiguous -model = stokes.constitutive_model +```bash +osascript -e 'display notification "msg" with title "title" sound name "Glass"' # macOS +notify-send "title" "msg" # Linux ``` --- -## Test Classification - -### By Complexity Level (pytest markers) -- `@pytest.mark.level_1`: Quick core tests (seconds) -- `@pytest.mark.level_2`: Intermediate tests (minutes) -- `@pytest.mark.level_3`: Physics/solver tests (minutes to hours) +## Tests -### By Reliability Tier -- `@pytest.mark.tier_a`: Production-ready (TDD-safe) -- `@pytest.mark.tier_b`: Validated (use with caution) -- `@pytest.mark.tier_c`: Experimental (development only) +Files are `tests/test_NNNN_description.py` and carry both a level +(`level_1`/`level_2`/`level_3`) and a tier (`tier_a`/`tier_b`/`tier_c`). ```bash -# Quick validation -pytest -m "level_1 and tier_a" - -# Full validation -pytest -m "tier_a or tier_b" +pytest -m "level_1 and tier_a" # quick +pytest -m "tier_a or tier_b" # full validation ``` -**Details**: `docs/developer/TESTING-RELIABILITY-SYSTEM.md` +Tier A is hardened and reviewed — safe to build code around. Tier C is not mature +enough to drive coding. Every bug fix ships the regression test that would have +caught it, written first and shown to fail. Validate a new test's own correctness +before changing library code to satisfy it. ---- - -## On-Demand Documentation References - -When working on specific subsystems, these documents provide detailed guidance. -**Read them on demand using the Read tool** — do NOT load them all at conversation start. - -> **AI Assistant Protocol**: When you need deeper context, explicitly tell the user -> what you're reading and why. Use the Read tool to load the specific file. -> Example: "Let me check the units design doc for this..." - -### Units & Scaling -- `docs/developer/design/UNITS_SIMPLIFIED_DESIGN_2025-11.md` - **Authoritative** units architecture -- `docs/developer/ai-notes/COORDINATE-UNITS-TECHNICAL-NOTE.md` - Coordinate unit handling -- `docs/developer/design/WHY_UNITS_NOT_DIMENSIONALITY.md` - Design rationale - -### Testing -- `docs/developer/TESTING-RELIABILITY-SYSTEM.md` - Test tier classification (A/B/C) -- `docs/developer/ai-notes/TEST-CLASSIFICATION-2025-11-15.md` - Current test status - -### Code Style, Workflow & Patterns -- `docs/developer/guides/branching-strategy.md` - Branching, releases, API change discipline -- `docs/developer/UW3_Style_and_Patterns_Guide.md` - Development standards - -### Data Access & Variables -- `docs/developer/subsystems/data-access.md` - Data access patterns, self-validating cache -- `docs/developer/UW3_Developers_NDArrays.md` - NDArray_With_Callback internals - -### Architecture & Design -- `docs/developer/design/ARCHITECTURE_ANALYSIS.md` - System structure analysis -- `docs/developer/design/MATHEMATICAL_MIXIN_DESIGN.md` - Mathematical objects internals -- `docs/developer/design/GEOGRAPHIC_COORDINATE_SYSTEM_DESIGN.md` - Spherical/planetary meshes -- `docs/developer/design/SYMBOL_DISAMBIGUATION_2025-12.md` - Multi-mesh symbol identity -- `docs/developer/TEMPLATE_EXPRESSION_PATTERN.md` - Solver template expressions - -### Coordinates & Mesh -- `docs/developer/design/COORDINATE_MIGRATION_GUIDE.md` - Coordinate system changes -- `docs/developer/design/mesh-geometry-audit.md` - Mesh geometry patterns - -### Development History -- `docs/developer/ai-notes/historical-notes.md` - Completed migrations, fixed bugs +Every test file must be reachable from a glob in `scripts/test.sh`; +`scripts/check_test_coverage.py` enforces it. --- -## Quick Reference +## Finding the rest -### Build & Test Commands -```bash -./uw build # Rebuild after source changes (preferred) -./uw test # Run test suite -pixi run -e default python # Run Python in environment -``` +`docs/developer/index.md` carries the **authority map** — one governing document +per topic. Read it rather than guessing which of several documents on a topic is +current. Development history and completed migrations are in +`docs/developer/ai-notes/historical-notes.md`. -### Key Files -- `src/underworld3/mpi.py` - Parallel safety implementation -- `src/underworld3/scaling/` - Units system -- `src/underworld3/utilities/mathematical_mixin.py` - Mathematical objects -- `src/underworld3/function/expressions.py` - UWexpression (lazy evaluation, symbol disambiguation) -- `src/underworld3/function/_function.pyx` - UnderworldFunction (mesh variable symbols) -- `src/underworld3/discretisation/enhanced_variables.py` - EnhancedMeshVariable (units, math ops, persistence) - -### Historical Notes -For development history, completed migrations, and fixed bugs: -See `docs/developer/ai-notes/historical-notes.md` - ---- +Key implementation files: -*Reorganized 2025-12-13: Historical content moved to docs/developer/ai-notes/historical-notes.md* +| | | +|---|---| +| `src/underworld3/mpi.py` | parallel-safe output | +| `src/underworld3/scaling/` | units system | +| `src/underworld3/function/expressions.py` | UWexpression, lazy evaluation | +| `src/underworld3/function/_function.pyx` | mesh-variable symbols | +| `src/underworld3/utilities/mathematical_mixin.py` | mathematical objects | +| `src/underworld3/utilities/rotated_bc.py` | rotated free-slip | +| `src/underworld3/discretisation/enhanced_variables.py` | units, math ops, persistence | diff --git a/docs/developer/guides/development-setup.md b/docs/developer/guides/development-setup.md index acb18c9b..cbef13f3 100644 --- a/docs/developer/guides/development-setup.md +++ b/docs/developer/guides/development-setup.md @@ -2,43 +2,92 @@ title: "Development Environment Setup" --- -# Setting Up Underworld3 Development Environment - -```{note} Coming Soon -This section will cover: -- Pixi environment setup and management -- Required dependencies and versions -- IDE configuration and tools -- Building from source -- Docker development containers - -**Priority**: High - needed for new developer onboarding -**Current Status**: Placeholder - needs development +# Setting up an Underworld3 development environment + +Underworld3 builds Cython extensions against a specific PETSc, so the build is +not as forgiving as a pure-Python package. The constraints below are the ones +that cost real time when they are broken. + +## Everyday commands + +```bash +./uw build # rebuild after ANY source change +./uw test # run the test suite +pixi run -e default python # a Python in the environment ``` -## Quick Commands +## Rebuild after every source change -For now, the essential commands are: +Underworld3 is *installed* into the pixi environment, not imported from `src/`. +Editing a file under `src/underworld3/` changes nothing until you rebuild: ```bash -# Build Underworld3 -pixi run underworld-build +./uw build +``` + +Check what you are actually running with `uw.__file__` — it must point into +`.pixi/envs//lib/python3.12/site-packages/underworld3/`, never `src/`. -# Run tests -pixi run underworld-test +`./uw build` passes `--no-cache-dir`, because the version is always `0.0.0` and +pip will otherwise reuse a stale wheel. If you still suspect stale code, clear +the intermediate build tree as well: -# Python environment -pixi run -e default python script.py +```bash +rm -rf build/lib.* build/bdist.* +./uw build ``` -## Environment Information +A `.pyx` change that appears not to take effect is almost always this: see also +[the JIT cache notes](../subsystems/jit-cache.md). -From CLAUDE.md context: -- Uses pixi for environment management -- Key dependencies: underworld3, petsc4py, numpy, scipy -- Must use `pixi run underworld-build` after ANY source code changes -- No in-place builds due to Cython/PETSc complexity +## Never use an editable install ---- +`pip install -e .` is prohibited. There are no exceptions, and the reason is +that the damage outlives the install: + +- the `.pth` files it writes **contaminate every pixi environment** sharing the + source directory, and worktrees share environments by symlink; +- they **persist after uninstall**, silently redirecting imports back to `src/` + even once a proper `./uw build` has run; +- `.so` files left in the source tree get loaded by an environment expecting a + different PETSc arch, which surfaces as a `dlopen` symbol error rather than + as anything that names the real cause. + +Always `./uw build`, which runs a non-editable `pip install .`. Where `./uw` is +unavailable: + +```bash +pixi run -e pip install . --no-build-isolation --no-cache-dir +``` + +### Recovering from editable-install contamination + +```bash +find .pixi/envs -name "__editable__*underworld*" -delete # stale .pth, all envs +find src/underworld3 -name "*.so" -delete # .so belong in site-packages +rm -rf build/ +./uw build +``` + +## PETSc is not relocatable + +``` +/Users/lmoresi/+Underworld/underworld-pixi-2/petsc/ +``` + +**Do not move this directory.** PETSc hardcodes paths at configure time, so +moving it breaks petsc4py and every pixi task that depends on it, and the only +repair is a full rebuild of roughly an hour. Worktrees share this one PETSc by +symlink for exactly that reason; everything else in a worktree environment is +its own. + +## Worktrees have their own environments + +Each worktree carries its own pixi environment — its own site-packages and its +own compiled extensions — with only PETSc shared. `./uw build` installs the +source of whichever worktree you run it from, into that worktree's environment. +So: enter the worktree first, then build, then run. Building from the main +checkout does nothing for a worktree. -*This document will be expanded to include complete setup instructions.* \ No newline at end of file +See [Branching and Release Strategy](branching-strategy.md) for the worktree +lifecycle and the branch policy.