From 3546fe94222be993b60ae4e6793b8cab3157b00c Mon Sep 17 00:00:00 2001 From: "rearden-grok[bot]" <317016512+rearden-grok[bot]@users.noreply.github.com> Date: Mon, 21 Sep 2026 15:37:50 -0700 Subject: [PATCH] docs: keep one current contract for agents A stronger model preloads every linked file and reconciles every paraphrase. Put the planning cycle first, keep ship commands in the skill, and describe schema bumps from the live version. Unscheduled plans say they are not the task. --- .agents/skills/ship-pr/SKILL.md | 119 ++++--------- AGENTS.md | 55 +++--- CONTRIBUTING.md | 16 +- SCHEMA.md | 55 +++--- SCHEMA_HISTORY.md | 22 +-- crates/rbitcoin-consensus/AGENTS.md | 2 + crates/rbitcoin-esplora/AGENTS.md | 2 + crates/rbitcoin-net/AGENTS.md | 2 + crates/rbitcoin-query/AGENTS.md | 2 + crates/rbitcoin-rpc/AGENTS.md | 2 + crates/rbitcoin-store/AGENTS.md | 2 + docs/ORIENT.md | 24 ++- docs/README.md | 6 +- docs/errata.md | 17 ++ docs/external_findings/README.md | 4 +- docs/how-we-plan.md | 265 +++++++++++++++------------- docs/invariants.md | 11 +- docs/io-modality.md | 29 ++- docs/peer-clients.md | 5 +- docs/personal-node-plans/README.md | 7 +- docs/quality.md | 20 ++- docs/road-to-1.0.md | 4 +- 22 files changed, 336 insertions(+), 335 deletions(-) diff --git a/.agents/skills/ship-pr/SKILL.md b/.agents/skills/ship-pr/SKILL.md index 346cb27a83..c1dd0860f3 100644 --- a/.agents/skills/ship-pr/SKILL.md +++ b/.agents/skills/ship-pr/SKILL.md @@ -12,84 +12,46 @@ One session worktree, one topic branch per plan, many small commits (each already passed local CI except coverage), required checks green. A plan is not done until those checks are green. Do not merge unless asked. -Owner facts for tests and coverage stay in [`TESTING.md`](../../../TESTING.md). -Do not copy them here. +When to run which command: +[`docs/how-we-plan.md`](../../../docs/how-we-plan.md#agent-contract). +Suite and budgets: [`TESTING.md`](../../../TESTING.md). +Do not copy those facts here. ## Session worktree -Reuse one worktree per session. Do not add a worktree per PR. - -`rearden-grok[bot]` on this operator VM: silo, editor tree, ENOSPC, identity, -and HTTPS push are [`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). If -you are not that identity, ignore that file; use `$PWD/target/dev`. - -```bash -# once per session -git fetch origin -git worktree add /tmp/rbtc- origin/master -cd /tmp/rbtc- -git switch -c / -export CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-$PWD/target/dev}" - -# next PR in the same session (do not add another worktree) -git fetch origin -git switch -C / origin/master -``` - -| Rule | Detail | -|------|--------| -| Base | Current `origin/master` (or `main`) | -| Branch | Topic name. Never commit the plan onto `master` | -| Worktree | One `/tmp/rbtc-` for the session | -| `CARGO_TARGET_DIR` | `$PWD/target/dev` unless already set. `rearden-grok[bot]`: [`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md) | -| Remotes | Worktrees share `origin`. Never `git remote set-url origin` | - -### After merge +One `/tmp/rbtc-` for the session. The next PR is `git switch -C` +from current `origin/master`, not another worktree. Never commit the plan +onto `master`. Worktrees share `origin`. Never `git remote set-url origin`. -Delete the topic branch only. Keep the session worktree until the session ends. -Do not delete a branch that still has an open PR. Do not `git push --delete master`. +`rearden-grok[bot]`: silo, identity, HTTPS `HEAD:` push, and session +end are [`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). If you are not +that identity, ignore that file. Use `$PWD/target/dev` and push the topic +branch with ordinary `git push`. -```bash -git fetch origin --prune -git switch -C / origin/master -git branch -d / -# rearden-grok[bot]: delete the topic branch over HTTPS (rearden-vm-HOST.md) -``` - -```bash -# session end -git worktree remove /tmp/rbtc- -git fetch origin --prune -``` - -`rearden-grok[bot]` session-end extras (test dirs, shared silo): -[`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). Other identities: -ignore that file. +After merge, delete the topic branch only. Keep the session worktree until +the session ends. Do not delete a branch that still has an open PR. Do not +`git push --delete master`. ## Local tests From `nix-shell` (CI pins rustc 1.95.0). `CARGO_TARGET_DIR` as in Session worktree. Same **commands** as the required jobs, not the GitHub Actions -`env:` -(leave `CARGO_INCREMENTAL` unset; that is [`ci.yml`](../../../.github/workflows/ci.yml) only). -When to run which command: -[`docs/how-we-plan.md`](../../../docs/how-we-plan.md) (The cycle). -Suite and budgets: [`TESTING.md`](../../../TESTING.md). +`env:` (leave `CARGO_INCREMENTAL` unset; that is +[`ci.yml`](../../../.github/workflows/ci.yml) only). | When | Run | |------|-----| | Inner loop (Red / Green) | Targeted `cargo test -p … -- --quiet`. `cargo check -p --lib`. Not `--tests` after every edit | | After Green | `cargo test --workspace --quiet` | | After Refactor, before commit | Local CI except coverage (below) | +| Docs, comments, formatting only | Skip the suite. `cargo fmt` if rustfmt would touch the tree | | Core functional CI red | [core-functional skill](../core-functional/SKILL.md) | | Overlay functional CI red | [overlay-functional skill](../overlay-functional/SKILL.md) | | Never local | `./scripts/coverage.sh`, `nix build .#rbitcoin-musl`, host IBD | -Redirect every command above (stdout and stderr to `/tmp`, print `EXIT`, -search then tail on failure). Do not `tee` or `| tail` as the only sink, -and do not load the log wholesale. Recipe: -[`docs/how-we-plan.md`](../../../docs/how-we-plan.md) (Agent RAM). Do not -copy that snippet here. +Redirect every cargo, clippy, deny, and rustc command (stdout and stderr +to `/tmp`, print `EXIT`, search then tail on failure). Recipe: +[`docs/how-we-plan.md`](../../../docs/how-we-plan.md#agent-ram). Local CI except coverage: @@ -105,12 +67,10 @@ cargo test --workspace --quiet If the slice changed `flake.nix`, `nix/`, or the NixOS module, also `nix build .#checks.x86_64-linux.nixos-module-eval --no-link`. Label the PR `nixos-module-runtime` when systemd deps, users/groups, firewall, or the VM -start argv changed — that runs `nixos-module-runtime` (qemu). Do not wait -out the VM test locally. +start argv changed. Do not wait out the VM test locally. Coverage and native `windows` / `macos` stay GitHub Actions; -`ci-os-smoke.sh` is the local stand-in. Do not wait out a host IBD or a -coverage run in the agent VM. +`ci-os-smoke.sh` is the local stand-in. ## Push and poll @@ -128,9 +88,7 @@ Label `nixos-module-runtime` when the NixOS module VM test should run (not eval). Poll with `--interest nixos-module-runtime`. It is not a required check. -`rearden-grok[bot]` fetch/push (`gh-login.sh`, HTTPS `HEAD:`, never -`git push origin`, no `-u`): [`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). -If you are not that identity, ignore that file. +Bot fetch and push: [`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). ```bash gh pr create --repo reardencode/rbitcoin --head / --title "…" --body "…" @@ -146,11 +104,10 @@ operator (`security_events` is read). An alert that only fires in `#[cfg(test)]` or a test module: stop. Do not rename tests or shuffle literals. Ask the operator to dismiss. Production CodeQL is a real finding — fix it. -**Fail-fast poll.** Do not use `gh pr checks --watch` as the only waiter. It -blocks until every check finishes, including slow `coverage` and CodeQL, even -after a job this change exercises is already red. `./scripts/pr-checks-watch.sh` -exits 1 as soon as any required job fails, and as soon as a `--interest` job -fails. On exit 1: stop polling, fetch that job’s log, and start the fix. +**Fail-fast poll.** `./scripts/pr-checks-watch.sh` exits 1 as soon as any +required job fails, and as soon as a `--interest` job fails. On exit 1: +stop polling, fetch that job’s log, and start the fix. `gh pr checks --watch` +waits out `coverage` and CodeQL after a job this change already failed. ```bash gh api --allow-escape-sequences repos/reardencode/rbitcoin/actions/jobs//logs @@ -159,14 +116,10 @@ gh run view --repo reardencode/rbitcoin --job --log-failed **Unmergeable means tests do not run.** If `mergeable` is `CONFLICTING` or `mergeStateStatus` is `DIRTY` (or `BEHIND` when the branch is not on current -`origin/master`), required `test` CI does not start. Rebase, lease-force the -topic branch, then poll. Do not `gh run rerun` to wake jobs that never queued. - -```bash -git fetch origin -git rebase origin/master -# rearden-grok[bot]: lease-force over HTTPS (rearden-vm-HOST.md) -``` +`origin/master`), required `test` CI does not start. Rebase onto +`origin/master`, lease-force the topic branch, then poll. Do not `gh run rerun` +to wake jobs that never queued. Bot lease-force: +[`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). | Rule | Detail | |------|--------| @@ -185,16 +138,16 @@ When required checks are green locally and CI only needs a re-run: |----|--------| | `gh run rerun [--failed]` | Empty commit whose only purpose is to wake Actions | | GitHub Actions re-run failed or all jobs | Noise commits (`ci: bump`, `trigger`) | -| Amend the tip and `--force-with-lease` the topic branch over HTTPS | Force-push `master` / `main` | +| Amend the tip and `--force-with-lease` the topic branch | Force-push `master` / `main` | ```bash gh run rerun --failed - -# If the API rejects: amend the tip (no empty commit) and lease-force the topic branch only -git commit --amend --no-edit -# rearden-grok[bot]: lease-force over HTTPS (rearden-vm-HOST.md) ``` +If the API rejects, amend the tip (no empty commit) and lease-force the +topic branch only. Bot lease-force is +[`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). + If CI `coverage` fails, add a pin and push. Floor: [`TESTING.md`](../../../TESTING.md). Operator musl binaries are [`docs/releases.md`](../../../docs/releases.md). diff --git a/AGENTS.md b/AGENTS.md index e6f7f40074..c08ca012c5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,9 +1,10 @@ # Agent notes Documentation map (one owner per fact): [`docs/README.md`](docs/README.md). -Before exploring, read [`docs/ORIENT.md`](docs/ORIENT.md). -This file is the harness-injected hard-rule contract. Design lives in the -owner docs; do not grow a second design book here. +When the task area is unclear, [`docs/ORIENT.md`](docs/ORIENT.md) routes. +Open the one row that matches the change. This file is the harness-injected +hard-rule contract. Design lives in the owner docs; do not grow a second +design book here. Process playbooks are Agent Skills under [`.agents/skills/`](.agents/skills/). Read the matching `SKILL.md` when the task needs it. Do not paste bash, @@ -15,22 +16,27 @@ If you are acting as `rearden-grok[bot]`, also read [`rearden-vm-HOST.md`](rearden-vm-HOST.md). If you are not that identity, ignore that file. -## Language, comments, composition +## Language and shape Write clear, concrete technical English. Keep Core-aligned terms where we match Bitcoin Core. Do not inject moralizing or political framing, or soften consensus and security language. If a rename is not clearer engineering, keep the existing term. -Comments that restate what, why, or weird are a smell. Prefer names, types, -and structure. Keep `//` only for an invariant, protocol rule, `SAFETY`, or -library quirk. Crate and public rustdoc (`//!` / `///`) is not this rule. -Full text: [`CONTRIBUTING.md`](CONTRIBUTING.md) principle 7. - -Prefer composition (has-a) over inheritance; avoid tall trees. Build -immutable structures once, then compose them. If a map needs extra fields, -wrap it on read rather than mutating members in place. -Control flow: principle 10 and [`docs/code-shape.md`](docs/code-shape.md). +Open [`CONTRIBUTING.md`](CONTRIBUTING.md) principles 7–11 and +[`docs/code-shape.md`](docs/code-shape.md) when editing Rust. Short form: + +- `//` only for an invariant, protocol rule, `SAFETY`, or library quirk. + Crate and public rustdoc (`//!` / `///`) is not this rule (principle 7). +- Tests assert shipped behavior, not repo text. No `*_for_test` backdoors. + IO pins use session or table stats or on-disk state (principle 8). +- A RAM or CPU trade is named (principle 9). +- Control flow and composition: principle 10 and `docs/code-shape.md`. +- Crate `pub` is the cross-crate graph only. No unused `pub`. No + `dead_code` silence. `#[cfg(test)]` on production items is a smell, + including fuzz-only exports (principle 11). +- One logical change per commit. The message says what and why. Not WIP, + misc, or a drive-by rename mixed with behavior. ## Store and IBD @@ -65,8 +71,8 @@ Perf A/B is operator-host only. One plan step is one Red → Green → Refactor turn, committed before the next step. Keep `--lib` compiling (wrap the old API, switch one caller). Inner loop is `cargo test -p --lib` / `cargo check -p --lib`. -Do not `cargo check --tests` after every edit. Owner: -[`docs/how-we-plan.md`](docs/how-we-plan.md). Commands: +Do not `cargo check --tests` after every edit. Cycle and Agent RAM: +[`docs/how-we-plan.md`](docs/how-we-plan.md#agent-contract). Commands: [`.agents/skills/ship-pr/SKILL.md`](.agents/skills/ship-pr/SKILL.md). **Agent RAM:** never load `cargo test`, clippy, deny, or rustc stdout into @@ -81,24 +87,11 @@ identities: ignore that file; use `$PWD/target/dev`. One production implementation at the lowest crate that owns the concept. Extract is a move: [`docs/code-shape.md`](docs/code-shape.md). Core-facing -RPC / P2P / Electrum / Esplora: [`COMPAT.md`](COMPAT.md). - -Tests assert shipped behavior, not repo text -([`CONTRIBUTING.md`](CONTRIBUTING.md) principle 8). Budgets, no `*_for_test` -backdoors, no production-scale default fixtures: [`TESTING.md`](TESTING.md). -Tests use session or table instance stats or on-disk state, not thread-local -hot-path IO probes. - -Crate `pub` is the cross-crate graph only. Unused `pub` is forbidden. -`#[cfg(test)]` on production items is a smell, including fuzz-only exports. -Do not leave dead code or silence `dead_code`. A RAM or CPU trade is named -([`CONTRIBUTING.md`](CONTRIBUTING.md) principle 9). - -One logical change per commit. The message says what and why. Not WIP, misc, -or a drive-by rename mixed with behavior. +RPC / P2P / Electrum / Esplora: [`COMPAT.md`](COMPAT.md). Budgets and +fixtures: [`TESTING.md`](TESTING.md). Before the first edit under `crates//`, read `crates//AGENTS.md` -when it exists. +when it exists, and open the read-first row that matches the change. ## Ship, release, Core functional diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fe108a3d60..2913a65436 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -246,16 +246,12 @@ digging into a monolithic job log. Label **`nixos-module-runtime`** to run the NixOS module qemu test (not eval). That job is not required. -**Agents** implement in **one git worktree per session**, a topic branch per -PR, and **one PR** per plan. Each plan step follows -[`docs/how-we-plan.md`](./docs/how-we-plan.md) and is committed only after -local CI except coverage. Coverage and native `windows` / `macos` stay -GitHub Actions. After merge they delete the topic branch and keep the -session worktree. Never `git remote set-url origin`. Bot push uses an -explicit HTTPS URL (not `git push origin`). Fail-fast poll is -`./scripts/pr-checks-watch.sh`, not `gh pr checks --watch`. Start at -[`docs/ORIENT.md`](./docs/ORIENT.md). -Ship commands: [`.agents/skills/ship-pr/SKILL.md`](./.agents/skills/ship-pr/SKILL.md). +**Agents** open [`docs/ORIENT.md`](./docs/ORIENT.md) when the area is +unclear, follow the agent contract in +[`docs/how-we-plan.md`](./docs/how-we-plan.md#agent-contract), and use +[`.agents/skills/ship-pr/SKILL.md`](./.agents/skills/ship-pr/SKILL.md) to +open or update a pull request. Humans use ordinary git. The bot host file +does not apply to them. Humans who want the same gates offline (Nix optional; rustup 1.95 is enough): diff --git a/SCHEMA.md b/SCHEMA.md index 2821db81a4..0715eb48ce 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -86,6 +86,14 @@ Older versions and migration notes live in [`SCHEMA_HISTORY.md`](./SCHEMA_HISTOR ## Schema 17 freeze +Class A shape settled here at schema 17 and is still the live layout under +the `SCHEMA_VERSION` at the top of this file (24). Schemas 18–24 already +shipped; they are open rules above and history in +[`SCHEMA_HISTORY.md`](./SCHEMA_HISTORY.md). A byte-incompatible change bumps +from that live constant +([Changing durable bytes](#changing-durable-bytes)). It does not target +schema 18. + ### What 17 locks (on-disk) | Object | Frozen choice | @@ -113,17 +121,19 @@ catalogs, or 16-layout Class A with creates is **refused**. ### New script kinds without a wipe Kind nibble **10–15** is **Corrupt** on this binary (no implicit width). A new -consensus script type does **not** force a 17 datadir wipe: +consensus script type does **not** force a wipe of the Class A shape settled +at 17: -| Path | On-disk | Old 17 binary | New binary | -|------|---------|---------------|------------| +| Path | On-disk | This binary | A later binary | +|------|---------|-------------|----------------| | **RAW** | kind 0 + CompactSize + bytes | already decodes | same | -| **Soft-18** | new kind nibble + known width; `SCHEMA_VERSION = 18` | **refuses** 18 `meta` (or unknown kind) | reads 17 files; writes 18 | +| **Kind nibble** | new kind + known width; bump `SCHEMA_VERSION` | refuses the new `meta` (or unknown kind) | reads these files; writes the new version | -Use RAW when the type is rare. Use soft-18 when the type is common enough to -pay a width table. Soft-18 is **not** a silent in-place rewrite of 17 files. -Inwit `create_fk` Δfk (parked) is the same class: **18 or an inwit-only -rewrite**, not a silent 17 mutate. +Use RAW when the type is rare. Use a kind-nibble bump when the type is common +enough to pay a width table. That bump is not a silent in-place rewrite of +existing files. Inwit `create_fk` Δfk (parked) is the same class: a +`SCHEMA_VERSION` bump or an inwit-only rewrite, not a silent mutate of +existing rows. ### Field widths (10 years) @@ -140,19 +150,20 @@ Assume ~400k–700k creates/day. Ten years ≈ +1.5e9…2.6e9 creates on top of | `tx.head` bits | 25-bit segments | Roll + seal; no mono-file widen | | SH megakey page | 4 KiB delta stream | Page chain; not a single-integer cap | | `sp_tweaks` off | u32 per segment | Already segmented | -| Script kind | 4 bits | 0–9 used; 10–15 reserved Corrupt; extension = RAW or soft-18 | +| Script kind | 4 bits | 0–9 used; 10–15 reserved Corrupt; extension = RAW or a kind-nibble bump | Practical risks are **loc overflow without ovf** (sentinel with missing sidecar) and **Bitcoin timestamp 2106** (consensus, every node). -### What would force schema 18 +### What forces a schema bump A **byte-incompatible** change to Class A / OA / body / idx / SH catalog -layout, or anything that cannot soft-open 17 files. +layout, or anything that cannot soft-open the live files. Bump from the +live `SCHEMA_VERSION`, not from 17. -| Change | 18? | Notes | -|--------|-----|-------| -| New implicit-width script kind | Optional | RAW = no bump; nibble = soft-18 (17 refuses 18) | +| Change | Bump? | Notes | +|--------|-------|-------| +| New implicit-width script kind | Optional | RAW = no bump; nibble = next `SCHEMA_VERSION` (this binary refuses it) | | Inwit Δfk | Yes or inwit-only rewrite | Parked; cold stem | | Idx not stride-8 / not u32 | Yes | Would retire the 8-align pad | | Packed Class A again / merge stems | Yes | Wipe | @@ -161,10 +172,11 @@ layout, or anything that cannot soft-open 17 files. | Fuse8 envelope v3 | No | Soft-migrate like v1→v2 (log + rewrite; no wipe) | | Independent rolls / L2 strong / no DONTCACHE | No | Writer/RAM only | -Parked size work that is **not** 17: inwit Δfk; drop 8-align pad on empty -inwit / zero-out spent (needs a different idx encoding); `txid.body` -compression. Do not chase `inwit` size as an IBD **hot-set** win — put it -on a cold volume (`--datadir-cold`). Census: [Mainnet census](#mainnet-census-this-trees-reference-datadir-2026-08-13). +Parked size work that is **not** a live-layout tweak: inwit Δfk; drop 8-align +pad on empty inwit / zero-out spent (needs a different idx encoding); +`txid.body` compression. Do not chase `inwit` size as an IBD **hot-set** +win — put it on a cold volume (`--datadir-cold`). Census: +[Mainnet census](#mainnet-census-this-trees-reference-datadir-2026-08-13). Process: bump `SCHEMA_VERSION`, document this file + `SCHEMA_HISTORY.md` in the same commit, refuse or soft-open with a one-line operator message. Do @@ -458,9 +470,10 @@ one code to “extension follows” and still read `e=0..=9` records). Decode expands templates to wire scripts (P2TR is `5120||32`). XOR at rest covers hash/data only. Spender flags live only on `spent`. A new consensus -script type does **not** wipe a 17 datadir: encode it as kind 0 **RAW**, or -introduce an implicit-width nibble as **soft-18** (18 reads 17; this 17 -binary refuses 18 / unknown kind). See [Schema 17 freeze](#schema-17-freeze). +script type does **not** wipe the Class A shape settled at 17: encode it as +kind 0 **RAW**, or introduce an implicit-width nibble as a +`SCHEMA_VERSION` bump (the new binary reads these files; this binary refuses +the new `meta` or an unknown kind). See [Schema 17 freeze](#schema-17-freeze). ### Sole-spender slot (`spent.body`) diff --git a/SCHEMA_HISTORY.md b/SCHEMA_HISTORY.md index 5d87a5b5b2..2159d99911 100644 --- a/SCHEMA_HISTORY.md +++ b/SCHEMA_HISTORY.md @@ -130,14 +130,15 @@ Closed layout. SH run catalogs are unique `(scripthash, create_fk)` at `key_len=40`. Schema-16 `key_len=32` catalogs are refused. Hot Class A: thin LAYOUT17 `txout` meta, script kinds **0–9** (unknown -kind is Corrupt). A new consensus script is **RAW** (no bump) or -**soft-18** (18 reads 17; 17 refuses 18) — not a 17 datadir wipe. 8 B -spent slots, `spent.ovf` overflow. Inwit flags bits 4–7 and spent flags -other than `MULTI_SPENDER` are Corrupt. Inwit prevout is still -`create_fk:u64` + CompactSize vout (Δfk is an **18 / inwit-only** -follow-up). Writer/RAM on 17 (not on-disk): idx stems roll independently; -`strong_tx` always L2; no `RWF_DONTCACHE`. See -[`SCHEMA.md`](SCHEMA.md) (Schema 17 freeze). +kind is Corrupt). At schema 17, a new consensus script was **RAW** (no bump) +or a kind-nibble bump the then-current binary would refuse — not a 17 +datadir wipe. 8 B spent slots, `spent.ovf` overflow. Inwit flags bits 4–7 +and spent flags other than `MULTI_SPENDER` are Corrupt. Inwit prevout is +still `create_fk:u64` + CompactSize vout (Δfk was parked as an inwit-only +or later-version follow-up). Writer/RAM on 17 (not on-disk): idx stems roll +independently; `strong_tx` always L2; no `RWF_DONTCACHE`. See +[`SCHEMA.md`](SCHEMA.md) (Schema 17 freeze). The live bump rule is that +file, not this section. Megakey SH pages store ULEB128 fk0+deltas (`ver=1`). Leftover raw-u64 pages (`ver=0`, `n>0`) rematerialize. @@ -145,8 +146,9 @@ pages (`ver=0`, `n>0`) rematerialize. **Body orientation (still 17):** new creates write `scripthash.body/NN` + `scripthash.ovf/body`. Existing file `scripthash.body` stays shared (one writer). Mixed file+dir or a dir without `ovf/body` is Layout -refuse (wipe `store/scripthash*`). Not schema 18 — 18 remains the next -incompatible Class A / idx change. +refuse (wipe `store/scripthash*`). That body-dir change was not schema 18. +At the time, 18 was the next incompatible Class A / idx change; 18–24 have +since shipped. `archive_epoch` is gone. Create does not plant `wire/` or packed `tx.body`. Open unlinks leftover `archive_epoch`, `store/wire`, and diff --git a/crates/rbitcoin-consensus/AGENTS.md b/crates/rbitcoin-consensus/AGENTS.md index 3aed83f395..f7e7e6e945 100644 --- a/crates/rbitcoin-consensus/AGENTS.md +++ b/crates/rbitcoin-consensus/AGENTS.md @@ -6,6 +6,8 @@ Do not run the interpreter on an I/O thread. ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Rules we own vs Core corpora | [`docs/consensus-tests.md`](../../docs/consensus-tests.md) | diff --git a/crates/rbitcoin-esplora/AGENTS.md b/crates/rbitcoin-esplora/AGENTS.md index dc22078289..d6bcddd09d 100644 --- a/crates/rbitcoin-esplora/AGENTS.md +++ b/crates/rbitcoin-esplora/AGENTS.md @@ -6,6 +6,8 @@ need `--sh-index`. ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Shipped HTTP / WS surface | [`COMPAT.md`](../../COMPAT.md) | diff --git a/crates/rbitcoin-net/AGENTS.md b/crates/rbitcoin-net/AGENTS.md index be987182cb..a728f20ac8 100644 --- a/crates/rbitcoin-net/AGENTS.md +++ b/crates/rbitcoin-net/AGENTS.md @@ -5,6 +5,8 @@ P2P and IBD. Depends on query, store, consensus, and mempool. ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Roles, body queue, publish order | [`docs/concurrency.md`](../../docs/concurrency.md) | diff --git a/crates/rbitcoin-query/AGENTS.md b/crates/rbitcoin-query/AGENTS.md index 80c6c0d3fc..8e65f8756d 100644 --- a/crates/rbitcoin-query/AGENTS.md +++ b/crates/rbitcoin-query/AGENTS.md @@ -6,6 +6,8 @@ a query API already owns the fact. ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Stage IO, pin identity, leftover | [`docs/invariants.md`](../../docs/invariants.md) | diff --git a/crates/rbitcoin-rpc/AGENTS.md b/crates/rbitcoin-rpc/AGENTS.md index f28603bb1a..eab4fb32be 100644 --- a/crates/rbitcoin-rpc/AGENTS.md +++ b/crates/rbitcoin-rpc/AGENTS.md @@ -6,6 +6,8 @@ net, and consensus. The HTTP server is `src/server.rs`; methods are ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Method list, auth, permanent gaps | [`docs/rpc.md`](../../docs/rpc.md) | diff --git a/crates/rbitcoin-store/AGENTS.md b/crates/rbitcoin-store/AGENTS.md index 4c3ada705c..57755bfa94 100644 --- a/crates/rbitcoin-store/AGENTS.md +++ b/crates/rbitcoin-store/AGENTS.md @@ -6,6 +6,8 @@ and `rbitcoin-log`. `rbitcoin-query` and everything above it call this crate. ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Stage IO, leftover union, no silent fallback | [`docs/invariants.md`](../../docs/invariants.md) | diff --git a/docs/ORIENT.md b/docs/ORIENT.md index 9f32a33aa0..ac5e7d055d 100644 --- a/docs/ORIENT.md +++ b/docs/ORIENT.md @@ -1,7 +1,8 @@ # Agent orientation -Read this before a broad search when the task area is unclear. +Open this when the task area is unclear. Facts stay in the owner files in [`README.md`](./README.md). This page only routes. +Open the one row that matches the change. Leave the other owners closed. ## Crate graph @@ -30,8 +31,8 @@ Before the first edit in a crate, read `crates//AGENTS.md` when that file | Which head file (tx / header / scripthash) | [`heads.md`](./heads.md) | | Crash, tip-as-commit, kill-9 | [`crash-recovery.md`](./crash-recovery.md) | | Tests, budgets, coverage, fixtures | [`../TESTING.md`](../TESTING.md) | -| cargo-mutants (weekly sweep, PR advisory) | [`../TESTING.md`](../TESTING.md) (Mutation testing); snapshot [`mutants/`](./mutants/) | -| Multi-step plan (Red → Green → Refactor) | [`how-we-plan.md`](./how-we-plan.md) | +| A missed cargo-mutants case | [`../TESTING.md`](../TESTING.md) (Mutation testing). Open [`mutants/`](./mutants/) only for that miss | +| Multi-step plan (Red → Green → Refactor) | [`how-we-plan.md`](./how-we-plan.md#agent-contract) (stop before Rationale) | | cargo / clippy / deny / rustc logs (do not load into the session) | [`how-we-plan.md`](./how-we-plan.md) (Agent RAM) | | `rearden-grok[bot]` operator VM (ignore unless that identity) | [`../rearden-vm-HOST.md`](../rearden-vm-HOST.md) | | Intentional Core / Electrum / Esplora differences | [`../COMPAT.md`](../COMPAT.md) | @@ -43,9 +44,18 @@ Before the first edit in a crate, read `crates//AGENTS.md` when that file ## Ask first -Hard stops are the Store and IBD bullets and the Ship landmines -(fail-fast poll) in [`../AGENTS.md`](../AGENTS.md). +Hard stops in [`../AGENTS.md`](../AGENTS.md) are proceed-or-stop rules. `rearden-grok[bot]` also follows [`../rearden-vm-HOST.md`](../rearden-vm-HOST.md); other identities ignore that file. -Also ask before expanding who is trusted beyond the boundary in -[`../SECURITY.md`](../SECURITY.md). + +Ask before any of these, including when the change looks locally justified: + +- Replace a purpose-built IO machine with batched `pread` / `pwrite`. +- Bump or wipe the schema, or add an `RBITCOIN_*` knob. +- Widen who is trusted beyond [`../SECURITY.md`](../SECURITY.md). +- Merge, force-push `master`, or `cargo clean` the shared cargo silo. +- Start an item in [`quality.md`](./quality.md), [`peer-clients.md`](./peer-clients.md), + [`personal-node-plans/`](./personal-node-plans/), or [`road-to-1.0.md`](./road-to-1.0.md) + that the user did not name. +- Open a mainnet datadir, or treat an agent-VM run as a perf result. +- Label an ordinary net or RPC pull request `core-functional`. diff --git a/docs/README.md b/docs/README.md index 07ef2599d6..46d5e81755 100644 --- a/docs/README.md +++ b/docs/README.md @@ -33,7 +33,7 @@ into `AGENTS.md`. | Doc | Owns | |-----|------| -| [`ORIENT.md`](./ORIENT.md) | Agent task router: crate graph and read-first links. Not a second index of facts. | +| [`ORIENT.md`](./ORIENT.md) | Agent task router: crate graph and read-first links. Open one matching row. Not a second index of facts. | | [`architecture.md`](./architecture.md) | Why this node is different (Core / Fulcrum contrasts). No stage-IO table copy. | | [`concurrency.md`](./concurrency.md) | Writer roles, publish order, body-queue, pins. Links invariants for leftover union. | | [`invariants.md`](./invariants.md) | Confirm stage IO (the **only** copy), leftover union, store start states S0–S4, no silent fallbacks. | @@ -47,7 +47,7 @@ into `AGENTS.md`. | [`consensus-tests.md`](./consensus-tests.md) | Rules we own vs Core corpora. | | [`core-functional.md`](./core-functional.md) | Core v31.1 functional harness. | | [`overlay-functional.md`](./overlay-functional.md) | Private Tor / i2pd / cjdns mesh harness (labeled / nightly). | -| [`how-we-plan.md`](./how-we-plan.md) | Red → Green → Refactor planning contract; Agent RAM (redirect cargo/clippy/rustc stdout). | +| [`how-we-plan.md`](./how-we-plan.md) | Agent contract (cycle, Agent RAM, keep-compiling), then human rationale. | | [`personal-node-plans/`](./personal-node-plans/) | Home-node implementation plan group (SOCKS, overlays, wallet onions, ephemeral broadcast, inwit prune / `NETWORK_LIMITED`). Live flags stay OPERATOR/COMPAT when a slice ships. Do not copy step lists into quality.md until scheduled. | | [`releases.md`](./releases.md) | Tag `vX.Y.Z`, `vX.Y.x` patch line, `.99` bump, Highlights / GitHub notes. | | [`code-shape.md`](./code-shape.md) | Control flow, types, naming, composition (CONTRIBUTING principle 10). Named extracts: quality.md **Q-61** (Completed). Clippy: no workspace `allow` list; leftover lints are site-local with a reason. | @@ -56,7 +56,7 @@ into `AGENTS.md`. | [`reproducible-builds.md`](./reproducible-builds.md) | Pinned Nix / musl byte-identity. | | [`rust-bitcoin-limitations.md`](./rust-bitcoin-limitations.md) | Workarounds where rust-bitcoin is not Core-faithful. | | [`mempool-fee-estimation.md`](./mempool-fee-estimation.md) | Fee estimator notes. | -| [`errata.md`](./errata.md) | Known one-off store/confirm quirks. | +| [`errata.md`](./errata.md) | Known one-off store/confirm quirks. Retired confirm dual-path names. | | [`peer-clients.md`](./peer-clients.md) | Hornet Node and satd: what to steal (tests/ideas) and what not to copy. Ranked items stay here; do not copy into quality.md. | | [`lightning.md`](./lightning.md) | CLN and LDK as Bitcoin backends: five `bcli` calls, Esplora/Electrum chain sync, `--sh-index` API matrix. Not LND. | | [`mutants/`](./mutants/) | Generated cargo-mutants missed/timeout snapshot. How to run / CI: [`TESTING.md`](../TESTING.md). Not a quality backlog. | diff --git a/docs/errata.md b/docs/errata.md index 8b59356dbc..9a01f9a412 100644 --- a/docs/errata.md +++ b/docs/errata.md @@ -31,3 +31,20 @@ connected; n−1 is held in in-flight until after the child pin. Do not grow these maps to `Vec` unless a mainnet miss is shown to be this case. + +## Retired confirm dual paths + +These names are gone. A missed fact the pipeline promised is still +`StoreError::Corrupt("invariant: …")` ([`invariants.md`](./invariants.md)). +Do not treat the list as a set of identifiers to police: + +- Soft spentness recovery for a wrong or missing pin identity. +- Unpinned wire-corrected `create_fk` spentness. +- Load-stage `txid.body` identity fill after lookup promised the stamp. +- `ColdPinMode` Allow/Forbid cold denserels on load (load is range outs only). +- Denserels-as-spender-abs (schema 22 abs is `spent` loc off + `8×vout` only). +- `AssembleMode::Full` / `validate_block_connect` (confirm is optimistic + assemble, then `structural_validate_spends`). +- `archive_plan_batch_from_store` and production `Query` turning TxApply into + a dummy `Block`. `tx_apply_to_tx`, `connect_block`, and + `commit_class_a_only` are `rbitcoin_query::testutil::FixtureChain` only. diff --git a/docs/external_findings/README.md b/docs/external_findings/README.md index d243a4f6ad..faaaaf8fb5 100644 --- a/docs/external_findings/README.md +++ b/docs/external_findings/README.md @@ -35,7 +35,9 @@ rbitcoin reference, or redteam static analysis). Numbered reports live beside th **Policy:** Core `script_tests` / `tx_valid` / `tx_invalid` corpora must pass **every** data row with **no allowlist**. Do not commit if those tests fail. Findings stay -**fixed** with a named regression on the shipped path. +**fixed** with a named regression on the shipped path. A green regression means +the write-up stays closed. Do not re-derive the bug from the narrative unless +that test fails. **006–009:** consensus accept-invalid (zip 2026-08-10) — **fixed** in-tree. **010–011:** mempool remediation **fixed** (Coin spentness + structural tip checks + fee path). diff --git a/docs/how-we-plan.md b/docs/how-we-plan.md index d07bb9fef1..857a855303 100644 --- a/docs/how-we-plan.md +++ b/docs/how-we-plan.md @@ -5,16 +5,147 @@ Work lands as **small vertical slices**. Each slice is one This file owns that cycle. Hard-rule pointer: [`AGENTS.md`](../AGENTS.md). Commands: [`.agents/skills/ship-pr/SKILL.md`](../.agents/skills/ship-pr/SKILL.md). -Plans have more steps than a typical “phase 1 / phase 2” design. Each step -should leave the production path simpler and the suite a sharper pin. +**Agents** follow [Agent contract](#agent-contract) through +[Keep the tree compiling](#keep-the-tree-compiling). Do not preload +[Rationale](#rationale). + +**Humans** learning the method: stories, examples, and the checklist are +the rationale after the contract. + +## Agent contract + +Core functional scripts are not the in-tree Red test. When the labeled +`core-functional` job is red, reproduce locally +([`core-functional.md`](./core-functional.md)). + +## The cycle: Red → Green → Refactor + +One step is one turn of the loop, **committed before the next step starts**. +Red pins behavior, so Green is safe to be crude. The pin makes Refactor +safe. Refactor leaves the production path simpler and the suite sharper, so +the next Red is easier to name. Skip a phase and the loop degrades: no Red +is coding to logs; no Refactor is one-offs that accrete; no **test** +refactor is a slow suite full of brittle twins. Landing several slices and +then fighting CI is the same skip — the gates never ran under a small diff. + +```text +for each plan step: + 1. Red new test; targeted run; see red. No production edit yet. + 2. Green smallest production change; targeted run; see green. + 3. Suite cargo test --workspace; confirm green. + 4. Refactor fold one-offs; production and tests; still green. + 5. Gates local CI except coverage (commands in ship-pr). + 6. Commit then the next step. + +after the last step (optional): + holistic refactor → local CI except coverage → commit + then push and poll GitHub +``` + +Edits inside Red and Green stay targeted: `cargo test -p …` and +`cargo check -p --lib`. Do not `cargo check --tests` after every +edit. The workspace suite is step 3 (and again in step 5 if Refactor +changed code). + +### Agent RAM + +Never return full `cargo test`, clippy, deny, or rustc stdout through the +Shell tool into the agent session. Workspace `cargo test --workspace` is +the dangerous one (will OOM an agent turn). Targeted +`cargo test -p ` still redirect if output can be large. +Docs-only slices still must not dump rustc if they run fmt. + +Redirect **stdout and stderr** to a file under `/tmp` (`mktemp +/tmp/rbtc-agent-XXXXXX.log` or `/tmp/rbtc-agent-.log`). Do not `tee` +(that still streams into the tool result). Do not `| tail` as the only +sink (the compiler still writes a huge pipe into the tool). + +After the command, print **exit code** in the tool result +(`echo EXIT:$?`). Look at that first. + +- Exit 0 with `--quiet` (tests) or clippy `-D warnings` and no output: + green. Do **not** Read the log file. +- Exit ≠ 0: search the file for failure names (`FAILED`, `error:`, clippy + lint ids, first rustc error). Read at most **~80 lines of tail** (or a + tight `rg` window). Do not `Read` a multi-megabyte log. + +`--quiet` is enough to confirm green. Copy-paste shape (adapt the +`cargo …` line per command: clippy `-D warnings`, `cargo deny check`, +`cargo fmt --all -- --check`, `./scripts/ast-grep.sh`, +`./scripts/ci-os-smoke.sh`). Inner loop: +`cargo test -p --lib -- --quiet` redirected the same way. + +```bash +log=$(mktemp /tmp/rbtc-agent-XXXXXX.log) +set +e +cargo test --workspace --quiet >"$log" 2>&1 +ec=$? +set -e +echo EXIT:$ec +if [ "$ec" -ne 0 ]; then + rg -n 'FAILED|^error:|error\[' "$log" | head -n 40 + echo '--- tail ---' + tail -n 80 "$log" +fi +``` + +A sequence of gates can wrap that body in a `run` function (one call per +command). Do not paste a second copy into ship-pr; that file lists the +commands and links here. + +**Agent disk:** `rearden-grok[bot]` follows +[`rearden-vm-HOST.md`](../rearden-vm-HOST.md). Other identities: ignore +that file. Commands: +[`.agents/skills/ship-pr/SKILL.md`](../.agents/skills/ship-pr/SKILL.md). + +Pure docs, comments, or formatting skip Red, Green, and the workspace suite. +Still run `cargo fmt --all` if rustfmt would touch the tree, and the other +gates if the slice also changed Rust, scripts, or lint. + +If Refactor is empty, do not run the workspace suite twice: step 3 plus +fmt / deny / clippy / ast-grep / `ci-os-smoke.sh` is enough. + +Coverage (`./scripts/coverage.sh`) and a host IBD are never local. Native +`windows` / `macos` still run on GitHub Actions; `./scripts/ci-os-smoke.sh` +is the local stand-in. `nixos-module-eval` only when the slice changed +`flake.nix`, `nix/`, or the NixOS module. The qemu VM test +(`nixos-module-runtime`) is GitHub Actions on label **`nixos-module-runtime`** +and on Release tags — not local, not a required PR check. Push may wait until +several slices are committed; each commit must already have passed those +gates. + +A step is not done because it compiles, because the one-off is still there, +or because “CI at the end will catch it.” + + +### Keep the tree compiling + +Schema/API rewires stall when a session deletes the old type (`TxIdx`, a +body meta field, …) and then spends the rest of the turn on `unresolved` / +`dead_code` across store tests, query, consensus, and net. That is not TDD; +it is a compile-doom loop. `rbitcoin-store --tests` is a fat rustc unit — +do not use it as the edit cycle. + +| Do | Do not | +|----|--------| +| New type + its unit tests green, **then** a thin wrap on the old API | Delete the old module in the same dirty tree as all callers | +| Switch **one** caller crate per step; `--lib` stays green | One uncommitted tree spanning store + confirm + query + net + docs | +| Keep `--lib` compiling throughout; commit after local CI except coverage | Hours of un-gated WIP, or a slice commit that has not passed those gates | +| `cargo check -p --lib` (or `cargo test -p --lib `) | `cargo check -p rbitcoin-store --tests` or six-crate `--tests` after each edit | + +`--tests` / multi-crate check belongs after Green and in Gates, not in the +inner loop. Commands: +[`.agents/skills/ship-pr/SKILL.md`](../.agents/skills/ship-pr/SKILL.md). -Influences: Extreme Programming (stories, planning game, small releases, -TDD, continuous refactoring), INVEST stories, vertical slicing, YAGNI / -simple design. Adapted for a consensus + IBD codebase and agent-driven -execution. --- +## Rationale + +The contract above is the cycle. This part is why the repo plans in +small vertical slices, and how to read a step. It is not a second +procedure. + ## Why plan this way | Problem we had | Planning fix | @@ -181,104 +312,7 @@ even if the slices are “vertical.” --- -## The cycle: Red → Green → Refactor - -One step is one turn of the loop, **committed before the next step starts**. -Red pins behavior, so Green is safe to be crude. The pin makes Refactor -safe. Refactor leaves the production path simpler and the suite sharper, so -the next Red is easier to name. Skip a phase and the loop degrades: no Red -is coding to logs; no Refactor is one-offs that accrete; no **test** -refactor is a slow suite full of brittle twins. Landing several slices and -then fighting CI is the same skip — the gates never ran under a small diff. - -```text -for each plan step: - 1. Red new test; targeted run; see red. No production edit yet. - 2. Green smallest production change; targeted run; see green. - 3. Suite cargo test --workspace; confirm green. - 4. Refactor fold one-offs; production and tests; still green. - 5. Gates local CI except coverage (commands in ship-pr). - 6. Commit then the next step. - -after the last step (optional): - holistic refactor → local CI except coverage → commit - then push and poll GitHub -``` - -Edits inside Red and Green stay targeted: `cargo test -p …` and -`cargo check -p --lib`. Do not `cargo check --tests` after every -edit. The workspace suite is step 3 (and again in step 5 if Refactor -changed code). - -### Agent RAM - -Never return full `cargo test`, clippy, deny, or rustc stdout through the -Shell tool into the agent session. Workspace `cargo test --workspace` is -the dangerous one (will OOM an agent turn). Targeted -`cargo test -p ` still redirect if output can be large. -Docs-only slices still must not dump rustc if they run fmt. - -Redirect **stdout and stderr** to a file under `/tmp` (`mktemp -/tmp/rbtc-agent-XXXXXX.log` or `/tmp/rbtc-agent-.log`). Do not `tee` -(that still streams into the tool result). Do not `| tail` as the only -sink (the compiler still writes a huge pipe into the tool). - -After the command, print **exit code** in the tool result -(`echo EXIT:$?`). Look at that first. - -- Exit 0 with `--quiet` (tests) or clippy `-D warnings` and no output: - green. Do **not** Read the log file. -- Exit ≠ 0: search the file for failure names (`FAILED`, `error:`, clippy - lint ids, first rustc error). Read at most **~80 lines of tail** (or a - tight `rg` window). Do not `Read` a multi-megabyte log. - -`--quiet` is enough to confirm green. Copy-paste shape (adapt the -`cargo …` line per command: clippy `-D warnings`, `cargo deny check`, -`cargo fmt --all -- --check`, `./scripts/ast-grep.sh`, -`./scripts/ci-os-smoke.sh`). Inner loop: -`cargo test -p --lib -- --quiet` redirected the same way. - -```bash -log=$(mktemp /tmp/rbtc-agent-XXXXXX.log) -set +e -cargo test --workspace --quiet >"$log" 2>&1 -ec=$? -set -e -echo EXIT:$ec -if [ "$ec" -ne 0 ]; then - rg -n 'FAILED|^error:|error\[' "$log" | head -n 40 - echo '--- tail ---' - tail -n 80 "$log" -fi -``` - -A sequence of gates can wrap that body in a `run` function (one call per -command). Do not paste a second copy into ship-pr; that file lists the -commands and links here. - -**Agent disk:** `rearden-grok[bot]` follows -[`rearden-vm-HOST.md`](../rearden-vm-HOST.md). Other identities: ignore -that file. Commands: -[`.agents/skills/ship-pr/SKILL.md`](../.agents/skills/ship-pr/SKILL.md). - -Pure docs, comments, or formatting skip Red, Green, and the workspace suite. -Still run `cargo fmt --all` if rustfmt would touch the tree, and the other -gates if the slice also changed Rust, scripts, or lint. - -If Refactor is empty, do not run the workspace suite twice: step 3 plus -fmt / deny / clippy / ast-grep / `ci-os-smoke.sh` is enough. - -Coverage (`./scripts/coverage.sh`) and a host IBD are never local. Native -`windows` / `macos` still run on GitHub Actions; `./scripts/ci-os-smoke.sh` -is the local stand-in. `nixos-module-eval` only when the slice changed -`flake.nix`, `nix/`, or the NixOS module. The qemu VM test -(`nixos-module-runtime`) is GitHub Actions on label **`nixos-module-runtime`** -and on Release tags — not local, not a required PR check. Push may wait until -several slices are committed; each commit must already have passed those -gates. - -A step is not done because it compiles, because the one-off is still there, -or because “CI at the end will catch it.” +## How a step reads ### Red: name the contract @@ -392,30 +426,9 @@ Each step is independently green, gated, committed, and shippable. | Plan ignores test runtime | Explicit unit vs scenario choice per step | | Core functional as the default-CI Red | In-tree catalog journey; Core stays nightly | | Several slices uncommitted; first GitHub run is the fmt/clippy/suite gate | Each slice commits only after those gates | -| Delete a large type/module then `cargo check --tests` until the workspace builds | Keep-compiling facade (below) | +| Delete a large type/module then `cargo check --tests` until the workspace builds | Keep-compiling facade ([agent contract](#keep-the-tree-compiling)) | | Inner loop = `cargo check --tests` after every edit | `--lib` until that crate’s lib is green | -### Keep the tree compiling - -Schema/API rewires stall when a session deletes the old type (`TxIdx`, a -body meta field, …) and then spends the rest of the turn on `unresolved` / -`dead_code` across store tests, query, consensus, and net. That is not TDD; -it is a compile-doom loop. `rbitcoin-store --tests` is a fat rustc unit — -do not use it as the edit cycle. - -| Do | Do not | -|----|--------| -| New type + its unit tests green, **then** a thin wrap on the old API | Delete the old module in the same dirty tree as all callers | -| Switch **one** caller crate per step; `--lib` stays green | One uncommitted tree spanning store + confirm + query + net + docs | -| Keep `--lib` compiling throughout; commit after local CI except coverage | Hours of un-gated WIP, or a slice commit that has not passed those gates | -| `cargo check -p --lib` (or `cargo test -p --lib `) | `cargo check -p rbitcoin-store --tests` or six-crate `--tests` after each edit | - -`--tests` / multi-crate check belongs after Green and in Gates, not in the -inner loop. Commands: -[`.agents/skills/ship-pr/SKILL.md`](../.agents/skills/ship-pr/SKILL.md). - ---- - ## Checklist for authors (and agents) Before accepting a plan: diff --git a/docs/invariants.md b/docs/invariants.md index 06cb79449f..42f8928f2e 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -23,15 +23,8 @@ with `Err(…Corrupt("invariant: …"))` (and `debug_assert!` where useful). Do **One path.** Do not reintroduce a silent colder alternate when lookup or load was supposed to guarantee the fact. Test-only adapters stay in `*_testutil`; do not grow production APIs around fixture shapes. -**Killed dual paths (examples, not a name list to police):** soft spentness recovery for wrong/missing -pin identity; unpinned wire-corrected create_fk spentness; load-stage `txid.body` -identity fill after lookup promised stamp; `ColdPinMode` Allow/Forbid cold denserels -split on load (load is range **outs** only); denserels-as-spender-abs (schema 22 -abs is `spent` loc off + `8×vout` only); `AssembleMode::Full` / `validate_block_connect` -(confirm is optimistic assemble then `structural_validate_spends`); -`archive_plan_batch_from_store` and production `Query` TxApply→dummy `Block` -(`tx_apply_to_tx` / `connect_block` / `commit_class_a_only` are -`rbitcoin_query::testutil::FixtureChain` only). +Retired dual-path names live in [`errata.md`](./errata.md). The rule is the +sentence above. ## Failure style diff --git a/docs/io-modality.md b/docs/io-modality.md index 6e227be006..ca82f2d791 100644 --- a/docs/io-modality.md +++ b/docs/io-modality.md @@ -176,30 +176,21 @@ IOCP. Ring depth **128** (merge may grow). `RBITCOIN_IO=pread` forces libc. --- -## Historical record: head insert uring ~5× slower +## Head insert locality -| Commit | Note | -|--------|------| -| `0ee28c0` / `77cb2ab` | io_uring bulk / page-grouped RMW for `tx.head` insert | -| **`259b766`** (2026-07-23) | **Reverted to mmap-only head insert.** Host A/B: **io_uring head inserts ~5× slower on head ms/blk** than mmap Release. Bulk uring kept for **reads** only | -| `788936e` | Page-coalesce insert still via **plain map** `write_at` | -| `3a0c220` | **Body** FdOnly success (different pattern: linear append + bulk batch read) | -| `f829090` / `11134cb` | Segmented idx/head landed **after** the 5× failure | - -**Implication:** do not ship head demap without **operator host** A/B (musl -static). Prefer page-coalesced **pread→mutate→pwrite** over per-slot uring. -Segmented heads reduce grow/remap pain but do not free us from page locality. +Do not demap heads without an operator-host measurement (musl static). +A host A/B showed io_uring head inserts about 5× slower on head ms/blk than +a mapped insert. Prefer page-coalesced pread → mutate → pwrite over per-slot +uring. Segmented heads do not remove that page-locality constraint. --- -## End goal (phased) - -1. **FdOnly** for multi‑GiB random tables: `create.loc` → `tx.head` / header head → SH head/body / spenders. -2. **InRam** (explicit process buffers) for small Class C / mempool — not leftover MapFull “because small.” -3. **Remove `memmap2`** from the workspace. -4. Update this doc after **each** phase with host A/B results. +## Settled IO shape -Agent correctness tests under `/tmp` are required; **perf ship/fail is host-only**. +Phase 6 is done: the workspace has zero `memmap2` / `MmapMut` (top of this +file). Multi-GiB random tables stay FdOnly. Small Class C and the mempool +stay explicit process buffers. A host A/B comes before a demap. Agent +correctness tests under `/tmp` are required; perf ship/fail is host-only. --- diff --git a/docs/peer-clients.md b/docs/peer-clients.md index e448bd9acf..b84908dcf1 100644 --- a/docs/peer-clients.md +++ b/docs/peer-clients.md @@ -205,8 +205,9 @@ peer**. Copying that would mean reintroducing a coins view. ## Ranked if we spend time later -Do not treat this as Open rank. Promote into [`quality.md`](./quality.md) -only when scheduling a slice. +Not scheduled. Implement a row only when the user names it. Do not treat +this as Open rank. Promote into [`quality.md`](./quality.md) only when +scheduling a slice. | Rank | Item | Source | Lands in | |-----:|------|--------|----------| diff --git a/docs/personal-node-plans/README.md b/docs/personal-node-plans/README.md index cf8f258615..713a975b5d 100644 --- a/docs/personal-node-plans/README.md +++ b/docs/personal-node-plans/README.md @@ -1,5 +1,8 @@ # Personal wallet-node plans +**Not scheduled.** Implement a numbered file only when the user names that +plan. Do not start the next file because it is listed here. + Implementation plans for a **home node that cannot open clearnet listen ports**: P2P over Tor SOCKS / I2P SAM / CJDNS, Electrum and Esplora on onion (and I2P when SAM can forward), P2P inbound on a Tor onion, local tx announce on a @@ -11,7 +14,7 @@ is the plan group only. Do not copy step lists into [`quality.md`](../quality.md until a slice is scheduled (same pattern as [`peer-clients.md`](../peer-clients.md) ranked items). -Each numbered file is one plan: [`how-we-plan.md`](../how-we-plan.md) +Each numbered file is one plan: [`how-we-plan.md`](../how-we-plan.md#agent-contract) (Contract / Red / Green / Refactor / Verify). Commands: [`.agents/skills/ship-pr/SKILL.md`](../../.agents/skills/ship-pr/SKILL.md). Do not copy the cycle here. @@ -47,7 +50,7 @@ A node on CGNAT / no forwarded TCP still: the write thread joins) grows — unlikely here. - One product PR per numbered file. Each step is one cycle turn, committed before the next step starts, only after **local CI except coverage** - ([`how-we-plan.md`](../how-we-plan.md); commands in + ([`how-we-plan.md`](../how-we-plan.md#agent-contract); commands in [ship-pr](../../.agents/skills/ship-pr/SKILL.md)). Coverage and native `windows` / `macos` stay GitHub Actions. Optional holistic refactor after the last step, then push and poll. diff --git a/docs/quality.md b/docs/quality.md index 643b5e5711..abb990c3fe 100644 --- a/docs/quality.md +++ b/docs/quality.md @@ -5,11 +5,11 @@ in [`CHANGELOG.md`](../CHANGELOG.md). 1.0 product gates: [`road-to-1.0.md`](./road-to-1.0.md). Peer-node notes: [`peer-clients.md`](./peer-clients.md) (do not copy here). -**Last reaudit:** 2026-09-17. Schema **24**. Core functional **75** -`test_runner` jobs (**69** inventory `run`; Core expands transport twins and -`wallet_txn_*` flags) / **198** `skip`. Findings **001–023** fixed. Nightly -fuzz **20** jobs. Weekly mutants **8** shards (not required; owner -[`TESTING.md`](../TESTING.md)). Previous: 2026-09-15. +**Last reaudit:** 2026-09-17. Live schema: [`SCHEMA.md`](../SCHEMA.md). +Core functional inventory: [`core-functional.md`](./core-functional.md). +Findings **001–023** fixed. Nightly fuzz **20** jobs. Weekly mutants **8** +shards (not required; owner [`TESTING.md`](../TESTING.md)). Previous: +2026-09-15. | Section | Purpose | |---------|---------| @@ -24,14 +24,16 @@ evidence (failed Core corpus, new dual path, red required CI, MSRV drift). ## Open (priority order) +Counts and procedures stay in the owner doc. A row is the rank and the outcome. + | Rank | ID | Item | Done looks like | |-----:|----|------|-----------------| -| 1 | **Q-41** | Grow Core functional `run` set | Inventory `run` covers claimed wallet-client / P2P / mempool / buried-activation. Labeled `test_runner` prints **77** (**71** inventory `run` / **196** skip; 17 `rpc-missing`, 16 `core-log`, 68 `no-wallet`; Core expands transport twins and `wallet_txn_*` flags). COMPAT leftovers are `rpc-dialect`, not `rpc-missing`. `run` must hit node production (not only shim argv / dummy `blk*.dat` / Core decode dialect). Named Core-functional shim dialect (`RBITCOIN_RPC_WAIT_TIP_IDLE`, `RBITCOIN_RPC_PACKAGE_DIALECT`, proxy abort JSON) is documented, not a silent Core-clone. Next inventory `run`: recover to 72. `mempool_accept` stays skip (`policy-libre`). Unlabeled PRs stay cargo-only. Owner: [`core-functional.md`](./core-functional.md). | -| 2 | **Q-48** | BIP331 rust-bitcoin package types | Native BIP331 `NetworkMessage` when rust-bitcoin exposes it (**RB-007**). Local packages: RPC `submitpackage`, Esplora `POST /txs/package`, Electrum 1.6 `broadcast_package`. `protocol_max` is **1.6**; 1.7 `scriptpubkey.*` still missing. | +| 1 | **Q-41** | Grow Core functional `run` set | Inventory `run` covers claimed wallet-client / P2P / mempool / buried-activation, and each `run` hits node production. Owner: [`core-functional.md`](./core-functional.md). | +| 2 | **Q-48** | BIP331 rust-bitcoin package types | Native package types when rust-bitcoin exposes them (**RB-007**), then the local RPC / Esplora / Electrum package calls. Owner: [`rust-bitcoin-limitations.md`](./rust-bitcoin-limitations.md), [`COMPAT.md`](../COMPAT.md). | | 3 | **Q-31** | Hermetic tip fixtures | Frozen signet/mainnet tip packs for offline consensus/Electrum regression (no live API). Fuzz already merges tiny `signet_block_*.bin` / `mainnet_block_290329.bin`. Electrum hermetic packs still Open. | | 4 | **R-10** | Residual god-files | Peel **only** when a higher row needs a seam. Do not split `interpreter.rs` opcode `match` or io_uring machines. Named extracts: **Q-61** Completed. **Q-56** Completed (scriptnum + pack-ints). **Q-68** Completed (create.loc SIMD). | | 5 | **Q-67** | `asked_blocks` clone on hold | `hold_body` clones `asked_blocks` before `held_bodies` insert so the read lock does not overlap the write (`HeldBodies::insert` already takes `&HashSet`). Bound is `MAX_SERVE_BLOCKS` × peers. Follow-up: pass the read guard with a documented lock order, or keep the clone as a named trade. Owner: `crates/rbitcoin-net/src/chain.rs`. | -| 6 | **Q-69** | CLN / LDK chain backend | Operator can point Core Lightning `bcli` and ldk-node Esplora/Electrum at this node on regtest/signet. Electrum/Esplora **start without** `--sh-index`; SH-only APIs fail closed (`scripthash index disabled`). Channel watches (txid/outpoint) work. BDK address history still needs SH. No LND/ZMQ/BIP157. Owner: [`lightning.md`](./lightning.md). | +| 6 | **Q-69** | CLN / LDK chain backend | Operator can point CLN `bcli` and ldk-node Esplora/Electrum at this node. Owner: [`lightning.md`](./lightning.md). | R-ids were the 2026-08-12 slice. Canonical id is **bold**. Do not start **R-11+**. Next unused Q-id is **Q-70**. @@ -100,7 +102,7 @@ checklist. appender; ibd-confirm publishes script waves (no coordinator). Pins are plan/batch only. [`concurrency.md`](./concurrency.md), [`invariants.md`](./invariants.md). -- **On-disk:** [`SCHEMA.md`](../SCHEMA.md) current bytes (today **24**). +- **On-disk:** [`SCHEMA.md`](../SCHEMA.md) current bytes (live `SCHEMA_VERSION`). Soft-migrate durable side formats; bump or refuse; **no silent wipe**. Same commit as the format code. - **io_uring:** do not flatten a purpose-built machine to batched diff --git a/docs/road-to-1.0.md b/docs/road-to-1.0.md index 4ba136275a..275c16e9ba 100644 --- a/docs/road-to-1.0.md +++ b/docs/road-to-1.0.md @@ -10,8 +10,8 @@ Day-to-day ranked work stays in [`quality.md`](./quality.md). Electrum/Esplora need `--sh-index` (default off). BIP324 v2-only. Install is a GitHub Release (Linux musl; Windows/Darwin snapshots). Nightly differential fuzz vs Core v31.1 is continuous (**Q-30**). Core functional -inventory is **71** `run` / **196** `skip` (**Q-41**). Findings **001–023** -are fixed. +inventory counts and the `run` bar: [`core-functional.md`](./core-functional.md) +(**Q-41**). Findings **001–023** are fixed. 1.0 is the first **frozen-format, support-windowed** line. Not a Bitcoin Core clone, not a soak badge, not a desktop wallet.