diff --git a/.agents/skills/ship-pr/SKILL.md b/.agents/skills/ship-pr/SKILL.md index 346cb27a8..c1dd0860f 100644 --- a/.agents/skills/ship-pr/SKILL.md +++ b/.agents/skills/ship-pr/SKILL.md @@ -12,84 +12,46 @@ One session worktree, one topic branch per plan, many small commits (each already passed local CI except coverage), required checks green. A plan is not done until those checks are green. Do not merge unless asked. -Owner facts for tests and coverage stay in [`TESTING.md`](../../../TESTING.md). -Do not copy them here. +When to run which command: +[`docs/how-we-plan.md`](../../../docs/how-we-plan.md#agent-contract). +Suite and budgets: [`TESTING.md`](../../../TESTING.md). +Do not copy those facts here. ## Session worktree -Reuse one worktree per session. Do not add a worktree per PR. - -`rearden-grok[bot]` on this operator VM: silo, editor tree, ENOSPC, identity, -and HTTPS push are [`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). If -you are not that identity, ignore that file; use `$PWD/target/dev`. - -```bash -# once per session -git fetch origin -git worktree add /tmp/rbtc- origin/master -cd /tmp/rbtc- -git switch -c / -export CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-$PWD/target/dev}" - -# next PR in the same session (do not add another worktree) -git fetch origin -git switch -C / origin/master -``` - -| Rule | Detail | -|------|--------| -| Base | Current `origin/master` (or `main`) | -| Branch | Topic name. Never commit the plan onto `master` | -| Worktree | One `/tmp/rbtc-` for the session | -| `CARGO_TARGET_DIR` | `$PWD/target/dev` unless already set. `rearden-grok[bot]`: [`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md) | -| Remotes | Worktrees share `origin`. Never `git remote set-url origin` | - -### After merge +One `/tmp/rbtc-` for the session. The next PR is `git switch -C` +from current `origin/master`, not another worktree. Never commit the plan +onto `master`. Worktrees share `origin`. Never `git remote set-url origin`. -Delete the topic branch only. Keep the session worktree until the session ends. -Do not delete a branch that still has an open PR. Do not `git push --delete master`. +`rearden-grok[bot]`: silo, identity, HTTPS `HEAD:` push, and session +end are [`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). If you are not +that identity, ignore that file. Use `$PWD/target/dev` and push the topic +branch with ordinary `git push`. -```bash -git fetch origin --prune -git switch -C / origin/master -git branch -d / -# rearden-grok[bot]: delete the topic branch over HTTPS (rearden-vm-HOST.md) -``` - -```bash -# session end -git worktree remove /tmp/rbtc- -git fetch origin --prune -``` - -`rearden-grok[bot]` session-end extras (test dirs, shared silo): -[`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). Other identities: -ignore that file. +After merge, delete the topic branch only. Keep the session worktree until +the session ends. Do not delete a branch that still has an open PR. Do not +`git push --delete master`. ## Local tests From `nix-shell` (CI pins rustc 1.95.0). `CARGO_TARGET_DIR` as in Session worktree. Same **commands** as the required jobs, not the GitHub Actions -`env:` -(leave `CARGO_INCREMENTAL` unset; that is [`ci.yml`](../../../.github/workflows/ci.yml) only). -When to run which command: -[`docs/how-we-plan.md`](../../../docs/how-we-plan.md) (The cycle). -Suite and budgets: [`TESTING.md`](../../../TESTING.md). +`env:` (leave `CARGO_INCREMENTAL` unset; that is +[`ci.yml`](../../../.github/workflows/ci.yml) only). | When | Run | |------|-----| | Inner loop (Red / Green) | Targeted `cargo test -p … -- --quiet`. `cargo check -p --lib`. Not `--tests` after every edit | | After Green | `cargo test --workspace --quiet` | | After Refactor, before commit | Local CI except coverage (below) | +| Docs, comments, formatting only | Skip the suite. `cargo fmt` if rustfmt would touch the tree | | Core functional CI red | [core-functional skill](../core-functional/SKILL.md) | | Overlay functional CI red | [overlay-functional skill](../overlay-functional/SKILL.md) | | Never local | `./scripts/coverage.sh`, `nix build .#rbitcoin-musl`, host IBD | -Redirect every command above (stdout and stderr to `/tmp`, print `EXIT`, -search then tail on failure). Do not `tee` or `| tail` as the only sink, -and do not load the log wholesale. Recipe: -[`docs/how-we-plan.md`](../../../docs/how-we-plan.md) (Agent RAM). Do not -copy that snippet here. +Redirect every cargo, clippy, deny, and rustc command (stdout and stderr +to `/tmp`, print `EXIT`, search then tail on failure). Recipe: +[`docs/how-we-plan.md`](../../../docs/how-we-plan.md#agent-ram). Local CI except coverage: @@ -105,12 +67,10 @@ cargo test --workspace --quiet If the slice changed `flake.nix`, `nix/`, or the NixOS module, also `nix build .#checks.x86_64-linux.nixos-module-eval --no-link`. Label the PR `nixos-module-runtime` when systemd deps, users/groups, firewall, or the VM -start argv changed — that runs `nixos-module-runtime` (qemu). Do not wait -out the VM test locally. +start argv changed. Do not wait out the VM test locally. Coverage and native `windows` / `macos` stay GitHub Actions; -`ci-os-smoke.sh` is the local stand-in. Do not wait out a host IBD or a -coverage run in the agent VM. +`ci-os-smoke.sh` is the local stand-in. ## Push and poll @@ -128,9 +88,7 @@ Label `nixos-module-runtime` when the NixOS module VM test should run (not eval). Poll with `--interest nixos-module-runtime`. It is not a required check. -`rearden-grok[bot]` fetch/push (`gh-login.sh`, HTTPS `HEAD:`, never -`git push origin`, no `-u`): [`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). -If you are not that identity, ignore that file. +Bot fetch and push: [`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). ```bash gh pr create --repo reardencode/rbitcoin --head / --title "…" --body "…" @@ -146,11 +104,10 @@ operator (`security_events` is read). An alert that only fires in `#[cfg(test)]` or a test module: stop. Do not rename tests or shuffle literals. Ask the operator to dismiss. Production CodeQL is a real finding — fix it. -**Fail-fast poll.** Do not use `gh pr checks --watch` as the only waiter. It -blocks until every check finishes, including slow `coverage` and CodeQL, even -after a job this change exercises is already red. `./scripts/pr-checks-watch.sh` -exits 1 as soon as any required job fails, and as soon as a `--interest` job -fails. On exit 1: stop polling, fetch that job’s log, and start the fix. +**Fail-fast poll.** `./scripts/pr-checks-watch.sh` exits 1 as soon as any +required job fails, and as soon as a `--interest` job fails. On exit 1: +stop polling, fetch that job’s log, and start the fix. `gh pr checks --watch` +waits out `coverage` and CodeQL after a job this change already failed. ```bash gh api --allow-escape-sequences repos/reardencode/rbitcoin/actions/jobs//logs @@ -159,14 +116,10 @@ gh run view --repo reardencode/rbitcoin --job --log-failed **Unmergeable means tests do not run.** If `mergeable` is `CONFLICTING` or `mergeStateStatus` is `DIRTY` (or `BEHIND` when the branch is not on current -`origin/master`), required `test` CI does not start. Rebase, lease-force the -topic branch, then poll. Do not `gh run rerun` to wake jobs that never queued. - -```bash -git fetch origin -git rebase origin/master -# rearden-grok[bot]: lease-force over HTTPS (rearden-vm-HOST.md) -``` +`origin/master`), required `test` CI does not start. Rebase onto +`origin/master`, lease-force the topic branch, then poll. Do not `gh run rerun` +to wake jobs that never queued. Bot lease-force: +[`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). | Rule | Detail | |------|--------| @@ -185,16 +138,16 @@ When required checks are green locally and CI only needs a re-run: |----|--------| | `gh run rerun [--failed]` | Empty commit whose only purpose is to wake Actions | | GitHub Actions re-run failed or all jobs | Noise commits (`ci: bump`, `trigger`) | -| Amend the tip and `--force-with-lease` the topic branch over HTTPS | Force-push `master` / `main` | +| Amend the tip and `--force-with-lease` the topic branch | Force-push `master` / `main` | ```bash gh run rerun --failed - -# If the API rejects: amend the tip (no empty commit) and lease-force the topic branch only -git commit --amend --no-edit -# rearden-grok[bot]: lease-force over HTTPS (rearden-vm-HOST.md) ``` +If the API rejects, amend the tip (no empty commit) and lease-force the +topic branch only. Bot lease-force is +[`rearden-vm-HOST.md`](../../../rearden-vm-HOST.md). + If CI `coverage` fails, add a pin and push. Floor: [`TESTING.md`](../../../TESTING.md). Operator musl binaries are [`docs/releases.md`](../../../docs/releases.md). diff --git a/AGENTS.md b/AGENTS.md index e6f7f4007..c08ca012c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,9 +1,10 @@ # Agent notes Documentation map (one owner per fact): [`docs/README.md`](docs/README.md). -Before exploring, read [`docs/ORIENT.md`](docs/ORIENT.md). -This file is the harness-injected hard-rule contract. Design lives in the -owner docs; do not grow a second design book here. +When the task area is unclear, [`docs/ORIENT.md`](docs/ORIENT.md) routes. +Open the one row that matches the change. This file is the harness-injected +hard-rule contract. Design lives in the owner docs; do not grow a second +design book here. Process playbooks are Agent Skills under [`.agents/skills/`](.agents/skills/). Read the matching `SKILL.md` when the task needs it. Do not paste bash, @@ -15,22 +16,27 @@ If you are acting as `rearden-grok[bot]`, also read [`rearden-vm-HOST.md`](rearden-vm-HOST.md). If you are not that identity, ignore that file. -## Language, comments, composition +## Language and shape Write clear, concrete technical English. Keep Core-aligned terms where we match Bitcoin Core. Do not inject moralizing or political framing, or soften consensus and security language. If a rename is not clearer engineering, keep the existing term. -Comments that restate what, why, or weird are a smell. Prefer names, types, -and structure. Keep `//` only for an invariant, protocol rule, `SAFETY`, or -library quirk. Crate and public rustdoc (`//!` / `///`) is not this rule. -Full text: [`CONTRIBUTING.md`](CONTRIBUTING.md) principle 7. - -Prefer composition (has-a) over inheritance; avoid tall trees. Build -immutable structures once, then compose them. If a map needs extra fields, -wrap it on read rather than mutating members in place. -Control flow: principle 10 and [`docs/code-shape.md`](docs/code-shape.md). +Open [`CONTRIBUTING.md`](CONTRIBUTING.md) principles 7–11 and +[`docs/code-shape.md`](docs/code-shape.md) when editing Rust. Short form: + +- `//` only for an invariant, protocol rule, `SAFETY`, or library quirk. + Crate and public rustdoc (`//!` / `///`) is not this rule (principle 7). +- Tests assert shipped behavior, not repo text. No `*_for_test` backdoors. + IO pins use session or table stats or on-disk state (principle 8). +- A RAM or CPU trade is named (principle 9). +- Control flow and composition: principle 10 and `docs/code-shape.md`. +- Crate `pub` is the cross-crate graph only. No unused `pub`. No + `dead_code` silence. `#[cfg(test)]` on production items is a smell, + including fuzz-only exports (principle 11). +- One logical change per commit. The message says what and why. Not WIP, + misc, or a drive-by rename mixed with behavior. ## Store and IBD @@ -65,8 +71,8 @@ Perf A/B is operator-host only. One plan step is one Red → Green → Refactor turn, committed before the next step. Keep `--lib` compiling (wrap the old API, switch one caller). Inner loop is `cargo test -p --lib` / `cargo check -p --lib`. -Do not `cargo check --tests` after every edit. Owner: -[`docs/how-we-plan.md`](docs/how-we-plan.md). Commands: +Do not `cargo check --tests` after every edit. Cycle and Agent RAM: +[`docs/how-we-plan.md`](docs/how-we-plan.md#agent-contract). Commands: [`.agents/skills/ship-pr/SKILL.md`](.agents/skills/ship-pr/SKILL.md). **Agent RAM:** never load `cargo test`, clippy, deny, or rustc stdout into @@ -81,24 +87,11 @@ identities: ignore that file; use `$PWD/target/dev`. One production implementation at the lowest crate that owns the concept. Extract is a move: [`docs/code-shape.md`](docs/code-shape.md). Core-facing -RPC / P2P / Electrum / Esplora: [`COMPAT.md`](COMPAT.md). - -Tests assert shipped behavior, not repo text -([`CONTRIBUTING.md`](CONTRIBUTING.md) principle 8). Budgets, no `*_for_test` -backdoors, no production-scale default fixtures: [`TESTING.md`](TESTING.md). -Tests use session or table instance stats or on-disk state, not thread-local -hot-path IO probes. - -Crate `pub` is the cross-crate graph only. Unused `pub` is forbidden. -`#[cfg(test)]` on production items is a smell, including fuzz-only exports. -Do not leave dead code or silence `dead_code`. A RAM or CPU trade is named -([`CONTRIBUTING.md`](CONTRIBUTING.md) principle 9). - -One logical change per commit. The message says what and why. Not WIP, misc, -or a drive-by rename mixed with behavior. +RPC / P2P / Electrum / Esplora: [`COMPAT.md`](COMPAT.md). Budgets and +fixtures: [`TESTING.md`](TESTING.md). Before the first edit under `crates//`, read `crates//AGENTS.md` -when it exists. +when it exists, and open the read-first row that matches the change. ## Ship, release, Core functional diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fe108a3d6..2913a6543 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -246,16 +246,12 @@ digging into a monolithic job log. Label **`nixos-module-runtime`** to run the NixOS module qemu test (not eval). That job is not required. -**Agents** implement in **one git worktree per session**, a topic branch per -PR, and **one PR** per plan. Each plan step follows -[`docs/how-we-plan.md`](./docs/how-we-plan.md) and is committed only after -local CI except coverage. Coverage and native `windows` / `macos` stay -GitHub Actions. After merge they delete the topic branch and keep the -session worktree. Never `git remote set-url origin`. Bot push uses an -explicit HTTPS URL (not `git push origin`). Fail-fast poll is -`./scripts/pr-checks-watch.sh`, not `gh pr checks --watch`. Start at -[`docs/ORIENT.md`](./docs/ORIENT.md). -Ship commands: [`.agents/skills/ship-pr/SKILL.md`](./.agents/skills/ship-pr/SKILL.md). +**Agents** open [`docs/ORIENT.md`](./docs/ORIENT.md) when the area is +unclear, follow the agent contract in +[`docs/how-we-plan.md`](./docs/how-we-plan.md#agent-contract), and use +[`.agents/skills/ship-pr/SKILL.md`](./.agents/skills/ship-pr/SKILL.md) to +open or update a pull request. Humans use ordinary git. The bot host file +does not apply to them. Humans who want the same gates offline (Nix optional; rustup 1.95 is enough): diff --git a/SCHEMA.md b/SCHEMA.md index 2821db81a..0715eb48c 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -86,6 +86,14 @@ Older versions and migration notes live in [`SCHEMA_HISTORY.md`](./SCHEMA_HISTOR ## Schema 17 freeze +Class A shape settled here at schema 17 and is still the live layout under +the `SCHEMA_VERSION` at the top of this file (24). Schemas 18–24 already +shipped; they are open rules above and history in +[`SCHEMA_HISTORY.md`](./SCHEMA_HISTORY.md). A byte-incompatible change bumps +from that live constant +([Changing durable bytes](#changing-durable-bytes)). It does not target +schema 18. + ### What 17 locks (on-disk) | Object | Frozen choice | @@ -113,17 +121,19 @@ catalogs, or 16-layout Class A with creates is **refused**. ### New script kinds without a wipe Kind nibble **10–15** is **Corrupt** on this binary (no implicit width). A new -consensus script type does **not** force a 17 datadir wipe: +consensus script type does **not** force a wipe of the Class A shape settled +at 17: -| Path | On-disk | Old 17 binary | New binary | -|------|---------|---------------|------------| +| Path | On-disk | This binary | A later binary | +|------|---------|-------------|----------------| | **RAW** | kind 0 + CompactSize + bytes | already decodes | same | -| **Soft-18** | new kind nibble + known width; `SCHEMA_VERSION = 18` | **refuses** 18 `meta` (or unknown kind) | reads 17 files; writes 18 | +| **Kind nibble** | new kind + known width; bump `SCHEMA_VERSION` | refuses the new `meta` (or unknown kind) | reads these files; writes the new version | -Use RAW when the type is rare. Use soft-18 when the type is common enough to -pay a width table. Soft-18 is **not** a silent in-place rewrite of 17 files. -Inwit `create_fk` Δfk (parked) is the same class: **18 or an inwit-only -rewrite**, not a silent 17 mutate. +Use RAW when the type is rare. Use a kind-nibble bump when the type is common +enough to pay a width table. That bump is not a silent in-place rewrite of +existing files. Inwit `create_fk` Δfk (parked) is the same class: a +`SCHEMA_VERSION` bump or an inwit-only rewrite, not a silent mutate of +existing rows. ### Field widths (10 years) @@ -140,19 +150,20 @@ Assume ~400k–700k creates/day. Ten years ≈ +1.5e9…2.6e9 creates on top of | `tx.head` bits | 25-bit segments | Roll + seal; no mono-file widen | | SH megakey page | 4 KiB delta stream | Page chain; not a single-integer cap | | `sp_tweaks` off | u32 per segment | Already segmented | -| Script kind | 4 bits | 0–9 used; 10–15 reserved Corrupt; extension = RAW or soft-18 | +| Script kind | 4 bits | 0–9 used; 10–15 reserved Corrupt; extension = RAW or a kind-nibble bump | Practical risks are **loc overflow without ovf** (sentinel with missing sidecar) and **Bitcoin timestamp 2106** (consensus, every node). -### What would force schema 18 +### What forces a schema bump A **byte-incompatible** change to Class A / OA / body / idx / SH catalog -layout, or anything that cannot soft-open 17 files. +layout, or anything that cannot soft-open the live files. Bump from the +live `SCHEMA_VERSION`, not from 17. -| Change | 18? | Notes | -|--------|-----|-------| -| New implicit-width script kind | Optional | RAW = no bump; nibble = soft-18 (17 refuses 18) | +| Change | Bump? | Notes | +|--------|-------|-------| +| New implicit-width script kind | Optional | RAW = no bump; nibble = next `SCHEMA_VERSION` (this binary refuses it) | | Inwit Δfk | Yes or inwit-only rewrite | Parked; cold stem | | Idx not stride-8 / not u32 | Yes | Would retire the 8-align pad | | Packed Class A again / merge stems | Yes | Wipe | @@ -161,10 +172,11 @@ layout, or anything that cannot soft-open 17 files. | Fuse8 envelope v3 | No | Soft-migrate like v1→v2 (log + rewrite; no wipe) | | Independent rolls / L2 strong / no DONTCACHE | No | Writer/RAM only | -Parked size work that is **not** 17: inwit Δfk; drop 8-align pad on empty -inwit / zero-out spent (needs a different idx encoding); `txid.body` -compression. Do not chase `inwit` size as an IBD **hot-set** win — put it -on a cold volume (`--datadir-cold`). Census: [Mainnet census](#mainnet-census-this-trees-reference-datadir-2026-08-13). +Parked size work that is **not** a live-layout tweak: inwit Δfk; drop 8-align +pad on empty inwit / zero-out spent (needs a different idx encoding); +`txid.body` compression. Do not chase `inwit` size as an IBD **hot-set** +win — put it on a cold volume (`--datadir-cold`). Census: +[Mainnet census](#mainnet-census-this-trees-reference-datadir-2026-08-13). Process: bump `SCHEMA_VERSION`, document this file + `SCHEMA_HISTORY.md` in the same commit, refuse or soft-open with a one-line operator message. Do @@ -458,9 +470,10 @@ one code to “extension follows” and still read `e=0..=9` records). Decode expands templates to wire scripts (P2TR is `5120||32`). XOR at rest covers hash/data only. Spender flags live only on `spent`. A new consensus -script type does **not** wipe a 17 datadir: encode it as kind 0 **RAW**, or -introduce an implicit-width nibble as **soft-18** (18 reads 17; this 17 -binary refuses 18 / unknown kind). See [Schema 17 freeze](#schema-17-freeze). +script type does **not** wipe the Class A shape settled at 17: encode it as +kind 0 **RAW**, or introduce an implicit-width nibble as a +`SCHEMA_VERSION` bump (the new binary reads these files; this binary refuses +the new `meta` or an unknown kind). See [Schema 17 freeze](#schema-17-freeze). ### Sole-spender slot (`spent.body`) diff --git a/SCHEMA_HISTORY.md b/SCHEMA_HISTORY.md index 5d87a5b5b..2159d9991 100644 --- a/SCHEMA_HISTORY.md +++ b/SCHEMA_HISTORY.md @@ -130,14 +130,15 @@ Closed layout. SH run catalogs are unique `(scripthash, create_fk)` at `key_len=40`. Schema-16 `key_len=32` catalogs are refused. Hot Class A: thin LAYOUT17 `txout` meta, script kinds **0–9** (unknown -kind is Corrupt). A new consensus script is **RAW** (no bump) or -**soft-18** (18 reads 17; 17 refuses 18) — not a 17 datadir wipe. 8 B -spent slots, `spent.ovf` overflow. Inwit flags bits 4–7 and spent flags -other than `MULTI_SPENDER` are Corrupt. Inwit prevout is still -`create_fk:u64` + CompactSize vout (Δfk is an **18 / inwit-only** -follow-up). Writer/RAM on 17 (not on-disk): idx stems roll independently; -`strong_tx` always L2; no `RWF_DONTCACHE`. See -[`SCHEMA.md`](SCHEMA.md) (Schema 17 freeze). +kind is Corrupt). At schema 17, a new consensus script was **RAW** (no bump) +or a kind-nibble bump the then-current binary would refuse — not a 17 +datadir wipe. 8 B spent slots, `spent.ovf` overflow. Inwit flags bits 4–7 +and spent flags other than `MULTI_SPENDER` are Corrupt. Inwit prevout is +still `create_fk:u64` + CompactSize vout (Δfk was parked as an inwit-only +or later-version follow-up). Writer/RAM on 17 (not on-disk): idx stems roll +independently; `strong_tx` always L2; no `RWF_DONTCACHE`. See +[`SCHEMA.md`](SCHEMA.md) (Schema 17 freeze). The live bump rule is that +file, not this section. Megakey SH pages store ULEB128 fk0+deltas (`ver=1`). Leftover raw-u64 pages (`ver=0`, `n>0`) rematerialize. @@ -145,8 +146,9 @@ pages (`ver=0`, `n>0`) rematerialize. **Body orientation (still 17):** new creates write `scripthash.body/NN` + `scripthash.ovf/body`. Existing file `scripthash.body` stays shared (one writer). Mixed file+dir or a dir without `ovf/body` is Layout -refuse (wipe `store/scripthash*`). Not schema 18 — 18 remains the next -incompatible Class A / idx change. +refuse (wipe `store/scripthash*`). That body-dir change was not schema 18. +At the time, 18 was the next incompatible Class A / idx change; 18–24 have +since shipped. `archive_epoch` is gone. Create does not plant `wire/` or packed `tx.body`. Open unlinks leftover `archive_epoch`, `store/wire`, and diff --git a/crates/rbitcoin-consensus/AGENTS.md b/crates/rbitcoin-consensus/AGENTS.md index 3aed83f39..f7e7e6e94 100644 --- a/crates/rbitcoin-consensus/AGENTS.md +++ b/crates/rbitcoin-consensus/AGENTS.md @@ -6,6 +6,8 @@ Do not run the interpreter on an I/O thread. ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Rules we own vs Core corpora | [`docs/consensus-tests.md`](../../docs/consensus-tests.md) | diff --git a/crates/rbitcoin-esplora/AGENTS.md b/crates/rbitcoin-esplora/AGENTS.md index dc2207828..d6bcddd09 100644 --- a/crates/rbitcoin-esplora/AGENTS.md +++ b/crates/rbitcoin-esplora/AGENTS.md @@ -6,6 +6,8 @@ need `--sh-index`. ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Shipped HTTP / WS surface | [`COMPAT.md`](../../COMPAT.md) | diff --git a/crates/rbitcoin-net/AGENTS.md b/crates/rbitcoin-net/AGENTS.md index be987182c..a728f20ac 100644 --- a/crates/rbitcoin-net/AGENTS.md +++ b/crates/rbitcoin-net/AGENTS.md @@ -5,6 +5,8 @@ P2P and IBD. Depends on query, store, consensus, and mempool. ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Roles, body queue, publish order | [`docs/concurrency.md`](../../docs/concurrency.md) | diff --git a/crates/rbitcoin-query/AGENTS.md b/crates/rbitcoin-query/AGENTS.md index 80c6c0d3f..8e65f8756 100644 --- a/crates/rbitcoin-query/AGENTS.md +++ b/crates/rbitcoin-query/AGENTS.md @@ -6,6 +6,8 @@ a query API already owns the fact. ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Stage IO, pin identity, leftover | [`docs/invariants.md`](../../docs/invariants.md) | diff --git a/crates/rbitcoin-rpc/AGENTS.md b/crates/rbitcoin-rpc/AGENTS.md index f28603bb1..eab4fb32b 100644 --- a/crates/rbitcoin-rpc/AGENTS.md +++ b/crates/rbitcoin-rpc/AGENTS.md @@ -6,6 +6,8 @@ net, and consensus. The HTTP server is `src/server.rs`; methods are ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Method list, auth, permanent gaps | [`docs/rpc.md`](../../docs/rpc.md) | diff --git a/crates/rbitcoin-store/AGENTS.md b/crates/rbitcoin-store/AGENTS.md index 4c3ada705..57755bfa9 100644 --- a/crates/rbitcoin-store/AGENTS.md +++ b/crates/rbitcoin-store/AGENTS.md @@ -6,6 +6,8 @@ and `rbitcoin-log`. `rbitcoin-query` and everything above it call this crate. ## Read first +Open the row that matches the change. Leave the other owners closed. + | Change | Read | |--------|------| | Stage IO, leftover union, no silent fallback | [`docs/invariants.md`](../../docs/invariants.md) | diff --git a/docs/ORIENT.md b/docs/ORIENT.md index 9f32a33aa..ac5e7d055 100644 --- a/docs/ORIENT.md +++ b/docs/ORIENT.md @@ -1,7 +1,8 @@ # Agent orientation -Read this before a broad search when the task area is unclear. +Open this when the task area is unclear. Facts stay in the owner files in [`README.md`](./README.md). This page only routes. +Open the one row that matches the change. Leave the other owners closed. ## Crate graph @@ -30,8 +31,8 @@ Before the first edit in a crate, read `crates//AGENTS.md` when that file | Which head file (tx / header / scripthash) | [`heads.md`](./heads.md) | | Crash, tip-as-commit, kill-9 | [`crash-recovery.md`](./crash-recovery.md) | | Tests, budgets, coverage, fixtures | [`../TESTING.md`](../TESTING.md) | -| cargo-mutants (weekly sweep, PR advisory) | [`../TESTING.md`](../TESTING.md) (Mutation testing); snapshot [`mutants/`](./mutants/) | -| Multi-step plan (Red → Green → Refactor) | [`how-we-plan.md`](./how-we-plan.md) | +| A missed cargo-mutants case | [`../TESTING.md`](../TESTING.md) (Mutation testing). Open [`mutants/`](./mutants/) only for that miss | +| Multi-step plan (Red → Green → Refactor) | [`how-we-plan.md`](./how-we-plan.md#agent-contract) (stop before Rationale) | | cargo / clippy / deny / rustc logs (do not load into the session) | [`how-we-plan.md`](./how-we-plan.md) (Agent RAM) | | `rearden-grok[bot]` operator VM (ignore unless that identity) | [`../rearden-vm-HOST.md`](../rearden-vm-HOST.md) | | Intentional Core / Electrum / Esplora differences | [`../COMPAT.md`](../COMPAT.md) | @@ -43,9 +44,18 @@ Before the first edit in a crate, read `crates//AGENTS.md` when that file ## Ask first -Hard stops are the Store and IBD bullets and the Ship landmines -(fail-fast poll) in [`../AGENTS.md`](../AGENTS.md). +Hard stops in [`../AGENTS.md`](../AGENTS.md) are proceed-or-stop rules. `rearden-grok[bot]` also follows [`../rearden-vm-HOST.md`](../rearden-vm-HOST.md); other identities ignore that file. -Also ask before expanding who is trusted beyond the boundary in -[`../SECURITY.md`](../SECURITY.md). + +Ask before any of these, including when the change looks locally justified: + +- Replace a purpose-built IO machine with batched `pread` / `pwrite`. +- Bump or wipe the schema, or add an `RBITCOIN_*` knob. +- Widen who is trusted beyond [`../SECURITY.md`](../SECURITY.md). +- Merge, force-push `master`, or `cargo clean` the shared cargo silo. +- Start an item in [`quality.md`](./quality.md), [`peer-clients.md`](./peer-clients.md), + [`personal-node-plans/`](./personal-node-plans/), or [`road-to-1.0.md`](./road-to-1.0.md) + that the user did not name. +- Open a mainnet datadir, or treat an agent-VM run as a perf result. +- Label an ordinary net or RPC pull request `core-functional`. diff --git a/docs/README.md b/docs/README.md index 07ef2599d..46d5e8175 100644 --- a/docs/README.md +++ b/docs/README.md @@ -33,7 +33,7 @@ into `AGENTS.md`. | Doc | Owns | |-----|------| -| [`ORIENT.md`](./ORIENT.md) | Agent task router: crate graph and read-first links. Not a second index of facts. | +| [`ORIENT.md`](./ORIENT.md) | Agent task router: crate graph and read-first links. Open one matching row. Not a second index of facts. | | [`architecture.md`](./architecture.md) | Why this node is different (Core / Fulcrum contrasts). No stage-IO table copy. | | [`concurrency.md`](./concurrency.md) | Writer roles, publish order, body-queue, pins. Links invariants for leftover union. | | [`invariants.md`](./invariants.md) | Confirm stage IO (the **only** copy), leftover union, store start states S0–S4, no silent fallbacks. | @@ -47,7 +47,7 @@ into `AGENTS.md`. | [`consensus-tests.md`](./consensus-tests.md) | Rules we own vs Core corpora. | | [`core-functional.md`](./core-functional.md) | Core v31.1 functional harness. | | [`overlay-functional.md`](./overlay-functional.md) | Private Tor / i2pd / cjdns mesh harness (labeled / nightly). | -| [`how-we-plan.md`](./how-we-plan.md) | Red → Green → Refactor planning contract; Agent RAM (redirect cargo/clippy/rustc stdout). | +| [`how-we-plan.md`](./how-we-plan.md) | Agent contract (cycle, Agent RAM, keep-compiling), then human rationale. | | [`personal-node-plans/`](./personal-node-plans/) | Home-node implementation plan group (SOCKS, overlays, wallet onions, ephemeral broadcast, inwit prune / `NETWORK_LIMITED`). Live flags stay OPERATOR/COMPAT when a slice ships. Do not copy step lists into quality.md until scheduled. | | [`releases.md`](./releases.md) | Tag `vX.Y.Z`, `vX.Y.x` patch line, `.99` bump, Highlights / GitHub notes. | | [`code-shape.md`](./code-shape.md) | Control flow, types, naming, composition (CONTRIBUTING principle 10). Named extracts: quality.md **Q-61** (Completed). Clippy: no workspace `allow` list; leftover lints are site-local with a reason. | @@ -56,7 +56,7 @@ into `AGENTS.md`. | [`reproducible-builds.md`](./reproducible-builds.md) | Pinned Nix / musl byte-identity. | | [`rust-bitcoin-limitations.md`](./rust-bitcoin-limitations.md) | Workarounds where rust-bitcoin is not Core-faithful. | | [`mempool-fee-estimation.md`](./mempool-fee-estimation.md) | Fee estimator notes. | -| [`errata.md`](./errata.md) | Known one-off store/confirm quirks. | +| [`errata.md`](./errata.md) | Known one-off store/confirm quirks. Retired confirm dual-path names. | | [`peer-clients.md`](./peer-clients.md) | Hornet Node and satd: what to steal (tests/ideas) and what not to copy. Ranked items stay here; do not copy into quality.md. | | [`lightning.md`](./lightning.md) | CLN and LDK as Bitcoin backends: five `bcli` calls, Esplora/Electrum chain sync, `--sh-index` API matrix. Not LND. | | [`mutants/`](./mutants/) | Generated cargo-mutants missed/timeout snapshot. How to run / CI: [`TESTING.md`](../TESTING.md). Not a quality backlog. | diff --git a/docs/errata.md b/docs/errata.md index 8b59356db..9a01f9a41 100644 --- a/docs/errata.md +++ b/docs/errata.md @@ -31,3 +31,20 @@ connected; n−1 is held in in-flight until after the child pin. Do not grow these maps to `Vec` unless a mainnet miss is shown to be this case. + +## Retired confirm dual paths + +These names are gone. A missed fact the pipeline promised is still +`StoreError::Corrupt("invariant: …")` ([`invariants.md`](./invariants.md)). +Do not treat the list as a set of identifiers to police: + +- Soft spentness recovery for a wrong or missing pin identity. +- Unpinned wire-corrected `create_fk` spentness. +- Load-stage `txid.body` identity fill after lookup promised the stamp. +- `ColdPinMode` Allow/Forbid cold denserels on load (load is range outs only). +- Denserels-as-spender-abs (schema 22 abs is `spent` loc off + `8×vout` only). +- `AssembleMode::Full` / `validate_block_connect` (confirm is optimistic + assemble, then `structural_validate_spends`). +- `archive_plan_batch_from_store` and production `Query` turning TxApply into + a dummy `Block`. `tx_apply_to_tx`, `connect_block`, and + `commit_class_a_only` are `rbitcoin_query::testutil::FixtureChain` only. diff --git a/docs/external_findings/README.md b/docs/external_findings/README.md index d243a4f6a..faaaaf8fb 100644 --- a/docs/external_findings/README.md +++ b/docs/external_findings/README.md @@ -35,7 +35,9 @@ rbitcoin reference, or redteam static analysis). Numbered reports live beside th **Policy:** Core `script_tests` / `tx_valid` / `tx_invalid` corpora must pass **every** data row with **no allowlist**. Do not commit if those tests fail. Findings stay -**fixed** with a named regression on the shipped path. +**fixed** with a named regression on the shipped path. A green regression means +the write-up stays closed. Do not re-derive the bug from the narrative unless +that test fails. **006–009:** consensus accept-invalid (zip 2026-08-10) — **fixed** in-tree. **010–011:** mempool remediation **fixed** (Coin spentness + structural tip checks + fee path). diff --git a/docs/how-we-plan.md b/docs/how-we-plan.md index d07bb9fef..857a85530 100644 --- a/docs/how-we-plan.md +++ b/docs/how-we-plan.md @@ -5,16 +5,147 @@ Work lands as **small vertical slices**. Each slice is one This file owns that cycle. Hard-rule pointer: [`AGENTS.md`](../AGENTS.md). Commands: [`.agents/skills/ship-pr/SKILL.md`](../.agents/skills/ship-pr/SKILL.md). -Plans have more steps than a typical “phase 1 / phase 2” design. Each step -should leave the production path simpler and the suite a sharper pin. +**Agents** follow [Agent contract](#agent-contract) through +[Keep the tree compiling](#keep-the-tree-compiling). Do not preload +[Rationale](#rationale). + +**Humans** learning the method: stories, examples, and the checklist are +the rationale after the contract. + +## Agent contract + +Core functional scripts are not the in-tree Red test. When the labeled +`core-functional` job is red, reproduce locally +([`core-functional.md`](./core-functional.md)). + +## The cycle: Red → Green → Refactor + +One step is one turn of the loop, **committed before the next step starts**. +Red pins behavior, so Green is safe to be crude. The pin makes Refactor +safe. Refactor leaves the production path simpler and the suite sharper, so +the next Red is easier to name. Skip a phase and the loop degrades: no Red +is coding to logs; no Refactor is one-offs that accrete; no **test** +refactor is a slow suite full of brittle twins. Landing several slices and +then fighting CI is the same skip — the gates never ran under a small diff. + +```text +for each plan step: + 1. Red new test; targeted run; see red. No production edit yet. + 2. Green smallest production change; targeted run; see green. + 3. Suite cargo test --workspace; confirm green. + 4. Refactor fold one-offs; production and tests; still green. + 5. Gates local CI except coverage (commands in ship-pr). + 6. Commit then the next step. + +after the last step (optional): + holistic refactor → local CI except coverage → commit + then push and poll GitHub +``` + +Edits inside Red and Green stay targeted: `cargo test -p …` and +`cargo check -p --lib`. Do not `cargo check --tests` after every +edit. The workspace suite is step 3 (and again in step 5 if Refactor +changed code). + +### Agent RAM + +Never return full `cargo test`, clippy, deny, or rustc stdout through the +Shell tool into the agent session. Workspace `cargo test --workspace` is +the dangerous one (will OOM an agent turn). Targeted +`cargo test -p ` still redirect if output can be large. +Docs-only slices still must not dump rustc if they run fmt. + +Redirect **stdout and stderr** to a file under `/tmp` (`mktemp +/tmp/rbtc-agent-XXXXXX.log` or `/tmp/rbtc-agent-.log`). Do not `tee` +(that still streams into the tool result). Do not `| tail` as the only +sink (the compiler still writes a huge pipe into the tool). + +After the command, print **exit code** in the tool result +(`echo EXIT:$?`). Look at that first. + +- Exit 0 with `--quiet` (tests) or clippy `-D warnings` and no output: + green. Do **not** Read the log file. +- Exit ≠ 0: search the file for failure names (`FAILED`, `error:`, clippy + lint ids, first rustc error). Read at most **~80 lines of tail** (or a + tight `rg` window). Do not `Read` a multi-megabyte log. + +`--quiet` is enough to confirm green. Copy-paste shape (adapt the +`cargo …` line per command: clippy `-D warnings`, `cargo deny check`, +`cargo fmt --all -- --check`, `./scripts/ast-grep.sh`, +`./scripts/ci-os-smoke.sh`). Inner loop: +`cargo test -p --lib -- --quiet` redirected the same way. + +```bash +log=$(mktemp /tmp/rbtc-agent-XXXXXX.log) +set +e +cargo test --workspace --quiet >"$log" 2>&1 +ec=$? +set -e +echo EXIT:$ec +if [ "$ec" -ne 0 ]; then + rg -n 'FAILED|^error:|error\[' "$log" | head -n 40 + echo '--- tail ---' + tail -n 80 "$log" +fi +``` + +A sequence of gates can wrap that body in a `run` function (one call per +command). Do not paste a second copy into ship-pr; that file lists the +commands and links here. + +**Agent disk:** `rearden-grok[bot]` follows +[`rearden-vm-HOST.md`](../rearden-vm-HOST.md). Other identities: ignore +that file. Commands: +[`.agents/skills/ship-pr/SKILL.md`](../.agents/skills/ship-pr/SKILL.md). + +Pure docs, comments, or formatting skip Red, Green, and the workspace suite. +Still run `cargo fmt --all` if rustfmt would touch the tree, and the other +gates if the slice also changed Rust, scripts, or lint. + +If Refactor is empty, do not run the workspace suite twice: step 3 plus +fmt / deny / clippy / ast-grep / `ci-os-smoke.sh` is enough. + +Coverage (`./scripts/coverage.sh`) and a host IBD are never local. Native +`windows` / `macos` still run on GitHub Actions; `./scripts/ci-os-smoke.sh` +is the local stand-in. `nixos-module-eval` only when the slice changed +`flake.nix`, `nix/`, or the NixOS module. The qemu VM test +(`nixos-module-runtime`) is GitHub Actions on label **`nixos-module-runtime`** +and on Release tags — not local, not a required PR check. Push may wait until +several slices are committed; each commit must already have passed those +gates. + +A step is not done because it compiles, because the one-off is still there, +or because “CI at the end will catch it.” + + +### Keep the tree compiling + +Schema/API rewires stall when a session deletes the old type (`TxIdx`, a +body meta field, …) and then spends the rest of the turn on `unresolved` / +`dead_code` across store tests, query, consensus, and net. That is not TDD; +it is a compile-doom loop. `rbitcoin-store --tests` is a fat rustc unit — +do not use it as the edit cycle. + +| Do | Do not | +|----|--------| +| New type + its unit tests green, **then** a thin wrap on the old API | Delete the old module in the same dirty tree as all callers | +| Switch **one** caller crate per step; `--lib` stays green | One uncommitted tree spanning store + confirm + query + net + docs | +| Keep `--lib` compiling throughout; commit after local CI except coverage | Hours of un-gated WIP, or a slice commit that has not passed those gates | +| `cargo check -p --lib` (or `cargo test -p --lib `) | `cargo check -p rbitcoin-store --tests` or six-crate `--tests` after each edit | + +`--tests` / multi-crate check belongs after Green and in Gates, not in the +inner loop. Commands: +[`.agents/skills/ship-pr/SKILL.md`](../.agents/skills/ship-pr/SKILL.md). -Influences: Extreme Programming (stories, planning game, small releases, -TDD, continuous refactoring), INVEST stories, vertical slicing, YAGNI / -simple design. Adapted for a consensus + IBD codebase and agent-driven -execution. --- +## Rationale + +The contract above is the cycle. This part is why the repo plans in +small vertical slices, and how to read a step. It is not a second +procedure. + ## Why plan this way | Problem we had | Planning fix | @@ -181,104 +312,7 @@ even if the slices are “vertical.” --- -## The cycle: Red → Green → Refactor - -One step is one turn of the loop, **committed before the next step starts**. -Red pins behavior, so Green is safe to be crude. The pin makes Refactor -safe. Refactor leaves the production path simpler and the suite sharper, so -the next Red is easier to name. Skip a phase and the loop degrades: no Red -is coding to logs; no Refactor is one-offs that accrete; no **test** -refactor is a slow suite full of brittle twins. Landing several slices and -then fighting CI is the same skip — the gates never ran under a small diff. - -```text -for each plan step: - 1. Red new test; targeted run; see red. No production edit yet. - 2. Green smallest production change; targeted run; see green. - 3. Suite cargo test --workspace; confirm green. - 4. Refactor fold one-offs; production and tests; still green. - 5. Gates local CI except coverage (commands in ship-pr). - 6. Commit then the next step. - -after the last step (optional): - holistic refactor → local CI except coverage → commit - then push and poll GitHub -``` - -Edits inside Red and Green stay targeted: `cargo test -p …` and -`cargo check -p --lib`. Do not `cargo check --tests` after every -edit. The workspace suite is step 3 (and again in step 5 if Refactor -changed code). - -### Agent RAM - -Never return full `cargo test`, clippy, deny, or rustc stdout through the -Shell tool into the agent session. Workspace `cargo test --workspace` is -the dangerous one (will OOM an agent turn). Targeted -`cargo test -p ` still redirect if output can be large. -Docs-only slices still must not dump rustc if they run fmt. - -Redirect **stdout and stderr** to a file under `/tmp` (`mktemp -/tmp/rbtc-agent-XXXXXX.log` or `/tmp/rbtc-agent-.log`). Do not `tee` -(that still streams into the tool result). Do not `| tail` as the only -sink (the compiler still writes a huge pipe into the tool). - -After the command, print **exit code** in the tool result -(`echo EXIT:$?`). Look at that first. - -- Exit 0 with `--quiet` (tests) or clippy `-D warnings` and no output: - green. Do **not** Read the log file. -- Exit ≠ 0: search the file for failure names (`FAILED`, `error:`, clippy - lint ids, first rustc error). Read at most **~80 lines of tail** (or a - tight `rg` window). Do not `Read` a multi-megabyte log. - -`--quiet` is enough to confirm green. Copy-paste shape (adapt the -`cargo …` line per command: clippy `-D warnings`, `cargo deny check`, -`cargo fmt --all -- --check`, `./scripts/ast-grep.sh`, -`./scripts/ci-os-smoke.sh`). Inner loop: -`cargo test -p --lib -- --quiet` redirected the same way. - -```bash -log=$(mktemp /tmp/rbtc-agent-XXXXXX.log) -set +e -cargo test --workspace --quiet >"$log" 2>&1 -ec=$? -set -e -echo EXIT:$ec -if [ "$ec" -ne 0 ]; then - rg -n 'FAILED|^error:|error\[' "$log" | head -n 40 - echo '--- tail ---' - tail -n 80 "$log" -fi -``` - -A sequence of gates can wrap that body in a `run` function (one call per -command). Do not paste a second copy into ship-pr; that file lists the -commands and links here. - -**Agent disk:** `rearden-grok[bot]` follows -[`rearden-vm-HOST.md`](../rearden-vm-HOST.md). Other identities: ignore -that file. Commands: -[`.agents/skills/ship-pr/SKILL.md`](../.agents/skills/ship-pr/SKILL.md). - -Pure docs, comments, or formatting skip Red, Green, and the workspace suite. -Still run `cargo fmt --all` if rustfmt would touch the tree, and the other -gates if the slice also changed Rust, scripts, or lint. - -If Refactor is empty, do not run the workspace suite twice: step 3 plus -fmt / deny / clippy / ast-grep / `ci-os-smoke.sh` is enough. - -Coverage (`./scripts/coverage.sh`) and a host IBD are never local. Native -`windows` / `macos` still run on GitHub Actions; `./scripts/ci-os-smoke.sh` -is the local stand-in. `nixos-module-eval` only when the slice changed -`flake.nix`, `nix/`, or the NixOS module. The qemu VM test -(`nixos-module-runtime`) is GitHub Actions on label **`nixos-module-runtime`** -and on Release tags — not local, not a required PR check. Push may wait until -several slices are committed; each commit must already have passed those -gates. - -A step is not done because it compiles, because the one-off is still there, -or because “CI at the end will catch it.” +## How a step reads ### Red: name the contract @@ -392,30 +426,9 @@ Each step is independently green, gated, committed, and shippable. | Plan ignores test runtime | Explicit unit vs scenario choice per step | | Core functional as the default-CI Red | In-tree catalog journey; Core stays nightly | | Several slices uncommitted; first GitHub run is the fmt/clippy/suite gate | Each slice commits only after those gates | -| Delete a large type/module then `cargo check --tests` until the workspace builds | Keep-compiling facade (below) | +| Delete a large type/module then `cargo check --tests` until the workspace builds | Keep-compiling facade ([agent contract](#keep-the-tree-compiling)) | | Inner loop = `cargo check --tests` after every edit | `--lib` until that crate’s lib is green | -### Keep the tree compiling - -Schema/API rewires stall when a session deletes the old type (`TxIdx`, a -body meta field, …) and then spends the rest of the turn on `unresolved` / -`dead_code` across store tests, query, consensus, and net. That is not TDD; -it is a compile-doom loop. `rbitcoin-store --tests` is a fat rustc unit — -do not use it as the edit cycle. - -| Do | Do not | -|----|--------| -| New type + its unit tests green, **then** a thin wrap on the old API | Delete the old module in the same dirty tree as all callers | -| Switch **one** caller crate per step; `--lib` stays green | One uncommitted tree spanning store + confirm + query + net + docs | -| Keep `--lib` compiling throughout; commit after local CI except coverage | Hours of un-gated WIP, or a slice commit that has not passed those gates | -| `cargo check -p --lib` (or `cargo test -p --lib `) | `cargo check -p rbitcoin-store --tests` or six-crate `--tests` after each edit | - -`--tests` / multi-crate check belongs after Green and in Gates, not in the -inner loop. Commands: -[`.agents/skills/ship-pr/SKILL.md`](../.agents/skills/ship-pr/SKILL.md). - ---- - ## Checklist for authors (and agents) Before accepting a plan: diff --git a/docs/invariants.md b/docs/invariants.md index 06cb79449..42f8928f2 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -23,15 +23,8 @@ with `Err(…Corrupt("invariant: …"))` (and `debug_assert!` where useful). Do **One path.** Do not reintroduce a silent colder alternate when lookup or load was supposed to guarantee the fact. Test-only adapters stay in `*_testutil`; do not grow production APIs around fixture shapes. -**Killed dual paths (examples, not a name list to police):** soft spentness recovery for wrong/missing -pin identity; unpinned wire-corrected create_fk spentness; load-stage `txid.body` -identity fill after lookup promised stamp; `ColdPinMode` Allow/Forbid cold denserels -split on load (load is range **outs** only); denserels-as-spender-abs (schema 22 -abs is `spent` loc off + `8×vout` only); `AssembleMode::Full` / `validate_block_connect` -(confirm is optimistic assemble then `structural_validate_spends`); -`archive_plan_batch_from_store` and production `Query` TxApply→dummy `Block` -(`tx_apply_to_tx` / `connect_block` / `commit_class_a_only` are -`rbitcoin_query::testutil::FixtureChain` only). +Retired dual-path names live in [`errata.md`](./errata.md). The rule is the +sentence above. ## Failure style diff --git a/docs/io-modality.md b/docs/io-modality.md index 6e227be00..ca82f2d79 100644 --- a/docs/io-modality.md +++ b/docs/io-modality.md @@ -176,30 +176,21 @@ IOCP. Ring depth **128** (merge may grow). `RBITCOIN_IO=pread` forces libc. --- -## Historical record: head insert uring ~5× slower +## Head insert locality -| Commit | Note | -|--------|------| -| `0ee28c0` / `77cb2ab` | io_uring bulk / page-grouped RMW for `tx.head` insert | -| **`259b766`** (2026-07-23) | **Reverted to mmap-only head insert.** Host A/B: **io_uring head inserts ~5× slower on head ms/blk** than mmap Release. Bulk uring kept for **reads** only | -| `788936e` | Page-coalesce insert still via **plain map** `write_at` | -| `3a0c220` | **Body** FdOnly success (different pattern: linear append + bulk batch read) | -| `f829090` / `11134cb` | Segmented idx/head landed **after** the 5× failure | - -**Implication:** do not ship head demap without **operator host** A/B (musl -static). Prefer page-coalesced **pread→mutate→pwrite** over per-slot uring. -Segmented heads reduce grow/remap pain but do not free us from page locality. +Do not demap heads without an operator-host measurement (musl static). +A host A/B showed io_uring head inserts about 5× slower on head ms/blk than +a mapped insert. Prefer page-coalesced pread → mutate → pwrite over per-slot +uring. Segmented heads do not remove that page-locality constraint. --- -## End goal (phased) - -1. **FdOnly** for multi‑GiB random tables: `create.loc` → `tx.head` / header head → SH head/body / spenders. -2. **InRam** (explicit process buffers) for small Class C / mempool — not leftover MapFull “because small.” -3. **Remove `memmap2`** from the workspace. -4. Update this doc after **each** phase with host A/B results. +## Settled IO shape -Agent correctness tests under `/tmp` are required; **perf ship/fail is host-only**. +Phase 6 is done: the workspace has zero `memmap2` / `MmapMut` (top of this +file). Multi-GiB random tables stay FdOnly. Small Class C and the mempool +stay explicit process buffers. A host A/B comes before a demap. Agent +correctness tests under `/tmp` are required; perf ship/fail is host-only. --- diff --git a/docs/peer-clients.md b/docs/peer-clients.md index e448bd9ac..b84908dcf 100644 --- a/docs/peer-clients.md +++ b/docs/peer-clients.md @@ -205,8 +205,9 @@ peer**. Copying that would mean reintroducing a coins view. ## Ranked if we spend time later -Do not treat this as Open rank. Promote into [`quality.md`](./quality.md) -only when scheduling a slice. +Not scheduled. Implement a row only when the user names it. Do not treat +this as Open rank. Promote into [`quality.md`](./quality.md) only when +scheduling a slice. | Rank | Item | Source | Lands in | |-----:|------|--------|----------| diff --git a/docs/personal-node-plans/README.md b/docs/personal-node-plans/README.md index cf8f25861..713a975b5 100644 --- a/docs/personal-node-plans/README.md +++ b/docs/personal-node-plans/README.md @@ -1,5 +1,8 @@ # Personal wallet-node plans +**Not scheduled.** Implement a numbered file only when the user names that +plan. Do not start the next file because it is listed here. + Implementation plans for a **home node that cannot open clearnet listen ports**: P2P over Tor SOCKS / I2P SAM / CJDNS, Electrum and Esplora on onion (and I2P when SAM can forward), P2P inbound on a Tor onion, local tx announce on a @@ -11,7 +14,7 @@ is the plan group only. Do not copy step lists into [`quality.md`](../quality.md until a slice is scheduled (same pattern as [`peer-clients.md`](../peer-clients.md) ranked items). -Each numbered file is one plan: [`how-we-plan.md`](../how-we-plan.md) +Each numbered file is one plan: [`how-we-plan.md`](../how-we-plan.md#agent-contract) (Contract / Red / Green / Refactor / Verify). Commands: [`.agents/skills/ship-pr/SKILL.md`](../../.agents/skills/ship-pr/SKILL.md). Do not copy the cycle here. @@ -47,7 +50,7 @@ A node on CGNAT / no forwarded TCP still: the write thread joins) grows — unlikely here. - One product PR per numbered file. Each step is one cycle turn, committed before the next step starts, only after **local CI except coverage** - ([`how-we-plan.md`](../how-we-plan.md); commands in + ([`how-we-plan.md`](../how-we-plan.md#agent-contract); commands in [ship-pr](../../.agents/skills/ship-pr/SKILL.md)). Coverage and native `windows` / `macos` stay GitHub Actions. Optional holistic refactor after the last step, then push and poll. diff --git a/docs/quality.md b/docs/quality.md index 643b5e571..abb990c3f 100644 --- a/docs/quality.md +++ b/docs/quality.md @@ -5,11 +5,11 @@ in [`CHANGELOG.md`](../CHANGELOG.md). 1.0 product gates: [`road-to-1.0.md`](./road-to-1.0.md). Peer-node notes: [`peer-clients.md`](./peer-clients.md) (do not copy here). -**Last reaudit:** 2026-09-17. Schema **24**. Core functional **75** -`test_runner` jobs (**69** inventory `run`; Core expands transport twins and -`wallet_txn_*` flags) / **198** `skip`. Findings **001–023** fixed. Nightly -fuzz **20** jobs. Weekly mutants **8** shards (not required; owner -[`TESTING.md`](../TESTING.md)). Previous: 2026-09-15. +**Last reaudit:** 2026-09-17. Live schema: [`SCHEMA.md`](../SCHEMA.md). +Core functional inventory: [`core-functional.md`](./core-functional.md). +Findings **001–023** fixed. Nightly fuzz **20** jobs. Weekly mutants **8** +shards (not required; owner [`TESTING.md`](../TESTING.md)). Previous: +2026-09-15. | Section | Purpose | |---------|---------| @@ -24,14 +24,16 @@ evidence (failed Core corpus, new dual path, red required CI, MSRV drift). ## Open (priority order) +Counts and procedures stay in the owner doc. A row is the rank and the outcome. + | Rank | ID | Item | Done looks like | |-----:|----|------|-----------------| -| 1 | **Q-41** | Grow Core functional `run` set | Inventory `run` covers claimed wallet-client / P2P / mempool / buried-activation. Labeled `test_runner` prints **77** (**71** inventory `run` / **196** skip; 17 `rpc-missing`, 16 `core-log`, 68 `no-wallet`; Core expands transport twins and `wallet_txn_*` flags). COMPAT leftovers are `rpc-dialect`, not `rpc-missing`. `run` must hit node production (not only shim argv / dummy `blk*.dat` / Core decode dialect). Named Core-functional shim dialect (`RBITCOIN_RPC_WAIT_TIP_IDLE`, `RBITCOIN_RPC_PACKAGE_DIALECT`, proxy abort JSON) is documented, not a silent Core-clone. Next inventory `run`: recover to 72. `mempool_accept` stays skip (`policy-libre`). Unlabeled PRs stay cargo-only. Owner: [`core-functional.md`](./core-functional.md). | -| 2 | **Q-48** | BIP331 rust-bitcoin package types | Native BIP331 `NetworkMessage` when rust-bitcoin exposes it (**RB-007**). Local packages: RPC `submitpackage`, Esplora `POST /txs/package`, Electrum 1.6 `broadcast_package`. `protocol_max` is **1.6**; 1.7 `scriptpubkey.*` still missing. | +| 1 | **Q-41** | Grow Core functional `run` set | Inventory `run` covers claimed wallet-client / P2P / mempool / buried-activation, and each `run` hits node production. Owner: [`core-functional.md`](./core-functional.md). | +| 2 | **Q-48** | BIP331 rust-bitcoin package types | Native package types when rust-bitcoin exposes them (**RB-007**), then the local RPC / Esplora / Electrum package calls. Owner: [`rust-bitcoin-limitations.md`](./rust-bitcoin-limitations.md), [`COMPAT.md`](../COMPAT.md). | | 3 | **Q-31** | Hermetic tip fixtures | Frozen signet/mainnet tip packs for offline consensus/Electrum regression (no live API). Fuzz already merges tiny `signet_block_*.bin` / `mainnet_block_290329.bin`. Electrum hermetic packs still Open. | | 4 | **R-10** | Residual god-files | Peel **only** when a higher row needs a seam. Do not split `interpreter.rs` opcode `match` or io_uring machines. Named extracts: **Q-61** Completed. **Q-56** Completed (scriptnum + pack-ints). **Q-68** Completed (create.loc SIMD). | | 5 | **Q-67** | `asked_blocks` clone on hold | `hold_body` clones `asked_blocks` before `held_bodies` insert so the read lock does not overlap the write (`HeldBodies::insert` already takes `&HashSet`). Bound is `MAX_SERVE_BLOCKS` × peers. Follow-up: pass the read guard with a documented lock order, or keep the clone as a named trade. Owner: `crates/rbitcoin-net/src/chain.rs`. | -| 6 | **Q-69** | CLN / LDK chain backend | Operator can point Core Lightning `bcli` and ldk-node Esplora/Electrum at this node on regtest/signet. Electrum/Esplora **start without** `--sh-index`; SH-only APIs fail closed (`scripthash index disabled`). Channel watches (txid/outpoint) work. BDK address history still needs SH. No LND/ZMQ/BIP157. Owner: [`lightning.md`](./lightning.md). | +| 6 | **Q-69** | CLN / LDK chain backend | Operator can point CLN `bcli` and ldk-node Esplora/Electrum at this node. Owner: [`lightning.md`](./lightning.md). | R-ids were the 2026-08-12 slice. Canonical id is **bold**. Do not start **R-11+**. Next unused Q-id is **Q-70**. @@ -100,7 +102,7 @@ checklist. appender; ibd-confirm publishes script waves (no coordinator). Pins are plan/batch only. [`concurrency.md`](./concurrency.md), [`invariants.md`](./invariants.md). -- **On-disk:** [`SCHEMA.md`](../SCHEMA.md) current bytes (today **24**). +- **On-disk:** [`SCHEMA.md`](../SCHEMA.md) current bytes (live `SCHEMA_VERSION`). Soft-migrate durable side formats; bump or refuse; **no silent wipe**. Same commit as the format code. - **io_uring:** do not flatten a purpose-built machine to batched diff --git a/docs/road-to-1.0.md b/docs/road-to-1.0.md index 4ba136275..275c16e9b 100644 --- a/docs/road-to-1.0.md +++ b/docs/road-to-1.0.md @@ -10,8 +10,8 @@ Day-to-day ranked work stays in [`quality.md`](./quality.md). Electrum/Esplora need `--sh-index` (default off). BIP324 v2-only. Install is a GitHub Release (Linux musl; Windows/Darwin snapshots). Nightly differential fuzz vs Core v31.1 is continuous (**Q-30**). Core functional -inventory is **71** `run` / **196** `skip` (**Q-41**). Findings **001–023** -are fixed. +inventory counts and the `run` bar: [`core-functional.md`](./core-functional.md) +(**Q-41**). Findings **001–023** are fixed. 1.0 is the first **frozen-format, support-windowed** line. Not a Bitcoin Core clone, not a soak badge, not a desktop wallet.