diff --git a/.github/dependabot.yml b/.github/dependabot.yml index d4144ee..55d3cc2 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -15,7 +15,7 @@ updates: # Report only: open-pull-requests-limit 0 turns off version-update pull requests for pyproject.toml's # dependencies, while a vulnerable one is still reported (Dependabot alerts and security updates, once turned on - # in the repository's settings). rnfs relies on RNS and LXMF internals, so a new version is taken deliberately: + # in the repository's settings). rnfs relies on RNS internals, so a new version is taken deliberately: # packaging/update-lock.py, packaging/flatpak/generate-sources.py and tests/test_rns_compat.py (CONTRIBUTING.md) - package-ecosystem: pip directory: / diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index b0b1c7c..15efa75 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -5,6 +5,7 @@ on: branches: ["**"] # not tags: the release workflow runs these jobs itself before publishing pull_request: workflow_call: # from release.yml, which grants no more than contents: read + workflow_dispatch: # by hand, on any branch: Actions > tests > Run workflow, to try macOS and Windows before a PR # Read-only token: these jobs only check out and test the code permissions: @@ -35,8 +36,13 @@ jobs: python -m pip install --require-hashes --only-binary=:all: -r packaging/build-backend.lock python -m pip install --no-deps --no-build-isolation . python -m pip check + # tests/ci_run.py runs `unittest discover tests`, then once more, alone, any test that failed (up to 5): one + # that passes then is named in a warning on the run's page and summary instead of failing the run - name: Unit tests - run: python -m unittest discover tests + run: python -m tests.ci_run + # Windows takes 23-45 minutes as its runners vary (Linux and macOS about 9): 30 and then 45 stopped runs whose + # every test passed + timeout-minutes: 60 # On Windows the daemons are started in their own process groups and stopped with CTRL_BREAK_EVENT. Every # module gated on RNFS_E2E is listed here (test_packaging checks it): the embedded node and the security # regressions that need real RNS included @@ -44,8 +50,8 @@ jobs: env: RNFS_E2E: "1" run: >- - python -m unittest tests.test_e2e tests.test_client tests.test_embedded tests.security.test_h1_real_rns - tests.security.test_m1_real_rns tests.security.test_h2_display_page_pins + python -m tests.ci_run tests.test_e2e tests.test_client tests.test_embedded tests.security.test_h1_real_rns + tests.security.test_m1_real_rns tests.security.test_h2_display_page_pins tests.test_nomadnet_e2e timeout-minutes: 20 - name: Command works run: rnfs --version @@ -70,7 +76,7 @@ jobs: - name: Tests under coverage env: RNFS_E2E: "1" - run: python -m coverage run --source=rnfs -m unittest discover tests + run: python -m coverage run --source=rnfs -m tests.ci_run timeout-minutes: 30 - name: Report run: python -m coverage report --sort=cover diff --git a/.gitignore b/.gitignore index d44c34c..5d2b980 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,4 @@ build-dir/ .coverage .coverage.* htmlcov/ +bandit/ \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index e0caa48..e427b05 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,984 @@ [specification.md](specification.md) is the authority on behaviour. +## 0.2.0 + +The feature-complete release: the local API and the library for companion apps; providing at scale (sweep +providing, provider records on disk within a byte budget, storage tiers against floods); finding content when +identities are generated next to its key (the density check, disjoint lookup paths, the announce first-seen +log); LoRa islands (fast nodes sparing them, two-level records, partition mode); and new features (per-app API +tokens, encrypted folders, NomadNet add, gateway sites, name sync, name key backups, remote pin status, network +status, files inside folders provided by their own CIDs). LXMF messaging moved out of rnfs into an app of its own +(Removed, below). The wire format gained one optional request, `/range` (Added, below), which these nodes send +only to nodes that answer it; nothing else on the wire changed, and CIDs are unchanged: these nodes, 0.1.6, 0.1.5, +0.1.4 and 0.1.3 nodes work together (`tests/interop.py` against 0.1.6 passes). The local API stays compatible but +for LXMF: every route, field and status of 0.1.6 is unchanged but LXMF's, which are gone, and error bodies gain a +field. The upgrade of an `` is automatic and one-way. + +Upgrade notes: +- **State:** the state database gains tables (`pin_times`, `pin_labels`, `name_exports`, `apps`, `block_sizes`, + `routing`, `peer_scores`, `announce_first_seen`, `provider_standing`, `remote_pins_asked`, `remote_pins_served`, + `name_syncs`, `name_sync_files`, `published_heal`, `name_announces`, `remote_pin_retries`, `range_support`, + `publish_items`, `publish_stores`); an older rnfs can't open it afterwards. The publisher's `published` and + `published_earlier` rows move into `publish_items` (a row per provided item), leaving those tables empty. Pins + made before the upgrade have no `added` time, name keys made before it count as never exported (`rnfs status` + lists them until `rnfs name export` is run), and remote pins asked before it have no status entry. +- **LXMF:** rnfs no longer sends or receives LXMF (Removed, below). Pins of links auto-fetched over LXMF change + source from `lxmf:
` to `app:lxmf:
` (`pin ls` shows the new one), an app called `lxmf`'s, so + they stay pinned for an LXMF app to take over; nothing is released by the change, and an older `pins.json` + imported at the upgrade gets the same rewrite. `storage/lxmf_identity`, `lxmf/` and the inbox and outbox in + `rnfs.db` are kept, untouched. `lxmf_*` settings in `rnfs.toml` are ignored with a warning. If you rely on + `rnfs send` and `rnfs inbox`, don't upgrade before an LXMF app is there: the upgrade can't be undone. +- **Provider records:** the upgrade gives each publisher of the records a node stores a standing + (`provider_standing`, and indexes on `provider_records`): Earned if it was first seen at least 3 days before its + latest store and refreshed a record outside the newcomer pool a day or more after storing it, else Newcomer. The + newcomer pool's records are no longer pooled, and the remembered departed and evicted publishers are dropped. +- **`max_records` is retired:** a `max_records` left in `rnfs.toml` (or given to `rnfs.Node`) is ignored with a + warning instead of stopping the daemon; `record_budget` (bytes) replaces it. `rnfs config set max_records` is + refused as an unknown setting; `rnfs config unset max_records` removes it. +- **Blocks:** they move from `blocks/` into 1,024 subdirectories, `blocks//` (the CID's third- and + second-to-last characters), at the first start, which takes a while on a large store (a rename per block; the API + answers once it is done: 6 minutes for 883k blocks on a desktop SSD, and 6 for 857k on a 2-vCPU cloud VM, of which + 110 s the move; the log says how far the listing and the move got about every 30 seconds, and, if they take over 10 + seconds, the listing of every block that follows and the first working out of what the node provides). Each start + moves whatever is left, so an interrupted upgrade loses nothing, and a block Windows won't let rnfs move stays + readable until a later start moves it. An older rnfs doesn't find the moved blocks. A subdirectory of `blocks/` that + is a link, not a directory or another user's stops the start. The state database gains a `block_sizes` table, every + stored block's size, which the first start fills by listing the blocks. +- **`/daemon.json`:** the daemon writes it while it runs. Clients from 0.2.0 on read the API port from + it first, so one that can't read a newer daemon's `rnfs.toml` still finds it. +- **Settings:** `rnfs.toml` may now be rewritten by the daemon (`rnfs config set`, `POST /api/config/set`). Only the + assignments changed are replaced; comments and other lines are kept. Settings whose `api_port` equals a + non-zero `gateway_port` are now refused at start (such a daemon could never listen on both). +- **Routing state:** the state database gains tables for the routing table (`routing`), which is now kept across + restarts, for local scores (`peer_scores`) and for the announce first-seen log (`announce_first_seen`). +- **Files and folders inside pins are provided** (`provide_entries`, below, on by default): after the upgrade a node + that isn't slow also publishes the CID of every file and folder inside its own complete pins (not remote pins or + followed names), up to `provide_entries_max` (100,000), so its first rounds publish more keys. Set + `provide_entries = false` to keep 0.1.6's behaviour. + +Security: +- **A provider can't keep a fetch alive for ever by alternating unreachable and unestablished links** (REL-1): a "no + path" result (retried once, a minute later) reset the count of lost requests, and a link that didn't establish + (lost) re-armed the retry, so a provider whose results alternated the two, as a node ignoring link requests and + answering every other path request gets, never failed, and its fetch, thread and searches lived as long as it + liked. Only an answer, or an outright failure, resets the lost count now. +- **A request for nothing served can't cancel a download** (REL-2, specification section 3): a `/blocks` or `/range` + request took a response slot, displacing a response in progress when the pool was full, before the node asked + whether it serves anything requested. One ~100-byte request for a made-up CID from a fresh link cancelled an honest + slow-link download, and repeating it broke every one. Such a request is now answered at once, with the same answer + as before, without a slot. +- **A request of the wrong shape is dropped before RNS indexes it** (REL-3): RNS reads a request as + `[time, path hash, data]` unchecked, so one whose plaintext was an int, a short list or had a list as its path + raised on RNS's request thread, a traceback in the log for every such packet, before rnfs's per-link rate limit + saw it. The guards now drop it with a throttled verbose line; no handler could have served it. +- **A tar member of negative size is refused by rnfs itself** (REL-4): Python's tarfile parses a negative size from + a base-256 field or a pax `size=` record, and only interpreters with its own offset check (3.12.3 and later fixes) + refused such a stream; rnfs supports 3.11. Sealed folders and tar adds now refuse it on every interpreter. +- **A file in the rnfs dir can't be added through another name for it** (REL-8, completing REV1): the check that `add` + refuses the rnfs dir compared paths, and a file added as the add's root isn't walked (the walks skip the rnfs dir by + inode), so on a case-insensitive disk (macOS: `~/.RNFS/api_token`) or through a bind mount an app could add the + CLI's token, the node identity or a name key and read it back. The file's directories are now compared by device and + inode too, up to the rnfs dir's `files/`. +- **Gateway documents are revalidated, not cached as immutable** (REL-9, specification section 11): every `/rnfs/` + response was `immutable` for a year, though its Content-Security-Policy follows `gateway_scripts`, so a page cached + while scripts were allowed kept running them in the browser after they were turned off. HTML, SVG, other XML, PDF + and listings now carry `Cache-Control: public, no-cache` (the browser keeps them and asks again, a `304` from the + node), and a `304` carries no `Cache-Control`. Other files stay immutable. +- **Withdrawing and restoring the same records spends no write budget** (REL-13, completing REV2): a withdrawal + spent a row of the provider write budget and a restore before the next save spent nothing but gave nothing back, + though neither row is then written. One identity looping over its own 27 records spent the budget other publishers' + new records need, while writing almost nothing, and every other Newcomer's new record was refused (an Earned one's + too, from an Earned attacker). The row a withdrawal spent is now given back when it is undone. +- **A keeper can't make this node keep and serve a name below the sequence it knows** (REL-14): the keeper store's + rollback check, and the sequence a keeper query checks answers against, left out the name floor (the highest + sequence of a name resolved before, kept after its record left the table). A lying keeper's old, validly signed + record of such a name, once kept, was served onward by this node. The floor counts now. +- **A flood of fresh identities can't set the clock-offset estimate** (REL-15, specification section 9): it was the + median of the 128 senders heard most recently, so 128 announces of fresh identities emitted 2 h ahead pushed every + honest sender out, and the node warned that its correct clock was 2 h behind. Once 5 senior servers were heard the + estimate is theirs alone, and the table forgets other senders first. +- **What fuzzing the gateway, the API, settings and tar streams found** (REL-17): a web page could set any + Content-Type, control characters included, with the gateway's `?filename=data:,` (the type is now taken from + the name's extension alone, and a failed response's buffered status line is dropped rather than sent before the + error's); a control character in a request's query reached a redirect's `Location` (percent-encoded now); a request + line of four words ending `HTTP/0.9` was answered without a status line or headers (refused as HTTP/0.9 is); a + deeply nested JSON body answered 500 (400 now); a whole number past 64 bits in a setting raised `OverflowError`, and + `upload_budget = 10**400` passed validation, then stopped the node starting (settings hold 64-bit numbers, as + rnfs.toml does); and a tar member refused by name but declaring 2**88 bytes held a tar add's reader for good, as + tarfile skips it by reading its size and reads nothing for ever at the end (refused as an invalid tar now). Nothing + a remote node sends reached any of these. +- **A `/range` provider that disowns a block it was sending is scored as one that sent a bad block** (REL-20): it could + send ranges of a block, checked only once whole, then answer that it doesn't hold it, so the airtime was spent and it + failed only for that fetch, free to do the same in the next. Its score now falls by what a block not matching its CID + costs; it isn't banned, as content unpinned and collected while it was sent does the same. +- **NomadNet links are bounded per link** (specification sections 3 and 11), with no process-wide change: the answer + may be no larger than `max_download_size` and the free space above the disk reserve; each Resource segment may + decompress to one segment, and must follow the sequence an honest sender keeps (same original hash and total, + the next index, the segment count the total gives), else the transfer is cancelled and the link closed; what is + assembled may not outgrow the advertised total; a file's metadata must be a small flat map, checked before RNS + decodes it; a page must be one flat byte string, checked from its msgpack headers before RNS decodes it. The + daemon's process-wide decompression limit stays 0, and `rnfs.dht` links keep every rule they have. +- **A NomadNet page answer may be at most 16 MiB** (specification section 11): RNS reads an answer without metadata + (a page) whole into memory and decodes it, so a "page" as large as the free disk could exhaust a small node's + memory. One advertised larger is refused at its first segment, before any of it is transferred (the add fails + with too large); files, written to disk, keep the bound above. +- **Gateway sites don't render uploads as pages** (specification section 11, Sites). A site's pages run script in + its own origin, so a file someone else got into a site's DAG could be made to run script there: on site hosts + `?filename=` now only names a download (`Content-Disposition: attachment`; the type comes from the path's own + name), and a file inside the site's DAG whose name gives no type is sniffed only to an image, audio, video, font, + wasm or plain text, else served as `application/octet-stream`, never as HTML, SVG or PDF. A site's `index.html`, + the files its DAG names `.html` or `.svg`, and a site whose root is itself a file render as before; the path + gateway is unchanged. +- **App names stay private** (specification section 11): a request naming an app that doesn't exist in + `X-Rnfs-App` gets an `X-Rnfs-Proof` too, keyed by an HMAC of a secret made at each start and the name, which no + client can check, instead of none; so every answer, the open `GET /api/status` and the 403s included, looks the + same whether or not the app exists, and anyone who can reach the API port can no longer list the installed apps. +- **What an app's token can do, said plainly** (specification section 11, README): an app can do anything the CLI + can, managing apps included in effect. The `/api/apps` routes refuse its token, but it reads and writes files as + the daemon's user, which reaches the CLI's token; `rnfs app add` now says so. Give a token only to software you + would trust with the CLI. +- **Unpacking a sealed folder counts what its entries take on disk** (REV10; specification section 5, + Unpacking): the tar's headers are read first, before anything is stored, and every file and directory counts + at least one filesystem allocation unit (4 KiB or the filesystem's own, if larger) in the block store and + again as written, where the plaintext's bytes were counted before. A sender could seal a folder of many tiny + files that passed the check and then took 4 to 8 times as much, twice, filling the recipient's disk past the + reserve (1,000 one-byte files: 10.9 MB written where 2 MB was checked). Such a get is now refused (413, naming + `--no-extract`). +- **An unpack the process died in no longer leaves the plaintext stored** (REV10): its decrypted blocks used to + stay in the block store, unpinned, until a gc or an eviction. An unpack now marks the state database while it + runs, and a start that finds the mark deletes every block nothing reaches before serving anything (the + unrecorded blocks of a fetch the same crash interrupted too: a retry fetches them again). +- **Documented: a sealed file is only as safe as the recipient's node identity** (REV12), which is online while + the node runs and stored unencrypted in the rnfs dir (`storage/identity`): sealing protects against the + network, not against someone who can read the recipient's disk (README, Known limitations; specification + section 5). +- **`add` refuses the rnfs dir** (specification section 5): `rnfs add ~` pinned and provided the node identity, the + name keys, `api_token` and `rnfs.db`, each findable by its own CID with `provide_entries`, and an app token could + read the CLI's token through `add`. `add` (the CLI, the API, add jobs, sealed folders) now refuses the rnfs dir, a + directory holding it, anything inside `storage/`, `names/`, `state/`, `blocks/` or `lxmf/`, and anything else in + the rnfs dir but outside `files/`. Every walk (`add`, the shared folder, `name sync`, sealed folders) also skips, + with a warning, a directory that is the rnfs dir or one of those subdirectories by device and inode, which catches + a bind mount of them; one given as the root is refused. +- **Provider records have a write budget** (REV2; specification section 6, "Provider records"): with no + per-publisher cap left, one identity storing and withdrawing records in a loop made the store write about + 100,000 rows a second in-process, and 9.5 MB/s at 100 requests a second. New records and withdrawals now spend a + budget of 60 rows every 30 s open to all, plus 30 each kept for Earned and Known publishers, carried over for up + to 6 hours; past it new records are refused (`False`), withdrawals never are, and refreshes cost nothing. A + withdrawal followed by a store of the same key before the next save is a refresh, and a refresh is written once + it moves a record's last-stored time by 6 hours rather than 1. A Newcomer flood now makes the store write at most + about 150 kB/s (fresh identities, inserted and withdrawn, on a 1M-record database), a hundredth of what it did. +- **A full key can't be taken over** (REV3; specification section 6): Earned publishers never gave way to each other + at a full key, so about 1000 identities that each refreshed one key on 3 UTC days (from about 48 hours) held all + its 1000 publishers, and every new provider of it but Known ones was refused. Now Earned records give way to + Earned and Known storers (a random one outside the `K/2` oldest), and Newcomers keep a tenth of the key: below it + a Newcomer evicts a random Earned record, past it Newcomers give way first, so a flood of fresh identities holds + at most that share. +- **Provider standings without records are capped** (REV11; specification section 6): a publisher left without + records keeps its refresh days a week, and nothing else bounded those rows, so identities that each refreshed one + record a day later and withdrew it grew the state database (and took room from the record budget, which measures + it). At most 50,000 are kept (about 5 MB); past that the hourly pass forgets those with the fewest refresh days + first, so Earned publishers that left outlast such a flood. +- **Partition mode can't be forced from outside** (spec section 6, "LoRa islands"; REV4): a server beyond the island + could answer a slow node's `/find` with 3 made-up IDs, whose failures entered partition mode at once. Only failures + of servers the node knows (its routing table or first-seen log, or an identity or path RNS knows) count, and 3 in a + row ask for a probe of an off-island server that answered or announced itself recently, as silence does: partition + mode starts only if that probe fails (or nothing beyond the island was heard to probe). An ID that only ever failed + is never a probe target. +- **The announce first-seen log keeps its senior servers** (spec section 6, "Announce first-seen log"; REV7): it + forgot the least recently heard server first, so about 3 announces a second of fresh identities pushed every honest + server out between its 6-hourly announces, and none ever became senior (`provider_seniority`), which publishing and + searching's senior rule needs. When full it now forgets first a server silent for 2 × `announce_interval` (counted + from the start at the earliest), then the server not yet senior that it first heard most recently (in a flood, the + newcomer itself), and a senior server heard lately only when nothing else can go. At a tenth of the scale (6,553 + rows, 200 honest servers, a fresh identity every 3.3 s for 2 days) every honest server became senior, against none. +- **Made-up IDs can't inflate the network size estimate** (spec section 6, "Identities placed next to a key"; REV8): + a lookup took its sample, the distance to the 10th-closest server, over every ID replies named, so a server naming + made-up IDs just beyond the 3 closest (which a lookup completes without asking) inflated the estimate about 3 + times, weakening the density check. A sample now counts only servers that answered or whose identity is known, + from the closest up to the first that isn't (its distance scaled to the 10th), so a node that knows few identities, + on a LoRa island, doesn't underestimate the network instead. Over 300 servers with such a server in every lookup: + 288 -> 743 before, 288 -> 318 now. +- **RNS's identity blackhole is honoured** (specification section 10, "Blackholed identities"): RNS drops a + blackholed identity's announces and paths, but rnfs kept using what it held, so the node stayed in the routing + table, was asked in lookups, and its provider records were handed out and fetched from. The node now reads the + list (`rnpath -B`, blackhole sources; through rnsd too) every 5 minutes and treats a listed identity's node ID as + banned while it is listed (nothing of it is saved: it ends when the identity leaves the list), and answers `False` + to every request on a link identified as a listed identity. + +Removed: +- **LXMF messaging left rnfs** for an app of its own, built on the local API and the library (roadmap "LXMF moves + out of core"): rnfs is the core (content, the DHT, names, pinning, the local API, the library), and messaging is + a companion app's. Gone: the commands `rnfs send`, `inbox` (`inbox get`, `inbox rm`), `outbox`, `lxmf address` and + `lxmf sync`; the routes `GET /api/lxmf/address`, `POST /api/lxmf/send`, `GET /api/lxmf/inbox`, + `POST /api/lxmf/inbox/get`, `/api/lxmf/inbox/rm`, `GET /api/lxmf/outbox` and `POST /api/lxmf/sync` (now 404 + `no_such_endpoint`); `lxmf` among `GET /api/capabilities`' features; the `lxmf_address`, `lxmf_inbox` and + `untrusted_dropped` fields of `GET /api/status` and `rnfs status`; the `lxmf` dependency, from `pyproject.toml`, + the lock and every package (the `.deb`, the Flatpak, the Windows exe, the install script). This is a deliberate + break of the local API for LXMF only: every other route, field and status of 0.1.6 is unchanged. The + library's `lxmf` methods, their objects and the `inbox` events, added during 0.2.0's development, never shipped, + nor did the deprecation path first planned for this release (warnings, a `Deprecation` header). +- **What stays:** nothing on the network changes, and `rnfs:` URIs and the LXMF link map (the field an LXMF message + carries rnfs links in) stay part of the specification (section 13) as the contract an LXMF app and other clients + use; `rnfs.Node` and `rnfs.Client` still take an `rnfs:` URI as a target (`rnfs/uris.py`). Contacts of kind + `lxmf` stay in the address book, for an app to read which senders are trusted. On disk, nothing is deleted: + `storage/lxmf_identity` and `lxmf/` are left as they are (rnfs no longer creates or checks `lxmf/`), so an LXMF + app can adopt them and keep the node's LXMF address; the inbox and outbox stay in `rnfs.db`'s `lxmf_messages` + table (0.1.4's `lxmf_inbox.json` and `lxmf_outbox.json` are still imported into it), which nothing reads, + changes or deletes, for the app to import. +- **Pins of links fetched automatically over LXMF** stay pinned, as `app:lxmf:
` (upgrade notes, above): + an app called `lxmf` holds them, and nothing else releases them. An LXMF app registered under that name + (`rnfs app add lxmf`) takes them over; to release them without one, run `rnfs app add lxmf` then + `rnfs app rm lxmf`. +- **Retired settings:** `lxmf_enabled`, `lxmf_display_name`, `lxmf_propagation_node`, `lxmf_send_from_hint`, + `lxmf_sync_interval`, `lxmf_accept_from`, `lxmf_auto_fetch_from` and `lxmf_auto_fetch_quota`. One left in + `rnfs.toml` (or given to `rnfs.Node`) is ignored with a warning, as `max_records` is, instead of stopping the + daemon; `rnfs config set` refuses one as unknown, and `rnfs config unset` removes it. +- **If you used `rnfs send` or `rnfs inbox`:** use an LXMF app built on rnfs once one is released. Meanwhile an + `rnfs:` link can go through any LXMF client (Sideband, NomadNet, MeshChat) as text, and the recipient fetches + `rnfs:/?from=` with `rnfs get / --from `. If you rely on rnfs's own inbox + and auto-fetch, stay on 0.1.6 until then: an `` 0.2.0 has upgraded can't go back to it. + +Added, for apps: +- **App tokens:** an app can have a token of its own, kept across restarts (`rnfs app add`, below), sent as + `X-Rnfs-Api` with `X-Rnfs-App: `; the proof a client checks before sending it is keyed by the token's + verifier, which is all the daemon keeps of it (`rnfs.db`'s new `apps` table). `rnfs.Client(app=..., token=... + or token_file=...)` and `ApiConnection(app=, token=)`. The `api_token` stays the CLI's, unchanged, and a + request without `X-Rnfs-App` is checked exactly as before. +- **Managing apps:** `rnfs app add [-o file]` issues an app a token, printed once (alone on stdout, for + scripts) or written to a new file only you can read, and says on stderr that the app can do anything the CLI + can (export name keys, change settings, stop the daemon), managing apps included in effect; `rnfs app ls [--json]`; + `rnfs app rm ` revokes the token and releases every pin the app made. Routes `GET /api/apps`, + `POST /api/apps/add`, `POST /api/apps/rm`, for the CLI's token only (an app's gets 403 with the new error code + `forbidden`); capability `apps`. An app called `lxmf` holds the `app:lxmf:` pins rnfs's own LXMF left + (Removed, below). +- **An app's pins:** what an app adds, pins, imports or makes with `dir set`/`dir rm` (and its pin, add and import + jobs) is pinned with the source `app:`, or `app::` with `X-Rnfs-Pin-Key: ` + (`client.pin_key(key)`), shown by `pin ls`. Its `pin rm` releases only its own source; the CLI's still + releases only `user`'s, and a pin key with the CLI's token is refused (400). Removing the app releases them all + and cancels its jobs. +- **Discovery:** `/daemon.json`, written atomically (0600) once the API listens and removed at stop: the + daemon's PID, API and gateway ports, version, start time, rnfs dir, Reticulum config directory, whether it + attached to a shared instance, and node ID. `rnfs.apiclient.read_daemon_info()` reads it, refusing a symlink, + a file others could have written, or one whose process has ended. +- **Stopping:** `POST /api/shutdown` stops the daemon as SIGTERM does, after answering; `rnfs stop [--wait S]` + uses it and waits until the API stops answering. Windows has no other clean way to stop a background daemon. +- **Error codes:** every API error is `{"error", "code"}`, the code machine-readable (`rnfs.errors.CODES`): the + node's errors by status (`bad_request`, `not_found`, `too_large`, `denied`, `invalid_dag`, `busy`, + `storage_full`, `internal`) and the API's own refusals (`no_such_endpoint`, `method_not_allowed`, + `unauthorized`, `bad_host`, `timeout`, `length_required`, `body_too_large`, `too_many_connections`, + `unsupported`). So a 404 for an endpoint a daemon lacks (`no_such_endpoint`) is told from one for missing + content (`not_found`). `ApiError.code`, and `NoSuchEndpoint` and `Unsupported` (both `NotFound`s) in the + library, which also recognises an older daemon's unknown-endpoint 404 by its message. +- **No API stability until 1.0.0:** the library (`rnfs.Node`, `rnfs.Client`) and the local API stay provisional + (`rnfs.API_STATUS`), and may change in 0.2.x releases; `GET /api/capabilities` says what a daemon offers. +- **An OpenAPI reference for the local API:** `docs/openapi.json` (OpenAPI 3.1) describes every route, its + parameters and bodies (streamed ones too), its answers, the error codes, the token headers and the event + kinds, for apps in any language; tests check it against the code (routes, codes, kinds, + capabilities, and the fields that live answers carry). It is in the sdist too. +- **Capabilities:** `GET /api/capabilities` lists the daemon's routes, features, event kinds, error codes and + capabilities (`events`, `pin_details`, `cache`, `config`, `jobs_add`, `add_tar`, ...). +- **Events:** `GET /api/events?after=&kinds=&epoch=&wait=`, a long poll on the node's events (`rnfs/events.py`), + replacing an app's polling of each list: `job` (every state change, and a running job's progress at most once + a second), `pin` (added, removed, labelled), `cache` (added, removed), `name` (a newer record of a name the node holds + or watches) and `peer` (a peer first heard, coming online, going offline). Events are numbered within a run; + the last 1024 are kept, and a client that falls behind, or asks of an earlier run (`epoch`), is told `lost` + and reloads. At most 8 polls wait at once. +- **Pins:** `pin ls --json` and `GET /api/pin/ls` give each pin's `label`, `origin` (where it was added from, + kept apart from the label), `size`, `entries` (a directory's entry count, as a number) and `added` (when first + pinned). `rnfs pin label [label]` / `POST /api/pin/label` give a pin a label, which becomes its listed + name. `name` is otherwise unchanged. +- **The cache:** `rnfs cache ls [--json]` / `GET /api/cache/ls` list the cached fetches (downloads not pinned), + most recently used first, with size, completeness and whether the same content is pinned or provided; + `rnfs cache rm …` / `POST /api/cache/rm` evict those leading to a CID now. Apps no longer need to look in + `blocks/`. +- **Settings:** `GET /api/config` gives every setting with its schema (type, choices, default, unit, range, + description, group, whether it applies without a restart), the running value, the value in `rnfs.toml`, and + whether the command line overrides it or a restart is pending; `POST /api/config/set` changes `rnfs.toml` + (validated as a start would, nothing written otherwise; `null` removes a setting), keeping its comments and + other lines, and applies at once what can be. `rnfs config [list|get|set|unset]`. `rnfs.config.settings_schema()` + needs no daemon. Settings that now apply without a restart: `max_storage` (enforced at once), + `max_download_size`, `max_output_entries`, `max_upload_rate`, `remote_pin_quota`, `follow_quota`, `max_jobs` + (queued jobs start if the new limit allows), `max_finished_jobs`, `serve_interfaces`, `no_serve_interfaces`, + `quiet_hours`, `keep_max_names`, `keeper_query` and `scan_interval`. +- **Add and import as jobs:** `POST /api/jobs` takes `kind` `add` (`path` and `POST /api/add`'s options) and + `import` (`path`), with progress (`total_bytes` to read, `done_bytes` read, `rate`, `eta`), cancellation at + the next read, and the result `POST /api/add` / `/api/import` would give; persisted, and run again from the + start if interrupted. `rnfs add --background`, `rnfs import --background`. +- **Streaming, so a daemon that can't see the app's files (Flatpak) works fully:** `POST /api/add/stream?format=tar` + (a tar stream becomes a directory; links, devices, absolute or `..` paths and invalid names are skipped with + warnings; bounded by `max_output_entries` and the DAG depth limit; extended headers over 64 KiB, global headers + over 4 KiB in all, data after the tar's end, and compressed or malformed tars refused), + `GET /api/get/stream?format=tar` (a target as a tar, fetched first, every limit checked before the response + starts, with an exact length; files as stored; entries whose names hold `\`, `＼` or `／`, or start as a Windows + drive does, `C:`, are skipped and counted), + `GET /api/export/stream` (an archive, byte-identical to `rnfs export`) and `POST /api/import/stream`. +- **Peers:** `GET /api/peers` and `rnfs peers` gain `first_heard`, `rtt` and `rtt_at` (the round trip RNS measured + on the last link to the peer, `ping` included) and `online`, worked out by the daemon: an rnfs link is open, or + within 15 minutes of its last sign of life — an answer, a request served, a link, or an announce emitted within + that time (a path answer, or an announce relayed late, doesn't count). Decided on the monotonic clock, so a + wall-clock step changes nothing. + `rnfs peers` gains RTT and ONLINE columns. +- **rnfs.Client and rnfs.Node:** typed methods for what was only reachable through `Client.api`: `peers()`, + `stats()`, `contacts.*`, `names.keep/unkeep/kept` and `names.follow/unfollow/following`, with objects (`Peer`, + `Stats`, `Contact`, `Kept`, `Followed`); `jobs.add` and `jobs.import_archive`; and the additions above (events, + capabilities, pin labels, the cache, settings, streams). A Node raises what a Client gets back. +- **rnfs links in the library:** `rnfs.Link`, `rnfs.parse_uri`, `rnfs.format_uri`, and the LXMF link map's + `rnfs.encode_field` and `rnfs.decode_field` (spec section 13), for an LXMF app and other clients that carry rnfs + links, so they need no internal module (`rnfs/uris.py`). They need no node and load no RNS. +- **Starting:** while a daemon opens its block store, before its API listens, `rnfs status` and the other commands + say it is starting, naming its process, rather than that no daemon is running; the library raises + `DaemonStarting` (`.pid`), a `DaemonNotRunning`, so code catching that is unchanged. Told by the rnfs dir's + lock, which the daemon takes first (`apiclient.dir_holder`). +- **Per-call timeouts:** `with client.timeout(seconds):` bounds the calling thread's requests only, long polls + fitting inside it (a Node accepts it and does nothing); `ApiConnection.request/call/get/post(timeout=)`. +- **Gateway sites, an origin per root:** `http://.rnfs.localhost:8789/` serves `/` and + `http://.name.localhost:8789/` the name's target, so a page can `fetch()` files of its own DAG. + `` is the CID as `k` + base36 (56 or 60 characters: an rnfs-dag CID's usual form is too long for a + DNS label), accepted in these hosts only (`rnfs.content.cid.to_base36`); ETags, listings and logs still show + the usual base32 CID. Site responses keep the page's origin (`sandbox allow-same-origin …`), can be framed + only by their own origin, and carry `Origin-Agent-Cluster: ?1`; service worker scripts (403) and `?format=` + (400) are refused there, and paths are DAG paths only. Browsers resolve `*.localhost` themselves; other tools + should keep using the path gateway, which is unchanged. All `*.rnfs.localhost` sites share cookies (as do all + `*.name.localhost` sites): neither the gateway nor the API uses them. `rnfs url |[/path]` prints a + target's site URL. + +Added: +- **Name key backups:** losing a name's key loses the name for good. `rnfs name create` and `rnfs name import` + say so and print the `rnfs name export` command (the API's results carry it as `warning`); the node remembers + which keys were exported by `name export` or imported, and `rnfs status` (`GET /api/status`, + `Status.unexported_names`) lists the names whose key never was. `name list` entries and `rnfs.Name` gain + `exported`. +- **Clock problems are noticed:** RNS drops an announce emitted (by its sender's clock) no later than the one it took + a path from, so a Raspberry Pi without a clock that came back on an older time lost its name updates silently, for + up to a week. The node now remembers each name's last announce (`name_announces`) and holds the name's announces + while its clock is behind it: a publish is committed as usual, its result says why the announce is held + (`announce_held`, printed by `rnfs name publish`, a `RuntimeWarning` from `Name.publish`), and the announce goes + out once the clock passes the last one, or after a week held in all. `GET /api/status` (`Status`) gains + `announces_held` and `clock_offset`, the clock's offset from the network's estimated from the emission times of + the announces heard (the median over senders, so a few wrong clocks don't move it); a warning is logged, and + printed by `rnfs status`, while the clock is over 5 minutes behind. Only behind is reported: RNS delivers many + announces late (it holds those arriving on an interface that just came up), which looks like a clock ahead, so + `clock_offset` reads 0 then. The estimate never changes the clock or what the node does. Name syncs publish + through the same path. +- **Pin by NomadNet address:** `rnfs add :/page/` (a page as the node renders it) or + `:/file/` (a file it serves) fetches it from that NomadNet node over a link of rnfs's own and adds + it like a file, pinned `user` (with an app's token, that app's source) with the link as its origin in `pin ls`; the + usual `add` options apply (`--no-pin`, `--wrap`, `--name`, `--to`, `--chunker`, `--only-hash`, `--background`), and it + runs as an `add` job with progress. Each run takes a new snapshot. The link is anonymous unless `--identify`, which + identifies with an identity of its own (`storage/nomadnet_identity`, made on first use), never the node's; `rnfs + status` (`GET /api/status`, `Status`) shows its hash, `nomadnet_identity_hash`, for the NomadNet node's operator to + list, making it if there is none yet. A page the node answers with its "Request Not Allowed" page fails rather than + being pinned, naming that hash when the add identified. A local file named like such a link + is added as `./`. `POST /api/add` takes `nomadnet` (with `name` and `identify`) instead of `path`, as do add + jobs (`POST /api/jobs`); capability `add_nomadnet`; `rnfs.Node`/`rnfs.Client` `add_nomadnet()` and + `jobs.add_nomadnet()`. Request data (a page's fields or `var_` values) isn't supported. +- **Encrypted folders:** `rnfs add --to … ` (and `--to-self`, `--background`, `POST /api/add` with + `to`) seals a folder as one sealed file: the folder is packed as a tar, everything under one top directory + `/`, and sealed under the name `.tar`, so its file names and structure are hidden along with its + content (not its approximate total size). The sealed-file format is unchanged: a recipient on 0.1.x gets a tar it can + unpack itself. The tar starts with a marker, a pax global header holding only the POSIX `comment` `rnfs sealed + folder`, which extractors ignore. The folder is walked as `add` walks one (symlinks, devices and invalid names skipped + with warnings; names holding `\`, `＼` or `／`, or starting as a Windows drive does, skipped too); more than + `max_output_entries` entries or a tree deeper than `MAX_DEPTH` is refused, and so are a folder name over 251 bytes and + `--no-sealed-name` with a folder (the name is what makes a recipient's `get` unpack it). +- **Unpacking sealed folders:** `rnfs get` of a sealed folder (a sealed file whose sealed name ends in `.tar` and whose + content is a valid tar starting with the marker) unpacks it, by a recipient. A tar sealed as a file (by 0.1.x, or + `rnfs add --to bob backup.tar` now) has no marker and is still written as that file, byte for byte; `GET /api/info` + gives a recipient `sealed_folder`. The decrypted tar is read with `POST /api/add/stream?format=tar`'s rules (links, + devices, absolute or `..` paths skipped; `max_output_entries` and `MAX_DEPTH`) and written with `get`'s (temporary + name, unsafe names skipped). Without `-o` the output is named after the tar's top directory. The free-space check + counts the content twice (its blocks, then the files), and the decrypted blocks are deleted afterwards, never pinned + or provided. A marked `.tar` whose content isn't a valid tar is written as the file, with a warning; one past a limit + is refused. `rnfs get --no-extract` writes the `.tar` instead; `POST /api/get`, `POST /api/jobs` (kept with the job), + `Node/Client.get` and `jobs.get` take `no_extract`, and `Written` gains `extracted`. `POST /api/get` without + `no_extract` unpacks a sealed folder; content sealed before 0.2.0 is written as it always was. `GET /api/capabilities` + lists `sealed_folders`. `cat` still gives the decrypted tar, and the gateway and `--raw` the sealed bytes. `POST + /api/add/stream?format=tar` still can't seal (400). +- **Remote pin status** (local only; `/pin` and the wire are unchanged): a node records the remote pins it asks + for (node, CID, pin or release, when, whether accepted) and works out whether each accepted one is done by + looking the node up among the CID's providers: paced (in the background at most 60 lookups an hour, 4 at + once, each pin hourly while waiting and daily after; on slow nodes 4 an hour, one at a time), waiting for up + to 24 hours after asking, then missing if not found. So "done" lags the node's publishing and is only as good as + the DHT. A node serving remote pins records what became of each (waiting, fetching, pinned, dropped with why: + quota, not found, denied, ...; refused because denied or its queue was full; released), which was only logged. + At most 10 000 asked entries and 1 000 served outcomes are kept. `remote_pin` events report both. + `rnfs pin remote ls [NODE_ID] [CID] [--check] [--json]` lists the pins asked for (`--check` looks them up now: + one lookup each, at most 32 a call, 4 on a slow node); `rnfs pin remote ls --served [--json]` lists those served + and each identity's use of `remote_pin_quota`. `GET /api/pin/remote?node=&cid=&check=1` and + `GET /api/pin/remote/served`; `ApiConnection.pin_remote_ls` / `pin_remote_served`; `Node/Client.remote_pins()` + and `served_pins()` (`RemotePin`, `ServedPins`, `ServedPin`, `RemotePinUsage`); capability + `remote_pin_status`. +- **Publishing a folder to a name** (local behaviour built on `name publish`; nothing on the wire changes): + `rnfs name sync ` keeps a name pointing at a folder's current content. Every `scan_interval` the + folder is imported as the shared folder is (unchanged files recognised by their metadata, not read again; links, + special files and files with other hard links left out; a folder in the rnfs directory refused), with FastCDC, + so an edited file shares most of its blocks with the version before. When its root changes the name is + published to it, as `name publish` does (the previous version's `name:` pin released), once the folder has + stayed unchanged a minute and at most once an hour per name (RNS's transport nodes stop rebroadcasting a + destination announcing faster than that). A folder that is missing, unreadable, empty or whose import fails + pauses the sync, logged once, and nothing is published. `rnfs name unsync ` stops it, the name keeping + its target; `rnfs name rm` stops it too; `rnfs name syncs [--json]` lists the syncs and how they stand. Syncs + are kept in the state database and resume after a restart. In private mode `name sync` warns, as `name publish` + does, and the sync logs it once. `POST /api/name/sync {"alias", "path"}`, `POST /api/name/unsync {"alias"}`, + `GET /api/name/syncs`; `ApiConnection.name_sync`, `name_unsync`, `name_syncs`; `Node/Client.names.sync`, + `unsync` and `syncs()` (`FolderSync`); capability `name_sync`. An app's token may start and stop a sync, as it + may publish; the name's pin stays the name's. +- **Files and folders inside pins are provided** (new settings `provide_entries` and `provide_entries_max`; an + entry is an ordinary provider record of its CID): every file and directory a directory entry names inside a + complete pin of the user's own content (the user's, the shared folder's, names' and apps' pins: not those other + nodes asked for or followed names'), at every depth, can be found by its own CID rather than only as + `/`, and the same file in two folders from either. Not chunks or other intermediate nodes, and never + inside cached downloads, which provide only their root; serving is unchanged (an entry's blocks are its pin's). + Unset (the default), it is on up to `provide_entries_max` entry keys (100,000): pins oldest first, each pin's + entries all or none, a pin that would pass the cap left out (logged). On a slow node + it is on only while republishing every key is estimated to take at most 2% of the channel (from the slowest + interface's bitrate, the keys, and the network size the publish rounds' regions show: about 1,300 keys at 100 + servers and 40 from 1,000 at 1 kbit/s), and none are provided past that; once on, they stay on up to 2.5%. + `true` or `false` overrides the default; both settings apply at once when changed through the API. Unpinning a folder + withdraws its entries, except those still inside another pin. `rnfs status` (and `GET /api/status`, + `Status`) gains `provided_entries`, `provide_entries_off` (why none are provided) and `provide_entries_capped`; + `provided` counts entries too. Providing entries tells anyone looking up a file's CID that the node has it. +- **Network status** (spec section 11, "Network status"): `rnfs status` (`GET /api/status`, the library's + `Status.network` and `Status.record_store`) reports what the node sees of the network, with nothing sent to learn + it: `network` gives the number of DHT servers the density check estimates (`size_estimate`, null until 8 lookups + completed; `known_nodes` is the routing table's, to compare) and how many searches and published keys the check ran + on and fired on in the last hour and day; `record_store` gives what counts against `record_budget` (the state + database's size), the records the store would hold when full, whether it is full, and for each storage tier + (Known, Earned, Newcomer) its records and share against its floor and the new records refused and records evicted + in the last hour and day. `rnfs status` sums these up in + a few lines after the other fields (`network_size`, `density_check`, `partition_for` while in partition mode, + `record_store`, `record_tiers`, `records_refused`, `records_evicted`); every existing field and line is unchanged. + The counts are kept in memory (per minute over the last day; a restart starts them afresh), and the status reads + nothing more from the state database, so apps can call it as often as before. + +The DHT and provider records (local behaviour: nothing on the wire changes): +- The routing table is saved and loaded at start, so a restart doesn't make every server new again: servers keep + their seniority (which ranks providers and keepers), bans last their time, and `/find` replies still leave out + servers silent for over 2 × `announce_interval`, the time the node was down included. +- Local scores: each node keeps a private score per peer, earned by work it checks itself (blocks served that + verified, `/find` answers that led to providers who served or to closer servers that answered) and lost by + failures, bad blocks and silence (scores halve each week). Bootstrap nodes start with a small seed they keep + only by earning it. Nothing is sent; at most 4096 peers are scored. +- A join (at start, and again while too few servers are live) asks up to 6 servers of the saved routing table + first, best scored first, and bootstrap nodes only if fewer than two of those answer: a node that has joined + once is its own bootstrap list, so a restart doesn't flood path requests for bootstrap nodes on radio. After + such a join, a node that isn't slow still asks the default bootstrap nodes (until one answers, at most 2), so + one that saved only an island of servers reaches the rest of the network; a slow node no longer does. +- Choosing providers: within each seniority group (seniority stays first), providers with a better local score + are tried first, before fewer hops: a provider whose blocks verified before is preferred to an unknown one. +- The announce first-seen log: each node records when it first heard each server's announce (at most 65,536 + servers, about 12 MB when full: servers silent for 2 × `announce_interval` forgotten first, then the newest not + yet senior, Security above), kept across restarts, and can give + the closest servers to a key among those heard for at least `provider_seniority`, which identities generated + next to a key can't be. Publishing and searching use it (below). +- **Sweep providing:** republishing many keys takes far fewer lookups. One closest-nodes lookup now serves every + due key of its region (the keys sharing as many leading bits as the key looked up shares with its 10th-closest + server), with about 1.6 more `/find` a region to fill in its far half; each key still gets its own 10 closest + servers (100% of keys on their true 10 closest over a simulated 2000-server network, against about 97% with a + lookup per key). A round needs about servers ÷ 12.5 lookups however many keys are due: 1,000 keys on a + 1,000-server network take about 80 lookups instead of 1,000. Slow nodes' 10 lookups a round now count regions, + not keys, so a round publishes every due key of up to 10 regions. +- **Sweep providing on LoRa islands and at a first start:** a node that knows few identities (on an island, or just + started) found a key's 10 closest among them, padded with servers far from the key, so its regions held many keys + its lookup knew little about. Regions are now never shorter than the network size estimate implies, a key a + lookup until there is an estimate, and a region covers only the keys it can store on all but 2 of their 10 closest + servers it heard of (it heard of others whose identity it doesn't know: the others get lookups of their own). Over + test_topology's world of backbone and islands (1,000 / 3,000 servers), an island node's records reach 7.0 / 4.7 of + their true 10 closest instead of 3.8 / 2.3 (a lookup per key: 7.8 / 5.0), and a node's first round after its first + start 6.5 / 5.0 instead of 0.3 / 0.1, its second 7.8 / 6.9 instead of 1.9 / 0.7. It costs island nodes keys per + round (1.5 / 1.2 keys a region lookup instead of 6.6 / 4.9, so a slow island node's round of 10 lookups publishes + fewer keys); a backbone node, which knows every identity it hears of, runs as many lookups as before + (`tests/bench_publisher.py`, 2,000 servers: 159 for 100,000 keys and 176 for 1M, against 158 and 176 without the + size estimate). +- **Stores to long-known servers too:** each key is also stored on the 10 servers closest to it among those the + node first heard at least `provider_seniority` (1 day) ago and has heard in the last 2 × `announce_interval`. + Without an attack they are mostly the key's 10 closest already, so this adds few stores; identities generated + next to a key can't push the record off them. Withdrawals reach them too. +- **Content is far harder to hide by placing identities next to its key** (spec section 6, "Identities placed next + to a key"). Ten identities generated closer to a key than every honest server (about 4 s of one core each among + 10,000 servers) hid it almost completely from 0.1.x nodes (found 0–4% of the time); with an upgraded publisher + and searcher it is found 99–100% of the time over the fake network of `tests/test_eclipse.py` (98.9% on fast + nodes and 90.8% on slow ones across every attack of the eclipse simulation). Four local mechanisms, nothing on + the wire changed: + - a network size estimate: the median distance to the 10th-closest server over the node's last 16 complete + lookups for other keys, counting only servers the lookup verified (Security above), kept in the state + database's `meta` table so a restart has it at once (the density check waits for 8); + - the density check: when the servers found near a key are packed more than twice as densely as the estimate + allows (from the 10th on), a search waits until 5 servers of each of two rings beyond them have answered (those + ranked after the dense core, and those beyond where 10 honest servers would be expected), and a publisher gives + the key a widened lookup of its own and stores it on 10 servers of each ring too; + - disjoint lookup paths: a publisher's region lookup runs as 3 lookups that share no node (2 on slow nodes) and + stores each key on each path's 10 closest too; a search that finds no provider at all falls back to them, in + what is left of the same `lookup_timeout` and of its 80 `/find` (a search still sends at most 80 in all); + - searches also wait for the 3 closest of the 10 servers nearest the key that the announce first-seen log heard + at least `provider_seniority` ago (none extra without an attack: they are the closest anyway). + + The cost: a little more traffic per publish (over the fake network of 2,000 servers, +17 `/find` and +3 stores + to publish one key, once per republish; with sweep providing about 0.65 `/find` and 13 stores a key instead of + 0.22 and 10), +0.2 to +0.5 `/find` per search on average (the density check fires on 2–6% of searches without an + attack), and a search for content nobody provides now costs about 4 times as much (30 `/find` instead of 7: the + fallback). A slow node paces that fallback like its publish lookups (it counts as 2 of a round's lookups and runs + only if the pacing allows within the lookup's time), and counts widened lookups among its round's 10 (a flagged + key that doesn't fit waits for the next round). +- **Fast nodes spare LoRa islands most lookup traffic** (spec section 6, "Slow paths"; nothing on the wire changes): + a node that isn't slow asks servers behind a slow path (an RNS path of more than 3 hops, or its own last link to + the server within 24 hours had a round trip above 1 s and a rate estimate below `slow_bitrate`) after every other + server but unreliable ones, in searches and publish lookups, and stores to them only to make up 10. They are still + asked when a lookup has too few others; the routing table, `/find` replies, provider ranking and failure counts are + unchanged. In the topology simulation this cut the airtime internet nodes' lookups cost islands by 97% (27 s to + 0.6 s each; a 30-server island needed 135–141% of its channel for light internet use before), and internet nodes + still found island content 98% of the time. Over the fake network of `tests/test_topology.py` (1,000 servers, + 21% on islands): island `/find` per internet search 0.76 to 0, island stores per internet publish 2.5 to 0.07. A + server whose path was unknown is judged again once the query's path request is answered, before a link is opened. + `rnfs peers` is unchanged; the traffic counters now keep each peer's last link's rate estimate too (in memory). +- **Slow nodes place records better:** a slow node's lookups may make more path requests than its `alpha` (1): up to + 10 per publishing lookup (its 2 disjoint paths 5 each, and a widened lookup 10) and 3 per search (and 3 more over + the fallback's paths). A slow node hears only some servers' announces, so a publish lookup stopped at the closest + servers it knew: in the topology simulation an island publisher's records reached 1.3–2.7 of their 10 closest + servers, and internet nodes found island content 68–91% of the time; with the larger budget 3.7–4.4 and 99%, for + 11–19% more airtime per publish. Over the fake network of `tests/test_topology.py` (1,000 servers): 3.6 to 6.9 of + the 10 closest. Nodes that aren't slow are unchanged (`alpha` per lookup), as is the keeper query's own budget. +- **LoRa islands keep their own content findable** (spec section 6, "LoRa islands"; nothing on the wire changes): a + node with a slow interface, or naming `local_interfaces`, keeps a set of *local* servers, those whose RNS path goes + out on a named interface, or (none named) within `local_hops` (3) hops over an interface below `slow_bitrate`, + judged from the path to each server as its announces arrive and from one read of the path table every 10 minutes + (through `rnsd`, one RPC: a client's own path table starts empty). It also stores each key it provides on its + `local_replicas` (3, at most 5) closest local servers (*two-level records*: ordinary `/provide`s, withdrawn like + the rest; a local server no link could be opened to, though a path was found, is tried once more at once, since over + LoRa one lost link request left the island without its replica until the next republish), and a search asks the closest local server first (3 of them with a hop radius), its answer seeding the + rest of the lookup. In the topology simulation an island neighbour found content during a partition 100% of the + time instead of 55–72%, and island-to-island lookups took 42% less airtime, for 12% more publish airtime. Fast + nodes with no slow interface and no `local_interfaces` are unchanged. New settings `local_interfaces`, + `local_hops`, `local_replicas`; `rnfs status` gains `local_servers`. +- **Partition mode** (spec section 6, "LoRa islands"): a node with local servers that fails 3 times in a row to reach + off-island servers it knows, or hears no off-island announce or answer for over 4 times the usual gap between + off-island announces (at least 10 minutes), and then gets no answer from one it probes, uses its local servers + only for lookups and publishing, so a search during a gateway outage ends in the time its island takes instead of + minutes of link timeouts. Any off-island answer or fresh announce ends it, and a probe of one off-island server + every 5 minutes (on slow nodes after 15 minutes, doubling up to 2 hours) keeps it from sticking. Silence alone never + enters it, so a small connected network with sparse announces doesn't flap. `rnfs status` gains `partition` and + `partition_since`. +- **The heal republish:** keys whose last publish round reached no server beyond the island (during a partition) are + marked, and republished as soon as partition mode ends, paced as a slow node's rounds are, so the rest of the + network finds them within hours instead of after their next republish (3 days on a slow node). The marks are kept + across restarts (a new table, `published_heal`). + + Together, over the fake network of `tests/test_topology.py` (1,000 servers, 21% on 8 islands, an island's gateway + down): a neighbour found what an island node published before the partition 100% of the time instead of 67%, and + what it published during it 100% instead of 91%, with 0.2 timeouts and 33 s of simulated time per search instead + of 11.7 and 490 s; the backbone found what was published during the partition 0% of the time until the heal + republish, 99.5% after it. Over a simulated week with 5 or 20 servers beyond the island announcing every 6 hours, + partition mode was entered only while the gateway was down, and left within 49 minutes of its coming back. +- **`/provide` to 16 servers at once:** a node that isn't slow sends a publish round's `/provide`s to up to 16 + servers at once, each still over its own identified link with its batches in order (withdrawals first, stopping + at a refusal), so a round over many servers takes a fraction of the time. Slow nodes still send one at a time. +- **Slow nodes pace withdrawals:** unpinning many keys no longer sends every withdrawal at once over radio. A slow + node sends withdrawals along with its round's stores to the same servers, and each server only withdrawals + reach counts as one of the round's 10 lookups (up to 5 kept for them while any are queued), so the round waits + 30 s for it before the next. Other nodes still send withdrawals at once. +- Kept names' keys take their fixed ±10% republish jitter from the key's first byte rather than the whole key: 256 + slots still spread them out, and keys sharing their leading bits (the same servers are closest to them) come due + together. A key's interval may change once, by at most a fifth, at the upgrade; it stays below 3.5 days. +- **On disk:** records are read from the state database per `/find` through an index, with a small cache, instead + of all held in memory (about 500 bytes each before): memory no longer grows with the records or their publishers. + Changes are written together every 30 s; a republish is written only when it moves a record's last-stored time by + more than 6 hours, and a refused store writes nothing. A flood of 300 000 fresh identities into a full store of + 200 000 records cost about 70 µs a store and 112 transactions in all, on a desktop NVMe disk. +- **A byte budget instead of a record count:** records may take `record_budget` of disk (new, default 1 GiB: about + 7 million records at the 150 bytes a record takes, measured from the database), never the free disk the blocks' + reserve needs. `max_records` (100 000 records) goes. +- **No per-publisher caps while there is room:** anyone stores freely until the budget is full, so a large + publisher (a mirror) is no longer held to 1% of the store, nor a new one to a tenth of that. The newcomer pool and + the memories of departed and evicted publishers go too. +- **Storage tiers once the budget is full:** Known (contacts of kind `node`, private-mode members, bootstrap + nodes; 30% floor), Earned (publishers that refreshed a record on 3 distinct UTC days; 60%) and Newcomer (everyone + else; 10%). A store to a full node takes a record from a tier's largest holder: the lowest tier below the storer's + above its floor first, then, if the storer's own tier is below its floor, a higher tier above its own; else + within its own tier, refused for one holding as many as the largest, and for Earned publishers, which never evict + each other. While full, a Newcomer's record expires 4.5 days after it was last stored (otherwise every record + still lives `PROVIDER_TTL`, 7 days). A key still keeps at most 1000 publishers, now giving way by tier, with a + share kept for Newcomers (Security, above). +- **Resumable and compressed block transfers** (spec section 7, `/range`): a new optional request, `/range` + (`[cid, offset, max_bytes, flags]`, answered `[size, encoding, bytes]`), the one addition to the wire in 0.2.0; + `/blocks` is unchanged. Over slow links a node fetches each raw chunk of 4 KiB or more in ranges of about + `max(min_rate_sample, response_target` at the link's rate estimate) bytes on the wire (16 KiB at 1 kbit/s, about 2 + minutes, instead of a whole 64–128 KiB chunk), compressed with bz2, and keeps what arrived in a partial block + (`/state/partial`, at most 64, deleted at start), verified whole against the CID before it is stored and + never served; a provider whose response stalled gets ranges on any link. The first request of a fetch, before any + search, uses ranges by the same rule when it goes to the hint, a remembered provider or a node known to answer + `/range` (an open link otherwise gets `/blocks`, as an older node's silence would hold the search up): on a radio it + is most often what brings a single-chunk file (over LoRa between two Heltec V3 RNodes at 3.5 kbit/s, a 24 KB text came whole in + 88 s, where bz2 makes it 9.8 KB). Measured with the real fetcher over a fake + 1 kbit/s link: a stall anywhere in a 64 KiB chunk now resends 8 KiB on average (1.1 minutes) instead of 32 KiB + (4.4 minutes), at most 15.6 KB instead of 64.7 KB; a Gutenberg book (Pride and Prejudice, 772 KB) takes 2.8–2.9× + less airtime (1.74 hours to 0.63, counting about 0.7 KB of RNS overhead per response). A requester decompresses an + answer itself, to at most the range it asked for plus one byte, and refuses anything else (a decompression bomb + ends there; `tests/security/test_range_decompression.py`, `test_range_limits.py`). A responder answers `/range` + exactly as `/blocks`: the same response slots, pools, cooldowns, upload pacing, airtime controls and + `upload_budget`, and compresses at most 256 KiB a second. + Nodes before 0.2.0 leave `/range` unanswered (RNS drops a request for a path without a handler, which costs them + nothing and counts against nothing): a node asks such a provider with `/blocks` for the rest of the fetch after one + silence, which may be a lost packet, and records it as not answering after two silences in a row (in one fetch or + several; any answer clears a silence), in the new `range_support` table, asking afresh after 7 days: an older node + costs two request timeouts a week. + +Changed: +- **A slow node no longer publishes what it downloads, by default** (spec section 6, "Provide strategy"): + `provide_strategy` gains `auto`, the new default, which is `pinned` on a slow node and `roots` (as before) on + others. Over LoRa each download's root went to about 10 servers, an identified link each: at 977 bit/s the hour + after three 16 KB downloads the laptop transmitted 6.6% of the time, nearly all of it publishing their roots, and it + told every node which files its user fetched. A slow node set to `roots` keeps doing so. After the upgrade a slow node + with the setting unset withdraws its cached downloads' records (paced, as any withdrawal). +- **`provide_entries` covers the user's own content only** (a changed default): the files and folders inside pins of + the user, the shared folder, names and apps are provided by their own CIDs, but not those inside pins other nodes + asked for (`remote:`) or followed names (`follow:`), so a node doesn't publish the file CIDs of content others + asked it to keep; a pin the user holds too counts as the user's. Its cap of 100,000 entry keys is a new setting, + `provide_entries_max` (applied at once, like `provide_entries`), as a large mirror passes it: a pin whose entries + would pass the cap provides none of them. +- A pacing wait no longer closes the link: after a response on a slow link the fetcher waits as long again before + asking that provider again (the responder's upload pacing), and on LoRa a 64 KiB chunk takes about 9 minutes, so + the wait outlasted `link_idle_timeout` (5 minutes) and the idle link was closed, taking its rate estimate with it: + the next response was sized and paced as on a new link. The link now counts as in use while its provider is + waited for. +- Slow links are judged per provider, not per node: a gateway with a TCP and a LoRa interface isn't a slow node, + so it ran 3 parallel `/blocks`, probed open links for the root and duplicated any batch out for 30 s, all over + the radio. Now a provider is reached over a slow link when the link's rate estimate is below `slow_bitrate` (or, + without one, the next-hop interface is, or the provider is behind a slow path); providers over slow links + behind one next-hop interface share one request at a time, a node that isn't slow probes no open link over a + slow link (a slow node still probes one, as before: a hit saves a lookup over the radio), nothing is duplicated + onto a slow link, and on fast links a batch is duplicated only once it has taken 1.5 times its blocks' transfer + time at the rate estimate (and `deadline_slack`, as before). +- A lost packet no longer drops a provider: a `/blocks` request RNS times out (its packet or its answer's lost, + which RNS doesn't resend), whose link times out under it, or whose new link isn't established, failed its + provider for the rest of the fetch at once, so at 5% packet loss most 1 MiB fetches from one provider failed. + Now its link is closed (which starts no cooldown at the provider) and the request asked again on a new link + after the busy delay; the provider fails after `max_failures` such requests in a row. A link closed by either + end, no path, or a malformed answer still fails it at once. Measured over the fake link of + `tests/test_fetcher.py`'s `LossyLinkTest` (one provider, each request lost with probability 5%), 1 MiB fetches + succeeded 39% of the time before and 99.7% after (300 fetches each), or 97.6% when each new link's request and + proof are lost at 5% too. +- Idle links close before RNS keeps them alive. The node that opens a link sends a keepalive after + `max(5 s, RTT × 206)` without a packet (at most 360 s) and the other node answers, and keepalives don't count as + traffic, so a link left open for `link_idle_timeout` (5 minutes) on a 50 ms path carried about 58 packets, relayed + by every transport node on the way, against about 6 for the lookup that opened it. A link without a rate estimate + now closes just before its first keepalive (1 s and two round trips before the interval), if that is sooner: + reopening it costs about 4 packets. A link that carried `/blocks` and holds a rate estimate keeps + `link_idle_timeout`, since closing it resets the provider's estimate and RNS's transfer window. On radio paths + (RTT ≥ 1.75 s) nothing changes. Simulated over 8 hours of browsing through the gateway (lookups of 8 servers of + 300, fetches from 1 of 40 providers, half with a rate estimate; mean of 5 runs) on 50 ms paths, links are opened + 6,431 times rather than 4,490, 1,971 of them within 5 minutes of closing rather than 1,123, while keepalive + packets fall from 328,237 to 19,826, and link opening, closing and keepalive packets together from 346,199 to + 45,548. +- A node asked first for a fetch's first block (one with an open link, or a remote pin's asker) that had it was + asked for that block again in its next request, so it sent it twice: on LoRa, a whole chunk's airtime more. Now + it is asked for the blocks after it. +- A DHT client that only fetches (it provides nothing, owns no names, accepts no remote pins and isn't in private + mode) announces `rnfs.dht` only at startup, no longer every `announce_interval`; nothing needs to reach it. It + announces again at once when it starts to provide. Servers, and clients that provide, announce as before. +- Blocks are stored in subdirectories (`blocks//`): one directory of millions of files is slow to list, + and FAT32, common on the USB drives added to a Raspberry Pi, holds only about 10,900 such names per directory + (a 59–64 character name takes 6 of its 65,536 entries). The cost is at the durability points: storing blocks + syncs each subdirectory written to (up to 1,024, 16 at a time) where it synced one directory. Adding 64 MiB + (1,029 blocks) to a new store on a desktop NVMe disk: 1.02 → 1.41 s; a tree of 1,024 8 KiB files 0.95 → 1.43 s; + fetching them (each block synced on its own) unchanged at about 13.5 s (`tests/bench_block_writes.py`, + `today`, median of 3). +- `max_storage` counts what blocks take on disk: each block is a file of its own, so its size rounded up to the + filesystem's block or cluster (read once at the start: at least 4 KiB, 16–32 KiB on a FAT32 USB drive), not its + length. On FAT32 a store of small files took several times `max_storage`: 2,000 notes of 200 B–8 KB are 4.1 MiB + of blocks but 63 MiB of 32 KiB clusters (9.4 MiB at 4 KiB). Stores of large files barely change (64 KiB chunks + fill whole clusters: a mixed store of 297 MiB takes 298 MiB at 4 KiB, 312 MiB at 32 KiB), so cache is evicted a + little sooner. `rnfs status` (`GET /api/status`) gains `allocated_bytes`, what `max_storage` counts, and + `allocation_unit`; `used_bytes` stays the blocks' bytes. Quotas (`follow_quota`, `remote_pin_quota`) and + `max_download_size` still count bytes of content and blocks. Nothing is stored for it: it follows from the block + sizes rnfs keeps. +- `rnfs.apiclient` (the CLI, `rnfs.Client`) refuses a daemon older than 0.1.5 before sending it the token, rather + than sending the request on a second connection as it did for them until now (the fallback planned to go in + 0.2.0): stop such a daemon and start the new one. Every daemon of 0.1.5 or later works as before. +- `POST /api/import` (and `rnfs import`) refuse anything but a regular file, without waiting on a FIFO. +- `Config.validate` refuses `api_port` equal to a non-zero `gateway_port`. +- A request that times out raises `ConnectionFailed` saying the daemon "didn't answer within its timeout". +- `Config.load` reports invalid TOML as " isn't valid TOML: …" (still a `ValueError`). +- A fetch to pin that fails keeps what it got: `pin add`, `dir set` and `dir rm`, remote pins, followed names and + deny lists record it as a cached fetch, as `get` and `cat` already did, so trying again resumes rather than + starting over (a 4 MiB pin over LoRa failing at 90% fetched all of it again). Not when it failed on denied + content, an invalid DAG or a size bound or quota (`TooLarge`), which a retry can't get past. The record is cache + like any other (evictable, removed by `gc`), and never provided as a download of the user's. +- Remote pins don't give up on content not yet published. The node asked to pin asks the node that asked first + (its node ID follows from the identity of the `/pin`'s link), which serves what it added before its provider + records are out. A pin whose content can't be found waits and is tried again after 10 minutes, doubling up to 6 + hours, for up to 3 days, instead of being dropped at once; `pin remote ls --served` shows it waiting + (`not_found`) with when it is tried again (`retry_at`, a new field of `GET /api/pin/remote/served`). Pins + accepted after it aren't held up, and a release ends the wait at once. +- A `get`, `pin` or `export` job submitted with the new `retry: true` (`POST /api/jobs`; `retry=True` in the + library's `jobs.get`, `pin` and `export`) whose content can't be found is queued again the same way, keeping its + blocks, rather than failed: `retry_at` (and `retry_since` while it waits) are new fields of a job, and `rnfs jobs` + shows it waiting. Not for a path its directory shows doesn't exist. Retrying is opt-in: the API's default is + 0.1.6's, failing at once, so an app polling its job still sees it fail rather than stay queued for days. `rnfs + get`, `pin add` and `export` with `--background` opt in; in the foreground they still fail at once. +- **A running job says whether it is still looking for providers:** on a radio a search takes minutes before any + byte arrives, and a job showed only that it was running. A job now has `providers` (how many its fetch uses; null + before it has any, and for add and import jobs) and `searching` (true while its fetch searches the DHT), in + `GET /api/jobs`, its `job` events (a search beginning or ending is published at once) and the library's `Job`; + `rnfs get` and `rnfs jobs` show "looking for providers" until one is found, and `rnfs jobs show` the providers. +- Path requests are paced node-wide: since RNS 1.2.5 every interface limits the path requests arriving on it (3 a + second while it is under 2 hours old, as a hub's interface for a newly connected client is), and a fresh node's + join and first publish round sent dozens at once, so its hub dropped some and the queries failed after + `path_timeout`, enough for partition mode on a gateway. Every path request rnfs sends (link opens, a lookup's path + check, name resolves, NomadNet adds) now waits its turn, at least 1 s after the last, and while the busiest + interface already sends more than 1.5 a second (other applications' requests included, at most 15 s); the wait + isn't counted in `path_timeout`. A link that fails to establish costs a turn, as RNS then asks again by itself; + concurrent waits for one destination share one request, and none is sent again within 20 s of the last. In the + topology simulation a fresh fast node's hub no longer drops any of its requests and its first publish round takes + 41–45 s instead of 7 s (22–25 s with drops); island rounds take 60–80 s more of 25–31 minutes. +- A query that failed for want of a path no longer counts towards partition mode (it still counts in the routing + table): island relays in RNS's default `full` mode never forward a request for a path they lack, and a hub drops + requests over its limit, so a node with local servers could enter partition mode on a connected network (in the + topology simulation 65–72% of island publish rounds with `full` relays). A real partition is still detected, a + few seconds later (the first failure for each server is a link failure). +- Path requests only where the node can be: with its own RNS instance, a node asks for the path to a node a `/find` + reply named over a fast interface (at least `slow_bitrate`) only on that interface (a hub's or an AutoInterface's + peers count as one interface), so a gateway's lookups no longer send requests for internet nodes over its radio. + A node named over a slow interface, joins, names, providers and links to known nodes ask on every interface, and + nothing changes through a shared instance. +- Shorter path waits for lookup candidates on nodes that aren't slow: a node measures how long its path requests take + to be answered, and once 20 are measured a lookup's path check waits their 95th percentile, at least RNS's own 15 s, + instead of `path_timeout` (30 s): a candidate whose path doesn't come costs half as long, as the lookup has others + to ask. Names, nodes the user names, stores, fetches, joins and slow nodes still wait `path_timeout`. +- A server closing a link (RNS's `DESTINATION_CLOSED`: no room for another incoming link, or going away) is a + refusal, not a failure: a lookup's `/find` on such a link no longer counts against the server in the routing table, + its score or partition mode (a partition probe it ends counts as reaching the outside), and a `/blocks` provider + that closes the link is busy, asked again on a new link after the busy delay and failed after `max_failures` in a + row, instead of failing at once. A link that timed out, or that this node closed, counts as before. +- A slow node paces by its radio's measured load: RNodes report their own airtime and the channel's load (over 15 s + and an hour, through rnsd too), which a slow node reads at most every 15 s while it has paced work. While the + channel's load is above 25% its path requests and publishing rounds (the heal republish's too) are spaced further + apart in proportion (twice as far at 50%, at most 4 times), never so far that a record would go unrepublished for + half its 7-day lifetime; while its own airtime over the last hour is above 50% it refuses `/blocks` (a new airtime + refusal, `blocks_refused_airtime` in `rnfs status`, `airtime` in `GET /api/stats`'s `refused`); and while its own + transmissions took over 2% of the channel in the last hour, `provide_entries`' default doesn't turn entries on. A + figure of 0.0 counts as unknown (RNodes report it before their first measurement); other nodes and interfaces + without the figures keep the estimates they had. + +Faster: +- A long fetch looks for more providers. A fetch searched once (again only when every provider had failed), so a + slow provider was kept to the end: between cloud nodes a 30 MB fetch over a 4-hop route ran 29 minutes at 18 KB/s, + while a node 2 hops away, which had fetched the file in 26 s, provided it from then on. A node that isn't slow now + searches again while it has fewer providers than `max_parallel_providers`, a minute after the last search ended and + then twice as long each time, up to 10 minutes; what it finds joins the fetch, ranked as any provider. Slow nodes + don't (a search is a minute of airtime there), nor do fetches from a given node (`--from`). +- Fetching something again doesn't look it up again. Nothing was kept between fetches, so once its links had closed + a gateway reload or a `pin add` after `cat` repeated the lookup (about 13.5 `/find` on a fast node). The nodes + that served a root are now remembered for 3 hours after they last did (in memory, up to 10 per root and 4,096 + roots) and asked for its first block before the nodes with open links, after a remote pin's asker, under the same + slow-link rule; one that doesn't have it any more, or fails, is forgotten. Only nodes that served verified blocks + are remembered, never the providers a lookup reports. And a search that found nobody providing a root isn't + repeated for 10 minutes by the gateway or by the retries of followed names and deny lists, which then fail at + once; `get`, `cat`, `pin add`, the rest of the CLI and API, jobs and remote pins always search. +- Range reads (`cat`, the gateway, sealed reads) size their window by the providers' links: they fetched 32 chunks + (2 MiB) before handing over the first byte, about 5 hours at LoRa's 1 kbit/s. While no provider known is reached + over a fast link (or none is known yet), the window is one chunk with the next asked for ahead, so the first + byte comes after one chunk (about 9 minutes at 1 kbit/s); over fast links it stays 32. +- A start no longer lists every stored block before the API answers: their sizes are kept in the state database + (`block_sizes`) and read from there after a clean stop. After a crash or power loss the start lists them all, + as before, and corrects the table. Opening storage on a desktop NVMe disk (warm cache), small blocks: 100 000 + of them 0.50 → 0.13 s after a clean stop (0.65 s after an unclean one), a million 5.4 → 1.2 s (7.2 s after an + unclean one: 1,024 directories listed and the table compared). The first start, moving them out of 0.1.6's flat + layout, took 4.2 s and 74 s. A cold cache and a slow disk should gain more, as the table is one file read in + order (`tests/bench_startup.py`, `--cold` for a cache dropped as after a reboot). +- A start works out what the node provides faster, before the API answers. Its first refresh walked every pin's + DAG for the blocks the node serves, then read the folders of each of the user's own pins again for the files and + folders it provides (`provide_entries`); they now come from the first walk. That walk no longer remembers every + chunk it checked, only the DAG nodes, and DAG nodes decode without being encoded again to check they were written + in their one canonical form: the decoder takes each header only in its shortest form instead, which accepts + exactly the same blocks (a file node of 6 chunks 16.2 → 9.2 µs, of 256 chunks 588 → 346 µs, for every read). + Measured with `tests/bench_refresh.py` (565,000 blocks in 28 pins, one a Gutenberg-like folder of 70,000 files; + desktop NVMe disk, warm cache): the first refresh 6.26 → 3.81 s (5.30 s with the decoder alone, 4.33 s with the + one walk too), its peak memory 178 → 160 MB. A 2-vCPU cloud VM with 857,325 blocks spent about 79 s of an 88 s + start there: 54 s in the walk, 23 s reading the folders again. +- The publisher's state is on disk: where and when each provided key was last published lives in the state + database (`publish_items`, indexed by when each is next due) and is read per round, rather than held in memory + for every key. A wake asks the index when the next key is due, and a round reads the due keys only, at most + 50,000 of them (the rest in the rounds that follow at once, in slices of the key space, so the rounds need no + more region lookups than one would), so neither looks at every provided key; while a round runs only the stores + servers answered are saved, in a small table written in order, and each key's row is written once at its end. + Measured with `tests/bench_publisher.py` (a 2,000-server fake network, desktop NVMe disk), before → after: + memory per provided key 632 B → 213 B after a round and 1,326 → 187 B after a restart at 100,000 keys, 582 → + 161 B and 1,287 → 158 B at 1M (what is left is the provide set itself: a 4 GB Raspberry Pi held about 1.5–2M + keys; what bounds a node now is how many it can republish a day); a wake (`next_due`, a round + with nothing due) 80 → 0.1 ms at 100,000 keys and about 1,100 → 0.1 ms at 1M; disk 278 → 290 B a key (the due + index); bytes written per round 55–83 KB → 0.8–0.95 KB a key (SQLite's WAL and checkpoints: rows are written in + key order, and an item's row once a round instead of at each save). A first round of 1M keys took 20 rounds of + 50,000 and 175 region lookups, against one round of 157 (the same with the network size estimate given, as a + node's regions are sized: 176 and 159). +- An answer that comes back before RNS has registered its request is no longer lost. RNS registers a request only + after sending it and drops an answer it can't match, so on a fast link with a busy process (about 1 request in + 800 with 16 at once over local TCP and the CPU busy) the request waited out RNS's whole timeout, 11 s at least, + and failed. One such `/find` held back a whole publishing round: `tests/interop.py` sometimes took 10 to 25 s + longer to find a node providing what it had just added. The answer is now kept and handed over once the request + is registered, on `rnfs.dht` links and on the link of a NomadNet add, where a page small enough for one packet + (NomadNet's "Request Not Allowed" among them) could be lost the same way and the add fail. + +Fixed: +- **`rnfs.Client` no longer loads RNS for an `rnfs:` target with a path:** checking the path's names imported the DAG + module, which loads RNS's bundled msgpack and so all of RNS. The name rule now has a module of its own + (`rnfs/entry_names.py`), which the DAG, `rnfs:` links and NomadNet links share. +- **`cache rm` counts only the cached fetch's own blocks as its own**: blocks nothing used before (an add just + unpinned) are collected with it, and were all counted as the first CID's: on a cloud node a 435 B cached file + "removed 670 blocks, freed 42973080 bytes". They are counted apart, as `unreferenced_blocks` and + `unreferenced_bytes` (new in the answer of `POST /api/cache/rm` and `Uncached`), and `rnfs cache rm` says so. +- **A collection after a start no longer reads every pin's DAG again**: a sealed folder's unpack ends with one, and on + a cloud node with 857k blocks the first unpack after a start held its result for 3 minutes, its files written at + once. The DAGs the start's first refresh found complete are taken whole without being read. +- **A file-hash deny entry stops a one-block file everywhere** (spec section 8): a file of one block has its content's + hash as its CID's digest, and the specification says either entry stops it, but a `sha256:` entry was only checked + where content is assembled. A cloud node with such an entry refused the file to its own `cat` and gateway, yet + served it to `get --from` from another node and kept providing it. The raw CID a hash names is now denied with it. +- **The daemon's log reaches the journal as it is written.** Under systemd (and launchd, or the Windows logon task) + stdout is a pipe or a file, which Python fills in 8 KiB blocks, so `journalctl --user -u rnfs -f` showed nothing for + hours on a quiet node, and lines turned up in bursts stamped hours earlier, all of them at a stop. Seen upgrading a + cloud node with 857k blocks: its first start moved them for minutes with nothing in the journal. The daemon now + writes stdout a line at a time. +- **A link the other end lost is closed after one unanswered request** (spec section 3, Links). When the `rnsd` of a + node, or of a transport node on the way, restarted, the links through it were gone at the far end but still up at + this one until RNS's keepalive noticed, 12 minutes on a radio path; every request on them waited out its timeout. + Over LoRa, after the gateway's `rnsd` restarted, a slow node's every `/find` to the gateway took 21–23 s and failed, + so a search during the partition that followed found nothing in 3.5 minutes. A link that received nothing at all + while a request on it went unanswered is now closed, unless another request is using it or the request was a + `/range` to a node not known to answer it (an older node's silence, on a working link), and the next request opens a + new one. +- **No links accepted until the node can answer on them** (spec section 3, Links): the `rnfs.dht` destination + accepted links from its creation, at the start of the daemon, but requests only once the node was set up; RNS drops + a request without a handler, so through a start (80 s on a cloud node with 857k blocks working out what it + provides, minutes at the first start after this upgrade) every request other nodes sent waited out its timeout. + Over LoRa a slow node's join asked a gateway restarting meanwhile and waited 20 s for nothing. Links are refused + (RNS drops the link request, which fails in seconds) until the request server is in place. A stop now starts by + refusing new links and answering every request `False`: a request arriving while the state was being closed failed + in its handler, logged as an error ("the state database ... is closed", seen restarting a cloud node). +- **What a node publishes after starting cut off is healed too** (spec section 6, "The heal republish"): a node + that starts while its island is cut off never enters partition mode (its failures are all for want of a path, which + don't count) and its rounds find local servers only, so nothing marked their keys for the heal republish. Over LoRa + the gateway PC, restarted during an outage, stored a new key on its one neighbour, and once the backbone was back + nothing republished it beyond the island until its next republish, days later on a slow node. A round that reached + no off-island server now marks its keys while the routing table holds off-island servers but none has answered or + announced itself since the start; an island that knows of none, or a node that was heard from beyond it, marks + nothing more than before. +- **A provider out of reach for a minute isn't dropped from the fetch**: one no path was found to failed at once and + was never asked again during the fetch. Over LoRa, a fetch from a file's one provider, whose `rnsd` had just + restarted, failed after its search reported the provider again, back by then. A provider with no path is now asked + again a minute later, and fails only if it is unreachable then too. +- **A fetch uses the provider records its own node holds** (spec section 6, Finding providers): a search started + with nothing, though a DHT server holds records for the keys near it, and on an island a node holds the local + replicas of its neighbours' content. Over LoRa the laptop held the island record of a file and waited 51 s for its + search to find the provider it named. The providers this node's own `/find` answer would list are now candidates + as the search starts; the search still runs, so a record held here can't hide other providers. +- **A job's progress counts a block as its ranges arrive**: `rnfs jobs` and the job's progress events counted a + block only once it was whole. Over LoRa, where a 64 KB chunk comes in `/range` responses over two minutes, a job + resumed after a restart showed `0 B/s ETA ?` for 90 s while 56 KB of its next chunk had arrived. The bytes each + range brings now count towards `done_bytes` and the rate, once each. A `get` job also showed "finding the target" + until its first chunk was whole (it reads the file's start to tell whether it is sealed): its size, known from the + root, is now its total at once. +- A node attached to a shared instance whose `rnsd` restarted logged a warning with a traceback each time it read + the radios' load meanwhile (the RPC refused). That is now a verbose line; other failures to read the stats stay + warnings. The node itself carries on through the restart, as before (tested on the laptop: a fetch over LoRa + 12 s after `rnsd` came back). +- **An older node isn't taken for one that answers `/range` for good** (REL-5): a `/range` whose link the other end + closed before answering (a busy 0.1.6 node evicts the link first, as it doesn't count the unanswered `/range` as a + request) recorded the node as answering `/range`, a record never checked again, so every later fetch from it over a + slow link waited out its silences as lost requests and failed it. Only an answer, a refusal, a response started or + a malformed answer show a handler now. +- **A partial block the disk fails no longer loops** (REL-6): a `/range` whose bytes couldn't be written (a full or + read-only disk) left the partial empty but counted as progress, so the fetch asked for the same first range for + ever, a whole range of airtime each time over LoRa. That, or a partial that can't be read back, now has the block + asked whole with `/blocks` for the rest of the fetch. +- **Damaged local state no longer stops the node** (REL-18, found by fuzzing the state files, the state database and + `rnfs.toml`): each of these failed every start (or, in the provider store, every expiry pass and save), and is now + treated as damage: logged, and dropped, set aside or clamped. + - A legacy JSON state file or a saved row nested about 10,000 levels deep raised RecursionError: the file is moved + to `.damaged` and the row deleted, as when they aren't JSON (a `names.json` still stops startup, with a plain + error). An `rnfs.toml` nested a few hundred levels gave every CLI command a traceback: it is refused as invalid + TOML. + - Infinite provider times in `providers.json` or a 0.1.5 or 0.1.6 database failed the upgrade's provider standing + step (NOT NULL), leaving the schema as it was: those publishers now start as Newcomers, and their times are + clamped to now as any future time is. Finite times upgrade as before. + - An LXMF inbox or outbox file whose `next_id` was below -2^63 failed its import: a `next_id` below 1 counts as + missing. + - A provider record whose key isn't a 16-byte blob crashed the start, expiry and every later save of the records: + such rows are never read. A provider standing whose records, days or publisher isn't an integer (or an ID) in + range, or whose records summed past 2^63, stopped the start: it counts as no standing. + - A cut-short publish round's saved item list holding an item that didn't parse stopped the start: the list counts + as malformed whole. + - A file or a dangling link at `/partial` stopped the start: partial blocks are kept in memory, with a + warning. An empty partial block's first range was appended to a file left at its path: the file starts afresh. +- **A crash during a get job's write-out leaves nothing behind** (REL-10): `node.get` wrote under a random hidden name + beside the output (`.rnfs-get-.partial`, up to the download's size, there for a sealed file's whole fetch), + which nothing removed after a crash. A get job's write-out is now named after the job + (`.rnfs-job-.write.partial`), and the run that resumes it removes it, as it does the job's partial output. +- **A clock stepped forward doesn't end a retry** (REL-11): a job or remote pin first not found while a Raspberry Pi + without a real-time clock still had the time it booted with gave up at its next try once NTP had stepped the + clock days on, as if it had been retried for 3 days. A try more than 6 hours after it was due (that, or the node + off) no longer counts the time between. +- While a daemon was starting (minutes on a small node as RNS's transport starts), the CLI said "no API token …: is + the daemon running?", as the token is written once the API is up: it now says the daemon is starting, as it does + once the token is there and the port isn't yet. +- **A fast link's provider no longer refuses for a minute after a large response** (REL-12): the default upload cap + set when the next response may start from the link's rate estimate, which small `/range` responses had left below + `slow_bitrate` (RNS's measurement of a small transfer is mostly its ramp-up). After a 327 KB response that took 8 s + between two cloud nodes, Madrid refused the next for 57 s, and the requester, pacing by the 8 s, failed it after + three refusals, and the fetch failed (found by a soak of random fetches between the cloud nodes). Once a + response under the default cap concludes, the next may start twice the time it took after it started. +- **A name sync stops when its name's key is deleted by hand** (REL-16): the key's pins were released at the next + start but its sync stayed, paused, and a new, unrelated name later made or imported under the alias got the old + folder published to it at the folder's next change. Such a sync is now dropped at the start, with a warning; one + whose key file is there but can't be read is kept, as its pins are. +- **A provider whose responses a route cuts is asked in ranges** (REL-19): between cloud nodes, one route cut every + 64 KiB `/blocks` response from Montreal at the same point (52 of 64 KiB); each counted as the provider being busy, + and after three the fetch failed it, though ranges, half a chunk each, would have come through. A response that + stopped arriving partway now has the next request of that provider made in ranges, as after a stall. +- `announce_interval`, `republish_interval` and `scan_interval` had to be only above 0, so a typo such as + `announce_interval = 1e-300` would have announced in a loop, flooding a radio channel. They are now at least 60 s, + 10 minutes and 1 s: `config set` refuses less, and a smaller value already in rnfs.toml is raised to the minimum + with a warning, so an upgraded node still starts. +- An `rnfs.toml` starting with a UTF-8 byte-order mark, as some Windows editors save it, was refused as invalid TOML: + it is read, and `config set` writes it back without the mark. +- The API answered `HEAD` (always refused, 405) with a JSON body after the headers, which a client reads as the start + of the next response: now the headers only. +- The API's integer parameters, a request's `Content-Length` and `quiet_hours` took whatever Python's `int()` takes: + `1_0`, `+7`, spaces, other scripts' digits. Only ASCII digits are taken now, and a `Content-Length` of 2**63 or more, + which overflowed the body reader, is refused (400, as a negative one is; one that isn't a number stays 411). +- Path requests in a burst kept their spacing only as reserved: a thread that slept past its turn, as a loaded machine + does (80 ms on a CI runner), went with the next one's, two requests at once. The spacing now holds between the + requests that actually go. +- A fetch running for days kept every search it made again (one each 10 minutes that found its providers again): + it now keeps its first and the latest 8. +- **A stalled response no longer fails the other requests on its link** (REL-7): closing the link for it failed a + request waiting for its slot (another fetch from the same provider, which then failed the provider) and a `/find` + in progress on it. The link is closed only if nothing else uses it, and a request that wasn't sent because its link + closed meanwhile counts as lost, asked again on a new link. +- **A search isn't held for minutes by servers that don't answer** (specification section 6, Finding providers): + RNS gives a server it has a long path to the whole of its link establishment timeout, and on a LoRa island two + servers, one 34 hops away by RNS's path, held two of a search's three query places 127 and 217 s each; the node + that knew the provider was asked only after them, 6 minutes in, and a gateway request in a later round ran out its + 600 s. A search's query unanswered after 15 s (45 s on a slow node) now gives up its place, ranked with the + unreliable until it answers, and its answer still counts. A search that has found nobody still waits for such + queries, up to 3 times as long after each was sent, as one may hold the only record. Finding the closest nodes, + for publishing, is unchanged. +- **A get just after the add finds the content** (specification section 6, Finding providers): content added + elsewhere is found only once its publisher's records are out, which between cloud nodes took over 20 s when the + publish lookup waited on a server that didn't answer, so a `get` of a CID shared as it was added failed with "no + provider found" (several fetches of the soak, asked 20 s after the add). A fetch the user asked for that found no + provider at all now searches once more, a twentieth of `lookup_timeout` (30 s) later, before it fails; the + gateway's and the node's own retries don't. +- **A provider named by an answer that came after its lookup ended is used** (spec section 6, Finding providers): a + lookup completes once the closest nodes by rank have answered, with queries still out, and their answers' providers + were dropped. Over LoRa the PC searched for a file the laptop had just added: the laptop, one hop away on the radio + but ranked last as a slow path, answered with itself as provider 29 ms after the first lookup completed, and the + fetch failed a second later. Those providers are now reported once the search's runs are over. +- **One lost request no longer fails a provider at once**: a request left unanswered on a link that heard nothing + meanwhile closes the link (0.2.0, above), and the request then counted as a failure rather than as lost, so the + provider was dropped from the fetch. Over LoRa a lone lost `/range` looks just so, and a fetch from the file's one + provider failed. It now counts as lost: asked again on a new link, failed only `max_failures` times in a row. + ## 0.1.6 Security fixes, maintainability work and performance work. No wire-format or CID changes: these nodes, 0.1.5, @@ -120,7 +1098,7 @@ Security: and at most 64 KiB of the proof response is read. - After the proof, the client sends the request on a new connection only to a daemon that proved a version below 0.1.5 (those close every connection); a newer daemon that closes the proved connection gets - `ConnectionFailed`, and nothing is sent. The fallback for 0.1.4 daemons goes in 0.1.7. + `ConnectionFailed`, and nothing is sent. The fallback for 0.1.4 daemons goes in 0.2.0. - A followed name can no longer fill the follower's disk: its new target is fetched only if its size, and the bytes of its DAG's blocks, fit in `follow_quota` (default 1 GiB); otherwise the check fails, the follow's state says why, and the previous target stays pinned. While one target replaces another both are held, so a followed diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ae8d018..279eb01 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -19,12 +19,12 @@ Compatibility rules every change keeps: ```sh git clone https://github.com/afit21/rnfs && cd rnfs python3 -m venv .venv && . .venv/bin/activate -pip install --require-hashes --only-binary=:all: -r packaging/requirements.lock # rns, lxmf and theirs +pip install --require-hashes --only-binary=:all: -r packaging/requirements.lock # rns and its dependencies pip install --require-hashes --only-binary=:all: -r packaging/build-tools.lock # ruff, coverage, build, ... pip install --no-deps --no-build-isolation -e . ``` -The library and the CLI use the standard library besides RNS and LXMF; `rnfs.apiclient`, `rnfs.remote` +The library and the CLI use the standard library besides RNS; `rnfs.apiclient`, `rnfs.remote` (rnfs.Client) and the modules they import never import RNS. Keep it that way. ## Tests @@ -42,17 +42,23 @@ variables: | Variable | What it turns on | |---|---| -| `RNFS_E2E=1` | Real daemons and real RNS over local TCP, driven through the CLI or the library. Run them with `RNFS_E2E=1 python3 -m unittest tests.test_e2e tests.test_client tests.test_embedded tests.security.test_h1_real_rns tests.security.test_m1_real_rns tests.security.test_h2_display_page_pins` (3 to 4 minutes). CI's end-to-end step lists every module gated on it, and `tests/test_packaging.py` fails if a new one is missing from that list. | +| `RNFS_E2E=1` | Real daemons and real RNS over local TCP, driven through the CLI or the library. Run them with `RNFS_E2E=1 python3 -m unittest tests.test_e2e tests.test_client tests.test_embedded tests.security.test_h1_real_rns tests.security.test_m1_real_rns tests.security.test_h2_display_page_pins tests.test_nomadnet_e2e` (3 to 4 minutes). CI's end-to-end step lists every module gated on it, and `tests/test_packaging.py` fails if a new one is missing from that list. | | `RNFS_FUZZ_EXAMPLES=N` | A deeper, randomized run of `tests/security/test_fuzz_parsers.py` (e.g. 3000 examples per parser). That module needs hypothesis, which isn't one of rnfs's dependencies: without it every test there is skipped, as in CI. | | `RNFS_BENCH=1` | Prints the FastCDC chunking rate measured by `tests/test_chunker.py` (the test itself only checks a floor). | +| `RNFS_ECLIPSE=1` | A larger run of `tests/test_eclipse.py` (identities placed next to a key: 2,000 servers, more keys, slow nodes too, about 1.5 minutes) that prints what it measured: findability under attack, false positives, `/find` and stores per search and publish. | `tests/bench_block_writes.py` is a benchmark, not a test: run it by hand on the disks that matter (its docstring -says how, including on a Raspberry Pi SD card). Performance changes record their before and after numbers in -CHANGELOG.md. +says how, including on a Raspberry Pi SD card). So is `tests/bench_startup.py`, which times opening a large block +store at startup (after a clean stop, after a crash, and the first start on 0.1.6's layout), and +`tests/bench_publisher.py`, which measures what each provided key costs the publisher (memory, disk, writes per +round) and how long a wake takes, and `tests/bench_refresh.py`, which times a start's first walk of every pin for +what the node serves and provides (a store of about 565,000 blocks, a Gutenberg-like folder among its pins). +Performance changes +record their before and after numbers in CHANGELOG.md. `tests/interop.py` is a manual cross-version check, not a test either: it runs this tree's daemons against daemons of older releases over local TCP, in both directions, and prints a PASS/FAIL table (providers, get and cat, -transfers with and without `max_upload_rate` checked for refusals, names, keepers, LXMF). Run it before a release +transfers with and without `max_upload_rate` checked for refusals, names, keepers). Run it before a release with each supported old release extracted by `git archive`: mkdir -p /tmp/rnfs-0.1.5 && git archive v0.1.5 | tar -x -C /tmp/rnfs-0.1.5 @@ -72,6 +78,32 @@ Tests that guard compatibility, and must stay green: Never weaken or delete a test to make a change pass. If a change deliberately alters behaviour a test encodes, change the test in the same commit and say why in the commit message. +### Tests on every OS + +CI runs every test on Linux, macOS and Windows, and nearly every CI failure so far was a test written and run only +on Linux. So: +- **Paths.** Compare a path the code resolved with one the test built through `tests.support.real()`: macOS + reaches the temp directory through `/private/var`, and Windows may name it `RUNNER~1`. Write `/` in an expected + path only where the code does. +- **POSIX-only APIs** (`os.geteuid`, `os.fchmod`, `os.mkfifo`, `resource`, `fcntl`, signals Windows lacks, and the + rest of the list in `pyproject.toml`) fail ruff (TID251) unless the line says what guards it: `# noqa: TID251 - + `. Skip a test with `tests/support.py`'s `POSIX_MODES`, `PERMISSION_CHECKS`, `IS_ROOT`, `HAS_FIFO` or + `CAN_SYMLINK`. +- **Files.** Windows keeps no execute bit, can't remove or replace a file another handle has open (close stores, + databases and nodes before the temporary directory is removed), and reads `\` or `C:` in a name as a path; a link + to a directory needs `target_is_directory=True` and is removed with `os.rmdir` there. macOS refuses names that + aren't UTF-8, and paths over 1024 bytes. +- **Time.** Wait for a condition with `support.wait_for(..., what=...)`, never a fixed sleep: without `what=` its + result is all that can fail, so a call that ignores it must name what it waits for (`tests/test_suite_rules.py` + checks). Leave upper bounds a wide margin (CI's macOS runners are several times slower than a desktop) and lower + bounds a tick of slack (Windows' clock ticks every 15.6 ms before Python 3.13). + +CI runs each test that failed once more, alone, in a fresh process (`tests/ci_run.py`, up to 5 of them): one that +then passes doesn't fail the run, but is named in a warning on the run's page and summary. Fix it: it will fail +again. To run CI on a branch before a pull request, use Actions > tests > Run workflow. Every push to any branch +runs it too, so push work in progress often: a release that brings weeks of commits to macOS and Windows for the +first time is what has made releases fail. + ### Security regression tests Every security fix comes with a regression test in `tests/security/test__.py` (for example @@ -89,7 +121,8 @@ ruff check . ``` A broad `except Exception` in `rnfs/` carries `# noqa: BLE001 - `, an import that -can't be at the top `# noqa: E402`, and RUF100 flags a `noqa` that suppresses nothing. The companion NomadNet +can't be at the top `# noqa: E402`, a use of an API Windows lacks `# noqa: TID251 - ` ("Tests on every +OS", above; the same holds in `rnfs/`), and RUF100 flags a `noqa` that suppresses nothing. The companion NomadNet pages (`companion-nomadnet-sites/*/index.mu`) are linted as Python. CI also fails on merge-conflict markers. Style: match the surrounding code. Every function has a docstring, comments say why rather than what, and lines @@ -102,7 +135,7 @@ versions: | File | What | Regenerate with | |---|---|---| -| `packaging/requirements.lock` | rnfs's runtime dependencies (rns, lxmf and theirs), every file PyPI has for each | `python3 -I packaging/update-lock.py [rns==X lxmf==Y ...]` (no arguments: the newest versions `pyproject.toml` allows) | +| `packaging/requirements.lock` | rnfs's runtime dependencies (rns and its dependencies), every file PyPI has for each | `python3 -I packaging/update-lock.py [rns==X ...]` (no arguments: the newest versions `pyproject.toml` allows) | | `packaging/build-backend.lock`, `packaging/build-tools.lock` | setuptools; and build, twine, PyInstaller, ruff, coverage and their dependencies | `python3 -I packaging/update-lock.py --build pyinstaller==X ...` (run it on Linux with Python 3.12) | | `packaging/flatpak/python3-deps.json`, `build-sources.json` | the Flatpak's wheels, from the locks | `python3 -I packaging/flatpak/generate-sources.py` | @@ -151,7 +184,14 @@ route touches these places: tested on both by `tests/test_public.py`. 5. The CLI command, if any (`rnfs/cli.py`, or the feature's own commands), through `ApiConnection` rather than a hand-written path. -6. Tests of the handler (`tests/test_api.py` or the feature's test file), and a CHANGELOG line. +6. The OpenAPI reference, `docs/openapi.json`: the operation (method and path, `operationId`, a feature's + route tagged with and `x-rnfs-feature` naming its feature), its parameters or request body, its answer's + schema and the statuses it answers; `X-Rnfs-Pin-Key` if it pins for the caller; a new error code, event kind or + capability in `ErrorCode`, `EventKind` and `Event`, or `Capabilities`. A new field in `GET /api/status` goes + into the status part that produces it (`StatusCore` for the node's, `Status` for a feature's; a new + feature adds a part to `Status`'s `allOf`). `tests/test_openapi.py` fails, naming it, on a route, code, kind + or capability missing, and its live checks on a field the answers carry that isn't documented. +7. Tests of the handler (`tests/test_api.py` or the feature's test file), and a CHANGELOG line. ## Commits @@ -174,7 +214,8 @@ Steps: 1. Go through the checklist below. 2. Set `__version__` in `rnfs/__init__.py` to X.Y.Z, change CHANGELOG.md's first heading from `## X.Y.Z (unreleased)` to `## X.Y.Z`, and update README's release links and file names - (`tests/test_packaging.py` checks the three agree). + (`tests/test_packaging.py` checks the three agree), and `info.version` in `docs/openapi.json` + (`tests/test_openapi.py`). 3. Commit, let CI pass on `main`, then push the tag: `git tag vX.Y.Z && git push origin vX.Y.Z`. The workflow refuses a tag that isn't `v` + `__version__`, and a CHANGELOG whose section for it is still marked unreleased. @@ -195,8 +236,8 @@ Steps: - [ ] Every security fix has its `tests/security/test__*.py` regression test, and `tests/security/README.md` lists it. - [ ] CI runs every `RNFS_E2E` module. Lint passes. -- [ ] Before and after numbers for the performance changes (a desktop NVMe disk and a Raspberry Pi SD card), - recorded in CHANGELOG.md. +- [ ] Before and after numbers for the performance changes, on a desktop disk, recorded in CHANGELOG.md + (Raspberry Pi SD-card numbers aren't a release requirement). - [ ] specification.md updated for every change in behaviour, with its version and compatibility note at the top. - [ ] design.md updated for every change in how things work. - [ ] CHANGELOG.md's entry states whether the wire format or CIDs changed ("no wire-format or CID changes"), diff --git a/MANIFEST.in b/MANIFEST.in index 5650f6a..faeadbb 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -4,5 +4,7 @@ prune packaging # ...except the dependency lock, which install.sh uses when installing from an sdist include packaging/requirements.lock include packaging/build-backend.lock +# The local API's OpenAPI reference, for app authors +include docs/openapi.json # The example program for the public API (rnfs.Node, rnfs.Client) recursive-include examples *.py diff --git a/README.md b/README.md index 2d65a3e..1707e42 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ Different package options available at [latest release](https://github.com/afit2 **Windows:** -[`rnfs-0.1.6-windows-x64.exe`](https://github.com/afit21/rnfs/releases/download/v0.1.6/rnfs-0.1.6-windows-x64.exe), saved as `rnfs.exe` +[`rnfs-0.2.0-windows-x64.exe`](https://github.com/afit21/rnfs/releases/download/v0.2.0/rnfs-0.2.0-windows-x64.exe), saved as `rnfs.exe` somewhere on your PATH ([how](#windows)). **Linux:** @@ -93,7 +93,7 @@ Or installers for ([Debian & Ubuntu](#debian-and-ubuntu)). [Flatpak](#other-lin **macOS:** -`pipx install https://github.com/afit21/rnfs/releases/download/v0.1.6/rnfs-0.1.6-py3-none-any.whl` ([details](#macos)). +`pipx install https://github.com/afit21/rnfs/releases/download/v0.2.0/rnfs-0.2.0-py3-none-any.whl` ([details](#macos)). ## 2 - Start rnfs and leave it running in its own terminal: @@ -122,28 +122,28 @@ links that can change, sharing in the background), and [Commands](#commands) lis # Install -rnfs runs on Linux, macOS and Windows. Every package includes Reticulum and LXMF and gives you one `rnfs` +rnfs runs on Linux, macOS and Windows. Every package includes Reticulum and gives you one `rnfs` command. Your data in `~/.rnfs` is never touched by an update or uninstall. | System | Download | |---|---| -| Windows (64-bit) | [`rnfs-0.1.6-windows-x64.exe`](https://github.com/afit21/rnfs/releases/download/v0.1.6/rnfs-0.1.6-windows-x64.exe) ([how](#windows)) | -| Debian 12+, Ubuntu 24.04+, Raspberry Pi OS (64-bit) | [`rnfs_0.1.6_amd64.deb`](https://github.com/afit21/rnfs/releases/download/v0.1.6/rnfs_0.1.6_amd64.deb) or [`rnfs_0.1.6_arm64.deb`](https://github.com/afit21/rnfs/releases/download/v0.1.6/rnfs_0.1.6_arm64.deb) ([how](#debian-and-ubuntu)) | -| Other Linux (x86_64) | [`rnfs-0.1.6-x86_64.flatpak`](https://github.com/afit21/rnfs/releases/download/v0.1.6/rnfs-0.1.6-x86_64.flatpak) ([how](#other-linux-flatpak)), or the [install script](#install-script) | -| macOS, or any system with Python 3.11+ | [`rnfs-0.1.6-py3-none-any.whl`](https://github.com/afit21/rnfs/releases/download/v0.1.6/rnfs-0.1.6-py3-none-any.whl) ([how](#macos)) | +| Windows (64-bit) | [`rnfs-0.2.0-windows-x64.exe`](https://github.com/afit21/rnfs/releases/download/v0.2.0/rnfs-0.2.0-windows-x64.exe) ([how](#windows)) | +| Debian 12+, Ubuntu 24.04+, Raspberry Pi OS (64-bit) | [`rnfs_0.2.0_amd64.deb`](https://github.com/afit21/rnfs/releases/download/v0.2.0/rnfs_0.2.0_amd64.deb) or [`rnfs_0.2.0_arm64.deb`](https://github.com/afit21/rnfs/releases/download/v0.2.0/rnfs_0.2.0_arm64.deb) ([how](#debian-and-ubuntu)) | +| Other Linux (x86_64) | [`rnfs-0.2.0-x86_64.flatpak`](https://github.com/afit21/rnfs/releases/download/v0.2.0/rnfs-0.2.0-x86_64.flatpak) ([how](#other-linux-flatpak)), or the [install script](#install-script) | +| macOS, or any system with Python 3.11+ | [`rnfs-0.2.0-py3-none-any.whl`](https://github.com/afit21/rnfs/releases/download/v0.2.0/rnfs-0.2.0-py3-none-any.whl) ([how](#macos)) | -These links are for 0.1.6; for a newer version, take the same files from the +These links are for 0.2.0; for a newer version, take the same files from the [latest release](https://github.com/afit21/rnfs/releases/latest), which also has the source archive -(`rnfs-0.1.6.tar.gz`) and `SHA256SUMS` ([checking a download](#checking-a-download)). +(`rnfs-0.2.0.tar.gz`) and `SHA256SUMS` ([checking a download](#checking-a-download)). ## Windows -The exe is a single file with Python, Reticulum and LXMF inside. In PowerShell, from your download folder: +The exe is a single file with Python and Reticulum inside. In PowerShell, from your download folder: ```powershell $bin = "$env:USERPROFILE\.local\bin" New-Item -ItemType Directory -Force $bin | Out-Null -Move-Item .\rnfs-0.1.6-windows-x64.exe "$bin\rnfs.exe" -Force +Move-Item .\rnfs-0.2.0-windows-x64.exe "$bin\rnfs.exe" -Force Unblock-File "$bin\rnfs.exe" [Environment]::SetEnvironmentVariable("Path", [Environment]::GetEnvironmentVariable("Path", "User") + ";$bin", "User") # open a new terminal, then: @@ -160,12 +160,13 @@ rnfs daemon ## Debian and Ubuntu -Pick `amd64` or `arm64` to match `dpkg --print-architecture`. Ubuntu 22.04 has Python 3.10, which is too old; -use the [install script](#install-script) there. +Pick `amd64` or `arm64` to match `dpkg --print-architecture`. Ubuntu 22.04's Python is 3.10, which is too old: +there, run `sudo apt install python3.11 python3.11-venv` and then the [install script](#install-script), which finds +`python3.11`. ```sh -curl -LO https://github.com/afit21/rnfs/releases/download/v0.1.6/rnfs_0.1.6_amd64.deb -sudo apt install ./rnfs_0.1.6_amd64.deb +curl -LO https://github.com/afit21/rnfs/releases/download/v0.2.0/rnfs_0.2.0_amd64.deb +sudo apt install ./rnfs_0.2.0_amd64.deb systemctl --user enable --now rnfs # run it as a service; logs: journalctl --user -u rnfs -f loginctl enable-linger $USER # keep it running while you're logged out ``` @@ -176,9 +177,9 @@ loginctl enable-linger $USER # keep it running while you're logged ou ## Other Linux (Flatpak) ```sh -curl -LO https://github.com/afit21/rnfs/releases/download/v0.1.6/rnfs-0.1.6-x86_64.flatpak +curl -LO https://github.com/afit21/rnfs/releases/download/v0.2.0/rnfs-0.2.0-x86_64.flatpak flatpak remote-add --user --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo -flatpak install --user rnfs-0.1.6-x86_64.flatpak +flatpak install --user rnfs-0.2.0-x86_64.flatpak alias rnfs='flatpak run io.github.afit21.rnfs' # add to ~/.bashrc rnfs daemon ``` @@ -194,7 +195,7 @@ with `flatpak uninstall --user io.github.afit21.rnfs`. ```sh brew install pipx && pipx ensurepath # then open a new terminal -pipx install https://github.com/afit21/rnfs/releases/download/v0.1.6/rnfs-0.1.6-py3-none-any.whl +pipx install https://github.com/afit21/rnfs/releases/download/v0.2.0/rnfs-0.2.0-py3-none-any.whl rnfs daemon ``` @@ -205,7 +206,7 @@ rnfs daemon https://raw.githubusercontent.com/afit21/rnfs/main/contrib/io.github.afit21.rnfs.plist launchctl load -w ~/Library/LaunchAgents/io.github.afit21.rnfs.plist ``` -- pipx (and plain `pip install`) resolves rnfs's dependencies unpinned, at the newest Reticulum, LXMF and +- pipx (and plain `pip install`) resolves rnfs's dependencies unpinned, at the newest Reticulum and cryptography rnfs allows, not at the tested versions and hashes in `packaging/requirements.lock` that the other packages and the install script use. @@ -241,18 +242,18 @@ Each release's `SHA256SUMS` lists every file's checksum. With it next to your do ```sh sha256sum -c SHA256SUMS --ignore-missing # Linux; macOS: shasum -a 256 -gh attestation verify rnfs_0.1.6_amd64.deb --repo afit21/rnfs +gh attestation verify rnfs_0.2.0_amd64.deb --repo afit21/rnfs ``` A matching checksum shows the file is whole, not who made it. The signed build provenance, checked with the [GitHub CLI](https://cli.github.com/) (2.49 or newer, logged in), shows it was built by this repository's -release workflow. On Windows, `Get-FileHash .\rnfs-0.1.6-windows-x64.exe` gives the checksum. +release workflow. On Windows, `Get-FileHash .\rnfs-0.2.0-windows-x64.exe` gives the checksum. ## From source ```sh git clone https://github.com/afit21/rnfs && cd rnfs -pip install --require-hashes -r packaging/requirements.lock # the tested Reticulum and LXMF (optional) +pip install --require-hashes -r packaging/requirements.lock # the tested Reticulum (optional) pip install -e . rnfs daemon python3 -m unittest discover tests # the tests @@ -275,9 +276,9 @@ Everything else is in the rnfs directory (`--dir`, default `~/.rnfs/`), which on | Path | Contents | |---|---| | `rnfs.toml` | your [settings](#settings) | -| `storage/identity`, `storage/lxmf_identity` | the node's identity (your node ID comes from it) and your LXMF identity | -| `blocks/` | stored blocks | -| `state/rnfs.db` | pins, records, jobs, names' sequences, contacts, the LXMF inbox and outbox, deny lists (SQLite; an older version's JSON files are imported once and kept as `*.migrated`) | +| `storage/identity` | the node's identity (your node ID comes from it); `storage/lxmf_identity` and `lxmf/`, if there, are what rnfs's own LXMF left before 0.2.0 ([LXMF moved out](#lxmf-moved-out)) | +| `blocks/` | stored blocks, one file per block in 1,024 subdirectories (`blocks//`) | +| `state/rnfs.db` | pins, records, jobs, names' sequences, contacts, deny lists, the stored blocks' sizes (SQLite; an older version's JSON files are imported once and kept as `*.migrated`) | | `names/` | name keys: back them up with `rnfs name export` | | `files/` | the shared folder: anything put here is added, pinned and shared | | `api_token`, `daemon.lock` | the running daemon's API token and lock | @@ -332,8 +333,25 @@ talks to it. Your node needs Reticulum interfaces that reach other nodes: [reticulum.network/connect.html](https://reticulum.network/connect.html) for the public backbone. - **Radio:** add an `RNodeInterface` for LoRa ([Reticulum manual](https://reticulum.network/manual/interfaces.html)). +**Running a LoRa island's gateway.** A node on the radio finds other nodes with path requests, and Reticulum never +repeats a path request on the interface it came in on: a request reaches only the nodes in radio range, and those +with transport enabled answer from the paths they already know. So the gateway (the node with the radio and an +internet interface) should forward requests for paths it doesn't know to the internet side: run it with +`enable_transport = Yes`, and give its radio interface `mode = gateway` (or `recursive_prs = yes`). A relay +bridging two radios needs the same on both. That serves the nodes in the gateway's (or the relay's) radio range; +nodes farther out find only the servers whose announces their neighbours heard, however many path requests they +send, as a relay with one radio can't pass a request on. (In Reticulum 1.5.6 write `mode = gateway`: +`interface_mode = gateway` isn't read correctly.) + +**A node one radio hop from its gateway.** By default every server within 3 hops over a slow interface counts as +your island's (`local_hops`), which suits a mesh of relays; but a node whose only way out is one gateway reaches +the servers on the gateway's internet hub in 2 or 3 hops too, and counts them as local. It then asks them first and, +while the island is cut off, keeps asking them in vain. On such a node set `local_hops = 1`, or name the radio +interface (`local_interfaces`) on the gateway itself, where rnfs sees which servers are behind the radio. + Your node then joins the DHT through the project's built-in bootstrap nodes and the rnfs nodes it hears -announcing. `rnfs status` shows your node ID and `known_nodes`; `rnfs peers` lists the nodes it knows. To also +announcing. `rnfs status` shows your node ID and `known_nodes`, and once a few lookups have completed, about how +many DHT servers the whole network has (`network_size`); `rnfs peers` lists the nodes it knows. To also join through a node you choose, run `rnfs daemon --bootstrap ` (or set `bootstrap_nodes`); `default_bootstrap = false` stops it contacting the built-in ones. @@ -347,7 +365,9 @@ rnfs get --from # ask a node you know has it, without th ``` In a browser, `http://127.0.0.1:8789/rnfs//` shows a directory's listing or its `index.html`, so a folder -with a website in it works as a site. +with a website in it works as a site. `rnfs url ` prints its address on a subdomain of its own, +`http://k….rnfs.localhost:8789/`, where its pages can also read its files with `fetch()` (names: +`http://.name.localhost:8789/`). ### 3. Share files @@ -360,7 +380,14 @@ rnfs add report.pdf --wrap # in a directory, so the link keeps its name: ` stops sharing, and `rnfs gc` frees space used by downloads you didn't pin. By default your node also shares what you download while it's cached, so others can see what you -fetched; `provide_strategy = "pinned"` shares only what you pin. +fetched, unless it is a slow radio node, which shares only what you pin (publishing each download costs airtime); +`provide_strategy = "pinned"` shares only what you pin on any node, `"roots"` downloads too. Every file and folder inside what you pin is also +findable by its own CID (`provide_entries`), so a file you shared in a folder can be fetched without the folder's +CID; on a slow radio node, only while there are few of them. Content other nodes asked yours to keep (remote pins) +and names you follow are shared only as a whole. That also lets anyone who has one of those files look +up its CID and see that your node holds it, so pin only what may be known to be on your node, or set +`provide_entries = false`. `add` refuses the rnfs dir (`~/.rnfs`) and any folder holding it, such as `~`: it holds +your node's keys and API token. ### 4. Names: a link that can change @@ -369,6 +396,7 @@ A CID always means the same content. A **name** is a fixed ID you can point at n ```sh rnfs name create site # prints the name (32 hex characters) rnfs name publish site # again later for a new version +rnfs name sync site ~/site # or keep it pointing at a folder, published as it changes rnfs contacts add site # a nickname, so others can type "site" rnfs get site/index.html # whatever the name points at now rnfs name follow # keep the latest version pinned @@ -391,16 +419,17 @@ IP networks. It is **not** compatible with IPFS. The full comparison is in | Kademlia DHT | For provider records only, `K` = 10, server and client modes, Sybil resistance from seniority | | Bitswap | Multi-source, resumable, verified `/blocks` requests over Reticulum, sized to each link's measured rate | | IPNS | Signed records in Reticulum announces, re-served by **keepers** so a name outlives its owner's announces | -| Pinning, remote pinning, GC | The same, with pin sources (user, folder, name, remote, follow, LXMF) | +| Pinning, remote pinning, GC | The same, with pin sources (user, folder, name, remote, follow, apps) | | CAR files, HTTP gateway | `export` / `import` archives; a local gateway with sandboxed pages and trustless `?format=raw` / `archive` | | MFS | `dir set` / `dir rm` make edited copies; a name is the mutable root | -| Pubsub, DNSLink | Dropped (floods are unaffordable on radio; Reticulum has no DNS). Use `name follow` and LXMF links | +| Pubsub, DNSLink | Dropped (floods are unaffordable on radio; Reticulum has no DNS). Use `name follow` and `rnfs:` links | | Private networks | Reticulum interface access codes, plus rnfs private mode (members only) | # Settings Settings go in `~/.rnfs/rnfs.toml`; command-line options override them. Times are in seconds, sizes in bytes, -rates in bits/s, and an unknown setting is an error. These are the ones you're most likely to change; every +rates in bits/s, and an unknown setting is an error (one a newer version retired, such as `max_records`, is ignored +with a warning). These are the ones you're most likely to change; every setting and its default is in [specification.md, section 12](specification.md#12-constants). ```toml @@ -411,7 +440,6 @@ provide_strategy = "pinned" # don't share your downloads upload_budget = 5_000_000 # upload at most ~5 MB per hour no_serve_interfaces = ["RNode"] # don't upload over the LoRa radio quiet_hours = "22:00-06:00" -lxmf_accept_from = [""] ``` | Setting | Default | What it does | @@ -420,54 +448,75 @@ lxmf_accept_from = [""] | `bootstrap_nodes`, `default_bootstrap` | `[]`, `true` | Node IDs to join through first; whether to also use the built-in ones | | `slow` | `"auto"` | A slow node uses one provider at a time and fewer parallel queries; `auto` when every interface is under `slow_bitrate` (100 kbit/s) | | `announce_interval` | `21600` (6 h) | How often the node and its names announce; keep it ≥ 1 h | -| `provide_strategy` | `"roots"` | `"roots"` shares pins and every complete download while it's cached (even a `cat` or `stat`), so **anyone can learn what you downloaded**; `"pinned"` shares only pins | -| `max_storage` | 10 GiB | Cached downloads are evicted above this; pins never are. The disk keeps max(256 MiB, 2%) free | +| `provide_strategy` | `"auto"` | `"roots"` shares pins and every complete download while it's cached (even a `cat` or `stat`), so **anyone can learn what you downloaded**; `"pinned"` shares only pins; `"auto"` is `"pinned"` on a slow node, `"roots"` on others | +| `provide_entries` | unset | Also share every file and folder inside your pins under its own CID (not inside downloads, remote pins or followed names). Unset: on, for up to `provide_entries_max` of them, oldest pins first; on a slow node only while republishing them would take under 2% of the radio channel (about 40 keys in all on a large network: `rnfs status` says when they're off and why). `true` or `false` overrides it. Each one tells anyone who looks that your node has that file | +| `provide_entries_max` | `100000` | The most files and folders `provide_entries` shares; a pin that would pass it shares none of its own (`rnfs status`: `provide_entries_capped`). Raise it for a large mirror | +| `max_storage` | 10 GiB | Cached downloads are evicted above this; pins never are. It counts disk use: each block rounded up to the filesystem's cluster, which on a FAT32 drive (16–32 KiB) makes small files take several times their size (`rnfs status`: `allocated_bytes`). The disk keeps max(256 MiB, 2%) free | | `max_download_size` | none | Refuse fetches larger than this | +| `record_budget` | 1 GiB | Disk the provider records your node stores for others (as a DHT server) may take; when full, your node contacts and publishers seen for days keep their place, and strangers share a small part | | `max_jobs` | 2; 1 on slow nodes | Transfers running at once | | `upload_budget` | none | Bytes per hour your node uploads in total | | `serve_interfaces`, `no_serve_interfaces` | all, none | Reticulum interfaces to upload on, or never upload on (names or prefixes) | | `quiet_hours` | off | Local times with no uploads, e.g. `"22:00-06:00"` | +| `local_interfaces` | `[]` | On a LoRa island: the Reticulum interfaces (names or prefixes; with `rnsd`, as its config names them) whose servers are your island's. Empty: servers within `local_hops` (3) over a slow interface. Set it on a gateway, or behind a big public hub | +| `local_hops`, `local_replicas` | `3`, `3` | How far a slow interface's servers count as local when `local_interfaces` is empty; and how many of them also store each thing you share (0 to 5), so your island finds it while cut off | | `remote_pin_identities`, `remote_pin_quota` | `[]`, 1 GiB | Identities allowed to ask this node to pin (`pin remote add`), and the storage each may use | | `deny_lists` | `[]` | [Deny lists](#keeping-content-off-your-node) to apply | | `keep_max_names` | `100` | Names you keep or follow for others | | `follow_quota` | 1 GiB | Storage each followed name's target may use; a larger one isn't fetched and the previous one stays pinned (up to twice this while it changes). `0` stops fetching followed names | -| `lxmf_accept_from` | everyone | Only keep LXMF messages from these addresses (recommended if you only expect links from people you know) | -| `lxmf_auto_fetch_from`, `lxmf_auto_fetch_quota` | `[]`, 1 GiB | Senders whose links are fetched and pinned automatically, and the storage each may use | -| `lxmf_propagation_node` | none | Propagation node for offline delivery (`send --propagate`) and periodic downloads | -| `lxmf_send_from_hint` | `true` | Links you send carry your node ID, so recipients can fetch from you; this links your LXMF address to your node | | `api_port`, `gateway_port` | `8788`, `8789` | Local API and gateway ports (127.0.0.1 only) | | `gateway_scripts` | `true` | Let gateway pages run scripts | | `debug_log` | `false` | Write `~/.rnfs/debug.log` (readable by you only) for diagnosing problems (`daemon --debug-log`) | | `verbosity` | `0` | Extra logging: 1 per-query detail, 2 RNS debug output (`daemon -v`, `-vv`) | +**Changing settings while the daemon runs.** `rnfs config` lists every setting with the value the daemon runs +with (`*` set in `rnfs.toml`, `!` changed there but waiting for a restart, `^` set on the daemon's command line, +which wins); `rnfs config get ` describes one. `rnfs config set ` (a TOML value: `true`, +`10`, `"text"`, `["a", "b"]`; a bare word is text) and `rnfs config unset ` edit `rnfs.toml` in place, +keeping your comments, after checking the whole file as a start would. Some apply at once: `max_storage` (enforced +then), `max_download_size`, `max_jobs`, `quiet_hours`, the serve interfaces, the quotas and a few more (`config get` +says which); the rest at the next start. Apps do the same through `GET /api/config` and `POST /api/config/set`, which +carry each setting's type, range, unit, default and description. + **Private mode.** `private = true` with a `members` list (identity hashes, from each node's `rnfs status`) makes the node link to, serve and keep records for members only. A name's announce still reaches everyone, so outsiders can see which CID it points at and when it changes (not fetch it), and `rnfs name publish` warns of this in private mode. -LXMF senders you've named (in `lxmf_accept_from`, `lxmf_auto_fetch_from`, or as an LXMF contact) are trusted. -Anyone can make new LXMF addresses for free, so messages from everyone else only push out each other's, within -fixed limits; set `lxmf_accept_from` if strangers' messages aren't wanted. +**LXMF moved out of rnfs.** Until 0.2.0 rnfs could send and receive links over LXMF +itself (`rnfs send`, `inbox`, `outbox`, `lxmf address`, `lxmf sync`, the `lxmf_*` settings and the +`/api/lxmf/` routes). That is now a separate app built on rnfs's local API and library, and +rnfs 0.2.0 no longer has any of it; `rnfs:` links stay rnfs's (`rnfs.Node` and `rnfs.Client` take one as a +target). An upgrade keeps what your node's LXMF had: its identity (`storage/lxmf_identity`) and state +(`lxmf/`) are left untouched, so an LXMF app can adopt them and keep your LXMF address; your inbox and outbox stay +in `state/rnfs.db` for it to import; your LXMF contacts stay in `rnfs contacts`; and `lxmf_*` settings left in +`rnfs.toml` are ignored with a warning (remove them with `rnfs config unset`). Files fetched automatically from +LXMF senders stay pinned, as the pins of an app called `lxmf` (`app:lxmf:
` in `rnfs pin ls`): an LXMF app +registered under that name takes them over, and if you don't want them, `rnfs app add lxmf` then +`rnfs app rm lxmf` releases them. # Commands -Every command except `daemon` talks to the running daemon, so run them as the daemon's user. `--dir DIR` and -`--api-port N` pick another daemon; `rnfs --help` shows every option. A **target** is +Every command except `daemon` and `url` talks to the running daemon, so run them as the daemon's user. +`--dir DIR` and `--api-port N` pick another daemon; `rnfs --help` shows every option. A **target** is `[/path]` or `/path`; where a node ID is expected, a node contact's nick works too. | Command | Description | |---|---| | `rnfs daemon` | Run a node with its API and gateway (`--bootstrap`, `--rnsconfig`, `--private`, `--debug-log`, `-v`…) | -| `rnfs status` / `peers` / `stats` | Node ID and identity hash, storage, jobs, LXMF address / known nodes / traffic | +| `rnfs status` / `peers` / `stats` | Node ID and identity hash, storage, jobs, and what it sees of the network (estimated size, density-check alarms, partition mode, provider records against `record_budget` by tier, refusals and evictions) / known nodes, their round-trip time and whether each is online / traffic | | `rnfs ping NODE_ID` / `providers CID` | Round-trip time to a node / nodes providing a CID | | `rnfs add PATH\|-` | Add a file, directory or stdin, pin it and print its CID (options below) | +| `rnfs add DEST:/page/PATH` / `add DEST:/file/PATH` | Add a snapshot of a NomadNet page or file (`DEST` is the NomadNet node's 32-hex address), pin it and print its CID | | `rnfs get TARGET [-o OUT] [--from NODE_ID] [--background]` | Fetch and write out a file or directory | | `rnfs cat TARGET [--offset N] [--length N]` | Write a file, or part of one, to stdout | | `rnfs ls TARGET` / `stat TARGET` | List a directory / size and how much is stored locally | +| `rnfs url TARGET` | The target's gateway site URL (`http://k….rnfs.localhost:8789/`; a name's too); needs no daemon | | `rnfs dir set TARGET NAME CID` / `dir rm TARGET/NAME` | Make a copy of a directory with an entry added, replaced or removed | | `rnfs export TARGET -o FILE` / `import FILE` | Write or import an archive (USB sticks, sneakernet) | | `rnfs pin add TARGET` / `pin rm CID` / `pin ls` | Fetch and pin / unpin / list pins, their sources and where they came from | | `rnfs pin remote add\|rm NODE_ID CID` | Ask another node to pin or unpin (it must list you in `remote_pin_identities`) | +| `rnfs pin remote ls [NODE_ID] [CID] [--check]` / `pin remote ls --served` | The remote pins you asked for and whether each is done (the node found among the CID's providers; `--check` looks now) / those others asked of your node, what became of each and their use of `remote_pin_quota` | | `rnfs gc` / `verify` | Evict cached downloads / re-check every stored block | | `rnfs deny add\|rm ENTRY…` / `deny ls` | Edit your deny list / show it and the lists in use | | `rnfs jobs [list\|show\|wait\|cancel\|clear] [ID]` | Background transfers: progress, rate, time left | @@ -475,24 +524,31 @@ Every command except `daemon` talks to the running daemon, so run them as the da | `rnfs name export\|import ALIAS` | Back up or restore a name's private key | | `rnfs name keep\|unkeep\|kept NAME` | Store and serve someone's name record, so it resolves while they're offline | | `rnfs name follow\|unfollow\|following NAME` | Keep the latest version a name points at pinned | -| `rnfs contacts add NICK HEX` / `rm` / `ls` | Nicknames for names, node IDs and LXMF addresses | -| `rnfs lxmf address` / `lxmf sync` | Your LXMF address / download waiting messages now | -| `rnfs send ADDRESS TARGET [-m TEXT] [--propagate]` | Send an `rnfs:` link over LXMF | -| `rnfs inbox` / `inbox get N` / `inbox rm N` / `outbox` | Received links (each line starts with the sender's address and whether it is `untrusted`, `trusted` or a contact; the display name the sender chose comes last, quoted), fetching them (a name link whose name can't be resolved needs `--use-sent-cid`), sent messages | +| `rnfs name sync ALIAS FOLDER` / `name unsync ALIAS` / `name syncs` | Keep your name pointing at a folder's current content: published when it changes, once unchanged a minute, at most once an hour / stop (the name keeps its target) / list the syncs and how they stand | +| `rnfs contacts add NICK HEX` / `rm` / `ls` | Nicknames for names, node IDs and LXMF addresses (an LXMF app reads them) | `add` options: `--wrap` keeps the file's name in the link; `--only-hash` prints the CID without storing; `--no-pin` keeps it as evictable cache; `--chunker fastcdc` lets an edited version share most blocks; `--to RECIPIENT…` (after the path) seals a file so only those node IDs, LXMF addresses or contacts can read it, -and `--to-self` adds you. +and `--to-self` adds you; a folder is sealed as one file, a tar named `.tar`, hiding its names too. + +A NomadNet link (`rnfs add e166ecdbc405bcc9762b6767617fc949:/page/index.mu`) is fetched from that node as it serves +it now and added like a file, with the link shown in `pin ls`; run it again for a newer snapshot. The same options +apply, and `--name N` wraps it under that name. Requests are anonymous; `--identify` identifies with rnfs's own +NomadNet identity (`storage/nomadnet_identity`, not your node's) for pages the node serves only to identities it +lists: `rnfs status` shows its hash (`nomadnet_identity_hash`), for the node's operator to list. A local file whose name looks like such a link is added as `./NAME`. `get` without `-o` uses the target's name only if it is a safe file name (printable, not starting with `.`), otherwise its CID, and writes under a temporary name until the transfer is complete. It decrypts sealed files -you can open; `--raw` gives the stored bytes, which is what scripts that show content to others should use. +you can open, and unpacks a sealed folder to its directory (`--no-extract` keeps the `.tar`); `--raw` gives the +stored bytes, which is what scripts that show content to others should use. **Gateway** (`http://127.0.0.1:8789`): `/rnfs/[/path]` and `/name/[/path]` serve files and -directories; `?filename=` sets the type and download name; `?format=raw` and `?format=archive` return data you -can check against the CID. Pages are sandboxed and can't read the gateway's responses, and sealed files are -never decrypted there. +directories; `?filename=` sets the type and download name (on a site, below, only the download name); +`?format=raw` and `?format=archive` return data you can check against the CID. Pages are sandboxed and can't read the gateway's responses, and sealed files are +never decrypted there. Each CID and name also has a site of its own, `http://.rnfs.localhost:8789/[path]` +(`rnfs url ` prints it) and `http://.name.localhost:8789/[path]`, whose pages can read their own +files; browsers resolve `*.localhost` themselves, other tools should use the paths above. # Building apps on rnfs @@ -503,7 +559,8 @@ errors: - **`rnfs.Client`** uses the user's running daemon (their content, pins and names). It never loads Reticulum, so it starts fast. -**The API is provisional until rnfs 0.2** (`rnfs.API_STATUS`). +**The API is provisional until rnfs 1.0.0** (`rnfs.API_STATUS`): the library and the local API may change in +0.2.x releases too; `capabilities()` (`GET /api/capabilities`) says what a daemon offers. ```python import RNS, rnfs @@ -521,12 +578,31 @@ with rnfs.Node("~/.myapp/rnfs") as node: # or: rnfs.Client(), for the d node.jobs.get(f"rnfs:name/{site.hex}", "/tmp/copy").wait() ``` -- An embedded node is a DHT client with LXMF off unless you pass `settings={"dht_mode": "server", ...}`. It +- An embedded node is a DHT client unless you pass `settings={"dht_mode": "server", ...}`. It never stops your Reticulum. Never point it at the daemon's `~/.rnfs/`: a directory is used by one node at a time. - Callbacks run on one event thread; keep them short. +- Beyond content, pins, names and jobs: `peers()` and `stats()`, the address book (`contacts.add/get/list/remove`), + and keeping and following names (`names.keep/unkeep/kept`, `names.follow/unfollow/following`). +- To follow what changes instead of polling lists: `events.watch(callback, kinds=...)` (jobs, pins, the cache, names, + peers, remote pins; an event with `event.lost` means "read everything again"), or `events.since(...)`. Also + `capabilities()`, pin labels (`label_pin`), the cache (`cache()`, `uncache(cid)`), settings (`config()`, + `config_set({...})`), tars and archives as streams (`add_tar`, `get_tar`, `export_stream`, `import_stream`), and + for a `Client` `shutdown()` and `daemon_info()`. +- A `Client` can have a token of its own, so the user sees and removes what your app pinned: the user runs + `rnfs app add myapp -o ~/.myapp/token`, and you use `rnfs.Client(app="myapp", token_file="~/.myapp/token")`. + Its pins get the source `app:myapp` (`with client.pin_key("inbox"): ...` makes them `app:myapp:inbox`'s, a group + your app can hold and release on its own); `client.unpin()` releases only your app's, and `rnfs app rm myapp` + revokes the token and releases them all. The token isn't scoped: your app can do anything the user's `rnfs` + command can, and so in effect manage apps too. `rnfs app` itself refuses an app's token, but an app can read + and write files as the daemon's user (`add`, `get`, `export`), which is enough to reach the command line's own + token. Give a token only to software you would trust with `rnfs` itself. +- A `Client` waits as long as each request takes unless given `timeout=`; `with client.timeout(5): ...` bounds + just the calls your thread makes inside it (a `Node` accepts it too, and does nothing). - `examples/rnfs_share.py` is a runnable example using either class; design.md's "Public API" section is the - reference. Other languages can use the local API's JSON endpoints (listed in `rnfs/api.py`). + reference. Other languages can use the local API's JSON endpoints: `docs/openapi.json` is an OpenAPI 3.1 + reference for every route (parameters, answers, error codes, the token headers and the events), which OpenAPI + tools can read or turn into a client. # Keeping content off your node @@ -555,10 +631,9 @@ cheaply. with `dir set` and `name publish`. Library nodes `rnfs name follow` it, so each always holds the latest edition, and readers `rnfs get library/maps/region.pdf`. -**Sharing with a friend over LXMF.** `rnfs send /report.pdf -m "as promised"`. -Sideband, NomadNet and MeshChat show the link as text; their rnfs node lists it in `rnfs inbox` and fetches it -with `rnfs inbox get 1`, asking your node first. Add friends to `lxmf_auto_fetch_from` to fetch their links -automatically. +**Sharing with a friend.** Send them `rnfs:/report.pdf?from=` any way you like (Sideband, +NomadNet and MeshChat show it as text, and an LXMF app built on rnfs sends and fetches such links for you); they +fetch it with `rnfs get /report.pdf --from `, asking your node first. **Private group.** `rnfs add minutes.pdf --to bob --to-self` seals a file so only those people can read it, while any node can carry it. For a fully closed group, run every @@ -576,7 +651,9 @@ resolvable through your node while they're offline. `rnsd`, every link looks like it arrives on the local interface. **A website on the mesh.** `rnfs add mysite/`, then `rnfs name create site && rnfs name publish site `. -Visitors open `http://127.0.0.1:8789/name//` on their own node, and each `name publish` updates it. +Visitors open `http://127.0.0.1:8789/name//` on their own node, and each `name publish` updates it; or +`rnfs name sync site mysite/` publishes the folder whenever it changes (at most once an hour, and only once it has +been unchanged for a minute, so a half-copied site isn't published). NomadNet pages can't open `rnfs:` links, so show the link as text: `rnfs:name//index.html`. # Running several nodes @@ -598,7 +675,10 @@ rnfs --dir /tmp/b --api-port 9002 get -o out - Sybil attacks can make a CID hard to find (`get --from` bypasses the DHT), but data can never be forged. - Sealed files hide content and file name, not size or recipients, and don't prove who sealed them. `get` of a directory leaves sealed files inside it encrypted. -- Under the default `provide_strategy = "roots"`, anyone can learn what your node downloaded. -- LXMF messages over 256 kB are refused, unsent messages are lost if the daemon restarts, and offline delivery - needs `lxmf_propagation_node`. A message rnfs can't keep (a stranger's during a flood) is lost, though the - sender's app shows it as delivered. +- A sealed file is readable by whoever holds the recipient's node identity, which is online whenever the node + runs and stored unencrypted in the rnfs dir (`storage/identity`): sealing protects against the network, not + against someone with access to the recipient's disk. +- Under `provide_strategy = "roots"`, the default on nodes that aren't slow, anyone can learn what your node + downloaded. With + `provide_entries` (on by default), anyone can learn which files your pinned folders hold by looking up each + file's CID, without knowing the folder. diff --git a/companion-nomadnet-sites/display-mu-file/index.mu b/companion-nomadnet-sites/display-mu-file/index.mu index f2b64f9..1cc98ad 100755 --- a/companion-nomadnet-sites/display-mu-file/index.mu +++ b/companion-nomadnet-sites/display-mu-file/index.mu @@ -169,7 +169,7 @@ def take_lock(): handle = open(os.path.join(PAGE_DIR, "lock"), "a+") try: try: - import fcntl + import fcntl # noqa: TID251 - ImportError on Windows, handled below except ImportError: # Windows import msvcrt msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1) diff --git a/companion-nomadnet-sites/file-index-node/README.md b/companion-nomadnet-sites/file-index-node/README.md index 8749b5e..bc9408d 100644 --- a/companion-nomadnet-sites/file-index-node/README.md +++ b/companion-nomadnet-sites/file-index-node/README.md @@ -14,9 +14,9 @@ NomadNet. Edit `DESCRIPTIONS`, `HIDDEN` and `SHOW_UNNAMED` at the top of `index. Anyone who can reach your NomadNet node can open the page. It lists a pin only if all of these are true: - **You made it.** Its sources in `rnfs pin ls` include `user` (`rnfs add`, `rnfs pin add`, ...) or `folder` - (your shared folder). Pins that exist only because a contact sent you a file (`lxmf:`), a name you follow - updated (`follow`), or another node asked you to pin it (`remote:`) are never listed, so the page doesn't - reveal who sends you what. + (your shared folder). Pins that exist only because a contact sent you a file (`app:lxmf:`, an LXMF app's, + or `lxmf:` before rnfs 0.2.0), a name you follow updated (`follow`), another node asked you to pin it + (`remote:`) or an app pinned it (`app:`) are never listed, so the page doesn't reveal who sends you what. - **It is complete** on this node. - **It isn't sealed.** A sealed (encrypted) file's name is meant to stay private, so sealed files are left off. The page recognises one by its first bytes. It reads them once per file with `rnfs cat --raw`, which diff --git a/companion-nomadnet-sites/file-index-node/index.mu b/companion-nomadnet-sites/file-index-node/index.mu index 90f9aab..6b757f3 100755 --- a/companion-nomadnet-sites/file-index-node/index.mu +++ b/companion-nomadnet-sites/file-index-node/index.mu @@ -249,7 +249,7 @@ def take_lock(): handle = open(LOCK, "a+") try: try: - import fcntl + import fcntl # noqa: TID251 - ImportError on Windows, handled below except ImportError: # Windows import msvcrt msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1) diff --git a/design.md b/design.md index 1a0b6a7..d8f1c2c 100644 --- a/design.md +++ b/design.md @@ -12,6 +12,10 @@ authority on behaviour; this file fixes module boundaries and the interfaces bet helpers for settings that depend on slow nodes or links. Loaded from `/rnfs.toml` plus command-line overrides (and `defaults` beneath the file: an embedded node's); `validate()` checks every core setting's type and range (`WHOLE_NUMBERS`, `RATES`, `DURATIONS`, `FLAGS`, `IDENTITY_LISTS`), and the daemon exits on an invalid one. + An unknown setting is refused too, but a retired one (`RETIRED_SETTINGS`, name -> what replaced it: `max_records` + since 0.2.0) is left out by `Config.combine`, wherever it comes from, with a warning in `Config.warnings` (an + embedded node's own defaults silently), so an `rnfs.toml` an older version accepted still starts the daemon; the + settings schema doesn't list it, so `rnfs config set` refuses it and `unset` removes it. Reticulum is started from `reticulum_dir`, by default RNS's own config directory, so the node attaches to `rnsd` with the matching RPC key. `member_set` and `remote_pin_set` are `members` and `remote_pin_identities` as frozensets of lowercase hex, normalised once there for the node, the link pool and the request server. @@ -19,9 +23,8 @@ authority on behaviour; this file fixes module boundaries and the interfaces bet hashes, LXMF addresses). `parse_hex16(text, what, *, brackets=False, upper=True)` takes whitespace around the digits, RNS's `` only where `brackets`, and nothing between them (`bytes.fromhex` alone would skip whitespace there); `is_hex16` is its check and `normal_hex16` the lowercase form. `api.parse_hex_id`, - `names.parse_name`, `identity.parse_node_id`, `contacts.parse_hex`, `lxmf.parse_address` and `Config.validate` - use it. Persisted rows are checked against its patterns (`HEX16`, `HEX16_LOWER`) as they are, unstripped, so - what loaded before still loads. + `names.parse_name`, `identity.parse_node_id`, `contacts.parse_hex` and `Config.validate` use it. Persisted rows are + checked against its patterns (`HEX16`, `HEX16_LOWER`) as they are, unstripped, so what loaded before still loads. - `rnfs/errors.py` — `RnfsError` and subclasses `InvalidDag`, `NotFound`, `TooLarge`, `StorageFull`, `Busy`, `Denied`, `Cancelled` (a fetch stopped by its observer), `Closed` (an `rnfs.Node` or `rnfs.Client` used after `close()`). @@ -30,13 +33,18 @@ authority on behaviour; this file fixes module boundaries and the interfaces bet strings only at the user edge (CLI, API, gateway, logs) and binary only on the wire. `Cid.from_block_name` is the block store's scan at startup: it accepts exactly what `parse` accepts, but checks the canonical form with one regular expression and decodes with `int(..., 32)` instead of re-encoding, keeping the name as the written form. + `to_base36` and `from_base36` are the gateway's host form (`k` + base36, section 4), which only the gateway's + host parser (`gateway.parse_host`) reads: exact length and digits by a regular expression (`int(..., 36)` alone + would also take `_`, signs, whitespace and non-ASCII digits), then `Cid.from_bytes`, then the canonical check. - `rnfs/log.py` — logging through RNS's logger. `log()` escapes non-printable characters in every message (remote names can't inject terminal escapes or fake lines); `cap_warnings` keeps a result's first `MAX_WARNINGS` (100) warnings plus a count line and `warnings_total`. With a debug sink set (`set_debug_sink`), every message also goes to it, whatever RNS's loglevel. `ThrottledLog(interval, clock)` logs one line about a kind of event at most every interval (60 s), with a count of those left out; `due()` is the check alone. The response guards, the request rate limit (`RequestServer._over_rate`), `RnsPaths` and the - peers listing use it. An unexpected exception caught by a broad `except` (a `# noqa: BLE001` reason says why + peers listing use it. `SlowStep(doing, done, after=10, interval=30)`, a context manager, says a start step's + `doing` once it has run `after` seconds, that it still is every `interval`, and `done` with the time it took (a + quick step says nothing): storage's listing of every block, and the node's first `Publisher.refresh`. An unexpected exception caught by a broad `except` (a `# noqa: BLE001` reason says why it's caught) is logged at `LOG_WARNING` inside the `except`, so the debug log gets its traceback: among them the fetcher's `/blocks` worker, the lookup's routing-table callbacks and `LinkPool.request`. - `rnfs/debuglog.py` — the debug log file (`debug_log`, `daemon --debug-log`), `/debug.log` (0600), @@ -52,12 +60,13 @@ authority on behaviour; this file fixes module boundaries and the interfaces bet `sync_directory` (fsync a directory after a rename into it); `printable` (Python escapes for non-printables, used by the log, the API's log lines and the CLI); `unsafe_name_reason` (one plain path component on this OS; `walker` uses it) and `safe_default_name(name, fallback)`, which every default output name chosen by someone - else goes through (DAG last names, sealed names, LXMF link names): printable, no leading `.` or `-`, no `\`, + else goes through (DAG last names, sealed names): printable, no leading `.` or `-`, no `\`, a safe component, else the fallback (the CID). Text files are opened with `encoding="utf-8"`. Node IDs, DHT keys and names are 16-byte `bytes`. Identity hashes (for `members`, `remote_pin_identities`, -pin source `remote:`) are RNS identity hashes, and LXMF addresses (`lxmf_*` settings, pin source -`lxmf:
`) are `lxmf.delivery` destination hashes, all written as 32 hex characters. +pin source `remote:`) are RNS identity hashes, and LXMF addresses (contacts of kind `lxmf`, the pin +source `app:lxmf:
` rnfs's own LXMF left, `storage.LXMF_PREFIX`) are `lxmf.delivery` destination hashes, +all written as 32 hex characters. - `rnfs/statedb.py` — the state database, `/state/rnfs.db` (standard `sqlite3`, WAL mode, `busy_timeout`; created 0600, which SQLite copies to its `-wal` and `-shm` files; on Windows the state @@ -69,9 +78,12 @@ pin source `remote:`) are RNS identity hashes, and LXMF addresses (`lx never part of one or one without those before it); a sqlite failure raises `StateDbError`, an `OSError`. Text that may hold lone surrogates (file names, remote strings) goes through `dump`/`load` (JSON text); sequences are decimal text, since they pass SQLite's 64-bit integers. Without `_sqlite3` the daemon stops - with a message saying so. + with a message saying so. `connect_memory(stores)` makes a private in-memory database with only the tables of + the schema steps of those labels (a memory-only `ProviderStore`). -Persistent state lives in the state database. Each store keeps its state in memory, which is all reads use, +Persistent state lives in the state database. Each store but the provider store (which reads its records per key, +`providers.py` below) and the publisher (which reads its items' rows per round, `publisher.py` below) keeps its state +in memory, which is all reads use, loads it once at startup (checking every row as the JSON loaders checked every record: a malformed row is logged, skipped and deleted) and then writes only what changed: a pin is one row inserted or deleted, batched stores write the records changed since their last save. The shared parts are in statedb: `open_store(state_dir, @@ -102,6 +114,7 @@ rnfs/ constants.py config.py ids.py errors.py log.py shared foundation debuglog.py the debug log file, /debug.log (debug_log) fsutil.py cross-platform file operations: replace/remove retried on Windows, O_BINARY, IS_WINDOWS + retry.py when content not found yet is tried again (remote pins, jobs): next_retry, no imports statedb.py the state database, /state/rnfs.db: schema, transactions, JSON state migration content/ no networking; uses RNS's bundled msgpack and RNS logging, so importing it loads RNS cid.py CIDs (section 4) @@ -111,14 +124,26 @@ rnfs/ seal.py sealed files: SealingReader, open_sealed/SealedFile, recipients (section 5, Sealed files) walker.py Expect, reader checks, path lookup, byte ranges, listing, writing out (sections 5, 7) edit.py dir set / dir rm (section 11) - blockstore.py one file per CID, atomic, verified (section 8) + blockstore.py one file per CID in 1,024 subdirectories, atomic, verified (section 8) storage.py Storage: blocks + pins (sources) + cached fetches + protection + eviction/GC (section 8) archive.py export/import stream; archive_size for an exact Content-Length (section 8) - shared_folder.py importing /files/ with pin source "folder" (section 8) + tarstream.py directories as tar streams: build_tar (an upload), plan/length/chunks (a download), + folder_chunks (a local folder, for sealing) + shared_folder.py importing /files/ with pin source "folder" (section 8); FolderImporter, the + import of one entry, which name sync reuses dht/ no RNS dependency routing.py RoutingTable: buckets, seniority, failures, unreliable nodes, bans (section 6) - providers.py ProviderStore: records, TTL, max_records eviction, /find reply selection (section 6) - lookup.py iterative lookups with alpha/beta, time budget, unknown-ID budget (section 6) + routing_state.py RoutingState: the routing table saved in the state database (section 6) + scores.py Scores: local scores per peer, decay, bootstrap seeds, the capped saved table (section 6) + firstseen.py FirstSeenLog: when each server's announce was first heard; closest old servers (section 6) + providers.py ProviderStore: records on disk, TTL, record budget and storage tiers, /find replies (section 6) + lookup.py iterative lookups with alpha/beta, time budget, unknown-ID budget, the senior rule and the + density check's rings (section 6) + density.py SizeEstimate: the network size from recent lookups; the density check and its rings (6) + disjoint.py DisjointLookup: one lookup over paths that share no node (section 6) + lookups.py Lookups: which lookups searching and publishing run; Search: the fallback to disjoint paths (6) + sweep.py sweep providing: regions of due keys, the far half's extra /find, Sweep (section 6) + recent.py RecentCounts: events per minute over the last day, for the network status (section 11) network/ everything that talks to RNS protocol.py message encoding/validation, batch sizes (sections 3, 6, 7, 8) rates.py rate estimates, slow links, slow node detection (section 2); the /blocks size rules both @@ -127,18 +152,35 @@ rnfs/ guards.py guards rejecting oversized, compressed or request Resource advertisements (on every link in the daemon, on rnfs.dht links only in an embedded node) and, on rnfs.dht links, responses that fail a msgpack header check before RNS decodes them (3, 10); the - half-open incoming link cap per destination (3) + same for NomadNet links this node opened, against a page's shape; the half-open + incoming link cap per destination (3) + nomadnet.py fetching a page or file from a NomadNet node over a link of its own, bounded per link + (Transfer: segment sequence and decompression bound, a page's size, metadata check) (11) server.py RequestServer: handlers, every section 3 limit, airtime policy, feature paths (3, 7) - client.py PeerClient: /find, /provide, /blocks (with stall detection), /pin, ping (sections 3, 6, 7) + client.py PeerClient: /find, /provide, /blocks and /range (with stall detection), /pin, ping (3, 6, 7) announces.py rnfs.dht announces carrying the DHT mode, and the announce handler (section 3) identity.py node identity file, node ID helpers mesh.py provider ranking (PathInfo, provider_rank, RnsPaths), airtime policy, Traffic (3, 6) - names.py NameManager: name keys, records, publish, resolve, sequences, keeper hook (section 9) + names.py NameManager: name keys, records, publish, resolve, sequences, keeper hook, announces + held while the clock is behind a name's last announce (section 9) + clockskew.py ClockSkew: the clock's offset from heard announces' emission times, for a warning; no + RNS (section 9) + name_sync.py NameSync: names kept pointing at a folder's content, settle time, hourly limit (section 9) denylist.py DenyList: denied CIDs and file hashes, local entries, accepted lists (section 8) fetcher.py fetching a target from providers; observing() for job progress (sections 6, 7) + ranges.py /range on the requesting side: which nodes answer it (persisted), partial blocks, one range + fetched into one (section 7) publisher.py provide strategy, batched /provide, republish, withdrawal, extra key sources (section 6) node.py RnfsNode: wires everything; the facade below node_content.py node side of --no-pin, sealing/unsealing, raw blocks and archive streams + apps.py AppRegistry: the apps rnfs app add issued tokens to, their verifiers (section 11) + uris.py rnfs: URIs and the LXMF link map (Link, parse_uri, format_uri, encode_field, + decode_field): the interop contract LXMF apps use, exported as rnfs.Link and so on; no RNS + (section 13) + entry_names.py the entry name rule (check_name, is_valid_name), which rnfs.content.dag re-exports; no RNS, + so uris.py and nomadnet.py check names without loading it (section 5) + lxmf_messages.py the lxmf_messages table rnfs's LXMF left (before 0.2.0) for an LXMF app: only the import of + 0.1.4's lxmf_inbox.json and lxmf_outbox.json into it (section 13) api.py local API, 127.0.0.1:8788 (section 11) apiclient.py the local API's protocol layer (ApiConnection) under rnfs.Client and the CLI: token, proof check, typed errors (section 11) @@ -156,7 +198,6 @@ rnfs/ keepers.py /name, kept names (section 9) follow.py followed names (section 9) contacts.py address book (section 11) - lxmf.py rnfs: URIs, LXMF send/inbox/outbox/auto-fetch (section 13) deny.py deny_lists sources: fetched, checked, followed; rnfs deny (section 8) examples/ rnfs_share.py: the public API with either class install.sh Linux installer of the latest release (or --main): venv, command, optional systemd user @@ -177,7 +218,10 @@ packaging/ requirements.lock (hash-pinned runtime dependencies), upda nodes `FileNode(children)`, `DirectoryNode(entries)`, `IndexNode(children)`, each with `child_cids` (the CIDs it lists, in order) and `names` (its entries' names or children's first names; none for a file node). - `encode(node) -> bytes`; `decode(block) -> node`, strict (types, shortest headers, re-encodes identically), - raising `InvalidDag`. Nodes are immutable and decoding is a pure function of the bytes, so `decode` remembers + raising `InvalidDag`. Re-encoding identically is checked without encoding: the reader takes each length or + integer only in the form `umsgpack.packb` writes it (`_SHORTEST`, the least value each longer header carries), + and nothing else in a node can be written two ways, so it accepts exactly the blocks a re-encode check would + (tests/test_dag.py `ShortestHeaderTest` compares the two on every header form and every byte changed). Nodes are immutable and decoding is a pure function of the bytes, so `decode` remembers the nodes of the blocks it decoded last (`MEMO_BLOCKS` 256, `MEMO_BYTES` 2 MiB of blocks, least recently used forgotten first; a block that fails is checked again each time): a read meets the same nodes many times. `check_name(name)` raises `ValueError`. @@ -211,11 +255,26 @@ packaging/ requirements.lock (hash-pinned runtime dependencies), upda `storage.py` — `Storage(blocks_dir, state_dir, max_storage)`, thread-safe: - blocks: `has(cid)`, `get(cid, verify=False) -> bytes | None` (with `verify`, a block no longer matching its CID is deleted), `put(cid, block)` (verifies, even a block already stored; atomic, the file fsynced), - `used_bytes()`. Adds, the shared folder and archive import store through `with session.writer() as put:`, + `used_bytes()` (the blocks' bytes), `allocated_bytes()` (what they take on disk: each block's size rounded up + by `fsutil.allocated` to `allocation_unit`, which `fsutil.allocation_unit(blocks_dir)` gives once at the start, + kept as a running total beside `used_bytes` and derived from the block sizes at each start, so nothing extra is + stored or stat'ed). Adds, the shared folder and archive import store through `with session.writer() as put:`, which writes blocks on up to 16 threads (`WRITE_THREADS`, at most 32 queued) so their fsyncs overlap, and has stored every one, or raised the first error, before the block exits: nothing is pinned before its blocks - are. It is handed only blocks built from, or checked against, their CIDs. `BlockStore.sync()` syncs the blocks directory; `Storage` calls it before saving pins or - cache records, so they never refer to block names a power loss could undo. + are. It is handed only blocks built from, or checked against, their CIDs. `BlockStore.sync()` syncs the + subdirectories blocks were renamed into since the last sync (16 at a time, `SYNC_THREADS`: they share the + filesystem's journal commits), and `blocks/` itself when a subdirectory was made; `Storage` calls it before + saving pins or cache records, so they never refer to block names a power loss could undo. +- block layout (`blockstore.py`): `blocks//`, `xy = shard_of(name)`, the written CID's `[-3:-1]`. `put` + writes its temporary file in the block's subdirectory, made 0700 on first use (`_shard_dir`, under a small + lock, since writers run in parallel). Opening the store lists `blocks/` itself only: `.partial-` files there + are deleted, each subdirectory is checked (`_check_shard`: `lstat`, then on POSIX an `O_NOFOLLOW` descriptor's + `fstat` for the owner and an `fchmod` to 0700; a link, non-directory or another user's raises RnfsError and + stops the start), and every canonical-CID file there (0.1.6's flat layout) is moved into its subdirectory + (`fsutil.replace`; if the target exists, the flat copy is deleted) before the subdirectories and `blocks/` are + synced. A move that fails leaves the CID in `_flat`, where `path()` looks first, until a start moves it, or + `delete` removes it. `scan(tidy)` lists every subdirectory (no links followed; a CID in another block's + subdirectory is ignored) and with `tidy` deletes the `.partial-` files it meets, which only a start may do. - pins: `pin(cid, source)`, `pin_many(cids, source)` (one save for them all; archive import), `unpin(cid, source) -> bool`, `pins() -> dict[Cid, set[str]]` (a copy), `pinned_cids() -> frozenset[Cid]` (the same set until pins change: the publisher's refresh reads it after every add), `has_pin(cid, source)`, @@ -235,7 +294,8 @@ packaging/ requirements.lock (hash-pinned runtime dependencies), upda - `check_disk(nbytes)` raises `StorageFull` when storing `nbytes` would leave less than the disk reserve (max(256 MiB, 2% of the filesystem)) free. `max_storage` itself never refuses a fetch: it is enforced by eviction once fetches end. -- `evict() -> set[Cid]` enforce `max_storage` (LRU cached fetches); `gc() -> stats` evicts every cached fetch; +- `evict() -> set[Cid]` enforce `max_storage` over `allocated_bytes()` (LRU cached fetches; the fetcher's + `_make_room` and the node's `_enforce_storage` compare `allocated_bytes()` with it too); `gc() -> stats` evicts every cached fetch; both delete unreachable blocks and report targets no longer complete, so the node can withdraw them. Both first read, without the lock, the DAG nodes they will walk (`_warm`, filling the `_links` and `_steps` memos, its path walk the one `_path_cids` makes under the lock, `_walk_path`; completeness via `dag_blocks`), @@ -250,10 +310,29 @@ packaging/ requirements.lock (hash-pinned runtime dependencies), upda since the last `save()`; those still cached are always the most recently used, so a save writes them with new `used` values (the table's LRU order) and deletes the others. - `verify() -> stats` re-checks every block, deleting bad ones. +- The block index, `_sizes` (CID -> size), is also the `block_sizes` table (schema step `block_sizes`). `_put` + and `_delete` note each change in `_sizes_dirty` (a size, or None); `_save_pins` writes them in the pin's durable + transaction and `save()` with the cache records, each taking and writing them under the lock (`save()` syncs + the blocks first, outside it), so an older batch never lands after a newer one. The table is trusted only after + a clean stop: `close()` marks the store closed (`_put` raises `Closed`, `_delete` deletes nothing), syncs the + blocks, writes what is left and sets the meta value `(block_sizes, clean)` to 1 in one durable transaction, + unless a put is still writing (`_writing`, a thread that outlived the node's stop) or `BlockStore.untidy` (a + partial file it couldn't delete). At the next start `_index` clears the mark durably before the block store + is opened (so the move of flat blocks, or anything after, can't be trusted to a stale table), then loads the + table, adding the flat blocks `BlockStore` found (`found_flat`: an older rnfs may have stored some since); with + no mark, a 0, or a row that doesn't parse, it lists every block (`scan(tidy=True)`) and writes the rows that + differ from the table (all of them after a bad row). A row whose file is missing is found when the block is + read, as a damaged block is (`get` deletes it from the index). A million small blocks (NVMe, warm cache) are + indexed in 1.2 s from the table against 5.4 s for 0.1.6's listing; a start after an unclean stop takes 7.2 s + (1,024 directories listed, the table read and compared). - A stored DAG node that no longer matches its CID (`DAMAGED`) has unknown children: while anything live reaches one, `evict()` and `gc()` delete nothing and log it once (`_held_back`) until `verify()` removes it or a fetch replaces it. Fetches treat such a block as not stored: deleted and fetched again. -- `dag_blocks(cid) -> frozenset[Cid]` the blocks of a complete DAG (memoised), used for "what a node serves". +- `dag_blocks(cid) -> frozenset[Cid]` the blocks of a complete DAG (memoised), used for "what a node serves"; + `dag_entries(cid)` the CIDs its directory entries name (every file and folder below it, none for a file), which + the same walk collects from the directory nodes it decodes and memoises beside it (`_complete_entries`, forgotten + with `_complete`), so the publisher's entries need no second walk. The walk keeps the DAG node expectations it + checked, not raw blocks' (checking one again is a size lookup, and they are most of a large DAG). For an incomplete DAG it remembers the block it found missing (`_missing`, `MISSING_MEMO` 10 000, oldest forgotten first): a CID's content never changes, so the DAG stays incomplete while that block isn't stored, and `is_complete` is a lookup until it arrives. The walks under the lock take a DAG memoised as complete @@ -279,66 +358,315 @@ before they are pinned. ## DHT layer interfaces -`routing.py` — `RoutingTable(own_id, config, clock=time.monotonic, admit=None)` (`admit(node_id)`: private mode's -member check, asked only for a node not in the table and without the table's lock held, since it may block on -RNS; the table is checked again once it answers): +`routing.py` — `RoutingTable(own_id, config, clock=time.monotonic, admit=None, wall_clock=time.time)` +(`admit(node_id)`: private mode's member check, asked only for a node not in the table and without the table's +lock held, since it may block on RNS; the table is checked again once it answers): `heard(node_id) -> bool` (proven server: announce in server mode, or answered `/find`), `answered(node_id) -> bool` (`heard`, and reliable again), `failed(node_id) -> bool` (True if dropped), `unreliable(node_id)` (its last `max_failures` queries or more failed, in the table or not; announces don't clear it: `_unanswered`, at most `MAX_UNRELIABLE` (4096) nodes, least recently failed forgotten first), `may_retry(node_id)` (due again after `AVOID_AFTER` (300 s), doubling per further failure up to `AVOID_MAX` (3600 s); True claims the retry, pushing the next one back a further backoff, so concurrent lookups don't all retry it), `remove(node_id)`, `ban(node_id)`, -`is_banned(node_id)`, `closest(key, count=K, for_reply=False)`, -`seniority(node_id) -> float | None` (seconds in the table), `is_senior(node_id, min_age)`, `standing(node_id, +`is_banned(node_id)` (also True while `blackholed(node_id)`, an attribute the node sets to +`Blackhole.node_listed`: `heard` and `answered` refuse such a node too, and nothing of it is saved), +`closest(key, count=K, for_reply=False)`, `seniority(node_id) -> float | None` (seconds in the table), `join_order()` (the servers a join asks first, before +the node orders them by score), `is_senior(node_id, min_age)`, `standing(node_id, min_age) -> int` (0 senior and healthy, 1 healthy, 2 stale, 3 absent: ranks keepers), `standings(min_age) -> dict` (every node's standing now: the keeper query's snapshot from before its lookup), `contains(node_id)`, `nodes()`, `node_count()`, `live_count()` (nodes `/find` replies would include). Every `heard` forgets expired bans and dropped nodes' seniority, looking only at the front of each: both are kept in expiry order (an entry set again is moved to the end), and their readers check expiry too, so an entry out of order (after a clock step or a changed setting) only lingers in memory. - -`providers.py` — `ProviderStore(max_records, state_dir, ttl=PROVIDER_TTL, clock=time.time, rng=random, -max_per_publisher=None, max_per_key=1000, own_id=None, seniority=0)` (`state_dir` holds the state database): `store(key, publisher) -> bool`, `withdraw(key, publisher)`, `provide(publisher, -keys, withdrawn) -> int`, `retain(keep) -> int` (drops every record of publishers `keep` rejects: private mode's -non-members), `for_reply(key, exclude=lambda id: False) -> list[bytes]`, `providers(key, exclude) -> list[bytes]`, `expire()`, -`record_count()`, `allowance(publisher)`, `save()`, `close()`. Local limits against flooding: one publisher holds at most max(10, -`max_records` // 100) records once seen for `seniority` (the node passes `provider_seniority`) and a tenth of -that (at least 10) before, the node's own (`own_id`) excepted. A publisher at its limit has its expired records -swept at most once per batch, and not at all before `sweep_after` (the earliest its oldest record can expire, -from its last sweep), so each further key of a refused `/provide` costs one check; a key keeps at most 1000 publishers: a newcomer to a full key evicts a random -one, never one of the `K/2` oldest records. The node's own record (`own_id`) isn't counted toward the 1000 and is -never a victim, here or in the pool (`_most_pooled_by_others` steps past it when it alone holds the most), so -Sybil stores can't push it out of its own store; its records may exceed the pool rather than evict each other. A tenth of the store is a newcomer pool evicting by fair share: each -publisher's pooled keys sit in an ordered dict (oldest pooled first), and publishers in lists by how many they -hold (`_holders`, with each one's index, and `_most_pooled`), so a full pool picks one of the biggest holders at -random in O(1); one already holding the most is refused. The newest publisher with records outside the pool comes -from a heap of (first seen, arrival) with stale entries dropped lazily. First-seen times of departed publishers -(seen for `seniority`) are kept for a TTL, at most `max_records` // 10, in departure order; those of younger -publishers whose last record the pool evicted (`_evicted`) for a TTL too, at most `pool_size` // 10 (at least -10), forgetting the one first seen most recently when full (an O(n) scan, only on such evictions). Times later than now -(a clock corrected backwards) are clamped to now at load and at the first `expire()` after the clock went backwards: -the store notes the latest clock reading any time it holds came from (`_latest`, via `_now()`), and an `expire()` -with the clock not behind it skips the clamping pass, which would change nothing (about half of a pass at 100 000 -records, held under the lock). Records are added only through -`_insert`, by `store` and by the load alike, so the indexes beside `_records` (each publisher's keys and count -outside the pool, the counts, `_pooled`, `_holders`, `_newest`) are kept in step in one place; -`_check_consistency()` recomputes them from `_records`, and test_providers runs it after every test. +Persistence (0.2.0): the table runs on the monotonic clock and converts to wall clock (`wall_clock() - clock()` as +the offset) only for saved rows. `row(node_id)` gives `(node, IN_TABLE, first seen, last heard, failures)`, +`(node, DROPPED, first seen, when dropped, 0)` or `(node, BANNED, None, ban end, 0)`, or None; `track(changed)` +sets the listener told, under the table's lock (`lock`, an RLock), each node whose row changed; `restore(rows)` +loads saved rows into an unused table (times in the future clamped to now, a ban to at most `ban_time` ahead; +expired drops and bans, malformed rows, our own ID, nodes `admit` now rejects and a full bucket's youngest left +out) and returns the nodes whose saved rows must be written again or deleted. `routing_state.py` — +`RoutingState(table, state_dir)`, a `node.Store`: loads the `routing` table into the table, then saves the rows +changed through a SNAPSHOT `RowStore` taking the table's lock; it imports the state database so `routing.py`, +which lookups import, needn't. + +`providers.py` — `ProviderStore(state_dir, record_budget=1 GiB, *, ttl=PROVIDER_TTL, clock=time.time, rng=random, +max_per_key=1000, own_id=None, known=lambda id: False, free_space=None, record_bytes=None, monotonic=time.monotonic, +write_budget=WRITE_BUDGET)` (`state_dir` holds the +state database; None: a private in-memory one, `statedb.connect_memory`, so tests run the same code): +`store(key, publisher) -> bool`, `withdraw(key, publisher)`, `provide(publisher, keys, withdrawn) -> int`, +`retain(keep) -> int` (drops every record and the standing of publishers `keep` rejects: private mode's +non-members), `for_reply(key, exclude=lambda id: False) -> list[bytes]`, `providers(key, exclude) -> list[bytes]`, +`record(key, publisher)`, `expire()`, `maintain()`, `is_full()`, `record_count()`, `publisher_count()`, +`tier_counts()`, `usage()`, `first_seen(publisher)`, `standing(publisher)`, `save()`, `close()`. + +Records live on disk and are read per key. Nothing in memory grows with the records or publishers: +- `_cache`: the records of recently used keys (publisher -> (first stored, last stored)), LRU, at most + `CACHE_KEYS` (4096) keys and `CACHE_ROWS` (50 000) rows, read through `provider_records`' primary key (joined + with `provider_standing`, so a record of a publisher without a standing, or malformed, is dropped and deleted). +- `_dirty`: the records changed since the last save (None: deleted), by key, and `_dirty_by_publisher`; a key read + from disk gets them laid over it, so the cache can drop any key. `_pinned_standings` holds the changed standings + until the save, `_standings` an LRU of others (`STANDING_CACHE`, 4096). +- `save()` writes all of it in one transaction (NORMAL), then measures (`_measure`): the node's `maintain()` + call, every 30 s (`RECORDS_INTERVAL`), expires and saves; a store saves itself once `FLUSH_CHANGES` (20 000) + changes wait, and past `MAX_CHANGES` (saves failing) refuses others' new records rather than hold more. A refresh + writes nothing unless it moves the last-stored time by more than 6 hours (`REFRESH_WRITE`), or its day is a new + refresh day; a refusal writes nothing. +- The write budget (`_WriteBudget`, REV2), full or not: new records and withdrawals spend rows from pools refilled + on `monotonic` per `WRITE_WINDOW` (30 s, the node's save interval), `WRITE_BUDGET` = (60, 30, 30): the first + open to every tier, the others kept for Earned and Known, who spend their own first; each pool starts full and + carries at most `WRITE_CARRY` (720) windows, 6 hours. `_store` refuses a new record of another publisher when its + pools are empty, as `_FULL` (so `provide` stops looking for the rest of the batch), and spends a row once it is + added; `_withdraw` spends one if any is left, never refusing. Every record inserted is deleted at most once, so + the rows written stay within twice the budget. A withdrawn record is kept in `_withdrawn` until the save (each is + a deleted row in `_dirty`, so bounded with it): stored again before then, `_store` adds it back with its + first-stored time (its last-stored time moved only past `REFRESH_WRITE`) as a refresh, spending nothing. + Measured on a 1M-record database, a new record costs about 12 kB of writes (WAL and checkpoint) for one publisher's + and 36 kB for a fresh identity's (its standing rows too), so a Newcomer flood writes at most about 150 kB/s; + without the budget one identity's store-and-withdraw loop wrote 10-20 MB/s. + +Each publisher's standing (`_Standing`: first seen, records held, tier, refresh days and the last of them) keeps +the counts the budget and tiers need: `_tier_records` (records per tier, this node's in a tier of its own, `OWN`) +and `_publishers`, read from `provider_standing` at load (`SUM ... GROUP BY tier`), and changed with each record +(`_count_records`). A publisher left without records keeps its standing only if it has refresh days, until a TTL +after the last (`maintain`'s hourly `_forget_departed`), and at most `MAX_DEPARTED` (50,000, about 5 MB) of them: +right after the hourly pass's save, `_cap_departed` counts them on disk (by `provider_standing_tier`) and forgets the +excess, fewest refresh days first, then the longest unrefreshed (REV11: they were capped only by the TTL, so +identities refreshing one record a day later and withdrawing it grew them). The tier is worked out when the +publisher stores (`_tier_of`: `own_id`, then `known(publisher)`, then `EARNED_DAYS` refresh days); a contact added or +removed since is noticed when the publisher stores, when it is picked to evict from (`_largest` checks and moves it), +or by the hourly check of those written as Known. + +Room: `_capacity()` is `record_budget` over the bytes a record takes, or the records held at the last measure plus +the free disk beyond the reserve (`free_space()`; the node passes `shutil.disk_usage` of the state directory less +`storage.disk_reserve`) over it, whichever is less. The bytes a record takes are measured at each save as the +database's pages in use over the records (at least `MEASURE_RECORDS`, 10 000: below that the database is mostly +the rest of the state), so indexes, standings, fragmentation and the other state count; `record_bytes` fixes it +(tests). A full store (`_full`) first sweeps for expired records, at most once a minute; then `_reclaim` applies +the tiers' rules (FLOORS 10/60/30%), evicting from a tier's largest holder the record it refreshed least recently +(`_oldest_key`, by `provider_records_publisher`). A tier's largest holder (`_largest`) is the larger of the changed +standings' heap (`_heaps`, stale entries dropped as they surface) and the first standing on disk, by +`provider_standing_tier`, not changed since (`_disk_top`, a page of 64 at a time, kept until the save changes the +table); both, and `_oldest_key`'s position in each publisher's records, are reset by a save. So a flood's store +costs a few reads and O(log n) in memory: about 70 µs on a desktop, a third of them SQLite's. Once a batch's key is +refused for room, `provide` refuses its other new keys without looking again (NEW-8). + +A key keeps at most `max_per_key` publishers (`_key_room`): the K/2 oldest records stay, and a random one of the rest +gives way, by tier in an order the storer's tier and the Newcomers' share of the key (`KEY_NEWCOMER_SHARE`, 10%, +counted among the rest) set: a Newcomer takes Earned then Newcomer records while Newcomers are below their share, else +only Newcomer ones; an Earned storer Earned then Newcomer ones while Newcomers are within their share, else Newcomer +then Earned ones; a Known storer the same, then Known ones (REV3: Earned records never gave way to each other, so +1000 Earned identities could hold a key against all but Known publishers). +This node's records are never counted there, never evicted and never refused. Expiry (`_expire`) reads expired +records through `provider_records_last`, `SWEEP_BATCH` at a time with the lock let go between batches, and, while +full, Newcomers' records last stored 4.5 days ago (`_NEWCOMERS_EXPIRED`, joined with their standing; the tier +checked again before each goes). Times later than now (a clock corrected backwards) are clamped to now at load +(if the latest last-stored time is in the future) and at the first `expire()` after the clock went backwards: the +store notes the latest clock reading any time it holds came from (`_latest`, via `_now()`), and an `expire()` +with the clock not behind it skips the clamping pass. `_check_consistency()` recomputes the counts and standings +from the records on disk and in memory (`_merged`) and checks the caches against them; test_providers runs it +after every test. + +For the node's status (`usage()`, specification section 11 "Network status"): `_refused` and `_evicted` are a +`recent.RecentCounts` per tier (on `monotonic`, `time.monotonic` unless given), fed by `_refuse` (every `_FULL` or +`_REFUSED` `_store` returns for another's new record, by the storer's tier; also the keys of a batch refused without +looking for room again) and `_count_eviction` (`_evict_from`'s record and a full key's victim, by the evicted +publisher's tier as last worked out). `usage()` takes `_count()`, `_capacity()`, `_tier_records` and `_publishers` +under the lock and reads nothing from the database (`_reported_usage`): `bytes` is the state database's pages in use +at the last `_measure` plus `RECORD_BYTES_ESTIMATE` (180, measured about 171) for each record stored since, which is +what counts against `record_budget`; `capacity` is the records held plus the budget's room left (within the free +disk) at `RECORD_BYTES_ESTIMATE` a record, or the records held once full; each tier's `share` is its records over it. +`_capacity()` itself divides the database by at least `MEASURE_RECORDS` records, so while the records are few next to +the rest of rnfs's state (a million blocks' sizes take tens of megabytes) it is a fraction of the real capacity: +right for deciding when the database reaches the budget, wrong as a figure to show. With a fixed `record_bytes` +(tests), `bytes` and `capacity` are the records times it and `_capacity()`. `lookup.py` — network access is injected: `FindQuery = Callable[[node_id, key], tuple[list[bytes], list[bytes]] | None]` returning (providers, nodes). `Lookup(key, start_nodes, query, own_id, alpha, beta, k, timeout, is_known, on_answer, on_failure, exclude, report, -unreliable, may_retry, max_queries)` with `find_closest() -> list[bytes]` and `find_providers(on_provider, should_stop) --> list[bytes]`. Besides completion and the deadline, a run (`_iterate`) ends once it has sent `max_queries` -(`MAX_QUERIES`, 8 × K) queries and their answers are in ("query cap"), or after `STALL_WINDOWS` (3) × max(alpha, -beta) answers in a row that leave the bit length of the beta-th closest known distance (`_reach`) unchanged ("no -progress"). A run ranks the nodes `unreliable` named when it learnt of them (`doubtful`) after every other, for -its queries, its termination and `find_closest`'s result; one that distance alone would rank within the query -window is asked again when `may_retry` says so, as a query beyond `alpha` (`retrying`) that the run waits for -only if the node comes to rank within the window by `rank` (it is then counted in flight). `is_known(id)` -says whether RNS knows its identity; at most `alpha` unknown IDs are tried per lookup (`allow_unknown(id)`, also -used by the fetcher for provider IDs). `listing_nodes()` is the nodes that have answered with at least one -provider, and `listers(provider)` those that listed a given provider, so the keeper search can tell who vouches -for its candidates. +unreliable, may_retry, max_queries, on_useful, size, seniors, claim, taken, on_complete, unknowns, slow_path, +check_path, first, allowed, named)` with `find_closest() -> list[bytes]` and `find_providers(on_provider, should_stop) -> list[bytes]`. Besides +completion and the deadline, a run (`_iterate`) ends once it has sent `max_queries` (`MAX_QUERIES`, 8 × K) queries and +their answers are in ("query cap"), or after `STALL_WINDOWS` (3) × max(alpha, beta) answers in a row that leave the bit +length of the beta-th closest known distance (`_reach`) unchanged ("no progress"). A run ranks its nodes by `tier()`, +then distance (`rank`), for its queries, its termination, the checks' nodes and `find_closest`'s result: `FIRST` +(answered, or nothing against it), `SLOW_PATH` (`slow_path` said so and it hasn't answered in the lookup: `slow`, kept +as `_last_slow`), `UNRELIABLE` (`unreliable` named it when the run learnt of it: `doubtful`). `slow_path` (the node's +`mesh.SlowPaths`, given only on nodes that aren't slow) is asked once per node per lookup (`_is_slow`, cached in +`_slow`). `check_path` (the node's `_path_check`) is asked on the query's worker thread (`_send(check=True)`) before a +first-tier node that hasn't answered is queried, once per node (`_checked`): True puts `_SLOWER` on the run's queue +instead of an answer, and the run gives the query back (`queried`, `sent`, `in_flight`), adds the node to `slow` and +ranks again, so it is asked at once (unchecked) only if it still ranks among the nodes to query; False counts as a +failed query, not sent. An unreliable node that distance alone would rank within the query window is asked again when +`may_retry` says so, as a query beyond `alpha` (`retrying`) that the run waits for only if the node comes to rank within +the window by `rank` (it is then counted in flight). `named(id, by)` (the node's `PathRequests.named`, given by +`_wired_lookup` once Reticulum runs) is told by `_send`, before a query to a node charged to the unknown-ID budget +(`_allowed_unknown`), which node's answer first named it (`_named_by`, kept across runs and by `ask`). `is_known(id)` +says whether RNS knows its identity; at most +`unknowns` unknown IDs are tried per lookup (`allow_unknown(id)`, also used by the fetcher for provider IDs; `alpha` if +not given, never fewer). `listing_nodes()` is the nodes that have answered with at least one provider, and +`listers(provider)` those that listed a given provider, so the keeper search can tell who vouches for its candidates +(and the fetcher whose answers led to a provider that served). `on_useful(id)` (local scores): a run notes which node's +answer first named each node (`introduced`); when a node answers that is closer to the key than the node that named it, +that node is passed to `on_useful`, once per lookup (`_useful`, `_credit`). For sweep providing, `candidates()` gives +every node `find_closest` chooses from (`_choices`), with its tier (nodes named only by late answers judged by +`_is_slow`), plus the answers that came after the run ended (a run that completes leaves queries out; the workers keep +every answer about the key in `_replies`), and `ask(node, key)` sends one more `/find` about a nearby key within the +deadline, counted through `on_answer`/`on_failure` (a node still out at the deadline stays out of the candidates, +`_out`) and merging the nodes it names. The checks against identities placed next to a key (section 6) live in the run +too: `seniors` (the announce first-seen log's senior servers, added to the known nodes) and `size` (the size estimate, +which turns the density check on) make `required()` name the nodes that must answer besides the `beta` closest: the +`beta` closest seniors known, and `RING_ANSWERS` of each ring beyond a dense core (`density.rings` over the known nodes +by distance, ordered by tier; `dense` notes that it fired). They are queried among the window's nodes in rank order; +those sent outside the window (`extras`) count towards `max_queries` but never against the stall rule. `claim`/`taken` +(a disjoint lookup's) drop a node another path owns before querying it and leave such nodes out of replies. +`on_complete(key, kth)` is told the run's sample after each run that ended "complete" (`ended` holds how the last run +ended); the size estimate samples it. The sample is `density.verified_sample(known, verified())`: over the known +nodes by distance (failed ones are out of `known`), the run of those `verified()` accepts (answered in any run, or +`is_known`: a node still out at completion counts if known, as real servers often answer late), from the closest up +to the first it doesn't, at most K long, its last distance d_j scaled by (K − 1/3) / (j − 1/3); none under +`MIN_VERIFIED` (3). Made-up IDs a reply names just beyond the beta closest, left unqueried by completion, end the run +instead of counting (REV8); `known_distances()` and `verified()` give the same view to `DisjointLookup`. `first` (a +search's closest local servers, specification section 6 +"LoRa islands") are added to the first run's known nodes and launched before the window's while any is left unqueried; +a reply listing a provider empties them (`first.clear()`), and a later run doesn't have them (`_first` is emptied). +They skip `check_path`, and count as `extras` when outside the window. `allowed(id)` (partition mode's local-only rule) +is part of `_usable`, so nodes it rejects never join a run, and is asked again just before each query: a node rejected +since it joined is dropped then, as a claimed one is; `_choices` leaves out unanswered nodes it rejects. + +`density.py` — `SizeEstimate(state_dir=None, samples=16, minimum=8)`, a `node.Store`: `note(key, kth)` keeps the +latest sample per key, at most `samples` (an `OrderedDict`, least recent first); `estimate(key=None)` is (K − 1/3) +× 2^128 over the median of the samples of other keys, None below `minimum`. Saved as one blob (key and d_K, 32 bytes +a sample) in the `meta` table (`size_estimate`, `samples`) with the batched saves, only when changed; a malformed +blob is ignored. It imports the state database only when given a state directory, since lookups import this module +and `rnfs.statedb` loads RNS. `dense_core(distances, size)` (last rank j >= K closer than j × 2^128 / (T × size), +stopping past 2K once a rank lies 4 times further out), `radius(size)` and `rings(nodes, distances, size, count, +later)`: `count` from the rank after the core and from the first node beyond the radius, each ring ordered by +`later` (a lookup's tier), stably, without duplicates. `DENSITY_THRESHOLD` (2), `RING_ANSWERS` (5). +`verified_sample(distances, verified)`: a lookup's sample, d_j of the verified run from the closest (at most K, +none under `MIN_VERIFIED`, 3) scaled to rank K; the samples stay d_K's stand-ins, so the saved blob is unchanged. +`CheckCounts(clock=time.monotonic)`: `checked` and `flagged`, two `recent.RecentCounts`; `note(flagged)` counts one +check, `status()` gives the four `density_*` fields of the node's status. + +`disjoint.py` — `DisjointLookup(key, start_nodes, make_path, paths, seniors=(), on_complete)`: `make_path(start, +seniors, claim, taken)` builds each path's `Lookup` (the node's `_new_lookup`, so answers and failures still reach +the routing table and scores), dealt every paths-th start node and senior; `Claims` maps each claimed node to its +path under a lock. `find_closest` runs the paths at once (the first in the caller's thread, the others on +`rnfs-lookup-path` threads) and merges their results (the K best of their union by (tier, distance): each +path's own K best holds every node of the merged K best). `find_providers` runs every path on a thread, each +reporting into one queue; the caller's thread passes providers on (once each, `_reported`) and evaluates +`should_stop`, setting the paths' stop flag and waking them. `candidates()` merges the paths' (a node takes the +lowest tier a path knowing it gives it); `path_candidates()` each path's own; `ask` goes through the path knowing +the node; `remaining`, `allow_unknown` (the first path's budget), `listing_nodes`, `listers`, `closest_answered`, +`dense`, `ended`. It samples the size estimate once per call if every path completed: `verified_sample` over the +paths' `known_distances()` merged, a node verified if any path's `verified()` accepts it. + +`lookups.py` — `Lookups(new_lookup, start, sizes, seniors, paths=DISJOINT_PATHS, pace=None, publish_unknowns=None, +search_unknowns=None)`: the node's recipe. +`search(key)` is a `Search` whose first lookup has `size=sizes.estimate(key)` and `seniors=seniors(key)` and whose +fallback is `_disjoint(key, size, seniors, time left, queries left)` (the queries left, the first lookup's +`max_queries` less what it sent (`Lookup.sent`), split among the paths, and no fallback when that leaves a path +none); `publish(key)` is `_disjoint(key)` (no checks: the region's keys are checked one by one afterwards); +`flags(key, region)` runs `dense_core` over the region's nodes; `widen(key, region)` runs a `Lookup` with both checks +for the key and returns `K` of each ring from its candidates and the region's. `_disjoint` gives a plain Lookup for +one path. `Search(first, fallback, pace)` quacks like a Lookup: `find_providers` runs the first lookup (the fallback +once escalated), then, if no provider was ever reported, time and queries are left and `should_stop` is False, +waits for `pace()` (slow nodes: `Publisher.pace_lookups`) only while the wait fits in the time left, and runs the +fallback; providers are passed on once each across both. So a search sends at most `MAX_QUERIES` `/find` in all, as +before (DHT-1). `Lookups.checks` (a `density.CheckCounts`, its own unless given) counts how often the density check +ran and fired: `Search(..., checks)` counts itself once as checked after a `find_providers` or `find_closest` if its +first lookup has a `size`, and once as flagged once `dense` (any of its lookups' `dense`); `flags` counts every key it +checks. `DISJOINT_PATHS` 3, `SLOW_DISJOINT_PATHS` 2. The unknown-ID budgets: `publish_unknowns` for +`publish` (split among its paths) and `widen`, `search_unknowns` for a search's first lookup and again for its +fallback (split among its paths); `split_budget(total, paths)` deals a budget out, the first paths taking the +remainder, and `DisjointLookup` makes its paths in order, so `_disjoint`'s `make_path` takes the next share from an +iterator. `None` passes no `unknowns` (alpha per Lookup). The node passes `SLOW_PUBLISH_UNKNOWNS` (10) and +`SLOW_SEARCH_UNKNOWNS` (3) on slow nodes, `None` otherwise. + +`sweep.py` — no RNS. `find_region(lookup, key, own_id, members, size=None) -> Region` runs a `Lookup`'s +`find_closest`, sets the region's prefix length p from the farthest of the `K` closest found (0 with none); with +`size` (the network size estimate, read once the lookup has run; `Region.heard` is kept only then), p is at least +`size_bits(estimate)` +(⌊log2(N ÷ `K`) − `SIZE_MARGIN`⌋, margin 1), and while the estimate is `None` the region is `single` (the key alone: +`Region.__contains__`, nothing explored, its p not taken as `Sweep.estimate` nor by the publisher's network size). +When `members(p)` (due keys sharing it) is 2 or more, it calls `explore` with p as found (the far half of the region +the lookup found, which a narrower region's keys may reach into): breadth first over parts of the region, starting +with the far half, it asks the candidate nearest each part's centre (`ask`, by tier first) about that +centre; a reply naming `K` servers inside the part queues its two halves; a server outside the part hands over to +the one it named inside it (its bucket holding the part may have been full); a part gives up after +`REGION_FIND_TRIES` (3) unanswered; at most `MAX_REGION_FINDS` (7) in all. `Region` holds the key, p, the +lookup's result, every candidate learned and the finds; `closest_to(key)` ranks the candidates for any key in it +(`heapq.nsmallest` by (tier, distance); never fewer than the lookup found for its own key). `Sweep(keys, +estimate)` is one round's bookkeeping, under a condition: `take()` gives the next key in the given order that is +neither covered nor inside a claim and claims the keys sharing `estimate − 1` of its bits (everything before any +region is known, so the first lookup runs alone), waiting while every key left is claimed; `cover(region)` covers +the region's uncovered keys that `Region.knows` (a bisect over the sorted keys, then: at most `UNSTORABLE` (2) of +the key's `K` closest of `Region.heard` are in `Region.unstorable`, what the lookup heard of (`Lookup.heard()`, every +path's for a `DisjointLookup`: named in any answer, also refused unknown IDs, not failed ones) but `nodes` leaves +out as their identity isn't known; always for the key looked up, or when no more than that is unstorable at all, as +on a node that knows every identity; the rest are taken in turn), releases the claim and updates `estimate`; +`members`, `release`, `stop`. With disjoint paths, `find_region` also keeps each path's candidates +(`Region.paths`, from `path_candidates()`), and `Region.path_closest(key)` gives each path's K closest together; the +publisher fills `Region.rings` (key -> ring servers of its widened lookup) and `Region.deferred`. `Sweep.spend(limit)` +counts a widened lookup if the round's lookups (`lookups`: regions plus `widened`) are under `limit`. + +`scores.py` — `Scores(state_dir=None, seeded=(), clock=time.time, max_rows=MAX_SCORES)`, a `node.Store`: +`score(node_id)` (0 without a row), `band(node_id)`, `bands()`, and `served`, `useful_answer`, `failed`, +`bad_block` adding `SERVED` (1), `USEFUL_ANSWER` (0.5), `FAILED` (−1) and `BAD_BLOCK` (−32). A row holds the score +as of `updated` (it decays by half each `SCORE_HALF_LIFE`, 7 days, computed when read or changed: nothing runs +on a timer), `useful` (last raised) and `seeded`. Rows sit in an `OrderedDict`, least recently useful first: a +raise moves a row to the end, a row made by lowering goes to the front, and past `MAX_SCORES` (4096) the front is +forgotten. `seeded` (the node's `_bootstrap_set()`: `bootstrap_nodes` and the defaults in use) gives each without a +row `BOOTSTRAP_SEED` (6); a row seeded before whose node isn't a bootstrap node any more has the seed's decayed +share taken off (exact, as every contribution decays alike). Saved in `peer_scores` through a SNAPSHOT `RowStore`. +The node wires it: lookups' `on_failure` is `_query_failed` (the routing table's `failed`, then `scores.failed`), +`on_useful` is `scores.useful_answer`, and the join's unanswered bootstrap nodes go through `_query_failed` too; +the fetcher (`Fetcher(scores=)`, in memory only if not given) calls `served` for each response with verified +blocks and `useful_answer` once per fetch for each of its lookup's `listers` of that provider (`Fetch._served`), +`failed` in `_fail` and `bad_block` with the ban. + +`local.py` — local servers (specification section 6, "LoRa islands"), no RNS. `LocalServers(max_servers=1024, +max_judged=4096)`: `note(id, local)` (a judgement from the path to a server; an `OrderedDict` of judgements and one of +the local servers, least recently judged first, each capped), `replace(judged)` (a scan's judgements: the local set +becomes exactly those it found local, and earlier local ones it didn't see are judged not local), `forget(id)`, +`clear()`, `is_local(id)` (a dict lookup, no lock), `judged(id)` (True, False, or None if never judged), +`closest(key, count, exclude)`, `nodes()`, `len()`. `judge_all(paths, is_server, is_local)` judges a scan's +`PathInfo`s of the destinations that are known servers. `first_count(named)`: `LOCAL_FIRST` (1) when +`local_interfaces` names the island, else `LOCAL_FIRST_RADIUS` (3). `Lookups(local_first=)` passes those nodes as a +search's first lookup's `first`. `PartitionMode(is_local, slow, on_change, clock=monotonic, wall_clock, known, +on_probe, no_path)`: `active`, `since` (wall time), `entered`; `allows(id)` (not active, or local); `answered(id)` +(off-island: resets the failures, notes contact, ends it), `failed(id)` (off-island, `known(id)` and not +`no_path(id)`, a failure for want of a path: the `FAILURES_TO_ENTER`-th in +a row sets `_wanted`, a probe due at once, and calls `on_probe()`, or with no `_targets()` enters it), `announced(id)` +(a fresh off-island announce: a gap sample in `_gaps`, a deque of `GAP_SAMPLES`, only if no partition mode since the +last (`_clean`); contact; ends it); `silence_limit()` (None below `MIN_GAPS`, else max(`MIN_SILENCE`, +`SILENCE_FACTOR` × the mean)); `probe_due()` (active: `_next_probe` passed; else `_wanted`, or silence since +`_last_contact` over the limit); `probe_target()` (`_targets()`: the off-island servers of `_recent`, heard or +answered, most recent first, those not in `_unreached` (failed since) before those in it, each at most +`RECENT_OFF_ISLAND`, taken in turn; an ID only in `_unreached` is never one, REV4); `probed(id, answered)` (answered: +as `answered`; else outside partition mode it enters; in it the next probe is `PROBE_INTERVAL`, 300 s, later, or on +slow nodes `SLOW_PROBE_INTERVAL`, 900 s, doubling to `SLOW_PROBE_MAX`, 7200 s, reset at each entry); `no_target()` +(silence counts from now; if `_wanted`, enters it); `state()` (`partition`, `partition_since`). +`on_change(active)` runs outside its lock. + +`firstseen.py` — `FirstSeenLog(state_dir=None, clock=time.time, max_rows=MAX_FIRST_SEEN, +seniority=DEFAULT_SENIORITY, silence=DEFAULT_SILENCE)` (the node passes `provider_seniority` and 2 × +`announce_interval`), a `node.Store`: +`heard(node_id)` (the node's `_heard_server`, for every server-mode announce or path response, before the routing +table), `forget(node_id)` (`_heard_client`), `first_heard(node_id)`, `len()`, and `closest(key, count=K, min_age=0, +exclude, heard_within=None)`: the count closest to key of the servers first heard at least min_age ago (publishers +and searchers pass `provider_seniority`), heard within `heard_within` seconds if given, not excluded (`exclude` runs +under the log's lock). Memory: a dict node ID -> first heard, in the order first heard (loaded `ORDER BY +first_heard`), an `OrderedDict` node ID -> last heard, least recently heard first (a heard moves to the end; loaded as +the `last_heard` the save wrote), and the same IDs in a sorted list (`bisect.insort`, a memory move: about 7 µs for a +new server at 65,536, 1 µs for one heard before, 17 µs when evicting). Past `MAX_FIRST_SEEN`, 65,536, `_evict` +forgets `_victim(now)` until it fits (REV7): the front of the last-heard order if silent for `silence` since +`max(last heard, _loaded)` (`_loaded`: the construction's time), else the newest of the first-heard order if first +heard less than `seniority` ago (if the newest is senior, all are), else the front of the last-heard order; each an +O(1) look. With `seniority` 0 it is least recently heard first, as before the fix. The IDs sharing the key's first p +bits are a contiguous run of the sorted list, all closer to the key than any outside it: `closest` binary-searches the +narrowest run holding count IDs (`_run`, two bisects), then widens it a bit at a time until it holds count old +enough, and ranks only that run (about 20 µs at 65,536). Saved in `announce_first_seen` through a SNAPSHOT +`RowStore`: an announce only marks its row, and a row's `last_heard` is the time of the save writing it (within 30 s +of the announce), which orders forgetting and is the last-heard time after a restart. ## Network layer interfaces @@ -354,18 +682,107 @@ non-members (`_is_member_publisher`: this node, a publisher whose `/provide` the this run, or a node ID whose recalled identity is a member, as routing admission checks). In private mode the node also drops non-members' records at start (`ProviderStore.retain`). -`client.py` — `PeerClient(links, config)`: -`find(node_id, key) -> (providers, nodes) | None`, `provide(node_id, keys, withdrawn) -> set[bytes]` (identified -link, batched; the keys the node may have recorded: every key, or those of the batches up to and including the -first refused or unanswered one), `blocks(node_id, cids, on_progress=None) -> BlocksResult` (status OK / BUSY / -FAILED / STALLED; blocks verified against their CIDs; `.blocks: dict[Cid, bytes]`, `.missing` answered None, -`.unsent` cut by the budget, `.bad` hash mismatches, `.elapsed` from sending to the answer, which the fetcher -paces by), `link_mdu(node_id)` (an open link's, which the fetcher sizes batches by), `pin(node_id, cid, pin) -> bool`, `ping(node_id) -> float | -None`, `has_link(node_id)`, `close(node_id)`. `links.py` links only to the `rnfs.dht` destination of the identity -RNS recalls, and only if its hash is the node ID asked for (in private mode, also only if that identity is on -`members`); the idle sweep spares a link `LinkPool.get()` handed out within `link_idle_timeout`. - -`announces.py` — `DhtAnnouncer(destination, config)` sets `app_data` to the mode and runs the announce schedule; +`client.py` — `PeerClient(links, config)`: `find(node_id, key) -> (providers, nodes) | None`, `provide(node_id, keys, +withdrawn) -> set[bytes]` (identified link, batched; the keys the node may have recorded: every key, or those of the +batches up to and including the first refused or unanswered one), `blocks(node_id, cids, on_progress=None) -> +BlocksResult` (status OK / BUSY / FAILED / STALLED / LOST, unanswered as a lossy link loses a request +(`PeerClient._lost`: RNS gave up on it with the link up, or the link timed out under it, `links.timed_out`; or a new +link to a node with a path and identity wasn't established, `LinkPool.unestablished`), which the fetcher retries on a +new link; blocks verified against their CIDs; `.blocks: dict[Cid, bytes]`, `.missing` answered None, `.unsent` cut by +the budget, `.bad` hash mismatches, `.elapsed` from sending to the answer, which the fetcher paces by), +`link_mdu(node_id)` (an open link's, which the fetcher sizes batches by), `closed_by_node(node_id)` (this thread's +last `find` went to node_id and failed because it closed the link, `links.closed_by_other`: RNS's `teardown_reason` +naming the other end by its role, or `LinkPool.refused`, a link closed as it was established, which charges no path +request; a thread-local `(node ID, closed)`, read by the node's `_query_failed`, which then counts nothing, and by +`probe_off_island`, which takes it as reached; a `/blocks` on such a link is `BUSY` with `.reason` "closed", which the +fetcher counts as busy, never as pacing), `block_range(node_id, cid, held, max_bytes, +compress, block_size, on_progress) -> RangeResult` (0.2.0: the next range after the `held` bytes, checked by +`protocol.parse_range_response`; `.block` the whole block once a range ends it, verified; `.bad` if that fails; +`.silent` when RNS timed it out with the link up, as an older node leaves `/range`; counted in traffic only once the +block verifies, and a silence not at all), `pin(node_id, cid, pin) -> bool`, +`ping(node_id) -> float | None`, `has_link(node_id)`, `close(node_id)`. `links.py` links only to the `rnfs.dht` +destination of the identity RNS recalls, and only if its hash is the node ID asked for (in private mode, also only if +that identity is on `members`); the idle sweep (`LinkPool.sweep`) closes a shared link with no request in progress +once it has had no data, and hasn't been handed out by `LinkPool.get()`, for its idle limit (`links.idle_limit`: +`link_idle_timeout` if it holds a rate estimate, `PeerLink.rate`, or RNS gives no `Link.keepalive`; else at most +`Link.keepalive` less `KEEPALIVE_MARGIN`, 1 s, and two round trips, so before RNS's first keepalive, 0.2.0), and +not within `link_idle_timeout` of the end of a `hold`. The `rnfs-link-sweeper` thread sweeps when the next link kept +is due (`LinkPool.next_sweep`, at least `MIN_SWEEP_WAIT`, 0.25 s, at most a quarter of `link_idle_timeout` within 1 to +30 s), and is woken (`LinkPool.wake`) by each request on a shared link ending and each link `get()` hands out. +A request on a shared link that RNS fails (`RECEIPT_FAILED`, nothing of an answer started) while the link, still +active, received nothing since it was sent (`Link.no_inbound_for()` longer than the request took: `_went_dead`) closes +that link (not one opened since), unless another thread uses it or the caller expected silence (`silence_expected`: a +`/range` to a node not known to answer it, `PeerClient.block_range(answers=)`), 0.2.0: the far end lost it, which RNS +would notice only at its keepalive. The exchange is marked `went_dead`, and `PeerClient._lost` counts it `LOST` (asked +again on a new link, failing the provider only `max_failures` times in a row), not `FAILED`: over LoRa a lone lost +request looks the same. +Every link `LinkPool` opens gets an `EarlyResponses` (`PeerLink.early`, 0.2.0) in place of its `handle_response` +(RNS looks it up on the link): a response no `Link.pending_requests` entry matches is kept (at most +`MAX_EARLY_RESPONSES`, 16, for `EARLY_RESPONSE_LIFE`, 30 s), and `LinkPool._request`, once `Link.request` has +returned the registered receipt, hands RNS the one kept for it (`deliver`), both under one lock, so an answer that +beat its receipt (RNS registers it only after sending) isn't dropped. NomadNet fetches (`nomadnet.py`) install one on +their own link too (0.2.0). + +`paths.py` — `PathRequests(transport=lambda: RNS.Transport, reticulum=RNS.Reticulum.get_instance, clock, sleep, +spacing=SPACING, slow_bitrate=None)`, the node's path-request budget (spec section 3, "Path requests"): `LinkPool` makes +it (`LinkPool.paths`, over its own module's RNS, which tests replace) and the node hands it to `RnsNetwork(paths=)` +(names) and `network/nomadnet.fetch(paths=)`. `await_path(destination, timeout, cancel)` (a known path returns at once; +else a turn, then RNS's `Transport.await_path` with the whole timeout), `await_known(destination, known, timeout, +cancel)` (a turn, `request_path`, then `known()` polled every `POLL`: `LinkPool._recall`'s identity wait), +`send(destination)` (a turn, then `request_path`, unwaited: names), `charge(destination)` (`LinkPool._open` and +NomadNet's `open_link` on a link not established: a turn taken, and the destination counted as requested). A turn +(`_take_turn`) reserves `max(now, _next_turn)` and moves `_next_turn` a `spacing` on, so turns go in order of asking and +never closer; `ready()` (the path or identity arrived) or `cancel` end the wait with nothing sent, and a turn lost so +leaves a gap. At its turn a request goes unless `busiest_pr_frequency(get_interface_stats())` is over `CEILING` (1.5), +else it reserves the next turn, for `CEILING_HOLD` (15 s) at most. `_waits` keeps one owner per destination; later +callers wait on its event (polling `ready()`). `_sent` (an `OrderedDict`, at most `MAX_TRACKED`, older than +`PATH_REQUEST_MI` forgotten) makes a wait within `PATH_REQUEST_MI` of the last request poll for `timeout` without +sending. `lacked(destination)`: the last `await_path` or `await_known` for it ended without (`_lacked`, at most +`MAX_TRACKED`). `named(node_id, by)` keeps a scope for node_id's next request (`_named`, `HINT_LIFE`, 300 s; popped by +the request). `_scope` turns it into interfaces only with the node's own instance (`is_connected_to_shared_instance` +false) and a `slow_bitrate` (`LinkPool` passes the config's): the `next_hop_interface` of the path to `by`, if it is +fast (`_fast`: a finite `bitrate` at least `slow_bitrate`; slow or unknown: every interface) and among the interfaces a +request can leave by (`_sendable`: `OUT`, `online`, not `detached`, no `spawned_interfaces`, not a local interface type +nor in `local_client_interfaces`); a spawned interface (a `parent_interface`) widens to every sendable interface of that +parent. A scope covering every sendable interface is no scope. One interface: RNS's `await_path(on_interface=)`; +several: `request_path(on_interface=, tag=)` on each with one random tag, then polled. `scoped` counts scoped requests. +`_own_wait` records, for a request it sent and saw answered, the seconds from sending to the answer in `_responses` (a +deque of `MAX_SAMPLES`, 100) unless `measure` is false (`await_path`, `await_known`); `candidate_wait(path_timeout)` is +path_timeout below `MIN_SAMPLES` (20), else the `PERCENTILE` (0.95) of them within `PATH_REQUEST_TIMEOUT` (15 s, RNS's) +and path_timeout. `LinkPool.await_path(node_id, timeout=None)` passes `measure` only for a wait of at least +path_timeout; the node's `_path_check` (fast nodes' lookups only) asks it with `_candidate_wait()`. +Counters `requests`, `charged`, `joined`, `reused`, `held`, `waited` are for tests and measurements. `stretch()` (1 by +default; on a slow node `ChannelLoad.stretch`) multiplies `spacing` for each turn and `charge` (`_spacing`, read outside +the lock; a value that isn't a finite number above 1 counts as 1). + +`blackhole.py` (0.2.0) — `Blackhole(reticulum, clock, wall_clock, refresh=REFRESH)`, RNS's identity blackhole as +the node honours it (spec section 10, "Blackholed identities"), made in `RnfsNode._setup` over `self.reticulum` +(nothing listed without one). `refresh()` reads `get_blackholed_identities()` (through a shared instance one RPC), +keeps the identity hashes `listed(entries, now)` takes (16-byte keys whose `until` hasn't passed) and their node IDs +(`Destination.hash(identity hash, "rnfs", "dht")`, each hashed once while listed), swaps both frozensets whole, and +returns the node IDs newly listed; a read that fails is logged (throttled) and keeps the last list. The maintenance +pass (`RnfsNode.maintain`) calls `refresh_if_due()` (every `REFRESH`, 300 s) and removes each newly listed node from +the routing table. The checks, `node_listed` (`RoutingTable.blackholed`, so `is_banned` everywhere) and +`identity_listed` (`RequestServer.blackholed`: `_guarded` answers False for a request on a link identified as a +listed identity, after the rate limit, before private mode's check), read the frozensets only, never RNS. + +`channel.py` (0.2.0) — `ChannelLoad(reticulum, clock, poll=POLL)`, a slow node's radios' measured load (spec section 3, +"Measured airtime"), made in `RnfsNode._start_network` on a slow node only. `load()` reads +`reticulum.get_interface_stats()` (through a shared instance one RPC) at most every `POLL` (15 s), lazily: only a +caller asks (a path request's turn, a publishing round's end, a `/blocks` request, a refresh), and one caller reads +while the others use the last reading. `measured_load(stats)` takes the highest of each figure (`airtime_short`, +`airtime_long`, `channel_load_short`, `channel_load_long`) among online entries that aren't local interfaces, a figure +counting only as a finite number in (0, 100] (`_percent`: an RNode's 0.0 before its first measurement is unknown); +`Load.channel` is the short channel figure while known, else the long one. A stats failure is logged (throttled) and +reads `UNKNOWN`. Its hooks: `stretch()` (`stretch_for(load)`: 1 up to `BUSY_LOAD`, 25%, then `load / 25`, at most +`MAX_STRETCH`, 4) is `PathRequests.stretch`; `stretch_wait(wait, slack)` (`max(wait, min(wait × stretch, slack))`) is +`Publisher.stretch`; `over_budget()` (`airtime_long` above `AIRTIME_BUDGET`, 50%) is `AirtimePolicy.over_airtime`; +`airtime_share()` (`airtime_long / 100`, None if unknown) is `Publisher.measured_airtime`. + +`announces.py` — `DhtAnnouncer(destination, config, needs_reach=None)` sets `app_data` to the mode and runs the +announce schedule; a client announces periodically only while `needs_reach()` (the node's `_needs_reach`: +provides, owns names, accepts remote pins, private mode) holds, checked every `REACH_CHECK` (60 s), announcing at +once when it starts to; `DhtAnnounceHandler(on_server, on_client, is_banned)`. ## Node facade (used by api.py, gateway.py and cli.py) @@ -382,13 +799,18 @@ check goes. `verify_daemon` checks `X-Rnfs-Proof` before anything else of the an `503` is `Busy` and every other status `AuthFailed`, both in the client's own words, so a program on the port chooses neither the error nor its text; it reads at most `PROOF_READ_LIMIT` (64 KiB), and connecting and proving take at most `PROOF_TIMEOUT` (15 s, or the client's timeout if shorter), after which the request gets the client's -own timeout. A daemon that closes the proved connection gets the request on a new one only if the version it proved -is below 0.1.5 (`KEEPS_PROOF_OPEN_SINCE`; such daemons close every connection), else `ConnectionFailed`: a new -connection could reach another listener. That fallback goes in 0.1.7. `request()` maps failures to `DaemonNotRunning`, `ConnectionFailed` (`_exchange`), `AuthFailed` (a +own timeout. A daemon that closes the proved connection gets nothing more (`ConnectionFailed`): a new connection could reach +another listener. A daemon proved older than 0.1.5 (`OLDEST_DAEMON`; such daemons close every connection) is +refused by `check_daemon` before the token is sent; until 0.2.0 the client sent it the request on a new connection. `request()` maps failures to `DaemonNotRunning`, `ConnectionFailed` (`_exchange`), `AuthFailed` (a failed proof, or 403) and the API's statuses (`_raise_for`: `STATUS_ERRORS`, `error_for`, the one mapping); `Stream` reads a body, raising -`ConnectionFailed` if it ends short. Its methods (`status`, `add_path`, `add_stream`, `fetch`, `cat`, `info`, -`ls`, `stat`, `dir_set`, `dir_rm`, `export`, `import_archive`, `gc`, `verify`, `pin_*` (with `pin_remote`), +`ConnectionFailed` if it ends short. `ApiConnection(..., app=, token=)` (0.2.0) is an app's: its token is given +(never read again) or in `token_file` (read and checked as `api_token` is, `read_token`), every request carries +`X-Rnfs-App` (`APP_HEADER`, `_identity_headers`), and `verify_daemon` checks the proof keyed by +`app_proof_key(token)`; `rnfs.Client(app=, token=)` passes them on. Its methods (`status`, `add_path`, `add_stream`, `fetch`, `cat`, `info`, +`ls`, `stat`, `dir_set`, `dir_rm`, `export`, `import_archive`, `add_tar`, `get_tar`, `export_stream`, +`import_stream` (uploads bytes or a binary file, chunked when their length is unknown; downloads a `Stream`), `gc`, +`verify`, `pin_*` (with `pin_remote`, `pin_remote_ls`, `pin_remote_served`), `providers`, `ping`, `name_*` (with `name_watch`, `name_import`, `name_export`, `name_rm`), `submit_job`, `job`, `jobs`, `cancel_job`, `clear_jobs`, `deny*`) cover every core route (`ApiHandler.ROUTES`; checked by test_apiclient), so `remote.py` writes no path itself, and send only the parameters given; `ApiError` is an `RnfsError`, and each status's error a subclass of the @@ -400,50 +822,169 @@ subclasses it, turning each `ApiError` into `fail(str(e))`, so its messages are | Method | Returns | |---|---| -| `add_path(path, wrap=False, only_hash=False, pin=True, to=None, to_self=False, seal_name=True, chunker=None)` | `{"cid", "size", "warnings": [str]}` (+ `"sealed"`); file or directory, pinned `user` (or cached with `pin=False`) | +| `add_path(path, wrap=False, only_hash=False, pin=True, to=None, to_self=False, seal_name=True, chunker=None)` | `{"cid", "size", "warnings": [str]}` (+ `"sealed"`); file or directory, pinned `user` (or cached with `pin=False`); with `to`/`to_self` a directory is packed as one tar and sealed as `.tar` (`node_content.add_sealed_path`, `tarstream.folder_chunks`) | | `add_stream(stream, size=None, name=None, only_hash=False, …)` | as above; `name` wraps it as that entry (sealed: the sealed name) | -| `get(target, output, from_node=None, raw=False)` | `{"cid", "warnings"}` (+ `"warnings_total"` past 100; + `"sealed", "name"`, the name only if `safe_default_name` accepts it); fetches and writes the file or directory under `.rnfs-get-.partial` beside `output`, then renames it to `output` (`rename_new`), decrypting a sealed file unless `raw`; a directory output is first checked to still be the one `write_out` made (`walker.check_unchanged`) and, on Windows, made to inherit its folder's access list (`winacl.inherit_from_parent`; if that fails it stays private, with a warning logged) | -| `info(target, for_gateway=False, from_node=None, raw=False)` | `{"cid", "type": "file"\|"directory", "size", "executable"}` (+ `"sealed", "recipient", "stored_size", "name"`; the sealed name is None unless `safe_default_name` accepts it) | +| `add_nomadnet(link, wrap=False, only_hash=False, pin=True, to=None, to_self=False, seal_name=True, chunker=None, name=None, identify=False, progress=None)` | as `add_stream` (+ `"name"`, the snapshot's name, and `"link"`): a snapshot of what a NomadNet node serves at `link` (`node_content.add_nomadnet`, "Adding from NomadNet" below); recorded in pin_names as the link; `progress(received, total)` is an add job's | +| `add_tar(stream, name=None, only_hash=False, pin=True, chunker=None)` | as above (+ `"warnings_total"` past 100): the directory a tar stream holds (`POST /api/add/stream?format=tar`), built by `tarstream.build_tar` as it is read; at most `max_output_entries` files and directories; recorded in pin_names as `name` or `(tar upload)` | +| `get(target, output, from_node=None, raw=False, no_extract=False)` | `{"cid", "warnings"}` (+ `"warnings_total"` past 100; + `"sealed", "name"`, the name only if `safe_default_name` accepts it; + `"extracted"` for a sealed folder unpacked, `name` then the directory's); fetches and writes the file or directory under `.rnfs-get-.partial` beside `output`, then renames it to `output` (`rename_new`), decrypting a sealed file unless `raw` and unpacking a sealed folder unless `no_extract` (`node_content.get_sealed`); a directory output is first checked to still be the one `write_out` made (`walker.check_unchanged`) and, on Windows, made to inherit its folder's access list (`winacl.inherit_from_parent`; if that fails it stays private, with a warning logged) | +| `info(target, for_gateway=False, from_node=None, raw=False)` | `{"cid", "type": "file"\|"directory", "size", "executable"}` (+ `"sealed", "recipient", "stored_size", "name"`, and for a recipient `"sealed_folder"`; the sealed name is None unless `safe_default_name` accepts it) | | `stored_head(target, length)` | a file's first `length` bytes if stored, else None; never fetches (the user's own gateway `HEAD` and ranges not at 0); `Denied` as a read: a denied CID met, or while there are file hashes a file that fails `_check_file` (one not wholly stored gives None) | | `read(target, offset=0, length=None, for_gateway=False, from_node=None, raw=False)` | iterator of bytes of a file (range fetched only; sealed files decrypted unless `raw` or `for_gateway`) | | `read_file(target, offset=0, length=None, from_node=None, raw=False)` | `(info, chunks)`: `info()` and `read()` in one fetch, resolving the target once; `chunks` reads `[offset, offset + length)` cut to the info's size only once iterated (a directory's raises `ValueError`), and closing it ends the fetch (`GET /api/cat`, which used to call `info` and `read`: two fetches) | | `block(cid, for_gateway=False)` | the CID's own block, verified (`cat --block`, `?format=raw`: the gateway calls it only for the user's own GET) | -| `export_stream(target, start, for_gateway=False)` | fetches the DAG, then writes its archive to `start(exact length)` (`?format=archive`; a `start` returning `None` writes nothing). The gateway calls it only for the user's own GET (never for a limited request or a HEAD, stored or not), inside `fetcher.observing(ClientWatch)`, which cancels the fetch once the client disconnects | +| `export_stream(target, start, for_gateway=False)` | fetches the DAG, then writes its archive to `start(exact length)` (`?format=archive`; a `start` returning `None` writes nothing). The gateway calls it only for the user's own GET (never for a limited request or a HEAD, stored or not), inside `fetcher.observing(ClientWatch)`, which cancels the fetch once the client disconnects. Built on `export_chunks` | +| `export_chunks(target, for_gateway=False)`, `get_tar(target)` | `(info, chunks)`, as `read_file`: the DAG fetched and checked first (`node_content._fetched_dag`: `max_download_size`, `max_output_entries`, the deny list's file hashes), then `info` (`{"roots", "blocks", "length"}` / `{"cid", "length", "skipped"}`) and an iterator of the archive's or tar's bytes, exactly `length` of them; closing it ends the fetch (`GET /api/export/stream`, `GET /api/get/stream?format=tar`, which run the fetch inside `observing(ClientWatch)` as the gateway does) | | `ls(target, for_gateway=False)` | `[{"name", "kind": "file"\|"executable"\|"directory", "size", "cid"}]` | | `stat(target)` | `{"cid", "size", "blocks", "local_blocks", "local_bytes"}` | | `is_local(target)` | `bool`: everything needed is stored, so no network fetch | | `dir_set(target, name, child_cid, executable=False)`, `dir_rm(target)` | `{"cid"}`, pinned `user`; `dir_rm` refuses a target that names no entry (`edit.check_entry_target`: ValueError, before fetching anything), so the API and `rnfs.Node` don't check it themselves | -| `pin_add(target)` | `{"cid"}` once complete | -| `pin_rm(cid)` | `None`; NotFound if no `user` source | +| `pin_add(target, source="user")` | `{"cid"}` once complete | +| `pin_rm(cid, source="user")` | `None`; NotFound if `source` doesn't hold it | +| `source=` (0.2.0) | taken by `add_path`, `add_stream`, `add_tar`, `dir_set`, `dir_rm`, `pin_add`, `pin_rm`, `import_archive` and `import_stream`: the source their caller pins and releases with, `user` (the default: the CLI, an embedded node) or an app's `app:[:]` (`ApiHandler.pin_source()`; anything else is ValueError, `node._caller_source`). After pinning with an app's source the node checks the app still exists (`_app_still_there`): `app_rm` forgets an app before it looks for its pins, so a pin a request already let in makes after that is released there and then, with NotFound | | `pin_ls(cid=None)` | `[{"cid", "sources": [str], "complete": bool, "type": "file" \| "directory" \| None, "sealed": bool \| None, "name": str \| None}]`; `type` and `name` only when the root block is stored, `name` where the pin was added from (pin_names; for a sealed add, the local path or the name given, kept for the local user), its shared-folder entry, or a directory's single entry or entry count; `sealed` from the file's first bytes (`seal.MAGIC`), False for directories, None if the root or first chunk isn't stored. A complete pin's `type`, name from inside and `sealed` are read from its blocks once, then remembered while it stays complete (`_pin_details`, `PIN_MEMO` 16 384, forgotten when released). Callers that show pins to others must skip `sealed` entries' names and content | -| `pin_remote(node_id, cid, pin)` | `bool` accepted | +| `pin_remote(node_id, cid, pin)` | `bool` accepted; recorded (`AskedPins.record`) | +| `pin_remote_ls(node_id=None, cid=None, check=False)` | `[{"node", "cid", "pin", "accepted", "asked", "state", "done", "checked", "seen"}]`, most recently asked first; `check` runs `AskedPins.check_now` first (Remote pin status, below) | +| `pin_remote_served()` | `{"pins": [{"identity", "cid", "state", "reason", "at", "detail", "retry_at"}], "usage": [{"identity", "used", "quota", "pinned", "waiting", "allowed"}], "quota"}` | | `gc()` / `verify()` | `{"removed_blocks", "freed_bytes"}` / `{"checked", "bad": [cid]}` | | `providers(cid)` | `[node_id bytes]` | | `ping(node_id)` | round-trip seconds, or None | -| `name_create(alias)`, `name_import(alias, key_bytes)` | `{"alias", "name"}` | +| `name_create(alias)`, `name_import(alias, key_bytes)` | `{"alias", "name", "warning"}` (`warning`: how to back the key up, `names.key_backup_warning`) | | `name_rm(alias)`, `name_export(alias)` | `None`, key bytes | -| `name_list()` | `[{"alias", "name", "cid" or None, "sequence"}]` | +| `name_list()` | `[{"alias", "name", "cid" or None, "sequence", "exported"}]` (`exported`: the key was exported or imported) | | `name_publish(alias, target)` | `{"name", "cid", "sequence"}`, plus `"warning"` in private mode (the name's announce reaches outsiders: `names.PRIVATE_NAME_WARNING`) | | `name_resolve(name, for_gateway=False)` | `{"name", "cid", "sequence", "age"}`; NotFound if unresolved | -| `export(target, output)` / `import_archive(path)` | `{"roots", "blocks"}` / `{"roots", "incomplete", "blocks"}` | -| `status()` | (API: without the token, only `{"version"}`) `{"version", "node_id", "identity_hash", "dht_mode", "slow", "private", "known_nodes", "records", "pins", "provided", "blocks", "used_bytes", "max_storage", "names"}` plus every feature's `status()` fields (traffic, jobs, LXMF); `identity_hash` is the RNS identity hash (None on offline test nodes) | +| `export(target, output)` / `import_archive(path, progress=None)` | `{"roots", "blocks"}` / `{"roots", "incomplete", "blocks"}`; `import_archive` opens `path` with `O_NONBLOCK` and refuses (ValueError) anything but a regular file, so a FIFO can't hold it (or an import job) | +| `import_stream(stream, label="(archive upload)", progress=None)` | as `import_archive`, of an archive read from `stream` to its end (`POST /api/import/stream`; `import_archive` opens the file and calls it). `progress` here and in `add_path` is told each read's length (`builder.BuildOptions.progress`, `builder.counted`) and may raise to stop: an add or import job's `Job.read` | +| `app_add(name)`, `app_ls()`, `app_rm(name)` | `{"name", "created", "token"}` / `[{"name", "created"}]` / `{"released"}`: `AppRegistry` (rnfs/apps.py); `app_rm` revokes the token, calls every feature's `app_removed(name)`, releases every `app:[:]` pin, then deletes the row | +| `status()` | (API: without the token, only `{"version"}`) `{"version", "node_id", "identity_hash", "nomadnet_identity_hash", "dht_mode", "slow", "private", "known_nodes", "records", "pins", "provided", "provided_entries", "provide_entries_off", "provide_entries_capped", "blocks", "used_bytes", "max_storage", "allocated_bytes", "allocation_unit", "names", "unexported_names", "announces_held", "clock_offset", "local_servers", "partition", "partition_since", "network", "record_store"}` plus every feature's `status()` fields (traffic, jobs); `identity_hash` is the RNS identity hash (None on offline test nodes); `nomadnet_identity_hash` the hash of `storage/nomadnet_identity` (`add --identify`), made by the first `status()` if there is none and kept, None if it can't be read or written; `network` is `_network_status()` (`sizes.estimate()` rounded, `len(sizes)`, `lookups.checks.status()`), `record_store` `records.usage()`: counts in memory, no read of the state database (specification section 11, "Network status") | `RnfsNode.offline(config, node_id, client, names_network)` builds a node without Reticulum or background threads, for tests (tests/test_node.py). The request server's backend is `node.backend` (`NodeBackend`). -`fetcher.py`: `Fetcher.open(root, names, from_node, pinned)` gives a `Fetch` context (a protection session) -with `resolve()`, `fetch_dag()`, `fetch_listing()`, `read_range()`; `publisher.py`: `Publisher.refresh()` -(recompute the provide set, withdraw at once; skipped when the node's generation, the deny list, the key -sources' keys and the strategy are as at the last one), `publish_due()`, `serves(cid)`, `run()`. The node calls the -first `refresh()` only after the features have registered their key sources, so a restart doesn't withdraw -kept names' keys. `publish_due` republishes `DhtKey`s every max(effective `republish_interval`, 3 days) -with a per-key ±10% jitter (capped below `PROVIDER_TTL / 2`), drops a withdrawal of any key it provides to -the same node in that round, runs up to `PUBLISH_LOOKUPS` (4) lookups at once, keeping the batches one at a -time would build, and on slow nodes runs one lookup at a time, at most 10 per round, waiting 30 s × the -round's lookups before the next (never-published items first, then the longest unpublished). When -`refresh()` finds a key provided again, it cancels that key's queued withdrawals (their nodes join its earlier -nodes) and re-records the node's own record, and a round sends only withdrawals still queued when it reaches -each node, so a withdrawal queued while the key was briefly unprovided never follows a store of it. -`protocol.provide_batches` puts withdrawals before stores, and a key in both lists is only stored. +`fetcher.py`: `Fetcher.open(root, names, from_node, pinned, hint)` gives a `Fetch` context (a protection session) with +`resolve()`, `fetch_dag()`, `fetch_listing()`, `read_range()`; `publisher.py`: `Publisher.refresh()` (recompute the +provide set, withdraw at once; skipped when the node's generation, the deny list, the key sources' keys, the +strategy and `_entries_input()` are as at the last one), `publish_due()`, `serves(cid)`, `run()`. The node calls the +first `refresh()` only after the features have registered their key sources, so a restart doesn't withdraw kept +names' keys. **Entries** (`provide_entries`, `_choose_entries` inside `_compute`): the entries of the provided CIDs +that are pins of the user's own content (`Storage.own_pinned_cids()`: CIDs one of whose sources passes +`storage.is_own_source`, so `user`, `folder`, `name:` and `app:`, not `remote:`, `follow:` or `deny`; memoised like +`pinned_cids()` until any pin's sources change; read once per compute, among the complete undenied DAGs) are found +once per pin +(`_walk_entries`: `Storage.dag_entries`, found by the walk `dag_blocks` made for the served set, so no block is +read again; memoised in `_entries_of` while the pin stays complete, as a CID's content never changes) and chosen by +`_capped_entries`: pins in `Storage.pin_order` (first pinned oldest first, those of before 0.2.0 first, then by +CID), each pin's entries whole or not at all while the union stays within `provide_entries_max` (100,000 by +default), a pin that doesn't fit logged and skipped. The choice is reused while `own_pinned_cids()` is the same +object, the complete pins are equal and the cap is the same, so a refresh caused by a fetch or a block stored walks +and chooses nothing; `_entries_input()` holds both settings, so changing either (live, through the settings feature, +whose hook refreshes) makes the next refresh compute. Entries are `Cid`s in +`_provided` like roots, so publishing, withdrawing (refresh's removed set) and persistence treat them alike; +`_entries` holds those that aren't roots too (`provided_entries()`, `entries_status()` for the node's status). +`_compute` asserts they are inside the served set, which doesn't change. On a slow node with the setting unset, the +entries are dropped whole when `republish_airtime(roots + keys + entries, network_size(...), bitrate) / +interval` exceeds `ENTRY_AIRTIME_ON` (2%), or `ENTRY_AIRTIME_OFF` (2.5%) while they are on (`_entries_on`); the +reason is kept in `_entries_off` and logged when it starts or stops applying. While they are off, they also stay off +while `measured_airtime()` (the node's `ChannelLoad.airtime_share` on a slow node; None, unknown, by default) is above +`ENTRY_AIRTIME_ON` (`_measured_busy`, part of `_entries_input()`, so its change makes a refresh compute). `bitrate` +is the slowest counted interface's (`rates.slowest_bitrate`, read once at start; the node passes it and `_known_servers`, the larger of +the routing table's size and the density check's estimate `sizes.estimate()`), and the +network size `network_size(_region_bits_mean, known)`: a running mean (weight 1/16) of the prefix length of the +regions each round found (`_learn_network_size`, in `publish_due`), at least `known_servers()`, at least +`UNKNOWN_NETWORK` (1000) before any. Because that changes without any pin, `_entries_input()` holds the network +size and interval on such a node, and its `run()` loop refreshes at every wake (a no-op unless they moved). The mean +and `_entries_on` are the publish state's meta values (`publisher`, `region_bits` and `entries_on`), so a restart +keeps the decision rather than withdrawing every entry. `publish_due` +republishes `DhtKey`s every max(effective `republish_interval`, 3 days) with a ±10% jitter fixed by the key's first +byte (`key_interval`: 256 slots, so a region's keys come due together; capped below `PROVIDER_TTL / 2`), drops a +withdrawal of any key it provides to the same node in that round, and finds the K closest by sweep providing +(`_sweep` over `dht.sweep`): due items in urgency order (never published first, then the longest unpublished), one +lookup per region, up to `PUBLISH_LOOKUPS` (4) at once on `rnfs-publish-lookup` threads (`_sweep_at_once`), the +batches built in due order whatever order the regions finished in; on slow nodes one region lookup at a time, at +most 10 per round, waiting 30 s × the round's lookups before the next, or `stretch(wait, slack)` of that (the node's +`ChannelLoad.stretch_wait`; the wait itself by default), slack being `PROVIDER_TTL / 2` less the larger of the age and +the interval of the record this round doesn't republish that is nearest to it; `_pace_cap` (a full round's wait, or +the last stretched one) bounds `_next_lookup` when the wall clock goes back. Slow nodes pace withdrawals with them: a +server only withdrawals reach counts as a lookup (those with the most keys queued first; while any are queued, the +round's lookups are limited to leave up to 5 for them), the others stay queued, and `next_due` counts queued +withdrawals as due only once the pacing allows; other nodes send them all at once. `_region_bits` carries the last +region's p to the next round. `set_network_size(estimate)` (the node passes `SizeEstimate.estimate`) gives +`find_region` the size estimate leaving out each key's own sample; without it regions are sized by their lookups +alone. `lookups_at_once` (constructor; `PUBLISH_LOOKUPS` if `None`) sets the region lookups run at once, 1 for tests +that measure placement. Each item's store set (`_store_set`) is its K closest from its region +(`Region.closest_to`), then each part added by `add_store_part(part)`, a `(key, region) -> servers` called without +the lock after the lookups (the node adds `_senior_servers`: `first_seen.closest(key, K, min_age=provider_seniority, +exclude=banned or, in private mode, not a member, heard_within=2 × announce_interval)`, and on a node that may have +local servers with `local_replicas` above 0, `_local_replicas`: `local.closest(key, local_replicas)`, the same +exclusions), without duplicates; an item +whose region found no server is sent nowhere and retried after `RETRY_INTERVAL`. The store set also takes each +disjoint path's K closest (`Region.path_closest`) and the ring servers of a widened lookup (`Region.rings`): +`set_islands(islands)` (the node's `PartitionMode`, on a node with local servers): while it is active `_store_set` +keeps only the servers `allows` takes, the round's withdrawal snapshot leaves out the others (they stay queued), and +`send` skips a server it no longer allows (as if refused, nothing recorded) without opening a link; each `/provide` +of keys reports `answered` (keys possibly recorded) or `failed` (none: no link). At a round's end `_mark_heal` marks +each item it published (its row's `heal`) whose accepted servers are all local, if partition mode was on at its +start or end or its store set held an off-island server, and clears the mark of one that reached an off-island server; +`heal()` (the node's `_partition_changed` on leaving partition mode) sets the retry time of every marked item to now +(one `UPDATE` through the partial index on marked rows, and the same on the states memory holds), so it is due at +once (and paced on slow nodes as any round); an item marked by a round that began in partition mode and ended out of +it is made due again the same way. `heal_marked()` counts them (`COUNT(*)` over that index, memory's states over +theirs). `set_widening(flags, widen)` (the +node passes `Lookups.flags` and `Lookups.widen`) has each lookup thread, once its +region is covered, check the keys it covered (`_widen`) and run the widened lookup of each flagged one within the +round's limit (`Sweep.spend`; a key over it goes in `Region.deferred` and isn't published this round, so it stays +due). The round's result counts `lookups`, `finds` (extra `/find`) and `widened`; slow nodes' pacing counts the +widened lookups with the regions. `pace_lookups(count)` lets lookups outside publishing (a slow node's search +fallback) take their share of that pacing: 0.0 and `_next_lookup` moved on by `count` × `SLOW_LOOKUP_PACE`, or the +seconds to wait. The `/provide`s go to up to `PROVIDE_PARALLEL` (16) servers at once on +`rnfs-publish-provide` threads (one at a time, in the publishing thread, on slow nodes): `_in_order` runs `send` +(which reads the withdrawals still queued for the server under the lock, then calls `client.provide`, one identified +link and the batches in order) and yields the outcomes in server order, so the publishing thread records them, and +saves every 10 servers or 30 s, as one at a time would. A stop or an exception starts no more; those in flight +finish and are recorded before the exception is raised. When `refresh()` finds a key provided again, it cancels that +key's queued withdrawals (their nodes join its earlier nodes) and re-records the node's own record, and a round +sends only withdrawals still queued when it reaches each node, so a withdrawal queued while the key was briefly +unprovided never follows a store of it. `protocol.provide_batches` puts withdrawals before stores, and a key in both +lists is only stored. + +**The publisher's state on disk.** What the publisher knows of each provided item is a `publish_items` row rather +than memory (0.1.x and the 0.2.0 betas held every item's last publish and servers in memory: about 1.3 KB an item +after a restart, so 1.5–2M items on a 4 GB Raspberry Pi): keyed by its DHT key, with the item itself (a binary CID, or +`00` and a `DhtKey`'s key), `due` (when it is next published: its retry time if any, else its last publish plus its +republish interval, 0 if never published; indexed, the index holding the key after it), `at`, `retry`, `nodes`, +`earlier` (server and time, packed) and `heal` (a partial index over the marked rows). An item's state is an `_Item`; +`_items` holds those changed since the last save (`None`: its row goes) and those the round in progress holds +(`_round`), over the rows (`_read`, batched `IN` queries by key; `_peek` reads without keeping, `_get` keeps the state +to change it). `next_due` asks the index for the earliest due row memory doesn't hold changed (`ORDER BY due LIMIT n ++ 1`, n being what memory holds) and takes memory's own; `publish_due` saves first if refresh added items, then +`_take_due` reads the due rows in the index's order, at most `ROUND_KEYS` (50,000), memory's states over theirs, +and sorts them as before (never published first, then the longest unpublished, then by key). Items published in one +round share a due time, and never-published ones have 0, so the index hands them out in key order: a first round of +more than `ROUND_KEYS` keys is cut into slices of the key space, which come due again as slices, and the rounds need +no more region lookups than one would (100,000 keys on 2,000 servers: 78 + 80 lookups against 157, with or without +the network size estimate). A round holds +about 2 KB per item while it runs, then lets go (`_end_round`). The first refresh looks through every row (by key, +`SCAN_BATCH` at a time) for items no longer provided (`_unprovided_rows`), deleting rows whose item doesn't parse; +later refreshes compare the provide sets. Added items go in `_added` and get a never-published row (`INSERT OR +IGNORE`, so a restart keeps those there) in the save refresh then makes. Saves stay at the old durability points +(a refresh that changed the provide set, every `SAVE_EVERY_NODES` nodes or `SAVE_EVERY_SECONDS` of a round, the +round's end, stop), through `_PublishState`, a `RowStore` whose snapshot is the changed items' rows, the added +items, the round's stores, withdrawals and meta values; once written, the states it wrote that haven't changed since +leave `_items` (`written`). Within a round, item rows aren't touched until its end, which writes each item once; +the saves in between write only `publish_stores`, append-only: the round's item list once, then a row per server +that answered with the 4-byte places of the items it recorded, emptied in the round's last save (`_ROUND_END`), and +merged into the items' rows by the next start after a crash (`_merge_stores`; an item without a row has its stores +queued for withdrawal). Every batch is written in key order: SQLite's cache (2 MB) spills pages of a large +transaction to the WAL, and random order wrote each page many times (about 20× for a round's rows). Measured with +`tests/bench_publisher.py` (a 2,000-server fake network, 10 servers a key), at 100,000 and 1M provided CIDs: memory +213 and 161 B a key after a round, 187 and 158 after a restart (the provide set itself), against 632 and 582, 1,326 +and 1,287 before; a wake (`next_due`, and `publish_due` with nothing due) 0.1 ms either way, against 80 ms and +1.1 s; 290 B a key on disk (`publish_items` 258, its due index 30); a round writes 820–950 B a key (the WAL and the +checkpoint copy, SQLite's `secure_delete` on), against 55–83 KB before. Times: a clock gone back by more than `CLOCK_SLACK` (60 s) since the latest time seen or +saved (meta `clock`) brings `at` and `retry` after it back to it, at once on disk (`_recompute`: one `UPDATE` and +the due times computed again, writing only the rows that change) and in memory; earlier servers' times are brought +back as rows are read, and expired ones left out. A republish interval other than the one the due times were +computed with (meta `interval`) computes them again the same way. Withdrawals stay in memory, as before (they are +sent at the next round on fast nodes). Errors (`errors.STATUSES`, the one table: `api.ERROR_STATUSES` is it, `apiclient.STATUS_ERRORS` and the gateway's `ERROR_MESSAGES` follow it): `ValueError` → 400 (with `BlockMismatch`, `SealError` and `NotRecipient`, which @@ -454,11 +995,19 @@ handed the store not matching its CID: a bug, not a bad request) as a warning. P as a request body. **Core API routes** (`rnfs/api.py` `ROUTES`): `GET /api/status` (open), `/api/info`, `/api/cat`, `/api/ls`, -`/api/stat`, `/api/pin/ls`, `/api/providers`, `/api/ping`, `/api/name/list`, `/api/name/resolve`, -`/api/name/watch` (a long poll for newer records: Public API, API additions); +`/api/stat`, `/api/pin/ls`, `/api/pin/remote` (`node`, `cid`, `check`), `/api/pin/remote/served`, +`/api/providers`, `/api/ping`, `/api/name/list`, `/api/name/resolve`, +`/api/name/watch` (a long poll for newer records: Public API, API additions), `/api/name/syncs`; `POST /api/add`, `/api/add/stream`, `/api/get`, `/api/dir/set`, `/api/dir/rm`, `/api/pin/add`, `/api/pin/rm`, `/api/pin/remote`, `/api/gc`, `/api/verify`, `/api/name/create`, `/api/name/rm`, `/api/name/export`, -`/api/name/import`, `/api/name/publish`, `/api/export`, `/api/import`. `GET /api/info` and `POST /api/get` take +`/api/name/import`, `/api/name/publish`, `/api/name/sync`, `/api/name/unsync` (Name sync, below), `/api/export`, +`/api/import`; since 0.2.0 `POST /api/pin/label`, `GET /api/cache/ls`, `POST /api/cache/rm`, `GET /api/capabilities`, +`GET /api/events` and `POST /api/shutdown` (Local API for apps, below); app tokens (`ApiHandler.CLI_ROUTES`, the +CLI's token only: an app's gets 403 `forbidden`) `GET /api/apps`, `POST /api/apps/add`, `/api/apps/rm` +(`app_ls`, `app_add`, `app_rm`; `ApiConnection.apps`, `app_add`, `app_rm`, which neither `rnfs.Node` nor +`rnfs.Client` has: managing apps is the CLI's, `rnfs app`); for clients that share no files with the +daemon (a sandboxed app), `GET /api/get/stream?format=tar`, `GET /api/export/stream`, `POST /api/import/stream`, +and `POST /api/add/stream?format=tar`. `GET /api/info` and `POST /api/get` take an optional `from` (node ID), as does `GET /api/cat`; `GET /api/name/resolve?name=` takes a name or a name contact's nick. @@ -469,19 +1018,24 @@ prints the version and says why). The daemon writes a new random token to `/daemon.lock` (`fsutil.lock_exclusive`: `flock`, or `msvcrt.locking` of a byte past the text on Windows; the file holds its process ID) and keeps the lock until it exits, so a second daemon -on the same dir stops at once, naming the first (`lock_holder`). A start runs in this order, in the daemon +on the same dir stops at once, naming the first (`lock_holder`). The lock is taken before the node opens its block +store, and the API listens only after, so a client whose connection is refused checks the lock +(`apiclient.dir_holder`: the process `lock_holder` names, if `_process_alive`, and then `fsutil.lock_held`, a +shared `flock` (or the byte's lock on Windows) taken and released at once, writing nothing) and raises +`DaemonStarting(pid)`, a `DaemonNotRunning`, in place of "no daemon is running": `rnfs status` and every other +command say the daemon is starting. The CLI's `ApiClient` is given `--dir` for it. A start runs in this order, in the daemon (`cli.run_daemon`) and in `rnfs.Node` alike: the root (`node.make_private_root`), then the settings (`Config.load`), then the subdirectories and the lock (`lock_rnfs_dir`: `_make_private_dirs(root)`, then `lock_exclusive`), so each step relies only on what an earlier one secured. `make_private_root` refuses to start (RnfsError) if the rnfs dir belongs to another user or can't be made 0700. `_make_private_dirs` (audit API-1) -opens each of `node.PRIVATE_SUBDIRS` (`state`, `blocks`, `storage`, `names`, `lxmf`) with `O_DIRECTORY | -O_NOFOLLOW`, creating it 0700 if missing: a symlink, a non-directory or one another user owns is refused with an -RnfsError naming `chown`, before Reticulum starts; a loose one is `fchmod`ed to 0700 with a notice, and if its old -mode let others write, one `lstat` pass (`_warn_foreign_entries`) warns about entries another user owns (a -planted raw block would be trusted by its length). `rnfs.toml`, and RNS's `config` when `reticulum_dir` is the -rnfs dir, are opened by `config.open_checked` (`O_NOFOLLOW | O_NONBLOCK`, so a symlink is refused and a FIFO -never waited on) and checked through the descriptor: a regular file, the user's own, and if group or other bits -are set `fchmod`ed to 0600 (`fsutil.check_owned_fd`, which the identity's check shares); at most +opens each of `node.PRIVATE_SUBDIRS` (`state`, `blocks`, `storage`, `names`; not `lxmf`, the LXMF state of rnfs +before 0.2.0, left as it is for an LXMF app) with `O_DIRECTORY | O_NOFOLLOW`, creating it 0700 if missing: a symlink, a +non-directory or one another user owns is refused with an RnfsError naming `chown`, before Reticulum starts; a loose one +is `fchmod`ed to 0700 with a notice, and if its old mode let others write, one `lstat` pass (`_warn_foreign_entries`) +warns about entries another user owns (a planted raw block would be trusted by its length). `rnfs.toml`, and RNS's +`config` when `reticulum_dir` is the rnfs dir, are opened by `config.open_checked` (`O_NOFOLLOW | O_NONBLOCK`, so a +symlink is refused and a FIFO never waited on) and checked through the descriptor: a regular file, the user's own, and +if group or other bits are set `fchmod`ed to 0600 (`fsutil.check_owned_fd`, which the identity's check shares); at most `MAX_SETTINGS_SIZE` (1 MiB) is read. What was changed goes to `config.warnings` (a plain attribute, not a field, so `rnfs.toml` can't set it), never to `rnfs.log`, which imports RNS: the daemon prints them (`cli.warn`), `rnfs.Node` logs them, the CLI client prints them and `rnfs.Client` (`apiclient.api_port`) raises them as @@ -511,13 +1065,142 @@ malformed `Range` headers are ignored. Neither server sends CORS headers. Gateway pages run sandboxed in an opaque origin, so they load their DAG's files as subresources but can't read any gateway response with `fetch()`; an earlier rule allowing a gateway `Referer` never fired, as browsers send no `Referer` from an opaque origin, and allowing `Origin: null` instead -would let any site's sandboxed frame read the gateway. A separate origin per root (a subdomain per CID) is the -possible future way to let pages read their own DAG. +would let any site's sandboxed frame read the gateway. + +**Gateway sites** (0.2.0) give each root an origin of its own, so its pages can read their own DAG. +`GatewayHandler.handle_request` starts with `gateway.parse_host`, which takes the path gateway's hosts through +`api.host_allowed` (unchanged: the API's check stays exact) and otherwise only `