Repository navigation
Conversation
Records the maintainer's answers to decisions Q2-Q16 (the recommended answers). Q1 stays open until the B benchmark has run.
The pinned flatpak-builder action restores and saves an Actions cache by default. Any default-branch workflow run can write that cache, so a poisoned .flatpak-builder directory could reach the attested release bundle. Set cache: false (and drop the now unused cache-key), and add a test_packaging guard that every flatpak-builder step sets it.
tests/fixtures/rnfs-0.1.5 is an rnfs directory as v0.1.5's own code leaves it: the 0.1.4 fixture upgraded by 0.1.5, plus the rows only 0.1.5 writes (local deny entries, a followed deny list pinned "deny", a remembered file hash). generate.py rebuilds it from git archive v0.1.5, and checks that 0.1.5 changes no row on a second start. expected.json holds every table's rows. test_upgrade_015 starts the current code on a copy and checks that the schema upgrades, quick_check passes, every 0.1.5 row is kept, nothing is set aside and a second start changes nothing. It also checks that the tables 0.1.5 created match a fresh database's. test_statedb pins a digest of the 16 released steps, so editing one in place fails (an upgraded database would never see the edit). Schema steps may now hold a callable taking the upgrade transaction, for a migration SQL alone can't express.
tests/fixtures/wire/vectors.json holds what 0.1.3 and 0.1.4 send for fixed inputs: /find, /provide, /blocks and /pin requests and responses (with the True/False reply), keepers' /name request and response, the rnfs.dht announce mode, name records at sequence 0, 1 and 2^64-1, and the LXMF field. generate.py runs the shared wire_cases.py on each release tree in a subprocess and checks that they agree byte for byte (0.1.5 agrees too). test_wire checks that the current code encodes each vector's inputs to its bytes and parses the bytes back to the inputs. Until now nothing caught a change that still round-trips with itself, such as an int width or a field order.
The end-to-end step ran only test_e2e and test_client. test_embedded's HostProgramTest (the only real-RNS test of rnfs.Node) and the real-RNS security regressions (test_h1_real_rns, test_m1_real_rns and test_h2_display_page_pins) were gated on RNFS_E2E but never selected. List them all and raise the step's timeout to 20 minutes. A test_packaging guard checks that every module reading RNFS_E2E appears in the step.
Enable ruff's BLE001, E402 and RUF100 together (RUF100 alone flags the noqa comments for rules that weren't on), and lint the companion NomadNet pages (index.mu scripts) as Python. Each of the 63 broad excepts in rnfs and the companion pages now carries a noqa: BLE001 with its reason. Tests get a per-file BLE001 ignore, so their four noqa comments become plain ones. Also: log.py's re-exported RNS levels move above its first statement (E402); two unneeded noqa: E402 and one noqa: BLE001 on an except that re-raises are dropped; and test_builder's noqa: B902 (a flake8 code ruff doesn't implement) becomes a comment. The rest of bugbear waits, as planned.
Add coverage to update-lock.py's build tools and regenerate the build locks with the existing tools pinned, so the only new entry is coverage 7.16.2 (build-backend.lock changes only in its header). A new Linux job runs the whole suite under coverage with RNFS_E2E=1 and prints the report, with no threshold. The lint job's comment now lists the rules TS10 added.
Response is shared by the request handler, the next request on the same link, the monitor thread, RNS's packet and Resource threads and link-closed callbacks, and its correctness rests on idempotence flags and on the order fields are assigned in. Write that model down in the class docstring: which fields are set once at construction or at handover (and in what order), which only the monitor writes, which move forward only, which are check-and-set under RequestServer.lock, and where a stale read is harmless. Documentation only; NET-1 changes how a rejected response is judged next.
An rnfs link in an LXMF message's field carries both the name and the CID the sender says it pointed at. When the name couldn't be resolved, inbox get quietly fetched the sender's CID and wrote it under the link's file name, and `rnfs inbox` never showed that CID: any sender, a stranger included, could pose as a well-known name. - _resolve(link, persist, allow_sent_cid) returns (cid, names, fell_back) and raises NotFound, naming --use-sent-cid, for an unresolvable name unless the fallback is allowed. A name that resolves always wins. - inbox_get(..., use_sent_cid=False) resolves names while planning, so nothing is written and no job submitted when one fails. With the flag, a link that fell back gets a "warnings" entry in its result, which cli.print_warnings shows. - CLI: `rnfs inbox get --use-sent-cid`. API: optional use_sent_cid on POST /api/lxmf/inbox/get; without it, an unresolvable name is now 404 (approved default change, Q3). - Auto-fetch (trusted senders) keeps the fallback and records "sent_cid": true with the pin; the listing shows "sent as <cid>" for name links and "auto-fetch pinned (sent CID; name unresolved)". No wire change: the LXMF field and the URIs are untouched. The two existing _run_inbox_get tests build their args by hand, so they gain use_sent_cid=False (argparse always sets it now).
ask_keepers was 138 lines of six nested closures (established, add, enough, may_ask, next_batch, ask) sharing found, unknown, used, results and answered. That state now lives in a small _KeeperQuery class: search (find providers, rank them, record who vouches for each), next_batch, ask (run on a thread per keeper), settled and records. ask_keepers keeps only the batch loop. Behaviour is unchanged: the same calls in the same order, the same deadline, the same log line. keepers had no test file of its own: tests/test_keepers.py tests the query directly (the search and who vouches, batch order across sources, the unknown-keeper budget, which answers are kept, and when they settle a name). This is the refactor DHT-1's behaviour change builds on.
L17 cooled down a requester only when it rejected a response's Resource with nothing sent. One that took the first window of parts and then cancelled (on RNS 1.5.6 any receiver cancel reaches the sender as REJECTED) could ask again at once, so each round made the node read, hash and pack every block and build and encrypt a whole Resource for the cost of one window. _concluded now replaces _sent_any with _made_progress: the part sent is the more of Response.sent and the Resource's get_progress() x size, and a REJECTED response under NO_PROGRESS_FRACTION (a quarter) of its size starts the existing flat blocks_cooldown on the link and the identity (maintainer decision Q4). Unreadable progress or size still counts as progress. A FAILED response (the link closed) deliberately starts none: an honest reader that gives up early closes its links, and in private mode a cooldown would hold back its own identity. Honest 0.1.5 fetchers never call Resource.cancel(), so the only honest cancel is RNS running out of retries on a lossy path, after which the requester moves to another provider as for any refusal. No wire change. The spec's section 3 Cooldown bullet, threat table and Limits, and design.md, record the rule, the residuals, and the constraint for plan A1: a reader that gives up must close its links, never cancel a Resource. reject() moves from test_l17 into tests/security/_server_helpers.py for the new test_net1_cancelled_responses.py; test_l17 keeps passing, with only its docstring and a comment updated.
While the deny list had a sha256: entry, a file was hashed whole before any of it was sent or imported, and the hash read every occurrence of repeated content: six blocks (one 64 KiB chunk and five file nodes, each listing the one below 256 times) declare 2^56 bytes, so a 1-byte cat, a gateway Range request, ?format=archive or an import of the 128 KB archive hashed for about a year, holding a gateway slot all along. _check_file now takes a getter over the file's whole stored DAG. Before hashing a file whose hash isn't known, it refuses one declaring more than DENY_HASH_RATIO (16) times the bytes of its distinct chunks plus DENY_HASH_SLACK (64 MiB), as Denied (451), so _refusing collects what a refused import stored. The new walker.leaf_bytes counts those bytes from the file nodes only (each distinct node read once, each distinct raw child's size once, a claim over MAX_BLOCK_SIZE invalid), never reads a raw block, and stops once the bytes needed are met. cat, the gateway and a sealed get (_check_fetched) now run fetch_dag before the check unless the hash is known, so the check walks stored blocks, and max_download_size and the disk reserve apply as for get. get's write-out, export_stream and import_archive already had the whole DAG stored. Honest files repeat far less; the cost is that a highly repetitive honest file over 64 MiB (a zero-filled disk image) can't be read while there are file-hash entries (decision Q2). Spec sections 8, 10 and 12 and design.md describe the rule and the constants.
…ore real progress
The user's own HEAD, or a range not starting at byte 0, takes a file's Content-Type from its first bytes only if they are stored, through node_content.stored_head, which read them past the deny list: a HEAD sniffed a denied stored chunk (a user pin keeps its blocks), and a HEAD of a stored file whose content hash is denied answered 200 where GET answered 451. stored_head now wraps its getter (node_content.denying) to raise Denied for a denied CID, and while the deny list has file hashes it runs CONT-1's bounded _check_file over stored blocks before reading. That check counts the chunks the file nodes list, so it needs the file wholly stored: one that isn't is treated as not stored (None, typed by name), so a HEAD still never fetches. pin ls passes deny=False and is unchanged: it is the user's own list, and shouldn't hash every pinned file. Spec section 11's gateway bullet and design.md say so.
Sybil server IDs ground near a name's keeper key could answer /find slowly with ever closer IDs, so the lookup never completed. With fewer than 3 x alpha keepers the search never had enough candidates to stop, used the whole lookup_timeout, and the /name queries got 1 s: a keeper whose link takes seconds to set up was asked, but its answer dropped. Every keeper-fallback resolve, follow check and retry_missing for the name failed, each after blocking for over 10 minutes. - /name budget: each batch is waited for max(the lookup's time left, name_wait), name_wait = path_timeout + LinkPool's establish grace + 5 hops of RNS's per-hop link timeout + LinkPool's request deadline without an RTT: 87.5 s at the defaults. - Soft search budget: the search stops when enough() holds, or once candidates exist and lookup_timeout / 5 has passed. Candidates are still asked only once it stops, so reply order doesn't choose them. If the candidates run out unsettled, with batches left, a search that was stopped early (and some of whose candidates were asked) resumes for new candidates, within the lookup's deadline. - Lookup bounds (decision Q10): a run sends at most max_queries /find queries (8 x K, retries included) and ends once their answers are in, and ends after 3 x max(alpha, beta) answers in a row that don't shorten the bit length of the beta-th closest known distance. Over a simulated 2000-node network, 2500 honest lookups (alpha and beta 3) all completed, within 12 queries and at most 8 such answers in a row. No wire change: /name and /find requests and responses are as before; peers see fewer /find requests. Specification sections 6 (Termination) and 9 (Resolve) and design.md's lookup and keepers notes are updated.
…a HEAD of a denied file
…dget; lookups are bounded
write_out created a directory, then its children later by path, and never
checked the directory was still the one it made. Anyone who could rename
inside the output (a group member with umask 002 in a shared setgid
directory; on Windows, any user a folder's inherited ACL grants Modify)
could swap partial/sub for a symlink or junction in between, and get then
created the DAG's files, execute bit included, where they chose.
On POSIX (walker.FD_WRITE) the partial root is now made with mkdir, opened
O_DIRECTORY|O_NOFOLLOW, checked to be this user's, fchmod'ed to 0700
(setgid kept) and must then be empty. Entries are created relative to their
parent's descriptor (mkdir(dir_fd=), files O_CREAT|O_EXCL|O_NOFOLLOW), and
each directory is reopened from the root one component at a time and
checked against the dev/ino recorded when it was made. The stack holds
names and identities, not descriptors. The root gets its original mode back
when complete. On Windows the root is created with winacl.make_private_dir,
each file's final path must be inside it, and node.get restores inheritance
with the new winacl.inherit_from_parent before the rename (Q13; a failure
there is logged and the output stays private). node.get also checks the
partial is still the directory write_out made (walker.check_unchanged)
before renaming it, and doesn't delete whatever was put in its place.
A swap raises ValueError("output was changed while being written").
Single-file outputs, warnings and write order are unchanged.
tests/test_walker.py: the refused-names test patched open and os.mkdir by
path only; it now patches os.open/os.mkdir with dir_fd too and runs both the
descriptor and the path write-out. tests/test_get_partial.py: its write_out
stand-in passes the new keyword through. The l7 FakeAcl gains inherit().
rnfs.toml was read by path with no check off Windows: a 0666 file loaded silently, a symlink was followed, a FIFO was skipped and another user's file was used, although its settings choose the Reticulum config (and the programs it starts) and the API port the CLI sends its token to. Existing subdirectories of the rnfs dir were never checked either, so a user who could once write in a loose rnfs dir kept a symlinked, foreign-owned or 0777 blocks/, state/ or names/ after the root was tightened. - config.open_checked opens rnfs.toml with O_NOFOLLOW | O_NONBLOCK and checks it through the descriptor: a symlink, anything but a regular file and another user's file are refused; one others may access is made 0600 with a warning; at most 1 MiB is read. RNS's config gets the same check when reticulum_dir is the rnfs dir. - fsutil.check_owned_fd is the shared owner/permission check (POSIX mode bits, winacl.tighten on Windows); network/identity uses it too. - _make_private_dirs(root) opens state, blocks, storage, names and lxmf with O_DIRECTORY | O_NOFOLLOW, creating them 0700: a symlink, a non-directory or a foreign owner stops the start (RnfsError naming chown) before Reticulum starts; a loose one is made 0700 with a notice, and if others could write to it, entries another user owns are reported. On Windows existing subdirectories get the private DACL and links are refused. - A start runs root (node.make_private_root), then settings, then subdirectories, then the lock, in the daemon and in rnfs.Node. - ND6: warnings go to config.warnings (a plain attribute, not a setting) instead of rnfs.log, which imports RNS. The daemon and the CLI client print them, rnfs.Node logs them, rnfs.Client raises RuntimeWarnings. winacl.secure_settings, now unused, is removed. Tests changed with the behaviour: _make_private_dirs now takes the root, not a Config (test_node, test_cli, test_l7*), lxmf/ is among the created subdirectories (test_l7_windows_acl, test_l7r1), and test_l7r1's settings warning is read from config.warnings rather than the log (ND6).
…CONT-2, NET-1, DHT-1, FEAT-5)
… dead code (FT5)
Name collisions were handled three ways: two features defining the same
API route raised (on the first API request), a core route silently
shadowed a feature's, a feature request path equal to a core one was
silently dropped by the request server, and two features adding the same
path silently overwrote each other.
- The node now checks all of them when it loads its features, with one
error style, features.FeatureError("feature <name>: <problem>"), which
stops the node before anything runs. checked_classes() checks every API
route (features.api_routes(): GET or POST of a path under /api/ with a
handler, defined neither by rnfs.api.ApiHandler.ROUTES nor by another
feature); once the features are made, features.request_handlers()
checks their request paths ("/" and a name with a handler, neither one
of rnfs's own, CORE_REQUEST_PATHS, nor another feature's), and
NodeBackend.feature_requests() hands the request server those. The
FT4 load errors use the same style. No shipped feature has such a
route or path (tested).
- Lifecycle tests with DummyFeature: requests() reach the backend the
request server reads; start in order, stop in reverse, save in order,
close in reverse; one start() raising is logged and the others start
(the start loop moves into RnfsNode._start_features, unchanged); a
constructor raising stops the node and closes those made before it.
DeadlineFeature, a DummyFeature subclass loaded beside it, no longer
inherits its API route, which the node now refuses as defined twice.
- Escaping: jobs printed the daemon's job IDs, states and counts raw
(--background, `jobs cancel`, `jobs clear`, the progress line and the
Ctrl+C line); `rnfs peers` printed hops, failures and bad counts raw and
`rnfs stats` its totals, refusal reasons and counts. All now go through
printable. test_feature_output covers jobs, peers, stats and deny, and
FEAT-7's example and rnfs.Client log; the jobs, peers and stats cases
failed before.
- Dead or stale code: the redundant `from rnfs import fsutil` inside
inbox_get; the lxmf docstring calling LXMF optional (pyproject requires
it; the node only runs on without it); _api_sync reaching into
_require_network (now LxmfFeature.sync, as every other handler calls
the feature); jobs' getattr(config, "max_jobs", None) on a field Config
declares; contacts' @staticmethod _log (module-level, as elsewhere).
_RoundRobin.clear stays: tests/security/test_lxmf_auto_fetch_fairness.py
uses it.
…paths checked at load
With A1 an early-closing reader nearly always has its next window in flight, and ending the fetch closed that provider's shared link. Gateway readers close early all the time (browsers abort range requests when seeking), so any other fetch with a request on the same link, in progress or waiting for its /blocks slot, saw it fail and dropped the provider. A fetch now closes a link only when no other fetch is using it. LinkPool counts the threads with a request in progress on a link or waiting for a slot (PeerLink.users, entered and left around LinkPool.request); the new PeerClient.close_unless_used(node_id, own) tears the link down unless a thread not in own uses it. The fetch passes its own request threads (Fetch._threads), both when it ends (__exit__) and when NET-4 closes the slower copy of a duplicated batch. When another fetch uses the link, this fetch's request is left to finish: while the fetch goes on its blocks are stored as they arrive (as A1 does for late blocks) and a NET-4 loser's outcome is still not counted; after the fetch has ended they are dropped, which costs the provider the one extra window the plan allows. Nothing is cancelled: no Resource.cancel() (NET-1). FakeClient tracks the threads in its blocks() calls the same way. Spec section 7 item 7 says so. Tests: two reads from one provider over one link (the real PeerClient and LinkPool, RNS faked): one closes early mid-window while the other's request waits for the slot; the link stays up, the other read completes and its provider is neither failed, stalled nor busy, and no Resource is cancelled (this fails with the previous fetcher: the link was torn down). The existing test_closing_the_reader_tears_the_link_down still covers a link nothing else uses. NET-4: with another fetch on the slow provider's link, the link is left open and the slow request's stalled outcome is ignored. A1 numbers unchanged (48 MiB read_range on the simulated network: tmpfs 6.12 -> 4.73 s, links 75% -> 97% busy; NVMe 8.6-9.2 -> 6.0-6.3 s).
…k, end-game duplication
16-byte IDs written as hex (node IDs, names, identity hashes, LXMF addresses) were parsed by about eight hand-written checks with different rules for case, brackets and whitespace, so a fix in one left the others accepting different text, and in private mode the node and the link pool normalised the members list separately. rnfs/ids.py (stdlib only: config imports it, and the CLI and rnfs.Client load config without RNS) has parse_hex16(text, what, *, brackets, upper), its check is_hex16, normal_hex16, and the patterns HEX16 / HEX16_LOWER. api.parse_hex_id, names.parse_name, identity.parse_node_id, contacts.parse_hex, lxmf.parse_address and Config.validate's identity lists use it, with the same accept-sets as before (the API still refuses <brackets>; status codes are unchanged). config._HEX_ID stays as an alias. The persisted-row validators (storage._IDENTITY_HASH, jobs._NODE_ID, the lxmf inbox and link checks) use the patterns as they are, unstripped, so rows that loaded before still load and none newly does. Config.member_set and Config.remote_pin_set hold members and remote_pin_identities as frozensets of lowercase hex; the node, the link pool, the request server and _load_remote_pins use them instead of their own normalisation. The one narrowing the plan approves: parse_node_id used bytes.fromhex, which skips whitespace between digits, so a bootstrap_nodes entry such as "0011 2233 ..." was taken as a node ID; it is now skipped with the warning malformed entries already got.
"A node or name that missed two announces is gone or stale" was written as its own 2 in the routing table's silence limit, names.ANNOUNCE_VALIDITY, follow's and deny's STALE_ANNOUNCES, and the routing table's seeding at start. They are one rule of the specification (2 x announce_interval), so they now use constants.MISSED_ANNOUNCES; names.ANNOUNCE_VALIDITY stays as its alias, and follow's and deny's copies are gone. No value changes. KIND_NAMES, the names `ls` shows for dag.KIND_*, moves from node.py to dag.py next to the kinds; nothing imported it from node.py.
…ign.md (A1, A4, NET-4) The release checklist wants design.md to cover the fetcher's pipelining, and the items left it describing the old window-at-a-time read.
The behaviour changes someone upgrading needs to know (the settings and rnfs dir checks, the shared folder's guard, follow_quota, the deny-hash bound, the API default, the gateway and Fetch Metadata, the inbox line, install.sh, the Flatpak sandbox, the new state table) are listed once, with pointers to the full lines below.
…releases The release checklist asks for 0.1.6 against 0.1.5 and 0.1.4 nodes, and tests/test_e2e.py only runs daemons of the same code. tests/interop.py (a script, not a test module) starts a hub and nodes from this tree and from an old tree given with --old (a git archive extract), each run with its own tree first on PYTHONPATH and driven by its own CLI, and checks in both directions: versions, providers, get/cat of a file and a tree, honest transfers with default settings and with max_upload_rate set (no refusals, failures or bad blocks in the fetcher's counters or either log: NET-1, NET-2), names, keepers (publisher stopped, a fresh resolver gets the record from a keeper of the other version) and an LXMF link. It prints a PASS/FAIL table per old tree and exits non-zero on any FAIL.
The interop harness found 'publishing failed: StateDbError(... is closed)' at stop: the publisher was still connecting to a node that had gone, stop's wait for it ran out, the state database was closed, and closing the links last then freed it to save into the closed database. 0.1.5 stops in the same order, so this isn't a regression, only a race the harness's stopping order hits. Links are now closed before the threads are waited for, so a thread stuck on the network ends while the database is open; they are closed again at the end, for any opened since. Work still waiting on the network is abandoned at stop either way, as before.
With no Pi run, a model of an SD card (one flush at a time) showed the deferred design would pass the gate only in its syncfs form, which needs a Linux-only path, a fallback on stacked filesystems such as ecryptfs (where syncfs writes nothing back) and the power-loss recovery path. That is too large a durability change to rush into a security release.
Bump __version__, drop "(unreleased)" from CHANGELOG's 0.1.6 heading, point README's release links and file names at 0.1.6, and tick the release checklist's version item.
- test_blockstore: the S2 scan test wrote an upper-case copy of the DAG block's name, which on a case-insensitive filesystem (APFS, NTFS) is the block's own file, emptying it. Write that name only where it is a separate file. - test_packaging: install.sh's find_wheel joins with "/" on every platform (sh on Windows is Git Bash); expect that rather than os.path.join's backslash. - test_cont2_write_out_swap: import the POSIX-only resource module inside the one (POSIX-only) test that needs it, so the module loads on Windows.
main condensed the README. Its version is kept, with 0.1.6's README content carried into it in the same condensed style: the release links at 0.1.6, the Flatpak's narrower sandbox and its overrides (SUP-5), install.sh's provenance stop and the curl | sh trust note (SUP-3), the plain-http warning and the hardening drop-in (SUP-6), pipx's unpinned dependencies (SUP-4), CONTRIBUTING.md, follow_quota (DHT-3), private-mode name announces (DHT-4), the inbox line and --use-sent-cid (FEAT-2, FEAT-5), the deny-hash notice and bound (CONT-4, CONT-1), open()'s read-ahead (S9) and the badge's Python version. test_readme_pyinstaller_is_locked now checks the pin only where README gives one: the condensed README dropped the hand-built PyInstaller command.
On Windows the write-out goes by path (walker._PathOutput): a file created through a swapped-in link lands, empty, where the link leads, before its final path can be checked, and is left there unwritten, since removing it by path could be redirected too. The test asserted the target stays empty, which holds only where files are created relative to their parent's descriptor (POSIX). It now expects exactly one empty file on Windows. The get still fails, and leaves neither output nor partial.
TS1 runs these on every OS for the first time: - test_h1_real_rns stopped its daemons with SIGINT, which Windows can't send to one process. Start them in their own process group and send CTRL_BREAK_EVENT there, as test_e2e does (its POPEN_OPTIONS and INTERRUPT). - test_h2_display_page_pins' DisplayPageDaemonTest drives the companion page through a /bin/sh shim named rnfs on PATH; on Windows the page's subprocess finds the installed rnfs.exe instead, so it talked to the wrong node. Skip that class on Windows; the page's logic is still tested there by the module's other classes.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.