Repository navigation
Conversation
…XMF feature rnfs/uris.py now holds the rnfs: URIs (Link, parse_uri, format_uri) and the LXMF link-map field (encode_field, decode_field), and rnfs/lxmf_messages.py the import of 0.1.4's lxmf_inbox.json and lxmf_outbox.json into the lxmf_messages table with the record checks it applies. rnfs/features/lxmf.py uses both instead of its own copies. Behaviour-neutral: this prepares LXMF's removal from core, after which rnfs.Node's and rnfs.Client's rnfs: targets, the wire vectors of the LXMF field and the upgrade from 0.1.4's state files still need this code. The wire cases take the field from rnfs.uris, falling back to the LXMF feature on the old trees generate.py runs on.
RNS 1.2.5 and later limit the path requests arriving on each interface (8 a second, 3 on an interface under 2 hours old) and drop the excess, and rnfs paces none of its own: a publish or lookup burst can trip the limit at a hub, cost other users their path resolution, and count the dropped requests as server failures, partition mode included. 0.2.0 gains a node-wide path-request budget, documents RNS's limits in the spec, and re-runs the topology simulation with them modelled before the slow nodes' larger budget ships. Also planned for 0.2.0, all local: closing idle links before RNS's keepalives start (about 58 packets per idle link on a fast path), sending a path request only on the interface the node can be reached through when rnfs runs its own RNS instance, and remembering the providers each lookup found. One link for publishing, republishing less often on fast nodes and compressed /blocks responses are left open for the owner, each with its trade-off. Traffic classes and LXMF content destinations wait on Reticulum.
The LXMF wrapper app isn't ready, and the owner decided 0.2.0 removes LXMF from core anyway (roadmap "LXMF moves out of core"), in place of the deprecation path. Removed: rnfs/features/lxmf.py with its commands (send, inbox, inbox get/rm, outbox, lxmf address, lxmf sync), its /api/lxmf/ routes, the status fields lxmf_address, lxmf_inbox and untrusted_dropped, the "inbox" event kind, the library's lxmf methods and their objects (Lxmf, Fetched, LxmfAddress, MessageLink, Message, Outgoing, Sent, Synced), the ApiConnection, embedded and remote methods behind them, the refusal of `rnfs app add lxmf` and the start-up release of app:lxmf: pins, and the LXMF-specific deprecation code (config.LXMF_DEPRECATION and what used it). docs/openapi.json loses the lxmf routes, tag, schemas, status part and event kind. The lxmf_* settings join config.RETIRED_SETTINGS: one left in rnfs.toml (or given to rnfs.Node) is ignored with a warning, as max_records is, and `rnfs config set` refuses it as unknown. rnfs.Node's defaults no longer set lxmf_enabled. Kept: rnfs: URIs and the link-map field (rnfs.uris), which rnfs.Node and rnfs.Client targets and the wire vectors use; contacts of kind lxmf; app:lxmf: pins, now held only by an app called lxmf (adding and removing it releases them, tested in test_apps); the lxmf_messages table and the import of 0.1.4's inbox and outbox files into it (rnfs.lxmf_messages), which nothing else reads; storage/lxmf_identity and lxmf/, which rnfs no longer creates or tightens (node.PRIVATE_SUBDIRS), left for an LXMF app to adopt. The shared folder still refuses to scan lxmf/, which may hold LXMF keys. guards.py changes only in its docstring: the half-open limit on the LXMF delivery destination was applied by the feature itself. Tests: the LXMF feature's tests and its security regressions (L12, L12-R, L13, L13-R, FEAT-1 to FEAT-5, auto-fetch fairness, and the LXMF halves of NEW-2 and R-1) go with it, as do the LXMF internals in test_rns_compat and the LXMF scenario of tests/interop.py. The upgrade tests keep their LXMF expectations, read from the state database instead of the feature: 0.1.4's outbox records are now kept exactly as written, unfinished ones too, since nothing loads them. The subdirectory tests drop lxmf/ from what a start creates, and check it is left alone. New: tests/test_no_lxmf.py (no module imports LXMF; rnfs runs where it can't be imported), the retired settings in test_config and test_settings, and an e2e daemon started with lxmf_* settings in its rnfs.toml.
0.2.0 built a generic way to deprecate a feature for LXMF: Feature.deprecated and deprecated_since, the Deprecation header (RFC 9745) on its routes, capabilities' "deprecated" (Capabilities.deprecated, is_deprecated), the settings schema's "deprecated" (DEPRECATED_GROUPS, Setting.deprecated, `rnfs config get`'s line, GET /api/config's and POST /api/config/set's warnings) and the CLI's warn_deprecated with RNFS_NO_DEPRECATION_WARNINGS. With LXMF removed nothing is deprecated, so all of it was dead code, tested only through LXMF. It goes rather than stays: none of it was ever released (0.1.6 has neither the header nor the fields), so no app reads it, and a mechanism nothing uses would need tests of its own to stay honest. A later deprecation can bring back what it needs. docs/openapi.json loses the header, the "Deprecated routes" paragraph and the two fields; test_openapi now checks that no operation is marked deprecated, and test_api that capabilities has exactly its documented keys.
rnfs no longer imports LXMF, so pyproject.toml depends on rns alone, and nothing installs LXMF any more: the install script, the .deb, the Flatpak, the PyInstaller exe and the tests all install from packaging/requirements.lock. The lock was regenerated with packaging/update-lock.py, pinned to the versions it held (rns 1.5.6, cryptography 50.0.2, cffi 2.1.1, pycparser 3.0, pyserial 3.5): the only change is lxmf's entry and the command line in its header (lxmf depended on rns alone). The Flatpak's python3-deps.json was regenerated from it with packaging/flatpak/generate-sources.py (lxmf's wheel and requirement gone; build-sources.json unchanged). The PyInstaller spec no longer collects LXMF's submodules and sources, and the comments naming LXMF in build-deb.sh (and the .deb's copyright and description), the Flatpak manifest, install.sh, rnfs_entry.py (freeze_support stays: harmless, and needed should anything start a child process) and dependabot.yml are updated. test_packaging checks that neither pyproject.toml nor the lock names lxmf.
…CHANGELOG specification.md section 13 is rewritten: it defines rnfs: URIs and the LXMF link map as the interop contract an LXMF app and other clients use, says rnfs no longer sends or receives LXMF, and lists what rnfs's own LXMF left (app:lxmf: pins and how adding then removing an app called lxmf releases them, storage/lxmf_identity and lxmf/ left untouched, the lxmf_messages table kept for an app to import, the retired lxmf_* settings). The message, receiving and auto-fetch rules are gone, to be the app's. Every other LXMF mention follows: the pins, the reconcile at start, the half-open link limit, the threat table, privacy, the CLI and status, the private subdirectories, app tokens, capabilities (no `deprecated` or Deprecation header), events, contacts, and section 12's settings table, live list and retired settings. design.md drops the LXMF feature, its threads, locks, routes, objects and library methods, and the deprecation mechanism, and describes rnfs.uris, rnfs.lxmf_messages, the retired settings and what stays on disk. README removes LXMF usage and settings and says in one paragraph that LXMF moved out, what stays on disk and how to release the app:lxmf: pins. CONTRIBUTING's lock and interop lines no longer name LXMF. CHANGELOG 0.2.0 replaces "Deprecated" with "Removed" (what went and why, what stays, the retired settings, what users of rnfs send and inbox do now) and updates its introduction, upgrade notes and the 0.2.0 additions that named LXMF.
Each built section of the 0.2.0 plan now ends its heading with "(implemented)" and closes with an "As implemented" bullet: what was built where it differs from or adds to the plan, and the numbers measured (sweep placement, the eclipse and island simulations, record sizes on disk, startup times). The LXMF section is rewritten for its removal in 0.2.0, which replaced the deprecation path the plan described. The two sections still to do and "After 0.2.0" are unchanged.
Most CI failures on macOS and Windows were code written and run only on Linux, found weeks later at the first push. Calls such as os.geteuid, os.fchmod, os.mkfifo, resource, fcntl and the signals Windows lacks now fail ruff (TID251, the list in pyproject.toml) unless the line says what guards it: `# noqa: TID251 - <the guard>`. A new unguarded use fails on the machine it was written on. All 53 existing uses were checked and are guarded; each now says how. The test skip `POSIX_MODES and hasattr(os, "getuid") and os.getuid() != 0`, written out nine times, is tests.support.PERMISSION_CHECKS (and IS_ROOT).
Found by reviewing the 0.2.0 work, which hasn't yet run on CI: - test_tar_stream's test_skipped_entries made "back\slash" and "C:drive" on Windows too, which reads them as a path and as a stream of a file C, and a non-UTF-8 name on macOS, whose filesystem refuses it. They are made only where they can exist, as test_builder already does. - files_of (test_tar_stream, used by test_sealed_folders) keyed paths with the OS's separator while the expected dicts write "/", and test_sealed_folders expected an execute bit Windows doesn't keep. - test_name_sync linked to directories without target_is_directory, which Windows needs, and removed a link to a directory with os.remove. - test_nomadnet made a directory named with ":", which Windows forbids: no file can be named like a NomadNet link there, so it skips.
support.wait_for returns its condition's falsy value on a timeout unless it is given what=, so a call whose result nothing checks can't fail: five in test_announces and one in test_api waited up to 10 s and then passed whatever happened. They now name what they wait for, and so raise. tests/test_suite_rules.py fails on any such call added later. test_announces' check that a client stops announcing sampled its count after a fixed two intervals, which a slow runner can outlast while an announce is still under way; it now waits until a whole interval passes without one.
- test_api's slow upload paused 0.3 s against a 0.5 s body_timeout, and a macOS runner's scheduling ate the difference (a 408 as late as the last CI run). It now runs with a body_timeout of 3 s in a class of its own; its pauses still add up to more than request_timeout, which is what it shows. ConnectionTest's setup and helpers become ConnectionTestCase, shared by both. - test_remote_pins and test_publisher counted how many calls ran at once from calls that each slept 50 or 20 ms, so a runner slow to start threads saw fewer. Their fakes now hold each call until the expected number are inside together (up to 5 s): code that runs them one at a time still fails, only more slowly. - test_lookup gave three answering nodes 0.3 s, the time it waits out a hung one; now 2 s. - test_fetcher's ScoreTest compared scores to four places while they decayed on the wall clock; its scores now run on a clock that stands still.
CI's unit, end-to-end and coverage steps run through tests/ci_run.py, which runs the tests as before, then each that failed (up to 5) once more, alone, in a fresh process. One that passes then doesn't fail the run, but is named in a warning on the run's page and in its summary, to be fixed. A test that fails twice, more than 5 failures, a module that can't be imported or an unexpected success fail the run as before. A shared runner's hiccup in one timing-sensitive test has held up releases; this keeps it visible without blocking one. The tests workflow can also be run by hand on any branch, to try macOS and Windows before a pull request, and the unit tests step has a 30 minute timeout rather than the job's 6 hours. CONTRIBUTING.md gains "Tests on every OS": the paths, POSIX APIs, file and timing differences behind nearly every CI failure so far, the helpers for each, and pushing work in progress often, since a release that brings weeks of commits to macOS and Windows for the first time is what has made releases fail.
The owner wants 0.2.0 to be the last foreseeable feature release, so the "After 0.2.0" part moves into it: fewer name announces (to do once 0.1.5 and 0.1.6 resolvers are gone, since they would query keepers on every resolve otherwise), what waits on Reticulum (traffic classes, LXMF content destinations) and what isn't planned. The introduction says later 0.2.x releases are for fixes, the items still to do and those waiting on others.
Fewer name announces waits on 0.1.5 and 0.1.6 resolvers leaving the network, and traffic classes and LXMF content destinations wait on Reticulum, so none of them can ship in 0.2.0; they go under Future, with what isn't planned. 0.2.0 keeps everything buildable now and stays the last foreseeable feature release.
A read-only review of 0.2.0 (security against remote and local attackers, data efficiency and reliability, and what Reticulum offers) found work worth doing before the release. Five sections, in build order: security fixes (add refusing the rnfs dir, a write budget for provider records, full keys, forced partition mode, NomadNet answers in memory, gateway sites rendering uploads, the first-seen log, the size estimate, smaller ones); fetching reliably on LoRa (a lost packet dropping a provider, failed pins losing progress, remote pins, slow links judged per provider, range windows, clock problems); using more of Reticulum (the RNode's measured channel load, the identity blackhole, teardown reasons); scale (the publisher's state on disk, provide_entries' default scope and cap); and resumable, compressed block transfers in one new optional request, which settles the open compressed /blocks decision. Path requests RNS sends by itself join the pacing section, and links holding a rate estimate keep their idle time.
What is left of 0.2.0 is the roadmap's "(to do)" sections. The handoff says what to build and in which order, the rules (wire compatibility, upgrades, no attribution, nothing pushed), and what worked in the build so far: worktrees reset to dev, PYTHONPATH per worktree, integration branches, resuming agents after usage limits, the load-sensitive sweep test. The progress file points at it.
Records the order of the remaining work, the IDs given to the 0.2.0 review's security findings, and which agent builds which.
RNS 1.5.6 reads a response Resource without metadata (a page) whole into memory and unpacks it with umsgpack in Link.response_resource_concluded, before rnfs sees it. The fetch's only bound on its size was max_response_size: max_download_size, unset by default, and the free space above the disk reserve. So a NomadNet node could answer a page request with a "page" as large as the free disk and exhaust a Raspberry Pi's memory (about twice its size: the file read and its decoded copy). Transfer.resource_started now refuses the first segment of an answer without metadata whose advertised total size is over MAX_PAGE_SIZE (16 MiB), cancelling it before RNS requests any part and closing the link, as the other segment checks do; _outcome reports it as TooLarge. segment_refusal already makes every later segment carry the first's total and metadata flag, and resource_concluded keeps what is assembled within that total, so the first segment's advertisement bounds the page. File answers (with metadata) arrive as an open file and are copied in chunks, so they keep their bound.
tests/security/README.md gains the 0.2.0 review's section with the row for test_rev5_nomadnet_page_memory.py and its fix commit.
tarstream._add_member now asks _place where a member goes (making the directories it is or implies in the tree, warning about what is skipped) and builds only the file it is handed back. Behaviour is unchanged; the next commit's header scan places members by the same rules without building anything.
A request carrying X-Rnfs-Challenge and X-Rnfs-App: <name> got a proof keyed by that app's verifier when the app existed, and none when it didn't. The proof needs no token, and every response of the API's own carries it (the open status, 403s, 404s, 405s), so any local process could try names and read the installed apps off the header's presence. AppRegistry now makes a random secret when it is created (each start) and keys the proof for a name it has no app by with HMAC-SHA256(secret, name): the same status, headers, body and proof shape as for an existing app, the same proof each time within a start, one no client can check (the client's check fails as for a wrong token). proof_key computes the HMAC for every name, so the answer takes the same time either way (measured: about 1.2 us for both). test_api's test_proof_for_an_app and test_apps' registry test asserted that no proof or key is given for a missing app: that is the behaviour changed here, so they now assert an unverifiable 64-hex proof instead. tests/security/test_rev9_app_name_oracle.py is the regression test.
The docs, the OpenAPI reference and `rnfs app add`'s warning said an app's token can do anything the CLI's can "except manage apps". The /api/apps routes do refuse it (403 forbidden), but that is no boundary: an app's token reads files as the daemon's user (add, then cat) and writes new files wherever that user can (get, export), which is enough to run programs as that user and read api_token. Specification section 11, the README's apps section, the OpenAPI `app` scheme, the apiclient, api and apps docstrings, the 0.2.0 changelog entry and `rnfs app add`'s message now say an app can in effect manage apps too, and to give a token only to software one would trust with the CLI. test_cli's test_add_prints_the_token_once asserted the old "except manage apps" wording, which is what changed; it now asserts the new one.
…hem so (REV6) A site host's page keeps a real origin (CSP sandbox with allow-same-origin), so a file served there as HTML, SVG or XML runs script with that origin's storage. A site that holds files other people supplied let an attacker do that with a file of theirs: a link with ?filename=x.html set the type of any file, and a file whose path name gives no type was sniffed to text/html or image/svg+xml from its first bytes. On site hosts ?filename= now only names a download (Content-Disposition: attachment) and the type comes from the path's own last name, on the HEAD and later-range paths too. A file reached by a path inside the site's DAG is sniffed only to SITE_SNIFFED_TYPES (images, audio, video, woff2, wasm, UTF-8 plain text), anything else becoming application/octet-stream, so HTML, SVG and PDF are never sniffed there. It is an allow-list so a type sniff() learns later stays passive on sites. A site whose root is itself a file is sniffed as on the path gateway, HTML included: that file is its origin's only content, so it gains an attacker nothing, and a name can point straight at a one-page HTML file. A site's index.html and the files its DAG names .html or .svg render as before; the path gateway, whose pages run in an opaque origin, is unchanged. Regression test: tests/security/test_rev6_site_upload_render.py. Spec section 11 (Sites, Files), design.md, README and CHANGELOG updated.
…n ranges next, as after a stall
…at doesn't parse, not a RecursionError that stops every start
…as a Newcomer, not a NULL tier that fails every start
…ing, so one below -2^63 doesn't stop every start
…nding that doesn't parse is none, so neither stops the start, expiry or the saves
… whole, not kept for a server's row to index and stop the start
…n't a directory, and an empty one's file starts afresh rather than appending to one left behind
…changelog's promises
…ers that don't answer can't hold it for minutes
…ed as one that sent a bad block
… hours take ASCII digits only
…, so an upgraded node starts; config set refuses it
…ing status - Added `providers` and `searching` fields to job status in OpenAPI specification. - Updated job class to track the number of providers used and whether it is searching for more. - Enhanced job event reporting to include provider information. - Modified fetcher to report provider usage and searching status to observers. - Updated tests to verify the correct reporting of providers and searching status during job execution. - Introduced a new module for entry name validation to ensure compliance with naming rules. - Added `nomadnet_identity_hash` to status responses for better identification in NomadNet interactions.
…'s deeper parsing, slower runners)
… fails, so a get just after the add finds it
…ertions for job submissions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges dev into main for the 0.2.0 release. Once this is merged and CI passes on main, the release is published by pushing the
v0.2.0tag (CONTRIBUTING.md, "Releasing").The release notes are CHANGELOG.md's
## 0.2.0section. In short: the local API and library for companion apps, providing at scale, resistance to floods and to identities placed next to a key, support for LoRa islands, and new features (per-app tokens, encrypted folders, NomadNet add, gateway sites, name sync, remote pin status, network status). LXMF moves out of core.Compatibility: the only wire change is one optional request (
/range), and CIDs are unchanged. The state upgrade is automatic and one-way.This PR also runs the release builds (Flatpak, Windows exe, amd64/arm64 .deb, Debian 12 smoke test) on the 0.2.0 code for the first time, since it touches
packaging/.