feat: forward beta node logs from the daemon

[jacderida]

Linear issue

V2-1021 — https://linear.app/autonominetwork/issue/V2-1021

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Boundary check: this changes no node behaviour, no wire protocol, no stored-data format, no
payments/economics and no upgrade mechanism. The daemon reads node log files that already exist and
POSTs them to an HTTP endpoint. It never touches a node process, its arguments, its stdio or its
data directory — build_node_args is untouched, and enabling or disabling forwarding never restarts
a node.

Compatibility

• Wire: none. No change to the P2P protocol or to anything a node sends or receives. The only
  new network traffic is the daemon's own outbound HTTPS to the beta log endpoint, and only after
  the user opts in.
• Storage: none to any existing format. node_registry.json is unchanged — no new NodeConfig
  fields. Two new files are written, both created on demand and absent until the feature is used:
  config_dir()/log_forward.json (opt-in state and token, mode 0600) and
  data_dir()/log_forward_offsets.json (tail positions). An older daemon ignores both.
• API: additive only. Three new endpoints (GET /api/v1/logs/forward,
  POST /api/v1/logs/forward/enable, POST /api/v1/logs/forward/disable) and their OpenAPI schemas.
  No existing endpoint, request or response shape changes. One new CLI subcommand tree,
  ant node logs forward enable|disable|status; no existing command changes.

Semver impact

• breaking
• feature
• fix

Purely additive: a new module, a new CLI subcommand, new endpoints, and one new Error variant
(Error::LogForward). Nothing existing changed shape or behaviour.

Test evidence

Unit — cargo test -p ant-core --lib: 562 passed, 0 failed (107 new, in forward::). Covers
the config file (0600 permissions preserved across overwrite, corrupt-file handling, token never
serialized), line parsing in both ant-node log layouts, offset persistence and pruning, tailing
across rotation/truncation/partial writes, document construction against the index's field names,
queue bounding and per-position retry, and the bulk response classifier.

Integration — cargo test -p ant-core --test log_forward_integration: 8 passed. These run
against a real axum endpoint that reproduces the V2-1016 contract: HTTP 200 for a batch containing
failed documents, per-position items[].status, create semantics returning 409 on a repeated
_id, and the forced filter_path response shape. They assert:

• documents arrive framed as NDJSON create actions with Authorization: ApiKey,
  Content-Type: application/x-ndjson and the required trailing newline;
• tagging uses the mapped field names (@timestamp, service, binary_version, channel, os,
  arch), and host / beta_user are never sent;
• sub-INFO events are dropped client-side;
• a restart neither duplicates nor loses events — an event written while the forwarder was down is
  delivered exactly once, and no _id is written twice;
• a request that fails in flight is replayed and the endpoint answers the replay with a 409 rather
  than storing a second copy;
• documents are indexed by their own timestamp, verified with events either side of midnight;
• a daily rotation is followed into the new file;
• a panic plus its backtrace arrives as one document;
• enabling does not upload the pre-consent backlog.

Other suites, all passing: daemon_integration 6, node_add_integration 3, datamap_file 20,
merkle_unit 8, unit_self_encrypt 16, ant-cli 18.

Lint/format: cargo clippy --all-targets --all-features -- -D warnings clean;
cargo fmt --all -- --check clean.

Manual, on a fresh DigitalOcean droplet (Ubuntu 24.04, static musl build of this branch):
daemon starts, ant node logs forward status reports off, GET /api/v1/logs/forward returns the
expected JSON, and all three paths appear in /api/v1/openapi.json. Locally, against an isolated
XDG_CONFIG_HOME: enable without a token fails with the guidance message, enable --token writes
log_forward.json at 0600, and status prints a token fingerprint rather than the token.

Not run: the e2e_* suites, which stand up an in-process testnet with Anvil.

Note on a pre-existing flake: data::client::adaptive::tests::controller_perf_overhead_is_bounded
asserts 100k controller observations finish inside 500ms and fails under a fully parallel run on a
loaded machine. It is unrelated to this change — it fails identically with all 107 new tests
excluded (--skip forward::), passes in isolation at 0.12s, and adaptive.rs is untouched here.

New dependency

None. The implementation uses reqwest, tokio, serde, serde_json, futures, blake3 and
tracing, all already direct dependencies of ant-core.

ADR

n/a — Tier 1.

Mitigation / rollback

Forwarding is off unless the user runs ant node logs forward enable, so the blast radius before
opt-in is zero. After opt-in, disable stops it immediately and changes nothing else about any
node. The forwarder only reads log files, on its own task, with a bounded drop-oldest queue and
bounded retries, so a dead or slow endpoint costs dropped log lines and nothing more — it cannot
block, restart or slow a node. Reverting the two commits removes the module, the endpoints and the
subcommand; the two files it writes are inert to any other code path and can be deleted.

Notes for review

Two design points that are load-bearing and easy to miss:

1. The deterministic _id and the per-event index derivation are one mechanism, not two
   choices. _id uniqueness is per index, so each document is filed under an index derived from
   its own @timestamp rather than the wall clock. Deriving the index from the wall clock would
   send a batch replayed after midnight to a different daily index, where the duplicate would be
   accepted instead of rejected with a 409 — silently breaking the no-duplication guarantee.

2. The tailer holds a growing file's final event back for one poll so continuation lines can
   join it. That is what keeps a panic and its backtrace one document rather than twenty orphans,
   at the cost of one poll interval (5s) of latency on the tail of a burst.

Also worth knowing: node file logging is off by default, so enable forwards only nodes that
already have a log_dir and reports the ones it is skipping, pointing at --log-dir-path. Beta
onboarding docs need to carry that instruction or most users will forward nothing. Enabling before
starting a node captures its startup line (and so the version/commit/peer_id fields);
enabling afterwards joins the file at its end and skips it.

The beta endpoint was still being provisioned at the time of writing, so a smoke test against the
real logs.autonomi.com is outstanding. The wire contract itself is confirmed and is documented at
the top of ant-core/src/node/daemon/forward/es.rs.

[dirvine]

Code review — changes requested

Reviewed at 8658e16cbdaee8c2f2d7fb66d43498eb0f95c95a.

I found two blocking issues in the opt-in log-forwarding path:

1. Document IDs can collide across installations

document_id hashes only the local node_id, filename and byte offset. Those values are not globally unique: separate installations can each have node 1, the same daily filename and an event at the same offset.

All clients write to shared beta-nodes-YYYY.MM.DD indices, and HTTP 409 is treated as successfully delivered. A collision will therefore silently discard the later installation's event as an existing document.

Suggested fix: include a persistent, globally unique installation/source namespace in the document ID, then add a test showing that identical node/file/offset tuples from two installations produce different IDs.

2. Disabling forwarding does not stop in-flight delivery

stop() only cancels a token, while run_cycle() drains the queue without observing cancellation inside HTTP requests or retry backoffs. The disable handler returns after signalling cancellation rather than waiting for the sender to stop.

Uploads can consequently continue after the CLI reports that forwarding is disabled, potentially for multiple request timeouts/retries. For an opt-in telemetry feature, disable should provide a clear revocation boundary.

Suggested fix: propagate cancellation through queue draining, HTTP requests and retry sleeps; retain and await the forwarding task's JoinHandle; and test with a blocking sink that no request continues after disable returns.

Verification

• cargo test -p ant-core --lib forward:: --no-fail-fast — 107 passed
• cargo test -p ant-core --test log_forward_integration --no-fail-fast — 8 passed
• cargo test -p ant-cli --no-fail-fast — 18 passed
• cargo fmt --all -- --check — passed
• git diff --check origin/rc-2026.8.3...HEAD — passed

Current CI has passing format, Clippy, documentation, Ubuntu build and Ubuntu unit-test jobs. The Security Audit fails on RUSTSEC-2026-0042 (h2 0.4.14, fixed in 0.4.16); that dependency is inherited from the base rather than introduced by this PR. Several macOS/E2E jobs remain pending.

Independent review seats timed out without verdicts, so these findings are based on direct code inspection and the executed checks above rather than panel consensus.

[jacderida]

Thanks — both findings were correct and are fixed in 4bb4689.

1. Document ID collisions across installations

Confirmed, and it was worse than "can collide". Byte offset 0 of each day's file is reached by every node on every machine, so on any given day the first event from node 1 collided across the whole cohort and exactly one won. That first event is the startup line carrying version, commit and peer_id — the record the forwarding exists to collect.

IDs are now prefixed with a random 64-bit installation namespace, minted on first enable and persisted alongside the opt-in. Generated from random bytes rather than derived from hostname, MAC or username, so it separates installations without describing them. It is deliberately stable across restarts and token rotation — regenerating it would make a replayed batch look like new documents and duplicate them, which is the property the deterministic ID exists to provide.

Tests as suggested: identical node/file/offset tuples from two installations produce different IDs, plus an integration test driving two forwarders through the shared mock endpoint showing both events are stored rather than one being swallowed as a 409.

2. Disable does not stop in-flight delivery

Confirmed. Cancellation was only observed between poll cycles, so a disable issued mid-flush kept uploading through the remaining retry ladder — with the default policy up to three 30s timeouts plus backoff per batch, repeated for every queued batch — while the CLI had already reported it stopped.

Cancellation now reaches the delivery loop: checked before taking each batch, and raced against the delivery itself so the in-flight future is dropped and the HTTP request cancelled with it, discarding any pending backoff. ForwarderHandle retains its JoinHandle and stop_and_wait awaits the task, so the disable endpoint returns only once the sender has actually stopped. Starting a replacement forwarder awaits the old one for the same reason, so two never overlap.

Tested with a blocking sink as suggested: it holds a request open across a disable, stop_and_wait returns promptly, the blocked send is confirmed never to have completed, and no request starts afterwards. I checked the test earns its place — reverting the mid-delivery cancellation makes it fail.

Verification

570 lib tests, 9 log-forwarding integration tests, 6 daemon integration, 18 ant-cli — all passing. clippy --all-targets --all-features -D warnings and fmt --check clean.

The README's beta section now also documents the installation identifier and the disable guarantee, since both are things someone deciding whether to opt in should be told.

Two notes on the rest of your review

Agreed that RUSTSEC-2026-0042 (h2 0.4.14) comes from the base rather than this PR — rc-2026.8.3 carries it. Happy to bump it here if you'd prefer, but it seemed better not to fold an unrelated dependency change into this branch.

Separately, and not raised by you: the release-channel rule now exists in three places — install.sh, install.ps1 and ant-core/src/channel.rs — bound only by comments. Nothing tests that they agree. It's beyond this PR's scope but worth an issue before it grows a fourth copy.

[dirvine]

Re-reviewed at 4bb46899cc19518bf58440eb342deb8352f08084.

The two blocking findings from my earlier review are addressed:

• document IDs now have a stable, random per-installation namespace, with unit and integration coverage for equivalent local positions on separate installations;
• disable now propagates cancellation into delivery, drops an in-flight send/retry, retains the task handle, and waits for task exit before returning.

Local verification passed:

• 113 forwarding unit tests
• 9 log-forwarding integration tests
• 6 daemon integration tests
• 18 CLI tests
• workspace Clippy with all targets/features and -D warnings
• formatting and diff hygiene

I found two non-blocking robustness points: a hand-written/pre-field enabled config can restart with an empty installation ID until enable is run again, and the replacement worker is spawned before the old worker has exited, so the “two never overlap” wording is stronger than the ordering strictly guarantees. Neither creates a release blocker for this newly introduced feature, but both are worth tightening later.

The Security Audit failure remains the base-branch h2 0.4.14 advisory rather than a regression in this PR. macOS and E2E jobs are still queued/pending; this approval does not waive those checks.

Approved.