Skip to content

Own bounded static acquisition end to end #48

Description

@forkwright

Finding

Zetesis still does not own an enforced end-to-end static-acquisition mechanism. PR #78 landed the primitive safety floor on main at c922b43b909cf43179b2bab6730ca40e9f7edf0b, but the public Crawler boundary remains implemented outside the crate. Redirect validation, connect-to-validated-address behavior, streaming/decompression/extraction limits, and evidence-envelope construction are therefore still prose obligations for each consumer.

Primitive floor already landed

PR #78 now provides and gates:

  • immutable, externally non-forgeable ValidatedTarget values;
  • a separate non-serializable, non-cloneable local-target authorization capability instead of caller-controlled allow_local_targets data;
  • private PageContent fields whose constructors and deserializer share URL/content-type/body/text limits, without claiming post-allocation validation is streaming protection;
  • compile-fail doctests in the required hybrid gate.

Those invariants are no longer acceptance work for this issue and must not regress.

Remaining evidence

  • crates/sylloge/src/crawler.rs is still a public trait implemented outside the crate; its redirect and connect-time requirements are prose.
  • No crate-owned StaticAcquirer currently owns private no-auto-redirect transport, per-hop validation, validated-address connection, or bounded wire/decompression/extraction.
  • No versioned evidence envelope currently binds the hop chain, exact limit profile, raw/extracted digests, extractor identity/version, and deterministic schema-versioned fingerprint.
  • The required malicious-response and positive anonymous-public-GET fixtures do not yet exist.
  • Paid-provider authorization issue Make paid-research authorization atomic, identity-bound, and fleet-scoped #47 remains unrelated: anonymous static HTTP GET needs network authority and resource bounds, not paid-research budget approval.

Why this matters

Provider-controlled URLs and responses cross an SSRF and resource-exhaustion boundary. Leaving redirect handling, DNS pinning, decompression, extraction, and evidence identity to each consumer guarantees divergent implementations and makes a caller appear safe after only the first hop. Dioptron and Aletheia should consume one Zetesis producer rather than reproduce this state machine.

Remaining correction

Land one concrete, consumer-neutral StaticAcquirer whose public operation owns the entire state transition:

  • keep its one-hop transport private with automatic redirects disabled;
  • validate the initial target and every redirect before each hop, connect only to the validated address set, preserve the URL host for HTTP Host/TLS SNI, and impose a finite redirect limit;
  • stream and separately bound wire bytes, decompressed bytes, and extracted text; reject unsupported or ambiguous encodings and stop at the first exceeded limit;
  • return a versioned evidence envelope containing requested/final target, validated hop chain, fetch timestamp, response metadata, exact limit profile, raw-content digest, extracted-content digest, extractor identity/version, and a deterministic schema-versioned fingerprint;
  • execute malicious-response fixtures for retargeting, local redirects, DNS-answer changes, chunked overruns, decompression bombs, oversized extraction, and envelope/fingerprint stability in required CI;
  • prove with a positive fixture that anonymous GET to a public HTTP(S) target succeeds without paid-provider authorization or ledger, while local/private targets remain denied without the separate authority;
  • keep anonymous static GET independent of Make paid-research authorization atomic, identity-bound, and fleet-scoped #47 and expose the landed producer to Dioptron and Aletheia only through thin adapters.

Done when: the concrete acquirer—not a marker trait or empty shell—enforces this full path in crate-owned code, its positive anonymous-GET and negative security fixtures run in required CI, and consumers can persist the returned evidence envelope verbatim.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    auditCode audit findingsbugSomething isn't workingseverity:high

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions