diff --git a/docs/cloud-setup/README.md b/docs/cloud-setup/README.md index 4bf6acb..82135f8 100644 --- a/docs/cloud-setup/README.md +++ b/docs/cloud-setup/README.md @@ -11,8 +11,8 @@ Each guide here covers what a provider administrator configures so dfe-fetcher can pull data from that provider: the access it needs, how to grant it least-privilege, where to put credentials, and how to verify the connection. -Every source reads only -- the fetcher never writes, modifies, or deletes -anything on the provider side. +Every source is read-side: M365 starts its activity-feed subscription and +Pub/Sub acknowledges what it delivered. Nothing else changes provider state. For the fetcher's own configuration (scheduler, output, credential resolution, the `sources.db` and `sources.file` blocks), see @@ -79,6 +79,3 @@ stable. | Alpha | Code-complete but not production-validated. Behaviour and config may change. The default for a new source. | | Beta | Validated against a live service; hardening in progress. | | Stable | Production-ready. | - -Promote a source in its profile rather than recording its stage in these -guides. diff --git a/docs/cloud-setup/gcp-pubsub.md b/docs/cloud-setup/gcp-pubsub.md index b0a1004..0300120 100644 --- a/docs/cloud-setup/gcp-pubsub.md +++ b/docs/cloud-setup/gcp-pubsub.md @@ -234,7 +234,7 @@ Common failures: was never created. - Pull succeeds but returns nothing - the sink writer identity was not granted `roles/pubsub.publisher` on the topic (step 3), the `--log-filter` matches no - entries, or the window is genuinely quiet. Verify routing with the Cloud + entries, or the window is quiet. Verify routing with the Cloud console's log-sink view. - `invalid SA key / missing client_email` - the key JSON is malformed or the wrong file. diff --git a/docs/internals.md b/docs/internals.md index 24ce43a..bc740d4 100644 --- a/docs/internals.md +++ b/docs/internals.md @@ -50,7 +50,7 @@ flowchart TB subgraph "DFE pipeline" direction LR Kafka[(Kafka topics)] - Receiver[dfe-receiver] + Receiver[dfe-loader, a transform or the archiver] end API --> Rest @@ -369,6 +369,8 @@ up for `stable_after_secs`. ## Container Communication +The HTTP mode listener requires `ingest.auth_token`: every `/ingest` call carries `Authorization: Bearer `. With no token the listener refuses to start unless `ingest.allow_unauthenticated: true` is set. The `/livez` and `/readyz` probes are exempt. + ```mermaid flowchart TB subgraph "stdout mode" @@ -477,13 +479,16 @@ another followed by `.`. ### Sidecar Patterns For log sources that established tools handle well, use container extractors -with Filebeat, Fluentd, or similar collection agents: +with a collection agent that can print JSON lines or POST JSON, such as Vector +or Fluent Bit. Filebeat has no HTTP output (its outputs are Elasticsearch, +Logstash, Kafka, Redis, file and console, per Elastic's output docs), so ship +Beats data to dfe-receiver's Beats listener instead. | Source Type | Sidecar Tool | Communication | |-------------|-------------|---------------| -| Syslog | Filebeat (system module) | HTTP POST to `/ingest/syslog` | -| Windows Event Logs | Filebeat (winlogbeat) | HTTP POST to `/ingest/windows_events` | -| Custom app logs | Filebeat (file input) | stdout JSON lines | +| Syslog | Vector or Fluent Bit | HTTP POST to `/ingest/syslog` | +| Windows Event Logs | A forwarder that POSTs JSON (winlogbeat is a separate Beat with the same output limits) | HTTP POST to `/ingest/windows_events` | +| Custom app logs | A shipper that prints JSON lines | stdout JSON lines | | Network captures | Zeek / Suricata | stdout JSON lines | | Cloud provider logs | Cloud-specific CLI tools | stdout JSON lines | @@ -500,8 +505,7 @@ exponential backoff. A log file the fetcher itself can reach is a Typical fetcher volumes are orders of magnitude lower than the loader / receiver / transform / archiver tier. A single fetcher pod servicing one tenant's combined AWS / Azure / M365 / GCP / SaaS audit feeds normally -moves tens to a few thousand records per minute. The downstream pipeline -moves PB/hour. The fetcher is also I/O-bound waiting on remote cloud +moves tens to a few thousand records per minute. The fetcher is also I/O-bound waiting on remote cloud APIs, not CPU-bound parsing or routing. What this means in practice: @@ -520,8 +524,7 @@ We still care about correctness, cancellation safety, bounded retries, and not leaking tasks - those are correctness concerns, not perf concerns. We also still prefer the simple, idiomatic version of any pattern over the clever one. But the aggressive hot-path discipline -documented in the hyperi-ai Rust standards and applied across -dfe-loader / dfe-receiver / dfe-archiver does **not** need to be +applied across dfe-loader / dfe-receiver / dfe-archiver does **not** need to be applied symmetrically here. This trade-off is deliberate. It keeps a source a YAML profile rather than diff --git a/docs/reference/db-drivers.md b/docs/reference/db-drivers.md index ac46cbb..fe80c41 100644 --- a/docs/reference/db-drivers.md +++ b/docs/reference/db-drivers.md @@ -113,10 +113,9 @@ to be mounted into the image; the spec mechanism does not fetch it. ### Forms that need a token minted per connection -These are unsupported today, and the reason is the same in each case: the +These are unsupported, and the reason is the same in each case: the credential is a token valid for minutes to an hour, and `connection_string` -resolves once. Supporting them means re-resolving the string per connect, which -issue #111 also asks for. +resolves once. Supporting them means re-resolving the string per connect. - **Azure Database for PostgreSQL** with Entra. The user name is the Entra principal and the PASSWORD is an access token, obtained out of band and valid @@ -206,4 +205,4 @@ the first tick was the last acknowledged row's. SQL Server and Oracle have the same proof, gated on their operator-supplied driver being registered with the host's driver manager: absent the driver the test skips saying so, in CI too, because the driver is a click-through licence no shared runner -carries; the dialect each needs is pinned by unit test regardless. +carries; the dialect each needs is pinned by unit test regardless. The PostgreSQL and MariaDB ODBC proofs run by hand: CI and the release gate do not build the `db-odbc` feature. diff --git a/docs/reference/profile-grammar.md b/docs/reference/profile-grammar.md index 67c7802..545aa73 100644 --- a/docs/reference/profile-grammar.md +++ b/docs/reference/profile-grammar.md @@ -58,13 +58,13 @@ entry. | | `session_login` (a login call mints a session token, read from a JSON pointer or a response header and carried in a header of the appliance's naming) | none shipped | | Pager (`paginate.strategy`) | `none` | runzero, azure `log_analytics` | | | `link_header` (RFC 5988 `rel="next"`) | github, okta | -| | `cursor` (a token from the body or a header, injected into `query:`, `body:`, `body_replace:` or the path; a list cursor is comma-joined) | aws, bitwarden, crowdstrike, duo, gcp, google_workspace, onepassword, slack | +| | `cursor` (a token from the body or a header, injected into `query:`, `body:`, `body_replace:` or the path; a list cursor is comma-joined) | aws, bitwarden, crowdstrike, datadog, duo, gcp, google_workspace, onepassword, slack | | | `page_number` (bounded by `total_pages_at`) | cloudflare | | | `offset` (advanced by rows received, bounded by `total_at` or a short page) | crowdstrike | | | `request_path` (the next URL from the body or a header, relative resolved against the page) | azure, m365, salesforce | | Decoder (`rows.decoder`) | `json_array` (streamed) | github, okta, m365 | | | `ndjson` (streamed) | runzero, object_store | -| | `json_at` (a wrapped page, rows at a JSON pointer) | aws, azure, bitwarden, cloudflare, crowdstrike, duo, gcp, gcp_pubsub, google_workspace, onepassword, salesforce, slack | +| | `json_at` (a wrapped page, rows at a JSON pointer) | aws, azure, bitwarden, cloudflare, crowdstrike, datadog, duo, gcp, gcp_pubsub, google_workspace, onepassword, salesforce, slack | | | `document` (the whole body is one row) | pypi, crates_io, go_modules, aws `cloudwatch_metrics` | | | `json` (an array is its elements, anything else one row) | object_store | | | `lines` (plain text, one row per line) | go_modules, object_store |