Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions docs/cloud-setup/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@
Each guide here covers what a provider administrator configures so dfe-fetcher
can pull data from that provider: the access it needs, how to grant it
least-privilege, where to put credentials, and how to verify the connection.
Every source reads only -- the fetcher never writes, modifies, or deletes
anything on the provider side.
Every source is read-side: M365 starts its activity-feed subscription and
Pub/Sub acknowledges what it delivered. Nothing else changes provider state.

For the fetcher's own configuration (scheduler, output, credential resolution,
the `sources.db` and `sources.file` blocks), see
Expand Down Expand Up @@ -79,6 +79,3 @@ stable.
| Alpha | Code-complete but not production-validated. Behaviour and config may change. The default for a new source. |
| Beta | Validated against a live service; hardening in progress. |
| Stable | Production-ready. |

Promote a source in its profile rather than recording its stage in these
guides.
2 changes: 1 addition & 1 deletion docs/cloud-setup/gcp-pubsub.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ Common failures:
was never created.
- Pull succeeds but returns nothing - the sink writer identity was not granted
`roles/pubsub.publisher` on the topic (step 3), the `--log-filter` matches no
entries, or the window is genuinely quiet. Verify routing with the Cloud
entries, or the window is quiet. Verify routing with the Cloud
console's log-sink view.
- `invalid SA key / missing client_email` - the key JSON is malformed or the
wrong file.
Expand Down
21 changes: 12 additions & 9 deletions docs/internals.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ flowchart TB
subgraph "DFE pipeline"
direction LR
Kafka[(Kafka topics)]
Receiver[dfe-receiver]
Receiver[dfe-loader, a transform or the archiver]
end

API --> Rest
Expand Down Expand Up @@ -369,6 +369,8 @@ up for `stable_after_secs`.

## Container Communication

The HTTP mode listener requires `ingest.auth_token`: every `/ingest` call carries `Authorization: Bearer <token>`. With no token the listener refuses to start unless `ingest.allow_unauthenticated: true` is set. The `/livez` and `/readyz` probes are exempt.

```mermaid
flowchart TB
subgraph "stdout mode"
Expand Down Expand Up @@ -477,13 +479,16 @@ another followed by `.`.
### Sidecar Patterns

For log sources that established tools handle well, use container extractors
with Filebeat, Fluentd, or similar collection agents:
with a collection agent that can print JSON lines or POST JSON, such as Vector
or Fluent Bit. Filebeat has no HTTP output (its outputs are Elasticsearch,
Logstash, Kafka, Redis, file and console, per Elastic's output docs), so ship
Beats data to dfe-receiver's Beats listener instead.

| Source Type | Sidecar Tool | Communication |
|-------------|-------------|---------------|
| Syslog | Filebeat (system module) | HTTP POST to `/ingest/syslog` |
| Windows Event Logs | Filebeat (winlogbeat) | HTTP POST to `/ingest/windows_events` |
| Custom app logs | Filebeat (file input) | stdout JSON lines |
| Syslog | Vector or Fluent Bit | HTTP POST to `/ingest/syslog` |
| Windows Event Logs | A forwarder that POSTs JSON (winlogbeat is a separate Beat with the same output limits) | HTTP POST to `/ingest/windows_events` |
| Custom app logs | A shipper that prints JSON lines | stdout JSON lines |
| Network captures | Zeek / Suricata | stdout JSON lines |
| Cloud provider logs | Cloud-specific CLI tools | stdout JSON lines |

Expand All @@ -500,8 +505,7 @@ exponential backoff. A log file the fetcher itself can reach is a
Typical fetcher volumes are orders of magnitude lower than the loader /
receiver / transform / archiver tier. A single fetcher pod servicing one
tenant's combined AWS / Azure / M365 / GCP / SaaS audit feeds normally
moves tens to a few thousand records per minute. The downstream pipeline
moves PB/hour. The fetcher is also I/O-bound waiting on remote cloud
moves tens to a few thousand records per minute. The fetcher is also I/O-bound waiting on remote cloud
APIs, not CPU-bound parsing or routing.

What this means in practice:
Expand All @@ -520,8 +524,7 @@ We still care about correctness, cancellation safety, bounded retries,
and not leaking tasks - those are correctness concerns, not perf
concerns. We also still prefer the simple, idiomatic version of any
pattern over the clever one. But the aggressive hot-path discipline
documented in the hyperi-ai Rust standards and applied across
dfe-loader / dfe-receiver / dfe-archiver does **not** need to be
applied across dfe-loader / dfe-receiver / dfe-archiver does **not** need to be
applied symmetrically here.

This trade-off is deliberate. It keeps a source a YAML profile rather than
Expand Down
7 changes: 3 additions & 4 deletions docs/reference/db-drivers.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,10 +113,9 @@ to be mounted into the image; the spec mechanism does not fetch it.

### Forms that need a token minted per connection

These are unsupported today, and the reason is the same in each case: the
These are unsupported, and the reason is the same in each case: the
credential is a token valid for minutes to an hour, and `connection_string`
resolves once. Supporting them means re-resolving the string per connect, which
issue #111 also asks for.
resolves once. Supporting them means re-resolving the string per connect.

- **Azure Database for PostgreSQL** with Entra. The user name is the Entra
principal and the PASSWORD is an access token, obtained out of band and valid
Expand Down Expand Up @@ -206,4 +205,4 @@ the first tick was the last acknowledged row's. SQL Server and Oracle have
the same proof, gated on their operator-supplied driver being registered
with the host's driver manager: absent the driver the test skips saying so,
in CI too, because the driver is a click-through licence no shared runner
carries; the dialect each needs is pinned by unit test regardless.
carries; the dialect each needs is pinned by unit test regardless. The PostgreSQL and MariaDB ODBC proofs run by hand: CI and the release gate do not build the `db-odbc` feature.
4 changes: 2 additions & 2 deletions docs/reference/profile-grammar.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,13 +58,13 @@ entry.
| | `session_login` (a login call mints a session token, read from a JSON pointer or a response header and carried in a header of the appliance's naming) | none shipped |
| Pager (`paginate.strategy`) | `none` | runzero, azure `log_analytics` |
| | `link_header` (RFC 5988 `rel="next"`) | github, okta |
| | `cursor` (a token from the body or a header, injected into `query:`, `body:`, `body_replace:` or the path; a list cursor is comma-joined) | aws, bitwarden, crowdstrike, duo, gcp, google_workspace, onepassword, slack |
| | `cursor` (a token from the body or a header, injected into `query:`, `body:`, `body_replace:` or the path; a list cursor is comma-joined) | aws, bitwarden, crowdstrike, datadog, duo, gcp, google_workspace, onepassword, slack |
| | `page_number` (bounded by `total_pages_at`) | cloudflare |
| | `offset` (advanced by rows received, bounded by `total_at` or a short page) | crowdstrike |
| | `request_path` (the next URL from the body or a header, relative resolved against the page) | azure, m365, salesforce |
| Decoder (`rows.decoder`) | `json_array` (streamed) | github, okta, m365 |
| | `ndjson` (streamed) | runzero, object_store |
| | `json_at` (a wrapped page, rows at a JSON pointer) | aws, azure, bitwarden, cloudflare, crowdstrike, duo, gcp, gcp_pubsub, google_workspace, onepassword, salesforce, slack |
| | `json_at` (a wrapped page, rows at a JSON pointer) | aws, azure, bitwarden, cloudflare, crowdstrike, datadog, duo, gcp, gcp_pubsub, google_workspace, onepassword, salesforce, slack |
| | `document` (the whole body is one row) | pypi, crates_io, go_modules, aws `cloudwatch_metrics` |
| | `json` (an array is its elements, anything else one row) | object_store |
| | `lines` (plain text, one row per line) | go_modules, object_store |
Expand Down
Loading