Skip to content

D16: tap telemetry leaves the host the moment an event is stamped - #14

Merged
JBailes merged 7 commits into
mainfrom
feat/telemetry-handoff
Oct 30, 2026
Merged

JBailes merged 7 commits into
mainfrom
feat/telemetry-handoff

Conversation

@JBailes

@JBailes JBailes commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What

D16: every event the host stamps is described off the host the moment it is stamped.

When the host stamps an event's seq, it writes one 128-byte, content-free record into a single-producer ring in its own region. That happens before the tap runs and before routing. One telemetry process, granted by the embedding, holds the consumer end and exports the records. The host never blocks on it, never talks to a network, and makes no system call per event for it.

Before this, ring traffic telemetry existed only as an embedding's in-process tap callback, so nothing about ring traffic left the host.

Pieces

  • Spec: D16 in spec/decisions.md, covering the record, the region, the hand-off, loss, the grant and the threat model. The layout is in vectors/layout_vectors.tsv, and the C, Go and Rust layout tests check those rows in both directions.
  • C host (telemetry_host.c, the hooks in route.c and host.c): ouro_host_telemetry_enable and ouro_host_grant_telemetry, with the record written at all three stamp sites. A full ring counts the loss, and the next record that fits follows a gap record carrying the count.
  • C and Go readers: ouro_client_attach_telemetry / ouro_telemetry_*, and ring.AttachTelemetry / ConnectTelemetry / TelemetryReader. Both wait on the doorbell, and the Go reader also wakes when the host exits (via its pidfd).
  • Reference exporter: its core is the package go/otlpexport, wrapped by the command go/cmd/ouroboros-ring-otlp. Each wakeup sends one OTLP/HTTP logs request, and records are released only when the collector accepts them. There is no batching timer; the only timer is retry backoff while the collector is down. It lives in the Go module, so a consumer builds it at the ring release it already pins. Its OpenTelemetry versions match aimee's. check_go_no_cgo.sh now fails if go/ring ever depends on more than the standard library and golang.org/x/sys.
  • Runtime grant files: the new key telemetry=yes names the exporter principal. Any value other than yes or no is rejected as malformed policy.
  • Release: VERSION goes to 0.3.0.
  • Docs: docs/telemetry.md, linked from the architecture, routing and conformance pages.

Cost

From c/bench/dispatch.c: 87 ns per event without telemetry, 100 ns with it, with a consumer thread draining concurrently.

The first version cost 44 ns. Three changes brought it down:

  1. The host caches the consumer's tail.
  2. It reads the clock once per pump.
  3. It writes the record with a little-endian fast path that is tested byte-for-byte against the portable encoder.

The perf budget now gates the telemetry-on path too (dispatch_overhead_telemetry_ns).

Tests run

  • C: release, ASan/UBSan and TSan-arena ctest, all 19 pass. test_telemetry and the telemetry benchmark also pass under -fsanitize=thread. Also run: perf budget, install, the external module example, check_single_host.sh, and the vectors check.
  • Go: gofmt, vet, CGO_ENABLED=0 tests, race, check_go_no_cgo.sh, and conformance against the C host, including the new TestTelemetryConformance. It covers four things: an ungranted principal gets no region, the granted client reads its own notification and the C side's ack in seq order, neither contains a payload byte, and a sleeping reader wakes when the host is killed.
  • Exporter (go/otlpexport): race-enabled tests (a drain is one request, and records refused by the collector are not released), and TestExportFromCHost end to end against the C host.
  • Runtime: test_runtime covers the grant key. The granted principal is handed the region, another principal is not, and a bad value is refused.
  • Rust: fmt, clippy -D warnings, tests, c_host_conformance, and both module runtime harness runs.

When the host stamps an event's seq it now writes one 128-byte, content-free record (the stamped frame header, a realtime stamp, nothing of the payload) into a single-producer ring in its own region, before the tap runs and before routing. One telemetry process, granted by the embedding from its attach hook, holds the consumer end (OURO_ATTACH_CAP_TELEMETRY, two more descriptors) and exports the records off the host. The host never blocks on it: a full ring counts the loss and the next record that fits follows a gap record carrying the count. The consumer is woken by doorbell once per pump, only when asleep, and a later consumer resumes at the same tail.

Cost, from c/bench/dispatch.c: 87 ns per event without telemetry, 100 ns with it. The host caches the consumer index, reads the clock once per pump and writes the record with a little-endian fast path that matches the portable encoder byte for byte. The perf budget now gates the telemetry-on path too.
… layout

The layout table gains the D16 rows (the capability bit, the two descriptors, the region header and the record), and the C, Go and Rust layout tests hold them in both directions. The Go client asks for the grant through AttachTelemetry and reads the ring with a TelemetryReader under the D13 doorbell protocol. The conformance host turns telemetry on and grants one principal, and TestTelemetryConformance proves the Go reader against it: an ungranted principal gets no region, and the granted one reads its own notification and the C side's ack, in seq order, with no payload byte in either. Rust declares the layout and does not consume it; the reference exporter is Go.
exporters/otlp is the reference telemetry process: it attaches with the D16 grant (ring.ConnectTelemetry), sleeps on the telemetry doorbell, and on each wakeup sends every available record as one OTLP/HTTP logs request, releasing the records only once the collector accepts them. It has no batching timer; its only timer is the retry backoff while the collector is down. It is its own Go module, so the client keeps no OpenTelemetry dependency, and CI tests it under the race detector and end to end against the C host.

The Go reader also holds the host's pidfd (D15), so an exporter asleep on the doorbell wakes and exits when the host goes away. docs/telemetry.md explains the record, the hand-off, loss and the exporter, and the architecture, routing and conformance pages point at it.
A telemetry key set to yes in a principal's grant file has the runtime grant that slot the telemetry consumer end from its attach hook; any value other than yes or no is a malformed policy. test_runtime proves a granted principal is handed the region and another is not.
A separate exporters/otlp module could not be installed at a release without a second tag cut after the go/ tag, and every consumer would pin two things. As go/cmd/ouroboros-ring-otlp a consumer builds it at the ring release it already requires, and the OpenTelemetry versions match aimee's. The ring package imports none of them, and check_go_no_cgo.sh now fails if go/ring ever depends on more than the standard library and golang.org/x/sys.
An embedding with its own collector policy and identity, such as aimee, runs otlpexport.New(...).Run in a process of its own; cmd/ouroboros-ring-otlp is now a thin command around it. The end-to-end test against the C host moved with the core.
@JBailes
JBailes merged commit 9cc341a into main Oct 30, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant