Repository navigation
D16: tap telemetry leaves the host the moment an event is stamped - #14
Merged
Merged
Conversation
When the host stamps an event's seq it now writes one 128-byte, content-free record (the stamped frame header, a realtime stamp, nothing of the payload) into a single-producer ring in its own region, before the tap runs and before routing. One telemetry process, granted by the embedding from its attach hook, holds the consumer end (OURO_ATTACH_CAP_TELEMETRY, two more descriptors) and exports the records off the host. The host never blocks on it: a full ring counts the loss and the next record that fits follows a gap record carrying the count. The consumer is woken by doorbell once per pump, only when asleep, and a later consumer resumes at the same tail. Cost, from c/bench/dispatch.c: 87 ns per event without telemetry, 100 ns with it. The host caches the consumer index, reads the clock once per pump and writes the record with a little-endian fast path that matches the portable encoder byte for byte. The perf budget now gates the telemetry-on path too.
… layout The layout table gains the D16 rows (the capability bit, the two descriptors, the region header and the record), and the C, Go and Rust layout tests hold them in both directions. The Go client asks for the grant through AttachTelemetry and reads the ring with a TelemetryReader under the D13 doorbell protocol. The conformance host turns telemetry on and grants one principal, and TestTelemetryConformance proves the Go reader against it: an ungranted principal gets no region, and the granted one reads its own notification and the C side's ack, in seq order, with no payload byte in either. Rust declares the layout and does not consume it; the reference exporter is Go.
exporters/otlp is the reference telemetry process: it attaches with the D16 grant (ring.ConnectTelemetry), sleeps on the telemetry doorbell, and on each wakeup sends every available record as one OTLP/HTTP logs request, releasing the records only once the collector accepts them. It has no batching timer; its only timer is the retry backoff while the collector is down. It is its own Go module, so the client keeps no OpenTelemetry dependency, and CI tests it under the race detector and end to end against the C host. The Go reader also holds the host's pidfd (D15), so an exporter asleep on the doorbell wakes and exits when the host goes away. docs/telemetry.md explains the record, the hand-off, loss and the exporter, and the architecture, routing and conformance pages point at it.
A telemetry key set to yes in a principal's grant file has the runtime grant that slot the telemetry consumer end from its attach hook; any value other than yes or no is a malformed policy. test_runtime proves a granted principal is handed the region and another is not.
A separate exporters/otlp module could not be installed at a release without a second tag cut after the go/ tag, and every consumer would pin two things. As go/cmd/ouroboros-ring-otlp a consumer builds it at the ring release it already requires, and the OpenTelemetry versions match aimee's. The ring package imports none of them, and check_go_no_cgo.sh now fails if go/ring ever depends on more than the standard library and golang.org/x/sys.
An embedding with its own collector policy and identity, such as aimee, runs otlpexport.New(...).Run in a process of its own; cmd/ouroboros-ring-otlp is now a thin command around it. The end-to-end test against the C host moved with the core.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
D16: every event the host stamps is described off the host the moment it is stamped.
When the host stamps an event's
seq, it writes one 128-byte, content-free record into a single-producer ring in its own region. That happens before the tap runs and before routing. One telemetry process, granted by the embedding, holds the consumer end and exports the records. The host never blocks on it, never talks to a network, and makes no system call per event for it.Before this, ring traffic telemetry existed only as an embedding's in-process tap callback, so nothing about ring traffic left the host.
Pieces
spec/decisions.md, covering the record, the region, the hand-off, loss, the grant and the threat model. The layout is invectors/layout_vectors.tsv, and the C, Go and Rust layout tests check those rows in both directions.telemetry_host.c, the hooks inroute.candhost.c):ouro_host_telemetry_enableandouro_host_grant_telemetry, with the record written at all three stamp sites. A full ring counts the loss, and the next record that fits follows a gap record carrying the count.ouro_client_attach_telemetry/ouro_telemetry_*, andring.AttachTelemetry/ConnectTelemetry/TelemetryReader. Both wait on the doorbell, and the Go reader also wakes when the host exits (via its pidfd).go/otlpexport, wrapped by the commandgo/cmd/ouroboros-ring-otlp. Each wakeup sends one OTLP/HTTP logs request, and records are released only when the collector accepts them. There is no batching timer; the only timer is retry backoff while the collector is down. It lives in the Go module, so a consumer builds it at the ring release it already pins. Its OpenTelemetry versions match aimee's.check_go_no_cgo.shnow fails ifgo/ringever depends on more than the standard library andgolang.org/x/sys.telemetry=yesnames the exporter principal. Any value other thanyesornois rejected as malformed policy.VERSIONgoes to 0.3.0.docs/telemetry.md, linked from the architecture, routing and conformance pages.Cost
From
c/bench/dispatch.c: 87 ns per event without telemetry, 100 ns with it, with a consumer thread draining concurrently.The first version cost 44 ns. Three changes brought it down:
tail.The perf budget now gates the telemetry-on path too (
dispatch_overhead_telemetry_ns).Tests run
test_telemetryand the telemetry benchmark also pass under-fsanitize=thread. Also run: perf budget, install, the external module example,check_single_host.sh, and the vectors check.CGO_ENABLED=0tests, race,check_go_no_cgo.sh, and conformance against the C host, including the newTestTelemetryConformance. It covers four things: an ungranted principal gets no region, the granted client reads its own notification and the C side's ack in seq order, neither contains a payload byte, and a sleeping reader wakes when the host is killed.go/otlpexport): race-enabled tests (a drain is one request, and records refused by the collector are not released), andTestExportFromCHostend to end against the C host.test_runtimecovers the grant key. The granted principal is handed the region, another principal is not, and a bad value is refused.-D warnings, tests,c_host_conformance, and both module runtime harness runs.