[FLO-OBS-03] Prove opt-in OpenTelemetry logs, metrics, and traces

[szmyty]

Parent: #14
Depends on: FLO-OBS-01 / #15, FLO-OBS-02 / #16

Outcome

Prove complete, opt-in OpenTelemetry interoperability for suite logs, metrics, and traces, including bounded context propagation across a representative parent/child process boundary.

This checkpoint establishes backend-neutral protocol behavior. It does not select a hosted vendor or turn telemetry into durable workflow evidence.

Signal scope

Traces

• Define the suite span hierarchy from command/run through stage, provider resolution, process execution, validation, checkpoint assessment, and terminal outcome.
• Map reviewed fields to OpenTelemetry attributes with explicit stability and cardinality.
• Preserve causal links for retry, resume, fan-out, validation, and cancellation.
• Propagate W3C Trace Context only across allowlisted participating suite boundaries.
• Treat baggage as prohibited by default; permit only explicitly reviewed low-cardinality keys.
• Never propagate context into arbitrary configured processors merely because they are child processes.

Metrics

• Define counters, up/down counters, gauges/observables, and histograms for operation count, outcome, duration, provider resolution, retries, reuse, validation, cancellation, queue/concurrency, bytes/artifacts, and bounded resource pressure.
• Use stable units and bounded attribute sets.
• Exclude paths, artifact digests, trace/run IDs, error messages, user identities, provider argv, and source-derived content from metric labels.
• Define temporality and aggregation expectations without hard-coding one backend.

Logs

• Correlate structured log records with active trace/span context.
• Map severity and event names consistently.
• Preserve redaction, truncation, and stream ownership from [FLO-OBS-01] Define the suite observability, privacy, and stream contract #15.
• Do not require OpenTelemetry log export for local JSON Lines operation.

Export and configuration

• Support optional OTLP export through the Rust OpenTelemetry ecosystem.
• Evaluate both OTLP over HTTP and gRPC against dependency, TLS, MSRV, binary-size, and platform requirements; ship only modes justified by evidence.
• Define resource attributes such as service.name, service.version, suite identity, and process/runtime metadata.
• Keep exporters and network dependencies behind explicit features and runtime configuration.
• Default to no exporter and no network activity.
• Support a local/in-memory test exporter and a local collector fixture.
• Define batching, queue bounds, sampling, retry, timeout, backpressure, flush, and shutdown behavior.
• Ensure CLI completion cannot hang indefinitely waiting for export.
• Surface collector/export failure as a bounded observability diagnostic that does not falsify workload success. Any fail-closed mode must be explicit and separately typed.
• Document proxy, TLS, endpoint, header, and credential handling without logging their values.

Cross-process proof

Use a hermetic parent/child fixture representing flow invoking a participating holon:

• parent creates or receives context;
• child receives only the allowlisted propagation material;
• child spans join the same trace;
• non-participating fixture processes receive no trace context;
• malformed or unsupported context fails safely;
• sampling and exporter absence do not change operation semantics.

Do not require a public network, hosted account, or external processor.

Acceptance criteria

• Logs, metrics, and traces export to an ephemeral local collector when explicitly enabled.
• All export is inactive by default and offline tests prove no connection attempt occurs.
• Trace/log correlation and parent/child causal relationships are validated without parsing human output.
• Non-participating providers receive no trace context.
• Metric instruments, units, attribute sets, and cardinality budgets have conformance tests.
• Paths, content, secrets, environment values, argv, raw error messages, artifact digests, and run/trace IDs are absent from metric labels.
• Exporter queue saturation, unavailable endpoint, partial export, flush timeout, and shutdown are tested.
• Workload result/evidence is identical with exporting enabled, disabled, sampled, or unavailable.
• Feature-disabled builds retain the supported MSRV and do not pull exporter/network dependencies.
• Documentation includes local collector configuration and exact privacy implications.
• OpenTelemetry crate/protocol versions and upgrade policy are recorded.

Explicit non-goals

• Selecting Grafana, Honeycomb, Datadog, New Relic, Jaeger, or another hosted backend.
• Export enabled by default.
• Auto-uploaded diagnostic data.
• Arbitrary baggage propagation.
• Unbounded event or media-content export.
• Treating span completion as stage validation.
• Continuous profiling.
• Production deployment of a collector.

Roadmap-Track: FLO-OBS