Skip to content

feat(otel): propagate W3C trace context across runs, effects, worker tasks, and child runs - #44

Closed
dylnslck wants to merge 31 commits into
mainfrom
feat/otel-trace
Closed

dylnslck wants to merge 31 commits into
mainfrom
feat/otel-trace

Conversation

@dylnslck

Copy link
Copy Markdown
Contributor

Problem and change

Connects Looms execution to distributed telemetry via standard W3C tracecontext (traceparent and tracestate):

  1. Run start capture: Ingress captures traceparent and tracestate from HTTP headers or options on start/startRun/launchRun and commits them onto runtime.run.started and RunStartIdentity.
  2. Effect context propagation: Handlers receive ctx.traceparent with a derived child span ID (preserving trace ID and flags).
  3. Worker tasks: Dispatched worker tasks (EffectWorkerTask) receive the run's traceparent and tracestate for end-to-end telemetry across queue/HTTP boundaries.
  4. Child runs: Launches from effects inherit the parent run's trace ID with a freshly generated child span ID on runtime.run.launched.
  5. Trace carrier utilities: @looms/otel exports createTraceparent, childTraceparent, parseTraceparent, isValidTraceparent, injectTraceCarrier, and extractTraceCarrier.

Review notes

Checked with bun x tsc -p packages/{core,runtime,otel}/tsconfig.build.json --noEmit, bun x oxlint --deny-warnings on changed sources, and bun test src in packages/core, packages/otel, and packages/runtime. bun run verify was not run.

Existing contributor checklist

  • bun run verify passes
  • Changeset added if @swirls/looms behavior changes (bun run changeset)
  • Docs updated for API / architecture changes

Made with Cursor

dylnslck and others added 6 commits October 23, 2026 10:13
BREAKING CHANGE: POST /runs/:runId/effects/:effectId/:attempt/:action returns { status, cancelRequested } instead of the run state. createLooms().workerCallback and the HTTP client return the same shape. Heartbeats may carry details that the next attempt's task includes. failEffect and worker fail callbacks can mark an attempt non-retryable or set retryAfterMs.

Co-authored-by: Cursor <cursoragent@cursor.com>
Attributes are immutable labels on the start event, run state, and summary, and authorization can see them. Secrets stay in a side store for handlers and in-process dispatch, and are kept out of events, snapshots, and HTTP responses.

Co-authored-by: Cursor <cursoragent@cursor.com>
…b store

Strings over 64 KiB, and any event payload that would still exceed 1 MiB, are stored by SHA-256. Reads and snapshots resolve the references, and trim deletes a body once nothing retained points at it.

Co-authored-by: Cursor <cursoragent@cursor.com>
Start, spawn, and recovery can load a definition that was not registered on the module, pin that version on the run, and fail a missing start before any event is written.

BREAKING CHANGE: Evaluator definitions require version, and lookup takes the name and version. defineEvaluator still defaults the version to v1.

Co-authored-by: Cursor <cursoragent@cursor.com>
Spawn, map, fanout, and loop can start a child run that signals the parent when it finishes. Cancelling the parent either stops those children immediately or waits until they finish cancelling.

BREAKING CHANGE: RunLauncher implementations must provide signal and cancel.

Co-authored-by: Cursor <cursoragent@cursor.com>
…tasks, and child runs

Store W3C traceparent and tracestate on RunStartIdentity, pass child trace contexts to in-process effect handlers (EffectContext) and external worker tasks (EffectWorkerTask), link child runs spawned from effects with child span IDs, and export trace context parsing/injection utilities from @looms/otel.

Co-authored-by: Cursor <cursoragent@cursor.com>
@dylnslck dylnslck mentioned this pull request Oct 23, 2026
2 tasks done
dylnslck and others added 23 commits October 23, 2026 13:43
The fold assigns traceparent and tracestate after building the start identity. The runtime no longer copies trace context into the requested identity, because the retry check never compares it.

Co-authored-by: Cursor <cursoragent@cursor.com>
…full fold

Run attributes are immutable and recorded on runtime.run.started, so authorization reads seq 1 and only falls back to loading the run when the log is trimmed. Store failures now fail the request instead of authorizing without attributes. Also fixes the start-body attribute decode, which did not type-check.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… runs

Dropping a reference used to list and read every run's manifest, so each trim cost grew with the number of runs in the bucket. Each run now writes refs/<digest>/<run> markers, and collecting a body lists only that digest's markers.

The old scheme also lost bodies when two Durable Objects shared a bucket. retain now writes markers before checking that each new body exists and fails the append if one was deleted. collect lists markers again after deleting and restores the body if a reference appeared. BlobBytes gains exists, and R2BucketLike gains head, so the check does not download the body.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…text

A transport that persists tasks cannot carry secrets, so its worker needs another way to read them. POST /runs/:runId/effects/:effectId/:attempt/secrets returns the secret context only while that attempt is started and unsettled, and stale or cancelled otherwise. It uses worker access, so it stays closed until the host configures worker auth, and responses are not cacheable. client.workerSecrets wraps it.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The merge takes feat/child-runs as is. The next commit reimplements trace propagation on top of it.

Co-authored-by: Cursor <cursoragent@cursor.com>
POST /runs reads the traceparent and tracestate headers, and start, startRun, and client.startRun accept trace. A valid version-00 context is recorded on runtime.run.started and startIdentity.trace. Invalid values are dropped rather than stored.

The context reaches effect handlers as ctx.trace, worker tasks as task.trace, and launched and child runs unless the launch sets its own. It is passed unchanged: Looms records no spans, so minting span ids would point at parents that never exist. The object's keys are the W3C header names, so it works directly as an OpenTelemetry carrier.

The local executor now takes one RunContext (correlation id, attributes, secrets, trace) in place of separate positional arguments.

Co-authored-by: Cursor <cursoragent@cursor.com>
The retry is scheduled from the wake's clock, which is read before the event timestamp, so comparing against the event could fail by a millisecond. The assertion also no longer skips when the payload is missing.

Co-authored-by: Cursor <cursoragent@cursor.com>
@dylnslck

Copy link
Copy Markdown
Contributor Author

Superseded by squash-merge of the stack tip (#46), which already includes this change set after #34 landed on main.

@dylnslck dylnslck closed this Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant