Skip to content

Execution recording: trace_runs/trace_spans schema and the recording consumer #254

Description

@07prajwal2000

Why

record_execution is the switch for keeping a run in our database so it can be
replayed in the portal — as opposed to tracing_enabled, which ships spans to
the project's own OTEL backend and stores nothing here.

The flag reaches the compiled artifact (modules/compiler/service.ts
compiledRuntime.ts) and nothing consumes it. There is no table and no
consumer, so turning it on does nothing at all.

The wire contract already exists and is the right one. TraceRunPayload /
TraceSpanRecord (packages/common/otlp/types.ts) already carry, per span,
blockId, blockType, nullable customBlockId, timestamps, outcome,
branch, input, output, error and truncated — which is exactly what a
recording needs. OTLP export is one consumer of that payload, not its purpose.
FLUXIFY_TRACES was created as RetentionPolicy.Limits rather than Workqueue
specifically so a second consumer can sit alongside the exporter (#195), so this
issue needs no new transport, no new payload type and no new producer hook.

This is the storage half of #208.

What

Two tables.

  • trace_runs — project, route, routeVersion, start/end, outcome, status code,
    dropped/truncated state, parentRunId + parentSeq for async custom-block
    runs.
  • trace_spans — run, seq, parentSeq, blockId, blockType, nullable
    customBlockId, timestamps, outcome, branch, error, truncated payload.

Indexes: (projectId, startedAt desc) on runs. PRIMARY KEY (run_id, seq) on
spans — that PK is also the dedup key for at-least-once delivery and doubles as
the (runId, seq) index. Do not add a second one.

Span times are performance.now() readings and are meaningless off-box. Convert
against the run's startedAtWallMs / perfOrigin pair on write, so consumers
never have to carry the origin around.

A second durable consumer on FLUXIFY_TRACES, independent of the OTLP
exporter. It persists a run when the route has record_execution on, and drops
it otherwise — re-checked at consume time, not trusted from compile time, since
the flag can flip between the run being recorded and this consuming it (the
exporter already does this for destinations).

Retention. Recordings are debug data and must not grow without bound. Cap
rows per route and delete the oldest on insert; no sweeper job.

Notes

  • customBlockId is nullable on purpose: spans recorded inside a custom block
    carry that graph's node ids, so overlaying them on the route canvas highlights
    nothing unless the reader knows which graph a span belongs to. The viewer
    depends on this being stored faithfully.
  • routeVersion is the only forward-compatibility hook for route versioning.
    Store it even though nothing reads it yet.
  • Runs with endedAt IS NULL older than the route timeout are "incomplete". No
    sweeper and no abandoned status writes — a status nobody reads is not worth a
    background job.
  • Ack and drop on a malformed payload rather than redelivering. A bad body fails
    identically every time and the run has no value once its route has moved on.
  • No producer yet. Nothing in modules/requestRouter or packages/blocks
    records spans today; the only publisher is scripts/publishTraceRun.ts, a
    fixture. That script is how this consumer gets exercised end to end, and it is
    enough to build and verify everything here. Real traffic producing spans is
    separate work on the compiled runtime's hot path.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions