Skip to content

AUDIT.md

Latest commit

 

History

History
212 lines (175 loc) · 13.4 KB

AUDIT.md

File metadata and controls

212 lines (175 loc) · 13.4 KB
Raw

Audit & Step-Level Trace

Step-level traces are not optional — for two reasons that need the same record:

  • Agent development — to score the trajectory (not just the outcome), replay regressions, and debug, you need the exact ordered (tool · args → result · status) sequence with conflict-recovery visible.
  • Finance — auditability needs an immutable, attributed provenance chain that answers why is this cell this value, who set it, from which inputs, in what order.

The room traces table (the effects — lock/edit/release) and the agentRuns summary don't give you that. The step trace does.

What's persisted (live)

Budget-aware runs also persist stopReason, remainingMs, deadlineAt, and handoff when present. Long-running free-auto jobs add agentJobs and agentJobAttempts: the job row is the durable cursor, and each slice links back to an agentRuns / agentSteps trace.

Every runRoomAgent action (convex/agent.ts) writes two durable records and returns the runId.

agentRuns — per-run telemetry (convex/agentRuns.ts)

model · goal · steps · toolCalls · conflictsSurvived · inputTokens · outputTokens · costUsd · ms · exhausted. One row per run. agentRuns:list · surfaced in the UI trace-header chip.

agentSteps — APPEND-ONLY, tamper-evident step trace (convex/agentSteps.ts)

One row per tool call: { runId, idx, tool, args(JSON), result(JSON), status, ms, elementId?, recordHash, prevStepHash }.

Property How Standard
Append-only insert + read only; corrections are new compensating runs, never edits SEC 17a-4(f) / SOX §802
Tamper-evident recordHash = SHA-256(sorted-key serialization incl. prevStepHash); altering any past step breaks every later hash hash-chain / WORM-alternative
Honest status ok | conflict | locked | error, derived from the result — never "ok" on a failed CAS HONEST_STATUS
Per-cell provenance elementId on every write, indexed by_room_element data lineage
Attributed agentId per step; model + cost + tokens on the run SOX §302 who/what
Size-bounded args/result JSON capped at 2 KB BOUND_READ

Queries

  • agentSteps:byRun(runId) — full ordered trajectory (trajectory eval + replay).
  • agentSteps:byElement(roomId, elementId) — every write that touched a cell ("why is this value").
  • agentSteps:verify(runId) — re-walks the hash chain → { valid, brokenAt?, reason? }.

Verified end-to-end

byRun  → 9 steps: read_range → propose_lock → 5× edit_cell (each [ok], elementId set) → release_lock → say
         each prevStepHash == previous recordHash   (intact chain)
verify → { valid: true, steps: 9 }
byElement r_rev__variance → args {value:"+24.0%", baseVersion:7} result {ok:true, version:8} status:ok agent:agent_room

The standards this maps to

Finance audit (SOX §802/§404, SEC 17a-4(f))

Done: append-only, hash-chain tamper-evidence, who/what/when (agentId·ts·tool), honest status, per-cell provenance, run-level reproducibility (model·tokens·cost·goal). The current AgentTraceEvent { step, tool, args, result, ms } was a debug trace; the agentSteps record is the audit-grade superset.

Agent eval + observability (OTel GenAI semconv v1.41.1, trajectory eval)

The persisted trace lets the eval (evals/runEval.ts) score trajectory from a durable store (CI history, production replay) instead of only the in-memory AgentResult.trace. Field mapping to OTel GenAI: model → gen_ai.request.model, inputTokens/outputTokens → gen_ai.usage.input_tokens/output_tokens, each agentSteps row → an execute_tool span.

Honest gaps (production roadmap — NOT built)

  1. Explicit edit→read provenance link. Today an edit carries its baseVersion (in args) and elementId; the read that sourced it is a prior step with the same elementId — implicit, not a stored sourcedFromStepId. For mechanical "no edit without a fresh read" proof from the persisted store (the no-silent-clobber gate), add triggeredBy / inputRefs to each write step.
  2. valueBefore + derivation inputs. Stored: baseVersion + value written. Not stored: the prior cell value, and the source numbers the variance was computed from (those came from the context snapshot, not a tool call). Add valueBefore + the contextSnapshotRef for full input→output lineage.
  3. Run seed for deterministic replay. goal is stored; room_seed (element values+versions at start) and the injected concurrency script are not.
  4. Reproducibility depth. Add modelVersion (pinned snapshot), promptHash, decoding params, harnessVersion.
  5. Retention + OTel export. SOX wants 7-year protected retention; observability wants an OTel exporter emitting gen_ai.* spans. Neither is wired.

These are deliberately out of scope for the spike — the core audit contract (append-only · tamper-evident · attributed · status-honest · per-cell provenance · verifiable) is live and proven above.

Prior art: OTel GenAI semantic conventions; SEC Rule 17a-4(f); SOX §302/§404/§802; LangSmith/AgentEvals trajectory match. See docs/AGENT_EVAL.md for the eval harness.

Agent Task Ladder (the eval that differentiates models)

The keystone: move from "did the agent complete the flow?" to "did it complete the right task, with the right tool, without leaking private data, without corrupting shared state, within acceptable cost and latency?" A flat pass/fail can't show that — every modern model passes a floor task. A complexity ladder makes model quality, tool safety, collaboration behavior, and cost tradeoffs visibly diverge.

Each rung = a harness: starting state + task prompt + allowed tools + expected trace + expected final state + deterministic checks + semantic judge + cost/latency.

Levels

L name what it proves failure it catches
L0 schema / contract valid output object (visibility, traceRef, sourceRefs) invalid schema, missing trace, wrong visibility
L1 read, no mutation inspects state with the smallest context mutates on a read; over-fetches; leaks private into public
L2 single artifact edit safe CAS write, correct version + range wrong cell, no version check, no audit
L3 versioning / conflict survives duplicate + stale-version silent overwrite, ignored conflict
L4 multi-turn correction / blocked revises on user correction; drafts when locked commits wrong update; force-edits a locked range
L5 large-sheet range reads rows 120–140, not the whole sheet full-snapshot context blow-up
L6 long-horizon session compaction + locks + recovery over time context pollution, state drift

Harnesses (5)

spreadsheet · notebook · cross-collaboration · diligence/research · risk-attack (public/private boundary, prompt-override, private-leak, wrong-event-retrieval, trace honesty).

Scoring (two layers)

  • Deterministic (no LLM): correct version, affected-range, no-duplicate-apply, no-silent-overwrite, null-preserved, trace-present, no-full-snapshot, no private→public leak.
  • Semantic judge (cheap LLM, isolated yes/no): right entity, no invented specifics, honest uncertainty, useful explanation. Penalties: −20 private leak, −20 silent overwrite, −15 invented fact, −15 wrong entity, −10 full-snapshot-when-range-sufficed, −10 stale overwrite.

Status (2026-06-07)

  • Built + live: the spreadsheet/collab ladder L1–L6 (evals/ladder.ts, npm run ladder, npm run ladder:real). L5 uses a 600-row operating model with a narrow range context, exposes a full-sheet trap tool, and fails on full-snapshot access or excessive context size. L6 forces compaction plus repeated conflict recovery, and each edit must be immediately preceded by the exact read_range that supplies its baseVersion.
  • Model matrix: the real-model ladder keeps the failure heatmap behavior that made the eval useful: cheaper models can clear simple reads/writes and still fail under lock contention or long-horizon recovery. The multi-model research matrix + cost-quality charts remain in scripts/benchmark/.
  • Next milestones (sequenced): notebook / cross-collaboration / risk-attack harnesses, then per-rung cost/latency on the heatmap.

Folder target: src/eval/{cases,harnesses,judges,runners,reports} — evals/ladder.ts is the first runner; the others graduate into that structure as they land.

Hardening review (2026-06-07) — 5-domain adversarial deep read

Fanned out 5 parallel reviewers (SSRF · Convex authz · lock-release · ladder honesty · telemetry), each reading end-to-end and trying to break it. Triage + disposition:

Fixed

  • SSRF (P0×3) in fetchSource.ts — the regex denylist was bypassable. Rewrote with a canonical IP guard (unwraps IPv4-mapped IPv6 incl. Node's hex-normalized ::ffff:7f00:1; CIDR incl. CGNAT 100.64/10 + multicast), DNS resolve-and-reject-private (closes name→private + metadata.google.internal), per-hop redirect re-validation, and one 5 s total budget across redirects + body (shared AbortSignal). 13-vector hermetic regression test (tests/fetchSourceSsrf.test.ts). Legit fetch verified intact.
  • Ladder honesty (P2) — L1 now requires reading the target cell (was: any read); L4 requires the draft to contain the target op (was: any create_draft); the cost diagnostic bug (priceRun(""), both ternary branches empty) is fixed and cost is now shown per rung.

Verified false (stale review state)

  • "Convex build is RED" — the 9-min review caught a mid-edit tree; tsc -p convex/tsconfig.json and the app tsc both pass clean now. No action.

Sound (reviewer agreed)

  • Wrong-holder lock release — correct in both engine + Convex; the regression test asserts the rejection.

Later hardening (2026-06-07)

  • Member proof tokens now gate live room access — client-visible Convex reads/writes pass {actor, token} and the server verifies either ctx.auth.getUserIdentity().subject or a timing-safe salted token hash against the members row before returning room state, messages, traces, telemetry, locks, or accepting edits/messages/settings changes. Public member reads are sanitized and never return authToken/authTokenHash. Agent tool endpoints that still accept agent actors are internal* functions reached only from the already-authorized runRoomAgent action.
  • Pre-token demo rows have an admin repair path — seed:seedDemoRoom is idempotent and repairs an existing Q3DEMO; seed:backfillDemoAuthTokens explicitly backfills already-seeded rooms. Both are protected by SEED_ADMIN_TOKEN, no longer ship source-code bearer tokens, and clear legacy raw tokens unless the operator supplies a fresh host token to hash.
  • Maintenance and model-call surfaces are narrower — seed/reset/research mutations are admin-gated, room-code creation rejects duplicates, new rooms default to approval-gated agent writes, and live runRoomAgent requires a proven room member. Host review remains required to approve/reject pending proposals when autoAllow is off.
  • SSRF guard follow-up — IPv6 link-local /10, multicast, documentation, NAT64, 6to4, Teredo, IPv4-compatible hex mappings, DNS-private resolution, redirect-to-private/http, and DNS timeout behavior are regression tested. DNS validation now runs under the same total abort budget as redirects/body reads, fetch sockets are pinned with an undici dispatcher to the prevalidated addresses for each hop, and per-hop dispatchers are closed after use.
  • Failed model-loop telemetry is durable — if runAgent throws after authorization, the action records an agentRuns row, any partial trace/usage available from the runtime, and a hash-chained agentSteps row with tool: "run_error" and status: "error" before rethrowing to the caller.
  • Budget-aware long-run handoff is durable - live agent actions pass a deadline/reserve into the runtime, compact long contexts by default, and persist stopReason, remainingMs, deadlineAt, and handoff. If the action is about to hit its budget, runAgent emits a tool: "handoff" trace event instead of relying on Convex's hard action timeout.
  • Convex deployment analyzer path is part of verification - the live Convex agent modules use the standard action runtime plus src/nodeagent/models/convexModel.ts direct provider HTTP calls, avoiding the Node-action analyzer failure hit during codegen. npx convex codegen --dry-run --typecheck disable is now a required smoke for backend changes.
  • Long-running free-auto continuation is atomic - finishSlice records agentJobAttempts, stores cursor/handoff, clears the lease, and schedules the next slice inside the same mutation. The UI exposes cancel/retry plus latest-attempt status so operators can inspect the job without reading deployment logs.
  • Auth provider compatibility is live — member proof accepts ctx.auth.getUserIdentity().subject when a Convex Auth identity is present, with hashed member tokens retained for anonymous/demo sessions. Public room/member reads return sanitized records only, member tokens must satisfy a server-side strength floor before they are salted/hashed/accepted, and legacy raw member tokens have an admin migration path.