Add Logfire tracing, experiment engine, and setup walkthrough - #13
Merged
Conversation
Adds the six setup-card class names (setup-card, setup-key, setup-key-required, setup-key-optional, setup-command, setup-summary) to CONTRACT_CLASSES in styles.test.ts. Fails until styles.css defines them. Co-Authored-By: Claude <noreply@anthropic.com>
Covers GET /api/setup resolving to the parsed SetupStatus, a non-OK response rejecting as ApiError, and that the setup export exposes no write method (keys are set only via the CLI). Fails until setup and SetupStatus/SetupKey are added to client.ts/types.ts.
Covers the two new FileConfig fields (logfire_token, logfire_api_key), set_logfire_token/set_logfire_api_key, apply_logfire_token, the effective-presence helpers (gateway_key_present, logfire_token_present, logfire_api_key_present), require_gateway_key, and a guard asserting the logfire extra is present in the dev environment. Fails until src/valcore/config.py and pyproject.toml add the corresponding implementation.
Defines .setup-card, .setup-key, .setup-key-required/-optional, .setup-command, and .setup-summary using existing design tokens, so they satisfy the CONTRACT_CLASSES check ahead of the components that consume them. Co-Authored-By: Claude <noreply@anthropic.com>
Add SetupKey/SetupStatus types and a setup.get() client method for GET /api/setup, mirroring the overview client. No write method exists since keys are set only via the CLI.
Covers the not-yet-implemented ExperimentRun model plus Store.set_experiment/ get_experiment, the create_all regression guard (table reaches an existing database), and request_cancel raising ContractError for experiment-produced runs while leaving runner-produced run cancellation untouched.
Adds failing tests ahead of the spec-generics fix: dataset_to_evals must resolve OutputT from the dataset's label schema (Literal enum / float / str) rather than object, which infers to an empty JSON schema. Also pins the constraints the fix must preserve: byte-identical serialized output, an unaffected Dataset.from_dict import path, and no validation introduced for labels outside the declared categorical set.
Adds logfire_token and logfire_api_key to FileConfig with matching set_*/apply_logfire_token functions mirroring the gateway key path, plus effective-presence helpers (env-or-file) and require_gateway_key, which turns a missing gateway key into a clear ConfigError instead of a 500 and N per-row failures. Declares the logfire extra and adds it to the dev group so downstream span tests actually run.
dataset_to_evals built EvalsDataset[dict, object, dict]; TypeAdapter(object)
infers to {}, so a hosted push carried no expected-output schema even though
valcore knows the label space. OutputT is now Literal[labels] for categorical,
float for numeric, and str for the empty "no ground truth" schema. Serialized
bytes and the import path are unchanged.
Marks a Run as produced by the experiment engine via a separate table (init_db is a bare create_all, so a new Run column would not reach an existing database). request_cancel now raises ContractError for such runs since Dataset.evaluate has no cancellation hook.
Closes a coverage gap: set_experiment follows the same replace-existing-row shape as set_generation, but nothing asserted that a second call replaces the row rather than leaving two. Co-Authored-By: Claude <noreply@anthropic.com>
Fails until web/src/components/useSetup.ts exists. Covers gatewayReady derived from the gateway_api_key SetupKey, the don't-strand-the-user default (true while loading and after a rejected fetch), refetch, the stale-response guard on rapid refetches, and GATEWAY_BLOCKER wording.
One hook and shared GATEWAY_BLOCKER string so every gateway-gated action reads setup status consistently. gatewayReady defaults to true and stays true during loading or on fetch error so a transient failure never strands the user with disabled buttons.
Tests for src/valcore/tracing.py, written before the module exists (fails today with ImportError). Covers: silent no-op with no token, single UserWarning when a token is set but the logfire extra is absent, idempotent configure(), run/row span parent-child nesting and documented attributes via logfire.testing's capfire fixture, and a source scan guarding against client-side agent instrumentation ever being reintroduced.
Adds configure_tracing, run_span, and row_span so the Gateway's server-side LLM spans nest under valcore's own run/row context via the injected traceparent, without any client-side agent instrumentation.
…cing.py Adds regression coverage the original test plan didn't spell out: the no-token path must stay silent even when logfire looks absent, and the module must never import real logfire or swallow ImportError.
gatewayReady previously read only the last-loaded status, so a refetch that started after a real "unset" response kept gatewayReady false throughout the pending request and after a rejection — violating the don't-strand-the-user guarantee. Derive it from loading and error too, and add tests covering the refetch-in-flight and refetch-error cases that the original tests (loaded status always null) couldn't expose. Also flush the stale-response test's resolution inside act() so the assertion can't run before the (non-)update commits.
Pins push_dataset's behavior ahead of implementation: API key resolution (argument over config), the on_conflict -> on_case_conflict rename, the DatasetDetail -> dict shape with None for absent optional keys and no synthesized URL, ContractError wrapping of client failures, ConfigError for a missing key or missing logfire extra, and -- via a poison-pill stand-in for LogfireAPIClient -- that the async client is what actually gets constructed and awaited. All stubbed; no network calls or real credentials needed.
Adds logfire_io.push_dataset, an async function using AsyncLogfireAPIClient (not the blocking sync client) so uploads don't stall the event loop inside the FastAPI handler that will call it. Resolves the API key from the argument or stored config, lazily imports logfire.experimental.api_client so the module stays importable without the logfire extra, and wraps client failures in ContractError.
run_span/row_span now guard on the configured flag instead of always calling logfire.span, so an unconfigured process (with real logfire installed) never triggers LogfireNotConfiguredWarning. configure_tracing sets the idempotency flag only after logfire.configure succeeds, so a failed configuration attempt can be retried instead of permanently wedging the module. Tests mock logfire.configure in warning/token cases so the suite never makes a real network call.
…_dataset Use the client's async context manager so its httpx.AsyncClient is closed on every path, and move construction/context-entry inside the try so client failures there also surface as ContractError. Add tests covering cleanup on success and failure, and fix a stale "does not exist yet" docstring in the test module.
Extends the three generation forms' existing test files with cases for the upcoming gatewayReady gating: blocker shown and Generate disabled when the key is missing, precedence over each form's own validity blocker, and non-generative fields staying editable. useSetup is mocked per file so the "ready" cases reuse the pre-existing assertions unchanged. No implementation yet, so 9 of the new cases fail against the current components.
execute_run and _score_row don't yet wrap their bodies in tracing.run_span/row_span. These tests pin the "no behavior change" contract (unconfigured tracing, cancellation, per-row errors, only_row_ids all identical to today) and, with logfire.testing capturing, assert the expected span tree: one valcore.run parenting N valcore.score_row spans, carrying run_id/kind/status and every metrics key.
Covers expanded state (all commands shown, required vs optional keys marked), collapsed summary when all keys are set, per-key Copy, Recheck re-fetching and collapsing the card, unaffected stat-card rendering in both states, and a rejected setup fetch not blanking the page. Fails against the current OverviewPage since the setup card isn't implemented yet. Co-Authored-By: Claude <noreply@anthropic.com>
Extends EvaluatorFromDataset, RefinePanel, and EvaluatorsPage's generate-action tests, and adds RunLauncher.test.tsx (no prior suite existed), to cover the same gateway gate as the dataset forms: useSetup's gatewayReady disables the action and shows GATEWAY_BLOCKER when the key is unset, and each action's own validity governs it once ready. EvaluatorsPage's scratch-mode Create stays ungated. These tests fail until the components read useSetup and wire the gate in.
Covers execute_experiment's contract ahead of its implementation: metrics parity with runner.execute_run (categorical and numeric), NumericDelta vs runner._agreement, EqualsExpected-equivalent categorical agreement, no agreement for EVAL runs, one RunResult/event per case including recorded case failures, the ExperimentRun marker (case_count, blocked cancellation), and FAILED status on an incompatible dataset. Co-Authored-By: Claude <noreply@anthropic.com>
Covers GET /api/setup (effective key presence, no-leak, no-POST), the
POST /api/datasets/{id}/logfire/push endpoint, and the gateway-key
guard on all seven generative/run-launching handlers -- including the
regression that a keyless run must fail cleanly with zero RunResult
rows instead of one failure per row. None of this passes yet: the
setup route, the push endpoint, and the guard calls don't exist.
Co-Authored-By: Claude <noreply@anthropic.com>
DatasetGenerateForm, GenerateMoreRows, and DatasetFromEvaluator now read useSetup() and prepend GATEWAY_BLOCKER to their blockers list, disabling Generate whenever the gateway key is missing. Non-generative fields stay editable so a dataset can still be prepared before the key is configured.
Shows one row per setup key (label, purpose, required/optional, command with Copy) when any key is unset, and collapses to a quiet summary line once all are configured. A Recheck button re-runs useSetup's fetch.
execute_run now opens a valcore.run span around setup completion through the terminal status update, recording status and every metrics key before close; _score_row wraps its body in a valcore.score_row span. Both spans are no-ops when tracing is unconfigured, so behavior is unchanged.
EvaluatorFromDataset, RefinePanel, EvaluatorsPage's criteria-mode generate, and RunLauncher now read useSetup() and disable their gated action with GATEWAY_BLOCKER when the gateway key is unset, matching the dataset-forms gate. Manual version authoring, saving, and freezing stay ungated. Also completes VersionEditor.test.tsx's api/client mock with a `setup` export: RefinePanel (rendered in the version editor rail) now calls useSetup, which reads it. Co-Authored-By: Claude <noreply@anthropic.com>
…g tests Strengthens the runner-spans test suite with two cases the original plan didn't cover explicitly: that a row's span closes with the correct idx even when _score_row's except branch runs, and that an EVAL-kind run (no agreement evaluator, no metrics) still opens and closes a clean valcore.run span. Co-Authored-By: Claude <noreply@anthropic.com>
…ve routes
Adds GET /api/setup (read-only key presence) and POST
/api/datasets/{id}/logfire/push, guards the seven generative/run-launching
handlers with config.require_gateway_key() so a missing key surfaces as a
client error instead of a 500 or a run full of per-row failures, and wires
configure_tracing/instrument_fastapi into the app factory. Adds the
logfire[fastapi] sub-extra so instrument_fastapi has its opentelemetry
dependency.
Implements execute_experiment as a second execution mode alongside runner.execute_run, sharing metrics.compute_metrics so both engines report identical numbers over the same data. Adds NumericDelta (delegating to runner._agreement) as the numeric agreement evaluator and PersistResults as the CaseLifecycle persistence seam that writes one RunResult and emits one row event per case. Co-Authored-By: Claude <noreply@anthropic.com>
Covers two implied-but-unasserted behaviors: an omitted on_conflict must reach push_dataset as "update" (not None), and the gateway guard on generate_version must fire before store.get_evaluator resolves the id.
The interrupted-case path has no public way to trigger through execute_experiment (this engine has no cancellation), so PersistResults is driven directly to exercise it. Also fixes a pre-existing ruff import-sort violation in the same file. Co-Authored-By: Claude <noreply@anthropic.com>
The guard lived only inside the background task, where ConfigError was swallowed by the broad exception handler, so POST /api/runs returned 200 PENDING and persisted a run even with no gateway key. Move the guard to the top of create_run so a missing key fails synchronously with a 422 and no run is ever created.
…errors Address code review feedback on the experiment engine: validation runs now raise ContractError for unlabeled rows before entering RUNNING (matching runner.execute_run), and exceptions escaping Dataset.evaluate/teardown now transition the run to FAILED with finished_at and an error event instead of leaving it stuck RUNNING. Tests now inspect the evaluators actually passed to dataset_to_evals rather than only recomputed agreement, and use TestModel throughout (a minimal TestModel subclass injects the one failure needed per case-failure test) instead of FunctionModel. Co-Authored-By: Claude <noreply@anthropic.com>
… gateway guard Covers the CLI surfaces task: `experiment` (mirroring `run`'s output and agreeing with it on metrics), `logfire push`, `config set-logfire-token`/`set-logfire-key`, `config get` presence-only display of the new keys, `require_gateway_key` guards on `run`/`experiment`, and the once-per-invocation `configure_tracing` call in the CLI group callback. None of this is implemented yet.
…ay guard Wires up the CLI surfaces for the logfire integration: `experiment` mirrors `run` on top of the second Dataset.evaluate-based engine, `logfire push` delegates to logfire_io.push_dataset, `config set-logfire-token`/`set-logfire-key` persist the two Logfire credentials, and `require_gateway_key()` guards `run` and `experiment` so a keyless invocation fails fast instead of producing N failed rows. `configure_tracing` is now called once per CLI invocation.
require_gateway_key() guards run and experiment now, so every pre-existing happy-path CLI test failed on the guard before its injected agent ever ran. Add the same autouse _gateway_key_present fixture already used in test_api_runs.py for the identical guard on the API surface.
…experiment tests config get checked cfg.gateway_api_key/logfire_token directly, so an env-only gateway key or LOGFIRE_TOKEN reported as absent instead of present. Route through gateway_key_present/logfire_token_present/logfire_api_key_present and add a regression test for the environment-only case. Also switch the experiment command's tests from FunctionModel to TestModel per the test plan, keeping FunctionModel only for the pre-existing run tests.
Add a Setup section covering the Overview setup card and the CLI-only key commands, and a Logfire section covering the optional extra, the valcore.run/score_row span shape, valcore experiment, and logfire push.
…d guard gaps - Write the ExperimentRun marker before RUNNING/evaluate() start (using len(rows) as the initial case count, replaced with len(report.cases) after evaluation) so request_cancel raises for the whole active run instead of accepting a silent no-op cancellation request. - Wrap the RUNNING transition through the terminal status update in run_span, and handle evaluate()/eval-dataset-construction exceptions while the span is still open so status=failed is attached before it closes. - Call config.require_gateway_key() as the first statement in retry_failed so a keyless retry fails before mutating the prior run's status or results.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Logfire integration — run/row tracing spans around
runner.execute_run, a second executionengine built on
pydantic_evals.Dataset.evaluate, and dataset push to Logfire's hosted store —plus a setup walkthrough that surfaces gateway/Logfire key presence and gates every
gateway-dependent action across the API, CLI, and web UI.
Changes
src/valcore/tracing.py: shapesvalcore.run/valcore.score_rowspans around run execution;configure()stays a silent no-op without a token.src/valcore/experiment.py: new experiment engine onDataset.evaluatewithCaseLifecycle-based persistence, sharingmetrics.compute_metricswith the row-by-row runner.src/valcore/logfire_io.py: pushes valcore datasets to Logfire's hosted dataset store via the async API client.src/valcore/spec.py: fixes dataset generics so pushed datasets carry a real output schema instead ofobject.src/valcore/config.py: adds Logfire token/API key fields and effective-presence helpers.src/valcore/models.py: adds theExperimentRuntable for the pydantic-evals engine.src/valcore/runner.py: rewires execution to emit tracing spans and guard on gateway-key presence.src/valcore/api/routes/setup.py(new) androutes/datasets.py,routes/evaluators.py,routes/runs.py: read-only setup endpoint and gateway-key guards on generative and run routes.src/valcore/cli/main.py: adds theexperimentcommand, Logfire push, config setters, and gateway-guard messaging.web/src/components/useSetup.ts(new): hook exposing gateway readiness for gating UI actions.web/src/pages/OverviewPage.tsx: setup walkthrough card showing which keys are configured.web/src/components/DatasetFromEvaluator.tsx,DatasetGenerateForm.tsx,EvaluatorFromDataset.tsx,GenerateMoreRows.tsx,RefinePanel.tsx,RunLauncher.tsx: gate generate/refine/run actions on the gateway key.web/src/api/client.ts,web/src/api/types.ts,web/src/styles.css: read-only setup API client and setup-card styles.README.md,pyproject.toml,uv.lock: document the setup walkthrough and Logfire integration, add the optionallogfireextra.tests/andweb/src/**for every module above (tracing, experiment, logfire_io, spec, config, models, runner, API routes, CLI, and the web components/hook).Test plan
uv run pytest -qpasses, includingtests/test_experiment.py,tests/test_tracing.py,tests/test_logfire_io.py, andtests/test_api_setup.pyuv run ruff check .anduv run ruff format --check .passcd web && npm ci && npx vitest runpasses, includinguseSetup.test.tsxandOverviewPage.test.tsxcd web && npx tsc --noEmitpassesPYDANTIC_AI_GATEWAY_API_KEYis unset, and succeed with it set🤖 Generated by workbench pr_writer