+ The app and the CLI are two front ends over the same SQLite workspace. Authoring is
+ easier in the app; running, exporting, and anything scripted is easier from the command
+ line. Nothing has to be running for the CLI to work — it opens the database directly, so
+ runs do not depend on valcore serve.
+
+ Evaluators, versions, and datasets are addressable by name or by a unique id prefix, so + you rarely need a full id: +
++ An ambiguous value is an error that lists the candidates rather than picking one. Add + characters until it resolves. +
+--version — a version other than the active one
+ --kind validation — compare against labels instead of only recording
+ output
+ --concurrency — how many rows are in flight at once
+ --watch — one line per completed row, rather than a final summary
+ --json — machine-readable results on stdout
+ --min-accuracy — exit non-zero below a threshold, for CI
+
+ valcore config holds credentials and defaults — the gateway key, Logfire
+ tokens, the default model, and the path to the config file. Keys are only ever set from
+ the CLI; no secret crosses HTTP into the browser.
+
+ valcore list shows what the workspace holds, as a table or with{" "}
+ --json. valcore export and valcore import move
+ evaluators and datasets between machines. valcore logfire push sends a
+ dataset to Logfire's hosted store. valcore skills installs the bundled
+ agent skills so a coding agent can drive valcore for you.
+
+ Every command takes --db on the group, which is how you keep a scratch
+ workspace or a per-project database separate from the default under{" "}
+ ~/.valcore:
+
+ Back to
+ A dataset is a table of rows with named columns, and it arrives one of three ways. + Upload a CSV when you already have data. Create a blank dataset and author rows by + hand when you are working from a handful of known cases. Generate one from a + description when you need coverage you do not have yet. +
++ Generation is a starting point, not an answer: a generated dataset is an editable draft + like any other, and its rows are yours to correct. +
++ A generated dataset keeps the request that produced it — the description, the columns + asked for, and the per-column notes — as read-only provenance. Months later that is how + you know what these rows were meant to represent. Uploaded and blank datasets have no + such record, which is normal rather than missing. +
++ Prescribing a label mix is opt-in. Leave it off and the distribution follows whatever + the description asks for. Turn it on to pin the proportions — useful when you need a + rare category represented well enough to measure. The editor works in whole percents + and shows the apportioned row count beside each label, so the number the model is + actually told is never hidden behind a percentage. +
++ Labels are ground truth: what the right answer is, independent of what any judge says. + The labeling grid is built for getting through rows quickly with the keyboard. +
+j / k — move between rows
+ 1–9 — apply a categorical label
+ a — accept the suggested label
+ u — clear the label
+ ? — show the full shortcut list
+ + Every change saves immediately, applied optimistically and rolled back if the write + fails. Cells are editable in place, rows can be added or deleted, and per-column notes + record what a column is supposed to hold. +
++ A dataset needs no labels to be scored. An ordinary run just records what the judge + said. Labels are only required for a validation run, which compares the judge against + them to measure agreement. +
++ A dataset that turned out too small can be extended in place: generating more rows + reuses the original request so the additions match the shape and intent of what is + already there. +
++ Export writes a dataset out as a Python script or as a portable JSON package, and the + CLI reads it back: +
+
+ Work with data in
+ An evaluator is an LLM-as-judge: a prompt, a model, and an output shape that together + score one thing about a row of data. It reads the columns you give it and returns a + label — a category from a fixed set, or a number — plus its reasoning. +
++ An evaluator is not tied to a dataset. The same judge runs over any dataset that + supplies the columns it requires, which is what makes a judge comparable across data. +
++ The output field defines what a judge returns. Categorical fields declare their label + space up front — the set of allowed values — and that space is what the labeling grid + offers and what agreement metrics are computed over. Numeric fields declare a range + instead, and are scored with error measures rather than a confusion matrix. +
++ Declaring the label space is what lets valcore tell a disagreement from an invalid + answer. A judge that returns something outside the space is a failure, not a low score. +
++ Every change to a judge lands as a version, and one version is active — the one used + when you do not name another. Editing an unsaved draft edits in place. Editing a frozen + version copies it first, so a version that has already produced runs is never mutated + underneath them. +
++ That is the point of freezing: a run records which version produced it, so a score + stays attributable. When two versions exist you can read the exact difference between + them in the version diff rather than guessing from timestamps. +
++ A judge can be granted capabilities when reading the row is not enough. FileSystem + gives it a rooted directory to read from; Shell gives it an allow-listed set of + commands and a timeout. Both are opt-in per version and configured alongside the + prompt. +
++ Grant the narrowest thing that answers the question. A capability widens what a judge + can see, which also widens what can change between runs. +
++ An evaluator and a dataset have to agree on columns, so rather than retype that shape, + generate one from the other. A dataset generated from a version always gets that + version's required columns; you can name extra columns, and per-column notes say + what each should contain. Suggested labels are optional — ask for them when you want + the model to propose ground truth, and the label space comes from the evaluator. +
++ The reverse direction works too: generate an evaluator from a dataset and it is + drafted against that dataset's columns. The result is an editable draft, not a + saved version. +
+
+ Author judges in
+ Runs are driven from the command line, against the active version unless you name + another: +
+
+ See
+ The setup card on
+ valcore reaches models through the Pydantic AI Gateway, and that is currently the only + route — there is no direct-to-provider client and no per-provider key. Without it, + generation and runs are unavailable and the UI says why. +
++ Everything that does not call a model still works with no key at all: authoring by + hand, uploading a CSV, editing rows, hand-labeling, and every export. +
++ Run it with no argument to be prompted without the key echoing to your terminal or + landing in shell history. +
++ Create the key in the Pydantic AI Gateway and paste it into the command above. The + gateway documentation covers account setup and where keys are issued: +
+
+
+ Model strings are always gateway/<provider>:<model> — for
+ example gateway/anthropic:claude-sonnet-5, which is the default. Valid
+ providers are anthropic, openai, google,{" "}
+ google-cloud, bedrock, and groq. A string that
+ does not match this shape is rejected before any request is made, so a bare model name
+ fails immediately with a clear error rather than at call time.
+
+ With a write token configured, each run opens a valcore.run span carrying
+ the evaluator version, dataset, and concurrency, with one child span per scored row.
+ On close, the run span records its status and each agreement metric as attributes, so
+ a Logfire query can filter runs by accuracy directly.
+
+ A write token belongs to one Logfire project and is created from that project's + settings. Create the project first, then issue the token there: +
+
+
uv tool install 'valcore[logfire]'. The gateway already reports
+ the LLM calls themselves; valcore adds only the surrounding run and row context, and
+ deliberately does not re-report the calls, which would double-count tokens and cost.
+ + The API key is a separate credential from the write token, and it is only needed for + one thing: pushing a dataset to Logfire's hosted dataset store. +
+
+ It must carry the project:read_datasets and{" "}
+ project:write_datasets scopes. A key without them will authenticate and
+ then fail on the push. API keys are issued from your Logfire account settings:
+
+
+ Keys live in ~/.valcore/config.toml, written with mode 0600,
+ and are exported into the environment when a command runs. An already-exported
+ environment variable always wins over the stored value, which is what you want in CI:
+
To check what is currently configured without revealing anything:
+
+ The stored key is masked unless you pass --show-key. See{" "}
+
+ A run is one evaluator version scored over one dataset, recorded with its results. The + version is part of the record, so a score stays attributable after the judge moves on. +
+Runs are started from the command line or from the launcher in the app:
++ An ordinary run records what the judge returned for each row. That is enough to inspect + behavior, spot invalid outputs, and compare two judges against each other. +
++ A validation run additionally compares the judge against the dataset's labels and + reports agreement. It requires a fully labeled dataset, so the option is unavailable — + with the reason shown — when rows are still unlabeled. +
++ What you get depends on the output field. A categorical judge produces a confusion + matrix: rows are the label, columns are what the judge said, and the diagonal is + agreement. Reading off the diagonal tells you which categories a judge confuses, which + a single accuracy number hides. +
++ A numeric judge produces error measures instead — mean absolute error and root mean + square error. MAE is the typical miss; RMSE punishes large misses harder, so a gap + between them means a few rows are badly wrong rather than everything being slightly + off. +
++ Comparison puts two runs over the same dataset side by side, disagreements first, + because the rows where they differ are the only ones that explain a change. Comparing + runs over different datasets is refused rather than rendered — the numbers would not be + about the same thing. +
+
+ Open
+ valcore run uses valcore's own runner. valcore experiment{" "}
+ scores the same pairing through pydantic_evals.Dataset.evaluate instead,
+ which is the path to take when you want results inside the pydantic-evals ecosystem.
+
+ A validation run can fail on its own accuracy, which is what makes it usable as a + check rather than a report: +
+serve is up.
+
+ See
{children}
+
+