Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Either way you get an `valcore` command on your `PATH`.

## Web UI

`valcore serve` opens a dark-themed web UI with four surfaces:
`valcore serve` opens a dark-themed web UI with five surfaces:

- **Overview** — the landing page, summarizing what you have and pointing to the next
step.
Expand All @@ -65,6 +65,12 @@ Either way you get an `valcore` command on your `PATH`.
from a description.
- **Runs** — inspect completed runs, their metrics, and per-row scores, and compare runs
against each other.
- **Docs** — how the product works, in four tabs: Evals, Datasets, Runs, and CLI.

The docs surface explains the concepts the other four assume — label spaces, frozen
versions, run kinds, agreement metrics — so you can read them without leaving the app.
It covers the workflow; this README stays the reference for install, credentials,
portable packages, CI, and Logfire.

## Quickstart

Expand All @@ -80,6 +86,9 @@ valcore serve
and datasets in the UI, then drive runs from the command line. Both can be written by
hand or generated from a description; a generated result is an editable draft either way.

From there, the **Docs** tab in the app walks through the full workflow — authoring a
judge, getting labeled rows, and reading what a run measured.

## Seeding one from the other

An evaluator and a dataset have to agree on columns, so rather than retype that shape you
Expand Down
46 changes: 46 additions & 0 deletions web/src/App.test.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
// Route-table coverage. DocsPage.test.tsx mounts the docs routes itself, which proves the
// component but not that App declares them — this file is what fails if /docs is missing
// from the real route table.
//
// Only docs paths are exercised: the other pages fetch on mount, and this suite is about
// wiring, not about mocking every endpoint.
import { afterEach, describe, expect, it } from "vitest";
import { cleanup, render, screen } from "@testing-library/react";
import { MemoryRouter } from "react-router-dom";
import App from "./App";
import { DOCS } from "./docs/registry";

afterEach(() => {
cleanup();
});

function renderApp(path: string) {
return render(
<MemoryRouter initialEntries={[path]}>
<App />
</MemoryRouter>,
);
}

describe("App routes", () => {
it("serves the docs section at /docs", () => {
renderApp("/docs");

expect(screen.getByRole("heading", { level: 1 }).textContent).toBe(DOCS[0].title);
});

it("serves a docs tab at /docs/:slug", () => {
renderApp("/docs/runs");

expect(screen.getByRole("heading", { level: 1 }).textContent).toBe("Runs");
});

it("renders the docs section inside the app shell", () => {
renderApp("/docs");

// Two navs: the sidebar and the tab strip. If docs were declared outside the layout
// route, the sidebar would vanish while reading them.
expect(screen.getAllByRole("navigation")).toHaveLength(2);
expect(screen.getByRole("link", { name: "Docs" }).getAttribute("href")).toBe("/docs");
});
});
5 changes: 5 additions & 0 deletions web/src/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import OverviewPage from "./pages/OverviewPage";
import EvaluatorsPage from "./pages/EvaluatorsPage";
import DatasetsPage from "./pages/DatasetsPage";
import RunsPage from "./pages/RunsPage";
import DocsPage from "./pages/DocsPage";

// All routes are declared here up front. The route table is maintained alongside the
// pages, so adding or moving a route means editing this file.
Expand All @@ -19,6 +20,10 @@ export default function App() {
<Route path="/runs" element={<RunsPage />} />
<Route path="/runs/compare" element={<RunsPage />} />
<Route path="/runs/:id" element={<RunsPage />} />
{/* Both point at DocsPage: the bare path renders the first tab, so the section
always has content and a stale slug degrades instead of blanking. */}
<Route path="/docs" element={<DocsPage />} />
<Route path="/docs/:slug" element={<DocsPage />} />
</Route>
</Routes>
);
Expand Down
37 changes: 35 additions & 2 deletions web/src/components/Layout.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,47 @@ describe("Layout nav", () => {
expect(screen.getByRole("link", { name: "Datasets" })).toBeTruthy();
expect(screen.getByRole("link", { name: "Runs" })).toBeTruthy();
expect(screen.getByRole("link", { name: "Compare" })).toBeTruthy();
expect(screen.getByRole("link", { name: "Docs" })).toBeTruthy();
});

it("renders exactly the five expected nav links and nothing else", () => {
it("renders exactly the six expected nav links and nothing else", () => {
renderLayout("/");

// Guards against a stray link (e.g. the brand wordmark accidentally becoming
// a link, or a footer/version badge that this task must not add).
expect(screen.getAllByRole("link")).toHaveLength(5);
expect(screen.getAllByRole("link")).toHaveLength(6);
});

it("orders Docs directly under Overview, above the labelled groups", () => {
renderLayout("/");

// Docs sits with Overview as the two ungrouped entries at the top: it is read
// before you have anything to author or measure, so it should not be buried under
// the working surfaces.
expect(screen.getAllByRole("link").map((link) => link.textContent)).toEqual([
"Overview",
"Docs",
"Evaluators",
"Datasets",
"Runs",
"Compare",
]);
});

it("points Docs at /docs", () => {
renderLayout("/");

expect(screen.getByRole("link", { name: "Docs" }).getAttribute("href")).toBe("/docs");
});

it("keeps Docs active on a docs sub-route", () => {
renderLayout("/docs/datasets");

// Docs deliberately omits `end`: every /docs/:slug tab must keep the nav item
// lit, otherwise the sidebar goes blank-looking while reading any tab but the
// first.
expect(screen.getByRole("link", { name: "Docs" }).getAttribute("aria-current")).toBe("page");
expect(screen.getByRole("link", { name: "Overview" }).getAttribute("aria-current")).toBeNull();
});

it("points Overview at / and Compare at /runs/compare", () => {
Expand Down
10 changes: 8 additions & 2 deletions web/src/components/Layout.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { NavLink, Outlet } from "react-router-dom";
import {
CompareIcon,
DatasetIcon,
DocsIcon,
EvaluatorIcon,
OverviewIcon,
RunIcon,
Expand All @@ -11,9 +12,12 @@ import {
type NavItem = { to: string; label: string; Icon: ComponentType<{ className?: string }> };
type NavSection = { label: string; items: NavItem[] };

// Overview stands alone above the labelled groups; the two sections mirror the
// author-then-measure flow of the product.
// Overview and Docs are the two ungrouped entries at the top; the labelled sections
// below them mirror the author-then-measure flow of the product. Docs sits up here
// rather than at the bottom because it is what you read before you have anything to
// author or measure — a new user needs it first, not last.
const OVERVIEW: NavItem = { to: "/", label: "Overview", Icon: OverviewIcon };
const DOCS: NavItem = { to: "/docs", label: "Docs", Icon: DocsIcon };
const SECTIONS: NavSection[] = [
{
label: "Author",
Expand Down Expand Up @@ -56,6 +60,8 @@ export default function Layout() {
</div>
{/* `end` keeps the "/" link from matching every route and staying active. */}
<NavItemLink {...OVERVIEW} end />
{/* No `end`: every /docs/:slug tab keeps the nav item lit. */}
<NavItemLink {...DOCS} />
{SECTIONS.map((section) => (
<div key={section.label}>
<div className="nav-section-label">{section.label}</div>
Expand Down
2 changes: 2 additions & 0 deletions web/src/components/icons.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import {
ChevronIcon,
CompareIcon,
DatasetIcon,
DocsIcon,
EvaluatorIcon,
InfoIcon,
OverviewIcon,
Expand All @@ -26,6 +27,7 @@ const ICONS = [
["InfoIcon", InfoIcon],
["ChevronIcon", ChevronIcon],
["PlusIcon", PlusIcon],
["DocsIcon", DocsIcon],
] as const;

describe("icons", () => {
Expand Down
12 changes: 12 additions & 0 deletions web/src/components/icons.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -107,3 +107,15 @@ export function PlusIcon(props: IconProps): JSX.Element {
</Svg>
);
}

// An open book: two facing pages over a spine. Distinct from DatasetIcon's stacked
// rules at 16px, which matters because both sit in the same nav column.
export function DocsIcon(props: IconProps): JSX.Element {
return (
<Svg {...props}>
<path d="M12 6 C10 4.5 7.5 4 4 4 L4 18 C7.5 18 10 18.5 12 20" />
<path d="M12 6 C14 4.5 16.5 4 20 4 L20 18 C16.5 18 14 18.5 12 20" />
<line x1="12" y1="6" x2="12" y2="20" />
</Svg>
);
}
90 changes: 90 additions & 0 deletions web/src/docs/content/Cli.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
// The CLI tab. Deliberately not a mirror of the README's command table: the README stays
// canonical for the exhaustive listing (it is the PyPI long_description, where a reader
// cannot reach these pages), and this tab teaches the usage patterns that table does not.

import { CodeBlock, DocLink, DocNote, DocPage, DocSection } from "../primitives";

export function Cli(): JSX.Element {
return (
<DocPage>
<DocSection title="How the CLI relates to this app">
<p>
The app and the CLI are two front ends over the same SQLite workspace. Authoring is
easier in the app; running, exporting, and anything scripted is easier from the command
line. Nothing has to be running for the CLI to work — it opens the database directly, so
runs do not depend on <code>valcore serve</code>.
</p>
<CodeBlock>valcore serve</CodeBlock>
</DocSection>

<DocSection title="Naming things">
<p>
Evaluators, versions, and datasets are addressable by name or by a unique id prefix, so
you rarely need a full id:
</p>
<CodeBlock>valcore run 7f3a my-dataset</CodeBlock>
<p>
An ambiguous value is an error that lists the candidates rather than picking one. Add
characters until it resolves.
</p>
</DocSection>

<DocSection title="Flags that shape a run">
<ul>
<li>
<code>--version</code> — a version other than the active one
</li>
<li>
<code>--kind validation</code> — compare against labels instead of only recording
output
</li>
<li>
<code>--concurrency</code> — how many rows are in flight at once
</li>
<li>
<code>--watch</code> — one line per completed row, rather than a final summary
</li>
<li>
<code>--json</code> — machine-readable results on stdout
</li>
<li>
<code>--min-accuracy</code> — exit non-zero below a threshold, for CI
</li>
</ul>
</DocSection>

<DocSection title="Which group to reach for">
<p>
<code>valcore config</code> holds credentials and defaults — the gateway key, Logfire
tokens, the default model, and the path to the config file. Keys are only ever set from
the CLI; no secret crosses HTTP into the browser.
</p>
<CodeBlock>valcore config set-key</CodeBlock>
<p>
<code>valcore list</code> shows what the workspace holds, as a table or with{" "}
<code>--json</code>. <code>valcore export</code> and <code>valcore import</code> move
evaluators and datasets between machines. <code>valcore logfire push</code> sends a
dataset to Logfire&apos;s hosted store. <code>valcore skills</code> installs the bundled
agent skills so a coding agent can drive valcore for you.
</p>
</DocSection>

<DocSection title="Pointing at another database">
<p>
Every command takes <code>--db</code> on the group, which is how you keep a scratch
workspace or a per-project database separate from the default under{" "}
<code>~/.valcore</code>:
</p>
<CodeBlock>valcore --db ./evals.sqlite list evaluators</CodeBlock>
<DocNote>
For the complete command table, plus install, CI recipes, portable packages, and
Logfire setup, see the README in the repository. This tab covers the patterns, not
every flag.
</DocNote>
<p>
Back to <DocLink to="/docs/runs">Runs</DocLink> for what the run commands measure.
</p>
</DocSection>
</DocPage>
);
}
Loading
Loading