Skip to content

Let a board declare what to observe and what each stage is judged by #336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
takashi-matsuyama merged 3 commits into from
Oct 9, 2026
Merged

Let a board declare what to observe and what each stage is judged by #336

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
77e6bec
basou: Let a board declare what to observe and what each stage is jud…
takashi-matsuyama Oct 8, 2026
3c0a544
basou: Refuse an uppercase npm package name in the board's tests
takashi-matsuyama Oct 8, 2026
695279d
basou: Settle what the review found in the board's version 2 shapes
takashi-matsuyama Oct 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,31 @@
All notable changes to **basou** are recorded here. The project follows
[Semantic Versioning](https://semver.org/spec/v2.0.0.html) starting with v0.1.0.

## Unreleased

### Added

- **A board can declare what to observe outside basou, and what to look at
when judging each stage (`board_version: 2`, experimental).** `observe`
lists observations, each under the `key` its record holds it by, of seven
kinds: an npm package's published version (`npm_version`, `package`), a
GitHub repository's latest release, open issues or open pull requests
(`github_release`, `github_open_issues`, `github_open_prs`, `repo` as
`owner/name`), the conclusion of a workflow's latest run on a branch
(`github_ci`, `repo`, `workflow` as its file name, `branch`, by default
`main`), the version a web page shows (`page_version`, an `https://` URL
with no user name or password), and one only a person can make (`manual`,
`how`, in words). basou does not make them and sends nothing; each value is
held to its shape when the board is read, since the steps a judge follows
turn it into a command. A stage may add `look` (what to look at) and
`notes` (what to mind when judging it). A version 1 board is read as
before, and is told that version 2 reads these keys when it has them. A
record of a board that declares observations must give exactly those, one
not made as `null` with its error: one left out, or one the board does not
declare, is refused, nothing written. Records are now `record_version: 2`,
holding the observations and the stages' `look` and `notes` the board
declared; a record of version 1 is still read, compared with and drawn.

## 0.67.0 — 2026-10-08

### Added
Expand Down
34 changes: 32 additions & 2 deletions packages/cli/src/commands/board.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -925,7 +925,7 @@ describe("basou board record", () => {
expect(await readdir(join(repo, "board", "records"))).toEqual([name]);
const written = JSON.parse(await readFile(join(repo, result.record ?? ""), "utf8"));
expect(written).toMatchObject({
record_version: 1,
record_version: 2,
measure: { digest: input.measure_digest },
judged_by: { model: "Claude Opus 5.5", self_reported: true },
order_anomalies: [],
Expand All @@ -938,6 +938,36 @@ describe("basou board record", () => {
expect(process.exitCode ?? 0).toBe(0);
});

it("records a version 2 board with exactly the observations it declares", async () => {
const board = JSON.parse(boardYaml([MD]));
board.board_version = 2;
board.stages["05"].look = ["the published version"];
board.observe = [{ key: "npm_cli", kind: "npm_version", package: "@scope/cli" }];
const { repo, input } = await judged(undefined, JSON.stringify(board));
const { err } = capture();
await runBoardRecord({}, fed(repo, input));
expect(err.join("\n")).toContain(
"observed.npm_cli: the board declares it, so it must be given (null, with an error, when it was not observed)",
);
expect(err.join("\n")).toContain("nothing was written");
expect(process.exitCode).toBe(1);
process.exitCode = undefined;
const observed = {
npm_cli: { value: null, observed_at: "2026-10-08", source: "npm", error: "offline" },
};
const result = await doRunBoardRecord({}, fed(repo, { ...input, observed }));
const written = JSON.parse(await readFile(join(repo, result.record ?? ""), "utf8"));
expect(written).toMatchObject({
record_version: 2,
measure: { board_version: 2 },
observed,
declaration: {
observe: [{ key: "npm_cli", kind: "npm_version", package: "@scope/cli" }],
stages: { "05": { meaning: "stage 05", look: ["the published version"] } },
},
});
});

it("checks and measures but writes nothing with --dry-run, and reports as JSON", async () => {
const { repo, input } = await judged();
const { out } = capture();
Expand Down Expand Up @@ -983,7 +1013,7 @@ describe("basou board record", () => {
);
expect(out.join("\n")).toBe(`Recorded ${result.record}\nNot compared with a previous record.`);
await expect(readFile(join(deeper, result.record ?? ""), "utf8")).resolves.toContain(
'"record_version": 1',
'"record_version": 2',
);
});

Expand Down
5 changes: 3 additions & 2 deletions packages/cli/src/commands/board.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,9 @@ read (nothing is printed on stdout).`,
"after",
`
The input is a JSON object: measure_digest (the digest of the measurement the
judgement saw), observed, cells (every lane at every stage), prose, judged_by
and axis_review. The board is measured again, and nothing is written when the
judgement saw), observed (exactly the observations the board declares, when it
declares any), cells (every lane at every stage), prose, judged_by and
axis_review. The board is measured again, and nothing is written when the
digest differs. Records are never written under a .basou/ directory. Exit
codes: 0 when the record was written, or with --dry-run when everything
checked out (nothing is written then); 1 when it was refused (nothing is
Expand Down
265 changes: 256 additions & 9 deletions packages/core/src/board/declaration.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,17 +190,16 @@ describe("parseBoardDeclaration: accepted declarations", () => {

describe("parseBoardDeclaration: board_version", () => {
it("stops at an unknown board_version, naming it, whatever else is wrong", () => {
expect(errorsOf(parse({ ...board(), board_version: 2, title: "", extra: 1 }))).toEqual([
"board_version: this basou reads board_version 1, not 2",
expect(errorsOf(parse({ ...board(), board_version: 3, title: "", extra: 1 }))).toEqual([
"board_version: this basou reads board_version 1 or 2, not 3",
]);
});

it("refuses a board_version that is missing or not a number", () => {
const { board_version: _v, ...missing } = board();
expect(errorsOf(parse(missing))).toEqual(["board_version: Invalid input: expected 1"]);
expect(errorsOf(parse({ ...board(), board_version: "1" }))).toEqual([
"board_version: Invalid input: expected 1",
]);
const reads = "board_version: must be a board version this basou reads (1 or 2)";
expect(errorsOf(parse(missing))).toEqual([reads]);
expect(errorsOf(parse({ ...board(), board_version: "1" }))).toEqual([reads]);
});
});

Expand Down Expand Up @@ -1134,11 +1133,11 @@ describe("parseBoardDeclaration: what the review found", () => {
});

it("names an unknown board_version before a YAML warning", () => {
const plain = stringify({ ...board(), board_version: 2 });
const plain = stringify({ ...board(), board_version: 3 });
expect(plain).toContain("title: A board\n");
const text = plain.replace("title: A board\n", "title: !v2tag A board\n");
const text = plain.replace("title: A board\n", "title: !v3tag A board\n");
expect(errorsOf(parseBoardDeclaration(text, { manifestRepoPaths: REPOS }))).toEqual([
"board_version: this basou reads board_version 1, not 2",
"board_version: this basou reads board_version 1 or 2, not 3",
]);
});

Expand Down Expand Up @@ -1211,3 +1210,251 @@ describe("parseBoardDeclaration: what the review found", () => {
}
});
});

// A board of version 2, observing one thing of each kind.
function boardV2(): Record<string, unknown> {
const doc = board();
entry(doc.stages, "05").look = ["the published version on npm"];
entry(doc.stages, "05").notes = ["on main but not published is part"];
return {
...doc,
board_version: 2,
observe: [
{ key: "npm_cli", kind: "npm_version", package: "@scope/cli" },
{ key: "github_release", kind: "github_release", repo: "owner/name" },
{ key: "github_open_issues", kind: "github_open_issues", repo: "owner/name" },
{ key: "github_open_prs", kind: "github_open_prs", repo: "owner/name.js" },
{ key: "github_main_ci", kind: "github_ci", repo: "owner/name", workflow: "quality.yml" },
{ key: "site_en", kind: "page_version", url: "https://example.com/" },
{ key: "db_users", kind: "manual", how: "count the users in production, read only" },
],
};
}

// A version 2 board whose only observation is `observe`.
function withObserve(observe: Record<string, unknown>): Record<string, unknown> {
return { ...boardV2(), observe: [observe] };
}

describe("parseBoardDeclaration: board_version 2", () => {
it("accepts what to observe and what to look at for each stage, filling the default branch", () => {
const result = parse(boardV2());
if (!result.ok) throw new Error(result.errors.join("\n"));
const { declaration } = result;
expect(declaration.board_version).toBe(2);
expect(declaration.observe.map((o) => [o.key, o.kind])).toEqual([
["npm_cli", "npm_version"],
["github_release", "github_release"],
["github_open_issues", "github_open_issues"],
["github_open_prs", "github_open_prs"],
["github_main_ci", "github_ci"],
["site_en", "page_version"],
["db_users", "manual"],
]);
expect(declaration.observe[4]).toEqual({
key: "github_main_ci",
kind: "github_ci",
repo: "owner/name",
workflow: "quality.yml",
branch: "main",
});
expect(declaration.stages["05"]).toEqual({
meaning: "stage 05",
look: ["the published version on npm"],
notes: ["on main but not published is part"],
});
expect(declaration.stages["01"]).toEqual({ meaning: "stage 01" });
});

it("reads a version 2 board without observe as observing nothing", () => {
const { observe: _o, ...doc } = boardV2();
const result = parse(doc);
if (!result.ok) throw new Error(result.errors.join("\n"));
expect(result.declaration.observe).toEqual([]);
});

it("still reads version 1, as observing nothing", () => {
const result = parse(board());
if (!result.ok) throw new Error(result.errors.join("\n"));
expect(result.declaration.board_version).toBe(1);
expect(result.declaration.observe).toEqual([]);
});

it("says that version 2 reads observe and a stage's look and notes when version 1 has them", () => {
const doc = boardV2();
doc.board_version = 1;
entry(doc.stages, "03").colour = "red";
expect(errorsOf(parse(doc))).toEqual([
"stages.03: unknown key 'colour'",
"stages.05: 'look', 'notes' are read from board_version 2 on; this file is board_version 1",
"(top level): 'observe' is read from board_version 2 on; this file is board_version 1",
]);
});

it("tells a key version 2 reads apart from one no version reads, where it stands", () => {
const doc = board();
entry(doc.stages, "02").colour = "red";
entry(doc.stages, "02").look = ["x"];
entry(doc.stages, "03").observe = [];
doc.extra = 1;
doc.look = ["x"];
doc.observe = [];
expect(errorsOf(parse(doc))).toEqual([
"stages.02: unknown key 'colour'",
"stages.02: 'look' is read from board_version 2 on; this file is board_version 1",
"stages.03: unknown key 'observe'",
"(top level): unknown keys 'extra', 'look'",
"(top level): 'observe' is read from board_version 2 on; this file is board_version 1",
]);
});

it("refuses an unknown kind, a kind's missing keys, keys of another kind and a duplicate key", () => {
const doc = boardV2();
doc.observe = [
{ key: "a", kind: "rss_feed", url: "https://example.com/" },
{ key: "b", kind: "github_ci", repo: "owner/name" },
{ key: "c", kind: "npm_version", package: "x", repo: "owner/name" },
{ key: "d", kind: "manual" },
{ key: "d", kind: "manual", how: "ask" },
{ kind: "manual", how: "ask" },
{ key: "Bad", kind: "manual", how: "ask" },
];
expect(errorsOf(parse(doc))).toEqual([
"observe[0].kind: Invalid discriminator value. Expected 'npm_version' | 'github_release' | 'github_open_issues' | 'github_open_prs' | 'github_ci' | 'page_version' | 'manual'",
"observe[1].workflow: Invalid input: expected string, received undefined",
"observe[2]: unknown key 'repo'",
"observe[3].how: Invalid input: expected string, received undefined",
"observe[5].key: Invalid input: expected string, received undefined",
"observe[6].key: must start with a lowercase letter and use only a-z, 0-9, '_' and '-'",
"observe[4].key: duplicate observation key 'd' (first at observe[3])",
]);
});

it.each<[string, string]>([
["Cli", "uppercase"],
["@Scope/cli", "an uppercase scope"],
["scope/cli", "a slash without a scope"],
["-x", "a leading '-'"],
["@scope/-x", "a leading '-' after the scope"],
["a b", "a space"],
["a;rm", "a ';'"],
["@scope", "a scope alone"],
["", "nothing"],
["a".repeat(215), "too long"],
])("refuses the npm package %j (%s)", (pkg) => {
expect(errorsOf(parse(withObserve({ key: "n", kind: "npm_version", package: pkg })))).toEqual([
"observe[0].package: must be an npm package name such as name or @scope/name, in lowercase",
]);
});

it.each<[string]>([
["name"],
["-owner/name"],
["_owner/name"],
["owner/name/more"],
["owner/.."],
["owner/."],
["own er/name"],
["owner/na'me"],
[`${"o".repeat(65)}/name`],
[`owner/${"n".repeat(101)}`],
])("refuses the GitHub repository %j", (repo) => {
for (const kind of ["github_release", "github_open_issues", "github_open_prs"]) {
expect(errorsOf(parse(withObserve({ key: "g", kind, repo })))).toEqual([
"observe[0].repo: must be a GitHub repository written as owner/name",
]);
}
});

it("takes a managed user's owner, with '_', and names at their longest", () => {
const doc = boardV2();
doc.observe = [
{ key: "a", kind: "github_release", repo: "mona_corp/name" },
{ key: "b", kind: "github_release", repo: `${"o".repeat(64)}/${"n".repeat(100)}` },
{ key: "c", kind: "github_ci", repo: "o/n", workflow: `${"w".repeat(251)}.yml` },
];
expect(parse(doc).ok).toBe(true);
});

it("refuses a workflow that is not a file name and a branch a command could misread", () => {
const doc = boardV2();
doc.observe = [
{ key: "a", kind: "github_ci", repo: "o/n", workflow: ".github/workflows/q.yml" },
{ key: "b", kind: "github_ci", repo: "o/n", workflow: "quality" },
{ key: "c", kind: "github_ci", repo: "o/n", workflow: "-q.yml" },
{ key: "d", kind: "github_ci", repo: "o/n", workflow: "q.yml", branch: "-x" },
{ key: "e", kind: "github_ci", repo: "o/n", workflow: "q.yml", branch: "a..b" },
{ key: "f", kind: "github_ci", repo: "o/n", workflow: "q.yml", branch: "a b" },
{ key: "g", kind: "github_ci", repo: "o/n", workflow: "q.yml", branch: "release/" },
{ key: "h", kind: "github_ci", repo: "o/n", workflow: "q.yaml", branch: "release/1.x" },
{ key: "i", kind: "github_ci", repo: "o/n", workflow: "q.yml", branch: "a//b" },
{ key: "j", kind: "github_ci", repo: "o/n", workflow: "q.yml", branch: "v1." },
{ key: "k", kind: "github_ci", repo: "o/n", workflow: "Q.YML" },
{ key: "l", kind: "github_ci", repo: "o/n", workflow: `${"w".repeat(252)}.yml` },
];
const workflow = "must be the file name of a workflow, such as quality.yml";
const branch =
"must be a branch name such as main, using only A-Z, a-z, 0-9, '.', '_', '-' and '/'";
expect(errorsOf(parse(doc))).toEqual([
`observe[0].workflow: ${workflow}`,
`observe[1].workflow: ${workflow}`,
`observe[2].workflow: ${workflow}`,
`observe[3].branch: ${branch}`,
`observe[4].branch: ${branch}`,
`observe[5].branch: ${branch}`,
`observe[6].branch: ${branch}`,
`observe[8].branch: ${branch}`,
`observe[9].branch: ${branch}`,
`observe[10].workflow: ${workflow}`,
`observe[11].workflow: ${workflow}`,
]);
});

it.each<[string, string]>([
["http://example.com/", "must be an https:// URL"],
["HTTPS://example.com/", "must be an https:// URL"],
["ftp://example.com/", "must be an https:// URL"],
["example.com", "must be an https:// URL"],
["https://", "must name a host"],
["https:///example.com", "must name a host"],
["https://?a=1", "must name a host"],
["https://example.com/a b", "must be printable ASCII with no spaces"],
["https://example.com/é", "must be printable ASCII with no spaces"],
["https://example.com/'; rm -rf ~", "must be printable ASCII with no spaces"],
["https://example.com/a'b", "must not contain quotes, backslashes, '<' or '>'"],
["https://example.com/a\\b", "must not contain quotes, backslashes, '<' or '>'"],
["https://user:secret@example.com/", "must not carry a user name or a password"],
["https://user@example.com/", "must not carry a user name or a password"],
["https://:secret@example.com/", "must not carry a user name or a password"],
["https://@example.com/", "must not carry a user name or a password"],
['https://example.com/a"b', "must not contain quotes, backslashes, '<' or '>'"],
["https://example.com/a`b`", "must not contain quotes, backslashes, '<' or '>'"],
["https://example.com/<a>", "must not contain quotes, backslashes, '<' or '>'"],
["https://exa%zzmple.com/", "must be an https:// URL"],
])("refuses the page %j", (url, problem) => {
expect(errorsOf(parse(withObserve({ key: "p", kind: "page_version", url })))).toEqual([
`observe[0].url: ${problem}`,
]);
});

it("takes a page's query and fragment, and a manual observation's how as written", () => {
const doc = boardV2();
doc.observe = [
{ key: "p", kind: "page_version", url: "https://example.com:8443/ja/?a=1&b=2#v" },
{ key: "m", kind: "manual", how: "count with `psql`; it's read only, $(not run)" },
];
const result = parse(doc);
if (!result.ok) throw new Error(result.errors.join("\n"));
expect(result.declaration.observe).toEqual(doc.observe);
});

it("refuses an empty look or note of a stage", () => {
const doc = boardV2();
entry(doc.stages, "02").look = [""];
entry(doc.stages, "02").notes = "not a list";
expect(errorsOf(parse(doc))).toEqual([
"stages.02.look[0]: must be a non-empty string",
"stages.02.notes: Invalid input: expected array, received string",
]);
});
});
Loading
Loading