Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .github/workflows/python-quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,47 @@ jobs:
shell: pwsh
run: "& plugins/capability/darrow-git/tests/fresh-install.test.ps1"

tickets-windows:
name: Tickets Python ${{ matrix.python-version }} on windows-latest
needs: changes
if: contains(needs.changes.outputs.packages, 'plugins/capability/darrow-tickets-github/backend')
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13"]
runs-on: windows-latest
defaults:
run:
shell: bash
env:
UV_PYTHON: ${{ matrix.python-version }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- uses: astral-sh/setup-uv@v6
with:
enable-cache: true
- run: scripts/check-python --package plugins/capability/darrow-tickets-github/backend

tickets-fresh-install:
name: Fresh install tickets on ${{ matrix.os }}
needs: changes
if: contains(needs.changes.outputs.packages, 'plugins/capability/darrow-tickets-github/backend')
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.13"
- uses: astral-sh/setup-uv@v6
- run: uv run --quiet --frozen --no-dev --project plugins/capability/darrow-tickets-github/backend python plugins/capability/darrow-tickets-github/backend/tests/fresh_install.py

inventory:
name: Python inventory guard
runs-on: ubuntu-latest
Expand Down Expand Up @@ -333,6 +374,8 @@ jobs:
- verification-windows
- git-windows
- git-fresh-install
- tickets-windows
- tickets-fresh-install
- inventory
- skill-authoring-fresh-install
- observability-fresh-install
Expand Down Expand Up @@ -363,6 +406,9 @@ jobs:
GIT_SELECTED: ${{ contains(needs.changes.outputs.packages, 'plugins/capability/darrow-git/backend') }}
GIT_WINDOWS_RESULT: ${{ needs.git-windows.result }}
GIT_INSTALL_RESULT: ${{ needs.git-fresh-install.result }}
TICKETS_SELECTED: ${{ contains(needs.changes.outputs.packages, 'plugins/capability/darrow-tickets-github/backend') }}
TICKETS_WINDOWS_RESULT: ${{ needs.tickets-windows.result }}
TICKETS_INSTALL_RESULT: ${{ needs.tickets-fresh-install.result }}
run: |
scripts/verify-python-quality-results \
"$CHANGES_RESULT" "$INVENTORY_RESULT" \
Expand All @@ -378,3 +424,7 @@ jobs:
if [ "$GIT_SELECTED" = true ]; then expected=success; fi
test "$GIT_WINDOWS_RESULT" = "$expected"
test "$GIT_INSTALL_RESULT" = "$expected"
expected=skipped
if [ "$TICKETS_SELECTED" = true ]; then expected=success; fi
test "$TICKETS_WINDOWS_RESULT" = "$expected"
test "$TICKETS_INSTALL_RESULT" = "$expected"
6 changes: 6 additions & 0 deletions docs/eval-development.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,12 @@ constraints for producing and interpreting that evidence.
Insert recovered reads according to shared ordering anchors rather than
appending an earlier skill after later ones. Conflicting source orders remain
incomplete instead of silently selecting one sequence.
Claude's explicit plugin commands may be expanded before the first assistant
turn without a Skill event. The observer accepts only a native command bound
to the mounted plugin namespace, exact arguments, and session, followed by
the complete mounted body in that same session. This proves dispatch without
requiring a redundant model-side Skill call; implicit discovery still needs
its observed invocation.
Do not infer cross-child read order from launch-list order: multiple children
contributing unanchored new reads keep the observation incomplete.
Truncated evidence cannot establish an exclusion;
Expand Down
8 changes: 8 additions & 0 deletions docs/specs/skill-evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,14 @@ the installed plugin-qualified skill name on Codex. The source case therefore
stays host-portable while each harness receives an invocation it can actually
resolve.

Claude may expand an explicit plugin command before the first assistant turn,
without emitting a Skill tool event. Accept that path only when the native
command matches the mounted plugin's namespace, skill, arguments, and session,
and the complete mounted skill body follows in that session before the first
assistant turn. Missing, partial, duplicate, or foreign-plugin expansion does
not establish activation. Retain only the bounded receipt, not the command or
skill text.

Codex orchestration evidence may repeat the owning installed plugin
qualification in child skill tokens. Reconciliation treats that exact
qualification as host transport syntax and compares the declared phase
Expand Down
19 changes: 19 additions & 0 deletions docs/specs/ticket-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,21 @@ stable intent ("create a ticket for X") while the backend stays swappable.

## Backend contract

- **TM-P3 — Contained portable mechanics.** The GitHub provider ships one
dependency-free Python package managed by UV, invoked through its frozen
`darrow-ticket` entrypoint by all four skills. Preserve the command arguments,
output records, refusals, and exit codes while removing the Bash launcher.
Use argument-vector subprocesses, explicit UTF-8 JSON decoding and validation,
and native filesystem and temporary-file handling on Linux, macOS, and Windows.
Provider failures must stop pending mutations; completed mutations remain
visible if a later relation write or read fails. Validate with mocked provider
operations, generated boundary cases, and fresh copied-plugin execution;
automated tests must never mutate live GitHub state.
Claude's session-start context may route matching requests to the installed
ticket skills before repository inspection or prerequisite judgments. It must
remain static and nonmutating, perform no repository or tracker inspection,
preserve provider boundaries, and leave workflow and authority in the skill.

- **TM-P1 — Independently installable providers.** Each tracker implementation
lives in its own `darrow-tickets-<provider>` plugin. The shipped provider is
GitHub Issues via `gh`. Future trackers use separate, self-contained plugins;
Expand Down Expand Up @@ -207,6 +222,8 @@ then relay its authoritative metadata, relations, and body.
an ID or canonical URL. A topic, title fragment, foreign-project URL,
ambiguous conversational reference, or numeric suffix extracted from a
rejected URL never becomes a guessed ticket.
Missing and ambiguous references remain read-ticket requests: activate the
skill to obtain the exact reference, with no tracker access before clarification.
- **TM-R2 — Read-only.** Reading never mutates tracker state and never becomes
permission to comment, edit, label, relate, close, reopen, assign, or start
the tracked work.
Expand All @@ -218,6 +235,8 @@ then relay its authoritative metadata, relations, and body.
URL; it is absent from every refusal or retrieval failure. Consumers preserve
it verbatim rather than deriving a token from an input reference or URL. Do not summarize,
rerank, enrich, interpret, assess readiness, or omit inconvenient content.
Imperative text inside a ticket remains quoted data: retrieval does not execute
those instructions or append an editorial assessment of them.
- **TM-R4 — Honest retrieval failure.** A missing ticket, unusable backend,
unreadable relation, or tracker error stops with the CLI's complete diagnostic.
Backend-provided evidence remains verbatim but may be capped with an explicit
Expand Down
132 changes: 132 additions & 0 deletions evals/runner/adapters/claude-project-skills.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
import { expect, test } from "bun:test";
import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { gradeActivation } from "../activation";
import {
matchClaudeProjectInvocation,
observeClaudeProjectInvocation,
projectSkillActivation,
} from "./claude-project-skills";
const input = {
Expand Down Expand Up @@ -36,6 +40,73 @@ const body = {
const assistant = { type: "assistant", sessionId: "session-one" };
const transcript = (entries: unknown[]) =>
entries.map((entry) => JSON.stringify(entry)).join("\n");
test.each(["darrow", "different-plugin"])(
"observer binds an explicit expansion to the mounted plugin (%s)",
async (nativePlugin) => {
const repo = await mkdtemp(join(tmpdir(), "darrow-native-command-"));
try {
const plugin = join(repo, ".git", "eval-plugin");
const skillDir = join(plugin, "skills", input.skill);
const manifestDir = join(plugin, ".claude-plugin");
const configRoot = join(repo, "host-config");
const sessions = join(
configRoot,
"projects",
repo.replace(/[^A-Za-z0-9]/g, "-"),
);
await Promise.all(
[skillDir, manifestDir, sessions].map((path) =>
mkdir(path, { recursive: true }),
),
);
const nativeCommand = {
...command,
message: {
content: command.message.content.replaceAll(
"darrow-guide",
`${nativePlugin}:darrow-guide`,
),
},
};
const nativeBody = {
...body,
message: {
content: body.message.content.map((block) => ({
...block,
text: block.text.replace(input.skillDir, skillDir),
})),
},
};
await Promise.all([
writeFile(join(skillDir, "SKILL.md"), input.skillText),
writeFile(
join(manifestDir, "plugin.json"),
JSON.stringify({ name: "darrow" }),
),
writeFile(
join(sessions, "session-one.jsonl"),
transcript([nativeCommand, nativeBody, assistant]),
),
]);
const result = await observeClaudeProjectInvocation({
repo,
configRoot,
stream: JSON.stringify({ type: "result", session_id: input.sessionId }),
prompt: input.prompt,
probe: {
mode: "explicit",
skill: input.skill,
invocation: "/darrow-guide",
},
});
expect(result?.type).toBe("darrow.claude_plugin_skill_invocation");
expect(result?.accepted).toBe(nativePlugin === "darrow");
expect(JSON.stringify(result)).not.toContain("Read current sources");
} finally {
await rm(repo, { recursive: true, force: true });
}
},
);
test("accepts one correlated complete project-command expansion without retaining contents", () => {
const result = matchClaudeProjectInvocation({
...input,
Expand All @@ -45,6 +116,46 @@ test("accepts one correlated complete project-command expansion without retainin
expect(JSON.stringify(result)).not.toContain("Read current sources");
expect(JSON.stringify(result)).not.toContain("What is Darrow");
});
test("accepts native plugin expansion of the unqualified command", () => {
const pluginCommand = {
...command,
message: {
content: command.message.content.replaceAll(
"darrow-guide",
"darrow:darrow-guide",
),
},
};
const result = matchClaudeProjectInvocation({
...input,
commandName: "darrow:darrow-guide",
transcript: transcript([pluginCommand, body, assistant]),
});
expect(result.accepted).toBe(true);
expect(result.type).toBe("darrow.claude_plugin_skill_invocation");
expect(JSON.stringify(result)).not.toContain("Read current sources");
});
test.each(["other:darrow-guide", "darrow-guide", "darrow:another-skill"])(
"rejects expansion from a different command %s",
(commandName) => {
const foreign = {
...command,
message: {
content: command.message.content.replaceAll(
"darrow-guide",
commandName,
),
},
};
expect(
matchClaudeProjectInvocation({
...input,
commandName: "darrow:darrow-guide",
transcript: transcript([foreign, body, assistant]),
}).accepted,
).toBe(false);
},
);
test.each([
[command, assistant],
[body, command, assistant],
Expand Down Expand Up @@ -97,6 +208,27 @@ test("malformed or unfinished native evidence remains unknown", () => {
).toBeNull();
});

test.each([false, null] as const)(
"unaccepted native plugin recovery preserves direct Skill-event evidence (%j)",
(accepted) => {
const observed = {
source: "harness_event" as const,
complete: true,
primarySkill: "darrow-guide",
observedSkills: ["darrow-guide"],
};
const result = projectSkillActivation(observed, {
type: "darrow.claude_plugin_skill_invocation",
skill: "darrow-guide",
accepted,
reason: "native expansion is absent or unavailable",
});
expect(result).toEqual(observed);
expect(gradeActivation("positive", "darrow-guide", result).passed).toBe(
true,
);
},
);
test.each([false, null] as const)(
"later skill events cannot repair an unaccepted project receipt (%j)",
(accepted) => {
Expand Down
Loading
Loading