Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/darrow-guide/evals/fixtures/setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ for category in "$source_root"/plugins/*; do
mkdir -p "plugins/$(basename "$category")"
if [ -f "$category/README.md" ]; then cp "$category/README.md" "plugins/$(basename "$category")/README.md"; fi
for plugin in "$category"/*; do
[ -d "$plugin" ] || continue
[ -d "$plugin" ] && [ -f "$plugin/README.md" ] || continue
destination="plugins/$(basename "$category")/$(basename "$plugin")"
mkdir -p "$destination/.claude-plugin" "$destination/.codex-plugin"
cp "$plugin/README.md" "$destination/README.md"
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/python-quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ jobs:
- uses: astral-sh/setup-uv@v6
with:
enable-cache: true
- run: scripts/check-python --package plugins/foundation/darrow-skill-authoring/skills/author-agent-skill/backend
- run: scripts/check-python --package plugins/foundation/darrow-skill-authoring/backend

discovery-windows:
name: Discovery Python ${{ matrix.python-version }} on windows-latest
Expand All @@ -119,7 +119,7 @@ jobs:
- uses: astral-sh/setup-uv@v6
with:
enable-cache: true
- run: scripts/check-python --package plugins/capability/darrow-discovery/skills/plan-implementation/backend
- run: scripts/check-python --package plugins/capability/darrow-discovery/skills/work-through-decisions/backend

verification-windows:
name: Verification Python ${{ matrix.python-version }} on windows-latest
Expand Down Expand Up @@ -485,7 +485,7 @@ jobs:
with:
python-version: "3.13"
- uses: astral-sh/setup-uv@v6
- run: uv run --quiet --frozen --no-dev --project plugins/foundation/darrow-skill-authoring/skills/author-agent-skill/backend python plugins/foundation/darrow-skill-authoring/skills/author-agent-skill/backend/tests/fresh_install.py
- run: uv run --quiet --frozen --no-dev --project plugins/foundation/darrow-skill-authoring/backend python plugins/foundation/darrow-skill-authoring/backend/tests/fresh_install.py

observability-windows:
name: Observability Langfuse Python ${{ matrix.python-version }} on windows-latest
Expand Down Expand Up @@ -543,7 +543,7 @@ jobs:
with:
python-version: "3.13"
- uses: astral-sh/setup-uv@v6
- run: uv run --quiet --frozen --no-dev --project plugins/capability/darrow-discovery/skills/plan-implementation/backend python plugins/capability/darrow-discovery/skills/plan-implementation/backend/tests/fresh_install.py
- run: uv run --quiet --frozen --no-dev --project plugins/capability/darrow-discovery/skills/work-through-decisions/backend python plugins/capability/darrow-discovery/skills/work-through-decisions/backend/tests/fresh_install.py

verification-fresh-install:
name: Fresh install verification on ${{ matrix.os }}
Expand Down
8 changes: 6 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,12 @@ between capabilities and orchestration, read [`docs/design.md`](docs/design.md).

## Skill development

For skill creation, revision, or validation, follow
[`author-agent-skill`](plugins/foundation/darrow-skill-authoring/skills/author-agent-skill/SKILL.md).
For new skill creation, follow
[`create-agent-skill`](plugins/foundation/darrow-skill-authoring/skills/create-agent-skill/SKILL.md).
For a read-only skill audit, follow
[`audit-agent-skill`](plugins/foundation/darrow-skill-authoring/skills/audit-agent-skill/SKILL.md).
Implement audit findings or other changes to an existing skill as ordinary
engineering work against the findings and this repository's instructions.
Add or adjust the applicable invariant under `docs/specs/` before implementation.
Keep contextual judgment in the skill. Put repeatable, error-prone command and
tool-protocol mechanics behind narrow bundled scripts so the model supplies
Expand Down
7 changes: 5 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,11 @@ plugin content. In particular:

- Add or adjust the applicable invariant under `docs/specs/` before
implementation.
- For skill creation or revision, follow
[`author-agent-skill`](plugins/foundation/darrow-skill-authoring/skills/author-agent-skill/SKILL.md).
- For skill creation, follow
[`create-agent-skill`](plugins/foundation/darrow-skill-authoring/skills/create-agent-skill/SKILL.md).
For a read-only skill audit, follow
[`audit-agent-skill`](plugins/foundation/darrow-skill-authoring/skills/audit-agent-skill/SKILL.md).
Implement audit findings under the repository's ordinary engineering rules.
- Keep each plugin self-contained. Never reference files outside a plugin or
assume a sibling plugin is installed.
- Use imperative Conventional Commits without trailing periods, and no AI
Expand Down
17 changes: 13 additions & 4 deletions docs/eval-development.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@ constraints for producing and interpreting that evidence.
- Prefer observable repository state, external effects, and user-visible
outcomes over prose. When free-form text is the only public seam, assert the
smallest semantic decision rather than parsing a complete explanation.
- For a free-form saved document, use `semantic_artifact` to grade the artifact
itself. Its repository-relative `path` may contain one `*` in the filename and
must resolve to exactly one regular file. Keep deterministic checks for file
presence, status, and structural validation; `semantic_output_checks` grade
only the final response.
- Give one case one concrete state and one decision. When success would require
recognizing several conditional branches in one free-form answer, split the
branches into separate concrete-state cases.
Expand Down Expand Up @@ -67,6 +72,11 @@ constraints for producing and interpreting that evidence.
- A Codex case without the placeholder remains an implicit-discovery probe and
requires a completed mounted-skill body read. Do not use the explicit path to
make implicit selection pass.
A complete read of the byte-identical marketplace staging copy of an
installed plugin skill also counts: the staging path must come from that
eval's installed marketplace catalog, and the observed output must match
the installed body. Prepared target skill drafts elsewhere in the fixture
never count as activation.
- Missing, repeated, malformed, or unverified observation evidence stays
unknown. A failed compound shell command may still prove an earlier skill
read only when the command names a mounted skill path and its output contains
Expand Down Expand Up @@ -237,9 +247,8 @@ evaluation that stops at the first failure. `--only <question-id>` and
before continuing. Fixtures snapshot current public documentation, manifests,
and relevant code while excluding inventory, hidden checks, and delivery
conclusions from the participant repository.
The guide driver pins semantic grading to Codex `gpt-5.6-terra` / medium;
the shared runner's default lightweight grader remains unchanged. The route
was calibrated against retained correct and reversed-delegation diagrams.
The guide driver pins semantic grading to Codex `gpt-6-luna` / medium;
the shared runner's default lightweight grader remains unchanged.

## Live-run controls

Expand Down Expand Up @@ -287,7 +296,7 @@ preserving the selected cases.

Codex runs use independent defaults for each eval role:

- candidate: `gpt-5.6-terra` at `medium` effort;
- candidate: `gpt-6-luna` at `medium` effort;
- advisory quality judge: `gpt-5.6-sol` at `low` effort;
- gating semantic-output grader: `gpt-5.6-luna` at `low` effort.

Expand Down
Loading
Loading