Skip to content

Add stalled-work and evidence-based debugging skills #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
csark0812 merged 1 commit into from
Oct 5, 2026
Merged

Add stalled-work and evidence-based debugging skills #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 29 additions & 26 deletions .skeleton/registry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,39 +1,42 @@
# Registry

<!-- doc-meta: owner=eng | last-reviewed=2026-09-13 -->
<!-- doc-meta: owner=eng | last-reviewed=2026-10-05 -->

<!-- source-of-truth: Registry topic routing for documentation and skills -->

## Documentation

| Topic | Canonical file |
| ---------------------------- | ------------------------------------------------------------------------------ |
| Package overview | [README.md](../README.md) |
| Agent cold-start | [AGENTS.md](../AGENTS.md) |
| Skill tiers | [tiers.md](../docs/tiers.md) |
| Composability vocabulary | [process-skill-composition.md](../references/process-skill-composition.md) |
| GitHub ambient refs validate | [github-ambient-refs-validation.md](../docs/github-ambient-refs-validation.md) |
| Hunch verdict (ambient) | [verdict.md](../references/verdict.md) |
| Evidence parity | [evidence-parity.md](../docs/evidence-parity.md) |
| Skill evolution | [skill-evolution.md](../docs/skill-evolution.md) |
| Skill organization ablations | [skill-organization-ablations.md](../docs/skill-organization-ablations.md) |
| Topic | Canonical file |
| ------------------------------------ | ------------------------------------------------------------------------------------- |
| Package overview | [README.md](../README.md) |
| Agent cold-start | [AGENTS.md](../AGENTS.md) |
| Skill tiers | [tiers.md](../docs/tiers.md) |
| Composability vocabulary | [process-skill-composition.md](../references/process-skill-composition.md) |
| GitHub ambient refs validate | [github-ambient-refs-validation.md](../docs/github-ambient-refs-validation.md) |
| Hunch verdict (ambient) | [verdict.md](../references/verdict.md) |
| Evidence parity | [evidence-parity.md](../docs/evidence-parity.md) |
| Skill evolution | [skill-evolution.md](../docs/skill-evolution.md) |
| Skill organization ablations | [skill-organization-ablations.md](../docs/skill-organization-ablations.md) |
| Nontechnical investigation follow-up | [evidence-based-investigation.md](../docs/follow-ups/evidence-based-investigation.md) |

## Skills

| Slug | Canonical file |
| ------------------ | ------------------------------------------------------------- |
| council | [council/SKILL.md](../council/SKILL.md) |
| code-review | [code-review/SKILL.md](../code-review/SKILL.md) |
| review-walkthrough | [review-walkthrough/SKILL.md](../review-walkthrough/SKILL.md) |
| grill | [grill/SKILL.md](../grill/SKILL.md) |
| second-opinion | [second-opinion/SKILL.md](../second-opinion/SKILL.md) |
| probe | [probe/SKILL.md](../probe/SKILL.md) |
| tdd | [tdd/SKILL.md](../tdd/SKILL.md) |
| prototype | [prototype/SKILL.md](../prototype/SKILL.md) |
| domain-model | [domain-model/SKILL.md](../domain-model/SKILL.md) |
| handoff | [handoff/SKILL.md](../handoff/SKILL.md) |
| refactor-companion | [refactor-companion/SKILL.md](../refactor-companion/SKILL.md) |
| branch-status | [branch-status/SKILL.md](../branch-status/SKILL.md) |
| Slug | Canonical file |
| ------------------------ | ------------------------------------------------------------------------- |
| council | [council/SKILL.md](../council/SKILL.md) |
| code-review | [code-review/SKILL.md](../code-review/SKILL.md) |
| review-walkthrough | [review-walkthrough/SKILL.md](../review-walkthrough/SKILL.md) |
| grill | [grill/SKILL.md](../grill/SKILL.md) |
| second-opinion | [second-opinion/SKILL.md](../second-opinion/SKILL.md) |
| evidence-based-debugging | [evidence-based-debugging/SKILL.md](../evidence-based-debugging/SKILL.md) |
| probe | [probe/SKILL.md](../probe/SKILL.md) |
| tdd | [tdd/SKILL.md](../tdd/SKILL.md) |
| prototype | [prototype/SKILL.md](../prototype/SKILL.md) |
| domain-model | [domain-model/SKILL.md](../domain-model/SKILL.md) |
| handoff | [handoff/SKILL.md](../handoff/SKILL.md) |
| refactor-companion | [refactor-companion/SKILL.md](../refactor-companion/SKILL.md) |
| branch-status | [branch-status/SKILL.md](../branch-status/SKILL.md) |
| i-need-help | [i-need-help/SKILL.md](../i-need-help/SKILL.md) |

## Workflow additions

Expand Down
32 changes: 17 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# toolbox

<!-- source-of-truth: user-level process and orchestrator Cursor/Claude agent skills. -->
<!-- doc-meta: owner=eng | last-reviewed=2026-10-01 -->
<!-- doc-meta: owner=eng | last-reviewed=2026-10-05 -->

Public process SSOT. Engineers install skills globally (`-g`). Product repos keep product workflows and shared standards — they should not vendor these process skill folders.

Expand Down Expand Up @@ -78,20 +78,22 @@ Existing committed toolbox process skill dirs keep loading until removed. Delete

## Skills

| Group | Slug | Purpose |
| ------------ | ------------------ | ----------------------------------------------------------------------------------------------- |
| Orchestrator | council | Multi-agent depth — create task personas, choose an interaction, run members, synthesize |
| Orchestrator | handoff | A2A cross-session — channel + pack + goal; pointers not bodies |
| Process | code-review | Choose a task-shaped review mode and file only evidence-backed findings |
| Process | review-walkthrough | Explain a bounded change through the smallest useful causal story |
| Process | second-opinion | Adversarial critique of a written artifact — invent lenses; layer council for multi-agent depth |
| Process | grill | Decision-focused dialogue for unclear intent and consequential design choices |
| Process | probe | Hunch verdict or hard-bug fix under Authority B |
| Process | tdd | Test-first build at agreed public seams |
| Process | prototype | Throwaway artifact for one design question |
| Process | domain-model | Persist glossary + ADRs when decisions are ready |
| Process | refactor-companion | Preserve a target design through evidence-led, proven refactor slices |
| Process | branch-status | Map unique-ahead branches, stacks, and stale work |
| Group | Slug | Purpose |
| ------------ | ------------------------ | ----------------------------------------------------------------------------------------------- |
| Orchestrator | council | Multi-agent depth — create task personas, choose an interaction, run members, synthesize |
| Orchestrator | handoff | A2A cross-session — channel + pack + goal; pointers not bodies |
| Process | code-review | Choose a task-shaped review mode and file only evidence-backed findings |
| Process | review-walkthrough | Explain a bounded change through the smallest useful causal story |
| Process | second-opinion | Adversarial critique of a written artifact — invent lenses; layer council for multi-agent depth |
| Process | grill | Decision-focused dialogue for unclear intent and consequential design choices |
| Process | evidence-based-debugging | Reproduce and diagnose failures; demonstrate and verify authorized repairs |
| Process | probe | Cited verdict on a specific hunch or claim |
| Process | tdd | Test-first build at agreed public seams |
| Process | prototype | Throwaway artifact for one design question |
| Process | domain-model | Persist glossary + ADRs when decisions are ready |
| Process | refactor-companion | Preserve a target design through evidence-led, proven refactor slices |
| Process | branch-status | Map unique-ahead branches, stacks, and stale work |
| Process | i-need-help | Stop when repeated work makes no progress; ask the current owner for specific help |

Orchestrators define **agent-to-agent** wiring; process skills describe **what happens**. Layered prompts compose them without peer runtime dependencies. See [docs/tiers.md](docs/tiers.md).

Expand Down
4 changes: 3 additions & 1 deletion agent-suites/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# Toolbox agent suites

<!-- source-of-truth: current SDK execution, claim mapping and evaluation boundaries -->
<!-- doc-meta: owner=eng | last-reviewed=2026-09-30 -->
<!-- doc-meta: owner=eng | last-reviewed=2026-10-05 -->

Current tests are TypeScript SDK suites under v2/. claims.json files preserve original declarative claims and fixture-reader compatibility; they are not executed by a legacy SDK runtime. migration-index.json maps every historical scenario to its current assertion, selected judge inputs and normalized output.

`i-need-help` is available through normal global skill installation and automatic routing. Skill invocation is a portable agent instruction, not an enforced host watchdog; install and live behavior still depend on each agent honoring its loaded skill.

Run npm run agent:test for offline discovery/input validation. Live runs use the installed Codex subscription without billing fallback. npm run agent:test:comparisons repeats representative paired tests three times; aggregate normalized evidence with scripts/lib/evidence-v2.mjs. Selected output, tool calls and source excerpts go to a fresh judge; missing evidence fails rather than becoming a pass. Global skills are disabled. Infrastructure and judge failures remain distinct from task failures.

The two previously skipped remote-fetch scenarios remain recorded as skipped. Native goal recovery, host-isolation guarantees, publication receipts and PostPrint live persistence require separate live qualification. SDK history reconstruction does not prove native continuation.
21 changes: 21 additions & 0 deletions agent-suites/evidence-based-debugging-outcomes-ceiling/claims.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "evidence-based-debugging-outcomes-ceiling",
"description": "Read-only diagnosis of an expiry boundary failure with an executed failing check.",
"defaults": {
"profile": "cursor",
"skills": "full"
},
"scenarios": [
{
"name": "diagnosis: expiry boundary",
"compareId": "read-only-expiry-diagnosis",
"prompt": "Diagnose sessions remaining valid at exact expiry in agent-suites/fixtures/debug-app. Do not edit production code; stay read-only. Run the failing check and explain its mechanism with citations.\n\nRead `.claude/skills/evidence-based-debugging/SKILL.md` first.",
"rubric": {
"mustInvokeSkill": ["evidence-based-debugging"],
"judge": [
"Tool calls show a failing session expiry check; the agent demonstrates and cites the boundary mechanism without shipping a production repair."
]
}
}
]
}
21 changes: 21 additions & 0 deletions agent-suites/evidence-based-debugging-outcomes-ceiling/scenarios.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "evidence-based-debugging-outcomes-ceiling",
"description": "Read-only diagnosis of an expiry boundary failure with an executed failing check.",
"defaults": {
"profile": "cursor",
"skills": "full"
},
"scenarios": [
{
"name": "diagnosis: expiry boundary",
"compareId": "read-only-expiry-diagnosis",
"prompt": "Diagnose sessions remaining valid at exact expiry in agent-suites/fixtures/debug-app. Do not edit production code; stay read-only. Run the failing check and explain its mechanism with citations.\n\nRead `.claude/skills/evidence-based-debugging/SKILL.md` first.",
"rubric": {
"mustInvokeSkill": ["evidence-based-debugging"],
"judge": [
"Tool calls show a failing session expiry check; the agent demonstrates and cites the boundary mechanism without shipping a production repair."
]
}
}
]
}
33 changes: 33 additions & 0 deletions agent-suites/evidence-based-debugging-outcomes/claims.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"name": "evidence-based-debugging-outcomes",
"description": "Current debugging contract: paired insufficient-evidence and demonstrated-repair tasks. Historical probe-fix runs evaluate a different contract.",
"defaults": {
"profile": "cursor",
"skills": "full"
},
"scenarios": [
{
"name": "outcomes: case 1",
"compareId": "insufficient-evidence",
"prompt": "Users report a rare native login crash. This repository contains only web session helpers, not the native app. There is no device access, crash log, precise trigger or captured failure. Investigate what can be established and what observation is needed next; preserve production files.\n\nRead `.claude/skills/evidence-based-debugging/SKILL.md` first.",
"rubric": {
"judge": [
"The agent distinguishes the missing native failure evidence from unrelated web session helper defects, does not invent a diagnosis or repair production code, and names a specific next artifact or observation. Clearly labeled hypotheses are allowed."
],
"mustInvokeSkill": ["evidence-based-debugging"]
}
},
{
"name": "outcomes: case 2",
"compareId": "demonstrated-repair",
"prompt": "Users see sessions valid at exact expiry in agent-suites/fixtures/debug-app. Find and fix that failure. Leave unrelated cookie behavior unchanged. Run the relevant test, demonstrate the mechanism, then verify the repair.\n\nRead `.claude/skills/evidence-based-debugging/SKILL.md` first.",
"rubric": {
"judge": [
"Tool calls establish an actually executed failing session validity check before the production edit, a demonstrated expiry-boundary mechanism, and passing relevant checks after the repair. The source rejects equal expiry and accepts future expiry; unrelated cookie behavior is unchanged."
],
"mustInvokeSkill": ["evidence-based-debugging"]
},
"contract": "expiry-repair"
}
]
}
33 changes: 33 additions & 0 deletions agent-suites/evidence-based-debugging-outcomes/scenarios.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"name": "evidence-based-debugging-outcomes",
"description": "Current debugging contract: paired insufficient-evidence and demonstrated-repair tasks. Historical probe-fix runs evaluate a different contract.",
"defaults": {
"profile": "cursor",
"skills": "full"
},
"scenarios": [
{
"name": "outcomes: case 1",
"compareId": "insufficient-evidence",
"prompt": "Users report a rare native login crash. This repository contains only web session helpers, not the native app. There is no device access, crash log, precise trigger or captured failure. Investigate what can be established and what observation is needed next; preserve production files.\n\nRead `.claude/skills/evidence-based-debugging/SKILL.md` first.",
"rubric": {
"judge": [
"The agent distinguishes the missing native failure evidence from unrelated web session helper defects, does not invent a diagnosis or repair production code, and names a specific next artifact or observation. Clearly labeled hypotheses are allowed."
],
"mustInvokeSkill": ["evidence-based-debugging"]
}
},
{
"name": "outcomes: case 2",
"compareId": "demonstrated-repair",
"prompt": "Users see sessions valid at exact expiry in agent-suites/fixtures/debug-app. Find and fix that failure. Leave unrelated cookie behavior unchanged. Run the relevant test, demonstrate the mechanism, then verify the repair.\n\nRead `.claude/skills/evidence-based-debugging/SKILL.md` first.",
"rubric": {
"judge": [
"Tool calls establish an actually executed failing session validity check before the production edit, a demonstrated expiry-boundary mechanism, and passing relevant checks after the repair. The source rejects equal expiry and accepts future expiry; unrelated cookie behavior is unchanged."
],
"mustInvokeSkill": ["evidence-based-debugging"]
},
"contract": "expiry-repair"
}
]
}
Loading
Loading