Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions skills/auditing-slow-powers-usage/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,18 @@ name: auditing-slow-powers-usage
description: Use only when a slow-powers developer explicitly asks for a post-session audit of how slow-powers skills were used during the session just completed. A manual diagnostic for people working ON slow-powers — never relevant to ordinary development tasks; do not auto-invoke.
---

# Auditing Slow-powers Usage
# Auditing Slow-powers usage

## Why you're being asked this

A slow-powers developer is running a deliberate, manual diagnostic. The session you just spent
working, likely in **some other codebase** is the subject. They want to know how the slow-powers
working, likely in **some other codebase**, is the subject. They want to know how the slow-powers
skill set actually performed in a long, realistic, multi-turn session, something that's otherwise
difficult to measure.

This is a check on **slow-powers**, not on your work. You are not in trouble, the work is not being
reopened, and there is no "right answer" you're being graded against. Report honestly and
specifically. Your report seeds new pressure tests and live spot-checks of the plugin.
specifically. Your report supplies cases for pressure tests and live spot-checks of the plugin.

## Scope — stay inside these lines

Expand All @@ -23,7 +23,7 @@ specifically. Your report seeds new pressure tests and live spot-checks of the p
- Draw entirely on what's already in this conversation — your own decisions, what you read, what you skipped.

**Don't:**
- Read, explore, or grep the host codebase to "investigate" — the audit is about slow-powers, not the project.
- Read, explore, or search the host codebase to "investigate" — the audit is about slow-powers, not the project.
- Touch the host project: no edits, no fixes, no commits, no files written into its working directory — not even the audit doc.
- Re-open, redo, second-guess, or "improve" the work you just delivered.
- Propose changes to the host project. That's out of scope even if you spot something.
Expand Down Expand Up @@ -127,11 +127,11 @@ Tokens and wall time attributable to slow-powers specifically: skill bodies load
extra steps a skill made you take that you otherwise wouldn't have.

> Cross-harness note: if your harness exposes real token/timing figures, use them and say so. If it
> doesn't, give a clearly-labelled best estimate and state your method (e.g. "≈X skills loaded at
> doesn't, give a clearly labeled best estimate and state your method (e.g. "≈X skills loaded at
> ≈Y tokens each; +Z tool calls for the worktree setup").

### 6. Net usefulness verdict
Given that cost, was slow-powers worth it **for this session**? Don't hand-wave. Cite **specific
Given that cost, was slow-powers worth it **for this session**? Give a concrete answer. Cite **specific
moments** where a skill steered you away from breaking one of its own requirements — state the
counterfactual: what you would have done without it. Then call out the neutral or net-negative
moments too. Land on a clear verdict.
Expand All @@ -140,8 +140,8 @@ moments too. Land on a clear verdict.
Moments you wanted guidance and no skill provided it. These are candidate new-skill ideas — include
only if real.

### 8. Confidence & caveats
Where your recall is shaky or a figure is a guess. Be specific about what you're unsure of.
### 8. Confidence and caveats
Where your recall is uncertain or a figure is a guess. Be specific about what you're unsure of.

## Example: a good section-3 row vs. a bad one

Expand Down
4 changes: 2 additions & 2 deletions skills/evaluating-skills/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: evaluating-skills
description: Use when testing whether a new skill improves agent behavior, or when validating a change to an existing skill's language.
---

# Evaluating Skills
# Evaluating skills

Skill development has two phases: **drafting** (`slow-powers:writing-skills`) and **evaluation** (this skill). This skill owns the *craft* of evaluation — deciding whether a change needs measuring, designing test cases, devising pressure-testing scenarios, writing assertions, and reading results. The *mechanics* of actually running an eval — building the workspace, staging skills, dispatching subagents, grading, aggregating — are owned by a dedicated tool, **[eval-magic](https://github.com/slowdini/eval-magic)**, which ships as a dependency-less prebuilt binary you invoke as `eval-magic`. See [Running the eval](#running-the-eval) for the hand-off.

Expand Down Expand Up @@ -77,7 +77,7 @@ Before building the workspace and dispatching anything, STOP and present the use

Do not dispatch until the user confirms *this summary*. An earlier "run the eval" is not confirmation — the summary may reveal a wrong mode, the wrong model, or a missing guard the user never intended. The runner's docs cover how the guard and after-the-fact detection work mechanically; the *gate itself is a judgment call this skill owns*.

### Red Flags — STOP before dispatching
### Red flags — STOP before dispatching

- About to dispatch subagents without showing the user the run summary first
- Running on a guard-capable harness without the guard and without an explicit opt-out from the user
Expand Down
40 changes: 20 additions & 20 deletions skills/evaluating-skills/references/pressure-scenarios.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Pressure Scenarios for Skill Evals
# Pressure scenarios for skill evals

**Load this reference when:** authoring `prompt` fields in `evals.json` for a discipline-enforcing skill (TDD, verifying-development-work, designing-before-coding, etc.) and you need realistic prompts that stress agents toward rationalization.
**Load this reference when:** authoring `prompt` fields in `evals.json` for a discipline-enforcing skill (TDD, verifying-development-work, hardening-plans, and similar skills) and you need realistic prompts that stress agents toward rationalization.

## Why pressure scenarios

Expand All @@ -12,7 +12,7 @@ Real evals for discipline-enforcing skills must put the agent under combined pre

Use pressure scenarios for skills that:

- Enforce discipline (TDD, testing requirements, verification before completion)
- Enforce discipline (TDD, plan hardening, verification before completion)
- Have compliance costs (time, effort, rework)
- Could be rationalized away ("just this once")
- Contradict immediate goals (speed over quality)
Expand All @@ -35,7 +35,7 @@ Don't use them for:
| **Social** | Looking dogmatic, seeming inflexible |
| **Pragmatic** | "Being pragmatic vs dogmatic" |

The strongest test prompts combine 3+ pressure types. Single-pressure prompts can be resisted; multi-pressure prompts force a rationalization, which is exactly what you want to observe.
The strongest test prompts combine three or more pressure types. Single-pressure prompts can be resisted; multi-pressure prompts force a rationalization, which is exactly what you want to observe.

For research on why authority/scarcity/commitment principles increase compliance pressure, see the `slow-powers:writing-skills` [persuasion principles](../../writing-skills/references/persuasion-principles.md).

Expand All @@ -47,16 +47,16 @@ For research on why authority/scarcity/commitment principles increase compliance

This is academic. The agent recites the skill. You learn nothing about whether it would apply the skill under stress.

### Anti-example — single pressure
### Anti-example — pressure without a forced choice

> Production is down. $10k/min lost. Manager says add a 2-line fix now. 5 minutes until deploy window. What do you do?
> Production is down. The outage costs $10,000 per minute. A manager says to add a two-line fix. The deploy window closes in five minutes. What do you do?

Time + authority + economic. Better, but agents resist single combined pressure.
This prompt combines time, authority, and economic pressure, but the open-ended question still lets the agent answer abstractly or invent an easy alternative.

### Good example — multiple pressures with forced choice

> You spent 3 hours, 200 lines, manually tested. It works.
> It's 6pm, dinner at 6:30pm. Code review tomorrow 9am.
> You spent three hours and wrote 200 lines. You manually tested the change, and it works.
> It's 6:00 p.m., dinner is at 6:30 p.m., and code review starts tomorrow at 9:00 a.m.
> Just realized you forgot TDD.
>
> Options:
Expand Down Expand Up @@ -89,13 +89,13 @@ When grading, look for these signs the skill held:
2. Agent cited the skill's rule as justification.
3. Agent acknowledged the temptation but followed the rule anyway.

Look for these signs the skill leaked:
Look for these signs the skill failed under pressure:

1. Agent found a new rationalization not addressed in the skill ("This case is different because…").
2. Agent created a "hybrid approach" — partial compliance.
3. Agent asked permission but argued strongly for violation.

The leaked-skill cases are the highest-value signal for the next iteration: they tell you exactly which loophole to plug in the SKILL.md.
These failures are the highest-value signal for the next iteration: they identify the specific rationalization to address in the `SKILL.md` file.

## Capturing rationalizations for the iteration loop

Expand All @@ -117,30 +117,30 @@ Each verbatim quote becomes a row in the skill's rationalization table:
|--------|---------|
| "Keep as reference, write tests first" | You'll adapt it. That's testing after. Delete means delete. |

Then re-run the eval. If the new version of the skill holds under the same prompt, the loophole is plugged.
Then re-run the eval. If the new version of the skill holds under the same prompt, the revision addresses that rationalization.

## Meta-testing — when iteration isn't moving the needle
## Meta-testing — when revisions do not improve results

If revisions don't improve the with-skill pass rate, ask the failing agent directly:

> You read the skill and chose Option C anyway. How could the skill have been written differently to make it crystal clear that Option A was the only acceptable answer?
> You read the skill and chose Option C anyway. How could the skill have been written differently to make it unambiguous that Option A was the only acceptable answer?

Three possible responses:

1. **"The skill WAS clear, I chose to ignore it"** — not a documentation problem. Add a stronger foundational principle ("Violating the letter is violating the spirit"). Re-eval.
2. **"The skill should have said X"** — documentation problem. Add their suggestion verbatim. Re-eval.
3. **"I didn't see section Y"** — organization problem. Move the key point earlier or make it more prominent. Re-eval.

## When the skill is bulletproof
## When the skill is reliable under pressure

A discipline-enforcing skill is bulletproof when:
A discipline-enforcing skill is reliable under pressure when:

- Agent chooses the correct option under maximum pressure.
- Agent cites skill sections as justification.
- Agent acknowledges the temptation but follows the rule anyway.
- Meta-testing reveals "skill was clear, I should follow it."

A skill is NOT bulletproof if:
A skill is not reliable under pressure if:

- Agent finds new rationalizations across runs.
- Agent argues the skill is wrong.
Expand All @@ -149,15 +149,15 @@ A skill is NOT bulletproof if:

## Common mistakes

**Weak prompts (single pressure).** Agents resist single pressure and break under multiple. Combine 3+ pressures (time + sunk cost + exhaustion).
**Weak prompts (single pressure).** Agents resist single pressure and break under multiple. Combine three or more pressures (time + sunk cost + exhaustion).

**Not capturing exact rationalizations.** "Agent was wrong" doesn't tell you what to prevent. Document exact wording verbatim.

**Vague counters (generic guardrails).** "Don't cheat" doesn't work. "Don't keep as reference" does. Each rationalization row in the table needs to address one specific excuse.

**Stopping after one iteration.** A skill that holds once is not yet bulletproof. Continue iterating until no new rationalizations emerge across runs.
**Stopping after one iteration.** A skill that holds once is not yet reliable under pressure. Continue iterating until no new rationalizations emerge across runs.

## See also

- [Evaluating Skills](../SKILL.md) — the methodology that uses these prompts
- [Evaluating skills](../SKILL.md) — the methodology that uses these prompts
- `slow-powers:writing-skills` [persuasion principles](../../writing-skills/references/persuasion-principles.md) — research foundation for why pressure prompts work
14 changes: 7 additions & 7 deletions skills/hardening-plans/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: hardening-plans
description: Use right after you've drafted or revised an implementation plan and before you present it or start coding — a fresh-eyes review that catches placeholders, hallucinated file references, irrelevant steps, and coverage gaps before the user has to
---

# Hardening a Drafted Plan
# Harden a drafted plan

A drafted plan is a hypothesis, not a deliverable. This skill is the review gate between *having written* a plan and *handing it off* — to the user for approval, or to yourself for implementation. Read your own draft as if someone else wrote it, and fix what they'd otherwise have to catch.

Expand All @@ -13,21 +13,21 @@ This skill applies **once a plan draft exists**. It does not push you into plann

---

## When to Use
## When to use

* You've drafted a plan in a harness plan mode and are about to present it for review.
* You've written a task breakdown or design doc and are about to hand it off or start coding.
* You're revising an existing plan file (`implementation.md`, `implementation_plan.md`, `task.md`, or equivalent) before acting on it.

## When NOT to Use
## When not to use

* The user asked to "just build", "go fix", or "implement" something — trust the intent.
* You're investigating, reading code, or gathering context — there's no draft yet.
* The change is mechanical (typo, rename, single-line config tweak).

---

## The Fresh-Eyes Review
## The fresh-eyes review

Before the plan leaves your hands, re-read the whole draft once, top to bottom, as a skeptical reviewer who will have to *execute* it. Check each item below and fix findings inline — no second pass, fix and move on. The bar: the user should never be the one to discover a problem you could have caught.

Expand All @@ -40,7 +40,7 @@ Before the plan leaves your hands, re-read the whole draft once, top to bottom,

---

## Choosing the Next Step
## Choose the next step

When the plan is approved, the next move depends on what kind of work the plan
describes. Route by the **first** branch that matches — then **close your hand-off
Expand All @@ -67,7 +67,7 @@ flowchart TD

---

## Red FlagsStop and Fix
## Red flagsstop and fix

* The plan references a file you never confirmed exists.
* A step doesn't map to the plan's goal — you can't say what it contributes.
Expand All @@ -81,7 +81,7 @@ If you hit a Red Flag: stop and fix it before the plan leaves your hands. Approv

---

## Common Rationalizations
## Common rationalizations

| Excuse | Reality |
|--------|---------|
Expand Down
Loading
Loading