Encoding a required next skill in a plan generally proves very helpful - we have some intentional skill flows we want the agent to follow, and this is an easy point where the agent can get lost. However, our current instructions create a big of a trap. We explicitly provide two potential followup skills (working-in-isolation and working-with-tdd), which are far-and-away the most likely followup skills. However, there are cases where a plan has been written, and neither of those skills should be called next.
For instance, it's often useful to write a plan to break down a large project into more workable tickets. With slow-powers installed, the next skill should actually probably be writing-technical-docs. But the user could also have a wide range of skills installed, one of which might apply next (not to just list examples, but imagine a skill around making updates to a database - this could easily require a plan before executing, but benefit from the skill for implementation).
The trap is that, by listing two skills to choose from, those are the only skills that are considered for the followup. When both are rejected, the "followup skill" requirement is considered met, and alternative skills never even come to mind.
Proposal
Update hardening-plans's "required next skill" instructions, along with any general instructions we provide around skill next steps (i.e., in writing-skills), to be more robust, with an emphasis on the idea that a list of options should consider whether choices not on the list might apply (the case with choosing a next skill for hardening-plans). If so, include clear instructions about choosing an alternative, and deciding no choice applies in the case in question, to ensure that the agent has fully considered the question, and not just looked at the list of options.
Encoding a required next skill in a plan generally proves very helpful - we have some intentional skill flows we want the agent to follow, and this is an easy point where the agent can get lost. However, our current instructions create a big of a trap. We explicitly provide two potential followup skills (
working-in-isolationandworking-with-tdd), which are far-and-away the most likely followup skills. However, there are cases where a plan has been written, and neither of those skills should be called next.For instance, it's often useful to write a plan to break down a large project into more workable tickets. With slow-powers installed, the next skill should actually probably be
writing-technical-docs. But the user could also have a wide range of skills installed, one of which might apply next (not to just list examples, but imagine a skill around making updates to a database - this could easily require a plan before executing, but benefit from the skill for implementation).The trap is that, by listing two skills to choose from, those are the only skills that are considered for the followup. When both are rejected, the "followup skill" requirement is considered met, and alternative skills never even come to mind.
Proposal
Update
hardening-plans's "required next skill" instructions, along with any general instructions we provide around skill next steps (i.e., inwriting-skills), to be more robust, with an emphasis on the idea that a list of options should consider whether choices not on the list might apply (the case with choosing a next skill forhardening-plans). If so, include clear instructions about choosing an alternative, and deciding no choice applies in the case in question, to ensure that the agent has fully considered the question, and not just looked at the list of options.