Skip to content

--goal runs stop at the feasibility gate, and the cause is the size of the allowlist #25

Description

@Vallhalen

What

find-and-kick as a file passes its own gate. The same task as a --goal does not.

form verdict, 6 runs outcome
quackd run find-and-kick feasible 6 success 6
quackd run --goal "Find the ball and kick it." uncertain 5, feasible 1 aborted 5

Same robot, seed, model and words. --goal is the first thing in run --help.

Which difference does it

duck_from_goal differs from that duck file four ways at once, so I rebuilt the --goal
contract as a .duck file and changed one thing at a time. The reconstruction is exact: its
system prompt diffs to zero lines against a real --goal run, and it reproduces the same
5/1 split.

variant verdicts, 6 runs
0, reconstruction of the --goal contract uncertain 5, feasible 1
1, only the persona changed to "Determined and cheerful" uncertain 5, feasible 1
2, only the success criterion made concrete uncertain 5, feasible 1
3, only the allowlist narrowed to six, duck's verb names feasible 5, uncertain 1
6, only the allowlist narrowed to six, goal's own verb names feasible 6
4, only the five step strategy added feasible 3, uncertain 3
5, the shipped find-and-kick file feasible 6

The allowlist decides it. Persona and success criterion do nothing. Variant 6 exists
because variant 3 renamed two verbs as a side effect, and it rules naming out: the same six
verbs under the goal's own names give 6 of 6.

Why, from the pilot's own reasons

With fifteen verbs the pilot answers about the world:

"Since the camera currently detects nothing, I cannot determine feasibility yet."

"I cannot assess whether this body can do the task without first observing the position
and distance of the ball."

With six it answers about the body and the list:

"The robot can rotate to scan for the ball using the search_scan verb, move toward it
using go_to, and kick it using the kick verb. No payload, reach, or height requirements
exceed the robot's limits."

A short allowlist hands the pilot a route it can check against a datasheet. A long one
leaves the route open, so it looks at the world first and files "I have not looked yet" as
uncertain, which then asks a human and, with nobody there, aborts the run.

observe is in BEFORE_VERDICT precisely so a pilot can look before judging. On the long
list it says it cannot judge without observing instead of calling observe.

Suggestion, not a patch

I have not sent code for this because the fix is a judgement call, not a line. Candidates:
say in the gate's own prompt paragraph that an unexamined world is a reason to call
observe rather than to answer uncertain; or narrow what --goal allows by default. The
measurement says the second would work and the first is untested.

What this does not say

One model (Qwen3-32B-AWQ on vLLM), microduck:sim2d, one seed, six runs a cell, thinking
off. 48 repeated runs elsewhere in this measurement put the noise floor near plus or minus 2
per six run cell, and the swings here are 5.

Measurement and data from the same session as #24.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions