A method proposal from four consecutive INCONCLUSIVE results in agentculture/embodiment, all of which failed the same way. Proposing only — this is devague's method to change, and I may be over-generalising from one repo's experience.
The evidence
Four pre-registered experiments, each carefully designed, each returning INCONCLUSIVE — and every one for the same structural reason: the cells tied at a ceiling, so the design could not have resolved regardless of the truth.
| experiment |
tie |
ceiling |
association-work |
interaction 0.00 |
cortex challenges 12/12, same as the muse |
arena-series (muse factor) |
INCONCLUSIVE |
9/9 in every muse cell |
devague-legs |
identical verdicts on all 12 cases |
task too easy, and separable by input length alone |
muse-arms |
INCONCLUSIVE |
control arm 8/8 correct — no headroom to improve |
The uncomfortable reading: none of the four measured what it set out to measure. They measured task difficulty. Every arm looks identical at a ceiling whether or not the arms differ.
Each was pre-registered properly — thresholds fixed in advance, asserted by value in tests, results published either way. The discipline worked exactly as intended and still produced four non-results, because pre-registration protects against tuning after the data and says nothing about whether the design can resolve at all.
The proposal, in two parts
1. A ceiling lens in /challenge
/challenge's lens list covers adjacent systems, unstated assumptions, overlooked actors, security/migration/concurrency, observability, and cheap probes. It does not ask the question that killed all four:
Can this design resolve, or will every cell tie? What result would look identical whether the hypothesis is true or false?
For a spec that carries an experiment or any comparative claim, that lens is the difference between spending a cycle and wasting one. Concretely it asks: where is the ceiling, where is the floor, and is the expected effect inside the range the instrument can see?
2. A pre-registration must name its escalation path
Stronger, and I think the more valuable half:
State the ceiling risk in advance, and name the escalation path you will take if the arms tie.
That converts a wasted run into a staged one. Instead of publishing INCONCLUSIVE and starting the design over, you climb: raise difficulty by a pre-registered ladder until the arms separate. Separation is the result; exhausting the ladder is the fallback. Both outcomes are informative, and neither requires a post-hoc decision that would breach the pre-registration.
We are running exactly this now (embodiment#34): a rising ladder of scenario difficulty and tightening constraints, with the stopping rule climb until they separate fixed before the first dial.
Where it would live
/challenge's lens list — one bullet, plus a line in the worked example.- Whatever guidance says what a pre-registration contains. If that is currently practice rather than a document (it is, on our side — each of our
*-preregistration.md files restates the rules inline), this might be the thing worth codifying.
What I am not claiming
That this generalises beyond comparative/experimental specs — for an ordinary feature spec there is no ceiling to speak of, and the lens would be noise. It may belong as a conditional lens ("if the spec makes a comparative claim…") rather than an unconditional one.
Nor that a ladder is always available. Our devague-legs leg is bounded by case supply, not difficulty: its ground truth comes from committed history and only six approved deviations exist. A design can be unescalatable, and saying so in advance is itself worth the lens.
Happy to contribute the wording if the shape is right.
A method proposal from four consecutive
INCONCLUSIVEresults inagentculture/embodiment, all of which failed the same way. Proposing only — this is devague's method to change, and I may be over-generalising from one repo's experience.The evidence
Four pre-registered experiments, each carefully designed, each returning
INCONCLUSIVE— and every one for the same structural reason: the cells tied at a ceiling, so the design could not have resolved regardless of the truth.association-workarena-series(muse factor)INCONCLUSIVEdevague-legsmuse-armsINCONCLUSIVEThe uncomfortable reading: none of the four measured what it set out to measure. They measured task difficulty. Every arm looks identical at a ceiling whether or not the arms differ.
Each was pre-registered properly — thresholds fixed in advance, asserted by value in tests, results published either way. The discipline worked exactly as intended and still produced four non-results, because pre-registration protects against tuning after the data and says nothing about whether the design can resolve at all.
The proposal, in two parts
1. A ceiling lens in
/challenge/challenge's lens list covers adjacent systems, unstated assumptions, overlooked actors, security/migration/concurrency, observability, and cheap probes. It does not ask the question that killed all four:For a spec that carries an experiment or any comparative claim, that lens is the difference between spending a cycle and wasting one. Concretely it asks: where is the ceiling, where is the floor, and is the expected effect inside the range the instrument can see?
2. A pre-registration must name its escalation path
Stronger, and I think the more valuable half:
That converts a wasted run into a staged one. Instead of publishing
INCONCLUSIVEand starting the design over, you climb: raise difficulty by a pre-registered ladder until the arms separate. Separation is the result; exhausting the ladder is the fallback. Both outcomes are informative, and neither requires a post-hoc decision that would breach the pre-registration.We are running exactly this now (embodiment#34): a rising ladder of scenario difficulty and tightening constraints, with the stopping rule climb until they separate fixed before the first dial.
Where it would live
/challenge's lens list — one bullet, plus a line in the worked example.*-preregistration.mdfiles restates the rules inline), this might be the thing worth codifying.What I am not claiming
That this generalises beyond comparative/experimental specs — for an ordinary feature spec there is no ceiling to speak of, and the lens would be noise. It may belong as a conditional lens ("if the spec makes a comparative claim…") rather than an unconditional one.
Nor that a ladder is always available. Our
devague-legsleg is bounded by case supply, not difficulty: its ground truth comes from committed history and only six approved deviations exist. A design can be unescalatable, and saying so in advance is itself worth the lens.Happy to contribute the wording if the shape is right.