Skip to content

chore(loop): expose the engine model to workflow_dispatch, and fix the push-after-skipped-pull failure (D34) - #44

Merged
Alex1990 merged 3 commits into
mainfrom
chore/loop-model-input
Sep 11, 2026
Merged

Alex1990 merged 3 commits into
mainfrom
chore/loop-model-input

Conversation

@Alex1990

@Alex1990 Alex1990 commented Sep 11, 2026 •

Copy link
Copy Markdown
Owner

Executes the A1 sample-gathering test I proposed, and it paid for itself: the guard works, the serial lock is proven, and the test exposed a real defect (D34).

1. The guard holds — D19 verified for the first time

workflow_dispatch had no model input (LOOP_MODEL was a hardcoded job env), so a bad model could not be produced at all. Added an optional model input, defaulting to the same id as before; validation is unchanged.

Dispatching with nonexistent/model-xyz:

provider  model                         context  max-out  thinking  images
deepseek  deepseek-v4-flash             1M       384K     yes       no
deepseek  deepseek-v4-flash-vision-exp  1M       384K     yes       yes
deepseek  deepseek-v4-pro               1M       384K     yes       no
##[error]LOOP_MODEL=nonexistent/model-xyz not in pi's DeepSeek catalog (want 'model-xyz')
##[error]Process completed with exit code 1.

Fails loudly, names the want/actual, prints the catalog. Zero LLM tokens — the guard trips before the agent. This closes the "the guard was never tested with a real bad value" gap.

2. Serial lock — proven, with a caveat about how you read the timestamps

Two dispatches 25s apart:

workflow createdAt job created_at job completed_at
run 1 14:02:38 14:02:42 14:03:07
run 2 14:03:03 14:03:07 14:03:36

Run 2 entered the queue at 14:03:03, but its job was not created until 14:03:07 — exactly run 1's job completed_at — and "Set up job" logs at 14:03:10. A 4s queue.

The trap: at workflow level created == started for both runs, so reading those alone would look like they ran concurrently. Only the job-level timestamps show the serialization. Worth recording, because "R2 has no concurrent writes" rests on this.

3. D34 — a second red step that buried the first cause

Every one of the failing runs showed:

9 Push state to R2 → failure
[loop:r2] push ← /home/runner/work/tiny-oss/tiny-oss/state
aws: [ERROR]: The user-provided path /home/runner/work/tiny-oss/tiny-oss/state does not exist.
[loop:r2] error: [r2:push] aws cli exited with code 255

state/ is gitignored and only materialises from a pull. The push is if: always() — correct intent (a crashed run must still record its processing state for the next sweep) — but with Pull state from R2 skipped there is nothing to upload.

The verdict did not change (the job was already failing), but this is the phase where a human reads these logs daily: two red steps read like R2 also broke, and the second failure is pure noise on top of the real one.

Fix: the pull gets an id, the push is conditioned on

if: always() && steps.pull.outcome != 'skipped'

!= 'skipped' rather than == 'success' deliberately — a failed pull may still leave a usable tree, and recording post-crash state is precisely what always() is for.

Verified on the same bad model after the fix (34608064576): step 9 is now skipped, step 6 stays failure. The only red step is the one that genuinely failed.

4. What this test did not establish

Recording these because I proposed this test partly on the expectation it would cover them, and it does not:

A1 item status after this test
Crash path (task stays processing → sweep reclaims by TTL) still unsampled. The failure lands in Install engine (pi), before Run stage, so no task is ever claimed — steps 7 and 8 were both skipped. Needs a job cancelled after entry.mjs claims.
Failure classification (retry vs inbox) still unexercised. A guard failure is a workflow step failure: entry.mjs never ran, so no run row exists and the classification table was never consulted.

Both are now written into the A1 checklist with the reason they cannot be tested this way, so the next person does not repeat the attempt.

Verification

@github-actions

Copy link
Copy Markdown
Contributor

Coverage

Metric % Covered / Total
Lines 90.19% 1003/1112
Statements 89.06% 1083/1216
Functions 91.03% 193/212
Branches 76.43% 519/679

The D19 guard exists because pi silently falls back to its default model for an
unknown `--model`, so a typo would run every run with the wrong model. The guard
was never exercised by a real bad value, and there was no way to produce one:
`LOOP_MODEL` was a hardcoded job-level env value with no dispatch input.

Add an optional `model` input, defaulting to the same id as before, so the guard
can be tested end to end and a model can be switched for debugging without
editing the workflow. Validation is unchanged and still fails loudly.

Verified by dispatching with a deliberately unknown model; see the PR.
D34, found by the D19 guard test (two dispatches with a deliberately unknown
model). Both runs failed correctly at `Install engine (pi)`, but then
`Push state to R2` failed too:

  aws: [ERROR]: The user-provided path
  /home/runner/work/tiny-oss/tiny-oss/state does not exist.

`state/` is gitignored and only ever appears from a pull, so when an earlier
step fails there is nothing to upload. The push is `if: always()` so a run that
crashes or times out still records its processing state — that intent is right,
but it needs the pull to have happened.

The job was already failing, so the verdict did not change. It still matters: a
second red step buries the first cause and reads like R2 itself broke, and this
is the phase where a human reads these logs daily.

Condition on `steps.pull.outcome != 'skipped'` rather than `== 'success'`: a
failed pull may still leave a usable tree, and post-crash state is precisely
what `always()` is for.

Also records what the guard test did and did not establish:
- serial lock: verified — run 2's *job* was created exactly when run 1's job
  completed, a 4s queue that workflow-level timestamps hide
- crash path and failure classification: still unexercised, and now provably
  not testable this way — the failure precedes `entry.mjs`, so nothing is ever
  claimed and no run row is written
Also reconciles the A1 checklist with the guard test's findings, and keeps
'recorded runs' distinct from 'workflow executions': the three guard failures
die at `Install engine (pi)`, before `entry.mjs`, so they write no run row —
21 executions, 18 run records. Conflating the two would have made the engine
stability count wrong.
@Alex1990
Alex1990 force-pushed the chore/loop-model-input branch from 1191b1a to ed735a0 Compare September 11, 2026 14:22
@Alex1990
Alex1990 merged commit 3019498 into main Sep 11, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant