test: make single-task job fixtures assert their own output

[leongdl]

What changed

187 single-task job fixtures across base, EXPR, FEATURE_BUNDLE_1 and
REDACTED_ENV_VARS now assert their own output, and run_openjd_cli_tests.py now
requires a valid job test to exit 0.

1160 passed, 0 failed on openjd-rs and on openjd-cli 0.7.6. The diff is purely
additive: 187 files, +7856, −0.

Why

expected.output is only checkable by a runner that can read the implementation's
stdout. An implementation that can only observe task status — a service rather than
a CLI whose pipe the runner holds — gets "a task ran and exited 0", not "it produced the
right answer". {{Param.Version}} resolving to 2.0 instead of 1.0 exits 0 and
passes.

Each instrumented case's onRun takes one leading argument, an
OpenJDConformanceAssert embedded file, followed by its original command and args:

      actions:
        onRun:
          command: python
          args:
          - '{{Task.File.OpenJDConformanceAssert}}'
          - python
          - -c
          - print(r'OUTPUT:{{Param.Version}}')

The wrapper runs sys.argv[1:], echoes the output verbatim, compares it against
literals baked into itself, and exits non-zero on mismatch. expected.output /
expected.forbidden are untouched, so a log-scanning runner is unaffected — both
mechanisms now check the same thing by different means.

In-idiom rather than new: expected.taskFailure already lets exit status carry a
verdict.

928 of 1554 declared assertion lines are now checked inside the task:

bundle	cases	instrumented	lines checked	declared
base	167	61	114	453
EXPR	142	119	802	903
FEATURE_BUNDLE_1	13	6	10	16
REDACTED_ENV_VARS	8	1	2	23
WRAP_ACTIONS	65	0	0	139
TASK_CHUNKING	7	0	0	20

Why the argv is passed through rather than embedded

Embedding the original argv in the wrapper's source works for base and would have
quietly gutted several EXPR cases. expr1.3.2--list-flattens-in-args asserts
ARG0:--width … COUNT:10; its subject is how the implementation expands an args
list, flattening a list value into separate arguments and skipping a null one. Freezing
argv into the wrapper moves that expansion from the implementation into the fixture, so
the case keeps passing while testing nothing.

Passing argv through leaves the expansion where it belongs, and removes every escaping
hazard embedding had.

Two false passes found on the way

The runner never checked exit status for a valid case. A mutation probe — corrupt
what the job prints, leave the expected literal alone — passed:

EXPECTED = ["OUTPUT:hello"]   job prints OUTPUT:WRONG
-> OPENJD_CONFORMANCE_ASSERT_FAILED: missing expected output: OUTPUT:hello
-> Process exited with code: 1
-> ✓

Two independent causes, both fixed. The runner ignored returncode when no
expected.taskFailure was declared, so the verdict rested entirely on substring
matching; and the wrapper's diagnostic echoed the expected literal, so the log scan
matched the error message. Diagnostics now report expected output line 1 of 2 not found and never the text. The probe reports ✗ on both CLIs now.

The exit-status requirement is a hole in the shared runner independent of the fixtures,
so it is a separate commit.

46 wrappers that never ran. All 47 instrumentable WRAP_ACTIONS cases passed after
instrumentation, then the success marker showed 46 of them had not executed the wrapper
at all: onWrapTaskRun replaces the task action, so a hook like
args: ["-c", "print('TIMEOUT={{WrappedAction.Timeout}}')"] never runs
WrappedAction.Command. Shipping those would have added verification that looks present
and cannot fail — the same failure one layer up. WRAP_ACTIONS is excluded wholesale.

That is what OPENJD_CONFORMANCE_ASSERT_OK: <n> expected, <m> forbidden is for, and why
it is worth keeping.

Deliberately left status-only

left out	why
Multi-task cases	Each task sees only its own output; a per-task exit code cannot assert that N lines appeared across N tasks. Needs aggregation outside the job.
9 redaction cases	4--openjd-redacted-env expects SECRET_IS:********, the implementation's redaction of what the task printed. The task sees the real value. unset-takes-precedence is the one that does self-assert, because there the variable really is unset.
4 env-output cases	7.3--env-file-reference asserts a line printed by a job environment's onEnter; its task prints something else.
All of WRAP_ACTIONS	Above.
wrap-no-args	It asserts WrappedAction.Args surfaces as an empty list when the action has no args. Adding an argument changes the subject. Encoded as a rule: an action with no args is never instrumented.
expected.taskFailure cases	Already carry their verdict in exit status.

Every one of these except the first was found by instrumenting the case and watching it
fail, or watching it pass without asserting — not by reading the fixture.

Testing

	result
openjd-rs 2023-09/*	1160 passed, 0 failed
openjd-cli 0.7.6 2023-09/*	1160 passed, 0 failed
Assertion confirmed to have executed	180 of 187 on both CLIs
Mutation probe	✗ on both; ✓ before the fixes
Original argv, command and expected preserved	187 of 187, checked against the parent commit

The 7 not confirmed are Windows-only and skipped on a POSIX host
(7.3--path-param-mapping-windows, 7.3--rawparam-no-mapping-windows, four
expr2.3.2--uri-*-windows, 6.1--end-of-line-auto-windows). Each has a POSIX sibling
that passes and differs only in which literal list the wrapper selects, so the risk is
low — but they are genuinely unverified until CI runs them on windows-latest.

[mwiebe]

Almost the whole diff appears to repeat this boilerplate. This likely reduces the readability of these tests for someone casually browsing these files. Are there other ways we could approach this?

[leongdl]

Yeah, thanks to claude Fable :) I have a new steering prompt to DELETE all this cruff.

Sorry I have not had time to review this one but it is necessary to make the tests robust and useful for backend.

[leongdl]

Done — the boilerplate is gone. The machinery now lives in one flat file, conformance-tests/_conformance_assert.py, and a fixture pulls it in with dataFile:. This case is now:

      embeddedFiles:
      - name: OpenJDConformanceAssert
        type: TEXT
        dataFile: _conformance_assert.py
      - name: OpenJDConformanceExpect
        type: TEXT
        data: |
          {"expected": ["OUTPUT:hello"]}

Fixture insertions drop from 7,902 to 2,048, and a wrapper fix is now one edit instead of 187.

One thing I tried first and could not make work, since it would have needed no harness change at all: putting the expected lines directly in args and keeping the wrapper fully generic. 2023-09 format strings have no escape for {{, so the 15 cases that forbid a literal {{Param. in their output cannot express it there. Checked against both CLIs rather than assumed — a bare {{Param. is a template validation error, and neither \{\{ nor a doubled {{{{ escapes it; both pass through as those literal characters. JSON's own \uXXXX decodes the braces after the format-string pass, which is why the expect file is an embedded file rather than argv text.

[leongdl]

Update: the wrapper is now one shared flat file

Pushed refactor: move the self-asserting wrapper into one shared flat file in response to @mwiebe's point about the boilerplate. Note the PR description above still describes the previous shape — read this comment for what the diff actually does now.

The machinery was copied verbatim into all 187 instrumented fixtures: one distinct body, 5,236 duplicated lines, 66% of what the change added. It now lives in conformance-tests/_conformance_assert.py, and each fixture pulls it in with dataFile: on its OpenJDConformanceAssert embedded file. The case's own expected and forbidden lines move to a second embedded file, OpenJDConformanceExpect, as one line of JSON.

A fixture is now 8 lines of scaffolding:

      embeddedFiles:
      - name: OpenJDConformanceAssert
        type: TEXT
        dataFile: _conformance_assert.py
      - name: OpenJDConformanceExpect
        type: TEXT
        data: |
          {"expected": ["OUTPUT:hello"]}

	before	after
fixture insertions	7,902	2,048
whole PR	7,946	2,259
duplicated machinery lines	5,236	0 (one 60-line file)

dataFile is a harness convention, not a template schema change: it names a file relative to the conformance-tests root whose contents become the embedded file's data. Resolving it is the only preprocessing these tests require — inline_data_files() in the runner is twenty lines including the comment. Because data is required on an embedded file, a harness that skips the step submits an invalid template and fails loudly rather than silently reverting to a status-only verdict.

Why the literals are not simply in args

That shape would need no harness change at all, so I tried it first. It cannot work: 2023-09 format strings have no escape for {{, and 15 cases forbid a literal {{Param. in their output. Verified against both CLIs rather than reasoned about:

carrier in args	openjd-rs	openjd-cli 0.7.6
bare {{Param.	rejected, validation error	rejected, validation error
{{{{Param.	rejected	rejected
\{\{Param.	passes through as those literal characters	same
\u007b\u007bParam.	passes through as those literal characters	same

JSON's own \uXXXX escape decodes the braces after the format-string pass has run, so the expect file has to be an embedded file. That is the whole reason for the second file rather than two more argv elements.

Verification

Checked against the pre-migration commit, per fixture and per platform:

	result
literals the task reads are identical to the ones the old wrapper baked in	187 / 187
rest of the case unchanged — original command, args, expected block	187 / 187
expect file is valid JSON	187 / 187
fixtures that lost instrumentation	0

Those checks are themselves mutation-tested, because a migration verifier that cannot fail is worse than none. Three deliberate corruptions — altered literal, altered original command, dropped expected_windows key — were each caught, and the tree was restored and re-verified clean afterwards.

Conformance, on the committed tree:

	openjd-rs	openjd-cli 0.7.6
2023-09/*	1160 passed, 0 failed	1160 passed, 0 failed
assertion confirmed to have executed	180 of 187	180 of 187
cases passing without the success marker	0	0
mutation probe: corrupt what the job prints, keep the expected literal	caught in-task	caught in-task

Both runs produce the same 14 per-suite counts, matching the pre-migration baseline. The 7 not confirmed are the Windows-only cases, skipped on a POSIX host.

The 6 fixtures whose subject is embedded files or Task.File now carry two of them; all 6 pass on both CLIs, so that hazard is closed by test rather than by argument.

Still open

• Windows is unrun. The expected_windows selection path is exercised by the migration verifier, which evaluates the wrapper for both platforms, but nothing has executed on a Windows host. Same gap this PR already carried for its 7 Windows-only cases; genuinely unverified until CI runs windows-latest.
• Service-side harnesses need the dataFile resolution before they bump their pinned conformance SHA. For a status-only consumer this is the change that keeps the content verdict working. It fails closed, so a harness that misses it goes red rather than quietly weaker — but it does have to land first.

[leongdl]

CI is red, and it is not this PR

Attribution checked before saying so. The failure profile on this PR is byte-identical to the one on mainline HEAD, which contains none of this instrumentation:

	PR #174 (550f542)	mainline HEAD (1e4d49e)
Conformance Tests (Rust), ubuntu + macos	1159 passed, 2 failed	1159 passed, 2 failed
Conformance Tests (Rust), windows	1108 passed, 8 failed	1108 passed, 8 failed
failing test names	same 8	same 8

Same counts, same names. This PR adds no failures. The Python workflow was already red on the previous head bc609e1 as well.

Cause. .github/workflows/conformance_tests_rust.yml runs cargo install openjd-cli with no version pin, so it resolves the published crate — 0.1.10 for this run. Mainline commit 1e4d49e ("Specify list string escaping and nested repr_pwsh arrays") landed spec and fixtures for behaviour that published CLI does not implement yet; the implementation side exists in openjd-rs (51df99c fix(expr): escape list elements when converting a list to a string, #336) but is not released. So the spec repo's own CI is testing new fixtures against an older CLI.

Note also that expr2.2.1--string-conversion-list-escaping does not exist on this branch at all — it arrives in CI because a pull_request checkout merges head with base. That fixture failing here is mainline's, not mine.

Two of the names are cases this PR does instrument (expr2.2.6--repr-pwsh, and on Windows also expr1.2.6--list-type-inference, expr2.2.3--flatten, expr2.3.1--uri-path-scheme-detection). I checked those specifically rather than waving at the totals: all of them fail identically on mainline, where they carry no wrapper, so the instrumentation is not implicated.

Locally, against an openjd-rs build that includes #336, this branch is 1160 passed, 0 failed — and the same on openjd-cli 0.7.6 (Python). Both with the same 14 per-suite counts as the pre-instrumentation baseline.

Worth fixing separately: pinning cargo install openjd-cli --version in that workflow would stop mainline's spec landings from silently reddening every open PR. Happy to raise that as its own change if useful — it is not in scope here.