-
Notifications
You must be signed in to change notification settings - Fork 35
test: [expected failures] EXPR fixtures needing implementation fixes #164
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: mainline
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # The maximum <UserIdentifier> length for a let binding is 512 characters | ||
| # (Template Schemas §3.6.1). This identifier is 513 characters and must be | ||
| # rejected. The 512-character accept twin is in 3.6--let-boundary-edges.yaml. | ||
| specificationVersion: jobtemplate-2023-09 | ||
| extensions: | ||
| - EXPR | ||
| name: TestJob | ||
| steps: | ||
| - name: Step1 | ||
| let: | ||
| - aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa = 42 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - "-c" | ||
| - "print()" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # Proposed EXPR fixtures (expression language) — job_templates | ||
|
|
||
| Fixtures listed here are spec-correct but parked instead of live, either | ||
| because a reference implementation has a known divergence that would fail | ||
| the suite, or because the normative spec text they test needs to be | ||
| restored first. Placement is kind-level (`<component>/<kind>/proposed/`) | ||
| so promotion is a mechanical move up one directory; the runner does not | ||
| scan `proposed/` directories. This README is named per fixture family | ||
| (`README-expr-lang.md`) because other expected-failures PRs park fixtures | ||
| in this same directory with their own READMEs. | ||
|
|
||
| Observations below are per-implementation: `rs` = openjd-rs CLI, `py` = | ||
| openjd-model/cli for Python. Dual-implementation results are from the | ||
| 2026-08-12 sweep. | ||
|
|
||
| ## Integer overflow on the parameter-value axis (blocked on spec restoration) | ||
|
|
||
| RFC 0005's normative "64-bit Signed Integer Type" section — the only text | ||
| stating that integer overflow is an error for literals, arithmetic, and | ||
| conversions — was dropped from the published `2026-02-Expression-Language.md`. | ||
| Only the type-table row in §1.2.1 ("int | 64-bit signed integer values (−2⁶³ | ||
| to 2⁶³−1)") survives. These reject fixtures test behavior the published spec | ||
| arguably no longer requires, so they are parked until the spec text is | ||
| restored. | ||
|
|
||
| ### `expr2.1.1--int64-overflow-param-default.invalid.yaml` | ||
|
|
||
| - Requires: INT job parameter `default: 9223372036854775808` (2^63) rejected | ||
| at template validation. | ||
| - Observed: **accepted by BOTH rs and py** — `openjd check` exits 0. The | ||
| YAML parser yields an arbitrary-precision integer and no range check is | ||
| applied to parameter defaults. | ||
| - Classification: implementation bug in both (out-of-range acceptance), | ||
| plus blocked on spec text restoration. Near-duplicate of the base | ||
| expected-failures PR's `2.3--int-default-above-int64-max` — kept because | ||
| the EXPR declaration could plausibly route through a different (typed) | ||
| validation path; if the fix turns out to share one code path, drop this | ||
| copy. | ||
|
|
||
| ### expr2.1.1--int64-overflow-param-supplied (promoted) | ||
|
|
||
| - Verified green on BOTH implementations (each rejects a supplied | ||
| job-parameter value of 2^63 at job creation), so this fixture ships as a | ||
| live `jobs/` fixture in the expr-lang-gaps PR: it pins behavior both | ||
| implementations already agree on, independent of spec-text restoration. | ||
|
|
||
| ### `expr2.1.1--int64-overflow-task-range.invalid.yaml` | ||
|
|
||
| - Requires: task parameter range element 2^63 rejected. | ||
| - Observed: **rs rejects / py ACCEPTS** (2026-08-12 sweep) — this is a live | ||
| Python defect, not a spec-blocked pass. (An earlier revision of this | ||
| README claimed the fixture "passes today"; that was true only of rs, and | ||
| rs's rejection may itself fire via its 2^62 over-rejection bug — see the | ||
| jobs/proposed accept twin — i.e. possibly for the wrong reason.) | ||
| - Classification: implementation bug (py accepts out-of-range), plus | ||
| blocked on spec restoration; promote together with the int64-max accept | ||
| twin so the rejection is attributable to a real bounds check. | ||
|
|
||
| ## Implementation divergences | ||
|
|
||
| ### `expr2.1.3--membership-element-type-mismatch.invalid.yaml` | ||
|
|
||
| - Requires: `"a" in [1, 2]` rejected. Expression Language §2.1.3 defines list | ||
| membership only as `__contains__(list: list[T], item: T)` — there is no | ||
| signature for a string item against `list[int]`, and the implicit | ||
| string→int coercion of `"a"` errors (§1.2.3), so no resolution exists. | ||
| - Observed: **rs accepts** (evaluates to `false` at runtime — apparent | ||
| fallback to §1.2.5 cross-type equality semantics); **py rejects** | ||
| (spec-conformant). | ||
| - Classification: openjd-rs implementation bug (missing type check). The | ||
| fixture is spec-correct as written. Two caveats for promotion: (a) the | ||
| spec does not explicitly require the error at TEMPLATE VALIDATION time — | ||
| a runtime-error implementation would arguably conform yet fail this | ||
| .invalid fixture; (b) the fix converts silently-false membership tests in | ||
| existing templates into hard errors — schedulers should scan before | ||
| adopting. | ||
|
|
||
| ### `3.6.1--let-identifier-513.invalid.yaml` | ||
|
|
||
| - Requires: a 513-character let-binding `<UserIdentifier>` rejected | ||
| (Template Schemas §3.6.1: "Maximum length of `<UserIdentifier>`: 512 | ||
| characters"). | ||
| - Observed: **accepted by BOTH rs and py** (2026-08-12 sweep) — the | ||
| identifier length limit is not enforced by either implementation. | ||
| - Classification: implementation bug in both (missing limit check). The | ||
| 512-character accept twin is `3.6--let-boundary-edges.yaml`, added by the | ||
| expr-lang-gaps PR. |
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quorum verdict: GOOD-WITH-NITS — spec: EL §1.2.1 type-table row (the explicit overflow-is-error text was dropped from the published spec; README honestly gates promotion on restoring it — right call). Nits: sweep shows BOTH implementations fail; README reports only openjd-rs. Near-duplicate of PR #163's base 2.3--int-default-above-int64-max (only extensions: [EXPR] differs) — justified only if EXPR changes numeric handling; say so or dedupe. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # The int type is a 64-bit signed integer (Expression Language §1.2.1; | ||
| # RFC 0005 "64-bit Signed Integer Type", dropped from the published form). | ||
| # A job parameter default of 2^63 = 9223372036854775808 is out of range and | ||
| # must be rejected at template validation. YAML parsers yield | ||
| # arbitrary-precision integers silently, so this is the axis where an | ||
| # implementation is most likely to let an out-of-range value through. | ||
| specificationVersion: jobtemplate-2023-09 | ||
| extensions: | ||
| - EXPR | ||
| name: TestJob | ||
| parameterDefinitions: | ||
| - name: TooBig | ||
| type: INT | ||
| default: 9223372036854775808 | ||
| steps: | ||
| - name: Step1 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - "-c" | ||
| - "print(r'{{ Param.TooBig }}')" |
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quorum verdict: GOOD-WITH-NITS with a REQUIRED README correction (1 reviewer voted NOT-GOOD on this point) — spec: EL §1.2.1 (2^63 task-range element rejected). The README claims this is 'correctly rejected... passes today; parked pending spec restoration' — but the 2026-08-12 sweep shows the PYTHON CLI accepts 2^63 in a task range: there IS a live defect and this is a genuine expected failure. Update the entry (and note openjd-rs's rejection may fire via its over-rejecting 2^62 cap, i.e. the wrong reason — promote together with the int64-max accept twin). |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # The int type is a 64-bit signed integer (Expression Language §1.2.1; | ||
| # RFC 0005 "64-bit Signed Integer Type", dropped from the published form). | ||
| # A task parameter range element of 2^63 = 9223372036854775808 is out of | ||
| # range and must be rejected at template validation. | ||
| specificationVersion: jobtemplate-2023-09 | ||
| extensions: | ||
| - EXPR | ||
| name: TestJob | ||
| steps: | ||
| - name: Step1 | ||
| parameterSpace: | ||
| taskParameterDefinitions: | ||
| - name: TooBig | ||
| type: INT | ||
| range: "9223372036854775808" | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - "-c" | ||
| - "print(r'{{ Task.Param.TooBig }}')" |
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quorum verdict: GOOD-WITH-NITS — spec: EL §2.1.3 (contains(list[T], T); no signature for "a" in [1,2], and "a"→int coercion is defined to error) — the error mandate is solid; openjd-rs evaluating it to false is a real bug (matches sweep, rs-only). Nits: (1) the spec doesn't explicitly require the error at TEMPLATE VALIDATION time, which the .invalid fixture demands — a runtime-error implementation would arguably conform yet fail; note the timing assumption in the README. (2) Service risk worth recording: the fix converts silently-false membership tests into hard errors for running workloads — recommend a template scan before service adoption. Accept twin: mainline expr2.1.3--list-ops. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # List membership is __contains__(list: list[T], item: T) — the item type | ||
| # must match the list's element type (Expression Language §2.1.3). There is | ||
| # no signature for a string item against list[int], so '"a" in [1, 2]' must | ||
| # be rejected rather than evaluating to false. | ||
| specificationVersion: jobtemplate-2023-09 | ||
| extensions: | ||
| - EXPR | ||
| name: TestJob | ||
| steps: | ||
| - name: Step1 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - -c | ||
| - print(r'{{ "a" in [1, 2] }}') |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # Proposed EXPR fixtures (expression language) — jobs | ||
|
|
||
| Kind-level parking (`<component>/<kind>/proposed/`); promotion is a move up | ||
| one directory. The runner does not scan `proposed/`. Named per fixture | ||
| family because other expected-failures PRs park fixtures here with their | ||
| own READMEs. | ||
|
|
||
| ### `expr2.1.1--int64-max-task-range.test.yaml` | ||
|
|
||
| - Requires: task parameter range element 2^63−1 = 9223372036854775807 | ||
| **accepted** (Expression Language §1.2.1: int covers −2⁶³ to 2⁶³−1). | ||
| - Observed: **rs rejects** at template validation — `INT parameter 'Big' | ||
| range expression error: Integer overflow: result is outside the 64-bit | ||
| signed range`. Probing shows single-element ranges are accepted at | ||
| 4611686018427387903 (2^62−1) and rejected from 4611686018427387904 (2^62) | ||
| upward, suggesting an internal computation (e.g. a length or midpoint | ||
| calculation) overflows before the value itself is range-checked. | ||
| **py passes** (accepts the valid value). | ||
| - Classification: openjd-rs implementation bug (false reject inside the | ||
| valid int64 domain). Fixing it widens acceptance only. Promote together | ||
| with `../job_templates/proposed/expr2.1.1--int64-overflow-task-range` — | ||
| until this accept twin is green on rs, that reject twin passes on rs for | ||
| possibly the wrong reason (the same 2^62 cap). The job-parameter accept | ||
| twins at 2^63−1 are `expr2.1.1--int64-max-param-values.test.yaml` in the | ||
| expr-lang-gaps PR. |
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quorum verdict: GOOD (5/5) — spec: EL §1.2.1 + §3.4.1.1.1 (2^63-1 is a valid int and a valid range element — must be ACCEPTED). Pins openjd-rs's false-reject from 2^62 up (the probing detail in the README is excellent). Fix widens acceptance only — zero customer-breakage direction — and keeps the overflow twin honest (that twin passes for the wrong reason until this is green; promote as a pair). Nits: README cites accept twin ../jobs/expr2.1.1--int64-max-param-values.test.yaml, which lives on PR #160's branch, not here — dangling until #160 merges; state that Python passes. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # The int type is a 64-bit signed integer (Expression Language §1.2.1), so | ||
| # 2^63-1 = 9223372036854775807 must be accepted as a task parameter range | ||
| # element. This is the accept twin of expr2.1.1--int64-overflow-task-range; | ||
| # the job-parameter accept twins live in | ||
| # ../jobs/expr2.1.1--int64-max-param-values.test.yaml. | ||
| template: | ||
| specificationVersion: jobtemplate-2023-09 | ||
| extensions: | ||
| - EXPR | ||
| name: TestJob | ||
| steps: | ||
| - name: Step1 | ||
| parameterSpace: | ||
| taskParameterDefinitions: | ||
| - name: Big | ||
| type: INT | ||
| range: "9223372036854775807" | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - -c | ||
| - "print(r'TASK:{{ Task.Param.Big }}')" | ||
| expected: | ||
| output: | ||
| - "TASK:9223372036854775807" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Quorum verdict: GOOD (5/5, one attribution fix) — spec: §3.6.1 (UserIdentifier max 512 — an explicit numeric limit; 513 must be rejected). Identifier verified exactly 513 chars; accept twin at 512 in PR #160's 3.6--let-boundary-edges. Risk-free promotion once fixed. Fix: README reports only openjd-rs acceptance; the sweep shows BOTH implementations fail.