From 53ac70b621dbd0336f32f988fb0133f649eeb022 Mon Sep 17 00:00:00 2001 From: David Leong <116610336+leongdl@users.noreply.github.com> Date: Sat, 1 Aug 2026 19:36:02 -0700 Subject: [PATCH] test: Add expected-failure EXPR expression-language fixtures (parked in proposed/) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five parked fixtures with dual-implementation observations: the int64 param-default acceptance (both implementations), the openjd-rs 2^62 false-reject on task-range endpoints (Python accepts, spec admits any int64), the Python-only task-range 2^63 acceptance, the untyped membership acceptance in openjd-rs, and the unenforced 513-char let-identifier cap (both). Per-fixture output, spec citation, and classification in proposed/README.md. Companion to conformance-expr-lang-gaps. Review: quorum-review fixes — kind-level proposed/ placement with a family-named README (README-expr-lang.md) so co-located expected-failures PRs cannot add/add-conflict on a shared README path; corrected the int64-overflow-task-range entry (the Python CLI ACCEPTS 2^63 in a task range per the 2026-08-12 sweep — a live defect, not a spec-blocked pass, and rs's rejection may fire via its own 2^62 over-rejection); added dual-implementation attribution throughout (let-identifier-513 and int64-overflow-param-default fail BOTH); noted the membership fixture's validation-time assumption and silent-false-to-hard-error adoption risk; fixed accept-twin references to name the PR branches they live on. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com> --- .../3.6.1--let-identifier-513.invalid.yaml | 18 ++++ .../proposed/README-expr-lang.md | 87 +++++++++++++++++++ ...-int64-overflow-param-default.invalid.yaml | 23 +++++ ....1--int64-overflow-task-range.invalid.yaml | 22 +++++ ...bership-element-type-mismatch.invalid.yaml | 17 ++++ .../EXPR/jobs/proposed/README-expr-lang.md | 25 ++++++ .../expr2.1.1--int64-max-task-range.test.yaml | 27 ++++++ 7 files changed, 219 insertions(+) create mode 100644 conformance-tests/2023-09/EXPR/job_templates/proposed/3.6.1--let-identifier-513.invalid.yaml create mode 100644 conformance-tests/2023-09/EXPR/job_templates/proposed/README-expr-lang.md create mode 100644 conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.1--int64-overflow-param-default.invalid.yaml create mode 100644 conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.1--int64-overflow-task-range.invalid.yaml create mode 100644 conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.3--membership-element-type-mismatch.invalid.yaml create mode 100644 conformance-tests/2023-09/EXPR/jobs/proposed/README-expr-lang.md create mode 100644 conformance-tests/2023-09/EXPR/jobs/proposed/expr2.1.1--int64-max-task-range.test.yaml diff --git a/conformance-tests/2023-09/EXPR/job_templates/proposed/3.6.1--let-identifier-513.invalid.yaml b/conformance-tests/2023-09/EXPR/job_templates/proposed/3.6.1--let-identifier-513.invalid.yaml new file mode 100644 index 0000000..594296b --- /dev/null +++ b/conformance-tests/2023-09/EXPR/job_templates/proposed/3.6.1--let-identifier-513.invalid.yaml @@ -0,0 +1,18 @@ +# The maximum length for a let binding is 512 characters +# (Template Schemas §3.6.1). This identifier is 513 characters and must be +# rejected. The 512-character accept twin is in 3.6--let-boundary-edges.yaml. +specificationVersion: jobtemplate-2023-09 +extensions: +- EXPR +name: TestJob +steps: +- name: Step1 + let: + - aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa = 42 + script: + actions: + onRun: + command: python + args: + - "-c" + - "print()" diff --git a/conformance-tests/2023-09/EXPR/job_templates/proposed/README-expr-lang.md b/conformance-tests/2023-09/EXPR/job_templates/proposed/README-expr-lang.md new file mode 100644 index 0000000..36a680c --- /dev/null +++ b/conformance-tests/2023-09/EXPR/job_templates/proposed/README-expr-lang.md @@ -0,0 +1,87 @@ +# Proposed EXPR fixtures (expression language) — job_templates + +Fixtures listed here are spec-correct but parked instead of live, either +because a reference implementation has a known divergence that would fail +the suite, or because the normative spec text they test needs to be +restored first. Placement is kind-level (`//proposed/`) +so promotion is a mechanical move up one directory; the runner does not +scan `proposed/` directories. This README is named per fixture family +(`README-expr-lang.md`) because other expected-failures PRs park fixtures +in this same directory with their own READMEs. + +Observations below are per-implementation: `rs` = openjd-rs CLI, `py` = +openjd-model/cli for Python. Dual-implementation results are from the +2026-08-12 sweep. + +## Integer overflow on the parameter-value axis (blocked on spec restoration) + +RFC 0005's normative "64-bit Signed Integer Type" section — the only text +stating that integer overflow is an error for literals, arithmetic, and +conversions — was dropped from the published `2026-02-Expression-Language.md`. +Only the type-table row in §1.2.1 ("int | 64-bit signed integer values (−2⁶³ +to 2⁶³−1)") survives. These reject fixtures test behavior the published spec +arguably no longer requires, so they are parked until the spec text is +restored. + +### `expr2.1.1--int64-overflow-param-default.invalid.yaml` + +- Requires: INT job parameter `default: 9223372036854775808` (2^63) rejected + at template validation. +- Observed: **accepted by BOTH rs and py** — `openjd check` exits 0. The + YAML parser yields an arbitrary-precision integer and no range check is + applied to parameter defaults. +- Classification: implementation bug in both (out-of-range acceptance), + plus blocked on spec text restoration. Near-duplicate of the base + expected-failures PR's `2.3--int-default-above-int64-max` — kept because + the EXPR declaration could plausibly route through a different (typed) + validation path; if the fix turns out to share one code path, drop this + copy. + +### expr2.1.1--int64-overflow-param-supplied (promoted) + +- Verified green on BOTH implementations (each rejects a supplied + job-parameter value of 2^63 at job creation), so this fixture ships as a + live `jobs/` fixture in the expr-lang-gaps PR: it pins behavior both + implementations already agree on, independent of spec-text restoration. + +### `expr2.1.1--int64-overflow-task-range.invalid.yaml` + +- Requires: task parameter range element 2^63 rejected. +- Observed: **rs rejects / py ACCEPTS** (2026-08-12 sweep) — this is a live + Python defect, not a spec-blocked pass. (An earlier revision of this + README claimed the fixture "passes today"; that was true only of rs, and + rs's rejection may itself fire via its 2^62 over-rejection bug — see the + jobs/proposed accept twin — i.e. possibly for the wrong reason.) +- Classification: implementation bug (py accepts out-of-range), plus + blocked on spec restoration; promote together with the int64-max accept + twin so the rejection is attributable to a real bounds check. + +## Implementation divergences + +### `expr2.1.3--membership-element-type-mismatch.invalid.yaml` + +- Requires: `"a" in [1, 2]` rejected. Expression Language §2.1.3 defines list + membership only as `__contains__(list: list[T], item: T)` — there is no + signature for a string item against `list[int]`, and the implicit + string→int coercion of `"a"` errors (§1.2.3), so no resolution exists. +- Observed: **rs accepts** (evaluates to `false` at runtime — apparent + fallback to §1.2.5 cross-type equality semantics); **py rejects** + (spec-conformant). +- Classification: openjd-rs implementation bug (missing type check). The + fixture is spec-correct as written. Two caveats for promotion: (a) the + spec does not explicitly require the error at TEMPLATE VALIDATION time — + a runtime-error implementation would arguably conform yet fail this + .invalid fixture; (b) the fix converts silently-false membership tests in + existing templates into hard errors — schedulers should scan before + adopting. + +### `3.6.1--let-identifier-513.invalid.yaml` + +- Requires: a 513-character let-binding `` rejected + (Template Schemas §3.6.1: "Maximum length of ``: 512 + characters"). +- Observed: **accepted by BOTH rs and py** (2026-08-12 sweep) — the + identifier length limit is not enforced by either implementation. +- Classification: implementation bug in both (missing limit check). The + 512-character accept twin is `3.6--let-boundary-edges.yaml`, added by the + expr-lang-gaps PR. diff --git a/conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.1--int64-overflow-param-default.invalid.yaml b/conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.1--int64-overflow-param-default.invalid.yaml new file mode 100644 index 0000000..e651546 --- /dev/null +++ b/conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.1--int64-overflow-param-default.invalid.yaml @@ -0,0 +1,23 @@ +# The int type is a 64-bit signed integer (Expression Language §1.2.1; +# RFC 0005 "64-bit Signed Integer Type", dropped from the published form). +# A job parameter default of 2^63 = 9223372036854775808 is out of range and +# must be rejected at template validation. YAML parsers yield +# arbitrary-precision integers silently, so this is the axis where an +# implementation is most likely to let an out-of-range value through. +specificationVersion: jobtemplate-2023-09 +extensions: +- EXPR +name: TestJob +parameterDefinitions: +- name: TooBig + type: INT + default: 9223372036854775808 +steps: +- name: Step1 + script: + actions: + onRun: + command: python + args: + - "-c" + - "print(r'{{ Param.TooBig }}')" diff --git a/conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.1--int64-overflow-task-range.invalid.yaml b/conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.1--int64-overflow-task-range.invalid.yaml new file mode 100644 index 0000000..2bef7fe --- /dev/null +++ b/conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.1--int64-overflow-task-range.invalid.yaml @@ -0,0 +1,22 @@ +# The int type is a 64-bit signed integer (Expression Language §1.2.1; +# RFC 0005 "64-bit Signed Integer Type", dropped from the published form). +# A task parameter range element of 2^63 = 9223372036854775808 is out of +# range and must be rejected at template validation. +specificationVersion: jobtemplate-2023-09 +extensions: +- EXPR +name: TestJob +steps: +- name: Step1 + parameterSpace: + taskParameterDefinitions: + - name: TooBig + type: INT + range: "9223372036854775808" + script: + actions: + onRun: + command: python + args: + - "-c" + - "print(r'{{ Task.Param.TooBig }}')" diff --git a/conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.3--membership-element-type-mismatch.invalid.yaml b/conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.3--membership-element-type-mismatch.invalid.yaml new file mode 100644 index 0000000..3e099d4 --- /dev/null +++ b/conformance-tests/2023-09/EXPR/job_templates/proposed/expr2.1.3--membership-element-type-mismatch.invalid.yaml @@ -0,0 +1,17 @@ +# List membership is __contains__(list: list[T], item: T) — the item type +# must match the list's element type (Expression Language §2.1.3). There is +# no signature for a string item against list[int], so '"a" in [1, 2]' must +# be rejected rather than evaluating to false. +specificationVersion: jobtemplate-2023-09 +extensions: +- EXPR +name: TestJob +steps: +- name: Step1 + script: + actions: + onRun: + command: python + args: + - -c + - print(r'{{ "a" in [1, 2] }}') diff --git a/conformance-tests/2023-09/EXPR/jobs/proposed/README-expr-lang.md b/conformance-tests/2023-09/EXPR/jobs/proposed/README-expr-lang.md new file mode 100644 index 0000000..3b8201c --- /dev/null +++ b/conformance-tests/2023-09/EXPR/jobs/proposed/README-expr-lang.md @@ -0,0 +1,25 @@ +# Proposed EXPR fixtures (expression language) — jobs + +Kind-level parking (`//proposed/`); promotion is a move up +one directory. The runner does not scan `proposed/`. Named per fixture +family because other expected-failures PRs park fixtures here with their +own READMEs. + +### `expr2.1.1--int64-max-task-range.test.yaml` + +- Requires: task parameter range element 2^63−1 = 9223372036854775807 + **accepted** (Expression Language §1.2.1: int covers −2⁶³ to 2⁶³−1). +- Observed: **rs rejects** at template validation — `INT parameter 'Big' + range expression error: Integer overflow: result is outside the 64-bit + signed range`. Probing shows single-element ranges are accepted at + 4611686018427387903 (2^62−1) and rejected from 4611686018427387904 (2^62) + upward, suggesting an internal computation (e.g. a length or midpoint + calculation) overflows before the value itself is range-checked. + **py passes** (accepts the valid value). +- Classification: openjd-rs implementation bug (false reject inside the + valid int64 domain). Fixing it widens acceptance only. Promote together + with `../job_templates/proposed/expr2.1.1--int64-overflow-task-range` — + until this accept twin is green on rs, that reject twin passes on rs for + possibly the wrong reason (the same 2^62 cap). The job-parameter accept + twins at 2^63−1 are `expr2.1.1--int64-max-param-values.test.yaml` in the + expr-lang-gaps PR. diff --git a/conformance-tests/2023-09/EXPR/jobs/proposed/expr2.1.1--int64-max-task-range.test.yaml b/conformance-tests/2023-09/EXPR/jobs/proposed/expr2.1.1--int64-max-task-range.test.yaml new file mode 100644 index 0000000..12f165d --- /dev/null +++ b/conformance-tests/2023-09/EXPR/jobs/proposed/expr2.1.1--int64-max-task-range.test.yaml @@ -0,0 +1,27 @@ +# The int type is a 64-bit signed integer (Expression Language §1.2.1), so +# 2^63-1 = 9223372036854775807 must be accepted as a task parameter range +# element. This is the accept twin of expr2.1.1--int64-overflow-task-range; +# the job-parameter accept twins live in +# ../jobs/expr2.1.1--int64-max-param-values.test.yaml. +template: + specificationVersion: jobtemplate-2023-09 + extensions: + - EXPR + name: TestJob + steps: + - name: Step1 + parameterSpace: + taskParameterDefinitions: + - name: Big + type: INT + range: "9223372036854775807" + script: + actions: + onRun: + command: python + args: + - -c + - "print(r'TASK:{{ Task.Param.Big }}')" +expected: + output: + - "TASK:9223372036854775807"