Skip to content

test: [expected failures] EXPR fixtures needing implementation fixes - #164

Draft
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:conformance-expr-lang-expected-failures
Draft

test: [expected failures] EXPR fixtures needing implementation fixes#164
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:conformance-expr-lang-expected-failures

Conversation

@leongdl

@leongdl leongdl commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Important

Expected-failure fixtures — do not expect these to pass. Spec-correct fixtures that FAIL against at least one current implementation, parked in EXPR/proposed/ (not discovered by the runner) so merging keeps the suite green. Per-fixture output, spec citation, and classification in proposed/README.md. Promote each fixture unchanged when its fix lands.

Contents (5 fixtures, dual-implementation verified)

  • expr2.1.1--int64-overflow-param-default.invalid — INT param default: 2^63 accepted at check by BOTH implementations. The known live acceptance bug on the template-validation axis. Also blocked on restoring RFC 0005's dropped "64-bit Signed Integer Type" section to the published spec.
  • expr2.1.1--int64-max-task-range.test — openjd-rs FALSELY REJECTS a valid task-range endpoint of 2^63−1 ("Integer overflow"); probing shows acceptance ends at 2^62−1. Python accepts and runs. The spec admits any int64. Rust-side bug, the mirror image of the acceptance bug.
  • expr2.1.1--int64-overflow-task-range.invalid — task-range element 2^63: openjd-rs correctly rejects; the Python CLI accepts. Python-side bug.
  • expr2.1.3--membership-element-type-mismatch.invalid"a" in [1, 2] accepted by openjd-rs (evaluates to false at runtime); Python rejects at validation. §2.1.3 defines membership only as (list[T], T). Rust-side bug.
  • 3.6.1--let-identifier-513.invalid — a 513-char let identifier accepted by BOTH implementations; schema §3.6.1 caps <UserIdentifier> at 512. The 512 accept-twin is green in the companion PR.

Companion to the green-fixture PR from branch conformance-expr-lang-gaps.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5, one attribution fix) — spec: §3.6.1 (UserIdentifier max 512 — an explicit numeric limit; 513 must be rejected). Identifier verified exactly 513 chars; accept twin at 512 in PR #160's 3.6--let-boundary-edges. Risk-free promotion once fixed. Fix: README reports only openjd-rs acceptance; the sweep shows BOTH implementations fail.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §1.2.1 + §3.4.1.1.1 (2^63-1 is a valid int and a valid range element — must be ACCEPTED). Pins openjd-rs's false-reject from 2^62 up (the probing detail in the README is excellent). Fix widens acceptance only — zero customer-breakage direction — and keeps the overflow twin honest (that twin passes for the wrong reason until this is green; promote as a pair). Nits: README cites accept twin ../jobs/expr2.1.1--int64-max-param-values.test.yaml, which lives on PR #160's branch, not here — dangling until #160 merges; state that Python passes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — spec: EL §1.2.1 type-table row (the explicit overflow-is-error text was dropped from the published spec; README honestly gates promotion on restoring it — right call). Nits: sweep shows BOTH implementations fail; README reports only openjd-rs. Near-duplicate of PR #163's base 2.3--int-default-above-int64-max (only extensions: [EXPR] differs) — justified only if EXPR changes numeric handling; say so or dedupe.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS with a REQUIRED README correction (1 reviewer voted NOT-GOOD on this point) — spec: EL §1.2.1 (2^63 task-range element rejected). The README claims this is 'correctly rejected... passes today; parked pending spec restoration' — but the 2026-08-12 sweep shows the PYTHON CLI accepts 2^63 in a task range: there IS a live defect and this is a genuine expected failure. Update the entry (and note openjd-rs's rejection may fire via its over-rejecting 2^62 cap, i.e. the wrong reason — promote together with the int64-max accept twin).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — spec: EL §2.1.3 (contains(list[T], T); no signature for "a" in [1,2], and "a"→int coercion is defined to error) — the error mandate is solid; openjd-rs evaluating it to false is a real bug (matches sweep, rs-only). Nits: (1) the spec doesn't explicitly require the error at TEMPLATE VALIDATION time, which the .invalid fixture demands — a runtime-error implementation would arguably conform yet fail; note the timing assumption in the README. (2) Service risk worth recording: the fix converts silently-false membership tests into hard errors for running workloads — recommend a template scan before service adoption. Accept twin: mainline expr2.1.3--list-ops.

@leongdl

leongdl commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Quorum review (5 independent agents). 5 fixtures + README. Net: 2 GOOD, 3 GOOD-WITH-NITS, and two README corrections required: (1) the int64-overflow-task-range entry claims the fixture 'passes today' — the 2026-08-12 sweep shows the Python CLI ACCEPTS 2^63 in a task range (a live defect), so this is an expected failure, not a spec-blocked pass; (2) several entries report only openjd-rs observations where the sweep shows Python also fails (let-identifier-513, int64-overflow-param-default). Cross-PR blocker: this branch and PR #165 both create EXPR/proposed/README.md with different content — verified add/add merge conflict; whichever lands second conflicts. Standardize the proposed/ placement too (this branch is component-level, #166 is kind-level).

…in proposed/)

Five parked fixtures with dual-implementation observations: the int64 param-default acceptance (both implementations), the openjd-rs 2^62 false-reject on task-range endpoints (Python accepts, spec admits any int64), the Python-only task-range 2^63 acceptance, the untyped membership acceptance in openjd-rs, and the unenforced 513-char let-identifier cap (both). Per-fixture output, spec citation, and classification in proposed/README.md. Companion to conformance-expr-lang-gaps.

Review: quorum-review fixes — kind-level proposed/ placement with a
family-named README (README-expr-lang.md) so co-located expected-failures
PRs cannot add/add-conflict on a shared README path; corrected the
int64-overflow-task-range entry (the Python CLI ACCEPTS 2^63 in a task
range per the 2026-08-12 sweep — a live defect, not a spec-blocked pass,
and rs's rejection may fire via its own 2^62 over-rejection); added
dual-implementation attribution throughout (let-identifier-513 and
int64-overflow-param-default fail BOTH); noted the membership fixture's
validation-time assumption and silent-false-to-hard-error adoption risk;
fixed accept-twin references to name the PR branches they live on.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@leongdl
leongdl force-pushed the conformance-expr-lang-expected-failures branch from 270517e to 53ac70b Compare August 15, 2026 19:26
@leongdl

leongdl commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Quorum-review fixes applied and pushed (rebased onto mainline 3069673):

  • Restructured to kind-level parking (EXPR/job_templates/proposed/ + EXPR/jobs/proposed/) with a family-named README (README-expr-lang.md) — this removes the add/add merge conflict with test: [expected failures] function-library fixtures pending spec decisions #165, which previously created the same EXPR/proposed/README.md path with different content.
  • Corrected the expr2.1.1--int64-overflow-task-range entry (the review's required fix): per the 2026-08-12 sweep the Python CLI ACCEPTS 2^63 in a task range — a live defect, not a "passes today" spec-blocked entry; and openjd-rs's rejection may fire via its own 2^62 over-rejection (wrong reason), so it must be promoted together with the int64-max accept twin.
  • Dual-implementation attribution added throughout: 3.6.1--let-identifier-513 and expr2.1.1--int64-overflow-param-default fail BOTH implementations.
  • expr2.1.3--membership-element-type-mismatch: documented the validation-time-vs-runtime-error assumption and the silent-false→hard-error adoption risk.
  • Accept-twin references now name the PR branches they live on (no more dangling ../jobs/ pointers).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant