From 42efcb7148d75981031490c537372ecf5bfee6ee Mon Sep 17 00:00:00 2001 From: "John William Creighton (s243a)" Date: Fri, 11 Sep 2026 17:31:23 -0600 Subject: [PATCH 1/5] plawk: compile END-only programs (no main rule) `END { print "done" }` and every other END-only form declined (exit 3) -- the simplest awk one-liners, a real surprise for new users. Now they compile and match gawk: constant/NR/field/NF/length/arithmetic print, printf, multi-statement END lists, END-`if`, and the empty program `''` (which reads input and prints nothing, as gawk does). An END-only program is NOT "skip the loop": an END block sees NR (record count), $0/$N (LAST record), NF and length, so the record loop still runs -- consuming all stdin, counting NR, retaining the last record -- with only the per-record RULE CHAIN empty. On empty input NR is 0, $0 empty, NF 0. So it is the existing rules+END driver with an empty rule chain, not the BEGIN-only driver (which reads no input). Three coordinated changes, each gated on empty rules so no program with rules is touched (20/20 golden-corpus programs byte-identical): 1. plawk_scalar_state_plan/3 admits an empty plan when Rules == [] -- its guard otherwise requires some action / print-field / getline, none of which a rule-less program has. 2. a dedicated plawk_scalar_rule_chain_ir([], ...) clause emits `br label %continue_loop`, the terminator the empty `lowered_match:` block needs. Without it the general clause emits an unterminated block -- invalid LLVM, an exit-4 clang failure. The general clause keeps its `RuleCount > 0`. 3. the END-loop driver clause is guarded `Rules0 \== []`. Change 3 is the sharp one and the reason for a full exit-4 sweep. Relaxing the state-plan guard (change 1) made `END { while ... }` -- which the END-loop clause handles, cutting as soon as it sees a loop -- newly pass state_plan, commit past its cut, and emit a malformed loop driver: exit 4, a MISCOMPILE, on a program that used to decline cleanly. The guard keeps END-loop-with-no-rule declining (exit 3) as the follow-on it is. Lesson recorded in the handoff: relaxing a shared gate can convert a clause that declined-by-relying-on-that-gate-failing into one that commits and miscompiles; enumerate every such clause and sweep the whole surface for exit 4. Scope: END-only print / printf / if / empty-program compile and match gawk. Still declining cleanly (pinned, exit 3, never exit 4): END-only loop, END-only scalar assignment, for-in, getline, record builtins (substr($0,..)/toupper($1)). Parse-level (exit 2): BEGIN+assignment+END, END assoc assignment. Tests: tests/test_plawk_end_only.pl (29) -- the target matrix incl. empty input, the retained-record composition, the empty program, every boundary decline, an explicit "END-loop declines exit 3 not exit 4" pin, and an all-forms no-miscompile sweep. Two stale pins in test_plawk_end_if_nr.pl and test_plawk_end_field_reads.pl that asserted END-only declines are re-attributed (not deleted) to assert the status flip, output parity covered by the new suite. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01QrQistdoYMNpVtecwims6g --- docs/design/PLAWK_AWK_FEATURE_AUDIT.md | 2 +- docs/design/PLAWK_CAMPAIGN_HANDOFF.md | 23 +- .../codegen/llvm/plawk_native_codegen.pl | 26 ++ tests/test_plawk_end_field_reads.pl | 10 +- tests/test_plawk_end_if_nr.pl | 11 +- tests/test_plawk_end_only.pl | 299 ++++++++++++++++++ 6 files changed, 362 insertions(+), 9 deletions(-) create mode 100644 tests/test_plawk_end_only.pl diff --git a/docs/design/PLAWK_AWK_FEATURE_AUDIT.md b/docs/design/PLAWK_AWK_FEATURE_AUDIT.md index 5ee541b1d..dac56f775 100644 --- a/docs/design/PLAWK_AWK_FEATURE_AUDIT.md +++ b/docs/design/PLAWK_AWK_FEATURE_AUDIT.md @@ -17,7 +17,7 @@ only (runtime pending) · ❌ missing. | Feature | Status | Notes | |---|---|---| -| `BEGIN` / `END` | ✅ | incl. constant `print` (BEGIN/END literal print). **`$0` / `$N` in END now read the LAST RECORD** (`END { print $1 }` → `c` on `a 1/b 2/c 3`, like gawk), in a straight-line END print, a statement list (`END { print $1; print $2 }`) and a concatenation (`print $1 " / " $2`). The last record is *gone* by END — at `end_print` the transient buffer holds the `end_of_file` sentinel — so retention is explicit: the record loop copies each record into a reused, geometrically grown buffer (`@plawk_lastrec_store`, the same shape as `@wam_rt_set`, so **constant memory and one memcpy per record**; interning per record would grow the atom table with every distinct record), and END re-materialises those bytes in the shared transient buffer and hands the reserved transient atom id to `llvm_emit_atom_field_slice/5` — the *same* slicer every in-loop field read uses. Honours FS/OFS/ORS/RS; `$N` past NF is empty; empty input gives an empty `$0`. **Pay-per-use**: the globals and defines are emitted as program-level IR only when `plawk_end_term_mentions_field/1` (the #4100 END-loop safety gate, *inverted* rather than restated) fires, so every program without an END field read is byte-identical (15/15 golden corpus). **Also in an END `if` branch** (`END { if (n == 3) print $1 }` — which had been silently printing `end_of_file`, a pre-existing wrong output surfaced and fixed by this line) **and an END loop body** (`while` / `do-while` / C-`for`, nested, inside an `if` inside a loop, with `break` — flipping #4100's gate). Those two needed no emitter parameterisation: `plawk_end_lastrec_rewrite/2` rewrites `field(N)` → `end_lastrec_field(N)` in the END actions (a structural walk, matching the gate) and **two clauses** on the shared `plawk_emit_print_expr_for_context/6` know the record source, so the rule-body print emitter and `plawk_scalar_action_sequence_pairs//15` are untouched. The rewrite covers conditions too, which is fail-safe: no condition emitter has a clause for `end_lastrec_field(_)`, so `END { if ($1 == "c") … }` declines rather than miscompiling. **`NF` in END counts the retained record** (`END { print NF }` → 2), in all three contexts — and in the `if`-branch and loop-body ones it had been printing **1**, the field count of the `end_of_file` sentinel, since those drivers existed: the gate was named `plawk_end_term_mentions_field/1` and matched `field(_)` only, so `NF` walked past the gate built to catch record reads. It is now `plawk_end_term_reads_record/1`, named for the property, matching `field(_)`, `special('NF')` and `special(length)` (`NR`/`RT` are process state and stay out). **`printf` arguments too** — `printf "%s\n", $1`, `printf "%s\n", $0`, `printf "%d\n", NF`, and mixes with scalars — producing the same `string_ptr` / `slice_len`+`slice_ptr` / `i64` call-argument vocabulary a record-context printf produces, so the format rewriter needed no new cases. **`length` / `length($N)` in END counts the retained record too**, in every route at once — a straight-line print, a concatenation (`print "L=" length`), a statement list, the mixed and assoc routes, a `printf` argument, an `if` branch and a loop body. It cost ONE clause, because the change that carried it first collapsed the duplication that had been making every previous cell cost one per route. Two collapses: (a) `NF` had an `end_lastrec_nf` expression row that was its in-loop `nf` row with `%line` swapped for the retained record Value — `length` had no such row, which is the whole reason every END form of it declined — so both became entries in `plawk_record_i64_read/5`, ONE table of record-reading i64 leaves parameterised on *which record they read*, with ONE retained-record wrapper (`end_lastrec_read(Kind)`) covering every entry, and `plawk_end_lastrec_nf_lines//2` generalised to `plawk_end_lastrec_i64_lines//4` with a NameBase parameter so NF keeps its own temporaries and its IR stays byte-identical; (b) the THREE END print walkers each enumerated the print-field vocabulary as clause heads, and each clause was three steps — separator, one per-kind call, recurse — around exactly the call `plawk_end_field_print_lines/4` already makes for that kind inside a concat, so "what may be printed in END" was FOUR lists with nothing keeping them equal (they had already drifted twice: a string scalar in a concat printed its atom id, and NF reached the routes one at a time). **24 clauses became 13**, each walker keeping only what is genuinely its own — `concat`, the one kind that is not a single value, and for the assoc route the table reads. Two smaller instances of the same shape went with it: `plawk_end_list_printf_arg_ok/2`'s two chains each listed the record-reading printf arguments (they differ only in `var` and `NR`), factored so `length` was one row not two; and `plawk_rule_body_print_field/1`, a THIRD list reached by an END loop body, needed one row. **32 pre-existing golden-corpus programs are byte-identical** across both collapses, which is the check that says a collapse was a collapse and not a rewrite. Bare `length` needed no case of its own: it parses to `length(field(0))` and field 0 measures the whole record, so the awk shorthand is already the arity-1 instance of the general form. One unintended-but-correct side effect, pinned rather than left unremarked: delegating the assoc walker passes the EMPTY scalar plan (as it already did for concat parts), which makes the shared generic expression clause reachable, so `END { print 1 + 2, c["x"] }` now builds and agrees with gawk. Tests: `tests/test_plawk_end_length.pl`. Still declined: a field or `NF` in a loop/`if` **condition**, **builtins over the record** (`substr($0, …)`, `toupper($1)`), an **END-only** program (different driver, no retain), a record read in a **for-in CHAIN** body (`END { for (k in c) print k, NF }` — `NF` and `length` decline there alike, so it is the chain body's print vocabulary rather than either feature), the **associative** END-`if` branch (`plawk_assoc_end_if_branch_prints_ok/2` allows only string literals there — an unrelated pre-existing restriction), and under a **binary descriptor** | +| `BEGIN` / `END` | ✅ | incl. constant `print` (BEGIN/END literal print). **`$0` / `$N` in END now read the LAST RECORD** (`END { print $1 }` → `c` on `a 1/b 2/c 3`, like gawk), in a straight-line END print, a statement list (`END { print $1; print $2 }`) and a concatenation (`print $1 " / " $2`). The last record is *gone* by END — at `end_print` the transient buffer holds the `end_of_file` sentinel — so retention is explicit: the record loop copies each record into a reused, geometrically grown buffer (`@plawk_lastrec_store`, the same shape as `@wam_rt_set`, so **constant memory and one memcpy per record**; interning per record would grow the atom table with every distinct record), and END re-materialises those bytes in the shared transient buffer and hands the reserved transient atom id to `llvm_emit_atom_field_slice/5` — the *same* slicer every in-loop field read uses. Honours FS/OFS/ORS/RS; `$N` past NF is empty; empty input gives an empty `$0`. **Pay-per-use**: the globals and defines are emitted as program-level IR only when `plawk_end_term_mentions_field/1` (the #4100 END-loop safety gate, *inverted* rather than restated) fires, so every program without an END field read is byte-identical (15/15 golden corpus). **Also in an END `if` branch** (`END { if (n == 3) print $1 }` — which had been silently printing `end_of_file`, a pre-existing wrong output surfaced and fixed by this line) **and an END loop body** (`while` / `do-while` / C-`for`, nested, inside an `if` inside a loop, with `break` — flipping #4100's gate). Those two needed no emitter parameterisation: `plawk_end_lastrec_rewrite/2` rewrites `field(N)` → `end_lastrec_field(N)` in the END actions (a structural walk, matching the gate) and **two clauses** on the shared `plawk_emit_print_expr_for_context/6` know the record source, so the rule-body print emitter and `plawk_scalar_action_sequence_pairs//15` are untouched. The rewrite covers conditions too, which is fail-safe: no condition emitter has a clause for `end_lastrec_field(_)`, so `END { if ($1 == "c") … }` declines rather than miscompiling. **`NF` in END counts the retained record** (`END { print NF }` → 2), in all three contexts — and in the `if`-branch and loop-body ones it had been printing **1**, the field count of the `end_of_file` sentinel, since those drivers existed: the gate was named `plawk_end_term_mentions_field/1` and matched `field(_)` only, so `NF` walked past the gate built to catch record reads. It is now `plawk_end_term_reads_record/1`, named for the property, matching `field(_)`, `special('NF')` and `special(length)` (`NR`/`RT` are process state and stay out). **`printf` arguments too** — `printf "%s\n", $1`, `printf "%s\n", $0`, `printf "%d\n", NF`, and mixes with scalars — producing the same `string_ptr` / `slice_len`+`slice_ptr` / `i64` call-argument vocabulary a record-context printf produces, so the format rewriter needed no new cases. **`length` / `length($N)` in END counts the retained record too**, in every route at once — a straight-line print, a concatenation (`print "L=" length`), a statement list, the mixed and assoc routes, a `printf` argument, an `if` branch and a loop body. It cost ONE clause, because the change that carried it first collapsed the duplication that had been making every previous cell cost one per route. Two collapses: (a) `NF` had an `end_lastrec_nf` expression row that was its in-loop `nf` row with `%line` swapped for the retained record Value — `length` had no such row, which is the whole reason every END form of it declined — so both became entries in `plawk_record_i64_read/5`, ONE table of record-reading i64 leaves parameterised on *which record they read*, with ONE retained-record wrapper (`end_lastrec_read(Kind)`) covering every entry, and `plawk_end_lastrec_nf_lines//2` generalised to `plawk_end_lastrec_i64_lines//4` with a NameBase parameter so NF keeps its own temporaries and its IR stays byte-identical; (b) the THREE END print walkers each enumerated the print-field vocabulary as clause heads, and each clause was three steps — separator, one per-kind call, recurse — around exactly the call `plawk_end_field_print_lines/4` already makes for that kind inside a concat, so "what may be printed in END" was FOUR lists with nothing keeping them equal (they had already drifted twice: a string scalar in a concat printed its atom id, and NF reached the routes one at a time). **24 clauses became 13**, each walker keeping only what is genuinely its own — `concat`, the one kind that is not a single value, and for the assoc route the table reads. Two smaller instances of the same shape went with it: `plawk_end_list_printf_arg_ok/2`'s two chains each listed the record-reading printf arguments (they differ only in `var` and `NR`), factored so `length` was one row not two; and `plawk_rule_body_print_field/1`, a THIRD list reached by an END loop body, needed one row. **32 pre-existing golden-corpus programs are byte-identical** across both collapses, which is the check that says a collapse was a collapse and not a rewrite. Bare `length` needed no case of its own: it parses to `length(field(0))` and field 0 measures the whole record, so the awk shorthand is already the arity-1 instance of the general form. One unintended-but-correct side effect, pinned rather than left unremarked: delegating the assoc walker passes the EMPTY scalar plan (as it already did for concat parts), which makes the shared generic expression clause reachable, so `END { print 1 + 2, c["x"] }` now builds and agrees with gawk. Tests: `tests/test_plawk_end_length.pl`. Still declined: a field or `NF` in a loop/`if` **condition**, **builtins over the record** (`substr($0, …)`, `toupper($1)`), ~~an **END-only** program~~ **now compiles** (`END { print ... }`, printf, END-`if`, and the empty program `''` — the rules+END driver admits an empty rule chain via an empty scalar state plan + a dedicated `plawk_scalar_rule_chain_ir([], ...)` clause emitting `br label %continue_loop`; the record loop still runs so NR/$0/$N/NF/length are the last record, byte-identical on 20/20 existing programs; remaining END-only follow-ons are a **loop** `END { while ... }` (pinned as a clean exit-3 decline, guarded so it never miscompiles), a scalar **assignment**, and for-in/getline; tests `tests/test_plawk_end_only.pl` 29), a record read in a **for-in CHAIN** body (`END { for (k in c) print k, NF }` — `NF` and `length` decline there alike, so it is the chain body's print vocabulary rather than either feature), the **associative** END-`if` branch (`plawk_assoc_end_if_branch_prints_ok/2` allows only string literals there — an unrelated pre-existing restriction), and under a **binary descriptor** | | `/regex/` | ✅ | a bare `/re/` pattern matches the whole record with **full POSIX ERE**: a body with metacharacters lowers to `field_match(0, Regex)` and runs through the same regex engine as `~`/`!~` (`/foo.*bar/`, `/^(err\|warn):/`, character classes, alternation, groups). A metachar-free body keeps its fast native lowering (`^prefix` → `prefix/1`, all-literal → `contains/1`); `\/` is a literal slash | | `$N == "v"`, `$3 > 100` | ✅ | field-equality + numeric field guards; **string inequality `$N != "v"`** now parses too (the skip-a-value idiom `$1 != "header"`) — rewritten to `not_pat(field_eq(N,"v"))`, so it reuses the `==` guard codegen and composes with `!`/`&&`/`||`, `if` conditions, and for-in-END group-by. Numeric `$N != K` keeps the integer path. **Field string ordering `$N < "v"` / `<=` / `>` / `>=` landed too** — awk compares a field against a string constant lexically (never numeric, so `$1 < "10"` is byte-wise while `$1 < 10` stays the numeric path); lowered to a memcmp of the field slice against the interned literal (`@wam_atom_field_str_cmp_value`), composing with `!`/`&&`/`||`, `if` conditions, and for-in-END group-by | | `&& \|\| !` combinators | ✅ | awk precedence, parens, single-block lowering | diff --git a/docs/design/PLAWK_CAMPAIGN_HANDOFF.md b/docs/design/PLAWK_CAMPAIGN_HANDOFF.md index 4bbd261ad..08611dbf3 100644 --- a/docs/design/PLAWK_CAMPAIGN_HANDOFF.md +++ b/docs/design/PLAWK_CAMPAIGN_HANDOFF.md @@ -367,6 +367,21 @@ parse, not the gate you expected to fire. ## Verification practices (do not skip) +- **Relaxing a shared gate can turn a clean decline into a MISCOMPILE in a + neighbouring driver that relied on that gate failing.** END-only support needed + `plawk_scalar_state_plan/3` to admit an empty plan (`; Rules == []`). That made + `END { print ... }` compile — but the END-LOOP driver clause (`END { while ... }`) + cuts as soon as it sees a loop, then had been relying on `state_plan` *failing* for + a rule-less program to decline after the cut. With the guard relaxed, state_plan + succeeded, the clause committed past its cut, and it emitted a malformed loop + driver: exit 4, a miscompile, on a program that used to decline cleanly at exit 3. + The fix was a targeted `Rules0 \== []` guard on that clause so END-loop-with-no-rule + keeps declining. **When you relax a gate, enumerate every clause that used that + gate's failure as its own decline mechanism** — a cut-then-rely-on-a-later-goal- + failing clause is silently converted from "declines" to "commits and may miscompile". + A broad exit-4 sweep of the whole surface the relaxation touches is mandatory, not + optional: here it found exactly one such site, and the campaign's worst outcome + (invalid LLVM on a supported-looking program) was one un-run probe away. - **gawk 5.2 is the oracle.** Compare output *and* exit status. Probe harness pattern: write the program, `rm -f` the binary first (a declining build must not run a stale one), build, run, diff against `gawk`. @@ -690,8 +705,12 @@ condition emitter has a clause for `end_lastrec_field(_)` / `end_lastrec_nf`) · at parse time (#4169). The two share a spelling and nothing else: one needs the retained record, the other needs no runtime at all. · **builtins over the record** in END (`substr($0, …)`, `toupper($1)` — the gate retains for them, the -emitters have no clause; the *literal* forms of all five are done) · **END-only** -programs (a driver with no retain) · +emitters have no clause; the *literal* forms of all five are done) · ~~**END-only** +programs~~ **DONE** (see below; `END { print ... }`, printf, END-`if`, and the empty +program now compile — remaining END-only follow-ons are an END-only **loop** +(`END { while ... }`, pinned as a clean exit-3 decline, NOT a miscompile), an +END-only scalar **assignment** (`END { x = 5; print x }`), and for-in / getline in +END-only) · `printf` field args in the **assoc / mixed END chain** (a different driver, passes `no_end_record`) · the **associative** END-`if` branch (refused by `plawk_assoc_end_if_branch_prints_ok/2`, which allows only string literals there — diff --git a/examples/plawk/codegen/llvm/plawk_native_codegen.pl b/examples/plawk/codegen/llvm/plawk_native_codegen.pl index d3faece9f..e12aafd52 100644 --- a/examples/plawk/codegen/llvm/plawk_native_codegen.pl +++ b/examples/plawk/codegen/llvm/plawk_native_codegen.pl @@ -1061,6 +1061,15 @@ DriverIR ) :- plawk_end_actions_have_loop(EndActions), + % A loop in the END block is only supported alongside a main rule. END-only + % support (an empty scalar state plan, added for `END { print ... }`) makes + % plawk_scalar_state_plan/3 succeed for Rules0 = [], which would otherwise let + % this clause commit past its cut and emit a malformed loop driver -- exit 4, + % a miscompile -- for `END { while (...) ... }`. Require a main rule so that + % END-loop-with-no-rule declines cleanly (exit 3) as the follow-on it is, + % rather than compiling to invalid LLVM. END-only print / if is unaffected + % (those clauses are 788 / 851 / 920 / 980). + Rules0 \== [], !, plawk_end_loop_print_fields(EndActions, PrintFields), plawk_resolve_writebin_rules(BeginClauses, Rules0, Rules1, WritebinPlan), @@ -8090,6 +8099,12 @@ ; BodyPrintFields \== [] ; plawk_rules_have_record_getline(Rules) ; plawk_rules_have_main_getline(Rules) + % A rule-less program (END-only, or the empty program) has an empty scalar + % state plan and must still reach the driver: with no rules, none of the + % disjuncts above can fire, so without this the guard rejects exactly the + % programs whose slots legitimately come only from the END print fields (or + % from nothing at all). state_plan([], []) is the correct plan here. + ; Rules == [] ), findall(Name, ( member(Field, PrintFields), @@ -14898,6 +14913,17 @@ plawk_end_scalar_expr(Expr), plawk_expr_scalar_read_name(Expr, Name). +% END-only / rule-less program: the record loop still runs (an END block reads all +% input to fix NR / NF / $0), but there is no per-record rule chain. The general +% clause below cannot serve this: with zero rules it emits an EMPTY `lowered_match:` +% block, and the entry branch into the chain (`br label %rule_0_match`) that would +% normally terminate that block is part of the chain IR, so an empty chain leaves +% `lowered_match:` with no terminator -- invalid LLVM (clang: expected instruction +% opcode). Emit the loop-continue branch instead: with nothing to match, the match +% block jumps straight to `continue_loop`. RuleCount 0, no globals, no branch exits. +plawk_scalar_rule_chain_ir([], _StatePlan, _FieldSeparator, _OutputSeparator, + '', ' br label %continue_loop', 0, []) :- + !. plawk_scalar_rule_chain_ir(Rules, StatePlan, FieldSeparator, OutputSeparator, GlobalIR, ChainIR, RuleCount, BranchNextExits) :- plawk_scalar_planned_rules(Rules, PlannedRules, Controls), diff --git a/tests/test_plawk_end_field_reads.pl b/tests/test_plawk_end_field_reads.pl index 88d1491d4..f69ac3f58 100644 --- a/tests/test_plawk_end_field_reads.pl +++ b/tests/test_plawk_end_field_reads.pl @@ -610,9 +610,13 @@ build_status("{ n++ } END { print toupper($1) }\n", 3), !. -% An END-only program (no rules) uses a different driver, which does not retain. -test(end_only_program_field_read_declines) :- - build_status("END { print $1 }\n", 3), +% An END-only program (no rules) now COMPILES and retains the last record -- the +% END-only driver landed. `END { print $1 }` builds and prints the last record's $1. +% Pinned as a decline while END-only was a gap; re-attributed, not deleted. Output +% parity is in tests/test_plawk_end_only.pl (field_and_nf_of_last_record); here we +% pin the status flip in the suite that first pinned this shape. +test(end_only_program_field_read_now_compiles) :- + build_status("END { print $1 }\n", 0), !. % --- regressions: END without fields is unchanged ------------------------ diff --git a/tests/test_plawk_end_if_nr.pl b/tests/test_plawk_end_if_nr.pl index 183fe43a8..3fac57695 100644 --- a/tests/test_plawk_end_if_nr.pl +++ b/tests/test_plawk_end_if_nr.pl @@ -121,9 +121,14 @@ build_status("{ n++ } END { if (3 == NR) print \"yes\" }\n", 3), !. -% END-only programs (no rule) decline entirely -- separate driver gap. -test(end_only_nr_if_declines) :- - build_status("END { if (NR == 3) print \"yes\" }\n", 3), +% END-only programs now COMPILE -- the END-only driver landed. `END { if (NR == 3) ... }` +% builds and matches gawk (prints "yes" on a 3-record input). This was pinned as a +% decline while END-only was a gap; re-attributed, not deleted, when the gap closed. +% Output parity is covered in tests/test_plawk_end_only.pl (end_if_no_rule); here we +% pin the status flip so a regression that re-broke the END-only driver shows up in +% the suite that first pinned this shape. +test(end_only_nr_if_now_compiles) :- + build_status("END { if (NR == 3) print \"yes\" }\n", 0), !. % Assoc rules plus this scalar END-if select no driver: the assoc END-if route admits diff --git a/tests/test_plawk_end_only.pl b/tests/test_plawk_end_only.pl new file mode 100644 index 000000000..c7af5fa31 --- /dev/null +++ b/tests/test_plawk_end_only.pl @@ -0,0 +1,299 @@ +:- encoding(utf8). +% SPDX-License-Identifier: MIT OR Apache-2.0 +% Copyright (c) 2026 John William Creighton (@s243a) +% +% END-only programs: a program whose only action block is `END { ... }`, with no +% main rule. `END { print "done" }` and the like used to decline (exit 3) for every +% form; new users hit this on the simplest possible awk one-liners. +% +% --------------------------------------------------------------------------- +% AN END-ONLY PROGRAM STILL READS ALL INPUT +% +% The subtle part, and the reason this is not "skip the loop": an END block sees +% NR (total record count), $0/$N (the LAST record), NF and length -- so the record +% loop must still run, consuming all of stdin, counting NR and retaining the last +% record, with only the per-record RULE CHAIN empty. On empty input NR is 0, $0 is +% empty, NF is 0. gawk 5.x is the oracle for all of this. +% +% This is why an END-only program is the existing rules+END driver with an empty +% rule chain, not the BEGIN-only driver (which reads NO input). The fix was three +% coordinated pieces, each gated on the rules being empty so no program with rules +% is touched (20/20 golden-corpus programs byte-identical): +% +% 1. plawk_scalar_state_plan/3 admits an empty plan when Rules == [] (its guard +% otherwise requires some action / print / getline, none of which a rule-less +% program has). +% 2. a dedicated plawk_scalar_rule_chain_ir([], ...) clause emits +% `br label %continue_loop` -- the terminator the empty `lowered_match:` block +% needs. Without it the general clause emits an unterminated block: invalid +% LLVM, an exit-4 clang failure. The general clause keeps its `RuleCount > 0`. +% 3. the END-loop driver clause is guarded `Rules0 \== []`, so `END { while ... }` +% with no rule DECLINES cleanly (exit 3) instead of committing past its cut and +% miscompiling -- END-loop-with-no-rule is a follow-on, not part of this change. +% +% The distinction that matters most below: a supported END-only form must COMPILE +% and match gawk; an unsupported one must DECLINE cleanly (exit 3), never exit 4. + +:- use_module(library(plunit)). +:- use_module(library(process)). +:- use_module(library(filesex), [make_directory_path/1]). +:- use_module('../examples/plawk/parser/plawk_parser'). +:- use_module('../examples/plawk/codegen/llvm/plawk_native_codegen'). + +clang_available :- + catch(( process_create(path(clang), ['--version'], + [stdout(null), stderr(null), process(Pid)]), + process_wait(Pid, exit(0)) ), _, fail). + +% Three records; the LAST is "7 disk" (NR 3, $1 7, $2 disk, $0 "7 disk", NF 2, +% length 6). The last record differs from the first so a test cannot pass by +% reading the wrong record. +input("5 boot\n5 trace\n7 disk\n"). + +:- begin_tests(plawk_end_only). + +% --- the simplest END-only programs, which used to decline -------------- + +test(constant_print, [condition(clang_available)]) :- + run("END { print \"done\" }\n", "done\n"), + !. + +test(two_constants, [condition(clang_available)]) :- + run("END { print \"a\", \"b\" }\n", "a b\n"), + !. + +test(arithmetic, [condition(clang_available)]) :- + run("END { print 1 + 2 }\n", "3\n"), + !, + run("END { print 3 % 2 }\n", "1\n"), + !, + run("END { print -1 }\n", "-1\n"), + !. + +% --- NR: the record count, which needs the loop to have run -------------- + +test(nr_is_the_record_count, [condition(clang_available)]) :- + run("END { print NR }\n", "3\n"), + !. + +test(nr_on_empty_input_is_zero, [condition(clang_available)]) :- + run_with("", "END { print NR }\n", "0\n"), + !. + +test(nr_in_a_concat, [condition(clang_available)]) :- + run("END { print \"n=\" NR }\n", "n=3\n"), + !. + +% --- the retained last record: $0 / $N / NF / length -------------------- +% +% These are the reason an END-only program is not "skip the loop": the record loop +% must retain the last record for END to project from it. That capability already +% existed (END field reads); this confirms it composes with an empty rule chain. + +test(whole_last_record, [condition(clang_available)]) :- + run("END { print $0 }\n", "7 disk\n"), + !. + +test(field_and_nf_of_last_record, [condition(clang_available)]) :- + run("END { print $1, NF }\n", "7 2\n"), + !. + +test(length_of_last_record, [condition(clang_available)]) :- + run("END { print length }\n", "6\n"), + !, + run("END { print length($0) }\n", "6\n"), + !. + +test(length_on_empty_input_is_zero, [condition(clang_available)]) :- + run_with("", "END { print length }\n", "0\n"), + !. + +test(nr_field_and_nf_together, [condition(clang_available)]) :- + run("END { print NR, $1, NF }\n", "3 7 2\n"), + !. + +% bare `print` in END is `print $0` of the last record. +test(bare_print_is_whole_last_record, [condition(clang_available)]) :- + run("END { print }\n", "7 disk\n"), + !. + +% --- printf, and the multi-statement END list (clause 851) -------------- + +test(printf_in_end_only, [condition(clang_available)]) :- + run("END { printf \"%d\\n\", NR }\n", "3\n"), + !, + run("END { printf \"%s\\n\", $1 }\n", "7\n"), + !. + +test(multi_statement_end, [condition(clang_available)]) :- + run("END { print \"a\"; print \"b\" }\n", "a\nb\n"), + !, + run("END { print NR; print $1 }\n", "3\n7\n"), + !. + +% --- END-if with no rule (came along on the same state-plan path) ------- + +test(end_if_no_rule, [condition(clang_available)]) :- + run("END { if (NR == 3) print \"x\" }\n", "x\n"), + !. + +test(end_if_else_reads_the_record, [condition(clang_available)]) :- + run("END { if (NR > 1) print $1; else print \"few\" }\n", "7\n"), + !. + +test(end_if_on_empty_input, [condition(clang_available)]) :- + run_with("", "END { if (NR == 0) print \"empty\" }\n", "empty\n"), + !. + +% --- the empty program: reads all input, prints nothing ----------------- +% +% A scope expansion that fell out of admitting an empty rule chain, and it is +% correct gawk behaviour (`awk ''` drains input, prints nothing, exits 0). Pinned +% as intended rather than left unremarked. + +test(empty_program_reads_input_prints_nothing, [condition(clang_available)]) :- + run("\n", ""), + !, + run_with("", "\n", ""), + !. + +% --- REGRESSIONS: programs WITH rules are untouched --------------------- + +test(rules_with_end_unchanged, [condition(clang_available)]) :- + run("{ n++ } END { print n }\n", "3\n"), + !, + run("{ n++; c[$1]++ } END { print NR, c[\"5\"] }\n", "3 2\n"), + !. + +test(begin_only_unchanged, [condition(clang_available)]) :- + run("BEGIN { print \"hi\" }\n", "hi\n"), + !. + +test(rule_only_unchanged, [condition(clang_available)]) :- + run("{ print $1 }\n", "5\n5\n7\n"), + !. + +% END-loop WITH a rule still compiles -- the guard is on empty rules only. +test(end_loop_with_a_rule_still_compiles, [condition(clang_available)]) :- + build_status("{ n++ } END { while (i < n) { print i; i++ } }\n", 0), + !. + +% --- BOUNDARIES: unsupported END-only forms DECLINE, never miscompile --- + +% The miscompile that this change had to avoid: END-loop with no rule. It must +% decline cleanly (exit 3), NOT emit invalid LLVM (exit 4). This is the sharpest +% pin in the suite -- the whole END-loop clause guard exists for it. +test(end_only_loop_declines_cleanly_not_exit_4) :- + build_status("END { while (i < 3) { print i; i++ } }\n", 3), + !. + +% END-only with a scalar ASSIGNMENT is a follow-on (the assignment machinery is +% not wired for an END-only context yet). Declines, exit 3. +test(end_only_assignment_declines) :- + build_status("END { x = 5; print x }\n", 3), + !. + +% for-in over an (empty) array in END-only: declines, exit 3. +test(end_only_forin_declines) :- + build_status("END { for (k in a) print k }\n", 3), + !. + +% builtins over the retained record in END are a separate follow-on (the LITERAL +% forms fold; these read the record). Decline, exit 3. +test(end_only_record_builtins_decline) :- + build_status("END { print substr($0, 1, 3) }\n", 3), + !, + build_status("END { print toupper($1) }\n", 3), + !. + +% getline in an END block is unsupported: declines, exit 3. +test(end_only_getline_declines) :- + build_status("END { getline x; print x }\n", 3), + !. + +% --- clang never rejects a supported END-only program ------------------- +% +% Belt-and-braces over the whole supported surface: build each and require a +% binary. An exit-4 here would mean invalid LLVM slipped through; that is the +% failure the dedicated rule-chain clause exists to prevent. +test(no_supported_end_only_form_miscompiles, [condition(clang_available)]) :- + forall(member(Src, [ + "END { print \"done\" }\n", + "END { print NR }\n", + "END { print $0 }\n", + "END { print $1, NF }\n", + "END { print length }\n", + "END { printf \"%d\\n\", NR }\n", + "END { print \"a\"; print \"b\" }\n", + "END { if (NR == 3) print $1; else print NF }\n" + ]), build_status(Src, 0)), + !. + +% --- the empty rule chain lowers to a single loop-continue branch ------- +% +% The IR property that keeps the fix additive: with no rules, the record loop's +% match block is just a branch to continue_loop. Pinned on the emitted IR so a +% future change that reintroduces per-rule scaffolding here shows up. +test(empty_rule_chain_is_a_continue_branch) :- + build_ll("END { print \"done\" }\n", LL), + assertion(sub_string(LL, _, _, _, "br label %continue_loop")), + % ...and no rule-match scaffolding, since there are no rules. + assertion(\+ sub_string(LL, _, _, _, "rule_0_match")), + !. + +:- end_tests(plawk_end_only). + +% --- helpers --------------------------------------------------------------- + +odir(Dir) :- + current_prolog_flag(tmp_dir, Tmp), + directory_file_path(Tmp, 'uw_plawk_end_only', Dir), + ( exists_directory(Dir) -> true ; make_directory_path(Dir) ). + +run(Src, Expected) :- + input(Input), + run_with(Input, Src, Expected). + +run_with(Input, Src, Expected) :- + odir(Dir), + directory_file_path(Dir, 'eo_bin', Bin), + ( exists_file(Bin) -> delete_file(Bin) ; true ), + directory_file_path(Dir, 'eo', Prog0), + atom_concat(Prog0, '.plawk', Prog), + setup_call_cleanup(open(Prog, write, S, [encoding(utf8)]), + write(S, Src), close(S)), + atom_concat(Prog0, '_in.txt', In), + setup_call_cleanup(open(In, write, SI, [encoding(utf8)]), + write(SI, Input), close(SI)), + cli([build, Prog, '-o', Bin], 0), + process_create(Bin, [In], [stdout(pipe(PS)), stderr(std), process(Pid)]), + read_string(PS, _, Out), + close(PS), + process_wait(Pid, exit(0)), + ( Out == Expected + -> true + ; format(user_error, "~n~w~n got ~q~n expected ~q~n", + [Src, Out, Expected]), fail + ). + +build_status(Src, ExpectedStatus) :- + odir(Dir), + directory_file_path(Dir, 'eo_reject', Prog0), + atom_concat(Prog0, '.plawk', Prog), + setup_call_cleanup(open(Prog, write, S, [encoding(utf8)]), + write(S, Src), close(S)), + atom_concat(Prog0, '_bin', Bin), + ( exists_file(Bin) -> delete_file(Bin) ; true ), + cli([build, Prog, '-o', Bin], ExpectedStatus). + +build_ll(Src, LL) :- + plawk_parse_string(Src, Program), + plawk_program_native_driver_ir(Program, 'input.txt', IR), + atom_string(IR, LL). + +cli(Args, ExpectedStatus) :- + process_create(path(swipl), ['examples/plawk/bin/plawk' | Args], + [stdout(pipe(S)), stderr(std), process(Pid)]), + read_string(S, _, _), close(S), + process_wait(Pid, exit(Status)), + assertion(Status == ExpectedStatus). From 7891f31242c5a7fdd58e99e4596f508133339f7a Mon Sep 17 00:00:00 2001 From: "John William Creighton (s243a)" Date: Fri, 11 Sep 2026 20:46:38 -0600 Subject: [PATCH 2/5] plawk: pin the rule-less clause enumeration (no shape miscompiles) Strengthens the END-only suite with an explicit enumeration test: one rule-less program routed at each END-driver clause type -- scalar print/list/if, loop, the MIXED and ASSOC ends whose RuleCount>0 gates were deliberately NOT relaxed, BEGIN+end, getline and assignment -- each asserted to build to a real exit code (0/2/3) and NEVER 4. The load-bearing cases are the mixed and assoc rule-less shapes (`END { print c["x"] }`, `END { for (k in c) print k }`, `END { c["x"]++ }`): they confirm the two unrelaxed gates keep those clauses DECLINING on empty rules rather than committing past a cut and miscompiling -- the exact "did you enumerate every clause that relied on the shared gate failing" question the state-plan relaxation raises. All twelve build without exit 4. Complements the shape-level no-miscompile sweep already in the suite with a clause-level one. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01QrQistdoYMNpVtecwims6g --- tests/test_plawk_end_only.pl | 47 ++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/tests/test_plawk_end_only.pl b/tests/test_plawk_end_only.pl index c7af5fa31..06cd9ac20 100644 --- a/tests/test_plawk_end_only.pl +++ b/tests/test_plawk_end_only.pl @@ -229,6 +229,33 @@ ]), build_status(Src, 0)), !. +% --- clause enumeration: NO rule-less shape miscompiles (exit 4) --------- +% +% The sharpest risk in this change: relaxing the shared state-plan guard could let a +% rule-less program COMMIT past a cut in some driver clause other than the ones tested +% above and emit invalid LLVM. This pins the enumeration -- one rule-less program routed +% at each END-driver clause type (scalar print/list/if, loop, MIXED and ASSOC ends whose +% RuleCount>0 gates were deliberately NOT relaxed, BEGIN+end, binfmt, getline/redirect/ +% assignment). Every one must build to a real exit code (0 compile / 2 parse / 3 decline) +% and NEVER 4. The mixed/assoc cases are the load-bearing ones: they confirm the two +% unrelaxed gates keep those clauses declining instead of committing on empty rules. +test(no_rule_less_shape_miscompiles) :- + forall(member(Src, [ + "END { print \"x\" }\n", + "END { print \"a\"; print \"b\" }\n", + "END { if (NR == 3) print \"x\" }\n", + "END { while (i < 3) { print i; i++ } }\n", + "END { print c[\"x\"] }\n", + "END { print c[\"x\"], NR }\n", + "END { for (k in c) print k }\n", + "END { c[\"x\"]++; print c[\"x\"] }\n", + "BEGIN { FS=\":\" } END { print NR }\n", + "BEGIN { print \"start\" } END { print NR }\n", + "END { getline x }\n", + "END { x = 5 }\n" + ]), build_status_not_4(Src)), + !. + % --- the empty rule chain lowers to a single loop-continue branch ------- % % The IR property that keeps the fix additive: with no rules, the record loop's @@ -286,6 +313,26 @@ ( exists_file(Bin) -> delete_file(Bin) ; true ), cli([build, Prog, '-o', Bin], ExpectedStatus). +% Build and assert the exit code is anything but 4 (a clang miscompile). Used by the +% clause-enumeration test: a rule-less shape may compile, parse-fail, or decline, but +% must never produce invalid LLVM. +build_status_not_4(Src) :- + odir(Dir), + directory_file_path(Dir, 'eo_no4', Prog0), + atom_concat(Prog0, '.plawk', Prog), + setup_call_cleanup(open(Prog, write, S, [encoding(utf8)]), + write(S, Src), close(S)), + atom_concat(Prog0, '_bin', Bin), + ( exists_file(Bin) -> delete_file(Bin) ; true ), + process_create(path(swipl), ['examples/plawk/bin/plawk', build, Prog, '-o', Bin], + [stdout(pipe(Out)), stderr(std), process(Pid)]), + read_string(Out, _, _), close(Out), + process_wait(Pid, exit(Status)), + ( Status =\= 4 + -> true + ; format(user_error, "~nMISCOMPILE (exit 4): ~w~n", [Src]), fail + ). + build_ll(Src, LL) :- plawk_parse_string(Src, Program), plawk_program_native_driver_ir(Program, 'input.txt', IR), From e4e4fc28a2393fee733fab26e1ac9031f41432cb Mon Sep 17 00:00:00 2001 From: "John William Creighton (s243a)" Date: Fri, 11 Sep 2026 21:36:00 -0600 Subject: [PATCH 3/5] plawk: END-only with a scalar var declines cleanly (fix review-found exit-4) An ultra review found a blocking miscompile the first cut missed: END-only programs that read/write a scalar VARIABLE -- `END { print x }`, `END { printf "%d\n", x }`, `END { if (x == 0) ... }`, `END { if (NR>0) print x }`, `BEGIN{...} END { print x }` incl. BINFMT -- exited 4 (clang: use of undefined value '%rule_-1_in_slot_0'). Cause: admitting an empty rule chain let plawk_scalar_state_plan collect the scalar as a slot, so the state plan was NOT empty. The next-slot phi emitter then computed LastRuleIndex = RuleCount - 1 = -1 and referenced %rule_-1_match / %rule_-1_in_slot_N -- undefined SSA. My own exit-4 sweep missed it because every print case it tried used a literal / field / NR / NF / length, never a scalar variable; the defect lives in the interaction between an empty rule chain and a NON-empty state plan. Fix: guard the dedicated empty rule-chain clause to fire only when the state plan has NO slots (plawk_state_plan_slots(StatePlan, [])). A scalar-var END-only program therefore declines cleanly (exit 3) instead of miscompiling. Declining (rather than a pass-through phi) is the right scope: an unset scalar's value in END is context-dependent -- empty in string context, 0 in numeric, the uninitialised-scalar representation problem -- so a correct compile needs that settled first. Scalar-var END-only is now a pinned follow-on. Every non-scalar END-only form is unaffected and still compiles + matches gawk (literal/NR/field/NF/length print, printf, multi-statement, END-if, empty program). 20/20 golden-corpus programs remain byte-identical (the guard is inert for rule-bearing programs). The reviewer's entire blocking list now exits 3, zero exit-4 across a broad rule-less re-sweep. Tests: the enumeration test gains the scalar-var forms (the exact blind spot), and a new pin `end_only_scalar_var_read_declines_not_miscompiles` asserts exit 3 (not 4) for five scalar-var shapes, so the miscompile cannot silently return. Suite now 31 tests. Handoff records the lesson: an exit-4 sweep must vary the DATA a relaxation admits (does the END read a literal/field/special/scalar-VAR?), not just program shapes, because the defect can hide in a data-dependent downstream a clause-level enumeration never reaches -- and external review earns its keep precisely when the change already looks polished. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01QrQistdoYMNpVtecwims6g --- docs/design/PLAWK_AWK_FEATURE_AUDIT.md | 2 +- docs/design/PLAWK_CAMPAIGN_HANDOFF.md | 28 +++++++++++++++++-- .../codegen/llvm/plawk_native_codegen.pl | 13 ++++++++- tests/test_plawk_end_only.pl | 27 +++++++++++++++++- 4 files changed, 65 insertions(+), 5 deletions(-) diff --git a/docs/design/PLAWK_AWK_FEATURE_AUDIT.md b/docs/design/PLAWK_AWK_FEATURE_AUDIT.md index dac56f775..456aa367a 100644 --- a/docs/design/PLAWK_AWK_FEATURE_AUDIT.md +++ b/docs/design/PLAWK_AWK_FEATURE_AUDIT.md @@ -17,7 +17,7 @@ only (runtime pending) · ❌ missing. | Feature | Status | Notes | |---|---|---| -| `BEGIN` / `END` | ✅ | incl. constant `print` (BEGIN/END literal print). **`$0` / `$N` in END now read the LAST RECORD** (`END { print $1 }` → `c` on `a 1/b 2/c 3`, like gawk), in a straight-line END print, a statement list (`END { print $1; print $2 }`) and a concatenation (`print $1 " / " $2`). The last record is *gone* by END — at `end_print` the transient buffer holds the `end_of_file` sentinel — so retention is explicit: the record loop copies each record into a reused, geometrically grown buffer (`@plawk_lastrec_store`, the same shape as `@wam_rt_set`, so **constant memory and one memcpy per record**; interning per record would grow the atom table with every distinct record), and END re-materialises those bytes in the shared transient buffer and hands the reserved transient atom id to `llvm_emit_atom_field_slice/5` — the *same* slicer every in-loop field read uses. Honours FS/OFS/ORS/RS; `$N` past NF is empty; empty input gives an empty `$0`. **Pay-per-use**: the globals and defines are emitted as program-level IR only when `plawk_end_term_mentions_field/1` (the #4100 END-loop safety gate, *inverted* rather than restated) fires, so every program without an END field read is byte-identical (15/15 golden corpus). **Also in an END `if` branch** (`END { if (n == 3) print $1 }` — which had been silently printing `end_of_file`, a pre-existing wrong output surfaced and fixed by this line) **and an END loop body** (`while` / `do-while` / C-`for`, nested, inside an `if` inside a loop, with `break` — flipping #4100's gate). Those two needed no emitter parameterisation: `plawk_end_lastrec_rewrite/2` rewrites `field(N)` → `end_lastrec_field(N)` in the END actions (a structural walk, matching the gate) and **two clauses** on the shared `plawk_emit_print_expr_for_context/6` know the record source, so the rule-body print emitter and `plawk_scalar_action_sequence_pairs//15` are untouched. The rewrite covers conditions too, which is fail-safe: no condition emitter has a clause for `end_lastrec_field(_)`, so `END { if ($1 == "c") … }` declines rather than miscompiling. **`NF` in END counts the retained record** (`END { print NF }` → 2), in all three contexts — and in the `if`-branch and loop-body ones it had been printing **1**, the field count of the `end_of_file` sentinel, since those drivers existed: the gate was named `plawk_end_term_mentions_field/1` and matched `field(_)` only, so `NF` walked past the gate built to catch record reads. It is now `plawk_end_term_reads_record/1`, named for the property, matching `field(_)`, `special('NF')` and `special(length)` (`NR`/`RT` are process state and stay out). **`printf` arguments too** — `printf "%s\n", $1`, `printf "%s\n", $0`, `printf "%d\n", NF`, and mixes with scalars — producing the same `string_ptr` / `slice_len`+`slice_ptr` / `i64` call-argument vocabulary a record-context printf produces, so the format rewriter needed no new cases. **`length` / `length($N)` in END counts the retained record too**, in every route at once — a straight-line print, a concatenation (`print "L=" length`), a statement list, the mixed and assoc routes, a `printf` argument, an `if` branch and a loop body. It cost ONE clause, because the change that carried it first collapsed the duplication that had been making every previous cell cost one per route. Two collapses: (a) `NF` had an `end_lastrec_nf` expression row that was its in-loop `nf` row with `%line` swapped for the retained record Value — `length` had no such row, which is the whole reason every END form of it declined — so both became entries in `plawk_record_i64_read/5`, ONE table of record-reading i64 leaves parameterised on *which record they read*, with ONE retained-record wrapper (`end_lastrec_read(Kind)`) covering every entry, and `plawk_end_lastrec_nf_lines//2` generalised to `plawk_end_lastrec_i64_lines//4` with a NameBase parameter so NF keeps its own temporaries and its IR stays byte-identical; (b) the THREE END print walkers each enumerated the print-field vocabulary as clause heads, and each clause was three steps — separator, one per-kind call, recurse — around exactly the call `plawk_end_field_print_lines/4` already makes for that kind inside a concat, so "what may be printed in END" was FOUR lists with nothing keeping them equal (they had already drifted twice: a string scalar in a concat printed its atom id, and NF reached the routes one at a time). **24 clauses became 13**, each walker keeping only what is genuinely its own — `concat`, the one kind that is not a single value, and for the assoc route the table reads. Two smaller instances of the same shape went with it: `plawk_end_list_printf_arg_ok/2`'s two chains each listed the record-reading printf arguments (they differ only in `var` and `NR`), factored so `length` was one row not two; and `plawk_rule_body_print_field/1`, a THIRD list reached by an END loop body, needed one row. **32 pre-existing golden-corpus programs are byte-identical** across both collapses, which is the check that says a collapse was a collapse and not a rewrite. Bare `length` needed no case of its own: it parses to `length(field(0))` and field 0 measures the whole record, so the awk shorthand is already the arity-1 instance of the general form. One unintended-but-correct side effect, pinned rather than left unremarked: delegating the assoc walker passes the EMPTY scalar plan (as it already did for concat parts), which makes the shared generic expression clause reachable, so `END { print 1 + 2, c["x"] }` now builds and agrees with gawk. Tests: `tests/test_plawk_end_length.pl`. Still declined: a field or `NF` in a loop/`if` **condition**, **builtins over the record** (`substr($0, …)`, `toupper($1)`), ~~an **END-only** program~~ **now compiles** (`END { print ... }`, printf, END-`if`, and the empty program `''` — the rules+END driver admits an empty rule chain via an empty scalar state plan + a dedicated `plawk_scalar_rule_chain_ir([], ...)` clause emitting `br label %continue_loop`; the record loop still runs so NR/$0/$N/NF/length are the last record, byte-identical on 20/20 existing programs; remaining END-only follow-ons are a **loop** `END { while ... }` (pinned as a clean exit-3 decline, guarded so it never miscompiles), a scalar **assignment**, and for-in/getline; tests `tests/test_plawk_end_only.pl` 29), a record read in a **for-in CHAIN** body (`END { for (k in c) print k, NF }` — `NF` and `length` decline there alike, so it is the chain body's print vocabulary rather than either feature), the **associative** END-`if` branch (`plawk_assoc_end_if_branch_prints_ok/2` allows only string literals there — an unrelated pre-existing restriction), and under a **binary descriptor** | +| `BEGIN` / `END` | ✅ | incl. constant `print` (BEGIN/END literal print). **`$0` / `$N` in END now read the LAST RECORD** (`END { print $1 }` → `c` on `a 1/b 2/c 3`, like gawk), in a straight-line END print, a statement list (`END { print $1; print $2 }`) and a concatenation (`print $1 " / " $2`). The last record is *gone* by END — at `end_print` the transient buffer holds the `end_of_file` sentinel — so retention is explicit: the record loop copies each record into a reused, geometrically grown buffer (`@plawk_lastrec_store`, the same shape as `@wam_rt_set`, so **constant memory and one memcpy per record**; interning per record would grow the atom table with every distinct record), and END re-materialises those bytes in the shared transient buffer and hands the reserved transient atom id to `llvm_emit_atom_field_slice/5` — the *same* slicer every in-loop field read uses. Honours FS/OFS/ORS/RS; `$N` past NF is empty; empty input gives an empty `$0`. **Pay-per-use**: the globals and defines are emitted as program-level IR only when `plawk_end_term_mentions_field/1` (the #4100 END-loop safety gate, *inverted* rather than restated) fires, so every program without an END field read is byte-identical (15/15 golden corpus). **Also in an END `if` branch** (`END { if (n == 3) print $1 }` — which had been silently printing `end_of_file`, a pre-existing wrong output surfaced and fixed by this line) **and an END loop body** (`while` / `do-while` / C-`for`, nested, inside an `if` inside a loop, with `break` — flipping #4100's gate). Those two needed no emitter parameterisation: `plawk_end_lastrec_rewrite/2` rewrites `field(N)` → `end_lastrec_field(N)` in the END actions (a structural walk, matching the gate) and **two clauses** on the shared `plawk_emit_print_expr_for_context/6` know the record source, so the rule-body print emitter and `plawk_scalar_action_sequence_pairs//15` are untouched. The rewrite covers conditions too, which is fail-safe: no condition emitter has a clause for `end_lastrec_field(_)`, so `END { if ($1 == "c") … }` declines rather than miscompiling. **`NF` in END counts the retained record** (`END { print NF }` → 2), in all three contexts — and in the `if`-branch and loop-body ones it had been printing **1**, the field count of the `end_of_file` sentinel, since those drivers existed: the gate was named `plawk_end_term_mentions_field/1` and matched `field(_)` only, so `NF` walked past the gate built to catch record reads. It is now `plawk_end_term_reads_record/1`, named for the property, matching `field(_)`, `special('NF')` and `special(length)` (`NR`/`RT` are process state and stay out). **`printf` arguments too** — `printf "%s\n", $1`, `printf "%s\n", $0`, `printf "%d\n", NF`, and mixes with scalars — producing the same `string_ptr` / `slice_len`+`slice_ptr` / `i64` call-argument vocabulary a record-context printf produces, so the format rewriter needed no new cases. **`length` / `length($N)` in END counts the retained record too**, in every route at once — a straight-line print, a concatenation (`print "L=" length`), a statement list, the mixed and assoc routes, a `printf` argument, an `if` branch and a loop body. It cost ONE clause, because the change that carried it first collapsed the duplication that had been making every previous cell cost one per route. Two collapses: (a) `NF` had an `end_lastrec_nf` expression row that was its in-loop `nf` row with `%line` swapped for the retained record Value — `length` had no such row, which is the whole reason every END form of it declined — so both became entries in `plawk_record_i64_read/5`, ONE table of record-reading i64 leaves parameterised on *which record they read*, with ONE retained-record wrapper (`end_lastrec_read(Kind)`) covering every entry, and `plawk_end_lastrec_nf_lines//2` generalised to `plawk_end_lastrec_i64_lines//4` with a NameBase parameter so NF keeps its own temporaries and its IR stays byte-identical; (b) the THREE END print walkers each enumerated the print-field vocabulary as clause heads, and each clause was three steps — separator, one per-kind call, recurse — around exactly the call `plawk_end_field_print_lines/4` already makes for that kind inside a concat, so "what may be printed in END" was FOUR lists with nothing keeping them equal (they had already drifted twice: a string scalar in a concat printed its atom id, and NF reached the routes one at a time). **24 clauses became 13**, each walker keeping only what is genuinely its own — `concat`, the one kind that is not a single value, and for the assoc route the table reads. Two smaller instances of the same shape went with it: `plawk_end_list_printf_arg_ok/2`'s two chains each listed the record-reading printf arguments (they differ only in `var` and `NR`), factored so `length` was one row not two; and `plawk_rule_body_print_field/1`, a THIRD list reached by an END loop body, needed one row. **32 pre-existing golden-corpus programs are byte-identical** across both collapses, which is the check that says a collapse was a collapse and not a rewrite. Bare `length` needed no case of its own: it parses to `length(field(0))` and field 0 measures the whole record, so the awk shorthand is already the arity-1 instance of the general form. One unintended-but-correct side effect, pinned rather than left unremarked: delegating the assoc walker passes the EMPTY scalar plan (as it already did for concat parts), which makes the shared generic expression clause reachable, so `END { print 1 + 2, c["x"] }` now builds and agrees with gawk. Tests: `tests/test_plawk_end_length.pl`. Still declined: a field or `NF` in a loop/`if` **condition**, **builtins over the record** (`substr($0, …)`, `toupper($1)`), ~~an **END-only** program~~ **now compiles** (`END { print ... }`, printf, END-`if`, and the empty program `''` — the rules+END driver admits an empty rule chain via an empty scalar state plan + a dedicated `plawk_scalar_rule_chain_ir([], ...)` clause emitting `br label %continue_loop`; the record loop still runs so NR/$0/$N/NF/length are the last record, byte-identical on 20/20 existing programs; remaining END-only follow-ons are a **scalar-variable read/write** `END { print x }` (declines cleanly at exit 3 -- an ultra review caught that admitting it as a slot made the loop's next-slot phi reference `%rule_-1_*`, a miscompile; and an unset scalar is context-dependent, the uninitialised-scalar problem, so it needs that settled first), a **loop** `END { while ... }` (clean exit-3 decline, guarded so it never miscompiles), a scalar **assignment**, and for-in/getline; tests `tests/test_plawk_end_only.pl` 29), a record read in a **for-in CHAIN** body (`END { for (k in c) print k, NF }` — `NF` and `length` decline there alike, so it is the chain body's print vocabulary rather than either feature), the **associative** END-`if` branch (`plawk_assoc_end_if_branch_prints_ok/2` allows only string literals there — an unrelated pre-existing restriction), and under a **binary descriptor** | | `/regex/` | ✅ | a bare `/re/` pattern matches the whole record with **full POSIX ERE**: a body with metacharacters lowers to `field_match(0, Regex)` and runs through the same regex engine as `~`/`!~` (`/foo.*bar/`, `/^(err\|warn):/`, character classes, alternation, groups). A metachar-free body keeps its fast native lowering (`^prefix` → `prefix/1`, all-literal → `contains/1`); `\/` is a literal slash | | `$N == "v"`, `$3 > 100` | ✅ | field-equality + numeric field guards; **string inequality `$N != "v"`** now parses too (the skip-a-value idiom `$1 != "header"`) — rewritten to `not_pat(field_eq(N,"v"))`, so it reuses the `==` guard codegen and composes with `!`/`&&`/`||`, `if` conditions, and for-in-END group-by. Numeric `$N != K` keeps the integer path. **Field string ordering `$N < "v"` / `<=` / `>` / `>=` landed too** — awk compares a field against a string constant lexically (never numeric, so `$1 < "10"` is byte-wise while `$1 < 10` stays the numeric path); lowered to a memcmp of the field slice against the interned literal (`@wam_atom_field_str_cmp_value`), composing with `!`/`&&`/`||`, `if` conditions, and for-in-END group-by | | `&& \|\| !` combinators | ✅ | awk precedence, parens, single-block lowering | diff --git a/docs/design/PLAWK_CAMPAIGN_HANDOFF.md b/docs/design/PLAWK_CAMPAIGN_HANDOFF.md index 08611dbf3..886729c62 100644 --- a/docs/design/PLAWK_CAMPAIGN_HANDOFF.md +++ b/docs/design/PLAWK_CAMPAIGN_HANDOFF.md @@ -382,6 +382,29 @@ parse, not the gate you expected to fire. A broad exit-4 sweep of the whole surface the relaxation touches is mandatory, not optional: here it found exactly one such site, and the campaign's worst outcome (invalid LLVM on a supported-looking program) was one un-run probe away. + + **And the exit-4 sweep must vary the DATA the relaxation admits, not just the + program shapes.** My own sweep enumerated END-only shapes (print / printf / if / + loop / for-in) and driver clauses, and passed -- but every print case used a + literal, a field, NR, NF, or length. An ULTRA REVIEW caught what I missed: + `END { print x }`, a scalar VARIABLE, still exit-4'd. The reason is one level below + the dispatch clause: admitting an empty rule chain let plawk_scalar_state_plan + collect `x` as a slot, and the next-slot phi emitter then computed + LastRuleIndex = RuleCount - 1 = -1 and referenced `%rule_-1_*` -- undefined SSA. + The miscompile lived in the interaction between an empty rule chain and a NON-empty + state plan, a combination no shape-only sweep reaches unless it includes a program + whose END reads a scalar variable. Two lessons: (1) when a relaxation admits a new + input CLASS, enumerate the class's data dimensions (here: does the END read a + literal, a field, a special, or a scalar VAR?), because the defect can hide in a + data-dependent downstream (a phi index) that the dispatch-clause enumeration never + sees; (2) a second reviewer with fresh eyes found in one pass what my own thorough + verification did not -- the value of external review scales with how confident and + polished the change already looks. The fix guards the empty rule chain to fire only + when the state plan has no slots, so a scalar-var END-only program declines cleanly + (exit 3) as a follow-on; a correct pass-through phi also needs the + uninitialised-scalar representation settled (`print x` of an unset var is empty in + string context, 0 in numeric), which is why declining, not a quick phi patch, was + the right scope. - **gawk 5.2 is the oracle.** Compare output *and* exit status. Probe harness pattern: write the program, `rm -f` the binary first (a declining build must not run a stale one), build, run, diff against `gawk`. @@ -708,8 +731,9 @@ record** in END (`substr($0, …)`, `toupper($1)` — the gate retains for them, emitters have no clause; the *literal* forms of all five are done) · ~~**END-only** programs~~ **DONE** (see below; `END { print ... }`, printf, END-`if`, and the empty program now compile — remaining END-only follow-ons are an END-only **loop** -(`END { while ... }`, pinned as a clean exit-3 decline, NOT a miscompile), an -END-only scalar **assignment** (`END { x = 5; print x }`), and for-in / getline in +(`END { while ... }`, pinned as a clean exit-3 decline, NOT a miscompile), an END-only +**scalar-variable read/write** (`END { print x }` -- see the review-found miscompile below), +an END-only scalar **assignment** (`END { x = 5; print x }`), and for-in / getline in END-only) · `printf` field args in the **assoc / mixed END chain** (a different driver, passes `no_end_record`) · the **associative** END-`if` branch (refused by diff --git a/examples/plawk/codegen/llvm/plawk_native_codegen.pl b/examples/plawk/codegen/llvm/plawk_native_codegen.pl index e12aafd52..e4f3edd93 100644 --- a/examples/plawk/codegen/llvm/plawk_native_codegen.pl +++ b/examples/plawk/codegen/llvm/plawk_native_codegen.pl @@ -14921,8 +14921,19 @@ % `lowered_match:` with no terminator -- invalid LLVM (clang: expected instruction % opcode). Emit the loop-continue branch instead: with nothing to match, the match % block jumps straight to `continue_loop`. RuleCount 0, no globals, no branch exits. -plawk_scalar_rule_chain_ir([], _StatePlan, _FieldSeparator, _OutputSeparator, +plawk_scalar_rule_chain_ir([], StatePlan, _FieldSeparator, _OutputSeparator, '', ' br label %continue_loop', 0, []) :- + % Zero rules is supported only when the END touches no SCALAR variable -- i.e. the + % state plan has no slots. A rule-less program whose END reads/writes a scalar + % (`END { print x }`) gives the plan a slot, and the loop's next-slot phi would + % reference %rule_-1_* (LastRuleIndex = RuleCount-1 = -1) -- undefined SSA, an + % exit-4 miscompile. AND an unset scalar's value in END is context-dependent + % (empty in string context, 0 in numeric) -- the uninitialised-scalar + % representation problem -- so even a correct pass-through phi would need that + % settled first. Until then, decline cleanly (this clause fails, the general + % `RuleCount > 0` clause fails, the driver declines at exit 3) rather than + % miscompile or print a context-wrong value. Scalar-var END-only is a follow-on. + plawk_state_plan_slots(StatePlan, []), !. plawk_scalar_rule_chain_ir(Rules, StatePlan, FieldSeparator, OutputSeparator, GlobalIR, ChainIR, RuleCount, BranchNextExits) :- diff --git a/tests/test_plawk_end_only.pl b/tests/test_plawk_end_only.pl index 06cd9ac20..0b2a03196 100644 --- a/tests/test_plawk_end_only.pl +++ b/tests/test_plawk_end_only.pl @@ -211,6 +211,24 @@ build_status("END { getline x; print x }\n", 3), !. +% A scalar VARIABLE read/write in an END-only program declines cleanly (exit 3), NOT +% exit 4. This is the case an ultra review caught that the first cut missed: admitting +% an empty rule chain let the state plan collect the scalar as a slot, and the loop's +% next-slot phi then referenced %rule_-1_* (LastRuleIndex = RuleCount-1 = -1) -- +% undefined SSA, a clang miscompile. The dedicated empty rule-chain clause is now +% guarded to fire only when the state plan has NO slots, so a scalar-var END-only +% program declines. It is a genuine follow-on: an unset scalar's value in END is +% context-dependent (empty in string context, 0 in numeric -- the uninitialised-scalar +% representation problem), so even a correct pass-through phi needs that settled first. +% Pinned so the miscompile cannot silently return and the follow-on stays visible. +test(end_only_scalar_var_read_declines_not_miscompiles) :- + build_status("END { print x }\n", 3), + build_status("END { print x; print \"done\" }\n", 3), + build_status("END { printf \"%d\\n\", x }\n", 3), + build_status("END { if (x == 0) print \"zero\" }\n", 3), + build_status("END { if (NR > 0) print x }\n", 3), + !. + % --- clang never rejects a supported END-only program ------------------- % % Belt-and-braces over the whole supported surface: build each and require a @@ -252,7 +270,14 @@ "BEGIN { FS=\":\" } END { print NR }\n", "BEGIN { print \"start\" } END { print NR }\n", "END { getline x }\n", - "END { x = 5 }\n" + "END { x = 5 }\n", + "END { print x }\n", + "END { print x; print \"done\" }\n", + "END { printf \"%d\\n\", x }\n", + "END { if (x == 0) print \"zero\" }\n", + "END { if (NR > 0) print x }\n", + "BEGIN { FS=\":\" } END { print x }\n", + "BEGIN { BINFMT = \"i64\" } END { print x }\n" ]), build_status_not_4(Src)), !. From e5bcc55897c7bdb5adca84b4c24cad9fb14868f2 Mon Sep 17 00:00:00 2001 From: "John William Creighton (s243a)" Date: Sun, 13 Sep 2026 15:27:00 -0600 Subject: [PATCH 4/5] plawk: re-attribute two stale END-only decline pins in begin_only test_plawk_begin_only.pl pinned `BEGIN { print "b" } END { print "e" }` and `END { print "e" }` as clean declines (exit 3) while END-only was a gap. The END-only driver landed, so both now compile and match gawk. Re-attributed (not deleted) to assert the new behaviour: they build (exit 0) and print their literal output deterministically on empty stdin. begin_only now 31/31. Surfaced by the full sweep -- which also exposed a harness bug (fixed in the local sweep tooling, not this repo): under `set -e` a suite whose run_tests failed aborted the whole sweep at that suite instead of recording the failure and continuing, so these two stale pins had masqueraded as a sweep that "died at 11/195" rather than being reported as failures. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01QrQistdoYMNpVtecwims6g --- tests/test_plawk_begin_only.pl | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/tests/test_plawk_begin_only.pl b/tests/test_plawk_begin_only.pl index 8a40163ff..9645e187e 100644 --- a/tests/test_plawk_begin_only.pl +++ b/tests/test_plawk_begin_only.pl @@ -185,15 +185,17 @@ % --- clean declines ------------------------------------------------------- -% A zero-rule program WITH an END: awk reads input in this case (END sees NR), so -% the loop-free driver must not claim it. No driver handles zero rules plus a -% record loop yet, so it declines -- a follow-on, not a silent wrong answer. -test(begin_and_end_without_rules_declines) :- - build_status("BEGIN { print \"b\" }\nEND { print \"e\" }\n", 3), +% A zero-rule program WITH an END now COMPILES -- the END-only driver landed +% (tests/test_plawk_end_only.pl). awk reads input in this case (END sees NR), so +% unlike BEGIN-only these run the record loop with an empty rule chain. Was pinned +% here as a decline while END-only was a gap; re-attributed, not deleted, to assert +% the new behaviour. Output is literal-only, so it is deterministic on empty stdin. +test(begin_and_end_without_rules_now_compiles) :- + run("BEGIN { print \"b\" }\nEND { print \"e\" }\n", "b\ne\n", 0), !. -test(end_only_without_rules_declines) :- - build_status("END { print \"e\" }\n", 3), +test(end_only_without_rules_now_compiles) :- + run("END { print \"e\" }\n", "e\n", 0), !. % BEGIN has no record and no scalar slots, so a field or variable read declines From 62d9dfce7804c44e22817fe900142b039e3b066c Mon Sep 17 00:00:00 2001 From: "John William Creighton (s243a)" Date: Mon, 14 Sep 2026 16:46:30 -0600 Subject: [PATCH 5/5] plawk: strengthen END-only tests per the Astra review (SHIP-WITH-CHANGES) The review confirmed the scalar-slot fix is correct (no residual exit-4, guard covers all scalar dispatch paths, no bypass) and asked only for stronger regression coverage. Applied: - The scalar-var decline pin gains the four contexts the review named as missing: a concatenation (`print "n=" x`), an arithmetic operand (`print x + 1`), an ELSE branch (`if (NR==0) print "empty"; else print x`), and a union-BINFMT program. All decline cleanly at exit 3 (verified), never miscompile. - The no-miscompile enumeration gains an explicit redirection example (`print NR > "/dev/stdout"`, a parse error at 2) -- the comment claimed redirection coverage without a case. - build_status_not_4/1 now asserts the status is a KNOWN-GOOD value (memberchk in [0,2,3]) rather than merely `=\= 4`, which would have passed an unexpected exit 1 (a crash). Renamed intent in the comment. Suite stays at 31 tests, all passing. Also documents, in the handoff follow-ons, the EOF-sentinel defect both reviews surfaced: a literal `end_of_file` input line is mistaken for EOF by a text comparison in the runtime stream driver (src/unifyweaver/targets/wam_llvm_target.pl ~23909). It is PRE-EXISTING (the rule-bearing form is equally wrong) and needs an identity-based EOF check -- a runtime fix affecting all programs, deliberately out of scope here; END-only merely made it reachable in an END-only shape. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01QrQistdoYMNpVtecwims6g --- docs/design/PLAWK_CAMPAIGN_HANDOFF.md | 12 ++++++++++++ tests/test_plawk_end_only.pl | 12 +++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/design/PLAWK_CAMPAIGN_HANDOFF.md b/docs/design/PLAWK_CAMPAIGN_HANDOFF.md index 886729c62..243385650 100644 --- a/docs/design/PLAWK_CAMPAIGN_HANDOFF.md +++ b/docs/design/PLAWK_CAMPAIGN_HANDOFF.md @@ -690,6 +690,18 @@ the implementation changed about the design. The load-bearing facts: ## Remaining follow-ons +**EOF-sentinel: a literal `end_of_file` input line is mistaken for EOF (PRE-EXISTING, +not END-only).** `END { print NR, $0 }` on input `a\nend_of_file\nb\n` prints `1 a`; +gawk prints `3 b`. Two independent reviews (terra, astra) surfaced this. The shared +stream driver (`src/unifyweaver/targets/wam_llvm_target.pl`, ~line 23909) detects +end-of-input by comparing the record TEXT against `"end_of_file"` instead of by atom +identity -- the runtime already has a distinct EOF atom, so the fix is an +identity-based comparison plus a regression test. It predates the END-only work and +affects EVERY program (the rule-bearing `{ n++ } END { print NR, $0 }` is equally +wrong); END-only just made it reachable in an END-only shape. A runtime fix of its +own, deliberately out of scope for the END-only PR. + + **END record reads, what is left** — each pinned as a decline in `tests/test_plawk_end_field_reads.pl`. A field or `NF` in a loop / `if` **condition** (a fail-safe decline: the rewrite reaches conditions, and no diff --git a/tests/test_plawk_end_only.pl b/tests/test_plawk_end_only.pl index 0b2a03196..9d7ec7216 100644 --- a/tests/test_plawk_end_only.pl +++ b/tests/test_plawk_end_only.pl @@ -227,6 +227,10 @@ build_status("END { printf \"%d\\n\", x }\n", 3), build_status("END { if (x == 0) print \"zero\" }\n", 3), build_status("END { if (NR > 0) print x }\n", 3), + build_status("END { print \"n=\" x }\n", 3), + build_status("END { print x + 1 }\n", 3), + build_status("END { if (NR == 0) print \"empty\"; else print x }\n", 3), + build_status("BEGIN { BINFMT=\"case(i64 | i64)\" } END { print x }\n", 3), !. % --- clang never rejects a supported END-only program ------------------- @@ -277,7 +281,8 @@ "END { if (x == 0) print \"zero\" }\n", "END { if (NR > 0) print x }\n", "BEGIN { FS=\":\" } END { print x }\n", - "BEGIN { BINFMT = \"i64\" } END { print x }\n" + "BEGIN { BINFMT = \"i64\" } END { print x }\n", + "END { print NR > \"/dev/stdout\" }\n" ]), build_status_not_4(Src)), !. @@ -353,9 +358,10 @@ [stdout(pipe(Out)), stderr(std), process(Pid)]), read_string(Out, _, _), close(Out), process_wait(Pid, exit(Status)), - ( Status =\= 4 + ( memberchk(Status, [0, 2, 3]) -> true - ; format(user_error, "~nMISCOMPILE (exit 4): ~w~n", [Src]), fail + ; format(user_error, "~nUNEXPECTED STATUS ~w (want 0/2/3, never 4): ~w~n", + [Status, Src]), fail ). build_ll(Src, LL) :-