diff --git a/docs/design/PLAWK_AWK_FEATURE_AUDIT.md b/docs/design/PLAWK_AWK_FEATURE_AUDIT.md index 5ee541b1d..456aa367a 100644 --- a/docs/design/PLAWK_AWK_FEATURE_AUDIT.md +++ b/docs/design/PLAWK_AWK_FEATURE_AUDIT.md @@ -17,7 +17,7 @@ only (runtime pending) · ❌ missing. | Feature | Status | Notes | |---|---|---| -| `BEGIN` / `END` | ✅ | incl. constant `print` (BEGIN/END literal print). **`$0` / `$N` in END now read the LAST RECORD** (`END { print $1 }` → `c` on `a 1/b 2/c 3`, like gawk), in a straight-line END print, a statement list (`END { print $1; print $2 }`) and a concatenation (`print $1 " / " $2`). The last record is *gone* by END — at `end_print` the transient buffer holds the `end_of_file` sentinel — so retention is explicit: the record loop copies each record into a reused, geometrically grown buffer (`@plawk_lastrec_store`, the same shape as `@wam_rt_set`, so **constant memory and one memcpy per record**; interning per record would grow the atom table with every distinct record), and END re-materialises those bytes in the shared transient buffer and hands the reserved transient atom id to `llvm_emit_atom_field_slice/5` — the *same* slicer every in-loop field read uses. Honours FS/OFS/ORS/RS; `$N` past NF is empty; empty input gives an empty `$0`. **Pay-per-use**: the globals and defines are emitted as program-level IR only when `plawk_end_term_mentions_field/1` (the #4100 END-loop safety gate, *inverted* rather than restated) fires, so every program without an END field read is byte-identical (15/15 golden corpus). **Also in an END `if` branch** (`END { if (n == 3) print $1 }` — which had been silently printing `end_of_file`, a pre-existing wrong output surfaced and fixed by this line) **and an END loop body** (`while` / `do-while` / C-`for`, nested, inside an `if` inside a loop, with `break` — flipping #4100's gate). Those two needed no emitter parameterisation: `plawk_end_lastrec_rewrite/2` rewrites `field(N)` → `end_lastrec_field(N)` in the END actions (a structural walk, matching the gate) and **two clauses** on the shared `plawk_emit_print_expr_for_context/6` know the record source, so the rule-body print emitter and `plawk_scalar_action_sequence_pairs//15` are untouched. The rewrite covers conditions too, which is fail-safe: no condition emitter has a clause for `end_lastrec_field(_)`, so `END { if ($1 == "c") … }` declines rather than miscompiling. **`NF` in END counts the retained record** (`END { print NF }` → 2), in all three contexts — and in the `if`-branch and loop-body ones it had been printing **1**, the field count of the `end_of_file` sentinel, since those drivers existed: the gate was named `plawk_end_term_mentions_field/1` and matched `field(_)` only, so `NF` walked past the gate built to catch record reads. It is now `plawk_end_term_reads_record/1`, named for the property, matching `field(_)`, `special('NF')` and `special(length)` (`NR`/`RT` are process state and stay out). **`printf` arguments too** — `printf "%s\n", $1`, `printf "%s\n", $0`, `printf "%d\n", NF`, and mixes with scalars — producing the same `string_ptr` / `slice_len`+`slice_ptr` / `i64` call-argument vocabulary a record-context printf produces, so the format rewriter needed no new cases. **`length` / `length($N)` in END counts the retained record too**, in every route at once — a straight-line print, a concatenation (`print "L=" length`), a statement list, the mixed and assoc routes, a `printf` argument, an `if` branch and a loop body. It cost ONE clause, because the change that carried it first collapsed the duplication that had been making every previous cell cost one per route. Two collapses: (a) `NF` had an `end_lastrec_nf` expression row that was its in-loop `nf` row with `%line` swapped for the retained record Value — `length` had no such row, which is the whole reason every END form of it declined — so both became entries in `plawk_record_i64_read/5`, ONE table of record-reading i64 leaves parameterised on *which record they read*, with ONE retained-record wrapper (`end_lastrec_read(Kind)`) covering every entry, and `plawk_end_lastrec_nf_lines//2` generalised to `plawk_end_lastrec_i64_lines//4` with a NameBase parameter so NF keeps its own temporaries and its IR stays byte-identical; (b) the THREE END print walkers each enumerated the print-field vocabulary as clause heads, and each clause was three steps — separator, one per-kind call, recurse — around exactly the call `plawk_end_field_print_lines/4` already makes for that kind inside a concat, so "what may be printed in END" was FOUR lists with nothing keeping them equal (they had already drifted twice: a string scalar in a concat printed its atom id, and NF reached the routes one at a time). **24 clauses became 13**, each walker keeping only what is genuinely its own — `concat`, the one kind that is not a single value, and for the assoc route the table reads. Two smaller instances of the same shape went with it: `plawk_end_list_printf_arg_ok/2`'s two chains each listed the record-reading printf arguments (they differ only in `var` and `NR`), factored so `length` was one row not two; and `plawk_rule_body_print_field/1`, a THIRD list reached by an END loop body, needed one row. **32 pre-existing golden-corpus programs are byte-identical** across both collapses, which is the check that says a collapse was a collapse and not a rewrite. Bare `length` needed no case of its own: it parses to `length(field(0))` and field 0 measures the whole record, so the awk shorthand is already the arity-1 instance of the general form. One unintended-but-correct side effect, pinned rather than left unremarked: delegating the assoc walker passes the EMPTY scalar plan (as it already did for concat parts), which makes the shared generic expression clause reachable, so `END { print 1 + 2, c["x"] }` now builds and agrees with gawk. Tests: `tests/test_plawk_end_length.pl`. Still declined: a field or `NF` in a loop/`if` **condition**, **builtins over the record** (`substr($0, …)`, `toupper($1)`), an **END-only** program (different driver, no retain), a record read in a **for-in CHAIN** body (`END { for (k in c) print k, NF }` — `NF` and `length` decline there alike, so it is the chain body's print vocabulary rather than either feature), the **associative** END-`if` branch (`plawk_assoc_end_if_branch_prints_ok/2` allows only string literals there — an unrelated pre-existing restriction), and under a **binary descriptor** | +| `BEGIN` / `END` | ✅ | incl. constant `print` (BEGIN/END literal print). **`$0` / `$N` in END now read the LAST RECORD** (`END { print $1 }` → `c` on `a 1/b 2/c 3`, like gawk), in a straight-line END print, a statement list (`END { print $1; print $2 }`) and a concatenation (`print $1 " / " $2`). The last record is *gone* by END — at `end_print` the transient buffer holds the `end_of_file` sentinel — so retention is explicit: the record loop copies each record into a reused, geometrically grown buffer (`@plawk_lastrec_store`, the same shape as `@wam_rt_set`, so **constant memory and one memcpy per record**; interning per record would grow the atom table with every distinct record), and END re-materialises those bytes in the shared transient buffer and hands the reserved transient atom id to `llvm_emit_atom_field_slice/5` — the *same* slicer every in-loop field read uses. Honours FS/OFS/ORS/RS; `$N` past NF is empty; empty input gives an empty `$0`. **Pay-per-use**: the globals and defines are emitted as program-level IR only when `plawk_end_term_mentions_field/1` (the #4100 END-loop safety gate, *inverted* rather than restated) fires, so every program without an END field read is byte-identical (15/15 golden corpus). **Also in an END `if` branch** (`END { if (n == 3) print $1 }` — which had been silently printing `end_of_file`, a pre-existing wrong output surfaced and fixed by this line) **and an END loop body** (`while` / `do-while` / C-`for`, nested, inside an `if` inside a loop, with `break` — flipping #4100's gate). Those two needed no emitter parameterisation: `plawk_end_lastrec_rewrite/2` rewrites `field(N)` → `end_lastrec_field(N)` in the END actions (a structural walk, matching the gate) and **two clauses** on the shared `plawk_emit_print_expr_for_context/6` know the record source, so the rule-body print emitter and `plawk_scalar_action_sequence_pairs//15` are untouched. The rewrite covers conditions too, which is fail-safe: no condition emitter has a clause for `end_lastrec_field(_)`, so `END { if ($1 == "c") … }` declines rather than miscompiling. **`NF` in END counts the retained record** (`END { print NF }` → 2), in all three contexts — and in the `if`-branch and loop-body ones it had been printing **1**, the field count of the `end_of_file` sentinel, since those drivers existed: the gate was named `plawk_end_term_mentions_field/1` and matched `field(_)` only, so `NF` walked past the gate built to catch record reads. It is now `plawk_end_term_reads_record/1`, named for the property, matching `field(_)`, `special('NF')` and `special(length)` (`NR`/`RT` are process state and stay out). **`printf` arguments too** — `printf "%s\n", $1`, `printf "%s\n", $0`, `printf "%d\n", NF`, and mixes with scalars — producing the same `string_ptr` / `slice_len`+`slice_ptr` / `i64` call-argument vocabulary a record-context printf produces, so the format rewriter needed no new cases. **`length` / `length($N)` in END counts the retained record too**, in every route at once — a straight-line print, a concatenation (`print "L=" length`), a statement list, the mixed and assoc routes, a `printf` argument, an `if` branch and a loop body. It cost ONE clause, because the change that carried it first collapsed the duplication that had been making every previous cell cost one per route. Two collapses: (a) `NF` had an `end_lastrec_nf` expression row that was its in-loop `nf` row with `%line` swapped for the retained record Value — `length` had no such row, which is the whole reason every END form of it declined — so both became entries in `plawk_record_i64_read/5`, ONE table of record-reading i64 leaves parameterised on *which record they read*, with ONE retained-record wrapper (`end_lastrec_read(Kind)`) covering every entry, and `plawk_end_lastrec_nf_lines//2` generalised to `plawk_end_lastrec_i64_lines//4` with a NameBase parameter so NF keeps its own temporaries and its IR stays byte-identical; (b) the THREE END print walkers each enumerated the print-field vocabulary as clause heads, and each clause was three steps — separator, one per-kind call, recurse — around exactly the call `plawk_end_field_print_lines/4` already makes for that kind inside a concat, so "what may be printed in END" was FOUR lists with nothing keeping them equal (they had already drifted twice: a string scalar in a concat printed its atom id, and NF reached the routes one at a time). **24 clauses became 13**, each walker keeping only what is genuinely its own — `concat`, the one kind that is not a single value, and for the assoc route the table reads. Two smaller instances of the same shape went with it: `plawk_end_list_printf_arg_ok/2`'s two chains each listed the record-reading printf arguments (they differ only in `var` and `NR`), factored so `length` was one row not two; and `plawk_rule_body_print_field/1`, a THIRD list reached by an END loop body, needed one row. **32 pre-existing golden-corpus programs are byte-identical** across both collapses, which is the check that says a collapse was a collapse and not a rewrite. Bare `length` needed no case of its own: it parses to `length(field(0))` and field 0 measures the whole record, so the awk shorthand is already the arity-1 instance of the general form. One unintended-but-correct side effect, pinned rather than left unremarked: delegating the assoc walker passes the EMPTY scalar plan (as it already did for concat parts), which makes the shared generic expression clause reachable, so `END { print 1 + 2, c["x"] }` now builds and agrees with gawk. Tests: `tests/test_plawk_end_length.pl`. Still declined: a field or `NF` in a loop/`if` **condition**, **builtins over the record** (`substr($0, …)`, `toupper($1)`), ~~an **END-only** program~~ **now compiles** (`END { print ... }`, printf, END-`if`, and the empty program `''` — the rules+END driver admits an empty rule chain via an empty scalar state plan + a dedicated `plawk_scalar_rule_chain_ir([], ...)` clause emitting `br label %continue_loop`; the record loop still runs so NR/$0/$N/NF/length are the last record, byte-identical on 20/20 existing programs; remaining END-only follow-ons are a **scalar-variable read/write** `END { print x }` (declines cleanly at exit 3 -- an ultra review caught that admitting it as a slot made the loop's next-slot phi reference `%rule_-1_*`, a miscompile; and an unset scalar is context-dependent, the uninitialised-scalar problem, so it needs that settled first), a **loop** `END { while ... }` (clean exit-3 decline, guarded so it never miscompiles), a scalar **assignment**, and for-in/getline; tests `tests/test_plawk_end_only.pl` 29), a record read in a **for-in CHAIN** body (`END { for (k in c) print k, NF }` — `NF` and `length` decline there alike, so it is the chain body's print vocabulary rather than either feature), the **associative** END-`if` branch (`plawk_assoc_end_if_branch_prints_ok/2` allows only string literals there — an unrelated pre-existing restriction), and under a **binary descriptor** | | `/regex/` | ✅ | a bare `/re/` pattern matches the whole record with **full POSIX ERE**: a body with metacharacters lowers to `field_match(0, Regex)` and runs through the same regex engine as `~`/`!~` (`/foo.*bar/`, `/^(err\|warn):/`, character classes, alternation, groups). A metachar-free body keeps its fast native lowering (`^prefix` → `prefix/1`, all-literal → `contains/1`); `\/` is a literal slash | | `$N == "v"`, `$3 > 100` | ✅ | field-equality + numeric field guards; **string inequality `$N != "v"`** now parses too (the skip-a-value idiom `$1 != "header"`) — rewritten to `not_pat(field_eq(N,"v"))`, so it reuses the `==` guard codegen and composes with `!`/`&&`/`||`, `if` conditions, and for-in-END group-by. Numeric `$N != K` keeps the integer path. **Field string ordering `$N < "v"` / `<=` / `>` / `>=` landed too** — awk compares a field against a string constant lexically (never numeric, so `$1 < "10"` is byte-wise while `$1 < 10` stays the numeric path); lowered to a memcmp of the field slice against the interned literal (`@wam_atom_field_str_cmp_value`), composing with `!`/`&&`/`||`, `if` conditions, and for-in-END group-by | | `&& \|\| !` combinators | ✅ | awk precedence, parens, single-block lowering | diff --git a/docs/design/PLAWK_CAMPAIGN_HANDOFF.md b/docs/design/PLAWK_CAMPAIGN_HANDOFF.md index 4bbd261ad..243385650 100644 --- a/docs/design/PLAWK_CAMPAIGN_HANDOFF.md +++ b/docs/design/PLAWK_CAMPAIGN_HANDOFF.md @@ -367,6 +367,44 @@ parse, not the gate you expected to fire. ## Verification practices (do not skip) +- **Relaxing a shared gate can turn a clean decline into a MISCOMPILE in a + neighbouring driver that relied on that gate failing.** END-only support needed + `plawk_scalar_state_plan/3` to admit an empty plan (`; Rules == []`). That made + `END { print ... }` compile — but the END-LOOP driver clause (`END { while ... }`) + cuts as soon as it sees a loop, then had been relying on `state_plan` *failing* for + a rule-less program to decline after the cut. With the guard relaxed, state_plan + succeeded, the clause committed past its cut, and it emitted a malformed loop + driver: exit 4, a miscompile, on a program that used to decline cleanly at exit 3. + The fix was a targeted `Rules0 \== []` guard on that clause so END-loop-with-no-rule + keeps declining. **When you relax a gate, enumerate every clause that used that + gate's failure as its own decline mechanism** — a cut-then-rely-on-a-later-goal- + failing clause is silently converted from "declines" to "commits and may miscompile". + A broad exit-4 sweep of the whole surface the relaxation touches is mandatory, not + optional: here it found exactly one such site, and the campaign's worst outcome + (invalid LLVM on a supported-looking program) was one un-run probe away. + + **And the exit-4 sweep must vary the DATA the relaxation admits, not just the + program shapes.** My own sweep enumerated END-only shapes (print / printf / if / + loop / for-in) and driver clauses, and passed -- but every print case used a + literal, a field, NR, NF, or length. An ULTRA REVIEW caught what I missed: + `END { print x }`, a scalar VARIABLE, still exit-4'd. The reason is one level below + the dispatch clause: admitting an empty rule chain let plawk_scalar_state_plan + collect `x` as a slot, and the next-slot phi emitter then computed + LastRuleIndex = RuleCount - 1 = -1 and referenced `%rule_-1_*` -- undefined SSA. + The miscompile lived in the interaction between an empty rule chain and a NON-empty + state plan, a combination no shape-only sweep reaches unless it includes a program + whose END reads a scalar variable. Two lessons: (1) when a relaxation admits a new + input CLASS, enumerate the class's data dimensions (here: does the END read a + literal, a field, a special, or a scalar VAR?), because the defect can hide in a + data-dependent downstream (a phi index) that the dispatch-clause enumeration never + sees; (2) a second reviewer with fresh eyes found in one pass what my own thorough + verification did not -- the value of external review scales with how confident and + polished the change already looks. The fix guards the empty rule chain to fire only + when the state plan has no slots, so a scalar-var END-only program declines cleanly + (exit 3) as a follow-on; a correct pass-through phi also needs the + uninitialised-scalar representation settled (`print x` of an unset var is empty in + string context, 0 in numeric), which is why declining, not a quick phi patch, was + the right scope. - **gawk 5.2 is the oracle.** Compare output *and* exit status. Probe harness pattern: write the program, `rm -f` the binary first (a declining build must not run a stale one), build, run, diff against `gawk`. @@ -652,6 +690,18 @@ the implementation changed about the design. The load-bearing facts: ## Remaining follow-ons +**EOF-sentinel: a literal `end_of_file` input line is mistaken for EOF (PRE-EXISTING, +not END-only).** `END { print NR, $0 }` on input `a\nend_of_file\nb\n` prints `1 a`; +gawk prints `3 b`. Two independent reviews (terra, astra) surfaced this. The shared +stream driver (`src/unifyweaver/targets/wam_llvm_target.pl`, ~line 23909) detects +end-of-input by comparing the record TEXT against `"end_of_file"` instead of by atom +identity -- the runtime already has a distinct EOF atom, so the fix is an +identity-based comparison plus a regression test. It predates the END-only work and +affects EVERY program (the rule-bearing `{ n++ } END { print NR, $0 }` is equally +wrong); END-only just made it reachable in an END-only shape. A runtime fix of its +own, deliberately out of scope for the END-only PR. + + **END record reads, what is left** — each pinned as a decline in `tests/test_plawk_end_field_reads.pl`. A field or `NF` in a loop / `if` **condition** (a fail-safe decline: the rewrite reaches conditions, and no @@ -690,8 +740,13 @@ condition emitter has a clause for `end_lastrec_field(_)` / `end_lastrec_nf`) · at parse time (#4169). The two share a spelling and nothing else: one needs the retained record, the other needs no runtime at all. · **builtins over the record** in END (`substr($0, …)`, `toupper($1)` — the gate retains for them, the -emitters have no clause; the *literal* forms of all five are done) · **END-only** -programs (a driver with no retain) · +emitters have no clause; the *literal* forms of all five are done) · ~~**END-only** +programs~~ **DONE** (see below; `END { print ... }`, printf, END-`if`, and the empty +program now compile — remaining END-only follow-ons are an END-only **loop** +(`END { while ... }`, pinned as a clean exit-3 decline, NOT a miscompile), an END-only +**scalar-variable read/write** (`END { print x }` -- see the review-found miscompile below), +an END-only scalar **assignment** (`END { x = 5; print x }`), and for-in / getline in +END-only) · `printf` field args in the **assoc / mixed END chain** (a different driver, passes `no_end_record`) · the **associative** END-`if` branch (refused by `plawk_assoc_end_if_branch_prints_ok/2`, which allows only string literals there — diff --git a/examples/plawk/codegen/llvm/plawk_native_codegen.pl b/examples/plawk/codegen/llvm/plawk_native_codegen.pl index d3faece9f..e4f3edd93 100644 --- a/examples/plawk/codegen/llvm/plawk_native_codegen.pl +++ b/examples/plawk/codegen/llvm/plawk_native_codegen.pl @@ -1061,6 +1061,15 @@ DriverIR ) :- plawk_end_actions_have_loop(EndActions), + % A loop in the END block is only supported alongside a main rule. END-only + % support (an empty scalar state plan, added for `END { print ... }`) makes + % plawk_scalar_state_plan/3 succeed for Rules0 = [], which would otherwise let + % this clause commit past its cut and emit a malformed loop driver -- exit 4, + % a miscompile -- for `END { while (...) ... }`. Require a main rule so that + % END-loop-with-no-rule declines cleanly (exit 3) as the follow-on it is, + % rather than compiling to invalid LLVM. END-only print / if is unaffected + % (those clauses are 788 / 851 / 920 / 980). + Rules0 \== [], !, plawk_end_loop_print_fields(EndActions, PrintFields), plawk_resolve_writebin_rules(BeginClauses, Rules0, Rules1, WritebinPlan), @@ -8090,6 +8099,12 @@ ; BodyPrintFields \== [] ; plawk_rules_have_record_getline(Rules) ; plawk_rules_have_main_getline(Rules) + % A rule-less program (END-only, or the empty program) has an empty scalar + % state plan and must still reach the driver: with no rules, none of the + % disjuncts above can fire, so without this the guard rejects exactly the + % programs whose slots legitimately come only from the END print fields (or + % from nothing at all). state_plan([], []) is the correct plan here. + ; Rules == [] ), findall(Name, ( member(Field, PrintFields), @@ -14898,6 +14913,28 @@ plawk_end_scalar_expr(Expr), plawk_expr_scalar_read_name(Expr, Name). +% END-only / rule-less program: the record loop still runs (an END block reads all +% input to fix NR / NF / $0), but there is no per-record rule chain. The general +% clause below cannot serve this: with zero rules it emits an EMPTY `lowered_match:` +% block, and the entry branch into the chain (`br label %rule_0_match`) that would +% normally terminate that block is part of the chain IR, so an empty chain leaves +% `lowered_match:` with no terminator -- invalid LLVM (clang: expected instruction +% opcode). Emit the loop-continue branch instead: with nothing to match, the match +% block jumps straight to `continue_loop`. RuleCount 0, no globals, no branch exits. +plawk_scalar_rule_chain_ir([], StatePlan, _FieldSeparator, _OutputSeparator, + '', ' br label %continue_loop', 0, []) :- + % Zero rules is supported only when the END touches no SCALAR variable -- i.e. the + % state plan has no slots. A rule-less program whose END reads/writes a scalar + % (`END { print x }`) gives the plan a slot, and the loop's next-slot phi would + % reference %rule_-1_* (LastRuleIndex = RuleCount-1 = -1) -- undefined SSA, an + % exit-4 miscompile. AND an unset scalar's value in END is context-dependent + % (empty in string context, 0 in numeric) -- the uninitialised-scalar + % representation problem -- so even a correct pass-through phi would need that + % settled first. Until then, decline cleanly (this clause fails, the general + % `RuleCount > 0` clause fails, the driver declines at exit 3) rather than + % miscompile or print a context-wrong value. Scalar-var END-only is a follow-on. + plawk_state_plan_slots(StatePlan, []), + !. plawk_scalar_rule_chain_ir(Rules, StatePlan, FieldSeparator, OutputSeparator, GlobalIR, ChainIR, RuleCount, BranchNextExits) :- plawk_scalar_planned_rules(Rules, PlannedRules, Controls), diff --git a/tests/test_plawk_begin_only.pl b/tests/test_plawk_begin_only.pl index 8a40163ff..9645e187e 100644 --- a/tests/test_plawk_begin_only.pl +++ b/tests/test_plawk_begin_only.pl @@ -185,15 +185,17 @@ % --- clean declines ------------------------------------------------------- -% A zero-rule program WITH an END: awk reads input in this case (END sees NR), so -% the loop-free driver must not claim it. No driver handles zero rules plus a -% record loop yet, so it declines -- a follow-on, not a silent wrong answer. -test(begin_and_end_without_rules_declines) :- - build_status("BEGIN { print \"b\" }\nEND { print \"e\" }\n", 3), +% A zero-rule program WITH an END now COMPILES -- the END-only driver landed +% (tests/test_plawk_end_only.pl). awk reads input in this case (END sees NR), so +% unlike BEGIN-only these run the record loop with an empty rule chain. Was pinned +% here as a decline while END-only was a gap; re-attributed, not deleted, to assert +% the new behaviour. Output is literal-only, so it is deterministic on empty stdin. +test(begin_and_end_without_rules_now_compiles) :- + run("BEGIN { print \"b\" }\nEND { print \"e\" }\n", "b\ne\n", 0), !. -test(end_only_without_rules_declines) :- - build_status("END { print \"e\" }\n", 3), +test(end_only_without_rules_now_compiles) :- + run("END { print \"e\" }\n", "e\n", 0), !. % BEGIN has no record and no scalar slots, so a field or variable read declines diff --git a/tests/test_plawk_end_field_reads.pl b/tests/test_plawk_end_field_reads.pl index 88d1491d4..f69ac3f58 100644 --- a/tests/test_plawk_end_field_reads.pl +++ b/tests/test_plawk_end_field_reads.pl @@ -610,9 +610,13 @@ build_status("{ n++ } END { print toupper($1) }\n", 3), !. -% An END-only program (no rules) uses a different driver, which does not retain. -test(end_only_program_field_read_declines) :- - build_status("END { print $1 }\n", 3), +% An END-only program (no rules) now COMPILES and retains the last record -- the +% END-only driver landed. `END { print $1 }` builds and prints the last record's $1. +% Pinned as a decline while END-only was a gap; re-attributed, not deleted. Output +% parity is in tests/test_plawk_end_only.pl (field_and_nf_of_last_record); here we +% pin the status flip in the suite that first pinned this shape. +test(end_only_program_field_read_now_compiles) :- + build_status("END { print $1 }\n", 0), !. % --- regressions: END without fields is unchanged ------------------------ diff --git a/tests/test_plawk_end_if_nr.pl b/tests/test_plawk_end_if_nr.pl index 183fe43a8..3fac57695 100644 --- a/tests/test_plawk_end_if_nr.pl +++ b/tests/test_plawk_end_if_nr.pl @@ -121,9 +121,14 @@ build_status("{ n++ } END { if (3 == NR) print \"yes\" }\n", 3), !. -% END-only programs (no rule) decline entirely -- separate driver gap. -test(end_only_nr_if_declines) :- - build_status("END { if (NR == 3) print \"yes\" }\n", 3), +% END-only programs now COMPILE -- the END-only driver landed. `END { if (NR == 3) ... }` +% builds and matches gawk (prints "yes" on a 3-record input). This was pinned as a +% decline while END-only was a gap; re-attributed, not deleted, when the gap closed. +% Output parity is covered in tests/test_plawk_end_only.pl (end_if_no_rule); here we +% pin the status flip so a regression that re-broke the END-only driver shows up in +% the suite that first pinned this shape. +test(end_only_nr_if_now_compiles) :- + build_status("END { if (NR == 3) print \"yes\" }\n", 0), !. % Assoc rules plus this scalar END-if select no driver: the assoc END-if route admits diff --git a/tests/test_plawk_end_only.pl b/tests/test_plawk_end_only.pl new file mode 100644 index 000000000..9d7ec7216 --- /dev/null +++ b/tests/test_plawk_end_only.pl @@ -0,0 +1,377 @@ +:- encoding(utf8). +% SPDX-License-Identifier: MIT OR Apache-2.0 +% Copyright (c) 2026 John William Creighton (@s243a) +% +% END-only programs: a program whose only action block is `END { ... }`, with no +% main rule. `END { print "done" }` and the like used to decline (exit 3) for every +% form; new users hit this on the simplest possible awk one-liners. +% +% --------------------------------------------------------------------------- +% AN END-ONLY PROGRAM STILL READS ALL INPUT +% +% The subtle part, and the reason this is not "skip the loop": an END block sees +% NR (total record count), $0/$N (the LAST record), NF and length -- so the record +% loop must still run, consuming all of stdin, counting NR and retaining the last +% record, with only the per-record RULE CHAIN empty. On empty input NR is 0, $0 is +% empty, NF is 0. gawk 5.x is the oracle for all of this. +% +% This is why an END-only program is the existing rules+END driver with an empty +% rule chain, not the BEGIN-only driver (which reads NO input). The fix was three +% coordinated pieces, each gated on the rules being empty so no program with rules +% is touched (20/20 golden-corpus programs byte-identical): +% +% 1. plawk_scalar_state_plan/3 admits an empty plan when Rules == [] (its guard +% otherwise requires some action / print / getline, none of which a rule-less +% program has). +% 2. a dedicated plawk_scalar_rule_chain_ir([], ...) clause emits +% `br label %continue_loop` -- the terminator the empty `lowered_match:` block +% needs. Without it the general clause emits an unterminated block: invalid +% LLVM, an exit-4 clang failure. The general clause keeps its `RuleCount > 0`. +% 3. the END-loop driver clause is guarded `Rules0 \== []`, so `END { while ... }` +% with no rule DECLINES cleanly (exit 3) instead of committing past its cut and +% miscompiling -- END-loop-with-no-rule is a follow-on, not part of this change. +% +% The distinction that matters most below: a supported END-only form must COMPILE +% and match gawk; an unsupported one must DECLINE cleanly (exit 3), never exit 4. + +:- use_module(library(plunit)). +:- use_module(library(process)). +:- use_module(library(filesex), [make_directory_path/1]). +:- use_module('../examples/plawk/parser/plawk_parser'). +:- use_module('../examples/plawk/codegen/llvm/plawk_native_codegen'). + +clang_available :- + catch(( process_create(path(clang), ['--version'], + [stdout(null), stderr(null), process(Pid)]), + process_wait(Pid, exit(0)) ), _, fail). + +% Three records; the LAST is "7 disk" (NR 3, $1 7, $2 disk, $0 "7 disk", NF 2, +% length 6). The last record differs from the first so a test cannot pass by +% reading the wrong record. +input("5 boot\n5 trace\n7 disk\n"). + +:- begin_tests(plawk_end_only). + +% --- the simplest END-only programs, which used to decline -------------- + +test(constant_print, [condition(clang_available)]) :- + run("END { print \"done\" }\n", "done\n"), + !. + +test(two_constants, [condition(clang_available)]) :- + run("END { print \"a\", \"b\" }\n", "a b\n"), + !. + +test(arithmetic, [condition(clang_available)]) :- + run("END { print 1 + 2 }\n", "3\n"), + !, + run("END { print 3 % 2 }\n", "1\n"), + !, + run("END { print -1 }\n", "-1\n"), + !. + +% --- NR: the record count, which needs the loop to have run -------------- + +test(nr_is_the_record_count, [condition(clang_available)]) :- + run("END { print NR }\n", "3\n"), + !. + +test(nr_on_empty_input_is_zero, [condition(clang_available)]) :- + run_with("", "END { print NR }\n", "0\n"), + !. + +test(nr_in_a_concat, [condition(clang_available)]) :- + run("END { print \"n=\" NR }\n", "n=3\n"), + !. + +% --- the retained last record: $0 / $N / NF / length -------------------- +% +% These are the reason an END-only program is not "skip the loop": the record loop +% must retain the last record for END to project from it. That capability already +% existed (END field reads); this confirms it composes with an empty rule chain. + +test(whole_last_record, [condition(clang_available)]) :- + run("END { print $0 }\n", "7 disk\n"), + !. + +test(field_and_nf_of_last_record, [condition(clang_available)]) :- + run("END { print $1, NF }\n", "7 2\n"), + !. + +test(length_of_last_record, [condition(clang_available)]) :- + run("END { print length }\n", "6\n"), + !, + run("END { print length($0) }\n", "6\n"), + !. + +test(length_on_empty_input_is_zero, [condition(clang_available)]) :- + run_with("", "END { print length }\n", "0\n"), + !. + +test(nr_field_and_nf_together, [condition(clang_available)]) :- + run("END { print NR, $1, NF }\n", "3 7 2\n"), + !. + +% bare `print` in END is `print $0` of the last record. +test(bare_print_is_whole_last_record, [condition(clang_available)]) :- + run("END { print }\n", "7 disk\n"), + !. + +% --- printf, and the multi-statement END list (clause 851) -------------- + +test(printf_in_end_only, [condition(clang_available)]) :- + run("END { printf \"%d\\n\", NR }\n", "3\n"), + !, + run("END { printf \"%s\\n\", $1 }\n", "7\n"), + !. + +test(multi_statement_end, [condition(clang_available)]) :- + run("END { print \"a\"; print \"b\" }\n", "a\nb\n"), + !, + run("END { print NR; print $1 }\n", "3\n7\n"), + !. + +% --- END-if with no rule (came along on the same state-plan path) ------- + +test(end_if_no_rule, [condition(clang_available)]) :- + run("END { if (NR == 3) print \"x\" }\n", "x\n"), + !. + +test(end_if_else_reads_the_record, [condition(clang_available)]) :- + run("END { if (NR > 1) print $1; else print \"few\" }\n", "7\n"), + !. + +test(end_if_on_empty_input, [condition(clang_available)]) :- + run_with("", "END { if (NR == 0) print \"empty\" }\n", "empty\n"), + !. + +% --- the empty program: reads all input, prints nothing ----------------- +% +% A scope expansion that fell out of admitting an empty rule chain, and it is +% correct gawk behaviour (`awk ''` drains input, prints nothing, exits 0). Pinned +% as intended rather than left unremarked. + +test(empty_program_reads_input_prints_nothing, [condition(clang_available)]) :- + run("\n", ""), + !, + run_with("", "\n", ""), + !. + +% --- REGRESSIONS: programs WITH rules are untouched --------------------- + +test(rules_with_end_unchanged, [condition(clang_available)]) :- + run("{ n++ } END { print n }\n", "3\n"), + !, + run("{ n++; c[$1]++ } END { print NR, c[\"5\"] }\n", "3 2\n"), + !. + +test(begin_only_unchanged, [condition(clang_available)]) :- + run("BEGIN { print \"hi\" }\n", "hi\n"), + !. + +test(rule_only_unchanged, [condition(clang_available)]) :- + run("{ print $1 }\n", "5\n5\n7\n"), + !. + +% END-loop WITH a rule still compiles -- the guard is on empty rules only. +test(end_loop_with_a_rule_still_compiles, [condition(clang_available)]) :- + build_status("{ n++ } END { while (i < n) { print i; i++ } }\n", 0), + !. + +% --- BOUNDARIES: unsupported END-only forms DECLINE, never miscompile --- + +% The miscompile that this change had to avoid: END-loop with no rule. It must +% decline cleanly (exit 3), NOT emit invalid LLVM (exit 4). This is the sharpest +% pin in the suite -- the whole END-loop clause guard exists for it. +test(end_only_loop_declines_cleanly_not_exit_4) :- + build_status("END { while (i < 3) { print i; i++ } }\n", 3), + !. + +% END-only with a scalar ASSIGNMENT is a follow-on (the assignment machinery is +% not wired for an END-only context yet). Declines, exit 3. +test(end_only_assignment_declines) :- + build_status("END { x = 5; print x }\n", 3), + !. + +% for-in over an (empty) array in END-only: declines, exit 3. +test(end_only_forin_declines) :- + build_status("END { for (k in a) print k }\n", 3), + !. + +% builtins over the retained record in END are a separate follow-on (the LITERAL +% forms fold; these read the record). Decline, exit 3. +test(end_only_record_builtins_decline) :- + build_status("END { print substr($0, 1, 3) }\n", 3), + !, + build_status("END { print toupper($1) }\n", 3), + !. + +% getline in an END block is unsupported: declines, exit 3. +test(end_only_getline_declines) :- + build_status("END { getline x; print x }\n", 3), + !. + +% A scalar VARIABLE read/write in an END-only program declines cleanly (exit 3), NOT +% exit 4. This is the case an ultra review caught that the first cut missed: admitting +% an empty rule chain let the state plan collect the scalar as a slot, and the loop's +% next-slot phi then referenced %rule_-1_* (LastRuleIndex = RuleCount-1 = -1) -- +% undefined SSA, a clang miscompile. The dedicated empty rule-chain clause is now +% guarded to fire only when the state plan has NO slots, so a scalar-var END-only +% program declines. It is a genuine follow-on: an unset scalar's value in END is +% context-dependent (empty in string context, 0 in numeric -- the uninitialised-scalar +% representation problem), so even a correct pass-through phi needs that settled first. +% Pinned so the miscompile cannot silently return and the follow-on stays visible. +test(end_only_scalar_var_read_declines_not_miscompiles) :- + build_status("END { print x }\n", 3), + build_status("END { print x; print \"done\" }\n", 3), + build_status("END { printf \"%d\\n\", x }\n", 3), + build_status("END { if (x == 0) print \"zero\" }\n", 3), + build_status("END { if (NR > 0) print x }\n", 3), + build_status("END { print \"n=\" x }\n", 3), + build_status("END { print x + 1 }\n", 3), + build_status("END { if (NR == 0) print \"empty\"; else print x }\n", 3), + build_status("BEGIN { BINFMT=\"case(i64 | i64)\" } END { print x }\n", 3), + !. + +% --- clang never rejects a supported END-only program ------------------- +% +% Belt-and-braces over the whole supported surface: build each and require a +% binary. An exit-4 here would mean invalid LLVM slipped through; that is the +% failure the dedicated rule-chain clause exists to prevent. +test(no_supported_end_only_form_miscompiles, [condition(clang_available)]) :- + forall(member(Src, [ + "END { print \"done\" }\n", + "END { print NR }\n", + "END { print $0 }\n", + "END { print $1, NF }\n", + "END { print length }\n", + "END { printf \"%d\\n\", NR }\n", + "END { print \"a\"; print \"b\" }\n", + "END { if (NR == 3) print $1; else print NF }\n" + ]), build_status(Src, 0)), + !. + +% --- clause enumeration: NO rule-less shape miscompiles (exit 4) --------- +% +% The sharpest risk in this change: relaxing the shared state-plan guard could let a +% rule-less program COMMIT past a cut in some driver clause other than the ones tested +% above and emit invalid LLVM. This pins the enumeration -- one rule-less program routed +% at each END-driver clause type (scalar print/list/if, loop, MIXED and ASSOC ends whose +% RuleCount>0 gates were deliberately NOT relaxed, BEGIN+end, binfmt, getline/redirect/ +% assignment). Every one must build to a real exit code (0 compile / 2 parse / 3 decline) +% and NEVER 4. The mixed/assoc cases are the load-bearing ones: they confirm the two +% unrelaxed gates keep those clauses declining instead of committing on empty rules. +test(no_rule_less_shape_miscompiles) :- + forall(member(Src, [ + "END { print \"x\" }\n", + "END { print \"a\"; print \"b\" }\n", + "END { if (NR == 3) print \"x\" }\n", + "END { while (i < 3) { print i; i++ } }\n", + "END { print c[\"x\"] }\n", + "END { print c[\"x\"], NR }\n", + "END { for (k in c) print k }\n", + "END { c[\"x\"]++; print c[\"x\"] }\n", + "BEGIN { FS=\":\" } END { print NR }\n", + "BEGIN { print \"start\" } END { print NR }\n", + "END { getline x }\n", + "END { x = 5 }\n", + "END { print x }\n", + "END { print x; print \"done\" }\n", + "END { printf \"%d\\n\", x }\n", + "END { if (x == 0) print \"zero\" }\n", + "END { if (NR > 0) print x }\n", + "BEGIN { FS=\":\" } END { print x }\n", + "BEGIN { BINFMT = \"i64\" } END { print x }\n", + "END { print NR > \"/dev/stdout\" }\n" + ]), build_status_not_4(Src)), + !. + +% --- the empty rule chain lowers to a single loop-continue branch ------- +% +% The IR property that keeps the fix additive: with no rules, the record loop's +% match block is just a branch to continue_loop. Pinned on the emitted IR so a +% future change that reintroduces per-rule scaffolding here shows up. +test(empty_rule_chain_is_a_continue_branch) :- + build_ll("END { print \"done\" }\n", LL), + assertion(sub_string(LL, _, _, _, "br label %continue_loop")), + % ...and no rule-match scaffolding, since there are no rules. + assertion(\+ sub_string(LL, _, _, _, "rule_0_match")), + !. + +:- end_tests(plawk_end_only). + +% --- helpers --------------------------------------------------------------- + +odir(Dir) :- + current_prolog_flag(tmp_dir, Tmp), + directory_file_path(Tmp, 'uw_plawk_end_only', Dir), + ( exists_directory(Dir) -> true ; make_directory_path(Dir) ). + +run(Src, Expected) :- + input(Input), + run_with(Input, Src, Expected). + +run_with(Input, Src, Expected) :- + odir(Dir), + directory_file_path(Dir, 'eo_bin', Bin), + ( exists_file(Bin) -> delete_file(Bin) ; true ), + directory_file_path(Dir, 'eo', Prog0), + atom_concat(Prog0, '.plawk', Prog), + setup_call_cleanup(open(Prog, write, S, [encoding(utf8)]), + write(S, Src), close(S)), + atom_concat(Prog0, '_in.txt', In), + setup_call_cleanup(open(In, write, SI, [encoding(utf8)]), + write(SI, Input), close(SI)), + cli([build, Prog, '-o', Bin], 0), + process_create(Bin, [In], [stdout(pipe(PS)), stderr(std), process(Pid)]), + read_string(PS, _, Out), + close(PS), + process_wait(Pid, exit(0)), + ( Out == Expected + -> true + ; format(user_error, "~n~w~n got ~q~n expected ~q~n", + [Src, Out, Expected]), fail + ). + +build_status(Src, ExpectedStatus) :- + odir(Dir), + directory_file_path(Dir, 'eo_reject', Prog0), + atom_concat(Prog0, '.plawk', Prog), + setup_call_cleanup(open(Prog, write, S, [encoding(utf8)]), + write(S, Src), close(S)), + atom_concat(Prog0, '_bin', Bin), + ( exists_file(Bin) -> delete_file(Bin) ; true ), + cli([build, Prog, '-o', Bin], ExpectedStatus). + +% Build and assert the exit code is anything but 4 (a clang miscompile). Used by the +% clause-enumeration test: a rule-less shape may compile, parse-fail, or decline, but +% must never produce invalid LLVM. +build_status_not_4(Src) :- + odir(Dir), + directory_file_path(Dir, 'eo_no4', Prog0), + atom_concat(Prog0, '.plawk', Prog), + setup_call_cleanup(open(Prog, write, S, [encoding(utf8)]), + write(S, Src), close(S)), + atom_concat(Prog0, '_bin', Bin), + ( exists_file(Bin) -> delete_file(Bin) ; true ), + process_create(path(swipl), ['examples/plawk/bin/plawk', build, Prog, '-o', Bin], + [stdout(pipe(Out)), stderr(std), process(Pid)]), + read_string(Out, _, _), close(Out), + process_wait(Pid, exit(Status)), + ( memberchk(Status, [0, 2, 3]) + -> true + ; format(user_error, "~nUNEXPECTED STATUS ~w (want 0/2/3, never 4): ~w~n", + [Status, Src]), fail + ). + +build_ll(Src, LL) :- + plawk_parse_string(Src, Program), + plawk_program_native_driver_ir(Program, 'input.txt', IR), + atom_string(IR, LL). + +cli(Args, ExpectedStatus) :- + process_create(path(swipl), ['examples/plawk/bin/plawk' | Args], + [stdout(pipe(S)), stderr(std), process(Pid)]), + read_string(S, _, _), close(S), + process_wait(Pid, exit(Status)), + assertion(Status == ExpectedStatus).