fix(parser): stop the tokenizer hanging and overflowing on hostile input

[ferentinai]

Two ways a short, malformed command line could take the process down rather than
return an error. Both are in the tokenizer, both are reachable from a single
string, and neither depends on anything unusual about the environment.

I found these while evaluating brush-parser for a tool that parses command
lines it did not write, so untrusted input on a deadline is the case I care
about. They will matter less for interactive use, where you are already running
the shell, but they are cheap to fix either way.

1. Unterminated here tag with an unterminated expansion hangs and allocates forever

$ printf '<<E$[\t\t' | your-parser-harness

Seven bytes. Before this change that reaches roughly 10 GB resident within
seconds on my machine and never returns; I only ever saw it end by being killed.

next_token_until looks for a here-document end tag whenever the here state
is anything other than None. In NextLineIsHereDoc the body has not started,
so there is no end tag to find, but remove_here_end_tag still succeeds against
the empty current token. That path goes to the delimit_current_token branch
that only pushes onto pending_tokens_after and returns Ok(None), so no token
is produced, no here tag is consumed, and the state is exactly what it was. The
loop goes round again and allocates again.

consume_nested_construct then makes it unbounded rather than merely stuck,
because it accumulates each of those here-document tokens into
pending_here_doc_tokens.

The fix only attempts the end-tag lookup while we are actually inside a
here-document body. The input above now reports an unterminated here document.

For what it is worth, bash -n also rejects it:

$ printf '<<E$[\t\t' > t.sh && bash -n t.sh
t.sh: line 1: unexpected EOF while looking for matching `]'
t.sh: line 2: syntax error: unexpected end of file

Variants that hit the same path: $[, $( and $(( as the unterminated
expansion, <<- as well as <<. Curiously it needs exactly two trailing blanks;
one is fine and so are three, which is what made it look so arbitrary before the
cause was clear.

2. Nested expansions recurse without bound and abort on stack overflow

$ printf 'echo %s%s' "$(printf '$(%.0s' {1..8000})" "$(printf ')%.0s' {1..8000})" | your-parser-harness

consume_nested_construct and next_token_until are mutually recursive, and the
input alone decides how deep that goes. On my machine it aborts somewhere between
6500 and 7000 levels on an 8 MB main thread, but under 2000 on a 2 MB one, which
is a command line of about 4 KB. Where the parse happens changes the threshold by
roughly 4x, which is worth knowing if a caller parses on a worker thread.

This adds a depth bound and a TokenizerError::ExpansionNestingTooDeep rather
than letting it run into the guard page. I picked the limit by measurement rather
than taste: an unoptimized build uses roughly 8 KB of stack per level, so 64
levels stays around half a megabyte and is comfortable even on a small thread
stack. My first attempt at 256 still overflowed a 2 MB debug test thread, which
is how I arrived at the number.

If you would rather have this configurable through TokenizerOptions, or set to
a different value, say the word and I will rework it.

Not the same as #948

Worth stating explicitly since the symptom is identical. #948
(nproc(){ nproc; } then echo $(nproc)) is runtime recursion through a
self-referencing function during expansion, and it still reproduces with this
change applied. It is a different code path and this PR does not claim to fix it.

Testing

• cargo test -p brush-parser: 227 passed, 0 failed. That is the existing 224
  plus 3 added here.
• cargo clippy -p brush-parser --all-targets: clean under the crate's existing
  lint configuration.
• cargo test -p brush-shell --test brush-compat-tests: 1388 succeeded, 405
  failed, 366 known to fail, 43 skipped, which is byte for byte what a pristine
  checkout produces on the same machine. The here-document change is the risky
  one, so I ran that baseline specifically to show it moves nothing.
• A structured fuzzer over shell metacharacters, 500,000 iterations, found the
  first of these bugs originally and reports no hangs or panics afterwards.

Added tests cover the hanging inputs, the depth bound firing, and nesting within
the limit still tokenizing normally.

Assisted-by: Claude Opus 5 (Claude Code)

[github-actions]

Public API changes for crate: brush-parser

Removed items

-impl brush_parser::ast::AssignmentName
-pub fn brush_parser::ast::AssignmentName::base_name(&self) -> &str
-pub brush_parser::ast::CompoundCommand::ExtendedTest(brush_parser::ast::ExtendedTestExprCommand)
-pub fn brush_parser::word::parse_compound_assignment_value(&str, &brush_parser::ParserOptions) -> core::option::Option<alloc::vec::Vec<(core::option::Option<brush_parser::ast::Word>, brush_parser::ast::Word)>>
-pub fn brush_parser::word::parse_scalar_assignment(&str, &brush_parser::ParserOptions) -> core::result::Result<brush_parser::ast::Assignment, brush_parser::WordParseError>

Added items

+pub brush_parser::ast::Command::ExtendedTest(brush_parser::ast::ExtendedTestExprCommand, core::option::Option<brush_parser::ast::RedirectList>)
+pub brush_parser::TokenizerError::ExpansionNestingTooDeep(u32)

Performance Benchmark Report

Benchmark name	Baseline (μs)	Test/PR (μs)	Delta (μs)	Delta %
clone_shell_object	17.50 μs	17.48 μs	-0.02 μs	⚪ Unchanged
eval_arithmetic	0.15 μs	0.15 μs	-0.00 μs	⚪ Unchanged
expand_one_string	1.66 μs	1.64 μs	-0.02 μs	⚪ Unchanged
for_loop	31.34 μs	31.46 μs	0.11 μs	⚪ Unchanged
full_peg_complex	59.78 μs	58.41 μs	-1.37 μs	⚪ Unchanged
full_peg_for_loop	6.30 μs	6.20 μs	-0.10 μs	⚪ Unchanged
full_peg_nested_expansions	16.48 μs	16.54 μs	0.06 μs	⚪ Unchanged
full_peg_pipeline	4.37 μs	4.24 μs	-0.13 μs	🟢 -3.00%
full_peg_simple	1.86 μs	1.79 μs	-0.07 μs	🟢 -3.65%
function_call	3.48 μs	3.49 μs	0.01 μs	⚪ Unchanged
instantiate_shell	55.05 μs	54.81 μs	-0.24 μs	⚪ Unchanged
instantiate_shell_with_init_scripts	26362.55 μs	26987.91 μs	625.36 μs	⚪ Unchanged
parse_peg_bash_completion	2142.94 μs	2110.30 μs	-32.64 μs	⚪ Unchanged
parse_peg_complex	21.30 μs	20.17 μs	-1.13 μs	🟢 -5.31%
parse_peg_for_loop	2.08 μs	2.02 μs	-0.05 μs	🟢 -2.60%
parse_peg_pipeline	2.18 μs	2.05 μs	-0.13 μs	🟢 -5.78%
parse_peg_simple	1.16 μs	1.09 μs	-0.07 μs	🟢 -6.28%
run_echo_builtin_command	16.68 μs	17.05 μs	0.37 μs	⚪ Unchanged
tokenize_sample_script	3.46 μs	3.46 μs	-0.00 μs	⚪ Unchanged

Code Coverage Report: Only Changed Files listed

Package	Base Coverage	New Coverage	Difference
brush-builtins/src/trap.rs	🟢 86.57%	🟢 98.51%	🟢 11.94%
brush-core/src/expansion.rs	🟢 96.17%	🟢 96.92%	🟢 0.75%
brush-core/src/traps.rs	🟢 81.82%	🟢 86.87%	🟢 5.05%
brush-parser/src/ast.rs	🔴 48.14%	🔴 48.43%	🟢 0.29%
brush-parser/src/parser/mod.rs	🟠 69.53%	🟠 66.95%	🔴 -2.58%
brush-parser/src/parser/peg.rs	🟢 95.07%	🟢 95.04%	🔴 -0.03%
brush-parser/src/tokenizer.rs	🟢 93.7%	🟢 93.76%	🟢 0.06%
brush-parser/src/word.rs	🟢 94.02%	🟢 93.82%	🔴 -0.2%
Overall Coverage	🟢 76.15%	🟢 76.24%	🟢 0.09%

Minimum allowed coverage is 70%, this run produced 76.24%
Maximum allowed coverage difference is -5%, this run produced 0.09%

Test Summary: bash-completion test suite

Outcome	Count	Percentage
✅ Pass	1582	75.01
❗️ Error	17	0.81
❌ Fail	156	7.40
⏩ Skip	339	16.07
❎ Expected Fail	13	0.62
✔️ Unexpected Pass	2	0.09
📊 Total	2109	100.00