Skip to content

feat: handle trailing comments at EOF and complete EOF tolerance - #59

Merged
tesujimath merged 1 commit into
tesujimath:mainfrom
geek5nan:pr/eol-comment-eof
Apr 7, 2026
Merged

feat: handle trailing comments at EOF and complete EOF tolerance#59
tesujimath merged 1 commit into
tesujimath:mainfrom
geek5nan:pr/eol-comment-eof

Conversation

@geek5nan

@geek5nan geek5nan commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR addresses two related issues with files that don't end with a newline:

1. Trailing comments at EOF produce error tokens

The lex() function's final_eol mechanism (line 306) indicates the intent to support files without trailing newlines — it injects a synthetic Eol when the token stream ends. However, when the last line contains an inline comment (; ...) without a following \n, the ; cannot match Eol (which requires \n) and becomes an error token, defeating final_eol.

Fix: Add an EolComment token (#[regex(r";[^\r\n]*")]) that matches trailing comments at EOF. Logos longest-match ensures Eol still wins when \n is present. EolIndentHandler converts EolComment to a zero-width Eol before it reaches the parser, maintaining full transparency.

2. Complete EOF tolerance in parsers (extends #54)

PR #54 added choice((Eol, end())) at 4 structural points. This PR extends the same pattern to the remaining 11 directive types (balance, price, open, close, event, note, document, pad, query, custom, commodity) for consistent behavior across all directives.

3. Harden resolve_span against invalid byte ranges

resolve_span used direct slice indexing (source_content[byte_span.start..byte_span.end]) which can panic on invalid byte boundaries. Changed to .get().unwrap_or("") for safety.

Motivation

  • The final_eol mechanism already demonstrates the intent to handle missing trailing newlines
  • The Beancount Python parser accepts all these cases
  • 5 test cases previously marked as ANOMALY now pass correctly

Test plan

  • All 335 existing tests pass
  • 5 ANOMALY test cases (CommentEOF, ExtraWhitespaceComment, ExtraWhitespaceCommentIndented, ExtraWhitespaceTransaction, NoFinalNewline) now parse successfully
  • Verified with real-world ledger files containing inline comments and missing trailing newlines

Add EolComment token to match `;...` at EOF (without trailing newline),
preventing error tokens that break the final_eol mechanism.
EolIndentHandler converts EolComment to zero-width Eol for UTF-8 safety.

Also complete EOF tolerance in parsers.rs (11 remaining directive types
not covered by tesujimath#54) and harden resolve_span against invalid byte ranges.
@tesujimath
tesujimath merged commit 51c580f into tesujimath:main Apr 7, 2026
4 checks passed
@tesujimath

Copy link
Copy Markdown
Owner

This is great, thanks!

@geek5nan
geek5nan deleted the pr/eol-comment-eof branch April 8, 2026 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants