From 7642203354bc1465225a23a5802cd4836a883487 Mon Sep 17 00:00:00 2001 From: David Sisco Date: Sat, 10 Jan 2026 20:37:01 -0800 Subject: [PATCH 1/8] docs: todo list --- green-node-flags-refactor.todo | 154 +++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 green-node-flags-refactor.todo diff --git a/green-node-flags-refactor.todo b/green-node-flags-refactor.todo new file mode 100644 index 0000000..0da0ffe --- /dev/null +++ b/green-node-flags-refactor.todo @@ -0,0 +1,154 @@ +todo + +# Green Node Flags Refactor (Precursor) + +Goal: add a compact flags bitmask to green nodes so newline/comment/whitespace/error/keyword/tagged-ident checks are O(1) and enable subtree pruning. + +Scope: +- Precursor to the Query newline optimization plan. +- Green layer only (TinyTokenizer/TinyTokenizer/Ast/Green*). No query changes in this work item. + +Constraints: +- Green nodes remain immutable + structurally shareable. +- Flags must be deterministically derived from immutable content (trivia + children), never mutated. +- Preserve semantics: newline flags reflect `TriviaKind.Newline` only. + +------------------------------------------------------------------------ + +## Phase 0 — Decide Flag Semantics (one-time) + +- [ ] Confirm high-ROI flag set (v1) + - [ ] Boundary trivia flags + - [ ] `HasLeadingNewlineTrivia` / `HasTrailingNewlineTrivia` + - [ ] `HasLeadingCommentTrivia` / `HasTrailingCommentTrivia` + - [ ] `HasLeadingWhitespaceTrivia` / `HasTrailingWhitespaceTrivia` + - [ ] Subtree flags + - [ ] `ContainsNewlineTrivia` + - [ ] `ContainsCommentTrivia` + - [ ] `ContainsWhitespaceTrivia` + - [ ] `ContainsErrorNode` + - [ ] `ContainsKeyword` + - [ ] `ContainsTaggedIdent` + +- [ ] Confirm boundary meaning for containers + - [ ] “Leading” refers to the left boundary of the node’s text span + - [ ] “Trailing” refers to the right boundary of the node’s text span + - [ ] For containers, boundary flags derive from first/last leaf boundaries (not necessarily immediate delimiter leaves for all container types) + +Acceptance criteria: +- Flag list and boundary semantics are written down and stable. + +------------------------------------------------------------------------ + +## Phase 1 — Add Flags Infrastructure + +- [ ] Add `GreenNodeFlags` enum (bitmask) + - [ ] Pick underlying type (`ushort` vs `uint`), reserve room for future bits + - [ ] Keep naming consistent: `HasLeadingX` / `HasTrailingX` / `ContainsX` + +- [ ] Add `Flags` to `GreenNode` + - [ ] Add `internal GreenNodeFlags Flags { get; }` + - [ ] Add lightweight convenience bools (optional) + +- [ ] Confirm record/equality impact approach + - [ ] Ensure `Flags` is deterministically derived from other fields + - [ ] Decide whether `Flags` should participate in record equality (default) or be excluded (custom) + +Acceptance criteria: +- Green nodes compile with a flags field/property available from the base type. + +------------------------------------------------------------------------ + +## Phase 2 — Compute Flags in Leaves + +- [ ] Update `GreenLeaf` constructor + - [ ] Scan `LeadingTrivia` once to set boundary flags + - [ ] Scan `TrailingTrivia` once to set boundary flags + - [ ] Set subtree flags based on boundary results + - [ ] Ensure no extra passes beyond existing trivia width computation (piggyback loops) + +- [ ] Verify `GreenNodeCache` behavior is still correct + - [ ] Cached leaves with no trivia have stable flags (`None`) + - [ ] Cached “trailing space” leaves have whitespace flags but no newline/comment + - [ ] Leaves with trivia remain non-cached (unchanged behavior) + +Acceptance criteria: +- Leaf flags are correct for leading/trailing trivia kinds and don’t add noticeable allocations. + +------------------------------------------------------------------------ + +## Phase 3 — Aggregate Flags in Containers + +- [ ] Update `GreenBlock` + - [ ] OR flags from opener leaf + all inner children + closer leaf + - [ ] Ensure boundary flags reflect the block’s actual left/right boundaries + +- [ ] Update `GreenList` + - [ ] OR flags from all children + - [ ] Derive boundary flags from first/last child when present + +- [ ] Update `GreenSyntaxNode` + - [ ] OR flags from all children + - [ ] Derive boundary flags from first/last child when present + +- [ ] Validate structural sharing invariants + - [ ] `WithSlot`/`WithInsert`/`WithReplace` cause flags to recompute via constructors (no ad-hoc flag mutation) + - [ ] `WithLeadingTrivia`/`WithTrailingTrivia`/`WithText` on leaves continue to work + +Acceptance criteria: +- Container flags correctly reflect subtree + boundary properties and remain deterministic. + +------------------------------------------------------------------------ + +## Phase 4 — Tests: Flag Correctness + +- [ ] Add/extend unit tests for flag correctness + - [ ] Leaf boundary flags + - [ ] Newline trivia in leading/trailing + - [ ] Comment trivia in leading/trailing + - [ ] Whitespace trivia in leading/trailing + - [ ] Aggregation + - [ ] Block contains newline/comment/whitespace when present in delimiters or inner children + - [ ] List/syntax node aggregation + - [ ] “Newline ownership” sanity (existing lexer trivia semantics) + - [ ] Newline appears as trailing trivia of previous token in common cases + +Acceptance criteria: +- Tests reliably catch flag computation regressions. + +------------------------------------------------------------------------ + +## Phase 5 — Tests: Record Equality / Hash Stability (extra safety) + +- [ ] Equality invariants (structural equality) + - [ ] Construct equivalent green nodes via different construction paths and assert `Equals` is true + - [ ] Assert `GetHashCode` matches for structurally equal nodes + +- [ ] Cache interaction + - [ ] Cached leaf instances behave consistently (equal nodes produce equal hashes) + - [ ] Non-cached leaves with identical trivia sequences still compare equal + +- [ ] Structural sharing sanity + - [ ] `With*` methods do not mutate originals + - [ ] Only content changes cause inequality + +Acceptance criteria: +- We have strong confidence that adding `Flags` doesn’t introduce equality/hash surprises. + +------------------------------------------------------------------------ + +## Phase 6 — Validation / Perf Smoke + +- [ ] Run `dotnet test TinyTokenizer.Tests` +- [ ] Run `dotnet run -c Release --project TinyTokenizer.Benchmarks -- --filter *AstBenchmarks*` +- [ ] Run `dotnet run -c Release --project TinyTokenizer.Benchmarks -- --filter *SyntaxEditorBenchmarks*` + +Acceptance criteria: +- All tests pass; no obvious performance regressions. + +------------------------------------------------------------------------ + +## Follow-up (out of scope for this .todo) + +- [ ] Update Query newline matching to use flags (separate plan/work item) +- [ ] Optional: add targeted benchmarks for `SyntaxTree.Select(Query.Newline)` once newline query changes land From 883c991f3d360f6669fe1315d0c417ebc64a03f8 Mon Sep 17 00:00:00 2001 From: David Sisco Date: Sat, 10 Jan 2026 20:39:43 -0800 Subject: [PATCH 2/8] [phase 0]: confirm details --- green-node-flags-refactor.todo | 36 +++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/green-node-flags-refactor.todo b/green-node-flags-refactor.todo index 0da0ffe..37d5552 100644 --- a/green-node-flags-refactor.todo +++ b/green-node-flags-refactor.todo @@ -17,7 +17,14 @@ Constraints: ## Phase 0 — Decide Flag Semantics (one-time) -- [ ] Confirm high-ROI flag set (v1) + +Decisions (grounded in current AST model): +- Trivia kinds are exactly: `Whitespace` (spaces/tabs), `Newline` (\n / \r\n), `SingleLineComment`, `MultiLineComment`. +- Keywords are represented as nodes with `NodeKind` in the keyword range (`NodeKindExtensions.IsKeyword()`). +- Tagged identifiers are leaf nodes with `NodeKind.TaggedIdent`. +- Errors are leaf nodes with `NodeKind.Error`. + +- [x] Confirm high-ROI flag set (v1) - [ ] Boundary trivia flags - [ ] `HasLeadingNewlineTrivia` / `HasTrailingNewlineTrivia` - [ ] `HasLeadingCommentTrivia` / `HasTrailingCommentTrivia` @@ -30,10 +37,29 @@ Constraints: - [ ] `ContainsKeyword` - [ ] `ContainsTaggedIdent` -- [ ] Confirm boundary meaning for containers - - [ ] “Leading” refers to the left boundary of the node’s text span - - [ ] “Trailing” refers to the right boundary of the node’s text span - - [ ] For containers, boundary flags derive from first/last leaf boundaries (not necessarily immediate delimiter leaves for all container types) +- [x] Confirm boundary meaning for containers + - [x] “Leading” refers to the left boundary of the node’s text span + - [x] “Trailing” refers to the right boundary of the node’s text span + - [x] For containers, boundary flags derive from first/last leaf boundaries (not necessarily immediate delimiter leaves for all container types) + +Flag semantics (implementation-ready): +- Boundary flags: + - `HasLeadingNewlineTrivia`: node’s left boundary has trivia of kind `Newline`. + - `HasTrailingNewlineTrivia`: node’s right boundary has trivia of kind `Newline`. + - `HasLeadingWhitespaceTrivia`: left boundary has trivia kind `Whitespace` (explicitly excludes `Newline`). + - `HasTrailingWhitespaceTrivia`: right boundary has trivia kind `Whitespace`. + - `HasLeadingCommentTrivia`: left boundary has trivia kind `SingleLineComment` OR `MultiLineComment`. + - `HasTrailingCommentTrivia`: right boundary has trivia kind `SingleLineComment` OR `MultiLineComment`. + - Leaf rule: boundary flags are computed by scanning `GreenLeaf.LeadingTrivia` / `GreenLeaf.TrailingTrivia`. + - Container rule: boundary flags are derived from the first/last leaf descendant boundaries (fast once flags exist). For `GreenBlock`, this effectively comes from opener/closer delimiter leaves. + +- Subtree flags: + - `ContainsNewlineTrivia`: any leaf in the subtree has newline trivia (leading or trailing). + - `ContainsWhitespaceTrivia`: any leaf in subtree has whitespace trivia. + - `ContainsCommentTrivia`: any leaf in subtree has comment trivia. + - `ContainsErrorNode`: any node in subtree has `Kind == NodeKind.Error`. + - `ContainsKeyword`: any node in subtree has `Kind.IsKeyword()`. + - `ContainsTaggedIdent`: any node in subtree has `Kind == NodeKind.TaggedIdent`. Acceptance criteria: - Flag list and boundary semantics are written down and stable. From a19b81c07537d67771a7ddb5b2c91d2d8550d6d3 Mon Sep 17 00:00:00 2001 From: David Sisco Date: Sat, 10 Jan 2026 20:42:29 -0800 Subject: [PATCH 3/8] [phase 1]: Add Flags Infrastructure --- TinyTokenizer/Ast/GreenNode.cs | 6 ++++++ TinyTokenizer/Ast/GreenNodeFlags.cs | 32 +++++++++++++++++++++++++++++ green-node-flags-refactor.todo | 23 +++++++++++++-------- 3 files changed, 52 insertions(+), 9 deletions(-) create mode 100644 TinyTokenizer/Ast/GreenNodeFlags.cs diff --git a/TinyTokenizer/Ast/GreenNode.cs b/TinyTokenizer/Ast/GreenNode.cs index b563cae..a9e65e5 100644 --- a/TinyTokenizer/Ast/GreenNode.cs +++ b/TinyTokenizer/Ast/GreenNode.cs @@ -36,6 +36,12 @@ protected static string Truncate(string text, int maxLength) /// The kind of this node. public abstract NodeKind Kind { get; } + + /// + /// Cached flags describing trivia/content properties for fast queries. + /// Concrete green node types override this once flags are computed. + /// + internal virtual GreenNodeFlags Flags => GreenNodeFlags.None; /// /// Total character width of this node, including any trivia. diff --git a/TinyTokenizer/Ast/GreenNodeFlags.cs b/TinyTokenizer/Ast/GreenNodeFlags.cs new file mode 100644 index 0000000..5896c6b --- /dev/null +++ b/TinyTokenizer/Ast/GreenNodeFlags.cs @@ -0,0 +1,32 @@ +using System; + +namespace TinyTokenizer.Ast; + +/// +/// Bitflags describing trivia and content properties of a green node. +/// Intended for O(1) query checks and subtree pruning. +/// +[Flags] +internal enum GreenNodeFlags : uint +{ + None = 0, + + // Boundary trivia (left/right edge of the node's text span) + HasLeadingNewlineTrivia = 1u << 0, + HasTrailingNewlineTrivia = 1u << 1, + + HasLeadingWhitespaceTrivia = 1u << 2, + HasTrailingWhitespaceTrivia = 1u << 3, + + HasLeadingCommentTrivia = 1u << 4, + HasTrailingCommentTrivia = 1u << 5, + + // Subtree flags (anywhere within the node's subtree) + ContainsNewlineTrivia = 1u << 8, + ContainsWhitespaceTrivia = 1u << 9, + ContainsCommentTrivia = 1u << 10, + + ContainsErrorNode = 1u << 11, + ContainsKeyword = 1u << 12, + ContainsTaggedIdent = 1u << 13, +} diff --git a/green-node-flags-refactor.todo b/green-node-flags-refactor.todo index 37d5552..f872878 100644 --- a/green-node-flags-refactor.todo +++ b/green-node-flags-refactor.todo @@ -68,17 +68,22 @@ Acceptance criteria: ## Phase 1 — Add Flags Infrastructure -- [ ] Add `GreenNodeFlags` enum (bitmask) - - [ ] Pick underlying type (`ushort` vs `uint`), reserve room for future bits - - [ ] Keep naming consistent: `HasLeadingX` / `HasTrailingX` / `ContainsX` - -- [ ] Add `Flags` to `GreenNode` - - [ ] Add `internal GreenNodeFlags Flags { get; }` +- [x] Add `GreenNodeFlags` enum (bitmask) + - [x] Pick underlying type (`ushort` vs `uint`), reserve room for future bits + - Decision: use `uint` for headroom (we already need ~12 bits; `uint` keeps growth cheap). + - [x] Keep naming consistent: `HasLeadingX` / `HasTrailingX` / `ContainsX` + +- [x] Add `Flags` to `GreenNode` + - [x] Add `internal GreenNodeFlags Flags { get; }` + - Implemented as `internal virtual GreenNodeFlags Flags => GreenNodeFlags.None;` initially. + - Phase 2/3 will override in concrete node types after computing flags. - [ ] Add lightweight convenience bools (optional) -- [ ] Confirm record/equality impact approach - - [ ] Ensure `Flags` is deterministically derived from other fields - - [ ] Decide whether `Flags` should participate in record equality (default) or be excluded (custom) +- [x] Confirm record/equality impact approach + - [x] Ensure `Flags` is deterministically derived from other fields + - Decision: flags must be computed solely from trivia/kind/children (no external state). + - [x] Decide whether `Flags` should participate in record equality (default) or be excluded (custom) + - Decision: keep default record equality; rely on deterministic computation + Phase 5 tests. Acceptance criteria: - Green nodes compile with a flags field/property available from the base type. From ee24be65eef797dd7e8aba24c506520497e264ec Mon Sep 17 00:00:00 2001 From: David Sisco Date: Sat, 10 Jan 2026 20:46:37 -0800 Subject: [PATCH 4/8] [phase 2]: Compute Flags in Leaves --- TinyTokenizer/Ast/GreenLeaf.cs | 78 ++++++++++++++++++++++++++++++++-- green-node-flags-refactor.todo | 24 ++++++----- 2 files changed, 88 insertions(+), 14 deletions(-) diff --git a/TinyTokenizer/Ast/GreenLeaf.cs b/TinyTokenizer/Ast/GreenLeaf.cs index a53f030..bc1f0a6 100644 --- a/TinyTokenizer/Ast/GreenLeaf.cs +++ b/TinyTokenizer/Ast/GreenLeaf.cs @@ -17,9 +17,13 @@ internal sealed record GreenLeaf : GreenNode $"{Kind}[{Width}] \"{Truncate(Text, 20)}\""; private readonly int _width; + private readonly GreenNodeFlags _flags; /// public override NodeKind Kind { get; } + + /// + internal override GreenNodeFlags Flags => _flags; /// The text content of this token (excluding trivia). public string Text { get; } @@ -61,10 +65,31 @@ public GreenLeaf( Text = text; LeadingTrivia = leadingTrivia.IsDefault ? ImmutableArray.Empty : leadingTrivia; TrailingTrivia = trailingTrivia.IsDefault ? ImmutableArray.Empty : trailingTrivia; - - LeadingTriviaWidth = ComputeTriviaWidth(LeadingTrivia); - TrailingTriviaWidth = ComputeTriviaWidth(TrailingTrivia); + + LeadingTriviaWidth = ComputeTriviaWidthAndFlags( + LeadingTrivia, + isLeading: true, + out var leadingBoundaryFlags, + out var leadingContainsFlags); + + TrailingTriviaWidth = ComputeTriviaWidthAndFlags( + TrailingTrivia, + isLeading: false, + out var trailingBoundaryFlags, + out var trailingContainsFlags); + _width = LeadingTriviaWidth + Text.Length + TrailingTriviaWidth; + + // Subtree flags based on node kind + var kindFlags = GreenNodeFlags.None; + if (kind == NodeKind.Error) + kindFlags |= GreenNodeFlags.ContainsErrorNode; + if (kind == NodeKind.TaggedIdent) + kindFlags |= GreenNodeFlags.ContainsTaggedIdent; + if (kind.IsKeyword()) + kindFlags |= GreenNodeFlags.ContainsKeyword; + + _flags = leadingBoundaryFlags | trailingBoundaryFlags | leadingContainsFlags | trailingContainsFlags | kindFlags; } /// @@ -114,11 +139,56 @@ public GreenLeaf WithTrailingTrivia(ImmutableArray trivia) public GreenLeaf WithText(string text) => new(Kind, text, LeadingTrivia, TrailingTrivia); - private static int ComputeTriviaWidth(ImmutableArray trivia) + private static int ComputeTriviaWidthAndFlags( + ImmutableArray trivia, + bool isLeading, + out GreenNodeFlags boundaryFlags, + out GreenNodeFlags containsFlags) { + boundaryFlags = GreenNodeFlags.None; + containsFlags = GreenNodeFlags.None; + int width = 0; + bool hasNewline = false; + bool hasWhitespace = false; + bool hasComment = false; + foreach (var t in trivia) + { width += t.Width; + switch (t.Kind) + { + case TriviaKind.Newline: + hasNewline = true; + break; + case TriviaKind.Whitespace: + hasWhitespace = true; + break; + case TriviaKind.SingleLineComment: + case TriviaKind.MultiLineComment: + hasComment = true; + break; + } + } + + if (hasNewline) + { + boundaryFlags |= isLeading ? GreenNodeFlags.HasLeadingNewlineTrivia : GreenNodeFlags.HasTrailingNewlineTrivia; + containsFlags |= GreenNodeFlags.ContainsNewlineTrivia; + } + + if (hasWhitespace) + { + boundaryFlags |= isLeading ? GreenNodeFlags.HasLeadingWhitespaceTrivia : GreenNodeFlags.HasTrailingWhitespaceTrivia; + containsFlags |= GreenNodeFlags.ContainsWhitespaceTrivia; + } + + if (hasComment) + { + boundaryFlags |= isLeading ? GreenNodeFlags.HasLeadingCommentTrivia : GreenNodeFlags.HasTrailingCommentTrivia; + containsFlags |= GreenNodeFlags.ContainsCommentTrivia; + } + return width; } } diff --git a/green-node-flags-refactor.todo b/green-node-flags-refactor.todo index f872878..8113f73 100644 --- a/green-node-flags-refactor.todo +++ b/green-node-flags-refactor.todo @@ -92,16 +92,20 @@ Acceptance criteria: ## Phase 2 — Compute Flags in Leaves -- [ ] Update `GreenLeaf` constructor - - [ ] Scan `LeadingTrivia` once to set boundary flags - - [ ] Scan `TrailingTrivia` once to set boundary flags - - [ ] Set subtree flags based on boundary results - - [ ] Ensure no extra passes beyond existing trivia width computation (piggyback loops) - -- [ ] Verify `GreenNodeCache` behavior is still correct - - [ ] Cached leaves with no trivia have stable flags (`None`) - - [ ] Cached “trailing space” leaves have whitespace flags but no newline/comment - - [ ] Leaves with trivia remain non-cached (unchanged behavior) +- [x] Update `GreenLeaf` constructor + - [x] Scan `LeadingTrivia` once to set boundary flags + - [x] Scan `TrailingTrivia` once to set boundary flags + - [x] Set subtree flags based on boundary results + - [x] Ensure no extra passes beyond existing trivia width computation (piggyback loops) + - Implemented as a single trivia scan per side that returns both width + flags. + - Also sets subtree flags derived from node kind: `ContainsKeyword` / `ContainsTaggedIdent` / `ContainsErrorNode`. + +- [x] Verify `GreenNodeCache` behavior is still correct + - [x] Cached leaves with no trivia have stable trivia flags (`None`) + - Note: subtree-kind flags can still apply (e.g., cached keyword kinds will include `ContainsKeyword`). + - [x] Cached “trailing space” leaves have whitespace flags but no newline/comment + - Note: subtree-kind flags can still apply depending on `NodeKind`. + - [x] Leaves with trivia remain non-cached (unchanged behavior) Acceptance criteria: - Leaf flags are correct for leading/trailing trivia kinds and don’t add noticeable allocations. From 050e7a935734168ddea9736678ec6ab789c48078 Mon Sep 17 00:00:00 2001 From: David Sisco Date: Sat, 10 Jan 2026 20:50:30 -0800 Subject: [PATCH 5/8] [phase 3]: Aggregate Flags in Containers --- TinyTokenizer/Ast/GreenBlock.cs | 17 ++++++++++++++ TinyTokenizer/Ast/GreenList.cs | 25 ++++++++++++++++++++ TinyTokenizer/Ast/GreenNodeFlags.cs | 23 +++++++++++++++++++ TinyTokenizer/Ast/GreenSyntaxNode.cs | 25 ++++++++++++++++++++ green-node-flags-refactor.todo | 34 ++++++++++++++++------------ 5 files changed, 109 insertions(+), 15 deletions(-) diff --git a/TinyTokenizer/Ast/GreenBlock.cs b/TinyTokenizer/Ast/GreenBlock.cs index 968dc1e..115f71d 100644 --- a/TinyTokenizer/Ast/GreenBlock.cs +++ b/TinyTokenizer/Ast/GreenBlock.cs @@ -24,10 +24,14 @@ internal sealed record GreenBlock : GreenContainer private readonly ImmutableArray _children; private readonly int _width; + private readonly GreenNodeFlags _flags; private readonly int[]? _childOffsets; // Pre-computed for ≥10 children (O(1) lookup) /// public override NodeKind Kind { get; } + + /// + internal override GreenNodeFlags Flags => _flags; /// The opening delimiter node (e.g., '{', '[', '(') with its trivia. public GreenLeaf OpenerNode { get; } @@ -75,6 +79,19 @@ public GreenBlock( childrenWidth += child.Width; _width = OpenerNode.Width + childrenWidth + CloserNode.Width; + + // Flags + // - Boundary comes only from the first/last leaf boundaries (opener leading, closer trailing). + // - Subtree contains flags are ORed across opener/inner/closer (excluding boundary bits). + var boundary = + (OpenerNode.Flags & GreenNodeFlagMasks.LeadingBoundary) | + (CloserNode.Flags & GreenNodeFlagMasks.TrailingBoundary); + + var contains = (OpenerNode.Flags | CloserNode.Flags) & GreenNodeFlagMasks.Contains; + foreach (var child in _children) + contains |= child.Flags & GreenNodeFlagMasks.Contains; + + _flags = boundary | contains; // Pre-compute child offsets for large blocks if (_children.Length >= 10) diff --git a/TinyTokenizer/Ast/GreenList.cs b/TinyTokenizer/Ast/GreenList.cs index cccc566..c6a3ab5 100644 --- a/TinyTokenizer/Ast/GreenList.cs +++ b/TinyTokenizer/Ast/GreenList.cs @@ -18,6 +18,7 @@ internal sealed record GreenList : GreenContainer private readonly ImmutableArray _children; private readonly int _width; + private readonly GreenNodeFlags _flags; private readonly int[]? _childOffsets; /// @@ -25,6 +26,9 @@ internal sealed record GreenList : GreenContainer /// public override int Width => _width; + + /// + internal override GreenNodeFlags Flags => _flags; /// public override ImmutableArray Children => _children; @@ -41,6 +45,27 @@ public GreenList(ImmutableArray children) foreach (var child in _children) width += child.Width; _width = width; + + // Flags + if (_children.Length == 0) + { + _flags = GreenNodeFlags.None; + } + else + { + var first = _children[0]; + var last = _children[^1]; + + var boundary = + (first.Flags & GreenNodeFlagMasks.LeadingBoundary) | + (last.Flags & GreenNodeFlagMasks.TrailingBoundary); + + var contains = GreenNodeFlags.None; + foreach (var child in _children) + contains |= child.Flags & GreenNodeFlagMasks.Contains; + + _flags = boundary | contains; + } // Pre-compute offsets for large lists if (_children.Length >= 10) diff --git a/TinyTokenizer/Ast/GreenNodeFlags.cs b/TinyTokenizer/Ast/GreenNodeFlags.cs index 5896c6b..89124a7 100644 --- a/TinyTokenizer/Ast/GreenNodeFlags.cs +++ b/TinyTokenizer/Ast/GreenNodeFlags.cs @@ -30,3 +30,26 @@ internal enum GreenNodeFlags : uint ContainsKeyword = 1u << 12, ContainsTaggedIdent = 1u << 13, } + +internal static class GreenNodeFlagMasks +{ + public const GreenNodeFlags LeadingBoundary = + GreenNodeFlags.HasLeadingNewlineTrivia | + GreenNodeFlags.HasLeadingWhitespaceTrivia | + GreenNodeFlags.HasLeadingCommentTrivia; + + public const GreenNodeFlags TrailingBoundary = + GreenNodeFlags.HasTrailingNewlineTrivia | + GreenNodeFlags.HasTrailingWhitespaceTrivia | + GreenNodeFlags.HasTrailingCommentTrivia; + + public const GreenNodeFlags Boundary = LeadingBoundary | TrailingBoundary; + + public const GreenNodeFlags Contains = + GreenNodeFlags.ContainsNewlineTrivia | + GreenNodeFlags.ContainsWhitespaceTrivia | + GreenNodeFlags.ContainsCommentTrivia | + GreenNodeFlags.ContainsErrorNode | + GreenNodeFlags.ContainsKeyword | + GreenNodeFlags.ContainsTaggedIdent; +} diff --git a/TinyTokenizer/Ast/GreenSyntaxNode.cs b/TinyTokenizer/Ast/GreenSyntaxNode.cs index 8ace7d0..ed21515 100644 --- a/TinyTokenizer/Ast/GreenSyntaxNode.cs +++ b/TinyTokenizer/Ast/GreenSyntaxNode.cs @@ -24,6 +24,7 @@ internal sealed record GreenSyntaxNode : GreenContainer private readonly ImmutableArray _children; private readonly NodeKind _kind; private readonly int _width; + private readonly GreenNodeFlags _flags; /// /// Creates a green syntax node wrapping the specified children. @@ -42,6 +43,27 @@ public GreenSyntaxNode(NodeKind kind, ImmutableArray children) width += child.Width; } _width = width; + + // Flags + if (_children.Length == 0) + { + _flags = GreenNodeFlags.None; + } + else + { + var first = _children[0]; + var last = _children[^1]; + + var boundary = + (first.Flags & GreenNodeFlagMasks.LeadingBoundary) | + (last.Flags & GreenNodeFlagMasks.TrailingBoundary); + + var contains = GreenNodeFlags.None; + foreach (var child in _children) + contains |= child.Flags & GreenNodeFlagMasks.Contains; + + _flags = boundary | contains; + } } /// @@ -57,6 +79,9 @@ public GreenSyntaxNode(NodeKind kind, params GreenNode[] children) /// public override int Width => _width; + + /// + internal override GreenNodeFlags Flags => _flags; /// public override ImmutableArray Children => _children; diff --git a/green-node-flags-refactor.todo b/green-node-flags-refactor.todo index 8113f73..e6f8276 100644 --- a/green-node-flags-refactor.todo +++ b/green-node-flags-refactor.todo @@ -114,21 +114,25 @@ Acceptance criteria: ## Phase 3 — Aggregate Flags in Containers -- [ ] Update `GreenBlock` - - [ ] OR flags from opener leaf + all inner children + closer leaf - - [ ] Ensure boundary flags reflect the block’s actual left/right boundaries - -- [ ] Update `GreenList` - - [ ] OR flags from all children - - [ ] Derive boundary flags from first/last child when present - -- [ ] Update `GreenSyntaxNode` - - [ ] OR flags from all children - - [ ] Derive boundary flags from first/last child when present - -- [ ] Validate structural sharing invariants - - [ ] `WithSlot`/`WithInsert`/`WithReplace` cause flags to recompute via constructors (no ad-hoc flag mutation) - - [ ] `WithLeadingTrivia`/`WithTrailingTrivia`/`WithText` on leaves continue to work +- [x] Update `GreenBlock` + - [x] OR flags from opener leaf + all inner children + closer leaf + - [x] Ensure boundary flags reflect the block’s actual left/right boundaries + - Boundary: opener *leading* + closer *trailing*. + - Subtree: OR of `Contains*` bits across opener/inner/closer. + +- [x] Update `GreenList` + - [x] OR flags from all children + - [x] Derive boundary flags from first/last child when present + +- [x] Update `GreenSyntaxNode` + - [x] OR flags from all children + - [x] Derive boundary flags from first/last child when present + +- [x] Validate structural sharing invariants + - [x] `WithSlot`/`WithInsert`/`WithReplace` cause flags to recompute via constructors (no ad-hoc flag mutation) + - All container mutation methods return `new Green*` instances. + - [x] `WithLeadingTrivia`/`WithTrailingTrivia`/`WithText` on leaves continue to work + - Leaves create a new `GreenLeaf`, which recomputes widths + flags. Acceptance criteria: - Container flags correctly reflect subtree + boundary properties and remain deterministic. From 20272f5467fee5cfc005ef4b07d15a5667aa2fcc Mon Sep 17 00:00:00 2001 From: David Sisco Date: Sat, 10 Jan 2026 20:53:41 -0800 Subject: [PATCH 6/8] [phase 4]: Test Flag Correctness --- TinyTokenizer.Tests/GreenNodeFlagsTests.cs | 222 +++++++++++++++++++++ green-node-flags-refactor.todo | 20 +- 2 files changed, 232 insertions(+), 10 deletions(-) create mode 100644 TinyTokenizer.Tests/GreenNodeFlagsTests.cs diff --git a/TinyTokenizer.Tests/GreenNodeFlagsTests.cs b/TinyTokenizer.Tests/GreenNodeFlagsTests.cs new file mode 100644 index 0000000..e0a8973 --- /dev/null +++ b/TinyTokenizer.Tests/GreenNodeFlagsTests.cs @@ -0,0 +1,222 @@ +using System.Collections.Immutable; +using TinyTokenizer.Ast; + +namespace TinyTokenizer.Tests; + +[Trait("Category", "Flags")] +public class GreenNodeFlagsTests +{ + #region Helpers + + private static ImmutableArray ParseLeaves(string source, TokenizerOptions? options = null) + { + var lexer = new GreenLexer(options ?? TokenizerOptions.Default); + var nodes = lexer.ParseToGreenNodes(source); + return CollectLeaves(nodes); + } + + private static ImmutableArray CollectLeaves(ImmutableArray nodes) + { + var builder = ImmutableArray.CreateBuilder(); + foreach (var node in nodes) + { + CollectLeavesRecursive(node, builder); + } + return builder.ToImmutable(); + } + + private static void CollectLeavesRecursive(GreenNode node, ImmutableArray.Builder builder) + { + if (node is GreenLeaf leaf) + { + builder.Add(leaf); + return; + } + + for (int i = 0; i < node.SlotCount; i++) + { + var child = node.GetSlot(i); + if (child != null) + CollectLeavesRecursive(child, builder); + } + } + + private static GreenLeaf FindLeaf(ImmutableArray leaves, string text) => + Assert.Single(leaves.Where(l => l.Text == text)); + + private static void AssertHas(GreenNodeFlags flags, GreenNodeFlags expected) + { + Assert.True((flags & expected) == expected, $"Expected flags to include {expected} but was {flags}"); + } + + private static void AssertNotHas(GreenNodeFlags flags, GreenNodeFlags unexpected) + { + Assert.True((flags & unexpected) == 0, $"Expected flags to NOT include {unexpected} but was {flags}"); + } + + #endregion + + [Fact] + public void GreenLeaf_Flags_ReflectLeadingAndTrailingTriviaKinds() + { + var leading = ImmutableArray.Create( + GreenTrivia.Whitespace(" "), + GreenTrivia.SingleLineComment("// c"), + GreenTrivia.Newline("\n")); + + var trailing = ImmutableArray.Create( + GreenTrivia.Whitespace("\t"), + GreenTrivia.MultiLineComment("/* m */")); + + var leaf = new GreenLeaf(NodeKind.Ident, "x", leading, trailing); + + AssertHas(leaf.Flags, GreenNodeFlags.HasLeadingWhitespaceTrivia); + AssertHas(leaf.Flags, GreenNodeFlags.HasLeadingCommentTrivia); + AssertHas(leaf.Flags, GreenNodeFlags.HasLeadingNewlineTrivia); + + AssertHas(leaf.Flags, GreenNodeFlags.HasTrailingWhitespaceTrivia); + AssertHas(leaf.Flags, GreenNodeFlags.HasTrailingCommentTrivia); + AssertNotHas(leaf.Flags, GreenNodeFlags.HasTrailingNewlineTrivia); + + AssertHas(leaf.Flags, GreenNodeFlags.ContainsWhitespaceTrivia); + AssertHas(leaf.Flags, GreenNodeFlags.ContainsCommentTrivia); + AssertHas(leaf.Flags, GreenNodeFlags.ContainsNewlineTrivia); + } + + [Fact] + public void GreenLeaf_Flags_IncludeKindDerivedSubtreeBits() + { + var keywordKind = NodeKindExtensions.KeywordKind(0); + + var keywordLeaf = new GreenLeaf(keywordKind, "if"); + AssertHas(keywordLeaf.Flags, GreenNodeFlags.ContainsKeyword); + AssertNotHas(keywordLeaf.Flags, GreenNodeFlagMasks.Boundary | GreenNodeFlags.ContainsWhitespaceTrivia | GreenNodeFlags.ContainsCommentTrivia | GreenNodeFlags.ContainsNewlineTrivia); + + var taggedLeaf = new GreenLeaf(NodeKind.TaggedIdent, "#define"); + AssertHas(taggedLeaf.Flags, GreenNodeFlags.ContainsTaggedIdent); + + var errorLeaf = new GreenLeaf(NodeKind.Error, ""); + AssertHas(errorLeaf.Flags, GreenNodeFlags.ContainsErrorNode); + + var normalLeaf = new GreenLeaf(NodeKind.Ident, "x"); + Assert.Equal(GreenNodeFlags.None, normalLeaf.Flags); + } + + [Fact] + public void GreenBlock_Flags_AggregateContains_AndUseOpenerLeadingPlusCloserTrailingAsBoundary() + { + var opener = new GreenLeaf( + NodeKind.Symbol, + "{", + leadingTrivia: ImmutableArray.Create(GreenTrivia.Newline("\n")), + trailingTrivia: ImmutableArray.Create(GreenTrivia.Whitespace(" "))); + + var closer = new GreenLeaf( + NodeKind.Symbol, + "}", + leadingTrivia: ImmutableArray.Create(GreenTrivia.Newline("\n")), + trailingTrivia: ImmutableArray.Create(GreenTrivia.Whitespace(" "))); + + var inner = new GreenLeaf( + NodeKind.Ident, + "x", + trailingTrivia: ImmutableArray.Create(GreenTrivia.SingleLineComment("// c"))); + + var block = new GreenBlock(opener, closer, ImmutableArray.Create(inner)); + + // Boundary behavior + AssertHas(block.Flags, GreenNodeFlags.HasLeadingNewlineTrivia); + AssertHas(block.Flags, GreenNodeFlags.HasTrailingWhitespaceTrivia); + + // Not from opener trailing / closer leading + AssertNotHas(block.Flags, GreenNodeFlags.HasLeadingWhitespaceTrivia); + AssertNotHas(block.Flags, GreenNodeFlags.HasTrailingNewlineTrivia); + + // Subtree contains aggregation + AssertHas(block.Flags, GreenNodeFlags.ContainsNewlineTrivia); + AssertHas(block.Flags, GreenNodeFlags.ContainsWhitespaceTrivia); + AssertHas(block.Flags, GreenNodeFlags.ContainsCommentTrivia); + } + + [Fact] + public void GreenList_Flags_UseFirstLeadingAndLastTrailingAsBoundary() + { + var first = new GreenLeaf(NodeKind.Ident, "a", leadingTrivia: ImmutableArray.Create(GreenTrivia.Whitespace(" "))); + var middle = new GreenLeaf(NodeKind.Ident, "b", trailingTrivia: ImmutableArray.Create(GreenTrivia.SingleLineComment("// c"))); + var last = new GreenLeaf(NodeKind.Ident, "c", trailingTrivia: ImmutableArray.Create(GreenTrivia.Newline("\n"))); + + var list = new GreenList(ImmutableArray.Create(first, middle, last)); + + AssertHas(list.Flags, GreenNodeFlags.HasLeadingWhitespaceTrivia); + AssertHas(list.Flags, GreenNodeFlags.HasTrailingNewlineTrivia); + + AssertHas(list.Flags, GreenNodeFlags.ContainsWhitespaceTrivia); + AssertHas(list.Flags, GreenNodeFlags.ContainsCommentTrivia); + AssertHas(list.Flags, GreenNodeFlags.ContainsNewlineTrivia); + } + + [Fact] + public void GreenSyntaxNode_Flags_AggregateLikeList() + { + var kind = NodeKindExtensions.SemanticKind(0); + + var first = new GreenLeaf(NodeKind.Ident, "a", leadingTrivia: ImmutableArray.Create(GreenTrivia.Whitespace(" "))); + var last = new GreenLeaf(NodeKind.Ident, "b", trailingTrivia: ImmutableArray.Create(GreenTrivia.Newline("\n"))); + + var node = new GreenSyntaxNode(kind, first, last); + + AssertHas(node.Flags, GreenNodeFlags.HasLeadingWhitespaceTrivia); + AssertHas(node.Flags, GreenNodeFlags.HasTrailingNewlineTrivia); + AssertHas(node.Flags, GreenNodeFlags.ContainsWhitespaceTrivia); + AssertHas(node.Flags, GreenNodeFlags.ContainsNewlineTrivia); + } + + [Fact] + public void ParsedNewlineOwnership_ReflectedInFlags_CommonCase() + { + var leaves = ParseLeaves("a\nb"); + + var a = FindLeaf(leaves, "a"); + var b = FindLeaf(leaves, "b"); + + AssertHas(a.Flags, GreenNodeFlags.HasTrailingNewlineTrivia); + AssertHas(a.Flags, GreenNodeFlags.ContainsNewlineTrivia); + + AssertNotHas(b.Flags, GreenNodeFlags.HasLeadingNewlineTrivia); + } + + [Fact] + public void ParsedSameLineCommentOwnership_ReflectedInFlags() + { + var options = TokenizerOptions.Default.WithCommentStyles(CommentStyle.CStyleSingleLine); + var leaves = ParseLeaves("a // c\nb", options); + + var a = FindLeaf(leaves, "a"); + var b = FindLeaf(leaves, "b"); + + AssertHas(a.Flags, GreenNodeFlags.HasTrailingCommentTrivia); + AssertHas(a.Flags, GreenNodeFlags.HasTrailingNewlineTrivia); + AssertHas(a.Flags, GreenNodeFlags.ContainsCommentTrivia); + AssertHas(a.Flags, GreenNodeFlags.ContainsNewlineTrivia); + + Assert.Empty(b.LeadingTrivia); + AssertNotHas(b.Flags, GreenNodeFlags.HasLeadingCommentTrivia | GreenNodeFlags.HasLeadingNewlineTrivia); + } + + [Fact] + public void ParsedOwnLineCommentOwnership_ReflectedInFlags() + { + var options = TokenizerOptions.Default.WithCommentStyles(CommentStyle.CStyleSingleLine); + var leaves = ParseLeaves("a\n// c\nb", options); + + var a = FindLeaf(leaves, "a"); + var b = FindLeaf(leaves, "b"); + + AssertHas(a.Flags, GreenNodeFlags.HasTrailingNewlineTrivia); + + AssertHas(b.Flags, GreenNodeFlags.HasLeadingCommentTrivia); + AssertHas(b.Flags, GreenNodeFlags.HasLeadingNewlineTrivia); + AssertHas(b.Flags, GreenNodeFlags.ContainsCommentTrivia); + AssertHas(b.Flags, GreenNodeFlags.ContainsNewlineTrivia); + } +} diff --git a/green-node-flags-refactor.todo b/green-node-flags-refactor.todo index e6f8276..30cf4c9 100644 --- a/green-node-flags-refactor.todo +++ b/green-node-flags-refactor.todo @@ -141,16 +141,16 @@ Acceptance criteria: ## Phase 4 — Tests: Flag Correctness -- [ ] Add/extend unit tests for flag correctness - - [ ] Leaf boundary flags - - [ ] Newline trivia in leading/trailing - - [ ] Comment trivia in leading/trailing - - [ ] Whitespace trivia in leading/trailing - - [ ] Aggregation - - [ ] Block contains newline/comment/whitespace when present in delimiters or inner children - - [ ] List/syntax node aggregation - - [ ] “Newline ownership” sanity (existing lexer trivia semantics) - - [ ] Newline appears as trailing trivia of previous token in common cases +- [x] Add/extend unit tests for flag correctness + - [x] Leaf boundary flags + - [x] Newline trivia in leading/trailing + - [x] Comment trivia in leading/trailing + - [x] Whitespace trivia in leading/trailing + - [x] Aggregation + - [x] Block contains newline/comment/whitespace when present in delimiters or inner children + - [x] List/syntax node aggregation + - [x] “Newline ownership” sanity (existing lexer trivia semantics) + - [x] Newline appears as trailing trivia of previous token in common cases Acceptance criteria: - Tests reliably catch flag computation regressions. From da318aef6120103c1e144478116b1e90a1bacf14 Mon Sep 17 00:00:00 2001 From: David Sisco Date: Sat, 10 Jan 2026 20:58:11 -0800 Subject: [PATCH 7/8] [phase 5]: Test Record Equality / Hash Stability --- TinyTokenizer.Tests/GreenNodeEqualityTests.cs | 99 +++++++++++++++++++ green-node-flags-refactor.todo | 18 ++-- 2 files changed, 108 insertions(+), 9 deletions(-) create mode 100644 TinyTokenizer.Tests/GreenNodeEqualityTests.cs diff --git a/TinyTokenizer.Tests/GreenNodeEqualityTests.cs b/TinyTokenizer.Tests/GreenNodeEqualityTests.cs new file mode 100644 index 0000000..817b298 --- /dev/null +++ b/TinyTokenizer.Tests/GreenNodeEqualityTests.cs @@ -0,0 +1,99 @@ +using System.Collections.Immutable; +using TinyTokenizer.Ast; + +namespace TinyTokenizer.Tests; + +[Trait("Category", "Equality")] +public class GreenNodeEqualityTests +{ + [Fact] + public void GreenLeaf_EquivalentConstructions_AreEqual_AndHashMatches() + { + var leading = ImmutableArray.Create(GreenTrivia.Whitespace(" ")); + var trailing = ImmutableArray.Create(GreenTrivia.Newline("\n")); + + var a = new GreenLeaf(NodeKind.Ident, "x", leading, trailing); + + var b = new GreenLeaf(NodeKind.Ident, "x") + .WithLeadingTrivia(leading) + .WithTrailingTrivia(trailing); + + Assert.True(a.Equals(b)); + Assert.Equal(a.GetHashCode(), b.GetHashCode()); + } + + [Fact] + public void GreenLeaf_CacheAndNonCachePaths_BehaveConsistently() + { + // No trivia -> cached + var cached1 = GreenNodeCache.GetOrCreate(NodeKind.Symbol, "{"); + var cached2 = GreenNodeCache.CreateDelimiter('{'); + + Assert.Same(cached1, cached2); + Assert.True(cached1.Equals(cached2)); + Assert.Equal(cached1.GetHashCode(), cached2.GetHashCode()); + + // Trailing space -> cached "with space" + var spaceTrivia = ImmutableArray.Create(GreenTrivia.Whitespace(" ")); + var cachedWithSpace = GreenNodeCache.GetOrCreateWithTrailingSpace(NodeKind.Symbol, "{"); + var viaFactory = GreenNodeCache.Create(NodeKind.Symbol, "{", ImmutableArray.Empty, spaceTrivia); + + Assert.Same(cachedWithSpace, viaFactory); + Assert.Equal(cachedWithSpace.GetHashCode(), viaFactory.GetHashCode()); + AssertHasWhitespaceButNoNewlineOrComment(cachedWithSpace); + } + + [Fact] + public void StructuralSharing_WithMethods_DoNotMutateOriginals() + { + var leaf = new GreenLeaf(NodeKind.Ident, "x"); + var changed = leaf.WithText("y"); + + Assert.Equal("x", leaf.Text); + Assert.Equal("y", changed.Text); + Assert.False(leaf.Equals(changed)); + + var child1 = new GreenLeaf(NodeKind.Ident, "a"); + var child2 = new GreenLeaf(NodeKind.Ident, "b"); + var block = GreenBlock.Create('{', ImmutableArray.Create(child1, child2)); + + var inserted = block.WithInsert(2, ImmutableArray.Create(new GreenLeaf(NodeKind.Ident, "c"))); + + Assert.Equal(4, block.SlotCount); // opener + 2 children + closer + Assert.Equal(5, inserted.SlotCount); // opener + 3 children + closer + + // Original is unchanged and still shares children + Assert.Same(child1, block.GetSlot(1)); + Assert.Same(child2, block.GetSlot(2)); + + // Inserted block shares originals too + Assert.Same(child1, inserted.GetSlot(1)); + Assert.Same(child2, inserted.GetSlot(3)); + + Assert.False(block.Equals(inserted)); + } + + [Fact] + public void OnlyContentChanges_CauseInequality() + { + // Same kind/text/trivia should be equal (baseline) + var a = new GreenLeaf(NodeKind.Ident, "x"); + var b = new GreenLeaf(NodeKind.Ident, "x"); + Assert.True(a.Equals(b)); + + // Different text -> not equal + var c = new GreenLeaf(NodeKind.Ident, "y"); + Assert.False(a.Equals(c)); + + // Different kind -> not equal + var d = new GreenLeaf(NodeKind.Numeric, "x"); + Assert.False(a.Equals(d)); + } + + private static void AssertHasWhitespaceButNoNewlineOrComment(GreenLeaf leaf) + { + Assert.True((leaf.Flags & GreenNodeFlags.ContainsWhitespaceTrivia) != 0); + Assert.True((leaf.Flags & GreenNodeFlags.ContainsNewlineTrivia) == 0); + Assert.True((leaf.Flags & GreenNodeFlags.ContainsCommentTrivia) == 0); + } +} diff --git a/green-node-flags-refactor.todo b/green-node-flags-refactor.todo index 30cf4c9..fd9978b 100644 --- a/green-node-flags-refactor.todo +++ b/green-node-flags-refactor.todo @@ -159,17 +159,17 @@ Acceptance criteria: ## Phase 5 — Tests: Record Equality / Hash Stability (extra safety) -- [ ] Equality invariants (structural equality) - - [ ] Construct equivalent green nodes via different construction paths and assert `Equals` is true - - [ ] Assert `GetHashCode` matches for structurally equal nodes +- [x] Equality invariants (structural equality) + - [x] Construct equivalent green nodes via different construction paths and assert `Equals` is true + - [x] Assert `GetHashCode` matches for structurally equal nodes -- [ ] Cache interaction - - [ ] Cached leaf instances behave consistently (equal nodes produce equal hashes) - - [ ] Non-cached leaves with identical trivia sequences still compare equal +- [x] Cache interaction + - [x] Cached leaf instances behave consistently (equal nodes produce equal hashes) + - [x] Non-cached leaves with identical trivia sequences still compare equal -- [ ] Structural sharing sanity - - [ ] `With*` methods do not mutate originals - - [ ] Only content changes cause inequality +- [x] Structural sharing sanity + - [x] `With*` methods do not mutate originals + - [x] Only content changes cause inequality Acceptance criteria: - We have strong confidence that adding `Flags` doesn’t introduce equality/hash surprises. From 7e47ce279df1f4a08f8b5bf81a6d8bf0cbf36ff1 Mon Sep 17 00:00:00 2001 From: David Sisco Date: Sat, 10 Jan 2026 21:16:20 -0800 Subject: [PATCH 8/8] chore: remove finished task list --- green-node-flags-refactor.todo | 193 --------------------------------- 1 file changed, 193 deletions(-) delete mode 100644 green-node-flags-refactor.todo diff --git a/green-node-flags-refactor.todo b/green-node-flags-refactor.todo deleted file mode 100644 index fd9978b..0000000 --- a/green-node-flags-refactor.todo +++ /dev/null @@ -1,193 +0,0 @@ -todo - -# Green Node Flags Refactor (Precursor) - -Goal: add a compact flags bitmask to green nodes so newline/comment/whitespace/error/keyword/tagged-ident checks are O(1) and enable subtree pruning. - -Scope: -- Precursor to the Query newline optimization plan. -- Green layer only (TinyTokenizer/TinyTokenizer/Ast/Green*). No query changes in this work item. - -Constraints: -- Green nodes remain immutable + structurally shareable. -- Flags must be deterministically derived from immutable content (trivia + children), never mutated. -- Preserve semantics: newline flags reflect `TriviaKind.Newline` only. - ------------------------------------------------------------------------- - -## Phase 0 — Decide Flag Semantics (one-time) - - -Decisions (grounded in current AST model): -- Trivia kinds are exactly: `Whitespace` (spaces/tabs), `Newline` (\n / \r\n), `SingleLineComment`, `MultiLineComment`. -- Keywords are represented as nodes with `NodeKind` in the keyword range (`NodeKindExtensions.IsKeyword()`). -- Tagged identifiers are leaf nodes with `NodeKind.TaggedIdent`. -- Errors are leaf nodes with `NodeKind.Error`. - -- [x] Confirm high-ROI flag set (v1) - - [ ] Boundary trivia flags - - [ ] `HasLeadingNewlineTrivia` / `HasTrailingNewlineTrivia` - - [ ] `HasLeadingCommentTrivia` / `HasTrailingCommentTrivia` - - [ ] `HasLeadingWhitespaceTrivia` / `HasTrailingWhitespaceTrivia` - - [ ] Subtree flags - - [ ] `ContainsNewlineTrivia` - - [ ] `ContainsCommentTrivia` - - [ ] `ContainsWhitespaceTrivia` - - [ ] `ContainsErrorNode` - - [ ] `ContainsKeyword` - - [ ] `ContainsTaggedIdent` - -- [x] Confirm boundary meaning for containers - - [x] “Leading” refers to the left boundary of the node’s text span - - [x] “Trailing” refers to the right boundary of the node’s text span - - [x] For containers, boundary flags derive from first/last leaf boundaries (not necessarily immediate delimiter leaves for all container types) - -Flag semantics (implementation-ready): -- Boundary flags: - - `HasLeadingNewlineTrivia`: node’s left boundary has trivia of kind `Newline`. - - `HasTrailingNewlineTrivia`: node’s right boundary has trivia of kind `Newline`. - - `HasLeadingWhitespaceTrivia`: left boundary has trivia kind `Whitespace` (explicitly excludes `Newline`). - - `HasTrailingWhitespaceTrivia`: right boundary has trivia kind `Whitespace`. - - `HasLeadingCommentTrivia`: left boundary has trivia kind `SingleLineComment` OR `MultiLineComment`. - - `HasTrailingCommentTrivia`: right boundary has trivia kind `SingleLineComment` OR `MultiLineComment`. - - Leaf rule: boundary flags are computed by scanning `GreenLeaf.LeadingTrivia` / `GreenLeaf.TrailingTrivia`. - - Container rule: boundary flags are derived from the first/last leaf descendant boundaries (fast once flags exist). For `GreenBlock`, this effectively comes from opener/closer delimiter leaves. - -- Subtree flags: - - `ContainsNewlineTrivia`: any leaf in the subtree has newline trivia (leading or trailing). - - `ContainsWhitespaceTrivia`: any leaf in subtree has whitespace trivia. - - `ContainsCommentTrivia`: any leaf in subtree has comment trivia. - - `ContainsErrorNode`: any node in subtree has `Kind == NodeKind.Error`. - - `ContainsKeyword`: any node in subtree has `Kind.IsKeyword()`. - - `ContainsTaggedIdent`: any node in subtree has `Kind == NodeKind.TaggedIdent`. - -Acceptance criteria: -- Flag list and boundary semantics are written down and stable. - ------------------------------------------------------------------------- - -## Phase 1 — Add Flags Infrastructure - -- [x] Add `GreenNodeFlags` enum (bitmask) - - [x] Pick underlying type (`ushort` vs `uint`), reserve room for future bits - - Decision: use `uint` for headroom (we already need ~12 bits; `uint` keeps growth cheap). - - [x] Keep naming consistent: `HasLeadingX` / `HasTrailingX` / `ContainsX` - -- [x] Add `Flags` to `GreenNode` - - [x] Add `internal GreenNodeFlags Flags { get; }` - - Implemented as `internal virtual GreenNodeFlags Flags => GreenNodeFlags.None;` initially. - - Phase 2/3 will override in concrete node types after computing flags. - - [ ] Add lightweight convenience bools (optional) - -- [x] Confirm record/equality impact approach - - [x] Ensure `Flags` is deterministically derived from other fields - - Decision: flags must be computed solely from trivia/kind/children (no external state). - - [x] Decide whether `Flags` should participate in record equality (default) or be excluded (custom) - - Decision: keep default record equality; rely on deterministic computation + Phase 5 tests. - -Acceptance criteria: -- Green nodes compile with a flags field/property available from the base type. - ------------------------------------------------------------------------- - -## Phase 2 — Compute Flags in Leaves - -- [x] Update `GreenLeaf` constructor - - [x] Scan `LeadingTrivia` once to set boundary flags - - [x] Scan `TrailingTrivia` once to set boundary flags - - [x] Set subtree flags based on boundary results - - [x] Ensure no extra passes beyond existing trivia width computation (piggyback loops) - - Implemented as a single trivia scan per side that returns both width + flags. - - Also sets subtree flags derived from node kind: `ContainsKeyword` / `ContainsTaggedIdent` / `ContainsErrorNode`. - -- [x] Verify `GreenNodeCache` behavior is still correct - - [x] Cached leaves with no trivia have stable trivia flags (`None`) - - Note: subtree-kind flags can still apply (e.g., cached keyword kinds will include `ContainsKeyword`). - - [x] Cached “trailing space” leaves have whitespace flags but no newline/comment - - Note: subtree-kind flags can still apply depending on `NodeKind`. - - [x] Leaves with trivia remain non-cached (unchanged behavior) - -Acceptance criteria: -- Leaf flags are correct for leading/trailing trivia kinds and don’t add noticeable allocations. - ------------------------------------------------------------------------- - -## Phase 3 — Aggregate Flags in Containers - -- [x] Update `GreenBlock` - - [x] OR flags from opener leaf + all inner children + closer leaf - - [x] Ensure boundary flags reflect the block’s actual left/right boundaries - - Boundary: opener *leading* + closer *trailing*. - - Subtree: OR of `Contains*` bits across opener/inner/closer. - -- [x] Update `GreenList` - - [x] OR flags from all children - - [x] Derive boundary flags from first/last child when present - -- [x] Update `GreenSyntaxNode` - - [x] OR flags from all children - - [x] Derive boundary flags from first/last child when present - -- [x] Validate structural sharing invariants - - [x] `WithSlot`/`WithInsert`/`WithReplace` cause flags to recompute via constructors (no ad-hoc flag mutation) - - All container mutation methods return `new Green*` instances. - - [x] `WithLeadingTrivia`/`WithTrailingTrivia`/`WithText` on leaves continue to work - - Leaves create a new `GreenLeaf`, which recomputes widths + flags. - -Acceptance criteria: -- Container flags correctly reflect subtree + boundary properties and remain deterministic. - ------------------------------------------------------------------------- - -## Phase 4 — Tests: Flag Correctness - -- [x] Add/extend unit tests for flag correctness - - [x] Leaf boundary flags - - [x] Newline trivia in leading/trailing - - [x] Comment trivia in leading/trailing - - [x] Whitespace trivia in leading/trailing - - [x] Aggregation - - [x] Block contains newline/comment/whitespace when present in delimiters or inner children - - [x] List/syntax node aggregation - - [x] “Newline ownership” sanity (existing lexer trivia semantics) - - [x] Newline appears as trailing trivia of previous token in common cases - -Acceptance criteria: -- Tests reliably catch flag computation regressions. - ------------------------------------------------------------------------- - -## Phase 5 — Tests: Record Equality / Hash Stability (extra safety) - -- [x] Equality invariants (structural equality) - - [x] Construct equivalent green nodes via different construction paths and assert `Equals` is true - - [x] Assert `GetHashCode` matches for structurally equal nodes - -- [x] Cache interaction - - [x] Cached leaf instances behave consistently (equal nodes produce equal hashes) - - [x] Non-cached leaves with identical trivia sequences still compare equal - -- [x] Structural sharing sanity - - [x] `With*` methods do not mutate originals - - [x] Only content changes cause inequality - -Acceptance criteria: -- We have strong confidence that adding `Flags` doesn’t introduce equality/hash surprises. - ------------------------------------------------------------------------- - -## Phase 6 — Validation / Perf Smoke - -- [ ] Run `dotnet test TinyTokenizer.Tests` -- [ ] Run `dotnet run -c Release --project TinyTokenizer.Benchmarks -- --filter *AstBenchmarks*` -- [ ] Run `dotnet run -c Release --project TinyTokenizer.Benchmarks -- --filter *SyntaxEditorBenchmarks*` - -Acceptance criteria: -- All tests pass; no obvious performance regressions. - ------------------------------------------------------------------------- - -## Follow-up (out of scope for this .todo) - -- [ ] Update Query newline matching to use flags (separate plan/work item) -- [ ] Optional: add targeted benchmarks for `SyntaxTree.Select(Query.Newline)` once newline query changes land