Add SQL lexer with token scanning and position tracking

[rahulc0dy]

Issue Reference

• Fixes Implement Lexer / Scanner #10

Summary by CodeRabbit

• New Features

  • Added a full SQL lexer to recognize keywords, identifiers, numbers, strings, operators, punctuation, and both line and block comments.
  • Introduced single-element lookahead support for parsing streams.

• Bug Fixes / Reliability

  • Improved line/column tracking and error reporting for unterminated strings, comments, and unexpected characters.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Adds a position-tracked SQL lexer: token types, keyword map and lookup, scanners for identifiers/numbers/strings, whitespace/comment skipping, lexer error types, NextToken returning (Token, error), and a generic LookaheadIterator for parser peek semantics.

Changes

SQL Lexer Infrastructure

Layer / File(s)	Summary
Token types and representation internal/sql/lexer/tokens.go	TokenType enum with human-readable names and Token struct carrying Type, Literal, and 1-based Line/Col; string formatting methods.
Lexer state, cursor, and errors internal/sql/lexer/lexer.go (lines 1–110), internal/sql/lexer/errors.go	Lexer position tracking (absolute index, line/col), NewLexer, rune cursor helpers (peek, peekNext, advance) updating line/col, comment skipping (--, /*...*/), and LexError sentinel/errors.
Scanning: identifiers, keyword lookup, numbers, strings internal/sql/lexer/lexer.go (lines 112–202), internal/sql/lexer/keywords.go	makeToken; scanIdentifier with lookupIdent using an upper-case keywords map; scanNumber supports integers, floats and leading-dot floats; scanString supports single-quoted literals and doubled-quote escaping.
NextToken dispatch and operators internal/sql/lexer/lexer.go (lines 204–282)	NextToken() (Token, error) skips whitespace/comments, returns TOKEN_EOF at end, dispatches to scanners, recognizes punctuation, arithmetic, =, and multi-char comparisons (<=, <>, >=, !=); returns TOKEN_ILLEGAL plus LexError for unexpected input.
Generic lookahead iterator internal/sql/lexer/lookahead.go	LookaheadIterator[T] generic with single-element peek buffer; Peek, Next, Count, ExpectNextValue, and ExpectNextMatches helpers for parser consumption semantics.

Assessment against linked issues

Objective	Addressed	Explanation
Define token constants for SQL keywords, operators, identifiers [#10]	✅
Build lexer struct to read SQL strings rune-by-rune [#10]	✅
Implement NextToken method with whitespace handling and terminator detection [#10]	✅

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

internal/sql/lexer/lookahead.go (1)

16-17: ⚡ Quick win

Consider validating nextFn parameter.

If nextFn is nil, the iterator will panic later during the first Peek() or Next() call with a nil pointer dereference. Adding an early check would provide a clearer error message at construction time.

🛡️ Suggested validation

 func NewLookaheadIterator[T any](nextFn func() T) *LookaheadIterator[T] {
+	if nextFn == nil {
+		panic("lookahead: nextFn cannot be nil")
+	}
 	return &LookaheadIterator[T]{nextFn: nextFn}
 }

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: bc4ffb44-7265-4271-9e98-3f6725f40711

📥 Commits

Reviewing files that changed from the base of the PR and between 885d5cc and a0d4c72.

📒 Files selected for processing (6)

• internal/sql/lexer/errors.go
• internal/sql/lexer/keywords.go
• internal/sql/lexer/lexer.go
• internal/sql/lexer/lookahead.go
• internal/sql/lexer/position.go
• internal/sql/lexer/tokens.go

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

internal/sql/lexer/lexer.go (1)

22-22: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Typo in doc comment: "fro" → "for".

-// NewLexer creates a Lexer fro the given SQL source string
+// NewLexer creates a Lexer for the given SQL source string

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1099c6c0-fd02-47c0-b261-90848c729c04

📥 Commits

Reviewing files that changed from the base of the PR and between 07b2f6f and ab07f8a.

📒 Files selected for processing (3)

• internal/sql/lexer/.gitkeep
• internal/sql/lexer/errors.go
• internal/sql/lexer/lexer.go