Refactor regex quantifiers into base and mode [reduced-it] [databricks]

[wjxiz1992]

JaCoCo production line coverage: +61 lines (sql-plugin +61; shim 330, fix-line measurement against this PR diff)

Closes #15831
Contributes to #14733
Follow-up to #15478 and the review suggestion.

Description

AI assistance: The change and PR description were prepared with Codex assistance.

What this fixes

Replace the three regex quantifier AST subtypes and mode-copy helpers with one product representation, RegexQuantifier(base, mode). This is a semantics-preserving follow-up: accepted syntax, serialized regexes, fallback behavior, diagnostic positions, and the 999 repetition boundary are unchanged.

Approach

• Model repetition meaning with a sealed Base: ZeroOrOne, ZeroOrMore, OneOrMore, Fixed(length), or Variable(min, max).
• Model matching behavior independently with a sealed Mode: Greedy, Reluctant, or Possessive.
• Construct the complete pair directly in the parser and migrate transpiler, rewrite, serializer, and fuzzer consumers without adding a parallel parser path.
• Keep mutable source position outside the case-class constructor so diagnostics do not participate in semantic equality.

Because #15831 is explicitly a behavior-preserving refactor, there is no pre-fix GPU/CPU mismatch to reproduce. Validation instead compares the baseline and patched acceptance suites and reruns the focused CPU/GPU Python cases established by #15478.

Whole-system design assessment

Disposition: ARCHITECTURAL_FIX

The smallest coherent change is at the AST boundary: base and mode become explicit independent dimensions, and all parser/transpiler/rewrite/serialization consumers use that single representation. This removes subtype-specific reconstruction while preserving the existing public behavior and fallback diagnostics.

Tests added

• Scala unit: RegularExpressionParserSuite -> issue-15831: quantifier base and mode are independent semantic dimensions
• Existing Scala acceptance: RegularExpressionParserSuite, RegularExpressionTranspilerSuite, RegularExpressionRewriteSuite, and RegExpUtilsSuite
• Python IT: focused regexp_test.py selection covering negative-limit split, unsupported fallback, repetition replacement, possessive fallback, extract fallbacks, and lazy quantifiers

Local validation

Scala suites:

Tests: succeeded 183, failed 0, canceled 6, ignored 0, pending 0
All tests passed.
BUILD SUCCESS

JaCoCo production fix-line measurement:

MODULE_FIXLINE_COVERED[sql-plugin]=61 (of 71 added prod lines)
FIXLINE_COVERED=61 (of 71 added prod lines)

Python IT with Spark 3.3 and Python 3.10:

7 passed, 89 deselected, 2 warnings in 13.61s

Scalastyle:

Processed 1850 file(s)
Found 0 errors
BUILD SUCCESS

End-to-end CPU/GPU parity on the patched dist JAR:

All 7 focused comparisons passed. Logs show ProjectExec running on GPU for supported paths and the expected unchanged CPU-bridge fallbacks for unsupported patterns.

Performance impact

Regex parsing remains O(pattern length), with constant work per quantifier. This runs during expression translation, not per input row; it adds no JNI or GPU work. Direct construction removes the simple-quantifier mode-copy allocation, while brace quantifiers retain the same aggregate temporary-allocation count. A runtime benchmark is not warranted for this cold-path representation refactor.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge because no actionable correctness, security, or repository-rule violations remain.

Summary

• Replaces subtype-specific quantifiers with RegexQuantifier(base, mode).
• Updates parser and transpiler pattern matching for the unified representation.
• Expands parser coverage across all base/mode combinations and malformed counted quantifiers.
• Updates fuzz generation to construct valid quantifier ranges.
• Since the previous review, only the Base and Mode import location changed; no new behavioral issue was identified.

Diagram

%%{init: {'theme': 'neutral'}}%%
classDiagram
  class RegexQuantifier {
    +Base base
    +Mode mode
    +Option~Int~ position
    +minLength() Int
    +toRegexString() String
  }

  class Base
  class ZeroOrOne
  class ZeroOrMore
  class OneOrMore
  class Fixed {
    +Int length
  }
  class Variable {
    +Int minLength
    +Option~Int~ maxLength
  }

  class Mode
  class Greedy
  class Reluctant
  class Possessive

  RegexQuantifier --> Base
  RegexQuantifier --> Mode
  Base <|-- ZeroOrOne
  Base <|-- ZeroOrMore
  Base <|-- OneOrMore
  Base <|-- Fixed
  Base <|-- Variable
  Mode <|-- Greedy
  Mode <|-- Reluctant
  Mode <|-- Possessive

Loading

[Copilot]

🔵 Needs a closer look

It is a broad refactor in a core parsing/transpilation path where subtle semantic or diagnostic-position regressions are possible despite unit coverage.

Pull request overview

Refactors the regex quantifier AST model to a single RegexQuantifier(base, mode) representation, separating repetition shape (base) from matching behavior (mode), while keeping diagnostics positions and the cuDF 999-count boundary behavior intact.

Changes:

• Replaces the former quantifier subtype hierarchy with RegexQuantifier.Base + RegexQuantifier.Mode, constructed directly by the parser.
• Updates cuDF transpiler quantifier pattern matches to use the new base/mode decomposition.
• Updates Scala test suites and fuzz generation to build quantifiers via the new model and adds an independence/equality regression test.

File summaries

File	Description
tests/src/test/scala/com/nvidia/spark/rapids/RegularExpressionTranspilerSuite.scala	Updates fuzz regex generation to construct RegexQuantifier(base, mode) rather than subtype + mode-copy helpers.
tests/src/test/scala/com/nvidia/spark/rapids/RegularExpressionParserSuite.scala	Updates expected ASTs to the new quantifier model and adds a test asserting base/mode independence plus equality ignoring diagnostic position.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RegexParser.scala	Refactors parsing and transpilation logic to use RegexQuantifier(base, mode); introduces Base ADT and ports quantifier-related matching/serialization accordingly.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[wjxiz1992]

build

[wjxiz1992]

build

[wjxiz1992]

build

[wjxiz1992]

build

[igorpeshansky]

Would like to resolve the "closer to Java semantics" discussion eventually, but it doesn't have to be addressed in this PR. The rest are minor cleanups.

[igorpeshansky]

FWIW, I think sticking closer to Java semantics helps maintain cleaner code and avoids questions about consistency… Since these edge cases are only reachable from tests, it would be easiest to match Java to decide on the expected behavior.

[wjxiz1992]

build

[wjxiz1992]

build