Replace regex parsing and string concatenation with a real parser and SwiftSyntax

[BenjaminBriggs]

Summary

This is a change I've wanted to make for a while but never had the time to do it but using Cluade Fable has helped make this change in a day instead of a week. I ended up rewriting the generator by hand but used Fables atempt as a reference.
This branch replaces both halves of the codegen pipeline with type-safe equivalents, behind a characterization test suite built first:

• Parsing: the four regexes and remove-and-reparse recursion are gone. Proto source now goes through a fabel written lexer (Lexer.swift) and recursive-descent parser (ProtoParser.swift) producing a tree AST that is flattened into the existing models. Nesting is handled by actual recursion, malformed input throws ParseError with line/column instead of silently dropping fields, and the parser is deliberately lenient about what it skips (custom options, aggregate values, doc comments anywhere, empty statements, service/extend blocks) while staying strict about what it generates from.
• Generation: string concatenation is replaced by SwiftSyntax builders (Sources/SwiftBuffet/Generator/). Every fragment is parsed at generation time, and the assembled file is re-parsed with SwiftParser before being written, invalid Swift fails the codegen run, not when we try and build.

Bugs fixed along the way

• Five regex parser bugs (single-line/empty messages, indented or same-line closing braces, deep nesting, nested enums without indented closing braces), characterized as expected failures first, then fixed by the rewrite
• init?(proto:) assigned the wrong variable / used the Swift property name instead of the proto name in the URL, integer, and message branches (wrong for fields like description)
• field.type.contains("int") misclassified types like PrintJob as integers; integer conversion force-unwrapped (Int(exactly:)!) now classified by explicit type sets and converted via the matching Swift type (Int/UInt)
• Nested messages referenced nonexistent proto types (ProtoInner instead of ProtoOuter.Inner); models now carry the full parent path
• Name collisions (two types flattening to the same Swift name) now fail with a clear error instead of emitting duplicate symbols
• Single-case enums generated case = 0 (empty name); allow_alias enums generated duplicate raw values, both fixed
• oneof members are parsed as ordinary fields, preserving the output shape the legacy generator produced

Also in this branch

• Merge of feature/xcode-27-updates: swiftbuffet.json plugin configuration, Xcode project plugin support, quiet flag rename, repeated-URL field support, CI workflow
• Test suite migrated to Swift Testing (79 tests in 10 suites, including parameterized round-trip validation and a torture-test proto exercising the full grammar: hex/octal literals, escaped strings, custom options, leading-dot types, allow_alias, three-level nesting)
• swift-syntax pinned to 600.0.0..<602.0.0 (510 produces different formatting, verified empirically)
• DocC documentation catalog (getting started, CLI and plugin reference, generated-code anatomy, supported feature matrix) and a rewritten README with the project rationale
• Generator split into focused files with a ProtoConversion switch and result-builder style throughout

Breaking/visible changes for consumers

• Generated output changes cosmetically (SwiftSyntax formatting); a one-time diff in generated files
• Clean builds compile swift-syntax (~15s, cached thereafter)
• Malformed protos that previously half-generated now fail loudly with positions

Test plan

• 79 tests across 10 suites (swift test)
• Golden snapshot tests pin exact generator output; parser swap was verified byte-identical before the generator rewrite
• Round-trip suite: every corpus proto × all flag combinations re-parses cleanly
• Example project builds end-to-end through the plugin (exercises swiftbuffet.json)
• CI workflow added (runs both of the above on macos-15)

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR rewrites SwiftBuffet’s proto parsing and Swift code generation pipelines to be type-safe and validated: a new lexer + recursive-descent parser replaces regex parsing, and a SwiftSyntax-based generator replaces string concatenation (with a SwiftParser re-parse gate to prevent emitting invalid Swift).

Changes:

• Replace regex-driven proto parsing with a lexer (Lexer.swift) and recursive-descent parser (ProtoParser.swift) producing a tree AST flattened into existing models.
• Replace string-concatenation generation with SwiftSyntax builders under Sources/SwiftBuffet/Generator/, adding collision detection and SwiftParser validation.
• Add a comprehensive Swift Testing suite (goldens, round-trips, torture fixture, error tests) plus DocC + README updates, plugin config support, and CI.

Reviewed changes

Copilot reviewed 38 out of 39 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
Tests/UtilitiesTests.swift	Adds unit tests for enum-case prefix stripping and snake→camel conversion.
Tests/TortureProtoTests.swift	Adds a hostile-but-valid proto fixture exercising broad grammar + generation assertions.
Tests/RoundTripTests.swift	Round-trip matrix: corpus protos × generator flag combinations must re-parse as Swift.
Tests/ProtoParserErrorTests.swift	Adds negative tests validating ParseError line/column/expected tokens + skip behavior.
Tests/ProtoInitTests.swift	Validates generated init?(proto:) conversion logic for key field categories.
Tests/ParserTests.swift	Migrates parser tests to Swift Testing and new parseProto entrypoint.
Tests/ParserCorpusTests.swift	Adds characterization corpus pinning legacy parsing behavior and prior regex bugs.
Tests/LexerTests.swift	Adds lexer tokenization/positioning/error-behavior tests.
Tests/GoldenTests.swift	Adds golden snapshot tests for full pipeline output formatting/content.
Tests/GeneratorTests.swift	Migrates generator tests to Swift Testing and adds new regression coverage.
Sources/SwiftBuffet/Utilities.swift	Updates type utilities, enum prefix stripping rules, and integer type classification helpers.
Sources/SwiftBuffet/TimeInterval+String.swift	Removes legacy TimeInterval parsing helper (no longer used).
Sources/SwiftBuffet/Regex.swift	Removes legacy regex patterns used by the old parser.
Sources/SwiftBuffet/ProtoParser.swift	Adds recursive-descent parser implementing supported proto subset + skipping rules.
Sources/SwiftBuffet/Parser.swift	Replaces regex recursion with stable parseProto + flatten flow; updates CLI parsing entrypoint.
Sources/SwiftBuffet/Models.swift	Introduces parentPath for nested type identity and improves field name/type mapping helpers.
Sources/SwiftBuffet/main.swift	Updates CLI flags (--quiet, --include-protobuf, --store-backing-data) and new throwing generator call.
Sources/SwiftBuffet/Lexer.swift	Adds lexer/token model + ParseError with line/column.
Sources/SwiftBuffet/Generator/GeneratorErrors.swift	Adds GenerationError (SwiftParser diagnostics) + DuplicateTypeNameError.
Sources/SwiftBuffet/Generator/Generator+ProtoConversion.swift	Adds conversion taxonomy + statement templates for init?(proto:).
Sources/SwiftBuffet/Generator/Generator+Messages.swift	SwiftSyntax-based struct generation, property docs/deprecation, initializers.
Sources/SwiftBuffet/Generator/Generator+Enums.swift	SwiftSyntax-based enum generation incl. allow_alias collapsing + nesting via extensions.
Sources/SwiftBuffet/Generator/Generator.swift	Assembles/formats output and re-parses to enforce valid Swift emission.
Sources/SwiftBuffet/Generator.swift	Removes legacy string-concatenation generator implementation.
Sources/SwiftBuffet/Documentation.docc/SwiftBuffet.md	Adds DocC landing page.
Sources/SwiftBuffet/Documentation.docc/SupportedProtoFeatures.md	Documents supported/skipped/rejected proto features and limitations.
Sources/SwiftBuffet/Documentation.docc/PluginConfiguration.md	Documents swiftbuffet.json plugin configuration mapping to CLI flags.
Sources/SwiftBuffet/Documentation.docc/GettingStarted.md	Adds getting-started guide for plugin usage.
Sources/SwiftBuffet/Documentation.docc/GeneratedCode.md	Documents generated code anatomy, naming, type mapping, and bridging rules.
Sources/SwiftBuffet/Documentation.docc/CommandLineUsage.md	Documents CLI options, examples, and error modes.
Sources/SwiftBuffet/AST.swift	Adds AST node types + flatten into generator models.
README.md	Expands README with rationale, examples, plugin config info, and docs pointer.
Plugins/SwiftBuffetPlugin/Plugin.swift	Adds swiftbuffet.json support for SwiftPM + Xcode plugin execution and tracks it as an input.
Package.swift	Adds SwiftSyntax/SwiftParser dependencies and pins swift-syntax version range.
Package.resolved	Updates resolved dependencies to include swift-syntax.
Example/Sources/swiftbuffet.json	Adds example plugin configuration file.
Example/Package.swift	Excludes swiftbuffet.json from target sources/resources to avoid SwiftPM warnings.
Example/Package.resolved	Updates example’s resolved dependencies to include swift-syntax.
.github/workflows/ci.yml	Adds CI workflow running tests and building the example package.

Comments suppressed due to low confidence (1)

Sources/SwiftBuffet/Utilities.swift:39

• primitiveTypes is documented as excluding integer-like proto types (since they need Int/UInt conversion), but the list currently includes several integer flavors (sint32, sfixed32, sint64, sfixed64, fixed32, fixed64). This is misleading and makes it easy for future code to accidentally treat these as passthrough scalars.

let primitiveTypes = [
   "double",
   "float",
   "sint32",
   "sfixed32",

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.