Skip to content

In-house FlatBuffers toolchain: drop flatc + com.google.flatbuffers (ADR 0017)#148

Merged
dfa1 merged 1 commit into
mainfrom
fbs-gen
Jun 24, 2026
Merged

In-house FlatBuffers toolchain: drop flatc + com.google.flatbuffers (ADR 0017)#148
dfa1 merged 1 commit into
mainfrom
fbs-gen

Conversation

@dfa1

@dfa1 dfa1 commented Jun 23, 2026

Copy link
Copy Markdown
Owner

Replaces the external flatc compiler and the com.google.flatbuffers runtime with an in-house, MemorySegment-native FlatBuffers toolchain — mirroring the existing proto-gen. Implements ADR 0017.

Why

  1. MemorySegment-native end to end. The reader memory-maps the file into one MemorySegment; the generated readers/builder now operate on it directly — no ByteBuffer bridge in the codec, zero-copy slices.
  2. Unblocks JPMS / clean Maven Central. flatbuffers-java shipped as an automatic module (filename-derived name), which a published named module cannot reliably requires. With both wire formats self-hosted, no upstream automatic module remains → module-info.java is now viable.

Also drops the external flatc CLI (no brew install, no ValidateVersion version-guard strip hack) and restores the four .fbs schemas to upstream-pristine names (the Fbs prefix now lives in the generator).

What

  • fbs-gen module.fbs lexer/parser/AST + code generator (mirrors proto-gen).
  • io.github.dfa1.vortex.fbsrt runtimeFbsTable / FbsStruct (read) + FbsBuilder (write) over MemorySegment. Full FlatBuffers algorithm: vtable lookup/dedup, back-to-front build, alignment, default omission, unions, inline structs, strings.
  • Generated classes carry @Generated; output names unchanged (FbsArray, …), so no consumer churn beyond the API-shape moves (getRootAsX(MemorySegment), byte-vector xAsSegment()).
  • regenerate-sources profile runs fbs-gen (+ proto-gen) execs; flatc exec + antrun strip removed.
  • Cleanups: killed MemorySegment→ByteBuffer→MemorySegment double-conversions in the flat-segment decoders; migrated Layout.metadata to MemorySegment.

Validation

  • Byte-identical to flatc (builder validated against the flatc builder for the same sequences).
  • Full unit suite + 275 integration tests (Rust interop: Rust reads our output, we read Rust's) pass.

Follow-ups (not in this PR)

  • Add module-info.java (now unblocked) and non-export the fbs/proto packages.
  • Finish the ByteBuffer purge: decoder ctx.metadata() (~25 decoders) and ZonedStatsSchema.

🤖 Generated with Claude Code

… format (ADR 0017)

See ADR 0017. Replaces flatc + com.google.flatbuffers with an in-house, MemorySegment-native FlatBuffers toolchain (fbs-gen + fbsrt runtime), and migrates every wire-format metadata path (Layout, decode/encode nodes, DType.Extension, zone-map stats, postscript plumbing) off java.nio.ByteBuffer. Unblocks JPMS (no upstream automatic module). Full unit suite + 275 Rust-interop integration tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dfa1 dfa1 merged commit 5907302 into main Jun 24, 2026
6 checks passed
@dfa1 dfa1 deleted the fbs-gen branch June 24, 2026 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant