Skip to content

perf: proto schema thinning (dual-variant build tags) to cut decode GC - #388

Open
jfberry wants to merge 3 commits into
mainfrom
perf/proto-thinning
Open

perf: proto schema thinning (dual-variant build tags) to cut decode GC#388
jfberry wants to merge 3 commits into
mainfrom
perf/proto-thinning

Conversation

@jfberry

@jfberry jfberry commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

What & why

Golbat decodes protobuf from game clients but only reads ~420 of the POGO schema's ~15,800 fields. Every unread field is still decoded into allocated Go structs on every message — a large source of GC pressure (GMO decoding dominates allocation).

This PR makes schema thinning a working build option: a regenerated pogo variant that keeps only the fields Golbat reads. Removed fields keep their field numbers, so they become unknown fields on the wire and — paired with DiscardUnknown — are skipped instead of decoded. It's the static, dependency-free equivalent of lazy decoding: the decode cost of ~97% of the schema disappears, with byte-identical decoded values for the fields we do read.

How it ships — two build-tag variants

The .proto can't be redistributed (license), so the generated Go ships in two mutually-exclusive variants:

File Build tag Used by
pogo/vbase.pb.go //go:build !thin contributors, gopls, plain go build — full field discovery
pogo/vbase.thin.pb.go //go:build thin make golbat / Docker — the production win

Same package, same Go type names, only one compiled at a time → no proto-registry clash.

  • make golbat and the Dockerfile now build -tags thin (end users get the win by default).
  • make golbat-full and plain go build stay full (contributors / IDE).

Why it's safe

  • Field numbers preserved → wire compatibility unchanged; kept fields decode identically.
  • Golbat is read-only against client protos and never re-marshals them (the analyzer confirms — the only reflective escapes are one cosmetic .String() in a log and three proto.Marshal calls in test fixtures). So dropping unread fields can't lose data.
  • DiscardUnknown is included (it's coupled to thinning: without it, protobuf-go retains each removed field as a raw-bytes unknown blob, clawing back the win). decode.go's unmarshalClientProto routes every inbound client decode through DiscardUnknown; Golbat's own internal proto (GolbatInternal) intentionally keeps plain proto.Unmarshal.

The regeneration pipeline (scripts/thin.sh, maintainer-side)

Needs only protoc + Go — not the licensed .proto:

  1. tools/dumpdesc → full FileDescriptorSet, extracted from the already-compiled pogo.File_vbase_proto (so the thin variant is a guaranteed faithful subset of what vbase.pb.go ships).
  2. tools/protofields → the exact (message, field) set Golbat's code accesses (type-precise via go/types; includes _test.go so the suite runs under -tags thin).
  3. tools/protofields/prototrim → trims the descriptor set to that set, preserving field numbers + oneof structure.
  4. protoc --descriptor_set_inpogo/vbase.thin.pb.go.

Deterministic (repeated runs are byte-identical). Rerun whenever vbase.pb.go is regenerated or field usage changes.

Correctness gate

The differential is elegant: the full test suite compiled -tags thin IS the full-vs-thin comparison — the same tests decode real protos and assert entity outcomes, so passing under both tags proves thin decodes identically. CI should require all four:

go build ./...            go test ./...             # full
go build -tags thin ./...  go test -tags thin ./...  # thin (differential)

A contributor who reads a not-yet-thinned field gets a clean -tags thin compile error naming the exact field — never a silent wrong decode.

Reviewing

The two generated pogo/*.pb.go files are large; the reviewable change is the ~700 lines of tooling + wiring: tools/protofields/, tools/dumpdesc/, scripts/thin.sh, decode*.go (DiscardUnknown), Makefile, Dockerfile. See tools/protofields/README.md for the design.

jfberry and others added 3 commits July 29, 2026 11:15
… thinning

go/packages + go/types analyzer that records the exact (message, field) pairs
Golbat accesses, plus reflective escape hatches (ProtoReflect/String/proto.
Marshal/etc). Type-precise (no getter-name collisions; catches direct field
access too), replacing the grep heuristic that drove the thinning measurement.

Run against Golbat: 146 message types, 409 fields accessed, ~50%+ of accessed
messages' fields thinnable (PokemonProto 13/78, PokemonFortProto 23/45). Only
1 escape hatch and it's a cosmetic .String() in a log line on a cold path —
Golbat is clean for static thinning. Separate module (no deps added to golbat).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Make schema thinning a working build option. The full pogo/vbase.pb.go
(//go:build !thin, IDE/contributor field discovery) and a generated
pogo/vbase.thin.pb.go (//go:build thin) ship side by side; the thin variant
keeps only the 417/15807 descriptor fields Golbat's code reads (removes 97%),
so decode skips the rest as unknown fields (with DiscardUnknown) instead of
allocating their subtrees. Field numbers are preserved, so decoded values are
identical — the full test suite passes unchanged under -tags thin, which is the
full-vs-thin differential.

Pipeline (scripts/thin.sh, maintainer-only — needs the licensed .proto):
  protofields (incl. tests) -> used (message,field) set
  prototrim  -> trim FileDescriptorSet, keep field numbers + oneof structure
  protoc     -> pogo/vbase.thin.pb.go (//go:build thin)

- tools/protofields/prototrim: descriptor-set thinner. Exact protoc-gen-go
  GoCamelCase; keeps a whole real oneof when its wrapper field is accessed via
  a type switch; drops emptied oneof decls and remaps oneof_index.
- protofields: dedup accessed files per-file (not per-package) so test-augmented
  package variants are analyzed too (composite-literal field sets in _test.go).
- make golbat / Dockerfile now build -tags thin (end users); make golbat-full
  and plain go build stay full (contributors).

Gates (all green): go build ./... , go build -tags thin ./... ,
go test ./... , go test -tags thin ./...

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rom compiled pogo

Rebased onto main, which regenerated pogo/vbase.pb.go (new schema). The shipped
thin variant was generated from the old schema, so regenerate it.

Also make the thinning pipeline source-independent of vbase.proto: tools/dumpdesc
extracts the full FileDescriptorSet from the COMPILED pogo package
(pogo.File_vbase_proto) instead of running protoc on the licensed .proto. The
thin variant is therefore guaranteed a faithful subset of exactly what
pogo/vbase.pb.go ships, and scripts/thin.sh now needs only protoc + Go — no
.proto access. Deterministic: repeated runs are byte-identical.

Against main's schema: 149 message types / 416 fields accessed; thin keeps 422
of 15807 descriptor fields (removes 97%). All four gates green:
go build ./... , go build -tags thin ./... , go test ./... , go test -tags thin ./...

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jfberry
jfberry force-pushed the perf/proto-thinning branch from 9fe9653 to 5dc471b Compare July 29, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant