perf: proto schema thinning (dual-variant build tags) to cut decode GC - #388
Open
jfberry wants to merge 3 commits into
Open
perf: proto schema thinning (dual-variant build tags) to cut decode GC#388jfberry wants to merge 3 commits into
jfberry wants to merge 3 commits into
Conversation
… thinning go/packages + go/types analyzer that records the exact (message, field) pairs Golbat accesses, plus reflective escape hatches (ProtoReflect/String/proto. Marshal/etc). Type-precise (no getter-name collisions; catches direct field access too), replacing the grep heuristic that drove the thinning measurement. Run against Golbat: 146 message types, 409 fields accessed, ~50%+ of accessed messages' fields thinnable (PokemonProto 13/78, PokemonFortProto 23/45). Only 1 escape hatch and it's a cosmetic .String() in a log line on a cold path — Golbat is clean for static thinning. Separate module (no deps added to golbat). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Make schema thinning a working build option. The full pogo/vbase.pb.go (//go:build !thin, IDE/contributor field discovery) and a generated pogo/vbase.thin.pb.go (//go:build thin) ship side by side; the thin variant keeps only the 417/15807 descriptor fields Golbat's code reads (removes 97%), so decode skips the rest as unknown fields (with DiscardUnknown) instead of allocating their subtrees. Field numbers are preserved, so decoded values are identical — the full test suite passes unchanged under -tags thin, which is the full-vs-thin differential. Pipeline (scripts/thin.sh, maintainer-only — needs the licensed .proto): protofields (incl. tests) -> used (message,field) set prototrim -> trim FileDescriptorSet, keep field numbers + oneof structure protoc -> pogo/vbase.thin.pb.go (//go:build thin) - tools/protofields/prototrim: descriptor-set thinner. Exact protoc-gen-go GoCamelCase; keeps a whole real oneof when its wrapper field is accessed via a type switch; drops emptied oneof decls and remaps oneof_index. - protofields: dedup accessed files per-file (not per-package) so test-augmented package variants are analyzed too (composite-literal field sets in _test.go). - make golbat / Dockerfile now build -tags thin (end users); make golbat-full and plain go build stay full (contributors). Gates (all green): go build ./... , go build -tags thin ./... , go test ./... , go test -tags thin ./... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rom compiled pogo Rebased onto main, which regenerated pogo/vbase.pb.go (new schema). The shipped thin variant was generated from the old schema, so regenerate it. Also make the thinning pipeline source-independent of vbase.proto: tools/dumpdesc extracts the full FileDescriptorSet from the COMPILED pogo package (pogo.File_vbase_proto) instead of running protoc on the licensed .proto. The thin variant is therefore guaranteed a faithful subset of exactly what pogo/vbase.pb.go ships, and scripts/thin.sh now needs only protoc + Go — no .proto access. Deterministic: repeated runs are byte-identical. Against main's schema: 149 message types / 416 fields accessed; thin keeps 422 of 15807 descriptor fields (removes 97%). All four gates green: go build ./... , go build -tags thin ./... , go test ./... , go test -tags thin ./... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jfberry
force-pushed
the
perf/proto-thinning
branch
from
July 29, 2026 10:16
9fe9653 to
5dc471b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Golbat decodes protobuf from game clients but only reads ~420 of the POGO schema's ~15,800 fields. Every unread field is still decoded into allocated Go structs on every message — a large source of GC pressure (GMO decoding dominates allocation).
This PR makes schema thinning a working build option: a regenerated
pogovariant that keeps only the fields Golbat reads. Removed fields keep their field numbers, so they become unknown fields on the wire and — paired withDiscardUnknown— are skipped instead of decoded. It's the static, dependency-free equivalent of lazy decoding: the decode cost of ~97% of the schema disappears, with byte-identical decoded values for the fields we do read.How it ships — two build-tag variants
The
.protocan't be redistributed (license), so the generated Go ships in two mutually-exclusive variants:pogo/vbase.pb.go//go:build !thingo build— full field discoverypogo/vbase.thin.pb.go//go:build thinmake golbat/ Docker — the production winSame package, same Go type names, only one compiled at a time → no proto-registry clash.
make golbatand the Dockerfile now build-tags thin(end users get the win by default).make golbat-fulland plaingo buildstay full (contributors / IDE).Why it's safe
.String()in a log and threeproto.Marshalcalls in test fixtures). So dropping unread fields can't lose data.DiscardUnknownis included (it's coupled to thinning: without it, protobuf-go retains each removed field as a raw-bytes unknown blob, clawing back the win).decode.go'sunmarshalClientProtoroutes every inbound client decode throughDiscardUnknown; Golbat's own internal proto (GolbatInternal) intentionally keeps plainproto.Unmarshal.The regeneration pipeline (
scripts/thin.sh, maintainer-side)Needs only
protoc+ Go — not the licensed.proto:tools/dumpdesc→ fullFileDescriptorSet, extracted from the already-compiledpogo.File_vbase_proto(so the thin variant is a guaranteed faithful subset of whatvbase.pb.goships).tools/protofields→ the exact(message, field)set Golbat's code accesses (type-precise viago/types; includes_test.goso the suite runs under-tags thin).tools/protofields/prototrim→ trims the descriptor set to that set, preserving field numbers + oneof structure.protoc --descriptor_set_in→pogo/vbase.thin.pb.go.Deterministic (repeated runs are byte-identical). Rerun whenever
vbase.pb.gois regenerated or field usage changes.Correctness gate
The differential is elegant: the full test suite compiled
-tags thinIS the full-vs-thin comparison — the same tests decode real protos and assert entity outcomes, so passing under both tags proves thin decodes identically. CI should require all four:A contributor who reads a not-yet-thinned field gets a clean
-tags thincompile error naming the exact field — never a silent wrong decode.Reviewing
The two generated
pogo/*.pb.gofiles are large; the reviewable change is the ~700 lines of tooling + wiring:tools/protofields/,tools/dumpdesc/,scripts/thin.sh,decode*.go(DiscardUnknown),Makefile,Dockerfile. Seetools/protofields/README.mdfor the design.