Skip to content

fix(dev): build the managed dev Goose backend at opt-level 1 - #14

Merged
matt2e merged 1 commit into
mainfrom
optimized-goose
Aug 13, 2026
Merged

fix(dev): build the managed dev Goose backend at opt-level 1#14
matt2e merged 1 commit into
mainfrom
optimized-goose

Conversation

@matt2e

@matt2e matt2e commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

The managed dev Goose backend is built with cargo's dev profile at opt-level 0,
and unoptimized async poll frames are fat enough that deep async descents inside
goose serve overflow the 2 MiB tokio worker stack. One goose serve process
backs every session, so a single overflow takes them all down. No release build
has reproduced it — it is debug-build-only, and it is the frame size that crosses
the line, not the recursion depth.

This sets CARGO_PROFILE_DEV_OPT_LEVEL=1 for the managed checkout's builds.
Going to --release would also fix it, but the dev profile keeps debug=true,
so goose's own SIGSEGV/SIGBUS handler can still symbolize the backtrace it prints
on a crash. The knob is GOOSE_DEV_OPT_LEVEL; set it to 0 to get an unoptimized
backend back for stepping through goose in a debugger. It is validated against
cargo's accepted set (0, 1, 2, 3, s, z) because the export is
unconditional and cargo parses the variable while loading its config, before it
picks a profile — an unvalidated typo would fail cargo build --release too and
break just bundle, a lane that never reads [profile.dev].

Two parts are load-bearing:

The opt-level participates in the build-skip decision.
ensure-local-goose.sh skips cargo entirely when stamp.env matches the pin, so
an already-built opt-0 binary at the current pin would keep being handed to
just dev forever — a no-op on exactly the machines that hit the crash. The
stamp now records STAMP_OPT_LEVEL, and the comparison defaults a missing field
to 0 (what a pre-knob stamp was actually built at) rather than to the current
value, so existing stamps fail the match, --check-bin exits 2, and the next
just setup / just goose-sync rebuilds optimized. Release stamps skip the
comparison: --release compiles under [profile.release] and never reads the
knob, so it is not part of their identity and they do not pay a spurious rebuild.

The knob lives in the shared lib/goose-dev-paths.sh, and
regenerate-sdk-schema.sh passes the same value when it builds
generate-acp-schema. Both land in the same cargo target dir; disagreeing on the
dev profile would have each run invalidate the other's fingerprints and recompile
the shared dependency graph on every alternation.

Scope

This reaches the lanes that build goose at GOOSE_BUILD_PROFILE=debug:
just dev, a bare just setup, just goose-sync, just sync-schema, and
scripts/dev-e2e.sh, plus the two that stage a debug backend into a bundle —
just bundle-debug and just stage-sidecar.

It does not change what any real bundle ships: just bundle and everything
routing through it (bundle-macos, bundle-linux, bundle-linux-docker),
scripts/release/build-macos.sh, and .github/workflows/release.yml all already
build goose at GOOSE_BUILD_PROFILE=release, which compiles under
[profile.release] and ignores the knob. prepare-goose-sidecar.sh is unchanged
— it still falls back to --check-bin, which resolves whichever profile its
caller asked for. The Windows path (scripts/windows/Ensure-LocalGoose.ps1) is
also unchanged — it is an independent implementation with its own stamp.json
format, and the crash work is on macOS.

This buys stack headroom; it does not bound the recursion. The durable fix is
goose-side — a larger worker stack for the serve runtime, or breaking the deep
descents onto their own tasks — and should land at the next pin bump.

Related issue

None found. Searched open and closed issues/PRs in block/berd and
squareup/berd for stack-overflow reports and dev-profile changes; no duplicate
or prior art.

Testing

No UI change — this only affects how the dev backend is compiled.

  • bash -n on all three scripts.
  • Stubbed-cargo runs covering the stamp paths: a pre-knob debug stamp and an
    opt-0 stamp are both rejected with --check-bin exit 2; an opt-1 stamp is
    accepted; GOOSE_DEV_OPT_LEVEL=0 rejects the opt-1 stamp; release stamps are
    reused regardless of the knob.
  • Validation: 0, 1, 2, 3, s, z each reach cargo; bogus, 4, S,
    01, and a padded value each exit 1 with the message and zero cargo
    invocations — under GOOSE_BUILD_PROFILE=release and --check-bin as well as
    a default build. The accepted set was read off cargo's own error text, which
    also confirmed 4 parses as config and only fails later in rustc.
  • The four ensure-local-goose.sh literals pinned by
    scripts/release/tests/release-scripts.test.mjs still match after the
    usage-block reorder, checked by exact string comparison rather than vitest —
    node_modules is absent in this checkout.
  • The real rebuild was verified separately: opt=1 with debug=true, binary
    459 MB → 280 MB.

The dev backend is built with cargo's dev profile at opt-level 0, and
unoptimized async poll frames are fat enough that goose's extension-add →
rmcp OAuth-metadata-discovery descent (~100 frames deep, entered whenever a
bundled remote MCP's server answers 401) overflows the 2 MiB tokio worker
stack. One `goose serve` process backs every session, so a single overflow
takes them all down. No release build has reproduced it — it is
debug-build-only, and it is the frame size that crosses the line, not the
recursion depth.

Set `CARGO_PROFILE_DEV_OPT_LEVEL=1` for the managed checkout's builds. Going
to `--release` would also fix it, but the dev profile keeps `debug=true`, so
goose's own SIGSEGV/SIGBUS handler can still symbolize the backtrace it prints
on a crash. The knob is `GOOSE_DEV_OPT_LEVEL`; set it to 0 to get an
unoptimized backend back for stepping through goose in a debugger. It is
validated against cargo's accepted set (`0`, `1`, `2`, `3`, `s`, `z`) because
the export is unconditional and cargo parses the variable while loading its
config, before it picks a profile — an unvalidated typo would fail
`cargo build --release` too and break `just bundle`, a lane that never reads
[profile.dev].

Two parts that are load-bearing:

- The opt-level participates in the build-skip decision. ensure-local-goose.sh
  skips cargo entirely when stamp.env matches the pin, so an already-built
  opt-0 binary at the current pin would keep being handed to `just dev`
  forever — a no-op on exactly the machines that hit the crash. The stamp now
  records STAMP_OPT_LEVEL, and the comparison defaults a missing field to 0
  (what a pre-knob stamp was actually built at) rather than to the current
  value, so existing stamps fail the match, `--check-bin` exits 2, and the
  next `just setup` / `just goose-sync` rebuilds optimized. Release stamps
  skip the comparison: `--release` compiles under [profile.release] and never
  reads the knob, so it is not part of their identity and they do not pay a
  spurious rebuild.

- The knob lives in the shared lib/goose-dev-paths.sh, and
  regenerate-sdk-schema.sh passes the same value when it builds
  generate-acp-schema. Both land in the same cargo target dir; disagreeing on
  the dev profile would have each run invalidate the other's fingerprints and
  recompile the shared dependency graph on every alternation.

This reaches the lanes that build goose at GOOSE_BUILD_PROFILE=debug: `just
dev`, a bare `just setup`, `just goose-sync`, `just sync-schema`, and
scripts/dev-e2e.sh, plus the two that stage a debug backend into a bundle —
`just bundle-debug` and `just stage-sidecar`. It does not change what any real
bundle ships: `just bundle` and everything routing through it (`bundle-macos`,
`bundle-linux`, `bundle-linux-docker`), scripts/release/build-macos.sh, and
.github/workflows/release.yml all already build goose at
GOOSE_BUILD_PROFILE=release, which compiles under [profile.release] and ignores
the knob. prepare-goose-sidecar.sh is unchanged — it still falls back to
`--check-bin`, which resolves whichever profile its caller asked for. The
Windows path (scripts/windows/Ensure-LocalGoose.ps1) is also unchanged — it is
an independent implementation with its own stamp.json format, and the crash
work is on macOS.

This buys stack headroom; it does not bound the recursion. The durable fix is
goose-side — spawning `oauth_flow` as its own task, or giving the serve
runtime a larger worker stack — and should land at the next pin bump.

Verified: `bash -n` on all three scripts, plus stubbed-cargo runs covering the
stamp paths (pre-knob debug stamp and an opt-0 stamp both rejected with
`--check-bin` exit 2; opt-1 stamp accepted; GOOSE_DEV_OPT_LEVEL=0 rejects the
opt-1 stamp; release stamps reuse regardless of the knob) and the new
validation (0, 1, 2, 3, s, z each reach cargo; bogus, 4, S, 01 and a padded
value each exit 1 with the message and zero cargo invocations, under
GOOSE_BUILD_PROFILE=release and `--check-bin` as well as a default build). The
accepted set was read off cargo's own error text, which also confirmed 4 parses
as config and only fails later in rustc. The four ensure-local-goose.sh
literals pinned by scripts/release/tests/release-scripts.test.mjs still match
after the usage-block reorder, checked by exact string comparison rather than
vitest — node_modules is absent in this checkout. The real rebuild was verified
separately: opt=1 with debug=true, binary 459 MB → 280 MB.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
@matt2e
matt2e requested a review from a team August 13, 2026 02:23
@matt2e
matt2e merged commit 9c208b1 into main Aug 13, 2026
10 checks passed
@matt2e
matt2e deleted the optimized-goose branch August 13, 2026 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants