feat: unbundle Z3 from gateway runtime packaging

[pimlock]

Problem Statement

OpenShell currently keeps the Linux openshell-gateway release artifact on bundled-z3 so the gateway stays portable across Debian/RPM/container/tarball surfaces while preserving the glibc 2.28 floor from #1937 and PR #1934.

That is the right short-term compatibility fix, but it leaves OpenShell owning a vendored/static Z3 build inside the gateway artifact. Long term, the gateway should prefer a runtime/system Z3 dependency where the packaging surface can provide and update Z3, while still keeping the gateway's host compatibility contract explicit and tested.

The earlier spike in #1937 showed this is not just a feature-flag flip: unbundling worked for some package surfaces, but the first portable Linux GNU artifact became distro-sensitive because the gateway was linked against the Z3 SONAME from the build environment.

Proposed Design

Design and implement a packaging-aware unbundled Z3 strategy for openshell-gateway.

The desired end state should:

• build Linux gateway release/package artifacts without bundled-z3 where a packaging surface can provide a compatible runtime Z3;
• keep verify-glibc-symbols.sh 2.28 or an equivalent guardrail on GNU/Linux gateway artifacts;
• define the Z3 runtime contract separately for each published surface:
  • Debian packages: depend on the distro package that provides the expected Z3 shared library;
  • RPM packages: build or link in a way that matches the RPM target distro's Z3 shared library, rather than reusing a Debian-linked gateway binary;
  • Snap: continue staging the correct Z3 runtime package if applicable;
  • Homebrew: rely on the existing Homebrew z3 dependency if applicable;
  • container images: install or copy the matching Z3 runtime into the image;
  • standalone tarballs: either document the system dependency clearly or ship a matching sidecar runtime with explicit RPATH/RUNPATH behavior;
• add smoke tests for each package surface that run LD_BIND_NOW=1 openshell-gateway --version or a stronger startup probe;
• update install/support docs to describe whether each gateway artifact is self-contained or expects runtime Z3.

The implementation should avoid losing the Rocky/RHEL 8 compatibility work. If RPM artifacts need to build in a Fedora/RPM environment to link against Fedora Z3, that build still needs an explicit glibc-floor strategy such as cargo-zigbuild, a compatible sysroot, or another verifier-backed approach.

Alternatives Considered

Keep bundled Z3 for all gateway artifacts

This is the current short-term state from PR #1934. It gives one portable gateway artifact, avoids cross-distro Z3 SONAME differences, and keeps the glibc 2.28 verifier as the primary runtime guardrail.

The downside is that OpenShell continues to carry the vendored Z3 build path, including Zig C/C++ wrapper setup, CMake cache handling, longer builds, and more ownership of the Z3 runtime update story.

Build one unbundled Linux GNU artifact in the Debian/Ubuntu-style build environment

This was tried during the #1937 spike. The gateway built without bundled-z3, linked dynamically to Z3, and stayed within the intended glibc 2.28 symbol floor. Debian packaging worked with Depends: libz3-4; Snap and Homebrew also looked plausible because they already stage or depend on Z3.

The RPM smoke failed because the gateway required libz3.so.4, while Fedora 44 provided z3-libs-4.16.0-1.fc44 with a different shared-library contract. That made the supposedly portable gateway binary distro-sensitive again.

Ship a matching shared Z3 runtime next to the gateway

This could make tarballs and RPMs work with a Debian-linked gateway by carrying the matching Z3 shared object and wiring RPATH/RUNPATH.

The downside is that OpenShell would still own Z3 runtime provenance and security updates, and would need to account for Z3's own C++ runtime dependencies. That may be better than static bundling in some ways, but it is not the same as relying on the platform package manager.

Agent Investigation

Context extracted from #1937:

• feat: expand gateway glibc compatibility #1937 tracks lowering the gateway GNU/Linux glibc floor to 2.28 for Rocky/RHEL 8 class hosts.
• PR fix(linux): lower host glibc floor to 2.28 to support RHEL/Rocky 8 #1934 keeps the gateway on bundled-z3 for Linux release artifacts after making the explicit glibc 2.28 path work.
• The unbundled/system-Z3 spike showed:

  • Linux gateway artifacts can build with an explicit GNU Zig target without bundled-z3.

  • The resulting gateway can keep the intended GLIBC_2.28 floor.

  • Debian packaging worked with Depends: libz3-4.

  • Snap already stages libz3-4.

  • Homebrew already declares depends_on "z3".

  • RPM package smoke failed at gateway startup with:

    openshell-gateway: error while loading shared libraries: libz3.so.4: cannot open shared object file: No such file or directory
    

  • Installing Fedora z3-libs / z3-devel was not sufficient because the gateway had been linked in a Debian/Ubuntu-style environment and expected libz3.so.4.

• The current bundled-Z3 follow-up added Zig C/C++ wrapper setup for z3-sys, GLIBC symbol verification, and cache-key handling around the wrapper. That is useful short-term hardening, but it is also evidence that the vendored Z3 path has non-trivial build complexity.

Relevant references:

• feat: expand gateway glibc compatibility #1937
• feat: expand gateway glibc compatibility #1937 comment: feat: expand gateway glibc compatibility #1937 (comment)
• feat: expand gateway glibc compatibility #1937 bundled-Z3 follow-up: feat: expand gateway glibc compatibility #1937 (comment)
• PR fix(linux): lower host glibc floor to 2.28 to support RHEL/Rocky 8 #1934

Definition of Done

• Decide which gateway artifact surfaces should use runtime/system Z3 and which, if any, should remain bundled.
• Define the per-surface runtime dependency contract for Debian, RPM, Snap, Homebrew, container images, and standalone tarballs.
• Update gateway build/release workflows to build unbundled artifacts where appropriate without raising the GNU/Linux glibc floor above 2.28.
• Add CI checks that verify both the glibc floor and the expected Z3 linkage model for each surface.
• Add package/container smoke tests that catch missing Z3 shared-library dependencies with eager binding.
• Update install/support docs with the final Z3 runtime dependency story.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request