Skip to content

feat: make trace collection easy to disable (docs + ERYX_PROFILE_TRACE) - #424

Merged
sd2k merged 1 commit into
mainfrom
feat/trace-collection-knob
Sep 11, 2026
Merged

sd2k merged 1 commit into
mainfrom
feat/trace-collection-knob

Conversation

@sd2k

@sd2k sd2k commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Why

SandboxBuilder defaults to collect_trace: true, which installs Python's sys.settrace hook for every execution. The hook fires on every line/call/return, so its cost scales with the amount of Python executed and it dominates anything heavier than pass. Measured 2026-09-10 (stock wasmtime 48.0.1, Ryzen 9 7950X, fresh instance per call):

Workload Trace on Trace off
cold pass 1279 µs 1078 µs
IRM-like render (import json, string; d = json.loads(...); string.Template(...).substitute(...)) 3.7 ms 0.9 ms
total = sum(i * i for i in range(20_000)) 369 ms 3.8 ms (~96x)

The knob to turn it off already exists (SandboxBuilder::with_trace_collection(false), added in #270), but nothing pointed at it: the doc comment did not quantify the cost, the profiling harness and criterion bench always ran with it on, and the book never mentioned it. This PR makes the knob discoverable and adds an env switch for measurement. It does not change any default, so existing numbers stay comparable.

What changed

  • Rust API (crates/eryx/src/sandbox.rs): with_trace_collection doc comment now carries the cost table above, an example, and notes that InProcessSessions created from the sandbox inherit the setting while a TraceHandler keeps tracing on regardless. No new API: with_trace_collection already covers sandboxes and in-process sessions, and SessionExecutor never installs sys.settrace unless with_tracing(tx) is called explicitly.

  • Harness / benches: ERYX_PROFILE_TRACE=0 disables trace collection (same semantics as the knob in the closed perf(eryx): keep the instance heap mapped with glibc malloc tunables #419: any other value or unset keeps the default) in

    • crates/eryx/examples/profile_execution.rs
    • crates/eryx/examples/session_bench.rs
    • crates/eryx/benches/execution.rs (create_sandbox() helper, so every criterion group honours it)

    If perf/instantiation-overhead (perf(eryx): pool instances and pre-instantiate stores for stateless execution #411) merges, its profile_stateless.rs should get the same three lines.

  • Python: no API change, deliberately. eryx.Sandbox, SandboxFactory.create_sandbox and (via SessionExecutor) Session have had trace collection hardcoded off since Optimize callback setup and Python tracing #270, and pyeryx's ExecuteResult does not expose a trace field, so a collect_trace=True option would be a pure performance regression that exposes nothing. Instead the Sandbox docstring / _eryx.pyi now say so explicitly, and the two with_trace_collection(false) sites carry a comment explaining why.

  • Book (book/src/guide/sandboxes.md): new "Trace Collection" subsection under Execution Results with the cost and how to disable it.

How to disable tracing

let sandbox = Sandbox::embedded().with_trace_collection(false).build()?;
// InProcessSession::new(&sandbox) inherits this.
# Nothing to do: pyeryx never collects traces (ExecuteResult has no trace field).
sandbox = eryx.Sandbox()
# Profiling harness / criterion bench:
ERYX_PROFILE_TRACE=0 cargo run --release --example profile_execution --features embedded
ERYX_PROFILE_TRACE=0 cargo bench --package eryx --features embedded

Tests

  • test_trace_collection_can_be_disabled / test_trace_handler_without_result_collection in crates/eryx/tests/trace_events_precise.rs already cover the Rust behaviour; no new test was needed. No Python test was added because there is no Python-visible behaviour to assert (no option, no trace field).
  • cargo nextest run --workspace --features embedded --cargo-profile release: 624 passed, 0 failed
  • cargo clippy --workspace --all-targets --all-features -- -D warnings: clean
  • cargo fmt --all --check: clean
  • pyeryx mise run build + uv run pytest tests/ -q: 333 passed, 0 failed

🤖 Generated with Claude Code

`SandboxBuilder` defaults to `collect_trace: true`, which installs
`sys.settrace` for every execution. The hook's cost scales with the
amount of Python run: cold `pass` 1.28 ms vs 1.08 ms, a small
json + string.Template render 3.7 ms vs 0.9 ms, and
`sum(i * i for i in range(20_000))` 369 ms vs 3.8 ms (~96x). The knob
to turn it off (`with_trace_collection(false)`, #270) existed but was
not discoverable and the profiling harness always ran with it on.

- Document the cost and the session/handler interaction on
  `SandboxBuilder::with_trace_collection`.
- Add an `ERYX_PROFILE_TRACE=0` env switch to `profile_execution.rs`,
  `session_bench.rs` and the criterion bench's `create_sandbox()`.
  Default (unset) keeps tracing on so existing numbers stay comparable.
- Explain in the pyeryx `Sandbox` docstring / stubs that Python
  sandboxes never collect traces (hardcoded off since #270 because
  `ExecuteResult` has no trace field), and comment the two call sites.
- Add a "Trace Collection" subsection to the sandboxes guide.

No defaults change and no new API is added.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

@sd2k
sd2k merged commit ec5beab into main Sep 11, 2026
31 checks passed
@sd2k
sd2k deleted the feat/trace-collection-knob branch September 11, 2026 08:30
@eryx-release-plz eryx-release-plz Bot mentioned this pull request Sep 11, 2026
@eryx-release-plz eryx-release-plz Bot mentioned this pull request Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant