perf(eryx): pool instances and pre-instantiate stores for stateless execution

book/src/SUMMARY.md

@@ -24,6 +24,7 @@
 - [MCP Server](./guide/mcp-server.md)
 - [gRPC Server](./guide/grpc-server.md)
 - [Pre-compiling Runtimes](./guide/precompile.md)
+- [Performance Tuning](./guide/performance.md)
 - [CLI](./guide/cli.md)
 
 # API Reference

book/src/guide/performance.md

@@ -0,0 +1,82 @@
+# Performance Tuning
+
+Most of the cost of a short `Sandbox::execute()` is not running Python. It is
+creating the WebAssembly instance the code runs on and tearing it down again:
+mapping the pre-initialized heap image, building a function reference for
+every entry in the runtime's ~6k-entry function table, and, afterwards,
+resetting the memory slot. Running `pass` on a fresh instance takes about
+1.3 ms on a fast workstation; the Python part of that is roughly 0.2 ms.
+
+Eryx hides that cost in two ways, both on by default. This page explains what
+they do and the environment variables that tune them. All of these are read
+once, when the process-wide wasmtime engine is created, so set them in the
+environment of the process that embeds eryx (for `pyeryx`, the Python process).
+
+## Warm instances
+
+After every stateless execution, a background task instantiates a replacement
+store for the same runtime and parks it in a process-wide pool. The next
+`execute()` takes it instead of instantiating, and the used store is dropped on
+the background task too. Every execution still runs on an instance that has
+never run user code, so isolation is unchanged; only the timing moves.
+
+The pool is keyed by runtime component rather than by `Sandbox`, so the common
+pattern of building a short-lived sandbox per request (for example
+`SandboxFactory.create_sandbox()` from Python) benefits from it: the first
+request in a process pays for instantiation, later ones do not.
+
+| Variable | Default | Effect |
+|----------|---------|--------|
+| `ERYX_WARM_INSTANCES` | `1` | Instances to keep ready per runtime. `0` disables the pool. |
+
+One ready instance is enough for a caller that executes serially. If several
+tasks execute concurrently, raise it towards that concurrency; each ready
+instance costs about the resident size of the pre-initialized heap (tens of
+megabytes, mostly shared copy-on-write pages) plus wasmtime's per-instance
+metadata.
+
+The pool is only used on a multi-threaded Tokio runtime, where the background
+work actually runs in parallel. On a current-thread runtime it would only add to
+the next request's latency, so it stays off. `PythonExecutor::warm_instances_ready()`
+reports how many instances are waiting for that executor's runtime.
+
+## Instance allocation
+
+Wasmtime can allocate each instance's linear memory, tables, and async stacks
+on demand (a fresh `mmap` per instance) or from a pool of pre-reserved slots.
+Eryx uses the pooling allocator: a slot that has just been released is reused
+by the next instance, so its pages are still mapped, and on Linux only the
+pages the previous instance dirtied are reset.
+
+| Variable | Default | Effect |
+|----------|---------|--------|
+| `ERYX_ALLOCATOR` | `pooling` | `pooling` or `on-demand`. |
+| `ERYX_POOL_INSTANCES` | `1000` | Maximum instances alive at once. Every running `execute()` and every live session holds one; instantiation fails once the pool is full. |
+| `ERYX_POOL_KEEP_RESIDENT_MB` | `64` | How much of each linear memory to keep mapped between uses. |
+| `ERYX_POOL_PAGEMAP_SCAN` | `1` | `1`: reset only dirty pages (Linux 6.7+, via `PAGEMAP_SCAN`). `0`: `memcpy` the whole keep-resident budget instead, which trades a larger copy for fewer page faults on the next execution. |
+
+The pool reserves virtual address space for every slot up front — several
+terabytes with the defaults, which is normal for wasmtime deployments but can
+be refused by a host with a low `ulimit -v`. If the pool cannot be created, eryx
+logs a warning and falls back to on-demand allocation; execution behaves
+identically either way.
+
+If a process holds more than `ERYX_POOL_INSTANCES` sessions open at once,
+raise the limit or switch to `on-demand`. Note that the choice of allocator does
+not affect precompiled `.cwasm` artifacts; only compilation settings do.
+
+## Measuring
+
+`crates/eryx/examples/profile_stateless.rs` times the stateless path and is a
+convenient target for `perf` or `samply`:
+
+```bash
+cargo build --example profile_stateless --features embedded --release
+# 2000 executions of `pass`, with a 2 ms gap between them so the background
+# replenishment gets the same chance it has between real requests
+./target/release/examples/profile_stateless 2000 pass 2000
+ERYX_WARM_INSTANCES=0 ./target/release/examples/profile_stateless 2000 pass 2000
+```
+
+The criterion benchmarks (`cargo bench --package eryx --features embedded`)
+cover the same paths with callbacks registered.

book/src/guide/resource-limits.md

@@ -260,6 +260,10 @@ Fuel limits provide fine-grained control over execution by limiting the number o
 - Billing based on actual computation performed
 - Preventing CPU-intensive attacks
 
+Fuel counts the instructions your code executes; instantiating the sandbox's
+WebAssembly instance is not charged, so the number does not depend on whether
+the execution ran on a fresh or a pre-instantiated instance.
+
 <!-- langtabs-start -->
 ```rust
 # extern crate eryx;

book/src/guide/sandboxes.md

@@ -230,7 +230,7 @@ If you need state to persist across executions, use a [Session](./sessions.md).
 
 ## SandboxFactory for Fast Creation
 
-When creating many sandboxes, use `SandboxFactory` to pre-initialize Python and packages once, then quickly instantiate sandboxes from that snapshot:
+When creating many sandboxes, use `SandboxFactory` to pre-initialize Python and packages once, then quickly instantiate sandboxes from that snapshot. Executions on short-lived sandboxes also pick up a pre-instantiated WebAssembly instance from a process-wide pool; see [Performance Tuning](./performance.md) for how that works and how to size it.
 
 <!-- langtabs-start -->
 ```rust

crates/eryx-python/README.md

@@ -85,6 +85,15 @@ print(f"Execution took {(time.perf_counter() - start) * 1000:.1f}ms")
 For repeated sandbox creation with custom packages, see
 [`SandboxFactory`](#sandboxfactory) below.
 
+Each `execute()` on a fresh sandbox also has to instantiate the WebAssembly
+runtime. Eryx hides most of that by keeping a pre-instantiated instance ready
+in the background and handing it to the next execution; the environment
+variables that tune this (`ERYX_WARM_INSTANCES`, `ERYX_ALLOCATOR`, ...) are
+documented in the
+[Performance Tuning](https://docs.eryx.run/guide/performance.html)
+guide. Set them in the environment of the Python process before importing
+`eryx`.
+
 ## API Reference
 
 **Core Classes:**

crates/eryx/Cargo.toml

@@ -48,6 +48,10 @@ name = "precompile"
 name = "profile_execution"
 required-features = ["embedded"]
 
+[[example]]
+name = "profile_stateless"
+required-features = ["embedded"]
+
 [[example]]
 name = "resource_limits"
 required-features = ["embedded"]
@@ -115,7 +119,7 @@ tar.workspace = true
 target-lexicon = "0.13"
 tempfile.workspace = true
 thiserror.workspace = true
-tokio = { workspace = true, features = ["sync", "net", "io-util", "time"] }
+tokio = { workspace = true, features = ["sync", "net", "io-util", "time", "rt"] }
 tokio-rustls.workspace = true
 tokio-util.workspace = true
 tracing.workspace = true

crates/eryx/benches/execution.rs

@@ -158,8 +158,10 @@ fn bench_sandbox_creation(c: &mut Criterion) {
 
 /// Benchmark stateless execution via `Sandbox::execute()`.
 ///
-/// Each call creates a fresh WASM instance and initializes Python from scratch.
-/// This is ~500ms per execution due to Python interpreter initialization.
+/// Each call runs on a fresh WASM instance. Python itself is already
+/// initialized in the pre-initialized snapshot, so the per-call cost is
+/// instantiation (or taking a warm instance from the pool, since this runtime
+/// is multi-threaded), the per-execute callback setup, and the code itself.
 ///
 /// Use this when you need complete isolation between executions.
 fn bench_stateless_execution(c: &mut Criterion) {
@@ -168,7 +170,7 @@ fn bench_stateless_execution(c: &mut Criterion) {
 
     let mut group = c.benchmark_group("stateless_execution");
 
-    // Stateless execution is slow (~500ms), so reduce sample size
+    // Stateless execution is the slowest path, so reduce sample size
     group.sample_size(10);
     group.measurement_time(Duration::from_secs(10));
 

crates/eryx/examples/profile_stateless.rs

@@ -0,0 +1,70 @@
+//! Profiling harness for stateless (fresh-instance) execution overhead.
+//!
+//! Each iteration goes through the full `Sandbox::execute()` path: a new
+//! `Store`, `instantiate_async` from the cached `InstancePre`, one `execute`
+//! export call, and teardown. This is the path a `SandboxFactory` render
+//! takes, so it is the one to profile for per-request latency.
+//!
+//! Run with samply:
+//!   cargo build --example profile_stateless --features embedded --release
+//!   samply record ./target/release/examples/profile_stateless
+//!
+//! Or count page faults per execution:
+//!   perf stat -e page-faults ./target/release/examples/profile_stateless 1000
+//!
+//! Arguments: `[iterations] [code] [gap_us]`. `gap_us` sleeps between
+//! executions so background work (warm-instance replenishment) gets the same
+//! chance it has between real requests; only the executions are timed.
+
+use std::time::{Duration, Instant};
+
+use eryx::Sandbox;
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let mut args = std::env::args().skip(1);
+    let iterations: u32 = args.next().and_then(|s| s.parse().ok()).unwrap_or(2000);
+    let code = args.next().unwrap_or_else(|| "pass".to_string());
+    let gap = Duration::from_micros(args.next().and_then(|s| s.parse().ok()).unwrap_or(0));
+
+    let rt = tokio::runtime::Runtime::new()?;
+
+    rt.block_on(async {
+        eprintln!("Creating sandbox...");
+        let sandbox = Sandbox::embedded().build()?;
+        let executor = sandbox.executor();
+
+        eprintln!("Warming up (10 iterations)...");
+        for _ in 0..10 {
+            sandbox.execute(&code).await?;
+            std::thread::sleep(gap);
+        }
+
+        eprintln!("Profiling {iterations} iterations of {code:?} with a {gap:?} gap...");
+        let mut executing = Duration::ZERO;
+        let mut warm_hits = 0u32;
+
+        for _ in 0..iterations {
+            if executor.warm_instances_ready() > 0 {
+                warm_hits += 1;
+            }
+            let start = Instant::now();
+            sandbox.execute(&code).await?;
+            executing += start.elapsed();
+            std::thread::sleep(gap);
+        }
+
+        eprintln!("\nResults:");
+        eprintln!("  Time executing: {executing:?}");
+        eprintln!("  Iterations: {iterations}");
+        eprintln!("  Warm instance available: {warm_hits}");
+        eprintln!("  Average: {:?} per execution", executing / iterations);
+        eprintln!(
+            "  Throughput: {:.0} executions/sec",
+            iterations as f64 / executing.as_secs_f64()
+        );
+
+        Ok::<_, Box<dyn std::error::Error>>(())
+    })?;
+
+    Ok(())
+}

crates/eryx/src/lib.rs

@@ -68,6 +68,7 @@ mod schema;
 pub mod secrets;
 pub mod session;
 mod trace;
+mod warm;
 mod wasm;
 
 /// Pre-initialization support for capturing Python memory state.