Repository navigation
perf: run a direct call's callee lazily (RFC 0078 stage 2) - #94
Merged
Merged
Conversation
Record where WeavePy stands against CPython 3.14 (warm time 1.28x, peak memory 1.71x, startup 0.74x), the callgrind breakdown of the remaining gap, and the ranked plan for the structural changes.
A call between two compiled functions bound a pooled InlineAct, pushed it on the core loop's switch, and entered its caller on the lazy spine's pending list, about 480 instructions in the call helper alone. It now binds a per-depth LightAct record instead: the callee's frame and native state, with its code, globals and builtins borrowed from the callable and no switch or pending-list work at all. Nothing outside the call sees a light activation. Whatever needs the core loop's view of it first reifies it, and the light activations that called it, into the ordinary direct activations the old path bound: the callee leaving its native code, any helper that switches the running activation or publishes its caller (ensure_heavy), and a return that isn't clean. A site whose light calls keep being reified binds ordinary activations instead. Positional and keyword calls, operator dunders, property getters and callees with cells all take the light path; core_bind_kw_direct is gone. go retires 6% fewer instructions, tomllib_loads 3%, deepcopy 1%; the suite's geomean 0.7%. New burst-stat lines count light calls and reifications, and a regression test covers frames, tracebacks, locals, finalizer timing, the recursion limit and warning stack levels from hot direct calls.
Replace the light activations with one bind for every direct call between compiled functions (frame_jit::direct_enter): positional and keyword arguments, defaults, and fresh or closure cells, into a pooled or site-warm InlineAct pushed on the core loop's switch, as the core loop binds its own calls. The light records skipped the switch and pending-list registration, about 40 instructions a call, but anything that switched activations had to reify them first, which kept constructors, generator resumes and builtin lanes off them and took 15 call sites and a frame swap to get right. They also borrowed the callee's code, which a reassigned __code__ could free mid-call. With one representation, every shape runs direct and nothing needs reifying: core_bind_kw_direct, bind_call_cells and core_bind_plain's DIRECT variant are gone. Instructions are within 0.3% of before on the call-heavy workloads (the geomean of go, raytrace, deepcopy, tomllib_loads, mini_interpreter, sudoku, chaos and exceptions is 0.997). The next step emits the call in line from compiled code against this record.
Record the design the call-path measurements point to: activations as records on a per-thread stack, linked to their callers, that the interpreter, the frame JIT and generators all run, replacing the pooled InlineAct, the lazy pending list, the per-call native State and the frameless leaf paths. The RFC gives the measured per-call cost breakdown, a five-stage plan, and the expected effect.
…ipt work in the progress note
RFC 0078, stage 1: `Frame::stack` becomes a `FrameStack` with the operations frames use (push and pop, the depth, the base pointer the native code addresses, and the top-of-stack moves the call paths make), so that a later stage can keep it in an activation record without touching its users. It is still a heap buffer. The helpers that took a frame's stack as a `&mut Vec<Object>` take the new type; `kw_names_fill_locals`, which also binds from staging vectors, takes either through a small `Operands` trait. No behavior changes.
RFC 0078, stage 1: `Frame::locals` becomes a `FrameLocals`, which makes explicit each way the VM reaches a frame's locals, so that a later stage can keep them in an activation record: - `shared()`: the handle a frame object or a shell holds to read the live values (eight sites), and `is_shared()` in place of the `Rc::strong_count` tests that decide whether a return may recycle them; - `len()`, `as_mut_ptr()` and `slots()`: the running frame's unguarded reads and the native code's base pointer; - `storage()`: the vector itself, for the binding paths that size, fill and empty it (29 sites, the work stage 2 replaces); - `identity()`: the frame's identity while it runs, for the guards that key on it. The storage is still the shared `Rc<RefCell<Vec<Object>>>` (`SharedLocals`). No behavior changes; instructions are within layout noise of `main` (geomean 0.997).
RFC 0078, stage 2. A call between compiled functions bound its callee into an InlineAct and registered it with the core loop at once: the caller entered on the pending list, the callee pushed on the switch and made its running activation, all undone at the return, though the core loop needs that view only when the callee leaves its native code. The callee now runs lazily: bound as before, but its state points at its unregistered activation (`State::lazy`), and its caller stays the switch's running activation. `register_lazy` does the registration the call skipped, its lazy callers first, and is due - when the callee leaves for the core loop (any return but a clean one, any other status); - before a helper uses the switch for it (the call, generator, send, catch and return lanes: 30 points, each right before the use, so a builtin call in place pays nothing); - before anything links a newer activation onto the spine or reads it (`Interpreter::settle_lazy` in the pending-list enters, the shell pushes, the flushes and the stack walkers), so Python code a helper runs (a property, an `__eq__`, a finalizer) sees the callee in its place. A clean return of a callee that never registered skips the pop and the switch's restore. The regression test gains the cases where a helper's Python code walks the stack. Instructions: a direct call about 88 fewer; go 0.960, tomllib_loads 0.982, closures_decorators 0.986, geomean 0.997, none worse than 1.004.
A positional call of a function with `*args`, `**kwargs`, or keyword-only parameters (a decorator's `def wrapper(*args, **kwargs)`, `def f(a, *, key=None)`) took the general inline call every time: `core_call_plain` and the frame JIT's direct call declined extended parameters and closures, and `core_call_cached` re-derived the shape on each call. The cached plain call and the direct call now bind these shapes themselves (`fill_extended_raw`: the surplus as the `*args` tuple, the compiled keyword-only defaults, an empty `**kwargs` dictionary), along with the callee's cells, so `core_call_cached` goes away. A direct site carries the extended-parameter flag beside its cells flag, so the common shape pays nothing for the check. Frameless leaves with keyword-only parameters take their defaults at frame JIT leaf sites and fused method sites too, and a shape mismatch no longer reaches `pure_leaf_call`, whose bail counter could otherwise retire a leaf for good. Per call (instructions retired, quick build): the forwarding wrapper 3368 -> ~2460, `fva(1, 2, 3)` 1817 -> ~1245, `fkw(i)` 835 -> ~760. Suite geomean 0.999 (closures_decorators -1.3%, context_managers -1.8%).
Keyword calls that the call site's permutation can't bind (a callee with `*args`, a closure, a native caller) now share one by-name binder, `lean_bind_keywords`, which takes its positionals as slices: the `*args` surplus builds its tuple in place, the first keyword is tried at the parameter after the positionals, and keyword-only defaults are looked up only for the parameters left unbound. `call_python_kw_names` binds through it instead of computing a permutation per call, and a tier-2 dynamic keyword call of a plain function now goes there too instead of converting each keyword name into a `String` for the generic binder. `f(*args, **kwargs)` of a frameless leaf (the forwarding wrapper's inner call) evaluates without allocating: the operands leave the stack as a fixed array instead of a collected vector, the function and code are borrowed, and keyword-only parameters take their defaults. A static or class method read through an instance is remembered in the method-load site's slot (`MethodSlotFn::ClassFn` beside `Static`), so the frame JIT's method load serves it and the call stays native. Per call (instructions retired): the forwarding wrapper ~2460 -> ~2270, `o.sm(i)` 1280 -> ~790, `o.cm(i)` 1361 -> ~960, `fvk(1, 2, x=3)` in a tier-2 loop 5526 -> ~4530. Suite geomean 0.998 against main (closures_decorators -2.0%, context_managers -2.3%).
A `functools.partial` with more than one stored positional, or with stored keywords, was called through its native `__call__` from the slow step: the builtin lane's leaf probes, `dispatch_call`, and a nested call of the function. The `CALL` arms now send it down their Python path, where `core_partial_call` binds the stored positionals, the call's, and the stored keywords by name (as CPython's `partial_vectorcall` does) and runs the function as an inline activation, or a warm pure leaf in place. Anything that binder declines, errors included, still takes the native `__call__`. A pure leaf's keyword binding now has a pointer form, `leaf_bind_keywords`: each parameter points into the positionals, the keyword values, or the function's defaults, with no reference taken. `f(*args, **kwargs)` of a pure leaf with a non-empty mapping (the forwarding wrapper called with keywords) and a partial over a pure leaf evaluate through it in place. `core_call_ex` classifies its callee once instead of probing the generator and class paths for every function. `SlotStorage::leading_interned` reads a `__slots__` layout's leading values by key identity (its keys are interned), which the partial's field reads take on every call. Per call (instructions retired): `partial(f3, 1, 2)(3)` 4511 -> ~1510 (CPython 914), `partial(fkw, c=5)(i)` 7222 -> ~1910 (CPython 2134), the forwarding wrapper called with a keyword 4187 -> ~3400. Suite geomean 0.994 against main (closures_decorators -5.8%, context_managers -2.8%, xml_etree -1.9%).
Tier 2's dynamic call helper treated a keyword call of a Python function as free: the comment held that tier 1 paid the same generic binder. Tier 1 now binds those calls inline (by the site's permutation, by name, or as a frameless leaf), so a native loop that makes one per iteration is a thin driver around interpreter round trips, each paying the activation shell, the guard revalidation and the pin traffic. Such calls now count toward the call-density budget that retires the loop to tier 1, as positional generic calls already do. Keyword calls of native callees stay uncharged. Per call (instructions retired, microops loop): `fvk(1, 2, x=3)` 4530 -> ~2630 (it ran 5526 on main), `fkw(i, c=4)` 1226 -> ~905. The suite is unchanged (geomean 0.9995 against the previous commit, within noise).
A `CALL_KW` site's `CallPyKwNames` permutation (the keyword values' slots, computed once per site) refused callees with `*args` and closures, so `wrapper(x=1)` against a decorator's `def wrapper(*args, **kwargs)` and `f(1, 2, key=3)` against `def f(*args, key=None)` bound by name on every call and switched through the general inline call. The permutation now covers surplus positionals (they go to the `*args` tuple, which `kw_names_fill_locals` builds) and closures whose code has no cell variables of its own (their cells are the function's, bound as the lean cells), and the frame JIT's direct keyword call takes both, so these calls bind by moving their operands and run as direct calls. The full handler's keyword fast path and tier 2's site binding now run a closure's frame with its cells (`call_python_bound`), which they could not before because no closure ever got a permutation. Per call (instructions retired): `fvk(1, 2, x=3)` ~2630 -> ~2270 (5526 on main), the forwarding wrapper called with a keyword ~3360 -> ~3130 (4187 on main). Suite unchanged (geomean 1.0008 against the previous commit, within noise; 0.994 against main).
A native caller's call of a Python function (an `lru_cache` miss, a class's `__init__` from `instantiate`, a `key=` function that isn't a frameless leaf) copied its arguments into a fresh vector for `call_python_owned`, which only resized it for the exact positional shape. `call_python` now binds that shape itself into a pooled vector, so the call allocates nothing for its locals. A lean `__init__` with `*args` (`contextlib.suppress.__init__(self, *exceptions)`) built its surplus tuple from a collected vector; it now builds the tuple from the remaining arguments in place. Per call (instructions retired): a recursive `lru_cache(maxsize=0)` call pair 5604 -> ~5440, an `lru_cache` miss of a looping function 4864 -> ~4590. Suite geomean 0.998 against the previous commit (context_managers -0.7%), 0.992 against main.
This reverts eedc1bd ("perf: leave keyword-call-dense loops to tier 1"). Charging tier 2's keyword calls against the call-density budget retired whole native loops whose other calls tier 2 runs natively: the `call_overhead` bench fixture (positional, default, keyword and method calls in one loop) fell back to tier 1 and ran 2.2 to 2.6 times slower than main (3390 -> 8200 instructions per iteration). A keyword call costs tier 1 less only when it is nearly all the loop does, so the budget isn't the place to judge it.
# Conflicts: # crates/weavepy-vm/src/gen_fast.rs # crates/weavepy-vm/src/lib.rs
# Conflicts: # crates/weavepy-vm/src/frame_jit.rs
# Conflicts: # crates/weavepy-vm/src/frame_jit.rs
# Conflicts: # crates/weavepy-vm/src/frame_jit.rs # crates/weavepy-vm/src/lib.rs
# Conflicts: # crates/weavepy-vm/src/frame_jit.rs # crates/weavepy-vm/src/lib.rs
…vation-stack # Conflicts: # crates/weavepy-vm/src/lib.rs
# Conflicts: # crates/weavepy-vm/src/lib.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #93 (and through it #86); merge those first.
RFC 0078, stage 2 (revised in this PR's first commit from the measured direct-call costs): a call between compiled functions runs its callee lazily, as tier 2 already runs its native calls.
Why
A direct call bound its callee into an
InlineActand registered it with the core loop at once: the caller entered on the pending list, the callee pushed on the switch and made its running activation, all undone at the return. The core loop needs that view only when the callee leaves its native code. Callgrind put the direct call at about 520 instructions in its helpers, against about 100 for CPython's whole call and return, with no step above 20; the registration and its undoing were about 90 of them.What
State::lazy), and its caller stays the switch's running activation.frame_jit::register_lazydoes the registration the call skipped, its lazy callers first. It is due:Interpreter::settle_lazyin the pending-list enters, the shell pushes, the flushes and the stack walkers. Python code a helper runs (a property, an__eq__, a finalizer) so sees the callee in its place, which the light activations of perf: bind every direct call through one path #86's first attempt didn't guarantee.Measurements
Instructions per direct call: about 88 fewer (416 left in the helpers). Whole suite against #93's build:
go0.960,tomllib_loads0.982,closures_decorators0.986,sudoku0.992, geomean 0.997; none worse than 1.004.Validation
-D warningsis clean.tests/regrtest/test_light_direct_calls.pygains the cases where a helper's Python code walks the stack from a lazily running callee: a property, an__eq__, a finalizer, a raising property caught in the callee, a nested call through a property, a generator. Their frame chains match CPython's.test_sys,test_traceback,test_frame,test_inspect,test_warnings,test_sys_settrace,test_generators,test_coroutines,test_exceptions,test_contextlib,test_functools,test_monitoring,test_pdbandtest_bdbpass with no unexpected results.What's left
The rest of the direct call is spread thinly too: 19 instructions building the callee's
Stateper call, 20 in the entry helper's prologue and epilogue, about 30 of admission checks the site repeats per call, and about 80 of reference counts and moves. A persistent state per warm slot and the admission gates folded into one stamp are the next steps.