diff --git a/doc/coro-performance-baseline.md b/doc/coro-performance-baseline.md index 745e2ae02a..e26972398a 100644 --- a/doc/coro-performance-baseline.md +++ b/doc/coro-performance-baseline.md @@ -1085,3 +1085,71 @@ no `validReadyQueue` under normal handoff, distribution, or drain. Its active cost is now dominated by channel registration/cleanup, fixed source and fleet dispatch, frame allocation/free, and BDWGC marking/locking. Those are the next performance gates before broadening capability coverage. + +### Fused task envelope and physical M/P checkpoint + +The next allocation checkpoint uses merge `2a3eb6884` as its exact parent. A +spawned logical G previously owned three independently released native ranges: +the 264-byte scheduler G, a 160-byte runtime G/M/P/local sidecar, and its LLVM +coroutine-frame allocation. The first two have identical lifetimes and root +requirements, but simply concatenating them was not sufficient. The local +BDWGC `GC_size` boundary rounded the old pair to 272 + 160 = 432 bytes, while +both a 424-byte and a 416-byte combined object occupied the 448-byte size +class. Five-run gates observed the expected RSS regression and rejected both +intermediate layouts. + +The retained design removes physical M/P state from the logical sidecar. A +`coroRuntimeContext` now contains only the runtime G and its `LocalContext`; +the independently allocated native/executor placeholder keeps the full +`runtimeContext { logical core, M, P }`. Immediately around one physical +`llvm.coro.resume`, the logical G borrows the executor placeholder's actual M/P +and becomes `M.curg`; leave restores the exact placeholder and detaches the G. +Synchronous same-G C-to-Go reentry borrows the already-installed attachment. +Suspended and runnable logical Gs therefore retain no M/P, matching the +scheduler model instead of carrying one synthetic M/P pair per goroutine. + +The scheduler G remains at allocation offset zero, so the compiler-facing +spawn ABI is unchanged. The native64 104-byte logical runtime context follows +it in one 368-byte scanned/root allocation, and a compile-time offset equality +fixes the G-at-base invariant. Spawn clears that envelope once and retirement releases +it once with the exact backend size. The ordinary dynamic lifecycle therefore +drops from three allocator objects to two without a pool, retained cache, +target callback, or GC-specific pointer recovery. Native BDWGC reports the +368-byte request as an exact 368-byte object; WASI malloc and tinygogc consume +the same target-neutral layout. + +Five process-start `GOMAXPROCS=1` runs of the final candidate gave the following +peak-RSS medians. The Go column is the same five-run same-source build used by +the bounded-audit checkpoint. + +| parked Gs | Go gc RSS | task-envelope RSS | Go incremental / G | candidate incremental / G | +| ---: | ---: | ---: | ---: | ---: | +| 0 | 3,522,560 | 7,749,632 | - | - | +| 1,000 | 6,471,680 | 9,486,336 | 2,949 B | 1,737 B | +| 5,000 | 17,596,416 | 17,235,968 | 2,815 B | 1,897 B | +| 10,000 | 31,539,200 | 29,409,280 | 2,802 B | 2,166 B | + +At 10,000 parked goroutines the candidate is 2,129,920 bytes (6.75%) below Go +in total RSS and its incremental resident cost is about 22.7% lower. It is also +360,448 bytes below Go at 5,000, while the higher fixed runtime still loses at +1,000; the observed total-footprint crossover is therefore between 1,000 and +5,000 live Gs. + +Five final 10,000-G runs interleaved with the exact parent measured 30,375,936 +versus 29,442,048 bytes peak RSS (-933,888, -3.07%), 6.294 versus 6.277 billion +retired instructions (-0.28%), and 156.354 versus 156.989 ms workload time +(+0.41%, within the loaded-host range). Seven interleaved short-workload runs +showed compute +0.8%, spawn -1.3%, and unbuffered handoff -0.2%; their ranges +overlap, so only absence of a displaced common-path regression is claimed. +The stripped executable grew by 992 bytes (+0.02%); `__TEXT`, `__DATA_CONST`, +and `__DATA` segment reservations are unchanged. + +The full runtime suite, all 20 native-fleet E2Es (including foreign reentry, +blocking compensation, locked G/M replacement, cross-route channel/select, +quota changes, and shutdown), the three linked defer/panic/channel-spawn E2Es, +JS/WASM, WASI malloc, WASI tinygogc, Linux ARM/RISC-V, and Cortex-M baremetal +target builds passed. Both WASI profiles also executed successfully under +Wasmtime. These gates freeze two architectural requirements for later work: +logical G storage may not regain permanent M/P fields, and task/frame pooling +must demonstrate a benefit beyond this allocation fusion without retaining an +unbounded embedded-target cache. diff --git a/doc/llvm-coro-runtime-design.md b/doc/llvm-coro-runtime-design.md index d16b9301cd..f0918daab1 100644 --- a/doc/llvm-coro-runtime-design.md +++ b/doc/llvm-coro-runtime-design.md @@ -939,6 +939,16 @@ Runtime 可选配置每 G 的 `maxFrameDepth/maxFrameBytes`，用于资源受限 Native 上 M 是 pthread，P 数量通常受 GOMAXPROCS 控制。JS/WASM、单线程 WASI 和 baremetal 初期折叠为一个 M、一个 P、多个 G。 +当前runtime实现已把语言runtime sidecar按同一ownership拆开：可迁移逻辑G只保留 +`g + LocalContext`，不永久内嵌`m/p`；native/executor placeholder才拥有完整 +`{logical core, m, p}`。每次物理`llvm.coro.resume`前，逻辑G临时借用当前executor的 +真实M/P并成为`M.curg`，leave恢复exact placeholder并把G重新置为detached；同G同步 +C→Go reentry只借用已经安装的关系。native64上动态scheduler G与104-byte语言 +sidecar合并成368-byte scanned/root task envelope，G仍位于offset 0，故compiler +spawn ABI不变；wasm32/其他target按其pointer-size布局同一结构。 +该checkpoint通过native多executor/LockOSThread/replacement、WASM/WASI/tinygogc和 +baremetal门；后续不得为了方便又退化成per-G伪M/P。 + 这里的结构只表达ownership，不冻结字段排列。V2等待由稳定G内嵌的`ParkState`、直接park frame拥有的`WaitSetRecord`和source-owned `OperationRecord/ParkLink`共同表示；旧`parkGeneration + wakePending`只属于legacy单等待迁移层，不能继续作为channel、select、timer或I/O的新契约。 #### Execution-domain handoff diff --git a/internal/build/coro_native_e2e_helpers_test.go b/internal/build/coro_native_e2e_helpers_test.go index ecfd166ad3..199cac1851 100644 --- a/internal/build/coro_native_e2e_helpers_test.go +++ b/internal/build/coro_native_e2e_helpers_test.go @@ -29,6 +29,7 @@ import ( func coroNativeTaskContextRuntimeSources() []string { root := filepath.Join("..", "..", "runtime", "internal", "runtime") return []string{ + filepath.Join(root, "coro_task_allocation.go"), filepath.Join(root, "coro_task_context.go"), filepath.Join(root, "runtime_context.go"), filepath.Join(root, "runtime2.go"), diff --git a/runtime/coro_task_allocation_source_test.go b/runtime/coro_task_allocation_source_test.go new file mode 100644 index 0000000000..d0bd6a8efd --- /dev/null +++ b/runtime/coro_task_allocation_source_test.go @@ -0,0 +1,103 @@ +//go:build !llgo + +/* + * Copyright (c) 2026 The XGo Authors (xgo.dev). All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package runtime + +import ( + "strings" + "testing" +) + +func TestCoroSpawnFusesTaskAndRuntimeContextAllocation(t *testing.T) { + layout := readRuntimePollFile(t, "internal/runtime/coro_task_allocation.go") + for _, required := range []string{ + "type coroTaskAllocation struct", + "task coro.G", + "context coroRuntimeContext", + "coroTaskAllocationTaskOffset = unsafe.Offsetof(coroTaskAllocation{}.task)", + "var _ [coroTaskAllocationTaskOffset]byte = [0]byte{}", + } { + if !strings.Contains(layout, required) { + t.Errorf("combined task allocation lacks layout gate %q", required) + } + } + + spawn := readRuntimePollFile(t, "internal/runtime/coro_spawn.go") + for _, required := range []string{ + "coroalloc.AllocTask(allocationSize)", + "coro.Zero(raw, allocationSize)", + "coroBindTaskAllocationRuntimeContext(child, parent)", + "coroalloc.FreeTask(raw, allocationSize)", + } { + if !strings.Contains(spawn, required) { + t.Errorf("spawn path lacks combined allocation marker %q", required) + } + } + if strings.Contains(spawn, "coroalloc.AllocTask(taskSize)") || + strings.Contains(spawn, "coroBindRuntimeContext(child, parent, false)") { + t.Fatal("spawn path retained a separately allocated runtime sidecar") + } + + context := readRuntimePollFile(t, "internal/runtime/coro_task_context.go") + for _, required := range []string{ + "if ctx.g.coroEmbedded {", + "ctx == &(*coroTaskAllocation)(unsafe.Pointer(task)).context", + "if !embedded {", + "FreeRoot(unsafe.Pointer(ctx))", + } { + if !strings.Contains(context, required) { + t.Errorf("runtime-context release lacks shared-root gate %q", required) + } + } +} + +func TestCoroLogicalContextBorrowsPhysicalMPOnlyWhileRunning(t *testing.T) { + contextLayout := readRuntimePollFile(t, "internal/runtime/runtime_context.go") + for _, required := range []string{ + "type coroRuntimeContext struct", + "type runtimeContext struct", + "coroRuntimeContext\n\tm m\n\tp p", + "var _ [runtimeContextCoreOffset]byte = [0]byte{}", + } { + if !strings.Contains(contextLayout, required) { + t.Errorf("runtime context split lacks marker %q", required) + } + } + + lifecycle := readRuntimePollFile(t, "internal/runtime/coro_task_context.go") + for _, required := range []string{ + "mp := current.m", + "gp.m = mp", + "mp.curg = gp", + "mp.curg = previous", + "gp.m = nil", + } { + if !strings.Contains(lifecycle, required) { + t.Errorf("logical context lacks physical M borrow marker %q", required) + } + } + for _, forbidden := range []string{ + "gp, pp := &ctx.g, &ctx.p", + "setpstatus(pp, _Prunning)", + "setpstatus(pp, _Pidle)", + } { + if strings.Contains(lifecycle, forbidden) { + t.Errorf("logical context retained per-G physical P path %q", forbidden) + } + } +} diff --git a/runtime/internal/coro/spawn.go b/runtime/internal/coro/spawn.go index c23132a54e..8fad1a9899 100644 --- a/runtime/internal/coro/spawn.go +++ b/runtime/internal/coro/spawn.go @@ -28,9 +28,11 @@ const ( taskStorageReleased ) -// TaskStorageSize is the exact scanned/root allocation required for one -// independently scheduled G. The G begins at the allocation base so the C ABI -// can pass the returned address directly to a coroutine root factory. +// TaskStorageSize is the exact scheduler-owned prefix required for one +// independently scheduled G. The G begins at its runtime allocation base so +// the C ABI can pass the returned address directly to a coroutine root +// factory; a runtime adapter may retain target-specific task-local storage in +// the same scanned/root allocation after this prefix. func TaskStorageSize() uintptr { return unsafe.Sizeof(G{}) } diff --git a/runtime/internal/runtime/caller.go b/runtime/internal/runtime/caller.go index e159f50d08..ad039e8bec 100644 --- a/runtime/internal/runtime/caller.go +++ b/runtime/internal/runtime/caller.go @@ -403,7 +403,7 @@ func CoroPanicRecoverActive() bool { return false } task := (*coro.G)(gp.startarg) - ctx := (*runtimeContext)(coro.TaskLocal(task)) + ctx := (*coroRuntimeContext)(coro.TaskLocal(task)) return ctx == gp.context && validCoroRuntimeTaskContext(task, ctx) && coro.RecoverTraceActive(task) } diff --git a/runtime/internal/runtime/coro_current_task_route.go b/runtime/internal/runtime/coro_current_task_route.go index 58f5539ef3..f04cd1e53c 100644 --- a/runtime/internal/runtime/coro_current_task_route.go +++ b/runtime/internal/runtime/coro_current_task_route.go @@ -31,7 +31,7 @@ func coroCurrentTaskRouteV1() coro.RouteID { return 0 } task := (*coro.G)(gp.startarg) - if ctx := (*runtimeContext)(coro.TaskLocal(task)); ctx != gp.context || !validCoroRuntimeTaskContext(task, ctx) { + if ctx := (*coroRuntimeContext)(coro.TaskLocal(task)); ctx != gp.context || !validCoroRuntimeTaskContext(task, ctx) { return 0 } _, _, route, current := coro.CurrentExecutorDriver(task) diff --git a/runtime/internal/runtime/coro_panic_report_libc.go b/runtime/internal/runtime/coro_panic_report_libc.go index a92ac1e309..d11aa7a0ba 100644 --- a/runtime/internal/runtime/coro_panic_report_libc.go +++ b/runtime/internal/runtime/coro_panic_report_libc.go @@ -282,7 +282,7 @@ func coroTerminalWriteCString(text *c.Char) bool { // worker result records the prefix length explicitly; terminal reporting never // guesses whether an address is native by inspecting pointer bits and never // asks a worker thread to traverse Go scheduler state. -func coroTerminalWriteWorkerFaultFrames(ctx *runtimeContext) { +func coroTerminalWriteWorkerFaultFrames(ctx *coroRuntimeContext) { if ctx == nil { return } @@ -311,7 +311,7 @@ func coroTerminalWriteWorkerFaultFrames(ctx *runtimeContext) { } } -func coroTerminalWritePanicFrames(g *coro.G, ctx *runtimeContext) { +func coroTerminalWritePanicFrames(g *coro.G, ctx *coroRuntimeContext) { coroTerminalWriteString("\n\ngoroutine ") if ctx.g.goid == 0 { coroTerminalWriteUint(1) @@ -369,7 +369,7 @@ func __llgo_coro_program_report_panic_v1(gPointer unsafe.Pointer) { coroRuntimeAbort("invalid coroutine program panic report") } record, published := coro.LoadPanicRecord(g) - ctx := (*runtimeContext)(coro.TaskLocal(g)) + ctx := (*coroRuntimeContext)(coro.TaskLocal(g)) if !published || record.Status != coro.ExplicitStatusPanic || record.TypeWord == nil || !validCoroRuntimeContext(ctx) { coroRuntimeAbort("invalid coroutine program panic record") diff --git a/runtime/internal/runtime/coro_spawn.go b/runtime/internal/runtime/coro_spawn.go index 6c2d4e86b4..74c4f52502 100644 --- a/runtime/internal/runtime/coro_spawn.go +++ b/runtime/internal/runtime/coro_spawn.go @@ -33,27 +33,28 @@ func coroSpawnBeginV1(parentPointer unsafe.Pointer) (unsafe.Pointer, bool) { if !coro.CanBeginSpawn(parent) || !coroalloc.Ready() { return nil, false } - size := coro.TaskStorageSize() - raw := coroalloc.AllocTask(size) + taskSize := coro.TaskStorageSize() + allocationSize := uintptr(coroTaskAllocationSize) + raw := coroalloc.AllocTask(allocationSize) if raw == nil { return nil, false } - coro.Zero(raw, size) - child := (*coroG)(raw) - if !coro.BeginSpawn(parent, child, raw, size) { - coro.Zero(raw, size) - if !coroalloc.FreeTask(raw, size) { + coro.Zero(raw, allocationSize) + child, _, actualSize, allocationOK := coroTaskAllocationAt(raw) + if !allocationOK || actualSize != allocationSize || !coro.BeginSpawn(parent, child, raw, taskSize) { + coro.Zero(raw, allocationSize) + if !coroalloc.FreeTask(raw, allocationSize) { return nil, false } return nil, false } - if !coroBindRuntimeContext(child, parent, false) { + if !coroBindTaskAllocationRuntimeContext(child, parent) { rolled, rolledSize, ok := coro.RollbackSpawn(parent, child) - if !ok || rolled != raw || rolledSize != size { + if !ok || rolled != raw || rolledSize != taskSize { return nil, false } - coro.Zero(raw, size) - if !coroalloc.FreeTask(raw, size) { + coro.Zero(raw, allocationSize) + if !coroalloc.FreeTask(raw, allocationSize) { return nil, false } return nil, false @@ -90,12 +91,13 @@ func coroReleaseCompletedTask(g *coroG) bool { if !owned { return true } - raw, size, ok := coro.ReleaseTaskStorage(g) - if !ok { + raw, taskSize, ok := coro.ReleaseTaskStorage(g) + _, _, allocationSize, allocationOK := coroTaskAllocationAt(raw) + if !ok || !allocationOK || raw != unsafe.Pointer(g) || taskSize != coro.TaskStorageSize() { return false } - coro.Zero(raw, size) - return coroalloc.FreeTask(raw, size) + coro.Zero(raw, allocationSize) + return coroalloc.FreeTask(raw, allocationSize) } //export __llgo_coro_spawn_begin_v1 diff --git a/runtime/internal/runtime/coro_task_allocation.go b/runtime/internal/runtime/coro_task_allocation.go new file mode 100644 index 0000000000..d0589ae0a2 --- /dev/null +++ b/runtime/internal/runtime/coro_task_allocation.go @@ -0,0 +1,67 @@ +/* + * Copyright (c) 2026 The XGo Authors (xgo.dev). All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package runtime + +import ( + "unsafe" + + "github.com/goplus/llgo/runtime/internal/coro" +) + +// coroTaskAllocation is the runtime-owned physical envelope for one spawned +// logical G. task must remain the first field: compiler-generated coroutine +// factories receive the allocation base as their opaque G pointer, while the +// target-neutral scheduler owns only the fixed-size task prefix. The runtime +// context has the same lifetime and scanned/root requirement, so retaining it +// in the tail removes one allocator transaction without adding a pool, cache, +// target callback, or scheduler dependency on runtime types. +type coroTaskAllocation struct { + task coro.G + context coroRuntimeContext +} + +const ( + coroTaskAllocationTaskOffset = unsafe.Offsetof(coroTaskAllocation{}.task) + coroTaskAllocationContextOffset = unsafe.Offsetof(coroTaskAllocation{}.context) + coroTaskAllocationSize = unsafe.Sizeof(coroTaskAllocation{}) +) + +// Go preserves declaration order, but make the allocation-base ABI an actual +// compile-time equality rather than a convention checked for every spawn. +var _ [coroTaskAllocationTaskOffset]byte = [0]byte{} + +func coroTaskAllocationAt(raw unsafe.Pointer) (*coro.G, *coroRuntimeContext, uintptr, bool) { + if raw == nil || uintptr(raw)%unsafe.Alignof(coroTaskAllocation{}) != 0 { + return nil, nil, 0, false + } + allocation := (*coroTaskAllocation)(raw) + task := &allocation.task + context := &allocation.context + if unsafe.Pointer(task) != raw || + unsafe.Pointer(context) != unsafe.Add(raw, coroTaskAllocationContextOffset) { + return nil, nil, 0, false + } + return task, context, coroTaskAllocationSize, true +} + +func coroTaskAllocationContext(task *coro.G) (*coroRuntimeContext, bool) { + if task == nil { + return nil, false + } + actual, context, _, ok := coroTaskAllocationAt(unsafe.Pointer(task)) + return context, ok && actual == task +} diff --git a/runtime/internal/runtime/coro_task_context.go b/runtime/internal/runtime/coro_task_context.go index bcb508465a..303a7180d0 100644 --- a/runtime/internal/runtime/coro_task_context.go +++ b/runtime/internal/runtime/coro_task_context.go @@ -25,45 +25,106 @@ import ( "github.com/goplus/llgo/runtime/internal/coro" ) -// coroBindRuntimeContext creates the runtime sidecar which follows one -// stackless logical G across executor threads. The target-neutral scheduler -// retains only its opaque scanned pointer. -func coroBindRuntimeContext(task, parent *coro.G, main bool) bool { +func coroRuntimeContextParent(task, parent *coro.G) (*g, bool) { if task == nil || coro.TaskLocal(task) != nil { - return false + return nil, false } var parentG *g if parent != nil { - parentContext := (*runtimeContext)(coro.TaskLocal(parent)) + parentContext := (*coroRuntimeContext)(coro.TaskLocal(parent)) if !validCoroRuntimeTaskContext(parent, parentContext) { - return false + return nil, false } parentG = &parentContext.g + if parentG.m == nil || parentG.m.curg != parentG || parentG.m.p == nil || + parentG.m.p.m != parentG.m || readgstatus(parentG) != _Grunning || + readpstatus(parentG.m.p) != _Prunning { + return nil, false + } + } + return parentG, true +} + +func coroBindRuntimeContextAt(task *coro.G, parentG *g, ctx *coroRuntimeContext, embedded bool) bool { + if ctx == nil || ctx.g.context != nil || embedded != (parentG != nil) { + return false } - ctx := allocRuntimeContext() - gp := initRuntimeContext(ctx, parentG, _Grunnable) + gp := initCoroRuntimeContext(ctx, parentG, _Grunnable) gp.localContext = &ctx.local - gp.isMain = main + gp.isMain = !embedded + gp.coroEmbedded = embedded if coro.BindTaskLocal(task, unsafe.Pointer(ctx)) { gp.startarg = unsafe.Pointer(task) return true } - discardCoroRuntimeContext(ctx) + discardCoroRuntimeContext(ctx, !embedded) return false } -func validCoroRuntimeContext(ctx *runtimeContext) bool { - if ctx == nil || ctx.root == nil { +// coroBindRuntimeContext creates the independently rooted runtime sidecar for +// the static command G. Spawned tasks use coroBindTaskAllocationRuntimeContext +// so the sidecar shares their physical task allocation. +func coroBindRuntimeContext(task, parent *coro.G, main bool) bool { + if !main || parent != nil { + return false + } + parentG, ok := coroRuntimeContextParent(task, parent) + if !ok { return false } - gp, mp, pp := &ctx.g, &ctx.m, &ctx.p - return gp.context == ctx && gp.m == mp && mp.curg == gp && mp.p == pp && - pp.m == mp && gp.localContext == &ctx.local + size := unsafe.Sizeof(coroRuntimeContext{}) + raw := AllocRoot(size) + if raw == nil { + coroRuntimeAbort("failed to allocate coroutine runtime context") + return false + } + c.Memset(raw, 0, size) + ctx := (*coroRuntimeContext)(raw) + return coroBindRuntimeContextAt(task, parentG, ctx, false) } -func validCoroRuntimeTaskContext(task *coro.G, ctx *runtimeContext) bool { - return task != nil && validCoroRuntimeContext(ctx) && ctx.g.startfn == nil && - ctx.g.startarg == unsafe.Pointer(task) +// coroBindTaskAllocationRuntimeContext initializes the sidecar in the tail of +// a zeroed spawned-task envelope. The task allocation remains its sole +// physical owner and is released only after this logical context is detached. +func coroBindTaskAllocationRuntimeContext(task, parent *coro.G) bool { + if parent == nil { + return false + } + parentG, ok := coroRuntimeContextParent(task, parent) + if !ok { + return false + } + ctx, ok := coroTaskAllocationContext(task) + if !ok { + return false + } + return coroBindRuntimeContextAt(task, parentG, ctx, true) +} + +// validCoroRuntimeContext checks only state which follows the logical G. Its +// temporary physical M/P attachment is validated exactly once by enter/leave; +// parent spawn admission performs its own running-state check. +func validCoroRuntimeContext(ctx *coroRuntimeContext) bool { + if ctx == nil { + return false + } + gp := &ctx.g + return gp.context == ctx && gp.localContext == &ctx.local +} + +func validCoroRuntimeTaskContext(task *coro.G, ctx *coroRuntimeContext) bool { + if task == nil || !validCoroRuntimeContext(ctx) || ctx.g.startfn != nil || + ctx.g.startarg != unsafe.Pointer(task) { + return false + } + // This predicate runs around every physical coroutine resume. The task is + // already scheduler-valid and coroEmbedded selects the combined allocation + // representation. The command root is the only independently allocated + // logical-task context; every dynamic G must match the exact tail address. + if ctx.g.coroEmbedded { + return ctx == &(*coroTaskAllocation)(unsafe.Pointer(task)).context + } + return ctx.g.isMain } // coroEnterRuntimeContext installs task's runtime G only for the physical @@ -71,42 +132,57 @@ func validCoroRuntimeTaskContext(task *coro.G, ctx *runtimeContext) bool { // frame of the same logical G while its parent resume remains active below C; // that exact nested case borrows the existing install. func coroEnterRuntimeContext(task *coro.G) (coroRuntimeContextActivationV1, bool) { - ctx := (*runtimeContext)(coro.TaskLocal(task)) + ctx := (*coroRuntimeContext)(coro.TaskLocal(task)) if !validCoroRuntimeTaskContext(task, ctx) { return coroRuntimeContextActivationV1{}, false } - gp, pp := &ctx.g, &ctx.p + gp := &ctx.g current := getg() if current == gp { - if readgstatus(gp) != _Grunning || readpstatus(pp) != _Prunning { + if gp.m == nil || gp.m.curg != gp || gp.m.p == nil || gp.m.p.m != gp.m || + readgstatus(gp) != _Grunning || readpstatus(gp.m.p) != _Prunning { return coroRuntimeContextActivationV1{}, false } return coroRuntimeContextActivationV1{borrowed: true}, true } - if readgstatus(gp) != _Grunnable || readpstatus(pp) != _Pidle { + if current == nil || current.context == nil || current.startarg != nil || + gp.m != nil || readgstatus(gp) != _Grunnable || + current.m == nil || current.m.curg != current || current.m.p == nil || + current.m.p.m != current.m || readgstatus(current) != _Grunning || + readpstatus(current.m.p) != _Prunning { return coroRuntimeContextActivationV1{}, false } + mp := current.m casgstatus(gp, _Grunnable, _Grunning) - setpstatus(pp, _Prunning) + gp.m = mp + mp.curg = gp setg(gp) return coroRuntimeContextActivationV1{previous: unsafe.Pointer(current)}, true } func coroLeaveRuntimeContext(task *coro.G, activation coroRuntimeContextActivationV1) bool { - ctx := (*runtimeContext)(coro.TaskLocal(task)) + ctx := (*coroRuntimeContext)(coro.TaskLocal(task)) if !validCoroRuntimeTaskContext(task, ctx) { return false } - gp, pp := &ctx.g, &ctx.p - if getg() != gp || readgstatus(gp) != _Grunning || readpstatus(pp) != _Prunning { + gp := &ctx.g + if getg() != gp || gp.m == nil || gp.m.curg != gp || gp.m.p == nil || + gp.m.p.m != gp.m || readgstatus(gp) != _Grunning || readpstatus(gp.m.p) != _Prunning { return false } if activation.borrowed { return activation.previous == nil } + previous := (*g)(activation.previous) + mp := gp.m + if previous == nil || previous == gp || previous.context == nil || previous.startarg != nil || + previous.m != mp || readgstatus(previous) != _Grunning { + return false + } casgstatus(gp, _Grunning, _Grunnable) - setpstatus(pp, _Pidle) - setg((*g)(activation.previous)) + mp.curg = previous + gp.m = nil + setg(previous) return true } @@ -114,16 +190,17 @@ func coroLeaveRuntimeContext(task *coro.G, activation coroRuntimeContextActivati // has made the G terminal but before its scanned task allocation is cleared. func coroReleaseRuntimeContext(task *coro.G) bool { raw := coro.TaskLocal(task) - ctx := (*runtimeContext)(raw) + ctx := (*coroRuntimeContext)(raw) if !validCoroRuntimeTaskContext(task, ctx) { return false } - released, ok := coro.ReleaseTaskLocal(task) - if !ok || released != raw { + gp := &ctx.g + if gp.m != nil || readgstatus(gp) != _Grunnable { return false } - gp, mp, pp := &ctx.g, &ctx.m, &ctx.p - if readgstatus(gp) != _Grunnable || readpstatus(pp) != _Pidle { + embedded := ctx.g.coroEmbedded + released, ok := coro.ReleaseTaskLocal(task) + if !ok || released != raw { return false } if gp.localContext != nil { @@ -136,20 +213,17 @@ func coroReleaseRuntimeContext(task *coro.G) bool { } releasePanicPCStore(gp) gp.startarg = nil + gp.coroEmbedded = false casgstatus(gp, _Grunnable, _Gdead) - setpstatus(pp, _Pdead) - pp.m = nil - mp.p = nil - mp.curg = nil - gp.m = nil - root := ctx.root - ctx.root = nil + gp.context = nil releaseGAndCheckDeadlock() - FreeRoot(root) + if !embedded { + FreeRoot(unsafe.Pointer(ctx)) + } return true } -func discardCoroRuntimeContext(ctx *runtimeContext) { +func discardCoroRuntimeContext(ctx *coroRuntimeContext, freeContext bool) { if ctx == nil { return } @@ -158,10 +232,10 @@ func discardCoroRuntimeContext(ctx *runtimeContext) { ctx.g.localContext = nil } releasePanicPCStore(&ctx.g) - root := ctx.root - ctx.root = nil - if root != nil { - releaseG() - FreeRoot(root) + ctx.g.context = nil + ctx.g.coroEmbedded = false + releaseG() + if freeContext { + FreeRoot(unsafe.Pointer(ctx)) } } diff --git a/runtime/internal/runtime/coro_task_context_test_adapter.go b/runtime/internal/runtime/coro_task_context_test_adapter.go index bfb7e13b65..2cf0d96020 100644 --- a/runtime/internal/runtime/coro_task_context_test_adapter.go +++ b/runtime/internal/runtime/coro_task_context_test_adapter.go @@ -27,6 +27,10 @@ func coroBindRuntimeContext(task, parent *coro.G, main bool) bool { return task != nil } +func coroBindTaskAllocationRuntimeContext(task, parent *coro.G) bool { + return task != nil +} + func coroEnterRuntimeContext(task *coro.G) (coroRuntimeContextActivationV1, bool) { return coroRuntimeContextActivationV1{}, task != nil } diff --git a/runtime/internal/runtime/g_pthread.go b/runtime/internal/runtime/g_pthread.go index 6581917218..4cfcc34804 100644 --- a/runtime/internal/runtime/g_pthread.go +++ b/runtime/internal/runtime/g_pthread.go @@ -85,15 +85,21 @@ func destroyG(ptr c.Pointer) { if gp == nil { return } + if gp.startarg != nil { + // A managed logical G must restore the executor placeholder before + // pthread exit. Do not mutate or free either a standalone command + // context or an interior spawned-task context from the C destructor; + // physical-owner shutdown detects the missing resume/leave separately. + return + } if gp.panic_ != nil { c.Free(gp.panic_) gp.panic_ = nil } releasePanicPCStore(gp) ctx := gp.context - if ctx != nil && ctx.root != nil { - root := ctx.root - ctx.root = nil - FreeRoot(root) + if ctx != nil { + gp.context = nil + FreeRoot(unsafe.Pointer(ctx)) } } diff --git a/runtime/internal/runtime/proc.go b/runtime/internal/runtime/proc.go index b0ff7976e6..805010d393 100644 --- a/runtime/internal/runtime/proc.go +++ b/runtime/internal/runtime/proc.go @@ -31,7 +31,7 @@ func NewProc(fn goroutineFunc, arg unsafe.Pointer, stackSize uintptr) { ctx := gp.context releaseG() FreeRoot(arg) - FreeRoot(ctx.root) + FreeRoot(unsafe.Pointer(ctx)) panic("runtime: failed to create new OS thread") } } @@ -97,7 +97,7 @@ func mexit(mp *m) { gp := mp.curg pp := mp.p ctx := gp.context - root := ctx.root + root := unsafe.Pointer(ctx) releaseGAndCheckDeadlock() casgstatus(gp, _Grunning, _Gdead) @@ -107,13 +107,11 @@ func mexit(mp *m) { mp.p = nil mp.curg = nil gp.m = nil + gp.context = nil releasePanicPCStore(gp) setg(nil) - if root != nil { - ctx.root = nil - FreeRoot(root) - } + FreeRoot(root) } // GMPForTesting reports the current runtime ownership graph. It is kept @@ -125,10 +123,11 @@ func GMPForTesting() (goid, parentGoid uint64, mid int64, pid int32, gstatus, ps } mp := gp.m pp := mp.p - ctx := gp.context + core := gp.context + ctx := (*runtimeContext)(unsafe.Pointer(core)) return gp.goid, gp.parentGoid, mp.id, pp.id, readgstatus(gp), readpstatus(pp), - mp.curg == gp && pp.m == mp && ctx != nil && - &ctx.g == gp && &ctx.m == mp && &ctx.p == pp + mp.curg == gp && pp.m == mp && core != nil && + &ctx.coroRuntimeContext == core && &ctx.g == gp && &ctx.m == mp && &ctx.p == pp } // GStateForTesting reports the packed scheduler state without changing it. diff --git a/runtime/internal/runtime/runtime2.go b/runtime/internal/runtime/runtime2.go index 8828eb69ca..285a10f8b3 100644 --- a/runtime/internal/runtime/runtime2.go +++ b/runtime/internal/runtime/runtime2.go @@ -76,7 +76,7 @@ type g struct { // phase-overlays this otherwise idle pointer with the scheduler G identity. startarg unsafe.Pointer - context *runtimeContext + context *coroRuntimeContext // localContext follows the logical G. Native pthread execution installs a // stack-owned context at its outer Go entry; the stackless scheduler points // this field at the context embedded in the task's runtime sidecar. @@ -85,6 +85,10 @@ type g struct { goexit bool isMain bool paniconfault bool + // coroEmbedded occupies existing tail padding. It distinguishes a runtime + // context stored in a spawned task envelope from independently rooted + // pthread/bootstrap contexts without retaining a redundant root pointer. + coroEmbedded bool } // m represents the host execution resource running Go code. The platform diff --git a/runtime/internal/runtime/runtime_context.go b/runtime/internal/runtime/runtime_context.go index 9df77f0c29..8816667cc1 100644 --- a/runtime/internal/runtime/runtime_context.go +++ b/runtime/internal/runtime/runtime_context.go @@ -22,23 +22,30 @@ import ( c "github.com/goplus/llgo/runtime/internal/clite" ) -// runtimeContext keeps the G, M, and P for one native or stackless logical -// execution context in one allocation. Keeping this ownership core independent -// of pthread creation lets every scheduler backend reuse it directly. -type runtimeContext struct { +// coroRuntimeContext is the state which follows one stackless logical G. M and +// P are physical executor resources and are deliberately absent: the logical G +// borrows its current executor's M/P only for a physical coroutine resume. +type coroRuntimeContext struct { g g - m m - p p - // local is used only by a stackless logical G. Keeping it inside the rooted - // sidecar gives package-local blocks the same lifetime as that G. + // local gives package-local blocks the same lifetime as the logical G. local LocalContext +} - // root is non-nil for contexts passed through a host-thread API or retained - // by a stackless task. Such contexts remain visible to the collector until - // their owning M or logical G exits. - root unsafe.Pointer +// runtimeContext owns a native or executor-placeholder G together with its +// physical M/P. Keeping coroRuntimeContext as the first field gives g.context +// one common prefix while independently allocated native contexts retain their +// allocation-base identity. +type runtimeContext struct { + coroRuntimeContext + m m + p p } +const runtimeContextCoreOffset = unsafe.Offsetof(runtimeContext{}.coroRuntimeContext) + +// Native teardown frees g.context as the original independent-root base. +var _ [runtimeContextCoreOffset]byte = [0]byte{} + var sched struct { goidgen uint64 midgen int64 @@ -113,9 +120,7 @@ func allocRuntimeContext() *runtimeContext { return nil } c.Memset(root, 0, size) - ctx := (*runtimeContext)(root) - ctx.root = root - return ctx + return (*runtimeContext)(root) } func initRuntimeContext(ctx *runtimeContext, callergp *g, status uint32) *g { @@ -124,23 +129,33 @@ func initRuntimeContext(ctx *runtimeContext, callergp *g, status uint32) *g { return gp } +func initCoroRuntimeContext(ctx *coroRuntimeContext, callergp *g, status uint32) *g { + gp := initCoroRuntimeContextUntracked(ctx, callergp, status) + retainG() + return gp +} + +func initCoroRuntimeContextUntracked(ctx *coroRuntimeContext, callergp *g, status uint32) *g { + gp := &ctx.g + gp.atomicstatus = status + gp.goid = nextGoid(gp) + if callergp != nil { + gp.parentGoid = callergp.goid + } + gp.context = ctx + return gp +} + // initRuntimeContextUntracked initializes the executor-thread placeholder // installed in pthread TLS. It is a physical context used only while no // stackless logical G is active, so it must not participate in the logical-G // count or the main-Goexit deadlock decision. func initRuntimeContextUntracked(ctx *runtimeContext, callergp *g, status uint32) *g { - gp := &ctx.g + gp := initCoroRuntimeContextUntracked(&ctx.coroRuntimeContext, callergp, status) mp := &ctx.m pp := &ctx.p gp.m = mp - gp.atomicstatus = status - gp.goid = nextGoid(gp) - if callergp != nil { - gp.parentGoid = callergp.goid - } - gp.context = ctx - mp.curg = gp mp.p = pp mp.id = nextMid(mp)