Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions doc/coro-performance-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -1085,3 +1085,71 @@ no `validReadyQueue` under normal handoff, distribution, or drain. Its active
cost is now dominated by channel registration/cleanup, fixed source and fleet
dispatch, frame allocation/free, and BDWGC marking/locking. Those are the next
performance gates before broadening capability coverage.

### Fused task envelope and physical M/P checkpoint

The next allocation checkpoint uses merge `2a3eb6884` as its exact parent. A
spawned logical G previously owned three independently released native ranges:
the 264-byte scheduler G, a 160-byte runtime G/M/P/local sidecar, and its LLVM
coroutine-frame allocation. The first two have identical lifetimes and root
requirements, but simply concatenating them was not sufficient. The local
BDWGC `GC_size` boundary rounded the old pair to 272 + 160 = 432 bytes, while
both a 424-byte and a 416-byte combined object occupied the 448-byte size
class. Five-run gates observed the expected RSS regression and rejected both
intermediate layouts.

The retained design removes physical M/P state from the logical sidecar. A
`coroRuntimeContext` now contains only the runtime G and its `LocalContext`;
the independently allocated native/executor placeholder keeps the full
`runtimeContext { logical core, M, P }`. Immediately around one physical
`llvm.coro.resume`, the logical G borrows the executor placeholder's actual M/P
and becomes `M.curg`; leave restores the exact placeholder and detaches the G.
Synchronous same-G C-to-Go reentry borrows the already-installed attachment.
Suspended and runnable logical Gs therefore retain no M/P, matching the
scheduler model instead of carrying one synthetic M/P pair per goroutine.

The scheduler G remains at allocation offset zero, so the compiler-facing
spawn ABI is unchanged. The native64 104-byte logical runtime context follows
it in one 368-byte scanned/root allocation, and a compile-time offset equality
fixes the G-at-base invariant. Spawn clears that envelope once and retirement releases
it once with the exact backend size. The ordinary dynamic lifecycle therefore
drops from three allocator objects to two without a pool, retained cache,
target callback, or GC-specific pointer recovery. Native BDWGC reports the
368-byte request as an exact 368-byte object; WASI malloc and tinygogc consume
the same target-neutral layout.

Five process-start `GOMAXPROCS=1` runs of the final candidate gave the following
peak-RSS medians. The Go column is the same five-run same-source build used by
the bounded-audit checkpoint.

| parked Gs | Go gc RSS | task-envelope RSS | Go incremental / G | candidate incremental / G |
| ---: | ---: | ---: | ---: | ---: |
| 0 | 3,522,560 | 7,749,632 | - | - |
| 1,000 | 6,471,680 | 9,486,336 | 2,949 B | 1,737 B |
| 5,000 | 17,596,416 | 17,235,968 | 2,815 B | 1,897 B |
| 10,000 | 31,539,200 | 29,409,280 | 2,802 B | 2,166 B |

At 10,000 parked goroutines the candidate is 2,129,920 bytes (6.75%) below Go
in total RSS and its incremental resident cost is about 22.7% lower. It is also
360,448 bytes below Go at 5,000, while the higher fixed runtime still loses at
1,000; the observed total-footprint crossover is therefore between 1,000 and
5,000 live Gs.

Five final 10,000-G runs interleaved with the exact parent measured 30,375,936
versus 29,442,048 bytes peak RSS (-933,888, -3.07%), 6.294 versus 6.277 billion
retired instructions (-0.28%), and 156.354 versus 156.989 ms workload time
(+0.41%, within the loaded-host range). Seven interleaved short-workload runs
showed compute +0.8%, spawn -1.3%, and unbuffered handoff -0.2%; their ranges
overlap, so only absence of a displaced common-path regression is claimed.
The stripped executable grew by 992 bytes (+0.02%); `__TEXT`, `__DATA_CONST`,
and `__DATA` segment reservations are unchanged.

The full runtime suite, all 20 native-fleet E2Es (including foreign reentry,
blocking compensation, locked G/M replacement, cross-route channel/select,
quota changes, and shutdown), the three linked defer/panic/channel-spawn E2Es,
JS/WASM, WASI malloc, WASI tinygogc, Linux ARM/RISC-V, and Cortex-M baremetal
target builds passed. Both WASI profiles also executed successfully under
Wasmtime. These gates freeze two architectural requirements for later work:
logical G storage may not regain permanent M/P fields, and task/frame pooling
must demonstrate a benefit beyond this allocation fusion without retaining an
unbounded embedded-target cache.
10 changes: 10 additions & 0 deletions doc/llvm-coro-runtime-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -939,6 +939,16 @@ Runtime 可选配置每 G 的 `maxFrameDepth/maxFrameBytes`,用于资源受限

Native 上 M 是 pthread,P 数量通常受 GOMAXPROCS 控制。JS/WASM、单线程 WASI 和 baremetal 初期折叠为一个 M、一个 P、多个 G。

当前runtime实现已把语言runtime sidecar按同一ownership拆开:可迁移逻辑G只保留
`g + LocalContext`,不永久内嵌`m/p`;native/executor placeholder才拥有完整
`{logical core, m, p}`。每次物理`llvm.coro.resume`前,逻辑G临时借用当前executor的
真实M/P并成为`M.curg`,leave恢复exact placeholder并把G重新置为detached;同G同步
C→Go reentry只借用已经安装的关系。native64上动态scheduler G与104-byte语言
sidecar合并成368-byte scanned/root task envelope,G仍位于offset 0,故compiler
spawn ABI不变;wasm32/其他target按其pointer-size布局同一结构。
该checkpoint通过native多executor/LockOSThread/replacement、WASM/WASI/tinygogc和
baremetal门;后续不得为了方便又退化成per-G伪M/P。

这里的结构只表达ownership,不冻结字段排列。V2等待由稳定G内嵌的`ParkState`、直接park frame拥有的`WaitSetRecord`和source-owned `OperationRecord/ParkLink`共同表示;旧`parkGeneration + wakePending`只属于legacy单等待迁移层,不能继续作为channel、select、timer或I/O的新契约。

#### Execution-domain handoff
Expand Down
1 change: 1 addition & 0 deletions internal/build/coro_native_e2e_helpers_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import (
func coroNativeTaskContextRuntimeSources() []string {
root := filepath.Join("..", "..", "runtime", "internal", "runtime")
return []string{
filepath.Join(root, "coro_task_allocation.go"),
filepath.Join(root, "coro_task_context.go"),
filepath.Join(root, "runtime_context.go"),
filepath.Join(root, "runtime2.go"),
Expand Down
103 changes: 103 additions & 0 deletions runtime/coro_task_allocation_source_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
//go:build !llgo

/*
* Copyright (c) 2026 The XGo Authors (xgo.dev). All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package runtime

import (
"strings"
"testing"
)

func TestCoroSpawnFusesTaskAndRuntimeContextAllocation(t *testing.T) {
layout := readRuntimePollFile(t, "internal/runtime/coro_task_allocation.go")
for _, required := range []string{
"type coroTaskAllocation struct",
"task coro.G",
"context coroRuntimeContext",
"coroTaskAllocationTaskOffset = unsafe.Offsetof(coroTaskAllocation{}.task)",
"var _ [coroTaskAllocationTaskOffset]byte = [0]byte{}",
} {
if !strings.Contains(layout, required) {
t.Errorf("combined task allocation lacks layout gate %q", required)
}
}

spawn := readRuntimePollFile(t, "internal/runtime/coro_spawn.go")
for _, required := range []string{
"coroalloc.AllocTask(allocationSize)",
"coro.Zero(raw, allocationSize)",
"coroBindTaskAllocationRuntimeContext(child, parent)",
"coroalloc.FreeTask(raw, allocationSize)",
} {
if !strings.Contains(spawn, required) {
t.Errorf("spawn path lacks combined allocation marker %q", required)
}
}
if strings.Contains(spawn, "coroalloc.AllocTask(taskSize)") ||
strings.Contains(spawn, "coroBindRuntimeContext(child, parent, false)") {
t.Fatal("spawn path retained a separately allocated runtime sidecar")
}

context := readRuntimePollFile(t, "internal/runtime/coro_task_context.go")
for _, required := range []string{
"if ctx.g.coroEmbedded {",
"ctx == &(*coroTaskAllocation)(unsafe.Pointer(task)).context",
"if !embedded {",
"FreeRoot(unsafe.Pointer(ctx))",
} {
if !strings.Contains(context, required) {
t.Errorf("runtime-context release lacks shared-root gate %q", required)
}
}
}

func TestCoroLogicalContextBorrowsPhysicalMPOnlyWhileRunning(t *testing.T) {
contextLayout := readRuntimePollFile(t, "internal/runtime/runtime_context.go")
for _, required := range []string{
"type coroRuntimeContext struct",
"type runtimeContext struct",
"coroRuntimeContext\n\tm m\n\tp p",
"var _ [runtimeContextCoreOffset]byte = [0]byte{}",
} {
if !strings.Contains(contextLayout, required) {
t.Errorf("runtime context split lacks marker %q", required)
}
}

lifecycle := readRuntimePollFile(t, "internal/runtime/coro_task_context.go")
for _, required := range []string{
"mp := current.m",
"gp.m = mp",
"mp.curg = gp",
"mp.curg = previous",
"gp.m = nil",
} {
if !strings.Contains(lifecycle, required) {
t.Errorf("logical context lacks physical M borrow marker %q", required)
}
}
for _, forbidden := range []string{
"gp, pp := &ctx.g, &ctx.p",
"setpstatus(pp, _Prunning)",
"setpstatus(pp, _Pidle)",
} {
if strings.Contains(lifecycle, forbidden) {
t.Errorf("logical context retained per-G physical P path %q", forbidden)
}
}
}
8 changes: 5 additions & 3 deletions runtime/internal/coro/spawn.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,11 @@ const (
taskStorageReleased
)

// TaskStorageSize is the exact scanned/root allocation required for one
// independently scheduled G. The G begins at the allocation base so the C ABI
// can pass the returned address directly to a coroutine root factory.
// TaskStorageSize is the exact scheduler-owned prefix required for one
// independently scheduled G. The G begins at its runtime allocation base so
// the C ABI can pass the returned address directly to a coroutine root
// factory; a runtime adapter may retain target-specific task-local storage in
// the same scanned/root allocation after this prefix.
func TaskStorageSize() uintptr {
return unsafe.Sizeof(G{})
}
Expand Down
2 changes: 1 addition & 1 deletion runtime/internal/runtime/caller.go
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,7 @@ func CoroPanicRecoverActive() bool {
return false
}
task := (*coro.G)(gp.startarg)
ctx := (*runtimeContext)(coro.TaskLocal(task))
ctx := (*coroRuntimeContext)(coro.TaskLocal(task))
return ctx == gp.context && validCoroRuntimeTaskContext(task, ctx) && coro.RecoverTraceActive(task)
}

Expand Down
2 changes: 1 addition & 1 deletion runtime/internal/runtime/coro_current_task_route.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ func coroCurrentTaskRouteV1() coro.RouteID {
return 0
}
task := (*coro.G)(gp.startarg)
if ctx := (*runtimeContext)(coro.TaskLocal(task)); ctx != gp.context || !validCoroRuntimeTaskContext(task, ctx) {
if ctx := (*coroRuntimeContext)(coro.TaskLocal(task)); ctx != gp.context || !validCoroRuntimeTaskContext(task, ctx) {
return 0
}
_, _, route, current := coro.CurrentExecutorDriver(task)
Expand Down
6 changes: 3 additions & 3 deletions runtime/internal/runtime/coro_panic_report_libc.go
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@ func coroTerminalWriteCString(text *c.Char) bool {
// worker result records the prefix length explicitly; terminal reporting never
// guesses whether an address is native by inspecting pointer bits and never
// asks a worker thread to traverse Go scheduler state.
func coroTerminalWriteWorkerFaultFrames(ctx *runtimeContext) {
func coroTerminalWriteWorkerFaultFrames(ctx *coroRuntimeContext) {
if ctx == nil {
return
}
Expand Down Expand Up @@ -311,7 +311,7 @@ func coroTerminalWriteWorkerFaultFrames(ctx *runtimeContext) {
}
}

func coroTerminalWritePanicFrames(g *coro.G, ctx *runtimeContext) {
func coroTerminalWritePanicFrames(g *coro.G, ctx *coroRuntimeContext) {
coroTerminalWriteString("\n\ngoroutine ")
if ctx.g.goid == 0 {
coroTerminalWriteUint(1)
Expand Down Expand Up @@ -369,7 +369,7 @@ func __llgo_coro_program_report_panic_v1(gPointer unsafe.Pointer) {
coroRuntimeAbort("invalid coroutine program panic report")
}
record, published := coro.LoadPanicRecord(g)
ctx := (*runtimeContext)(coro.TaskLocal(g))
ctx := (*coroRuntimeContext)(coro.TaskLocal(g))
if !published || record.Status != coro.ExplicitStatusPanic ||
record.TypeWord == nil || !validCoroRuntimeContext(ctx) {
coroRuntimeAbort("invalid coroutine program panic record")
Expand Down
32 changes: 17 additions & 15 deletions runtime/internal/runtime/coro_spawn.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,27 +33,28 @@ func coroSpawnBeginV1(parentPointer unsafe.Pointer) (unsafe.Pointer, bool) {
if !coro.CanBeginSpawn(parent) || !coroalloc.Ready() {
return nil, false
}
size := coro.TaskStorageSize()
raw := coroalloc.AllocTask(size)
taskSize := coro.TaskStorageSize()
allocationSize := uintptr(coroTaskAllocationSize)
raw := coroalloc.AllocTask(allocationSize)
if raw == nil {
return nil, false
}
coro.Zero(raw, size)
child := (*coroG)(raw)
if !coro.BeginSpawn(parent, child, raw, size) {
coro.Zero(raw, size)
if !coroalloc.FreeTask(raw, size) {
coro.Zero(raw, allocationSize)
child, _, actualSize, allocationOK := coroTaskAllocationAt(raw)
if !allocationOK || actualSize != allocationSize || !coro.BeginSpawn(parent, child, raw, taskSize) {
coro.Zero(raw, allocationSize)
if !coroalloc.FreeTask(raw, allocationSize) {
return nil, false
}
return nil, false
}
if !coroBindRuntimeContext(child, parent, false) {
if !coroBindTaskAllocationRuntimeContext(child, parent) {
rolled, rolledSize, ok := coro.RollbackSpawn(parent, child)
if !ok || rolled != raw || rolledSize != size {
if !ok || rolled != raw || rolledSize != taskSize {
return nil, false
}
coro.Zero(raw, size)
if !coroalloc.FreeTask(raw, size) {
coro.Zero(raw, allocationSize)
if !coroalloc.FreeTask(raw, allocationSize) {
return nil, false
}
return nil, false
Expand Down Expand Up @@ -90,12 +91,13 @@ func coroReleaseCompletedTask(g *coroG) bool {
if !owned {
return true
}
raw, size, ok := coro.ReleaseTaskStorage(g)
if !ok {
raw, taskSize, ok := coro.ReleaseTaskStorage(g)
_, _, allocationSize, allocationOK := coroTaskAllocationAt(raw)
if !ok || !allocationOK || raw != unsafe.Pointer(g) || taskSize != coro.TaskStorageSize() {
return false
}
coro.Zero(raw, size)
return coroalloc.FreeTask(raw, size)
coro.Zero(raw, allocationSize)
return coroalloc.FreeTask(raw, allocationSize)
}

//export __llgo_coro_spawn_begin_v1
Expand Down
67 changes: 67 additions & 0 deletions runtime/internal/runtime/coro_task_allocation.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
/*
* Copyright (c) 2026 The XGo Authors (xgo.dev). All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package runtime

import (
"unsafe"

"github.com/goplus/llgo/runtime/internal/coro"
)

// coroTaskAllocation is the runtime-owned physical envelope for one spawned
// logical G. task must remain the first field: compiler-generated coroutine
// factories receive the allocation base as their opaque G pointer, while the
// target-neutral scheduler owns only the fixed-size task prefix. The runtime
// context has the same lifetime and scanned/root requirement, so retaining it
// in the tail removes one allocator transaction without adding a pool, cache,
// target callback, or scheduler dependency on runtime types.
type coroTaskAllocation struct {
task coro.G
context coroRuntimeContext
}

const (
coroTaskAllocationTaskOffset = unsafe.Offsetof(coroTaskAllocation{}.task)
coroTaskAllocationContextOffset = unsafe.Offsetof(coroTaskAllocation{}.context)
coroTaskAllocationSize = unsafe.Sizeof(coroTaskAllocation{})
)

// Go preserves declaration order, but make the allocation-base ABI an actual
// compile-time equality rather than a convention checked for every spawn.
var _ [coroTaskAllocationTaskOffset]byte = [0]byte{}

func coroTaskAllocationAt(raw unsafe.Pointer) (*coro.G, *coroRuntimeContext, uintptr, bool) {
if raw == nil || uintptr(raw)%unsafe.Alignof(coroTaskAllocation{}) != 0 {
return nil, nil, 0, false
}
allocation := (*coroTaskAllocation)(raw)
task := &allocation.task
context := &allocation.context
if unsafe.Pointer(task) != raw ||
unsafe.Pointer(context) != unsafe.Add(raw, coroTaskAllocationContextOffset) {
return nil, nil, 0, false
}
return task, context, coroTaskAllocationSize, true
}

func coroTaskAllocationContext(task *coro.G) (*coroRuntimeContext, bool) {
if task == nil {
return nil, false
}
actual, context, _, ok := coroTaskAllocationAt(unsafe.Pointer(task))
return context, ok && actual == task
}
Loading