Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions doc/coro-performance-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -978,3 +978,43 @@ Mach-O text/data sizes were unchanged. These noisy-host measurements establish
the removed complexity and absence of a storage regression, not a stable
throughput promise. The remaining large-G cost is linear channel close/wakeup,
task/frame allocation and runtime-context initialization.

### Lazy panic-snapshot checkpoint

The next memory checkpoint uses `c773ea52e` as its exact parent. Every runtime
G previously embedded a 544-byte `panicPCStore`, including its 64-PC array,
even though an ordinary parked logical G never panics. Replacing that payload
with one pointer removes 536 bytes from every 64-bit runtime context. The
bounded store is allocated and explicitly released only if a managed panic
snapshot is actually requested. Stackless language panics continue to use
their scheduler-owned logical frame trace and therefore do not allocate it.

The legacy native fault callback cannot allocate in signal context. It now
uses an already-owned G store when one exists and otherwise publishes into one
static emergency store. This preserves the pre-existing process-global,
best-effort concurrent-fault scope of the legacy fault source. A normal panic
replaces that emergency attachment with an owned store. Source gates require
the signal callback to use the allocation-free entry, while complete coroutine
profiles retain their signal-stack-free compiler outcome path.

Peak RSS from one bounded, process-start `GOMAXPROCS=1` run was:

| parked Gs | Go gc RSS | `c773ea52e` RSS | lazy snapshot RSS | lazy incremental / G |
| ---: | ---: | ---: | ---: | ---: |
| 0 | 3,555,328 | 7,716,864 | 7,733,248 | - |
| 1,000 | 6,471,680 | 10,665,984 | 9,748,480 | 2,015 B |
| 5,000 | 17,580,032 | 25,526,272 | 20,987,904 | 2,651 B |
| 10,000 | 31,539,200 | 40,779,776 | 32,751,616 | 2,502 B |

At 10,000 parked Gs the candidate removes 8,028,160 bytes (19.7%) from the
parent and is only 1,212,416 bytes (3.8%) above Go in total RSS. Its slope from
idle is about 10.6% below Go's 2,798 bytes/G, so the stackless implementation
now demonstrates a per-goroutine memory advantage even though its larger fixed
runtime still narrowly loses the total-footprint comparison at this scale.

The stripped executable shrank by 416 bytes; Mach-O `__TEXT`, `__DATA_CONST`,
and `__DATA` segment reservations were unchanged. Runtime tests, native/wasm
panic and recover IR tests, and the linked native ExplicitStatus panic E2E
completed. The legacy PCLN signal integration command remains blocked before
its fault subtest by the same unrelated `pclntab_external.go` uintptr-retention
audit on both this candidate and the exact parent.
2 changes: 1 addition & 1 deletion runtime/internal/lib/runtime/fault_unwind_llgo.go
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ func onFault(pc, fp uintptr, sig int32) {
n += m
}
faultN = int32(n)
rtdebug.StoreFaultPCs(faultPCs[:n])
rtdebug.StoreSignalFaultPCs(faultPCs[:n])
}
// Capture done: re-arm the recursion guard before this fault turns
// into an ordinary (recoverable) panic.
Expand Down
55 changes: 40 additions & 15 deletions runtime/internal/runtime/caller.go
Original file line number Diff line number Diff line change
Expand Up @@ -227,8 +227,8 @@ func Callers(skip int, pcs []uintptr) int {
func SavePanicCallerFrames() {
// A fault handler stores the fault-site snapshot right before it
// panics; the regular capture here must not overwrite it.
p := panicPCStoreForG()
if p.armed != 0 {
p := loadPanicPCStore(getg())
if p != nil && p.armed != 0 {
p.armed = 0
return
}
Expand All @@ -238,22 +238,29 @@ func SavePanicCallerFrames() {
}
}

func panicPCStoreForG() *panicPCStore {
return &getg().panicPCs
}

// StorePanicPCs replaces the goroutine's panic snapshot (a new panic
// supersedes the previous one) and resets the recover marks.
func StorePanicPCs(pcs []uintptr) {
storePanicPCs(pcs, 0)
}

// StoreFaultPCs is StorePanicPCs for fault handlers: the imminent
// panic's own capture is suppressed so the fault-site chain survives.
// StoreFaultPCs is StorePanicPCs for a managed fault path. The imminent
// panic's own capture is suppressed so the fault-site chain survives. Legacy
// asynchronous signal callbacks must use StoreSignalFaultPCs instead.
func StoreFaultPCs(pcs []uintptr) {
storePanicPCs(pcs, 1)
}

// StoreSignalFaultPCs publishes a legacy signal-handler snapshot without
// allocating. A G which already owns a lazy store reuses it; otherwise it uses
// the same process-global best-effort scope as the legacy fault source.
func StoreSignalFaultPCs(pcs []uintptr) {
p := signalSafePanicPCStore(getg())
if p != nil {
storePanicPCsInto(p, pcs, 1)
}
}

// StoreCoroWorkerFaultPCs joins the two bounded native identities returned by
// the C worker with the resumed coroutine's exact active frame and the current
// G's compiler-maintained logical callers. The worker never receives this
Expand Down Expand Up @@ -314,12 +321,23 @@ func StoreCoroWorkerFaultPCs(task *coro.G, faultPC, targetPC uintptr) bool {
}
}
StoreFaultPCs(pcs[:n])
panicPCStoreForG().native = int32(native)
store := loadPanicPCStore(getg())
if store == nil {
return false
}
store.native = int32(native)
return true
}

func storePanicPCs(pcs []uintptr, armed int32) {
p := panicPCStoreForG()
p := ensurePanicPCStore(getg())
if p == nil {
return
}
storePanicPCsInto(p, pcs, armed)
}

func storePanicPCsInto(p *panicPCStore, pcs []uintptr, armed int32) {
n := len(pcs)
if n > len(p.pcs) {
n = len(p.pcs)
Expand All @@ -336,13 +354,14 @@ func storePanicPCs(pcs []uintptr, armed int32) {
// PanicPCsAreFault reports whether the stored snapshot came from a
// hardware-fault context (captured without the program-text bound).
func PanicPCsAreFault() bool {
return panicPCStoreForG().fault != 0
p := loadPanicPCStore(getg())
return p != nil && p.fault != 0
}

// PanicPCs returns the goroutine's captured panic pcs (nil when none).
func PanicPCs() []uintptr {
p := panicPCStoreForG()
if p.n == 0 {
p := loadPanicPCStore(getg())
if p == nil || p.n == 0 {
return nil
}
return p.pcs[:p.n]
Expand All @@ -352,14 +371,20 @@ func PanicPCs() []uintptr {
// time; the snapshot stays spliceable exactly while one of them is live on
// the physical chain (the deferred function has not returned yet).
func MarkPanicRecoverFPs(fp1, fp2 uintptr) {
p := panicPCStoreForG()
p := loadPanicPCStore(getg())
if p == nil {
return
}
p.recFP1 = fp1
p.recFP2 = fp2
}

// PanicRecoverFPs returns the recover-time frame marks.
func PanicRecoverFPs() (uintptr, uintptr) {
p := panicPCStoreForG()
p := loadPanicPCStore(getg())
if p == nil {
return 0, 0
}
return p.recFP1, p.recFP2
}

Expand Down
12 changes: 8 additions & 4 deletions runtime/internal/runtime/coro_panic_report_libc.go
Original file line number Diff line number Diff line change
Expand Up @@ -283,15 +283,19 @@ func coroTerminalWriteCString(text *c.Char) bool {
// guesses whether an address is native by inspecting pointer bits and never
// asks a worker thread to traverse Go scheduler state.
func coroTerminalWriteWorkerFaultFrames(ctx *runtimeContext) {
if ctx == nil || ctx.g.panicPCs.fault == 0 || ctx.g.panicPCs.native <= 0 {
if ctx == nil {
return
}
count := int(ctx.g.panicPCs.native)
if count > len(ctx.g.panicPCs.pcs) || count > coroTerminalPanicFrameLimit {
store := loadPanicPCStore(&ctx.g)
if store == nil || store.fault == 0 || store.native <= 0 {
return
}
count := int(store.native)
if count > len(store.pcs) || count > coroTerminalPanicFrameLimit {
coroRuntimeAbort("invalid coroutine C fault trace prefix")
}
for index := 0; index < count; index++ {
pc := ctx.g.panicPCs.pcs[index]
pc := store.pcs[index]
if pc <= 1 {
coroRuntimeAbort("invalid coroutine C fault trace pc")
}
Expand Down
2 changes: 2 additions & 0 deletions runtime/internal/runtime/coro_task_context.go
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ func coroReleaseRuntimeContext(task *coro.G) bool {
c.Free(gp.panic_)
gp.panic_ = nil
}
releasePanicPCStore(gp)
gp.startarg = nil
casgstatus(gp, _Grunnable, _Gdead)
setpstatus(pp, _Pdead)
Expand All @@ -156,6 +157,7 @@ func discardCoroRuntimeContext(ctx *runtimeContext) {
releaseLocalBlocks(ctx.g.localContext)
ctx.g.localContext = nil
}
releasePanicPCStore(&ctx.g)
root := ctx.root
ctx.root = nil
if root != nil {
Expand Down
1 change: 1 addition & 0 deletions runtime/internal/runtime/g_pthread.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ func destroyG(ptr c.Pointer) {
c.Free(gp.panic_)
gp.panic_ = nil
}
releasePanicPCStore(gp)
ctx := gp.context
if ctx != nil && ctx.root != nil {
root := ctx.root
Expand Down
1 change: 1 addition & 0 deletions runtime/internal/runtime/proc.go
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ func mexit(mp *m) {
mp.p = nil
mp.curg = nil
gp.m = nil
releasePanicPCStore(gp)

setg(nil)
if root != nil {
Expand Down
10 changes: 5 additions & 5 deletions runtime/internal/runtime/runtime2.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,10 @@ const (
_Pdead = 4
)

// panicPCStore is part of a logical G's state. Keep the storage shape next to
// g rather than in caller.go so minimal scheduler/runtime islands that do not
// link the public stack-inspection implementation still retain the complete G
// layout.
// panicPCStore is the bounded traceback snapshot of a panic. G keeps only a
// lazy pointer because ordinary logical Gs never need this 64-PC payload. The
// legacy signal handler has a process-global, allocation-free emergency store;
// its fault snapshot was already process-global and deliberately best-effort.
type panicPCStore struct {
n int32
armed int32
Expand All @@ -63,7 +63,7 @@ type panicPCStore struct {
type g struct {
defer_ *Defer
panic_ unsafe.Pointer
panicPCs panicPCStore
panicPCs *panicPCStore
m *m

atomicstatus uint32
Expand Down
55 changes: 55 additions & 0 deletions runtime/internal/runtime/runtime_context.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,61 @@ var sched struct {
gstate uint64
}

// signalPanicPCStore is the allocation-free fallback used only when a legacy
// SA_SIGINFO callback faults before its current G has ever needed a panic
// snapshot. That callback's source snapshot is already process-global and
// documents concurrent faults as a lost race on a doomed process.
var signalPanicPCStore panicPCStore

func loadPanicPCStore(gp *g) *panicPCStore {
if gp == nil {
return nil
}
return gp.panicPCs
}

func ensurePanicPCStore(gp *g) *panicPCStore {
if gp == nil {
return nil
}
if gp.panicPCs != nil && gp.panicPCs != &signalPanicPCStore {
return gp.panicPCs
}
size := unsafe.Sizeof(panicPCStore{})
raw := AllocRoot(size)
if raw == nil {
coroRuntimeAbort("failed to allocate panic PC store")
return nil
}
c.Memset(raw, 0, size)
gp.panicPCs = (*panicPCStore)(raw)
return gp.panicPCs
}

func signalSafePanicPCStore(gp *g) *panicPCStore {
if gp == nil {
return nil
}
if gp.panicPCs == nil {
gp.panicPCs = &signalPanicPCStore
}
return gp.panicPCs
}

func releasePanicPCStore(gp *g) {
if gp == nil || gp.panicPCs == nil {
return
}
store := gp.panicPCs
gp.panicPCs = nil
if store == &signalPanicPCStore {
return
}
size := unsafe.Sizeof(panicPCStore{})
c.Memset(unsafe.Pointer(store), 0, size)
FreeRoot(unsafe.Pointer(store))
}

func allocRuntimeContext() *runtimeContext {
size := unsafe.Sizeof(runtimeContext{})
root := AllocRoot(size)
Expand Down
40 changes: 40 additions & 0 deletions runtime/signal_coro_source_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,46 @@ func TestRuntimeSignalCoroAdapterIsSignalSafeAndEventDriven(t *testing.T) {
}
}

func TestRuntimePanicPCSnapshotIsLazyAndSignalSafe(t *testing.T) {
runtime2 := readRuntimePollFile(t, "internal/runtime/runtime2.go")
if !strings.Contains(runtime2, "panicPCs *panicPCStore") ||
strings.Contains(runtime2, "panicPCs panicPCStore") {
t.Fatal("runtime G does not keep the bounded panic PC payload out of line")
}

context := readRuntimePollFile(t, "internal/runtime/runtime_context.go")
for _, required := range []string{
"var signalPanicPCStore panicPCStore",
"func ensurePanicPCStore(gp *g) *panicPCStore",
"raw := AllocRoot(size)",
"func signalSafePanicPCStore(gp *g) *panicPCStore",
"gp.panicPCs = &signalPanicPCStore",
"if store == &signalPanicPCStore",
} {
if !strings.Contains(context, required) {
t.Errorf("lazy panic PC ownership lacks %q", required)
}
}

caller := readRuntimePollFile(t, "internal/runtime/caller.go")
for _, required := range []string{
"func StoreSignalFaultPCs(pcs []uintptr)",
"p := signalSafePanicPCStore(getg())",
"storePanicPCsInto(p, pcs, 1)",
"p := ensurePanicPCStore(getg())",
} {
if !strings.Contains(caller, required) {
t.Errorf("panic PC capture lacks %q", required)
}
}

faultLegacy := readRuntimePollFile(t, runtimeFaultLegacySource)
if !strings.Contains(faultLegacy, "rtdebug.StoreSignalFaultPCs(faultPCs[:n])") ||
strings.Contains(faultLegacy, "rtdebug.StoreFaultPCs(faultPCs[:n])") {
t.Fatal("legacy signal callback can reach the allocating panic PC path")
}
}

func TestRuntimeSignalSourceSelectionUsesCompleteNativeCapability(t *testing.T) {
native := []string{"llgo", "llgo_coro", "llgo_coro_native_pipe", "llgo_coro_native_timer"}
tests := []struct {
Expand Down