Skip to content
7 changes: 7 additions & 0 deletions .agents/skills/mirai-agent-api/references/methods.md
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,13 @@ Reading a capture:
per second, the mean interval between two and its deviation (how even the pace is) and the
latency from the start of a frame to its image. With generation there are twice as many images as
frames.
- **The GPU of a frame.** A frame's `gpu_ms` and passes are those of its own submissions, at the
time they ran (`start_ms` since the frame began on the CPU). A frame whose submissions did not all
come back says `gpu_measured:false`, and the GPU numbers of the summary and of `profile.compare`
leave it out (`gpu_frames`).
- **Memory.** `profile.capture` with `memory:true` samples it at the end of each frame, as the part
`memory` of the frame, and the summary gives `memory`: the World's tables and storages (its peak
over the capture and its rise), the process and the GPU. Leave it off to measure cost.
- **Own time.** `self_ms` excludes the zones inside: a function that waits for its tasks has them
as children (`job`, `range`); what the workers run appears with `thread:"workers"`.
- **Conditions.** `notes` warns about what invalidates a reading: engine at `-o:minimal`,
Expand Down
24 changes: 18 additions & 6 deletions .agents/skills/mirai-core/references/host.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,12 +190,24 @@ entities or persistent data.
always covered, like the editor bars, is not drawn.

**Profiling (`profile.odin`).** `profile_start(app, frames)` records the following frames: zones of
the main thread, one per loop function, the tasks of each worker and what the renderer
measures. The first `profile_start` reserves the memory: recording does
not allocate, without recording a zone costs one branch and what does not fit is counted in `lost`.
Whoever reads the zones as they happen (a profiling plugin) calls `profile_request` in every frame
and reads them with its own cursor; without a recording or a request, the device writes no
per-pass timestamps. The editor exposes it with
the main thread, one per loop function, the tasks of each worker, what the renderer measures and the
GPU passes of every submission of the frame (a capture or a thumbnail besides the window). Those
come back `rhi.MAX_FRAMES_IN_FLIGHT` submissions later, by `rhi` frame number, and go to the frame
that submitted them; a frame is measured once all of its submissions came back, and a recording runs
a few frames past its last one, keeping no zones, until they do. The first `profile_start` reserves
the memory: recording does not allocate, without recording a zone costs one branch and what does not
fit is counted in `lost`. Whoever reads the zones as they happen (a profiling plugin) calls
`profile_request` in every frame and reads them with its own cursor; without a recording or a
request, the device writes no per-pass timestamps. A recording that asks for memory
(`profile_start(app, frames, memory)`, off by default) samples it at the end of each frame, inside
the frame as its part `memory`: what the tables and storages of the World that ran last hold
(`core.world_memory`, the game loop's World, else the editor's), the private and resident memory of
the process (`platform.process_memory`) and the device memory. `core.world_memory` adds up the
capacity of the arrays, maps and slices of the tables and storages and of what their elements and
map values hold, reading the containers that `world_info_index` found once in their types, so a
sample costs one read per container header, those of every slot of a pool included; the bytes of
strings are not in it (a string may be a literal or a view), nor what game code allocates outside
its storages. The editor exposes it with
[`profile.*`](../../mirai-agent-api/references/methods.md#profiling).

**Loads (`load_profile.odin`).** `app.load_profile` keeps the last load of each kind (scene, Play,
Expand Down
6 changes: 4 additions & 2 deletions .agents/skills/mirai-plugins/references/profiling.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,10 @@ View > Profiler panel. **Record** records frames into a
agent see the same captures, and the panel shows the last finished one.

- It shows the capture summary, a chart with one bar per frame colored by what it waited on (CPU,
GPU or pacing), the timeline of the chosen frame (main thread zones, one row per worker and the
GPU passes) and the most expensive zones per frame.
GPU or pacing), and a tab per track for the chosen frame: CPU (main thread zones, one row per
worker, the most expensive zones per frame), GPU (its passes where they ran, the most expensive
passes) and Memory (the World, the process and the GPU over the frames). Its recordings sample
memory; the tabs keep the height of the tallest, so the panel does not move under the pointer.
- It draws with [`editor.drawing`](../../mirai-editor/references/extensions.md#a-panel); what it
computes from a capture it computes when it starts showing it.

Expand Down
3 changes: 3 additions & 0 deletions .agents/skills/mirai-rendering/references/frame-graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,9 @@ or interleave its own passes.
of `tick_now` (QPC on Windows, `CLOCK_MONOTONIC_RAW` on Linux). Without the extension, Vulkan
places the last timestamp of the frame at the moment it collects it: it keeps the durations but
arrives late;
- `renderer_gpu_submissions`: a copy of the timings of each submission since
`renderer_gpu_clear`, with the `rhi` frame it was (`rhi.frame_timings`), for whoever accounts for
every submission of a host frame; `Stats.timings` is only the last one the GPU finished;
- visible, culled and occluded instances, probes and estimated rays per frame;
- what was discarded by capacity ([UI limits](ui.md#capacities));
- materials, textures, their memory, reduced textures and compression
Expand Down
36 changes: 36 additions & 0 deletions core/world_data.odin
Original file line number Diff line number Diff line change
Expand Up @@ -302,3 +302,39 @@ world_detach :: proc(info:^World_Info,world,shared:rawptr) {
}
}
}

// The bytes the tables and storages of a World hold behind their pointers:
// the capacity of the arrays, maps and slices of each table and storage of
// the place World, and of its entities, as the ops world_info_index compiled
// from their types read them. It reads capacities, not the allocator, so
// what game code allocates outside its storages is not in it. 0 for a
// World_Info without schema.
world_memory :: proc(info:^World_Info,data:rawptr) -> (bytes:int) {
if info==nil || data==nil || info.schema==nil {return}
ws:=info.schema
bytes=memory_read(ws.memory,ws.entities,entities_of(info,data))
for &t,i in info.tables {
if t.place!=.World {continue}
table:=table_of(data,&t)
bytes+=table_bytes(table,t.info.size)
held:=ws.tables[i].held
if held[0]==held[1] {continue}
for row in 0..<table.values.len {bytes+=memory_read(ws.memory,held,rawptr(uintptr(table.values.data)+uintptr(row*t.info.size)))}
}
for &s,i in info.storages {
if s.place==.World && !s.by_world {bytes+=memory_read(ws.memory,ws.storages[i].held,storage_of(data,&s))}
}
return
}

@(private="file")
table_bytes :: proc(t:^Raw_Table,size:int) -> (bytes:int) {
bytes=(t.values.cap+t.added.cap+t.edited.cap)*size
bytes+=cap(t.entities)*size_of(Entity)+cap(t.stamps)*size_of(u64)+cap(t.page_stamps)*size_of(u64)
bytes+=cap(t.off)*size_of(bool)+cap(t.turns)*size_of(u64)+cap(t.pending)*size_of(Pending)
bytes+=cap(t.edits)*size_of(Edit)+cap(t.gone.items)*size_of(Entity)+cap(t.owners)*size_of(Row_Owner)
bytes+=len(t.locks)*size_of(Lock)+cap(t.rows)*size_of([]i32)
for page in t.rows {bytes+=len(page)*size_of(i32)}
when CHECK_WRITES {bytes+=cap(t.settled.values)}
return
}
147 changes: 147 additions & 0 deletions core/world_memory_test.odin
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
package core

import "base:runtime"
import "core:testing"

@(private) Mem_Slot :: struct {points:[dynamic][3]f32,tag:u32,bytes:[]u8}
@(private) Mem_Node :: struct {kids:[dynamic]Mem_Node,weight:u32}
@(private) Mem_Pools :: struct {
slots:[dynamic]Mem_Slot,
fixed:[4]Mem_Slot,
names:map[u32]u32,
lists:map[u32][dynamic]u32,
tree:[dynamic]Mem_Node,
flat:[8]u32,
}

@(private)
@(test_world)
Mem_World :: struct {
using world:Test_World,
values:Table(u32),
pools:Mem_Pools,
}

// What a World's tables and storages hold grows with their rows and items,
// by the capacity they reserved.
@(test)
the_memory_of_a_world_follows_what_its_tables_and_storages_hold :: proc(t:^testing.T) {
info,described:=test_info(t,Mem_World)
if !described {return}
defer test_info_destroy(&info)
w:Mem_World
if !testing.expect(t,world_init(&info,.World,&w,context.allocator)) {return}
defer world_destroy(&info,.World,&w)
empty:=world_memory(&info,&w)
for i in 0..<100 {
e:=create(&w.entities)
add(&w.values,e,u32(i))
}
settle(&info,&w,nil)
rows:=world_memory(&info,&w)
testing.expectf(t,rows>=empty+100*size_of(Entity),"100 rows and their entities add to it: %d -> %d",empty,rows)
append(&w.pools.slots,Mem_Slot{})
for i in 0..<1000 {append(&w.pools.slots[0].points,[3]f32{f32(i),0,0})}
items:=world_memory(&info,&w)
testing.expect_value(t,items-rows,cap(w.pools.slots)*size_of(Mem_Slot)+cap(w.pools.slots[0].points)*size_of([3]f32))
}

// The ops world_info_index compiles read what reading the types at every
// step reads, in arrays of values that hold arrays, fixed or not, in the
// values of a map and in a type that holds itself.
@(test)
the_compiled_memory_ops_read_what_the_types_say :: proc(t:^testing.T) {
info,described:=test_info(t,Mem_World)
if !described {return}
defer test_info_destroy(&info)
w:Mem_World
if !testing.expect(t,world_init(&info,.World,&w,context.allocator)) {return}
defer world_destroy(&info,.World,&w)
defer { // world_destroy frees arrays and slices, not maps
delete(w.pools.names)
for _,list in w.pools.lists {delete(list)}
delete(w.pools.lists)
}
for i in 0..<5 {
append(&w.pools.slots,Mem_Slot{tag=u32(i)})
for _ in 0..<i*7 {append(&w.pools.slots[i].points,[3]f32{})}
w.pools.slots[i].bytes=make([]u8,i*13)
}
for &slot,i in w.pools.fixed {slot.bytes=make([]u8,i+1)}
for i in 0..<20 {w.pools.names[u32(i)]=u32(i)}
for i in 0..<6 {w.pools.lists[u32(i)]=make([dynamic]u32,i,i*5+1)}
append(&w.pools.tree,Mem_Node{})
for i in 0..<3 {
append(&w.pools.tree[0].kids,Mem_Node{weight=u32(i)})
for _ in 0..=i {append(&w.pools.tree[0].kids[i].kids,Mem_Node{})}
}
for _ in 0..<3 {create(&w.entities)}
ws:=info.schema
if !testing.expect(t,ws!=nil) {return}
for &s,k in info.storages {
if s.place!=.World {continue}
data:=rawptr(uintptr(&w)+s.offset)
testing.expect_value(t,memory_read(ws.memory,ws.storages[k].held,data),reflected(data,s.info))
}
entities:=rawptr(uintptr(&w)+info.entities)
testing.expect_value(t,memory_read(ws.memory,ws.entities,entities),reflected(entities,type_info_of(Entities)))
testing.expect(t,reflected(&w.pools,type_info_of(Mem_Pools))>5*size_of(Mem_Slot),"the pools hold something")
}

@(private="file")
reflected :: proc(data:rawptr,info:^runtime.Type_Info) -> int {bytes,_:=reflected_bytes(data,info);return bytes}

// What a value holds behind its pointers, read from its type at every step:
// what the compiled ops of world_memory must give.
@(private="file")
reflected_bytes :: proc(data:rawptr,info:^runtime.Type_Info) -> (bytes,elements:int) {
base:=runtime.type_info_base(info)
#partial switch v in base.variant {
case runtime.Type_Info_Struct:
for i in 0..<int(v.field_count) {
b,e:=reflected_bytes(rawptr(uintptr(data)+v.offsets[i]),v.types[i])
bytes+=b;elements+=e
}
case runtime.Type_Info_Array:
if value_flat(v.elem) {return}
for i in 0..<v.count {
b,e:=reflected_bytes(rawptr(uintptr(data)+uintptr(i*v.elem_size)),v.elem)
bytes+=b;elements+=e
}
case runtime.Type_Info_Enumerated_Array:
if value_flat(v.elem) {return}
for i in 0..<v.count {
b,e:=reflected_bytes(rawptr(uintptr(data)+uintptr(i*v.elem_size)),v.elem)
bytes+=b;elements+=e
}
case runtime.Type_Info_Dynamic_Array:
a:=cast(^runtime.Raw_Dynamic_Array)data
bytes=a.cap*v.elem_size
if a.len==0 || value_flat(v.elem) {return}
for i in 0..<a.len {
b,e:=reflected_bytes(rawptr(uintptr(a.data)+uintptr(i*v.elem_size)),v.elem)
bytes+=b;elements+=e+1
}
case runtime.Type_Info_Slice:
a:=cast(^runtime.Raw_Slice)data
bytes=a.len*v.elem_size
if a.len==0 || value_flat(v.elem) {return}
for i in 0..<a.len {
b,e:=reflected_bytes(rawptr(uintptr(a.data)+uintptr(i*v.elem_size)),v.elem)
bytes+=b;elements+=e+1
}
case runtime.Type_Info_Map:
m:=(cast(^runtime.Raw_Map)data)^
if m.data==0 {return}
capacity:=uintptr(runtime.map_cap(m))
bytes=int(runtime.map_total_allocation_size(capacity,v.map_info))
keys,values,hashes,_,_:=runtime.map_kvh_data_dynamic(m,v.map_info)
for i in 0..<capacity {
if !runtime.map_hash_is_valid(hashes[i]) {continue}
kb,ke:=reflected_bytes(rawptr(runtime.map_cell_index_dynamic(keys,v.map_info.ks,i)),v.key)
vb,ve:=reflected_bytes(rawptr(runtime.map_cell_index_dynamic(values,v.map_info.vs,i)),v.value)
bytes+=kb+vb;elements+=ke+ve+1
}
}
return
}
113 changes: 112 additions & 1 deletion core/world_schema.odin
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ World_Schema :: struct {
storages:[]Value_Schema, // by storage: the persist ones; the others have TYPE_NONE
records:[]Value_Schema, // by record type of the World_Info
table_hashes:[]Table_Hash, // sorted by hash: world_table_index_of_hash
memory:[]Memory_Op, // world_memory reads these instead of the types (memory_compile)
entities:[2]i32, // the ops of the Entities of a World
allocator:runtime.Allocator,
}

Expand All @@ -29,6 +31,7 @@ Value_Schema :: struct {
hash:u64, // of its layout: a value stored with another migrates by field names
layout:serial.Layout, // the bytes of a value that are saved: no transient field
entities:[]u32, // offsets of its Entity fields
held:[2]i32, // the ops of World_Schema.memory of what it holds behind pointers
}

// Completes the World_Info that the generated world returns: its tables and
Expand Down Expand Up @@ -119,9 +122,117 @@ world_schema_make :: proc(info:^World_Info,allocator:=context.allocator) -> ^Wor
return nil
}
}
ops:=make([dynamic]Memory_Op,allocator)
compiled:=make(map[typeid][2]i32,context.temp_allocator)
entities,ok:=memory_compile(&ops,&compiled,type_info_of(Entities))
ws.entities=entities
for t,i in info.tables {ws.tables[i].held,ok=memory_compile(&ops,&compiled,t.info);if !ok {break}}
for s,i in info.storages {if ok {ws.storages[i].held,ok=memory_compile(&ops,&compiled,s.info)}}
ws.memory=ops[:]
if !ok {
world_schema_destroy(ws)
return nil
}
return ws
}

// A container a value holds: where it is in the value and how to read what
// it holds. Its children are the ops of one element, read for each element
// it has: of a fixed array, a dynamic array or a slice, or a value of a map.
// A key holds no container: a type that does is not comparable.
@(private)
Memory_Op :: struct {
offset:uintptr,
kind:Memory_Kind,
count:int, // Array: elements
elem_size:int,
map_info:^runtime.Map_Info,
children:[2]i32, // the ops of one element, from its start
}
@(private)
Memory_Kind :: enum u8 {Dynamic,Slice,Map,Array}

// Appends the ops of what a type holds behind pointers: first its own, then
// those of each of their elements. held: the range of its own. A type is
// compiled once: its later uses, as an element of itself too, read that range.
@(private="file")
memory_compile :: proc(ops:^[dynamic]Memory_Op,compiled:^map[typeid][2]i32,info:^runtime.Type_Info) -> (held:[2]i32,ok:bool) {
if known,found:=compiled[info.id];found {return known,true}
own:=make([dynamic]Memory_Op,context.temp_allocator)
elems:=make([dynamic]^runtime.Type_Info,context.temp_allocator)
memory_collect(&own,&elems,info,0)
held={i32(len(ops)),i32(len(ops)+len(own))}
if _,error:=append(ops,..own[:]);error!=nil {return held,false}
if _,error:=map_insert(compiled,info.id,held);error!=nil {return held,false}
for elem,i in elems {
if elem==nil {continue}
children,done:=memory_compile(ops,compiled,elem)
if !done {return held,false}
ops[int(held[0])+i].children=children
}
return held,true
}

// The containers of a value at offset in it, not of their elements: elems
// gets, by op, the type of an element that holds containers.
@(private="file")
memory_collect :: proc(own:^[dynamic]Memory_Op,elems:^[dynamic]^runtime.Type_Info,info:^runtime.Type_Info,offset:uintptr) {
if value_flat(info) {return}
add :: proc(own:^[dynamic]Memory_Op,elems:^[dynamic]^runtime.Type_Info,op:Memory_Op,elem:^runtime.Type_Info) {
append(own,op)
append(elems,elem if elem!=nil && !value_flat(elem) else nil)
}
#partial switch v in runtime.type_info_base(info).variant {
case runtime.Type_Info_Struct:
for i in 0..<int(v.field_count) {memory_collect(own,elems,v.types[i],offset+v.offsets[i])}
case runtime.Type_Info_Array: add(own,elems,{offset=offset,kind=.Array,count=v.count,elem_size=v.elem_size},v.elem)
case runtime.Type_Info_Enumerated_Array: add(own,elems,{offset=offset,kind=.Array,count=v.count,elem_size=v.elem_size},v.elem)
case runtime.Type_Info_Dynamic_Array: add(own,elems,{offset=offset,kind=.Dynamic,elem_size=v.elem_size},v.elem)
case runtime.Type_Info_Slice: add(own,elems,{offset=offset,kind=.Slice,elem_size=v.elem_size},v.elem)
case runtime.Type_Info_Map: add(own,elems,{offset=offset,kind=.Map,map_info=v.map_info},v.value)
}
}

// The bytes of what the ops in held read at data.
@(private)
memory_read :: proc(ops:[]Memory_Op,held:[2]i32,data:rawptr) -> (bytes:int) {
for op in ops[held[0]:held[1]] {
at:=rawptr(uintptr(data)+op.offset)
count,first:=0,rawptr(nil)
switch op.kind {
case .Dynamic:
a:=cast(^runtime.Raw_Dynamic_Array)at
bytes+=a.cap*op.elem_size
count,first=a.len,a.data
case .Slice:
a:=cast(^runtime.Raw_Slice)at
bytes+=a.len*op.elem_size
count,first=a.len,a.data
case .Map:
bytes+=memory_read_map(ops,op,(cast(^runtime.Raw_Map)at)^)
case .Array:
count,first=op.count,at
}
if op.children[0]==op.children[1] {continue}
for i in 0..<count {bytes+=memory_read(ops,op.children,rawptr(uintptr(first)+uintptr(i*op.elem_size)))}
}
return
}

// The bytes of a map and of what its values hold.
@(private="file")
memory_read_map :: proc(ops:[]Memory_Op,op:Memory_Op,m:runtime.Raw_Map) -> (bytes:int) {
if m.data==0 || op.map_info==nil {return}
capacity:=uintptr(runtime.map_cap(m))
bytes=int(runtime.map_total_allocation_size(capacity,op.map_info))
if op.children[0]==op.children[1] {return}
_,values,hashes,_,_:=runtime.map_kvh_data_dynamic(m,op.map_info)
for i in 0..<capacity {
if runtime.map_hash_is_valid(hashes[i]) {bytes+=memory_read(ops,op.children,rawptr(runtime.map_cell_index_dynamic(values,op.map_info.vs,i)))}
}
return
}

// The schema of a world and the type in it of one of its values: of a table,
// a persist storage or a record, or of what they hold. ok is false for another.
world_value_type :: proc(info:^World_Info,type:typeid) -> (s:^serial.Schema,index:serial.Type_Index,ok:bool) {
Expand Down Expand Up @@ -194,7 +305,7 @@ world_schema_destroy :: proc(ws:^World_Schema) {
for &v in ws.storages {value_schema_destroy(&v,allocator)}
for &v in ws.records {value_schema_destroy(&v,allocator)}
delete(ws.tables,allocator);delete(ws.storages,allocator);delete(ws.records,allocator)
delete(ws.table_hashes,allocator)
delete(ws.table_hashes,allocator);delete(ws.memory,allocator)
delete(ws.choices,allocator);delete(ws.schema.named)
serial.schema_destroy(&ws.schema)
_=runtime.mem_free(ws,allocator)
Expand Down
Loading
Loading