Skip to content

Say who owns each zone of a profile, and compare against saved baselines - #13

Open
5hif7y wants to merge 1 commit into
mainfrom
profiler/audit
Open

5hif7y wants to merge 1 commit into
mainfrom
profiler/audit

Conversation

@5hif7y

@5hif7y 5hif7y commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Closes #9.

Rebuilt after the second review as the small PR it proposed. profile.audit, the budgets and the Performance section of the manifest and Project Settings are gone. What is left is what profile.capture + profile.compare did not give: the owner of a zone, and baselines that outlive the session.

Changes

  • Owners:
    • Step_Info gains plugin and game.
    • The generator writes the steps by field name, and tells a package of the game with one check (game_package) that types, records and steps share.
    • Api_Profile_Zone gains owner (Game, Plugin, Engine) and plugin, found once per capture from Step_Info.
    • GPU passes have no owner: a pass does not know who added it.
  • Saved baselines:
    • profile.save {id, name} keeps what profile.compare reads of a capture, as a baseline of the project, in .mirai/profiles/baselines.
    • profile.compare takes baseline, and profile.list names the baselines.
    • The file carries a format version and is validated: a corrupt file, an invalid one, or one of another version is refused, with a message to save it again.
  • Tests and prompt:
    • The generator tests find a step's row by name and check only its fields.
    • The prompt says the agent compares the numbers with the target the user gives.

Code without tests: +217/−53. editor/profile_baseline.odin is 122 of that, and editor/api_profile.odin is +58/−47 because compare now reads a stored summary of each capture. Tests: +92/−7.

Tests

Not tested

🤖 Generated with Claude Code

@5hif7y 5hif7y self-assigned this Nov 9, 2026
@5hif7y

5hif7y commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

Needs more testing as a user before merging.

The tests and the sessions listed in the description cover the engine side only. My lead and I will adapt it as needed, and tomorrow I will test it with a separate agent acting as a Mirai user.

@tomaszbk

Copy link
Copy Markdown
Member

Review: rework. This PR depends on #12. It conflicts with main in core/world.odin, cook/build/world.odin, editor/project_settings.odin, editor/ai_context.odin and other files.

  1. The memory budget measures the wrong process. The budget and the memory-rise finding read process_private of the editor process (editor/api_profile_audit.odin:234, :322).
    • The editor holds far more memory than the shipped game, so over says nothing about the game.
    • Any rise above 0 MB becomes a medium finding, so normal noise triggers it.
  2. Paired structs, and JSON built by cutting strings.
    • The PR brings back the paired Profile_Audit_Out / Api_Profile_Audit structs. main dropped these pairs in 67b5aca: methods now answer with their result structs.
    • It adds the baseline to the answer by cutting the last character off a JSON string (text[:len(text)-1], line 166).
  3. Generator. cook/build/world.odin:2000 writes the main field, which main has removed from Step_Info.
  4. Too much policy in the engine. Fixed severity thresholds and findings written as English sentences put game-specific judgment in the engine. A smaller surface would return the measurements and the budget and let the agent judge them.

@5hif7y

5hif7y commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

Rebuilt on top of the new #12, as ca7eaf89 and 9c23dc68, force-pushed to profiler/audit. It merges into main cleanly.

  1. The memory budget measured the editor. It now measures the game.

    • The memory budget is the peak of the World's memory (core.world_memory, from Record profiles in CPU, GPU and memory tracks #12), not process_private.
    • There is no "rise above 0 MB" finding. The answer gives the rise of the World's memory as a number.
  2. Paired structs and JSON cut as a string. One result struct, written with call_result.

    • The baseline is an optional field holding the compare struct of profile.compare. It is omitted when there is no baseline.
    • No JSON text is cut anywhere.
  3. The generator wrote main. The generator now writes the steps by field name (name=…, library=…, plugin=…, game=…), as it does for storages. A field core adds no longer shifts them. Step_Info gains plugin and game. Test: a_zone_belongs_to_whoever_declares_its_function.

  4. Too much policy. Removed. There is no severity, threshold or English sentence. profile.audit answers:

    • each budget, with what was measured and whether it is over;
    • what bounds the frames;
    • the costliest zones and GPU passes, with their share of the frame and their owner (game, project_plugin, plugin, engine, scene);
    • the rise of the World's memory;
    • the compare against a baseline, when one is given.

    The agent judges. The prompt says how to read owners and budgets, with no numeric thresholds.

Budgets: a Performance section in project.mirai with frame (p95 ms), memory (MB, the World's peak) and load (ms). 0 means no budget. It is validated by manifest_validate, shown in Project Settings and changed with project.set.

Baselines: stored as digests in .mirai/profiles/baselines, read with the call's allocator, and listed by profile.list. Test: a_baseline_keeps_what_compare_reads_of_a_capture.

Not tested: Linux and macOS (type-checked only). A user-side test with a separate agent is still pending, as said before.

@5hif7y
5hif7y requested a review from tomaszbk November 15, 2026 01:26
@5hif7y 5hif7y closed this Nov 15, 2026
@5hif7y 5hif7y reopened this Nov 15, 2026
@5hif7y

5hif7y commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

Follow-up: a live run of profile.audit. Run on all six PRs merged onto main 8c5a2a6, demo, a headless --copy session.

  1. Budgets: project.set {"performance":{"frame":3,"memory":20,"load":100}}.

  2. Audit in Edit, save:"before":

    Budget Measured Over
    frame_ms (budget 3) 4.37 yes
    memory_mb (budget 20) 37.05 yes
    load_ms (budget 100) 9,042 yes
    • load_ms is the first load of the session, imports included.
    • bound is gpu 199 / cpu 41.
    • world_rise_mb is 0.
  3. Play, then audit with baseline:"before":

    • memory_mb is 37.85 and load_ms 369 (the Play load).
    • The compare reports mode Edit → Play as a blocking difference and comparable:false, and still gives the deltas (p95 +2.5 ms, render +1.96 ms).
  4. Owners, with top:64: 9 zones are Game (game_seats, game_use, zombie_think, horror_lights, player_hud…), 35 Plugin (named: physics, navigation, locomotion…), 20 Engine. Engine steps such as camera_follow are Engine.

Not covered: a plugin of the project's own (Project_Plugin), since demo has none; a conversation with the editor's agent.

@tomaszbk

Copy link
Copy Markdown
Member

Review of the rebuilt branch: this does not fit Mirai as it is. I suggest a much smaller PR. The points of the first review are solved as written (no paired structs, no JSON cut as text, no severities). The problem now is the surface: five new concepts (budgets, baselines, digests, owners, audit) and +310 lines of code without tests, for little new information.

What is really new compared with profile.capture + profile.compare

Only three things:

  • the owner of each zone;
  • baselines that survive the session;
  • the check "measured > budget", which the agent can do itself.

Everything else in the answer is already in the summary of #12: p95, bound, zones, passes, the World's peak and rise, and the load time from profile.load.

Why the budgets do not work

  1. They do not belong in project.mirai (project/manifest.odin:81-92). They are numbers of one machine in a versioned file that every machine shares. The shipped game also copies the manifest as it is (cook/build/game.odin:21), so the doc line "the shipped game carries nothing of them" (projects.md:56-58) is not true.
  2. They measure the editor, not the game. In your own live run, the Edit audit gave load_ms 9,042, with the imports of a first load in it. A user who sets frame:16 gets the cost of the editor viewport and its panels.
  3. The memory budget counts only the tables of the World. A user who sets memory:512 expects the process and the GPU. Textures and meshes are not in it.

Other findings

  1. The zone list is computed twice (editor/api_profile_audit.odin:136-155). It rebuilds what profile_summary gives, with a near copy of Api_Profile_Zone.
  2. Every GPU pass has the owner Scene (editor/api_profile_audit.odin:146), so the field says nothing. Passes and jobs of plugins (fog, decals) come out as Scene or Engine.
  3. Baseline files have no version and are not validated (editor/profile_baseline.odin:84-93). A digest of an older format loads as zeros with no error, and no test gives it a corrupt file. (Path traversal is blocked by the name check, that part is fine.)
  4. The generator has the "is it of the game" check a third time (cook/build/world.odin:2392, next to :2377 and of_game at :2474).
  5. stages_test.odin:94-95 asserts whole generated lines, which change with every version.

What I would keep, as a new small PR on top of #12

  • Owner of a zone: add owner and plugin to the existing Api_Profile_Zone, from Step_Info.plugin/game. About 20 lines. A GPU pass gets an owner only if the pass really knows who added it; if not, leave the field out.
  • Saved baselines: let profile.compare take a baseline saved by name, with a version in the file and a test with a corrupt file.
  • Drop profile.audit, the Performance section of the manifest and of Project Settings, and the budgets. The agent compares the numbers it gets with the target the user gives it.

If you think budgets are worth having, they would need to be measured in the player, not in the editor, and live outside the versioned manifest. That is a different design, so let's agree on it in an issue before any code.

- Step_Info gains plugin and game; the generator writes the steps by
  field name and tells a package of the game with one check
  (game_package), which types, records and steps share.
- Api_Profile_Zone gains owner (Game, Plugin, Engine) and plugin, found
  once per capture from Step_Info. GPU passes have none: a pass does not
  know who added it.
- profile.save keeps what profile.compare reads of a capture as a
  baseline of the project, by name, in .mirai/profiles/baselines;
  profile.compare takes baseline. The file carries a format version and
  is validated: a corrupt one or one of another version is refused.
  profile.list names the baselines.
- Generator tests find a step's row by name and check only its fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5hif7y

5hif7y commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

Rebuilt as the small PR you proposed. b04fd42, from main b33d9a2, force-pushed to profiler/audit.

  1. Budgets in project.mirai. Removed, together with the Performance section of Project Settings, Budget_Invalid, the performance field of project.set and their docs. Nothing in the manifest changes.
  2. They measured the editor. There are no budgets now. The prompt says the agent compares the numbers with the target the user gives.
  3. The memory budget counted only the World's tables. There is no memory budget. The World's memory stays what Record profiles in CPU, GPU and memory tracks #12 reports, named as such.
  4. The zone list was computed twice. owner and plugin are two fields of the existing Api_Profile_Zone, filled where profile_summary builds it. The owner of each zone name is found once per capture, in profile_build.
  5. Every pass was Scene. Passes get no owner field. A zone is Game, Plugin (with its name), or Engine for anything that is not a loop function of the game or of a plugin.
  6. Baselines had no version and no validation. The file carries a version. Loading checks it and the contents (frames, times as milliseconds, names). A corrupt file, an invalid one and one of another version are refused with a message to save it again; a_baseline_keeps_what_compare_reads_and_refuses_another_file covers this.
  7. "Is it of the game" a third time. One game_package check in cook/build/world.odin, shared by of_game (types), records and steps.
  8. Whole generated lines in tests. stages_test, contracts_test and minds_test now find a step's row by name and check only the fields each test is about.

Budgets measured in the player are not proposed here.

@tomaszbk

Copy link
Copy Markdown
Member

Review of the rebuilt branch: ready after small fixes. This is exactly the scope we agreed on, thank you. Nothing of audit, budgets, Project Settings or the manifest is left, the game check lives in one place, and baselines have a version, a name check and a test with a corrupt file.

To fix before the merge

  1. A zone nobody can attribute says it is the engine's. profile_zone_owner (editor/profile.odin:488-498) returns Engine both for an engine loop function and for any zone that is not a loop function at all (a part of the frame, a job, the editor), and owner is required in Api_Profile_Zone (editor/api_profile.odin:86).
    • Scenario: work of a project plugin runs under a job zone, so it comes out as Engine. The agent prompt (editor/ai_context.odin:392) then tells the agent that this code is not the project's.
    • Fix: give owner only to loop-function zones and leave it out otherwise, the same rule as for GPU passes.

Smaller points

  1. profile.compare silently drops base when baseline is also given (editor/api_profile.odin:732-736). Refuse the pair with an error that says to pass one of them. It also skips the project check that profile.save makes (:718), so without a project the error names a relative path.
  2. The version of a baseline file is checked after its contents (editor/profile_baseline.odin:93-94). A future format with renamed fields gets "is not a baseline" instead of the message about its version. Check the version first.
  3. Nits:
    • Five leftover blank lines at editor/api_profile.odin:806-810.
    • The owner test (editor/profile_test.odin:363) still passes if zones a or b are missing from the summary: expect them to be there.
    • No generator test checks that a plugin's package gets its plugin name in Step_Info.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The agent of a project cannot tell what of the cost is its own, nor whether a change helped

2 participants