[hoangsnowy]

Change description

Plan 1.9 perf(bench), following owner directive 2 ("measure, then fix"). Later commits fix CodeQL alerts (0e90f06, which also enforces IDE0044 at build time) and harden the gate after review (6edae2d).

Commit 1: suite and gate

• tests/AgentOs.Benchmarks is a BenchmarkDotNet 0.15.8 console app (the latest stable as of 2026-11-01) added to AgentOs.slnx. It is not a test project, so dotnet test does not run it.
• 52 benchmarks with [MemoryDiagnoser] and realistic inputs. No network or DB: the EF settings store runs on the in-memory provider. They cover:
  • per-LLM-call lookups: CostCalculator (exact ids, dated and Vertex snapshots, unknown model, long-context tier) and ModelCapabilities.TemperatureFor;
  • the settings read path: SettingsService (tenant → platform → seed → default) and EfAppConfigStore cache hits;
  • prompt and reply text: UntrustedText, RepoInstructions, PullRequestText, and JsonExtractor on ~100 KB agent replies;
  • queues: InMemorySessionRunFeed and BufferedToolInvocationLog;
  • desktop folds: Insights ByRole/LatencyByRole with AgentRoles.Canonical, Activities SettingsSearch.Find, and Board › Knowledge KnowledgeLayout.Compute.
• .github/workflows/perf.yml runs on workflow_dispatch and nightly. It uses the ShortRun job, uploads the artifacts, then runs compare. Actions are pinned by SHA: checkout v7.0.1, setup-dotnet v6.0.0, upload-artifact v7.0.2.
• The comparer (Comparison/BaselineComparer.cs, pure BCL, unit-tested in AgentOs.Tests including a deliberate O(n²) regression case) fails a benchmark when it:
  • allocates more: 0 B → anything, or allocation growth above 20% plus a 64 B floor;
  • crashes: no statistics in the report, or the benchmark run exits non-zero;
  • is missing on a full run: --filter '*' passes --require-all;
  • got slower: Mean above +20% (with a 2 ns noise floor). This fails manual runs only; on the nightly it is an advisory ::warning (see the machine caveat below).
• Baseline: tests/AgentOs.Benchmarks/baseline/*.json (BenchmarkDotNet brief reports).
• Docs: docs/architecture.md §8.

Commit 2: fixes for waste the suite measured. Same machine, ShortRun, before → after:

Hot path	Before	After
SettingsService.GetAsync (every LLM call / policy check)	168–176 B, 282 ns (tenant hit)	0 B, 178 ns
EfAppConfigStore cache hit	104 B	32 B
UntrustedText.Wrap, 4 KB ticket body	9.31 µs, 23.7 KB	2.57 µs, 16.0 KB
UntrustedText.Wrap, 40 KB body (capped)	38.5 µs, 93.9 KB	8.7 µs, 62.8 KB
SettingsSearch.Find (per keystroke)	37.6 µs, ~33 KB	27–38 µs, 0.6–1.2 KB
AgentRoles.Canonical over the Insights name set	312 ns, 160 B	257 ns, 0 B
JsonExtractor.ExtractJson, fenced 100 KB reply	298 µs	273 µs (−8%: impossible parse attempts no longer throw)

• Behaviour: unchanged. One exception: the EF settings cache key is now a (tenant, key) tuple, which also removes a latent collision between "a:b"+"c" and "a"+"b:c".
• Tests: existing tests cover each path, and new tests pin the edge cases (C1/DEL controls, CR at the length cap, the 40-character kind cap, comment-prefixed JSON, "Qa Agent").

Machine caveat: the baseline was recorded on a 4-vCPU Intel Xeon 2.1 GHz cloud VM, not on a GitHub-hosted runner. ShortRun error bars on that VM were large, and timing-only regressions on shared runners need a re-run before they mean anything. Allocation checks don't depend on the machine. To re-record, use the benchmark-results artifact from a perf.yml run on main (see the docs).

Not covered: the plan entry also lists the event sink and ledger KPI queries. There is no Metrics module on main yet (plan 1.2), so those benchmarks will come with it.

Type of change

• New feature (new agent / endpoint / capability)
• Bugfix
• Refactor (no behavior change)
• Documentation / README
• Infrastructure / CI configuration

Related issues

Plan docs/plans/agentic-sdlc-v2.md item 1.9, now marked shipped.

Checklist

• dotnet build passes locally in Release mode (0 errors; the only warning is the existing ASPIRE010 from AppHost)
• dotnet test passes: 2014 passed, 86 skipped, 0 failed locally after rebasing on main at 877a714; CI is green on the current head; the Benchmarks project is not run
• No secrets committed (API keys, connection strings)
• README / docs updated if public behavior changed
• Measurable results provided when related to agent output quality (before/after table above)

🤖 Generated with Claude Code

https://claude.ai/code/session_016Twi9motDLcziFta5zJSDs

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.