feat: add @metamask/mwp-connect-bench private benchmark harness - #341
Open
adonesky1 wants to merge 7 commits into
Open
feat: add @metamask/mwp-connect-bench private benchmark harness#341adonesky1 wants to merge 7 commits into
adonesky1 wants to merge 7 commits into
Conversation
Private, unpublished playground package for measuring MetaMask Connect (MWP) deeplink connection latency against a MetaMask Mobile dev build in the iOS simulator. Includes: a NOT-FOR-MERGE instrumentation patch for metamask-mobile's Connection service ([MWPPerf] timing logs), a session seeder that persists synthetic trusted-mode sessions via real relay handshakes (no dapp required, no approval UI), a cold-start trial runner, and a log analyzer that reports median/p75 handshake durations plus resume-regression and contention indicators. Built to A/B metamask-mobile#32475 (defer session resume/reconnect while a new connect is in flight, WAPI-1566); reusable for any MWP connect-latency experiment (WAPI-1564).
- Add create_session_received marker to the instrumentation (when the wallet_createSession request reaches the wallet) and an "approval gate" metric (create_session_received - connect_start) to the analyzer, so the approval path can be measured, not just the handshake. - Add --with-request trial mode: embeds an inline wallet_createSession (direct-deeplink flow) so approval-path trials are possible. Each such trial pops a real approval; reject between trials. - Add a dedicated instrumentation patch variant for metamask-mobile#32470 (eager approval), whose refactored Connection service the base patch cannot target. Base patch verified against main, #32475, and #32473; variant verified against #32470. - Add a per-PR measurement guide (patch/trial-mode/metric/expected result for #32475, #32470, #32473) to the README.
…marked Release builds strip console.* via transform-remove-console and run without Metro, so the instrumentation now logs through a bound console.warn alias and appends JSON lines to <Documents>/mwp-perf.log (both patch variants). README documents the Release-build workflow (cold-start fidelity, log collection, arm-swap rebuild caveat).
Remove shebangs, replace process.exit with throw, rename short/denylisted identifiers, add curly braces and JSDoc description, fix import order. The ./lib.mjs extension imports keep a targeted inline disable — plain Node ESM requires the extension at runtime.
Deliver deeplinks with plain 'xcrun simctl' (terminate + openurl) so the harness has no dependency on private local tooling; simulator and bundle id are overridable via BENCH_SIM_DEVICE / BENCH_BUNDLE_ID. Add AGENTS.md with the dev-build and Release-build playbooks plus the pitfalls learned running the first A/B (dev-launcher deeplink swallowing, console stripping, rebuild after arm swap, session cap, trial spacing, throttle verification).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Explanation
Adds
playground/mwp-connect-bench— a private, unpublished package for measuring MetaMask Connect (MWP) deeplink connection latency against a MetaMask Mobile dev build in the iOS simulator, and for verifying connection-flow changes don't regress session resume.Why: we needed a repeatable rig to A/B MetaMask/metamask-mobile#32475 (defer session resume/reconnect while a new connect is in flight, WAPI-1566), and the pieces are reusable for any MWP connect-latency experiment (WAPI-1564 investigation).
What's inside:
patches/mwp-perf-instrumentation.patch— NOT-for-merge patch adding[MWPPerf]timing logs to metamask-mobile'sConnectionservice (create / connect-handshake / resume spans). Applied locally to the build under test.seed-sessions.mjs— persists N synthetic MWP sessions to create cold-start resume load. Works because trusted-mode handshakes require no dapp acknowledgement: a valid connect deeplink withoutinitialMessage(QR-style) does a real relay handshake and persists a session with no approval UI. Payloads pass the wallet'sisConnectionRequest()validation, including real compressed-secp256k1 peer keys (via this package's owneciesjsdep, pinned to the version already in the repo).run-trial.mjs— one cold-start measurement trial (force-kill app → fresh connect deeplink via themmdlsimulator CLI).analyze.mjs— parses tee'd Metro logs; reports median/p75 handshake durations, resume failures (regression check), and resume-during-connect counts (contention indicator).The README documents the full A/B protocol (single native build, JS-only arm swap, ABBA trial blocks, network throttling, median-based comparison) and trial hygiene (session cap, simulator persistence, cleanup).
Constraints/CI: playground packages are excluded from the publish/constraints checks (
yarn.config.cjs), andyarn constraintspasses. Lockfile updated for the new workspace.References
Checklist