Repository navigation
test(ui): profile documentation export phases without reducing coverage - #261
Merged
Merged
Conversation
johnny4young
marked this pull request as ready for review
October 7, 2026 02:14
…er labels - Fail the self-test when the UI test's phase, locale, appearance, or size labels drift from the summarizer's constants. - Read job logs with replacement decoding so a stray byte elsewhere in a large log cannot reject complete marker evidence. - Isolate the unknown-label guard in the self-test, cover non-numeric durations, and replace the bare assert so python -O cannot skip the positive check. - Describe the lint sub-check in the Makefile and link the profiling guide from the release documentation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add diagnostic timing to the existing documentation-export UI test so future CI optimization is based on observed bottlenecks. This PR is instrumentation only: no performance improvement is claimed or implemented.
Scope and preserved coverage
No production code, workflow matrix, thresholds, golden baselines, expected skips, dependency, signing, or sandbox changes.
Exact revision
Base main: bb8fea0
Published source head: e52a594
Tested tree: c516980c4630cc1eaf57f72a1fd92396274e1a28
All ten existing open PRs were rechecked before publication; none supplies these markers/parser. This independent PR excludes first-use work in #251 and dependency work in #252/#260. Authenticated publication preserves the exact tested tree; the remote commit differs from the earlier local candidate because GitHub supplied commit metadata.
Validation and limits
Passed locally on Linux: parser self-tests; git diff --check; hygiene and changelog/version checks; portable make lint prerequisites, including the 53-state screenshot contract. The QA archive sub-check explicitly skips without ditto/unzip.
Attempted but unavailable locally: make format and final Swift-format lint require xcrun; make build requires XcodeGen/full Xcode. Native compilation and exact-head hosted CI subsequently passed; actual per-OS diagnostic observations are recorded below. Parser success alone cannot establish a passing UI run or performance gain.
Reviewer checklist
No merge, release, deployment, force-push, actual user-content capture, or security-setting change.
Exact-head hosted qualification and measured baseline
CI run 37553875872 completed successfully at unchanged source head
e52a594338024d128d9465069d4869969b3e1550, tested treec516980c4630cc1eaf57f72a1fd92396274e1a28. All five jobs pass, including native strict Swift formatting, Debug/universal Release builds, coverage/performance/golden gates, both UI jobs, and Linux static/site checks. Controlled WebKit also passes. Local Xcode limitations are unchanged; hosted native execution supplies that evidence.Each OS's documentation export test passes all eight states and its separate cancellation case. Each ordinary UI run reports 108 cases, zero failures, and 26 intentional tour skips; the separate strict tour runs all 26 successfully and exports/validates all 53 required screenshots. Source comparison confirms every original DocumentationExportUITests line remains in original order; this Swift change is additions only.
Timing provenance
20260907.0337.1. Documentation export test: 348.478 s; sum of measured phases: 348.34006429166675 s.20260907.0351.1. Documentation export test: 379.103 s; sum of measured phases: 378.87820220833345 s.All 64 raw marker lines from each completed job were extracted unchanged and passed through the committed Python parser separately: exactly eight states × eight phases, with no duplicates or missing phases. Parser results are preserved below; original full logs are linked above. These are one instrumented run per OS, not matched before/after trials.
The folder-panel phase is the largest measured phase on both platforms: 138.8766798333334 s / 348.34006429166675 s = 39.87% on Sequoia and 156.05349120833336 s / 378.87820220833345 s = 41.19% on Tahoe. These percentages are of measured documentation-test segments, not workflow duration or potential savings. Export phase sums are 26.359654124999906 s and 22.368217000000072 s; they include automation overhead and are not pure rendering benchmarks.
Concrete next investigation: profile the folder-panel Go to Folder typing and readiness path while preserving real native panel interaction, all locales/sizes, assertions, screenshots, cancellation, and output checks. This PR deliberately makes no optimization or test-selection change. Cross-OS timings cannot establish a speedup; five matched pairs per OS remain the documented future qualification target.
Retained native artifacts
Artifacts were verified against this run/head and are subject to GitHub retention. This qualifies the instrumentation-only scope for human review, not an optimization benefit or a combined integration with another PR.
Exact parser results, seconds, one job attempt per OS
{ "sequoia": { "states": { "en-dark-1280": { "launch": 2.279791125000031, "navigation": 4.363394833333302, "description": 5.139703625000038, "package-setup": 7.469274833333316, "panel": 22.021975499999996, "export": 3.5320828750000146, "validation": 0.0005284583332922921, "termination": 1.0278351250000242 }, "en-dark-minimum": { "launch": 2.7560145833332967, "navigation": 9.311627041666668, "description": 5.077317833333325, "package-setup": 7.403052458333377, "panel": 16.807455833333336, "export": 3.5208908333333397, "validation": 0.0011454999999500615, "termination": 1.1017827500000408 }, "en-light-1280": { "launch": 1.7316769583333098, "navigation": 4.270962875000009, "description": 5.159194458333332, "package-setup": 7.343094166666674, "panel": 14.395339791666686, "export": 3.084414874999993, "validation": 0.0005963749999864376, "termination": 1.0216663333333145 }, "en-light-minimum": { "launch": 1.7540418333333605, "navigation": 9.286213250000003, "description": 5.22500195833328, "package-setup": 7.087882958333353, "panel": 11.682465833333367, "export": 2.8896588749999523, "validation": 0.0006012916666691126, "termination": 1.003700125000023 }, "es-dark-1280": { "launch": 2.2434853333332967, "navigation": 4.2771527500000275, "description": 5.230763499999966, "package-setup": 7.267681666666704, "panel": 23.532939083333304, "export": 2.9573490833333835, "validation": 0.0008770833333073824, "termination": 1.0930141249999679 }, "es-dark-minimum": { "launch": 2.3521852916667285, "navigation": 9.242664874999946, "description": 5.63882804166667, "package-setup": 7.2779436250000344, "panel": 15.631971541666644, "export": 3.456666416666735, "validation": 0.0005723333332525726, "termination": 1.0397507083333721 }, "es-light-1280": { "launch": 2.3667610000001105, "navigation": 4.560526833333256, "description": 5.479237958333329, "package-setup": 7.034189083333331, "panel": 16.918474375000073, "export": 3.3462300833332392, "validation": 0.0007847916667742538, "termination": 1.0071003749999363 }, "es-light-minimum": { "launch": 2.609224291666692, "navigation": 9.511713916666679, "description": 6.1748589999999695, "package-setup": 7.8241122500000415, "panel": 17.886057875000006, "export": 3.5723610833332486, "validation": 0.000826083333436145, "termination": 1.0533746666666275 } }, "phase_seconds": { "launch": 18.093180416666826, "navigation": 54.82425637499989, "description": 43.12490637499991, "package-setup": 58.70723104166683, "panel": 138.8766798333334, "export": 26.359654124999906, "validation": 0.005931916666668258, "termination": 8.348224208333306 }, "total_seconds": 348.34006429166675, "state_count": 8, "qualification": "diagnostic timings only; no controlled performance improvement claim" }, "tahoe": { "states": { "en-dark-1280": { "launch": 1.5257922083333142, "navigation": 4.844142708333379, "description": 6.784884916666613, "package-setup": 8.504778500000043, "panel": 27.168803166666635, "export": 2.680198541666698, "validation": 0.0004265833333079172, "termination": 1.0721147083333449 }, "en-dark-minimum": { "launch": 1.8394867499999918, "navigation": 9.692541541666685, "description": 7.160198208333327, "package-setup": 8.104247833333318, "panel": 18.355442625000023, "export": 2.6402034166666226, "validation": 0.0009663750000186155, "termination": 1.0353093333333163 }, "en-light-1280": { "launch": 1.6878318749999721, "navigation": 5.161220833333346, "description": 6.63823237500003, "package-setup": 8.491216541666631, "panel": 17.23140354166668, "export": 2.839315208333346, "validation": 0.0004067499999678148, "termination": 1.0299057500000117 }, "en-light-minimum": { "launch": 1.717336208333336, "navigation": 9.705470833333322, "description": 6.655541500000027, "package-setup": 8.84060012499998, "panel": 16.001727208333307, "export": 3.4695575416666884, "validation": 0.000528583333334609, "termination": 1.0288411666666661 }, "es-dark-1280": { "launch": 1.4277932916667169, "navigation": 4.872233666666602, "description": 6.182006041666682, "package-setup": 8.187423333333413, "panel": 21.870125083333278, "export": 2.6599800416666994, "validation": 0.0004202916666145029, "termination": 1.014872916666718 }, "es-dark-minimum": { "launch": 1.4479518333333772, "navigation": 9.57756100000006, "description": 6.387815583333236, "package-setup": 8.36662350000006, "panel": 16.847247666666703, "export": 2.6958349166666267, "validation": 0.00042862500004048343, "termination": 1.0480000000000018 }, "es-light-1280": { "launch": 1.4643467500000042, "navigation": 4.826985708333382, "description": 6.710479874999919, "package-setup": 8.351867083333332, "panel": 16.767045458333428, "export": 2.700813791666633, "validation": 0.0004127500000095097, "termination": 1.0142245416666356 }, "es-light-minimum": { "launch": 1.6389996249999967, "navigation": 9.885021833333326, "description": 6.661802000000023, "package-setup": 8.82349579166663, "panel": 21.8116964583333, "export": 2.6823135416667583, "validation": 0.0004293333332725524, "termination": 1.0432764166666857 } }, "phase_seconds": { "launch": 12.749538541666709, "navigation": 58.5651781250001, "description": 53.180960499999856, "package-setup": 67.67025270833341, "panel": 156.05349120833336, "export": 22.368217000000072, "validation": 0.004019291666566005, "termination": 8.28654483333338 }, "total_seconds": 378.87820220833345, "state_count": 8, "qualification": "diagnostic timings only; no controlled performance improvement claim" } }