Skip to content

hk: avoid native graphics cache reset work on AGX2 - #69

Closed
RyanTheTide wants to merge 2 commits into
aurora-wipfrom
perf/agx2-reset-only-20261025
Closed

RyanTheTide wants to merge 2 commits into
aurora-wipfrom
perf/agx2-reset-only-20261025

Conversation

@RyanTheTide

@RyanTheTide RyanTheTide commented Oct 24, 2026 •

Copy link
Copy Markdown

Draft: both Macs have complete 624,218-case coverage on the exact reset-only head with zero new bad results or skips. M1 Pro coverage spans two boots after the known pre-existing t-3018 wedge; the failed attempt is preserved below. Twelve G17P pairs leave the d1 performance signal inconclusive, so no clean G17P non-regression claim or merge recommendation is made.

Command-buffer resets on G13/G14 clear native graphics caches that those generations never populate. Leave those caches untouched after their zeroed allocation, while continuing to reset common recording state. Native graphics generations retain the existing invalidation.

This is the reset-only half of #68, at exactly 3d00b6d8ae7bea7b19fd700916a58748ad6b1010:

  • 51c0bfd9: avoid clearing native graphics caches on AGX2 resets.
  • 3d00b6d8: leave the native background/end-of-tile cache untouched on AGX2 resets.

The G13X control-stream ioctl split (af16d985) is excluded. That work remains in #68 pending a kernel teardown fix and a fresh qualification. This PR does not change submission grouping or query synchronization.

Qualification

Keep this PR draft for review. The exact-head results are recorded below; the combined-head measurements in #68 are historical comparisons.

  • M2 and M1 Pro: paired Fedora Honeykrisp/base 4911479b/reset-only gpubench comparisons, four alternating repeats per build. Report median of independent repeat medians and their ranges, with process-local ICDs, inactive displays, and cooldown gates.
  • M2 and M1 Pro: affected-Vulkan v2 CTS, 624,218 frozen cases per device, compared against base with exact case-set checks and explicit new-bad/new-skip deltas.
  • G17P: focused occlusion-query repeat against base with more repetitions and cooldown gating, to resolve the earlier combined-head d1 reading.

Both Macs have completed the benchmark comparisons and exact-head CTS case coverage with zero new bad results or skips. M1 Pro coverage uses explicit complete segments across two boots after the known pre-existing wedge; it is not an uninterrupted full run. G17P results are recorded below; the d1 signal remains inconclusive. No universal stock-parity or clean G17P non-regression claim is made.

M2 measurements

Microseconds, median of four independent repeat medians, with minimum/maximum repeat medians in brackets. Each build has its own interleaved Fedora Honeykrisp comparison. Base is 4911479b; reset-only is 3d00b6d8. Lower is better.

Device Workload Depth Stock for base Base Stock for reset Reset-only Reset/base Reset/stock
M2 G14G Occlusion overhead 1 75.55 [66.41, 83.23] 51.34 [47.66, 55.48] 76.79 [59.8, 81.22] 56.08 [48.73, 60.29] 1.0923 0.7302
M2 G14G Occlusion total 1 321.1 [317.7, 325.9] 303.2 [301.2, 308.3] 327.7 [318.1, 329.4] 307.7 [304.9, 311.3] 1.0148 0.9391
M2 G14G Occlusion total 16 89.02 [88.84, 89.54] 87.14 [86.81, 87.23] 89.13 [88.94, 90.48] 87.09 [87.05, 87.23] 0.9995 0.9772
M2 G14G Idle wake, 20 ms 1 368.3 [363.6, 369.8] 369 [366.2, 371.4] 371.8 [365.3, 373.4] 372.7 [370, 375.3] 1.0101 1.0026
M2 G14G Clear 1 170.5 [169.6, 175.7] 175.8 [175.4, 177.1] 173 [169.6, 175.8] 178.6 [176.9, 181.3] 1.0161 1.0324
M2 G14G Fill, 1080p 1 868.1 [865.1, 874.1] 931.5 [874.1, 980.4] 865.1 [859.1, 874.1] 971.1 [886.5, 1033] 1.0424 1.1225
M2 G14G Empty submit 32 1.188 [1.185, 1.196] 1.264 [1.262, 1.266] 1.186 [1.184, 1.187] 1.178 [1.176, 1.188] 0.9315 0.9929
M2 G14G Empty submit 64 1.181 [1.178, 1.19] 1.262 [1.26, 1.264] 1.18 [1.18, 1.183] 1.177 [1.175, 1.191] 0.9323 0.9974

Reset-only closes the empty-submit gap: 6.8% lower than base at both depths and within 1% of stock. The depth-1 query total remains faster than stock, while the base/reset repeat ranges overlap. Clear remains 3.2% slower than stock. The short-window fill median is 12.3% above stock and 4.2% above base, with overlapping base/reset ranges; the longer fill stability check below measures a much smaller remaining gap. These results do not claim that every workload reaches stock performance.

Method

  • gpubench 070c2579fcc2156f5ffef50eab06a8f41d3eab0e; fixed M2 binary SHA256 7619ae4f38e0ace5742d96b152aa70e3e7e7c906e8fbb60b1d58ea935343e9c6.
  • 300 ms warmup, five 250 ms samples, three timestamped samples. Four repeats in AB, BA, AB, BA order. At least 60 s cooldown between builds, gated within 2 C of the initial reference temperature.
  • Inactive display, process-local ICDs, SELinux enforcing, unchanged 7.3.0-aurora-73w boot. No installed package changes.
  • Fedora Honeykrisp 26.2.3-1.fc44, extracted from the verified rollback RPM. Base/reset use the same release recipe: plain build, Fedora compiler flags, b_ndebug=true, AArch64 offline SDK.
  • Reset overlay SHA256 99ec4b5755d868e5235a0ed630364fd62daa9081627c43005b61ab0364cad0b7; Vulkan library SHA256 646c07a99b4a395c3612219bd99b239f977f5f6f25db11732d0468d572ff88d5.
M2 raw repeat medians and underlying benchmark sample arrays (microseconds)
[{"device":"M2 G14G","test":"query","depth":1,"metric":"overhead_us","repeat_medians_us":{"stock-for-base":[83.2268,72.9014,78.2014,66.4057],"base":[52.7251,55.4808,47.6551,49.9485],"stock-for-reset":[74.708,81.2178,78.8798,59.8017],"reset":[54.3672,48.7305,60.2883,57.7828]},"samples_us":{"stock-for-base":[null,null,null,null],"base":[null,null,null,null],"stock-for-reset":[null,null,null,null],"reset":[null,null,null,null]}},{"device":"M2 G14G","test":"query","depth":1,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[325.9452,319.6931,322.5806,317.662],"base":[305.2503,308.2614,301.2048,301.2048],"stock-for-reset":[328.5151,326.7974,329.3808,318.0662],"reset":[305.6235,304.878,309.7893,311.3325]},"samples_us":{"stock-for-base":[[319.6931,333.7784,325.5208,325.9452,325.9452],[322.1649,331.1258,318.8776,319.6931,317.2589],[317.662,331.565,323.8342,322.5806,322.5806],[341.5301,318.8776,317.2589,317.662,317.662]],"base":[[305.2503,305.9976,300.8424,305.2503,322.1649],[312.5,306.3725,304.1363,312.5,308.2614],[305.2503,299.7602,302.2975,299.4012,301.2048],[306.3725,299.7602,301.9324,300.8424,301.2048]],"stock-for-reset":[[333.7784,323.8342,328.5151,320.5128,328.5151],[327.654,326.7974,324.6753,327.654,325.5208],[326.7974,329.3808,332.0053,324.6753,332.0053],[318.0662,322.1649,312.8911,313.2832,320.5128]],"reset":[[309.0235,300.12,305.2503,305.6235,308.642],[316.0556,308.2614,304.878,302.2975,299.4012],[302.6634,310.1737,314.4654,309.0235,309.7893],[312.5,313.6763,307.8818,311.3325,303.7667]]}},{"device":"M2 G14G","test":"query","depth":16,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[89.5415,88.8415,89.0948,88.9363],"base":[86.8056,87.2296,87.0777,87.1992],"stock-for-reset":[89.1902,89.0631,88.9363,90.4814],"reset":[87.2296,87.1384,87.0474,87.0474]},"samples_us":{"stock-for-base":[[89.5415,89.6379,88.968,89.7666,89.4134],[88.9047,88.8415,88.6211,89.0948,88.7784],[89.0631,89.0948,88.968,89.5736,89.4454],[88.8099,89.3176,89.1583,88.8415,88.9363]],"base":[[86.8056,88.5897,87.321,86.7754,86.6251],[87.2296,87.108,86.9868,87.9044,87.8117],[86.8357,87.2296,87.0474,87.321,87.0777],[86.9868,87.4432,87.1992,87.4126,86.9565]],"stock-for-reset":[[89.9928,88.8731,88.4956,89.4134,89.1902],[89.5736,88.968,88.8731,89.0631,89.3176],[89.2857,88.5583,88.9363,88.8099,89.0313],[90.7112,90.4814,90.6783,90.2527,90.2527]],"reset":[[87.5044,87.0171,87.2296,87.321,86.9868],[87.1688,87.0171,87.1384,87.2905,86.8056],[87.6885,86.4753,87.1384,86.7152,87.0474],[87.382,87.0474,88.2145,86.8056,87.0171]]}},{"device":"M2 G14G","test":"idle.wake","depth":1,"metric":"wake_latency_us","repeat_medians_us":{"stock-for-base":[363.5635,369.7715,368.4385,368.126],"base":[369.126,366.2095,371.3555,368.8965],"stock-for-reset":[372.897,365.2925,373.4385,370.647],"reset":[370.0225,372.897,372.5845,375.314]},"samples_us":{"stock-for-base":[null,null,null,null],"base":[null,null,null,null],"stock-for-reset":[null,null,null,null],"reset":[null,null,null,null]}},{"device":"M2 G14G","test":"render.light","depth":1,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[169.6065,175.6852,170.2997,170.6485],"base":[175.4386,177.0538,175.8087,175.8087],"stock-for-reset":[175.8087,169.6065,171.4678,174.581],"reset":[181.2908,178.9549,176.9285,178.3167]},"samples_us":{"stock-for-base":[[176.8034,170.4158,168.8049,167.336,169.6065],[175.9324,174.3375,176.8034,171.7033,175.6852],[169.3767,170.2997,170.6485,169.4915,171.5854],[169.1475,170.068,170.6485,173.9736,174.0947]],"base":[[176.4291,175.8087,175.1927,174.3375,175.4386],[182.7485,177.8094,175.4386,176.6784,177.0538],[177.683,174.4592,175.8087,172.8907,176.1804],[177.0538,176.3047,175.4386,174.4592,175.8087]],"stock-for-reset":[[175.8087,177.4308,172.4138,176.4291,141.1632],[167.8979,169.6065,170.1838,168.3502,175.1927],[171.4678,171.4678,173.0104,169.7217,173.6111],[171.4678,175.07,174.581,174.581,179.0831]],"reset":[[182.615,180.6358,176.3047,181.2908,181.5541],[178.8269,181.0282,176.8034,178.9549,179.8561],[179.0831,173.7318,176.9285,175.6852,178.9549],[175.6852,174.4592,179.7268,178.3167,182.7485]]}},{"device":"M2 G14G","test":"render.heavy","depth":1,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[874.1259,871.0801,865.0519,865.0519],"base":[874.1259,886.5248,976.5625,980.3922],"stock-for-reset":[874.1259,871.0801,859.1065,859.1065],"reset":[1033.0579,886.5248,909.0909,1033.0579]},"samples_us":{"stock-for-base":[[877.193,886.5248,874.1259,871.0801,868.0556],[865.0519,871.0801,862.069,880.2817,871.0801],[865.0519,868.0556,856.1644,871.0801,856.1644],[865.0519,868.0556,859.1065,865.0519,862.069]],"base":[[871.0801,874.1259,877.193,874.1259,984.252],[886.5248,886.5248,862.069,886.5248,874.1259],[874.1259,874.1259,976.5625,1028.8066,1041.6667],[883.3922,877.193,1008.0645,980.3922,988.1423]],"stock-for-reset":[[874.1259,886.5248,859.1065,886.5248,871.0801],[862.069,871.0801,859.1065,874.1259,871.0801],[859.1065,865.0519,859.1065,865.0519,859.1065],[856.1644,865.0519,859.1065,865.0519,859.1065]],"reset":[[925.9259,1131.2217,1054.8523,1012.1457,1033.0579],[886.5248,886.5248,862.069,883.3922,886.5248],[865.0519,874.1259,909.0909,1046.0251,1016.2602],[874.1259,932.8358,1041.6667,1033.0579,1037.3444]]}},{"device":"M2 G14G","test":"submit.empty","depth":32,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[1.1868,1.1886,1.1854,1.1958],"base":[1.263,1.2654,1.2662,1.2622],"stock-for-reset":[1.1859,1.1861,1.1843,1.1869],"reset":[1.1786,1.1885,1.1762,1.1766]},"samples_us":{"stock-for-base":[[1.185,1.1868,1.1837,1.1874,1.1878],[1.1827,1.1869,1.1901,1.1894,1.1886],[1.1835,1.187,1.1837,1.1871,1.1854],[1.1954,1.1958,1.1958,1.1961,1.1963]],"base":[[1.2507,1.2657,1.263,1.2636,1.2627],[1.2659,1.2695,1.2651,1.265,1.2654],[1.26,1.2684,1.2656,1.2662,1.267],[1.2591,1.2627,1.2612,1.2622,1.2629]],"stock-for-reset":[[1.1848,1.1872,1.1859,1.1856,1.1868],[1.1861,1.1849,1.186,1.1861,1.1866],[1.1835,1.1836,1.1843,1.1846,1.186],[1.184,1.1869,1.1888,1.1869,1.1898]],"reset":[[1.1776,1.1809,1.1786,1.1782,1.1789],[1.1869,1.1889,1.1882,1.1886,1.1885],[1.1742,1.1764,1.1756,1.1762,1.1774],[1.177,1.1778,1.1764,1.1766,1.1762]]}},{"device":"M2 G14G","test":"submit.empty","depth":64,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[1.1821,1.1801,1.1783,1.1896],"base":[1.2598,1.2638,1.261,1.2641],"stock-for-reset":[1.1798,1.1797,1.1801,1.1834],"reset":[1.1768,1.1907,1.177,1.1746]},"samples_us":{"stock-for-base":[[1.1778,1.1821,1.1861,1.1795,1.1859],[1.1752,1.1801,1.1861,1.1785,1.1868],[1.1748,1.1783,1.1806,1.1767,1.1869],[1.1859,1.1896,1.191,1.1876,1.2006]],"base":[[1.2473,1.2534,1.2598,1.265,1.2644],[1.24,1.259,1.2662,1.2638,1.2673],[1.2459,1.2548,1.2638,1.261,1.2624],[1.2543,1.2591,1.2641,1.2671,1.2737]],"stock-for-reset":[[1.1744,1.1798,1.183,1.1784,1.1844],[1.1738,1.1797,1.1825,1.1775,1.1858],[1.175,1.1801,1.1814,1.177,1.1876],[1.1761,1.1834,1.1835,1.177,1.1883]],"reset":[[1.1737,1.1768,1.1819,1.1767,1.1866],[1.185,1.1876,1.1907,1.1908,1.2003],[1.1743,1.177,1.1813,1.1751,1.1919],[1.1663,1.1736,1.1763,1.1746,1.1845]]}}]

The repeat index is the order of each array (1–4). A null sample array means that metric is reported by the benchmark as a derived median rather than a sample series; the repeat medians are retained.

M2 longer fill stability check

Same separate method as #68: 1 s warmup, five 2 s samples, timestamps disabled, four alternating stock/reset pairs with cooldown and dark checks. Stock: 1005 [1001, 1015] us; reset-only: 1018 [1016, 1023] us, 1.0124x stock. The 12.3% short-window reading does not persist under this method. A smaller measured fill gap remains; this comparison does not isolate its cause to the reset commits.

Longer fill raw repeat medians and samples (microseconds)
{"stock":{"repeat_medians_us":[1014.7133,1000.5003,1009.5911,1000.5003],"samples_us":[[1001.5023,1007.0493,1014.7133,1030.3967,1020.929],[1000.5003,1017.294,997.009,1013.1712,974.1841],[956.0229,1020.929,995.0249,1009.5911,1016.7768],[1000.5003,971.3453,1011.6338,993.0487,1015.744]],"median_us":1005.0457,"range_us":[1000.5003,1014.7133]},"tip":{"repeat_medians_us":[1015.744,1018.8487,1023.0179,1016.2602],"samples_us":[[1027.2214,1034.6611,1007.0493,1015.744,1013.1712],[1042.7529,1017.8117,1033.0579,1013.1712,1018.8487],[1012.1457,1033.0579,1023.0179,1031.9917,991.5716],[1015.2284,1020.4082,1013.6847,1029.3361,1016.2602]],"median_us":1017.5544500000001,"range_us":[1015.744,1023.0179]}}

Affected Vulkan CTS qualification

CTS 1.4.6.2, frozen affected-Vulkan v2 list: command buffers, queries, synchronization/synchronization2, render passes, image clears, blend, dynamic state, copies/blits. Eight jobs, 5,000-case chunks, grouping 250, process-local ICDs and SELinux enforcing.

Device Cases per build Base results Reset-only results New bad New skips
M2 624,218 241,612 Pass / 382,606 Skip 241,612 Pass / 382,606 Skip 0 0
M1 Pro 624,218 241,611 Pass / 382,606 Skip / 1 Flake 241,612 Pass / 382,606 Skip 0 0

M2: complete uninterrupted 125-chunk reset-only and fresh base runs on the same unchanged 7.3.0-aurora-73w boot. Exact case-set, kit, source/library hashes, grouping and job-count comparisons pass; all per-case statuses are identical. The list SHA256 is 093289224e73c9be47e60d2de3919e565622ab5a51575f1003d5d7c60a79ad2f and CTS binary SHA256 is a7d5927e16f1edcd99d0639c18289c5ec1729b43ccb82d714b4f6585cf86969d.

Filtered per-chunk kernel counts: reset-only 31 scheduler-teardown lines across 21 chunks; base 33 across 28 chunks. Both have zero firmware-crash, GPU-crashed and context-invalidation-failure messages. These are descriptive counts from one complete run per build, not an estimate of long-term crash probability.

M1 Pro measurements

Same four alternating repeat pairs and median/range method. Stock is Fedora Honeykrisp 26.1.3-1.fc44; the exact reset-only head is 3d00b6d8. The fixed M1 Pro benchmark binary SHA256 is 32fa2a735f6bbec36034540cd680028b756f1ca522ff516440af66b103811ce3.

Device Workload Depth Stock for base Base Stock for reset Reset-only Reset/base Reset/stock
M1 Pro G13X Occlusion overhead 1 43.72 [28.6, 74.61] 56.84 [50.85, 60.44] 27.79 [-57.25, 46.52] 56.65 [53.29, 60.72] 0.9966 2.0385
M1 Pro G13X Occlusion total 1 360.5 [339.7, 393.1] 381.4 [377.1, 383.4] 346.3 [260.4, 363.4] 383.8 [378.2, 388.8] 1.0062 1.1083
M1 Pro G13X Occlusion total 16 78.58 [78.35, 78.84] 78.73 [78.32, 78.96] 78.67 [78.32, 79.26] 79.23 [78.47, 79.39] 1.0063 1.0071
M1 Pro G13X Idle wake, 20 ms 1 519.1 [513.1, 530] 527.1 [521.5, 546.4] 520.6 [518.1, 525.6] 532.5 [526.9, 534.1] 1.0103 1.0229
M1 Pro G13X Clear 1 218.7 [212.6, 227.7] 223.4 [218.9, 229.4] 217.1 [200.5, 225.6] 224.8 [217.8, 230.6] 1.0063 1.0355
M1 Pro G13X Fill, 1080p 1 1416 [1397, 1515] 1416 [1412, 1429] 1421 [925.9, 1506] 1442 [1397, 1479] 1.0177 1.0147
M1 Pro G13X Empty submit 32 1.511 [1.493, 1.54] 1.486 [1.484, 1.488] 1.522 [1.493, 1.641] 1.399 [1.392, 1.407] 0.9413 0.9192
M1 Pro G13X Empty submit 64 1.565 [1.539, 1.585] 1.488 [1.485, 1.493] 1.602 [1.545, 1.657] 1.402 [1.397, 1.409] 0.9422 0.8753

Reset-only reduces empty-submit medians by about 5.8% versus base. It does not reproduce the combined head's G13X query-overhead improvement: base/reset d1 overhead medians are essentially unchanged. Stock short-window results are highly variable, including a negative derived query-overhead repeat and a much faster fill repeat; the ranges and all raw values are retained. Ratios against those stock medians are descriptive and do not establish precise regression sizes. These measurements do not establish full M1 Pro parity.

M1 Pro raw repeat medians and underlying benchmark sample arrays (microseconds)
[{"device":"M1 Pro G13X","test":"query","depth":1,"metric":"overhead_us","repeat_medians_us":{"stock-for-base":[29.8846,28.5984,74.6104,57.5537],"base":[60.4382,60.0203,50.8476,53.6586],"stock-for-reset":[27.3828,-57.2453,46.5153,28.1942],"reset":[54.3806,53.2924,60.7185,58.9129]},"samples_us":{"stock-for-base":[null,null,null,null],"base":[null,null,null,null],"stock-for-reset":[null,null,null,null],"reset":[null,null,null,null]}},{"device":"M1 Pro G13X","test":"query","depth":1,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[339.6739,346.2604,393.0818,374.8126],"base":[383.4356,383.4356,379.3627,377.0739],"stock-for-reset":[346.2604,260.4167,363.3721,346.2604],"reset":[378.2148,380.5175,388.8025,386.9969]},"samples_us":{"stock-for-base":[[338.7534,352.6093,339.213,340.5995,339.6739],[349.6503,349.6503,339.6739,346.2604,345.3039],[388.1988,388.8025,393.0818,399.361,393.0818],[386.9969,377.6435,335.5705,342.9355,374.8126]],"base":[[377.0739,383.4356,376.506,387.5969,390.0156],[374.2515,379.3627,383.4356,385.8025,388.8025],[374.8126,378.2148,379.3627,386.9969,388.1988],[374.8126,375.9398,411.1842,407.1661,377.0739]],"stock-for-reset":[[329.8153,348.1894,346.2604,340.5995,353.1073],[260.4167,260.4167,255.6237,265.9574,275.634],[339.6739,342.4658,368.7316,381.6794,363.3721],[351.6174,334.2246,338.295,355.6188,346.2604]],"reset":[[326.3708,341.5301,378.2148,396.1965,398.7241],[378.2148,385.8025,379.3627,380.5175,381.0976],[390.0156,386.3988,381.0976,388.8025,389.4081],[390.0156,386.9969,380.5175,396.8254,378.2148]]}},{"device":"M1 Pro G13X","test":"query","depth":16,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[78.4191,78.3453,78.7402,78.8395],"base":[78.3208,78.7154,78.964,78.7402],"stock-for-reset":[79.2644,78.3208,78.9889,78.3453],"reset":[79.3903,79.1891,79.2644,78.4683]},"samples_us":{"stock-for-base":[[78.2963,78.4191,78.0031,78.5669,78.8644],[78.6164,78.3453,78.2473,78.1006,78.964],[79.0889,78.7402,78.1983,79.3147,78.3208],[79.4913,78.8395,78.765,79.8467,78.765]],"base":[[77.9059,78.3208,77.6156,79.0139,78.7402],[78.3699,78.7154,79.2142,79.3399,78.5669],[78.4191,78.6411,78.964,79.0139,78.964],[78.7154,78.7402,79.466,79.3399,78.125]],"stock-for-reset":[[79.466,79.2644,79.2644,79.0139,79.0639],[78.2228,78.8892,78.4683,78.1739,78.3208],[79.5925,78.9889,79.164,78.8892,78.1494],[78.3453,78.5423,78.2718,78.3699,78.1739]],"reset":[[80.36,79.0139,79.4155,79.139,79.3903],[79.3147,79.1891,78.964,79.2644,79.0389],[79.5165,79.2644,79.0389,79.2393,79.5672],[79.8977,79.0139,78.4683,78.0275,78.4437]]}},{"device":"M1 Pro G13X","test":"idle.wake","depth":1,"metric":"wake_latency_us","repeat_medians_us":{"stock-for-base":[530.0415,521.0845,517.062,513.082],"base":[525.6065,521.4995,546.3725,528.662],"stock-for-reset":[519.832,525.6235,521.3735,518.082],"reset":[531.6025,534.061,533.4775,526.936]},"samples_us":{"stock-for-base":[null,null,null,null],"base":[null,null,null,null],"stock-for-reset":[null,null,null,null],"reset":[null,null,null,null]}},{"device":"M1 Pro G13X","test":"render.light","depth":1,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[218.5315,227.6867,218.9142,212.585],"base":[223.6136,229.3578,223.2143,218.9142],"stock-for-reset":[217.77,200.4812,216.4502,225.6318],"reset":[225.8356,230.6273,223.8138,217.77]},"samples_us":{"stock-for-base":[[218.5315,213.858,210.615,223.4138,238.3222],[211.3271,224.0143,227.6867,232.9916,227.8943],[226.0398,212.585,219.106,216.0761,218.9142],[216.0761,211.5059,212.585,218.5315,212.0441]],"base":[[223.6136,216.6378,238.7775,226.4493,213.858],[225.0225,223.6136,232.342,234.3018,229.3578],[225.6318,220.4586,217.3913,223.2143,241.3127],[218.9142,212.766,219.2982,220.0704,216.263]],"stock-for-reset":[[215.8895,207.9867,219.6837,222.0249,217.77],[222.618,220.8481,200.4812,162.8664,168.0108],[225.6318,218.7227,216.4502,212.766,214.0411],[222.8164,211.1486,225.6318,232.342,225.8356]],"reset":[[227.4795,217.3913,225.8356,217.2024,228.7283],[231.696,230.6273,223.6136,220.2643,231.9109],[220.4586,214.7766,224.4165,227.0663,223.8138],[217.77,216.8257,219.6837,223.6136,212.4044]]}},{"device":"M1 Pro G13X","test":"render.heavy","depth":1,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[1515.1515,1412.4294,1396.648,1420.4545],"base":[1428.5714,1420.4545,1412.4294,1412.4294],"stock-for-reset":[1404.4944,925.9259,1436.7816,1506.0241],"reset":[1479.2899,1412.4294,1470.5882,1396.648]},"samples_us":{"stock-for-base":[[1488.0952,1515.1515,1524.3902,1497.006,1515.1515],[1381.2155,1412.4294,1497.006,1479.2899,1404.4944],[1381.2155,1404.4944,1412.4294,1366.1202,1396.648],[1453.4884,1381.2155,1436.7816,1420.4545,1344.086]],"base":[[1404.4944,1428.5714,1445.0867,1428.5714,1396.648],[1420.4545,1404.4944,1445.0867,1453.4884,1420.4545],[1445.0867,1420.4545,1396.648,1404.4944,1412.4294],[1404.4944,1412.4294,1428.5714,1428.5714,1396.648]],"stock-for-reset":[[1404.4944,1396.648,1515.1515,1428.5714,1404.4944],[925.9259,1033.0579,1101.3216,915.7509,762.1951],[1445.0867,1461.9883,1404.4944,1428.5714,1436.7816],[1515.1515,1506.0241,1506.0241,1470.5882,1488.0952]],"reset":[[1479.2899,1461.9883,1524.3902,1506.0241,1470.5882],[1396.648,1453.4884,1412.4294,1420.4545,1146.789],[1428.5714,1470.5882,1461.9883,1515.1515,1470.5882],[1396.648,1428.5714,1445.0867,1366.1202,1077.5862]]}},{"device":"M1 Pro G13X","test":"submit.empty","depth":32,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[1.4926,1.5024,1.5205,1.5403],"base":[1.4838,1.4857,1.4872,1.4882],"stock-for-reset":[1.5384,1.6407,1.5059,1.4929],"reset":[1.404,1.4072,1.3943,1.3924]},"samples_us":{"stock-for-base":[[1.4942,1.5056,1.488,1.4925,1.4926],[1.5039,1.498,1.5219,1.5024,1.502],[1.5259,1.5205,1.5206,1.5205,1.5164],[1.5423,1.5391,1.5403,1.5357,1.5546]],"base":[[1.4838,1.4843,1.4814,1.483,1.4855],[1.4852,1.4857,1.4817,1.4883,1.4875],[1.4945,1.4871,1.485,1.4872,1.4907],[1.4867,1.4875,1.4882,1.4948,1.4894]],"stock-for-reset":[[1.5283,1.5287,1.5384,1.5399,1.544],[1.6407,1.6342,1.6519,1.641,1.6403],[1.502,1.5156,1.5059,1.5085,1.5052],[1.492,1.4929,1.4932,1.4933,1.4819]],"reset":[[1.4029,1.404,1.4045,1.4027,1.4044],[1.4067,1.4053,1.4072,1.4092,1.4076],[1.3842,1.3943,1.3952,1.3933,1.3966],[1.3918,1.3924,1.3906,1.3976,1.3931]]}},{"device":"M1 Pro G13X","test":"submit.empty","depth":64,"metric":"us_per_submit","repeat_medians_us":{"stock-for-base":[1.5854,1.5853,1.5392,1.5445],"base":[1.4847,1.4859,1.4933,1.4895],"stock-for-reset":[1.5454,1.6573,1.6066,1.5964],"reset":[1.4044,1.4087,1.3991,1.3973]},"samples_us":{"stock-for-base":[[1.5792,1.5854,1.5817,1.6046,1.5903],[1.5678,1.5814,1.5853,1.594,1.5921],[1.508,1.5338,1.5392,1.5616,1.5395],[1.536,1.5426,1.5445,1.5567,1.5685]],"base":[[1.4816,1.4829,1.4847,1.4897,1.4871],[1.4827,1.4876,1.4832,1.4895,1.4859],[1.4868,1.4878,1.4954,1.4976,1.4933],[1.4859,1.4887,1.4895,1.4934,1.4904]],"stock-for-reset":[[1.5229,1.5454,1.5454,1.563,1.5361],[1.6584,1.6573,1.6545,1.6541,1.6593],[1.5797,1.6066,1.6045,1.6202,1.6122],[1.5762,1.5964,1.5951,1.603,1.6241]],"reset":[[1.4019,1.4067,1.4089,1.404,1.4044],[1.4087,1.4078,1.4119,1.4075,1.4097],[1.3952,1.3991,1.4061,1.3993,1.3987],[1.3938,1.4054,1.3989,1.3972,1.3973]]}}]

Known M1 Pro firmware wedge

The first reset-only CTS attempt stopped in chunk 122 on boot 33d37add-35f4-40c6-810b-2cecaecf2818, kernel 7.3.0-aurora-73w, after the paired benchmarks and completed chunks 0–121. This is the pre-existing M1 Pro firmware issue tracked as t-3018, which also occurs on unchanged base after similar accumulated load. The failed attempt remains part of the qualification record.

Counts across the preserved attempt: 24 scheduler-teardown lines, 1 firmware-crash message, 7 GPU-crashed messages and 0 context-invalidation-failure messages. The failed chunk itself contains the firmware-crash message and 7 GPU-crashed messages. Failed-chunk results include 2,874 Missing, 3 Fail and 7 Crash statuses. Those results are preserved and excluded from successful coverage; chunks 122–124 were repeated successfully on fresh boot daed6f9c-7b4c-42ac-af14-ae3ab0663ca0, with the explicit segments compared against base. No crash dump is published.

M1 Pro complete case comparison

Coverage is exactly chunks 0–121 (610,000 names) from the original boot plus chunks 122–124 (14,218 names) from the fresh boot. Every selected chunk completed and its case list matches the frozen list. Source/library, kit, jobs, grouping and 7.3.0-aurora-73w kernel identities match the preserved complete fresh-base run. There are zero new bad results and zero new skips over all 624,218 exact names. The sole changed status is the base implicit binary-semaphore copy Flake becoming Pass.

The three fresh-boot chunks have zero teardown, firmware-crash, GPU-crashed or context-invalidation-failure messages. This establishes complete case coverage across two boots, not uninterrupted sustained-run stability. The failed original chunk and its Missing/Fail/Crash results remain preserved and reported above as the known pre-existing t-3018 issue.

G17P focused query repeat

Twelve alternating base/reset pairs, rather than the four pairs in #68. Same median-of-independent-repeat-medians method, with repeat ranges in microseconds. Both builds use CPU affinity 1–4, inactive display, initial cooldown and 24 temperature-gated waits of 60–166 s between builds, process-local ICDs and the same 7.3.0-aurora-73w-dj423 boot. All 72 benchmark rows succeeded. Fixed gpubench binary SHA256: 32fa2a735f6bbec36034540cd680028b756f1ca522ff516440af66b103811ce3. No system security setting was changed; the Neo used its existing configuration, as approved by the operator.

Device Workload Depth Base Reset-only Reset/base
G17P Occlusion overhead 1 31.03 [29.64, 34.58] 32.87 [29.6, 41.06] 1.0594
G17P Occlusion total 1 245.6 [244.1, 255.6] 247.9 [244.4, 256.9] 1.0094
G17P Occlusion total 16 63.13 [62.3, 64.09] 63.02 [62.2, 63.97] 0.9982
G17P Compute light 1 157.9 [155.5, 162.8] 158.6 [155.8, 168.1] 1.0048

The earlier combined-head d1 total reading was +2.0%; this exact reset-only repeat gives +0.94% total and +5.94% derived overhead, with overlapping repeat ranges. Eight of twelve paired totals are higher. The median paired total change is +0.39%; an exploratory paired bootstrap interval is [-0.29%, +4.19%] (50,000 resamples of paired log-ratios, seed 3022). That interval includes both zero and +2.0%. The repeat therefore does not establish a causal regression or rule out the earlier concern. The d16 total is 0.18% lower and compute total 0.48% higher. A clean G17P non-regression result is not established.

G17P raw repeat medians and benchmark sample arrays (microseconds; repeat order 1–12)
[{"device":"G17P","test":"query","depth":1,"metric":"overhead_us","repeat_medians_us":{"base":[32.6302,30.9873,29.6354,30.6183,32.3186,29.9105,34.5804,34.3193,31.7875,29.7324,30.83,31.0687],"reset":[35.689,32.8739,29.6026,32.8111,31.0885,33.154,32.0669,32.8696,32.8136,36.2842,36.7966,41.0609]},"samples_us":{"base":[null,null,null,null,null,null,null,null,null,null,null,null],"reset":[null,null,null,null,null,null,null,null,null,null,null,null]}},{"device":"G17P","test":"query","depth":1,"metric":"us_per_submit","repeat_medians_us":{"base":[246.3054,245.5796,245.3386,245.5796,247.2799,246.5483,255.6237,255.3626,245.098,244.1406,244.1406,244.3793],"reset":[256.1475,245.821,244.3793,247.0356,246.7917,253.8071,247.7701,248.0159,245.5796,256.9373,256.6735,255.102]},"samples_us":{"base":[[246.5483,246.3054,246.5483,244.6184,244.1406],[245.5796,247.5248,245.5796,244.1406,246.063],[245.098,245.3386,249.7502,245.5796,244.858],[248.5089,245.821,245.5796,245.098,244.6184],[252.2704,247.5248,247.0356,246.3054,247.2799],[246.063,247.2799,246.5483,247.0356,246.5483],[255.3626,254.5825,256.9373,255.6237,257.732],[255.3626,256.1475,254.3235,256.6735,253.2928],[245.821,245.5796,244.858,245.098,242.9543],[244.3793,242.9543,244.1406,246.5483,244.1406],[244.3793,248.2622,243.1907,244.1406,243.6647],[244.3793,244.3793,243.4275,246.063,243.1907]],"reset":[[256.6735,256.1475,256.1475,257.4665,255.102],[245.821,243.9024,246.3054,243.4275,248.0159],[246.063,246.063,243.6647,244.3793,244.3793],[253.0364,247.7701,246.7917,247.0356,245.821],[248.0159,246.7917,247.5248,246.5483,246.5483],[253.2928,254.842,253.5497,254.5825,253.8071],[247.7701,247.7701,246.3054,245.5796,248.5089],[247.2799,248.2622,247.0356,248.0159,250.7523],[242.9543,247.0356,245.5796,247.7701,245.098],[255.8854,259.6054,256.9373,256.9373,257.9979],[258.7992,254.842,256.6735,254.3235,257.732],[246.5483,245.3386,255.102,259.0674,258.5315]]}},{"device":"G17P","test":"query","depth":16,"metric":"us_per_submit","repeat_medians_us":{"base":[63.0199,62.6096,63.5809,63.4196,63.0994,63.7918,64.0861,63.1632,62.6409,63.5001,62.5156,62.2975],"reset":[63.4679,63.0994,62.7668,63.5324,63.1792,62.3441,63.9713,63.6456,62.2045,62.9406,62.2045,62.6881]},"samples_us":{"base":[[63.0199,63.1313,62.9247,63.0199,63.1313],[62.5939,62.7983,62.7668,62.4844,62.6096],[63.5001,63.6618,63.3874,63.5809,64.0205],[63.4196,63.4196,63.597,63.5163,63.2591],[63.0676,63.0994,63.1154,63.2591,63.0517],[63.8081,63.7918,63.5486,63.9223,63.7592],[64.0861,64.2343,64.1026,63.955,63.9877],[63.0676,62.9247,63.1951,63.4196,63.1632],[62.6881,62.7825,62.6096,62.6409,62.5939],[63.4035,63.6294,63.5001,63.2271,63.7105],[62.3441,62.7038,62.751,62.3597,62.5156],[62.2045,62.5,62.2975,62.5313,62.282]],"reset":[[63.6132,63.4679,63.4035,63.4679,63.597],[63.0994,63.0676,62.9882,63.1632,63.3232],[62.9406,62.6723,62.7195,62.7983,62.7668],[63.597,63.9059,63.3874,63.5324,63.4518],[63.1792,63.2431,63.1473,63.743,62.8615],[62.3441,62.4064,62.282,62.3441,62.282],[63.9713,64.9013,63.857,63.9059,64.0533],[63.6456,63.6456,63.5647,63.5647,63.9386],[62.1891,62.22,62.1118,62.5782,62.2045],[62.8615,62.9406,63.2111,62.8615,63.0358],[62.2665,62.1891,62.1427,62.3752,62.2045],[62.3286,62.6881,62.7983,62.5156,62.9089]]}},{"device":"G17P","test":"compute.light","depth":1,"metric":"us_per_submit","repeat_medians_us":{"base":[161.9171,157.3317,156.3477,155.9576,156.1524,161.9171,155.8603,159.7444,162.7604,161.8123,155.4726,158.4284],"reset":[155.8603,155.9576,157.6293,155.9576,155.7632,161.2903,160.2564,162.4431,168.1237,156.0549,159.6424,161.8123]},"samples_us":{"base":[[156.0549,156.1524,161.9171,163.6126,164.7989],[163.3987,163.7197,157.3317,156.25,156.3477],[156.3477,155.9576,159.6424,161.8123,155.9576],[155.8603,159.7444,155.9576,155.9576,156.5435],[156.1524,156.5435,156.1524,156.0549,155.6663],[161.0825,161.9171,166.4447,162.022,161.0825],[155.8603,155.8603,155.8603,158.831,161.4987],[159.7444,163.0789,155.9576,156.0549,161.8123],[155.4726,162.7604,163.3987,163.292,155.8603],[156.3477,155.5694,161.8123,162.8664,162.9726],[160.7717,159.8465,155.376,155.376,155.4726],[155.9576,158.4284,162.3377,156.0549,162.2323]],"reset":[[156.7398,155.6663,156.0549,155.8603,155.7632],[162.1271,162.9726,155.9576,155.6663,155.5694],[156.1524,158.932,155.7632,157.6293,162.7604],[155.9576,155.5694,156.1524,155.6663,156.0549],[155.7632,157.5299,155.5694,155.7632,155.6663],[161.2903,167.8979,168.1237,159.4388,158.831],[162.4431,162.5488,160.2564,155.6663,155.7632],[161.6031,160.7717,162.4431,168.1237,168.3502],[161.3944,161.1863,168.1237,168.4636,168.8049],[155.9576,156.0549,156.0549,155.8603,156.3477],[156.5435,160.6684,159.6424,157.3317,163.5056],[161.8123,161.0825,168.5772,162.8664,161.0825]]}}]

AGX2 never populates the native graphics loader, launch, combined-sampler or spilled-render-target caches. Their storage is already zeroed when the command buffer is allocated. Leave that contiguous storage untouched on G13 and G14 resets, and reset all common state on either side. Native graphics resets keep the full clear.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The per-recording background/end-of-tile cache is only populated by the native graphics path. G13 and G14 otherwise invalidate sixteen entries spread over 13.5 KiB on every command-buffer reset, despite all of them remaining unused. Keep the existing invalidation on other generations.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Author

Superseded by #87, merged into aurora-wip in the 2026-10-28 consolidation.

@RyanTheTide
RyanTheTide deleted the perf/agx2-reset-only-20261025 branch October 28, 2026 04:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant