ci: run the driver's native suites as their own group - #591
Merged
Merged
Conversation
native-rest was the longest test job: its suites took 275 to 391 s over the six main runs 36126756550 to 36132815983, against 188 to 261 s for the slowest simulation group. It ran the native crate's non-simulation suites and the driver's native_* suites as two nextest invocations one after the other, each at two threads. The two threads were a packing choice (b01ae55). The one suite that can't share the server, native_connections_live, already runs in emulator at one thread, and the other native-rest suites read only their own database and query ids. The driver's native_* suites become the driver-native group, run from the workspace archive at TEST_THREADS like a simulation group. The native crate's non-simulation suites join native-sim-e, whose filter now takes every native crate suite the lettered groups don't name. They run from the native archive, which that group already downloads. Modelled from each test's worst time over those six runs, longest test first over four slots: driver-native 162 s (longest test native_demo_live's sim run, 162 s), native-sim-e 230 s. The same model reproduces the observed suites step of every group in those runs within 4 to 17%. The test job count stays the same.
2 of 3 tasks
native_stream_live's pacing test asserts that the median send-to-visible latency over 100 tics stays under 5 ms (P50_LIMIT). In driver-native it ran beside three resident sessions on a four-core runner, and on run 36137795545 it measured a p50 of 5.61 ms (p99 21.78 ms) and failed. It passed in native-rest, where it had at most one neighbour. The limit stays. The suite moves to emulator, which runs one test at a time for the same reason the connection suite runs there: its check is about the server and the runner, so a neighbour changes what it measures. No other suite in driver-native or the simulation groups asserts a latency or a duration. native_session_live measures frame waits and prints them without asserting on them, sim_cost_live asserts only that a statement takes at least as long as its own analysis, and the remaining time limits are harness timeouts of 10 s or more.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes, and why
native-restis the longest test job. Over the six main runs 36126756550 to 36132815983, its suites step took 275 to 391 s, while the slowest simulation group took 188 to 261 s. It ran two nextest invocations one after the other, each at--test-threads 2: the native crate's non-simulation suites, then the driver'snative_*suites.The two threads were a packing choice, made when
native-restabsorbed driver-native (b01ae55). Two cases matter for moving to more threads:native_connections_livecounts the server's connections, so it can't run next to another test. It already runs inemulatorat one thread.session_live'ssystem.query_logcount filters on its database) or their own query id (native_session_live'sKILL QUERY WHERE query_id = ...).This PR makes two changes:
native_*suites become thedriver-nativegroup. It runs from the workspace archive atTEST_THREADS, like a simulation group.native-sim-e, which already downloads the native archive. Its filter becomespackage(clickdoom-native) and not (<lettered groups>), so it catches every native suite the other groups don't name.native_stream_livegoes toemulator, notdriver-native(second commit, 6a930eb). Its pacing test asserts a median send-to-visible latency under 5 ms (P50_LIMIT). On this PR's first CI run (36137795545) it ran beside three resident sessions indriver-nativeand measured p50 5.61 ms (p99 21.78 ms, max 29.24 ms), then failed. The limit stays.emulatorruns one test at a time, and the connection suite runs there for the same reason.No other suite in
driver-nativeor the simulation groups asserts a latency or a duration:native_session_livemeasures frame waits and prints them without asserting on them.sim_cost_liveasserts only that a statement takes at least as long as its own analysis.FRAME_TIMEOUT,VISIBLE_TIMEOUT,FIRST_ROW_TIMEOUT,TIC_TIMEOUT).The test job count stays the same, and
ci-passed'sneedsdoesn't change.group_coverage.rsfinds the catch-all by its new filter text.Evidence
Per-test times come from the nextest PASS lines of the six main runs. Each group is modelled as longest-test-first scheduling over four slots:
The same model against each group's observed suites step in those runs (6 runs x 8 groups) gives observed/model between 0.96 and 1.17, mostly 1.05 to 1.10. nextest's own startup accounts for the steady part of that gap.
native-sim-e's 411 tests are its 28 earlier ones plus the 383 native testsnative-restran. Of the 19 driver testsnative-restran, 15 are indriver-nativeandnative_stream_live's 4 are inemulator.What this PR's CI run should show
test (driver-native): the first run's suites step took 217.5 s with the stream suite in it, which is slower than the model's 151 to 162 s at four sessions at once. Expect about the same without that suite (about 3 s of tests), well undernative-rest's 275 to 391 s.test (emulator): 3 to 6 s longer than before, withnative_stream_livepassing alone.test (native-sim-e): about 230 to 250 s, up from 136 to 188 s.test-groups: 960 of 960 selected,emulator467 tests,driver-native15 tests.Invariants
None. Test grouping only.
Spec impact
Checks
make gates: not run. The checks above ran locallyWritten mostly by Claude Opus 5.5.