Report the test phase's sccache hit rate and warn on threshold breaches

.github/actions/ci-telemetry-finish/CHANGELOG.md

@@ -16,3 +16,11 @@ this file.
   The statistics come from the `sccache-report` file when present, otherwise
   from `sccache --show-stats`. New inputs `sccache-status` and
   `sccache-stats-file`.
+- Add the test phase's sccache figures (`sccache_test_hits`, `_misses`,
+  `_compile_requests`, `_hit_ratio`, `_problem`) as the difference between a
+  post-build `sccache-report` snapshot and the final statistics, and
+  `threshold_breaches`. New inputs `sccache-build-stats-file`,
+  `test-hit-ratio-min`, `test-min-compile-requests`, `min-free-disk-mib` and
+  `thresholds`. A breach on a push to the default branch prints a `::warning::`
+  and never fails the job. The fields are additive to schema 1 and null when no
+  snapshot is supplied.

.github/actions/ci-telemetry-finish/README.md

@@ -36,16 +36,36 @@ With sccache, wire the two `setup-rust` and `sccache-report` outputs in:
     sccache-status: ${{ steps.setup-rust.outputs.sccache-status }}
 ```
 
+To also measure the test phase, call `sccache-report` a second time right after
+the build step, writing to different files, and leave the final call after the
+tests as above:
+
+```yaml
+- run: cargo build --workspace --tests
+- uses: leynos/shared-actions/.github/actions/sccache-report@d26e53630893c942af44eb3db7efd4fcbe0c8093
+  with:
+    status: ${{ steps.setup-rust.outputs.sccache-status }}
+    stats-file: sccache-stats-build.json
+    text-file: sccache-stats-build.txt
+    summary: 'false'
+- run: cargo nextest run --workspace
+```
+
 Call finish last and under `if: always()`: a failed or cancelled job is the one
 whose resource use you most want to know. Only a time-out or a lost runner
 leaves no record.
 
 ## Inputs
 
-| Name               | Type   | Required? | Description                                                                                               | Default              |
-| ------------------ | ------ | --------- | --------------------------------------------------------------------------------------------------------- | -------------------- |
-| sccache-status     | string | no        | The `sccache-status` output of `setup-rust` (`started` or `fallback`). When empty the status is inferred. | `''`                 |
-| sccache-stats-file | string | no        | The JSON file `sccache-report` wrote. Read when it exists; otherwise finish asks sccache once.            | `sccache-stats.json` |
+| Name                      | Type   | Required? | Description                                                                                                                                | Default                    |
+| ------------------------- | ------ | --------- | ------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------- |
+| sccache-status            | string | no        | The `sccache-status` output of `setup-rust` (`started` or `fallback`). When empty the status is inferred.                                  | `''`                       |
+| sccache-stats-file        | string | no        | The JSON file `sccache-report` wrote. Read when it exists; otherwise finish asks sccache once.                                             | `sccache-stats.json`       |
+| sccache-build-stats-file  | string | no        | The JSON file the post-build `sccache-report` call wrote. Read when it exists; the test phase is the final statistics minus this snapshot. | `sccache-stats-build.json` |
+| test-hit-ratio-min        | string | no        | Warn when the test phase's hit ratio is below this fraction.                                                                               | `0.80`                     |
+| test-min-compile-requests | string | no        | The fewest test-phase compile requests worth judging.                                                                                      | `20`                       |
+| min-free-disk-mib         | string | no        | Warn when `min_free_disk_mib` is below this many MiB.                                                                                      | `3072`                     |
+| thresholds                | string | no        | Where thresholds apply: `main-push` (a push to the default branch), `always` or `never`.                                                   | `main-push`                |
 
 ## Outputs
 
@@ -104,6 +124,24 @@ but cannot be read or parsed is a problem to report, not a reason to ask again.
 Statistics that lack a field sccache has printed since 0.12, or give it the
 wrong type, are `unreadable`, not zero.
 
+### Test-phase and threshold fields
+
+Additive to schema 1, always present, and null whenever they cannot be
+established, so an existing reader sees no change.
+
+| Field                                                                       | Meaning                                                                                                                                                               |
+| --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `sccache_test_hits`, `sccache_test_misses`, `sccache_test_compile_requests` | The final counters minus the post-build snapshot's. Null without a `started` server or without the snapshot file.                                                     |
+| `sccache_test_hit_ratio`                                                    | Test-phase hits over hits plus misses; null with no attempts.                                                                                                         |
+| `sccache_test_problem`                                                      | Null, or why the snapshot could not be used: `counter-reset` (the server restarted between the calls), `snapshot-invalid`, `snapshot-unreadable` or `snapshot-shape`. |
+| `threshold_breaches`                                                        | Null when thresholds did not apply. Otherwise `none`, or a comma-separated subset of `test_hit_ratio` and `min_free_disk`.                                            |
+
+A breach prints a `::warning title=ci-telemetry-threshold::` annotation. It
+never fails the job. The defaults come from measured records: warm Linux jobs on
+`main` show whole-job hit ratios of 0.998 to 1.0 and a smallest healthy free
+disk of about 5.4 GiB, while the failure that motivated the disk threshold left
+about 2 GiB.
+
 How CPU time is counted, from `/proc/stat` deltas between samples:
 
 - busy is `user + nice + system + irq + softirq`;

.github/actions/ci-telemetry-finish/action.yml

@@ -19,6 +19,42 @@ inputs:
       otherwise finish asks sccache for its statistics once.
     required: false
     default: sccache-stats.json
+  sccache-build-stats-file:
+    description: >-
+      The JSON file a second `sccache-report` call wrote after the build
+      (`stats-file: sccache-stats-build.json`). When it exists, the test
+      phase's hits, misses and hit ratio are the final statistics minus this
+      snapshot. When it is absent the `sccache_test_*` fields are null.
+    required: false
+    default: sccache-stats-build.json
+  test-hit-ratio-min:
+    description: >-
+      Warn when the test phase's sccache hit ratio is below this fraction.
+      Judged wherever the `thresholds` input applies (by default a push to the
+      default branch, which is warm by construction), and only when the phase
+      made at least `test-min-compile-requests` compile requests.
+    required: false
+    default: '0.80'
+  test-min-compile-requests:
+    description: >-
+      The fewest test-phase compile requests worth judging; a smaller phase is
+      noise, not a cold cache.
+    required: false
+    default: '20'
+  min-free-disk-mib:
+    description: >-
+      Warn when the job's least free disk (`min_free_disk_mib`) falls below
+      this many MiB. Judged where `thresholds` applies.
+    required: false
+    default: '3072'
+  thresholds:
+    description: >-
+      Where the thresholds apply: `main-push` (a push to the default branch,
+      the default), `always`, or `never`. A breach is a `::warning::`
+      annotation and the record's `threshold_breaches` field. It never fails
+      the job.
+    required: false
+    default: main-push
 
 outputs:
   record-path:
@@ -41,6 +77,12 @@ runs:
         CT_FINISH: ${{ github.action_path }}/scripts/finish.py
         CT_SCCACHE_STATUS: ${{ inputs.sccache-status }}
         CT_SCCACHE_STATS_FILE: ${{ inputs.sccache-stats-file }}
+        CT_SCCACHE_BUILD_STATS_FILE: ${{ inputs.sccache-build-stats-file }}
+        CT_TEST_HIT_RATIO_MIN: ${{ inputs.test-hit-ratio-min }}
+        CT_TEST_MIN_COMPILE_REQUESTS: ${{ inputs.test-min-compile-requests }}
+        CT_MIN_FREE_DISK_MIB: ${{ inputs.min-free-disk-mib }}
+        CT_THRESHOLDS_MODE: ${{ inputs.thresholds }}
+        CT_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
       run: |
         set -euo pipefail
         py="$(command -v python3 || command -v python || true)"

.github/actions/ci-telemetry-finish/scripts/finish.py

@@ -24,6 +24,8 @@
 
 import sccache_stats
 import summarize
+import test_phase
+import threshold_settings
 
 STOP_WAIT_SECONDS = 15
 CPU_FIELDS = 10
@@ -231,7 +233,15 @@ def finish(directory: Path, env: typ.Mapping[str, str]) -> dict[str, typ.Any]:
         env.get("CT_SCCACHE_STATUS", ""),
         env.get("CT_SCCACHE_STATS_FILE", ""),
     )
-    return {**_measurements(directory, env), **sccache}
+    record = {**_measurements(directory, env), **sccache}
+    build = sccache_stats.read_stats_file(env.get("CT_SCCACHE_BUILD_STATS_FILE", ""))
+    record.update(
+        test_phase.collect(sccache, threshold_settings.snapshot_from_outcome(build))
+    )
+    thresholds = threshold_settings.thresholds_from_env(env)
+    job = threshold_settings.job_from_env(env)
+    record.update(test_phase.judge(record, thresholds, job))
+    return record
 
 
 def degraded_record(env: typ.Mapping[str, str], exc: Exception) -> dict[str, typ.Any]:
@@ -246,14 +256,17 @@ def degraded_record(env: typ.Mapping[str, str], exc: Exception) -> dict[str, typ
     }
 
 
-def print_marker(record: typ.Mapping[str, typ.Any]) -> None:
+def print_marker(record: typ.Mapping[str, typ.Any], env: typ.Mapping[str, str]) -> None:
     """Print the one marker line."""
     if record["sampler"] == "not-started":
         print(
             "::warning title=ci-telemetry::ci-telemetry-start did not run in "
             "this job; the record carries no measurements"
         )
     print(summarize.marker_line(record))
+    thresholds = threshold_settings.thresholds_from_env(env)
+    for line in threshold_settings.commands(test_phase.messages(record, thresholds)):
+        print(line)
 
 
 def print_summary(record: typ.Mapping[str, typ.Any]) -> None:
@@ -305,7 +318,7 @@ def emit(
     The marker goes first and cannot fail; each later step reports its own
     failure and the rest still run.
     """
-    print_marker(record)
+    print_marker(record, env)
     print_summary(record)
     path = directory / "record.json"
     for label, step in (