feat: validate end-to-end pack test flow#13
Open
JacobPEvans-personal wants to merge 6 commits into
Open
Conversation
Five focused changes so downstream packs can scaffold from this template with confidence the harness has teeth and the artifact ships clean: 1. Harness teeth meta-tests (tests/harness-teeth.test.ts) — prove every assertion guard (assertPartialMatch, assertRequiredFields, parseSimpleFilter) fires on its target failure mode. Pure unit; no Cribl required. Locks in the assertion contract against silent refactor regressions. 2. New-pack scaffolding runbook (docs/new-pack.md) — 6-step "I cloned the template, what do I edit" guide consolidating the org-admin token wrapper, required package.json fields, fixture authorship, and matrix override. 3. Tarball drift guard (tests/tarball-parity.test.ts) — pins the test-path whitelist (PACK_ROOT_ENTRIES in cribl-client.ts, now exported) to the release-path whitelist (INCLUDE in build-crbl.sh). Drift would silently ship a tarball CI never validated. 4. Eliminate silent skips in route filter tests — parseSimpleFilter returns a discriminated union; routes.test.ts fails loudly when it encounters a filter form the local matcher can't auto-evaluate. Authors must either simplify the filter or extend parse-filter.ts. 5. Multi-version Cribl matrix (.github/workflows/cribl-pack-test.yml) — cribl_versions input drives a parallel test matrix. Default: `latest` required + `3.5.4` best-effort (continue-on-error). Adding a version = one JSON entry. Per-pack override supported. Coverage matrix + version policy documented in docs/test-harness.md. Assisted-by: Claude <noreply@anthropic.com>
3.5.4 was the most recent previous-MAJOR patch — but in May 2026 that maps to a 2022-10-13 release that's nearly four years old. Nobody pinning Cribl in production runs anything close to that line, so the leg added zero signal and ~6 minutes of CI runtime per PR. Replace with `4.17.1` (released 2026-04-22) — the last patch of the previous MINOR. That's a real compatibility test: did the latest 4.18 patches change anything that breaks pipelines authored against 4.17? Also: 4.17.1 >= the template's minLogStreamVersion=4.17.0, so the leg actually passes by default rather than producing expected-failure noise the PR author has to mentally filter. Updated version policy in docs/test-harness.md accordingly — drop the "previous major" framing (only makes sense at major-version boundaries) and replace with "last patch of previous minor" + when-to-bump guidance. Assisted-by: Claude <noreply@anthropic.com>
Three corrections to the bundle:
1. Personal-setup leak in docs/new-pack.md: removed the
`security find-generic-password -s GH_PAT_ORG_ADMIN -a ai-cli-coder
-w ~/Library/Keychains/elevate-access.keychain-db` snippet. That's
one maintainer's local keychain wiring — has no business in a public
template that downstream packs scaffold from. Replaced with a
token-agnostic `GH_TOKEN=<org-admin-token>` placeholder + a sentence
explaining the required scope.
2. Matrix is now N/N-1/N-2 (latest + 4.17.1 + 4.16.1) instead of just
latest + previous-major (3.5.4 in the original, 4.17.1 alone after
the first fix). Three legs covers the realistic compatibility band
for packs anyone actually runs. Lowered the template's
minLogStreamVersion from 4.17.0 to 4.16.0 so all three legs pass by
default — downstream packs raise it if they need a newer feature.
3. Trimmed over-documentation that was DRY-violating the code:
- Deleted docs/new-pack.md entirely. The README's existing
Installation + After-scaffolding sections covered 90% of it; the
remaining gotcha (org-admin token) is now one sentence in the
README.
- Removed the "Coverage matrix" mega-table from test-harness.md.
Test names in harness-teeth.test.ts are the source of truth for
what each guard catches; transcribing them into a table makes
every test rename a doc edit.
- Shrunk the "Cribl version matrix" section to two paragraphs that
point at the workflow input default instead of transcribing it.
Self-documenting code stays self-documenting; docs cover only the
rationale and gotchas that aren't visible from a `cat`.
Assisted-by: Claude <noreply@anthropic.com>
Adds a repo-local renovate.json that extends dryvist's org default and defines a custom regex manager. The manager finds `"version":"X.Y.Z"` strings inside .github/workflows/cribl-pack-test.yml's matrix default and treats each as a docker dep on cribl/cribl, so Renovate opens a PR when a newer patch ships. Package rules constrain it to patch bumps only: - 4.17.1 → 4.17.2 ✓ (auto-merge, criblio is in trusted-org list) - 4.17.1 → 4.18.0 ✗ (minor/major bumps disabled) The `"latest"` entry isn't captured by the regex (no semver triple), so it stays untouched and continues to float at runtime. This keeps the user's "ALWAYS the last released version" principle enforced at the patch level. Minor-window shifts (when `latest` rolls over to a new minor) remain manual — they're a deliberate compat-band decision, not a routine bump. Assisted-by: Claude <noreply@anthropic.com>
Biome auto-bumped from 2.0.x to 2.4.13 via Renovate; the schema URL pinned to 2.3.6 then failed Biome's strict schema-version check. Bump the schema URL to match the installed CLI version, and apply Biome's suggested format for renovate.json to clear the lint failure on feat/validate-test-flow.
…ported
The previous regex used a lazy capture group ['"](.*?)['"] which let
the value span across quote boundaries — "a=='x' && b=='y'" then
matched as a simple filter with value "x' && b=='y" instead of being
flagged unsupported.
Use a back-referenced quote group ['"])([^'"]*)\2 so the opening and
closing quote must match and the value cannot contain a quote. Boolean
expressions, function calls, and other multi-clause filters now
correctly return {kind: 'unsupported'} as the contract documents.
Verified against the failing harness-teeth case plus the existing simple
and double-quoted shapes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five focused changes so downstream packs can scaffold from this template with confidence the harness has teeth and the artifact ships clean.
tests/harness-teeth.test.ts) — pure unit proof thatassertPartialMatch,assertRequiredFields, andparseSimpleFilterfire on the failure modes they claim to catch. No Cribl required.docs/new-pack.md) — 6-step "I cloned the template, what do I edit" guide consolidating org-admin-token usage, required manifest fields, fixture authorship, matrix override, and branch protection.tests/tarball-parity.test.ts) — pinsPACK_ROOT_ENTRIES(test-path whitelist, now exported) toINCLUDE=inscripts/build-crbl.sh(release-path whitelist). Drift would silently ship a tarball CI never validated.parseSimpleFilterreturns a discriminated union;routes.test.tsnowexpect.fail()s (rather than.skip()s) on filter forms the local matcher can't auto-evaluate..github/workflows/cribl-pack-test.yml) —cribl_versionsJSON-array input drives a parallel test matrix. Default:latest(required) +4.17.1(best-effort, last patch of previous minor as of 2026-05). Adding a version = one JSON entry. Per-pack override viawith: cribl_versions:.Coverage matrix + version policy documented in
docs/test-harness.md.Two commits — squash-merge to consolidate. The second (
fix(matrix):) corrects the initial commit's bad version pick (3.5.4 was years-old previous-major; 4.17.1 is the actually-useful previous-minor signal).Expected CI signal
Test pack pipelines (Cribl latest)— green (required)Test pack pipelines (Cribl 4.17.1)— green (best-effort; 4.17.1 ≥ template's minLogStreamVersion 4.17.0)continue-on-erroron the second leg is forward-looking: when a future Cribl patch breaks pipelines, that leg goes red and surfaces the regression without blocking merges.Test plan