Skip to content

ci: shard cmd/server Go tests and split the Go job (follow-up to #434/#436) #445

Closed

Description

@dborup

Problem

After #436 sharded Playwright E2E (3 shards of about 10 min each), the ✅ Go Build & Test job is the CI critical path at about 22 minutes. Measured step durations on a recent green master run:

Step Duration
Build and test Go server (with coverage) 1152 s (~19 min)
Build and test Go ingestor (with coverage) 162 s
Run JS unit tests (test-all.sh) 98 s
Non-root container start path, supervisord stubbed 31 s
anomaly library with -race 21 s
everything else < 15 s each

cmd/server is one large package, so go test ./... parallelism across packages does not help. Nearly all the time is that single package.

Proposal

  1. Measure first. Run the cmd/server tests with go test -json and record per-test durations from at least 3 green runs. Put the top-N table in the PR. Look for outliers (time.Sleep, real timeouts, large fixtures, -race overhead) that can be fixed outright.
  2. Shard cmd/server tests into N parallel jobs (start with 3–4), balanced by measured duration with a longest-first assignment. Each job runs go test -run '^(TestA|TestB|…)$' from a checked-in manifest, like scripts/e2e-shards.json from ci: measure and shard Playwright E2E suites (#434) #436.
  3. Move the other steps into their own jobs, running in parallel: the ingestor tests, test-all.sh/JS unit tests and the small library checks.
  4. Coverage: each shard writes its own coverprofile. A small merge job combines them so the coverage report and badges stay correct.
  5. Fail-closed gate: 🏗️ Build & Publish Docker Image must require every Go shard and every split job, with the same semantics as ci: parallelize independent checks with fail-closed gate #383 and ci: measure and shard Playwright E2E suites (#434) #436 (failure, skipped, cancelled or empty count as failure).
  6. Coverage guard: a check that every Test… function in cmd/server is assigned to exactly one shard, so new tests cannot silently drop out (mirroring ci: measure and shard Playwright E2E suites (#434) #436's E2E guard).
  7. Keep the fork guards (9 in deploy.yml, 1 in release-fast-path.yml). No publish or deploy from the fork. Do not touch squad-*.yml.

Acceptance

  • The per-test timing table and the shard manifest are in the PR.
  • PR-triggered runs show all shards green, and the Go critical path drops well below the ~19 min server step. Report before/after wall time and total runner minutes for at least two runs.
  • The unassigned-test guard and the fail-closed gate are covered by tests, with mutants shown red.
  • The coverage percentage is unchanged (within noise) versus before.

Activity

  1. added a commit that references this issue on Oct 14, 2026
  2. dborup commented on Oct 14, 2026

    @dborup
    OwnerAuthor

    Fixed by #446 (merged as 5025e17c36204382a6fb4839545329426af23211).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions