Skip to content

Improve GitHub CI/CD pipeline checkers (coverage, reliability, signal) #209

Description

@TheDarkSkyXD

Summary

Harden and expand StreamFusion’s GitHub Actions CI so PR/main checks are faster, less flaky, and catch more real regressions — especially Mobile/Android paths that currently burn signal on KVM flake while Verify already carries most correctness.

Current state (as of 2026-09-11)

Workflow: .github/workflows/build.yml (CI)

Job What it gates today
Verify workspace dependency review/audit/signatures, lockfile, lint/typecheck/test across core, integration-relay, mobile, desktop, worker; policy tests; evidence catalog; mobile Android bundle
Android development APK and API 30 install needs: verify; build debug APK; Enable KVM; android-emulator-runner API 30 install script
Build (windows / macos) needs: verify; desktop packaging builds

Also: Release workflow + Dependabot. Required checks / rulesets gate main.

Known pain (recent evidence)

  • Android KVM flake class on PR jobs (Enable KVM / emulator) while main can be FULL GREEN — false reds on otherwise MERGEABLE PRs (fix(superdev-mode): restore mixed SuperDev role model map #205, fix(logging): poll Chromium tailer against readPosition #208).
  • Verify is one monolithic job — any late failure (e.g. tests/policy/no-raw-timers) burns the whole matrix; Android/builds skip after Verify fail.
  • Limited path-aware / change-scoped feedback — Mobile-only PRs still pay full Desktop + Worker cost before Android.
  • Weak flake vs real classification in check output for humans and bots (SF Main CI Watcher / Babysitter).

Goals

  1. Stronger checkers — add or deepen automated gates for gaps we still miss in review.
  2. Higher signal — reduce flaky reds (especially Android KVM) without weakening real install/smoke coverage on main.
  3. Faster feedback — split/parallelize Verify where safe; cache; path filters where correctness allows.
  4. Clear contracts — document required checks, flake classes, and when a red is actionable vs mainline noise.

Proposed workstreams (implement incrementally; each should land as its own PR)

A. Reliability / flake

  • Isolate or harden Android KVM + API30 install (retry policy, clearer failure taxonomy, optional continue-on-error only where product-approved, or dedicated flake lane that is not a required PR check while remaining required on main).
  • Emit structured annotations (::error/job summaries) that name flake-class vs real for KVM/emulator failures.
  • Add a small contract test that the Android job’s failure modes are classified (script-level), so bots don’t expand KVM reds into product fixes.

B. Stronger / more checkers

  • Inventory gaps vs repo policy tests (e.g. timer allowlist, architecture boundaries) and promote any high-value local policy into CI early/fast jobs.
  • Consider dedicated fast policy/lint job (no full workspace install) that fails before heavy Verify.
  • Evaluate adding: CodeQL or Semgrep (security), workflow action pinning audit, mobile Detox/Maestro smoke only if stable on hosted runners, Desktop packaging smoke already covered by Build matrix.
  • Ensure change-gate tickets (Fix Chromium noise-filter tailer CI synchronization #201-class) have a clear “CI sync” checker path so main shippability stays owned.

C. Speed / structure

  • Split Verify into parallel shards (deps+audit | core+relay | mobile | desktop+worker) with a final ci-success aggregator for branch protection.
  • npm/ci caching strategy that respects --ignore-scripts + rebuild:dependencies contract.
  • Path filters for PR-only jobs where safe (never skip required security/audit on dependency changes).

D. Docs / bot contracts

  • Update CI docs for humans + agents: required checks list, flake classes, “do not expand KVM fail into product PR”.
  • Keep scripts/release-workflow.test.mjs (and peers) in sync with any workflow splits.

Out of scope

  • Emulator setup as project GitHub artifacts (CloudAgent-local only for feature visual verify).
  • Merging/landing without explicit go.
  • Replacing CodeRabbit / human review.

Acceptance

  • Written inventory of current required checks vs proposed required checks.
  • At least one reliability improvement for Android KVM flake class or an explicit product decision documented in-repo for PR vs main requiredness.
  • At least one new or deepened checker that catches a real class of regression earlier (policy/security/structure).
  • Verify (or its shards) still green on main; no silent weakening of dependency audit/signature gates.
  • Agent-facing note: how SF Main CI Watcher / Babysitter should read the new check names.

Notes for implementer

Prefer small PRs. Do not draft. Measure before/after on a recent PR that hit KVM flake (#205/#208 class). Coordinate with Mobile fan-out only if a check rename breaks bot watchers.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentTriaged and ready for an agent to implement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions