🩺 CI Health

[claude]

🩺 CI Health Triage — 2026-07-08 07:52 UTC

Verdict: ⚠️ attention — one recurring genuine failure, everything else green.

Genuine failure: this workflow itself

• CI Health Triage (schedule) failed 3 nights in a row — 07-05 08:03, 07-06 08:36, 07-07 08:10 — all with the identical error, right after Claude Code installed successfully:
  Action failed with error: Environment variable validation failed: Either ANTHROPIC_API_KEY, CLAUDE_CODE_OAUTH_TOKEN, or workload identity federation ... is required
• Today's run (07-08 07:51, #28926641029) got past that same step and is producing this digest — so the credential is reachable, just not reliably on every scheduled run.
• Fix to check: whether CLAUDE_CODE_OAUTH_TOKEN/ANTHROPIC_API_KEY is scoped to an environment that schedule-triggered runs don't get, or there's a rotation/timing race specific to on: schedule. 3 fails / 4 attempts so far.

Everything else: healthy

• PR fix: bug-hunt round — iOS re-onboarding (HIGH), memory recall parity, markdown escaping, lifecycle hygiene #10 (fix/bug-hunt-2026-07-07): CI, Security, Claude Code Review (Mac) all green across all 3 push rounds (08:44→18:12 on 07-07).
• push → master (07-07 15:47): CI + Security green.
• Nightly DMG (07-08 05:04, run): succeeded in ~12s — by design, skips the actual build/sign/notarize because signing secrets (CERT etc.) aren't configured (::notice:: Signing secrets not set yet — skipping the nightly DMG). Identical skip pattern/duration on 07-06 and 07-07 — not a regression.
• No flaky same-job-fail-then-pass pairs observed in the window.
• No notably slow trends.

(Note: PR #9's "Claude Code Review (Mac)" failed 3x on 07-06 ~20:16–22:05, but that's >24h outside this window and PR #9 has since merged — mentioned only for continuity, not counted in the verdict above.)

— 2026-07-08

[kpmmmurphy]

Root cause of the "credential flake" — resolved, not a flake.

CLAUDE_CODE_OAUTH_TOKEN was added to repo secrets at 2026-07-07T08:31:17Z (gh secret list). The last scheduled failure ran at 07-07 08:10:28Z — 21 minutes before the secret existed. All five scheduled failures (07-03 → 07-07) predate the token; the first scheduled run after it (07-08 07:51) succeeded.

So there is no environment-scoping or rotation race on on: schedule — the timeline is clean. Expect green from tomorrow onward; if a scheduled run fails env-validation now that the secret exists, that would be a real finding.

— posted from the working session (Kev-approved), 2026-07-08

[claude]

🩺 CI Health — 2026-07-09 08:22 UTC

✅ Healthy — zero failures across the last 60 runs (2026-07-07T17:19Z → 2026-07-09T08:12Z, ~39h window).

• Failures: none. Every CI, Security, Claude Code Review (Mac), Code Quality, and PR-gate run in the window is success.
• Flaky signals: none observed (no fail→pass re-run pairs).
• Slow trends: none — job durations look consistent with recent norm; nothing stands out.
• Nightly DMG: ✅ ran and completed (run 28995982681) — preflight found no signing secrets, so build/sign/notarize/publish steps skipped as designed (expected on this fork).
• PRs fix: bug-hunt round — iOS re-onboarding (HIGH), memory recall parity, markdown escaping, lifecycle hygiene #10, fix(memory): bench decisions — recall floor 0.39, distiller-classified MemoryKind, PR #9 attribution #12, feat: Phase 17a egress gate · memory kinds complete · B5 title embeddings · mobile CI #13 all merged clean (CI/Security/Review all green per-PR).

No action needed.

— 2026-07-09

[claude]

✅ Healthy — 2026-07-10 08:12 UTC

• 90 runs in the last 24h (CI, Security, Code Quality, PR checks, Nightly DMG). Only 2 failures, both already fixed.
• Genuine failures (resolved): Nightly DMG (workflow_dispatch) failed twice — 14:05 UTC and 15:04 UTC on 2026-07-09 — both ##[error] No signing certificate "Mac Development" found ... team ID 76DJH43A4P, exit code 65. Fixed same day via fix/nightly-dmg-manual-signing (PR fix(release): nightly DMG — manual Developer ID signing on CI #18, merged ~15:58 UTC). Subsequent runs (15:58 UTC workflow_dispatch, and this morning's 05:20 UTC scheduled run) both succeeded.
• Nightly DMG (latest, scheduled): ✅ built successfully — run 29071105533.
• Flaky signals: none. The one cancelled run (CI on feat/memory-graph-title-composition, 29030644033) was a normal concurrency-cancel superseded by a newer push on the same PR, not a failure.
• Slow trends: nothing notable observed.

No action needed — signing fix is already merged and verified by two green Nightly DMG runs.

[claude]

✅ Healthy — 2026-07-11 07:37 UTC

• Last 24h window (2026-07-10 07:37 → 2026-07-11 07:36): zero failures across CI, Security, Code Quality, and PR-gate runs. Runs include PR feat(site): GEO content pages (vs-ollama, private-alternative, local-LLM guide, MCP, privacy) #21 (marketing/geo-content-pages) — CI/Security/Code Quality/PR-gate all green.
• Nightly DMG: ✅ ran on schedule (run 29140580691) — 9s, configured=false/skip path hit: "master unchanged since the last release ... skipping tonight's DMG." This is the new stale-nightly-skip gate from perf(ci): stop paying for builds that can't break #17 (d930290) working as designed, not a failure.
• Flaky signals: none — no fail→pass re-run pairs in the window.
• Slow trends: none — CI/Security run durations (~13-22 min) consistent with recent norm.
• Yesterday's signing-cert failures (Nightly DMG, 2026-07-09) remain resolved per prior digest — no recurrence.

No action needed.

— 2026-07-11

[claude]

🩺 CI Health — 2026-07-12 07:47 UTC

✅ Healthy — no failures in the last 24h (60 runs checked, back to 2026-07-11 13:33 UTC).

• Genuine failures: none.
• Flaky signals: none. One cancelled CI run (29155350443, push fix/netlify-ignore-cwd→master @ 14:01) — a follow-up push at 14:05 landed while it was mid-build and superseded it via the concurrency group (in-flight xcodebuild/swift test jobs cancelled, everything before that was green). Normal supersession, not a bug.
• Slow trends: none notable. Nightly DMG build step took ~15.5 min (05:04:56→05:20:20), in line with a notarized DMG build.
• Nightly DMG: ✅ built successfully (secrets present, ran the full build·sign·notarize·publish job at 05:04 UTC).

No action needed.

[claude]

✅ healthy — no failures in the last 24h

• 56 runs (CI, Security, Code Quality, Claude Code Review, PR checks) across master + PR feat(mlx): launch-time persona-prefix warm — ~1.9–3.3 s off the first turn #27/Prompt-hardening v2 code-side: self-query router + quarantined index segregation #28 branches, all success. Zero failure conclusions in the window.
• No flaky signal — nothing to point to since nothing failed/re-ran.
• CI job duration stable: 10–14 min across all 10 recent runs, no slowdown trend.
• Nightly DMG (2026-07-13 05:10 UTC): ✅ built successfully in 16 min (sign + notarize + publish all completed).
• 16 "Claude Code Assistant" runs show skipped — expected (issue_comment trigger gating on bot-mention conditions), not a failure mode.

No action needed.

—2026-07-13 08:09 UTC

[claude]

✅ CI Health: healthy — 2026-07-14

• 60 runs, last 24h: 0 failures across CI, Security, Claude Code Review (Mac), Code Quality, PR checks.
• Nightly DMG (29307377382, 2026-07-14 05:00 UTC): ✅ built successfully.
• Flaky signals: none observed.
• "Claude Code Assistant" skipped runs: expected — these fire on issue_comment and skip when the comment doesn't mention @claude, not a failure signal.
• Nothing to fix.

2026-07-14 UTC

[claude]

🩺 CI Health — 2026-07-15 07:42 UTC

✅ Healthy — one isolated infra blip, everything else green.

• Genuine failures: none. Of ~60 runs in the last 24h, only one failed:
  • Code Quality: Push on master → job Analyze (ruby) (CodeQL init step), run 29373080550, 2026-07-14 22:28 UTC.
  • Root cause: Error: read ECONNRESET while streaming the CodeQL bundle download from github/codeql-action releases — a transient network reset during Setup CodeQL tools, not a code/config problem.
  • Not yet retried (no subsequent push to master since). No fix needed beyond a re-run if it recurs; only actionable if it repeats.
• Flaky signals: none observed — no job seen failing then passing on re-run in this window.
• Slow trends: none apparent from run durations.
• Nightly DMG: ✅ built, signed, notarized, and published successfully (run 29390346706, full pipeline incl. DateVer release — not skipped, secrets present).

Signed off: 2026-07-15

[claude]

✅ Healthy — 2026-07-16 07:47 UTC

• 60/60 recent runs, 0 failures. No genuine breakage across CI, Security, Code Quality, PR checks, or CodeQL (Analyze c-cpp/rust/actions/ruby/python/js-ts) in the last 24h.
• No flakiness detected — zero same-job fail→pass re-run pairs.
• 3 cancelled runs, all routine concurrency-group supersession from rapid follow-up pushes on master and feat/artifact-house-style (each cancelled run has a same-workflow success/newer run seconds-to-minutes later) — not a failure signal.
• Durations nominal: Swift · iOS + visionOS shells (xcodebuild) ~7-12 min, Swift · Mac MVP (swift test) ~5-7 min, consistent across the 3 sampled master-push runs — no slowdown trend.
• Nightly DMG: built. Run 29472843262, Build · sign · notarize · publish job ran 05:02–05:23 UTC (~21 min) and completed successfully — a real build+notarize, not a no-secrets skip.

No action needed.

[claude]

✅ Healthy — 2026-07-17 07:45 UTC

• 0 genuine failures in the last 24h across 60 runs (CI, Security, Code Quality, PR checks, Nightly DMG).
• Cancelled runs (5x, all named "CI"): consistent with concurrency-group supersession from rapid consecutive pushes to master/PR branches (e.g. 3 pushes to master within ~1min at 21:53-21:54 UTC on 2026-07-16, each cancelling the prior in-flight run). Not a flake — expected behavior.
• No flaky signal: no job failed-then-passed on re-run.
• Nightly DMG: ✅ built successfully (run 29556338482, ~21min, Build·sign·notarize·publish job green).
• Timing: CI end-to-end ~14min (dominated by "Swift · iOS + visionOS shells (xcodebuild)" ~14min, in line with norm); Nightly DMG ~21min — both consistent with recent history, no slowdown trend.

No action needed.

[claude]

✅ Healthy — 2026-07-18 07:37 UTC

• 0 genuine failures in the last 24h (2026-07-17 07:36 to 2026-07-18 07:36 UTC), across ~22 runs: CI, Security, Code Quality, Claude Code Review, PR feat(avatar): Gecko's comeback + Colobus — two new voice-mode companions #53/feat(site): Meet the Companions — cast page with real on-device renders #54/feat(mlx): Ternary-Bonsai-27B family mapping + on-device trial evidence #55 checks, Nightly DMG.
• No flaky signal: no job failed-then-passed on re-run; nothing cancelled in this window (the previously-noted concurrency cancellations were about 34h ago, outside this window).
• Nightly DMG: built successfully (run 29631418487, 2026-07-18 04:57 UTC).
• Claude Code Assistant skipped runs (5x, issue_comment-triggered): expected, condition-gated, not failures.
• Timing: nothing standing out as slower than recent norm.

No action needed.

[claude]

CI Health -- 2026-07-19 07:47 UTC

Verdict: HEALTHY

• Window: last 60 runs, 2026-07-18 14:30 to 2026-07-19 07:47 UTC (~17h). 0 failures.
• 43 success, 15 skipped (all 'Claude Code Assistant', gated on issue_comment trigger conditions -- expected), 1 cancelled, 1 in-progress (this triage run).
• The 1 cancelled run ('CI' on feat/kokoro-mlx-backend, run 29651088250, 16:00 UTC) was superseded by a follow-up push 11 min later (run 29651465003) which passed -- normal supersede behavior, not flaky.
• Nightly DMG (run 29674270137, 05:03 UTC): success -- built/signed/notarized/published, no skipped steps.
• No flaky same-job fail-then-pass patterns and no notably slow jobs observed in this window.

No action needed.

[claude]

🩺 CI Health — 2026-07-20 08:06 UTC

✅ Healthy — no genuine failures in the last 60 runs (~16h window, 2026-07-19 16:24 → 2026-07-20 08:04).

• Conclusions: 41 success, 16 skipped, 2 cancelled, 0 failure.
• Cancelled (2): both CI on PR M1K3 hardening: conversation JSON import, speak() serialisation, voice effect-chain tuning #66 (claude/todo-implementation-358e3n), each superseded within minutes by a newer push on the same branch (normal concurrency-cancel behavior, not a flake).
• Skipped (16): all Claude Code Assistant on issue_comment events — expected conditional skips (comment didn't match the trigger), not errors.
• Nightly DMG: ✅ built, signed, notarized, and published — 2026.07.20 release, 31M DMG, run took ~21 min.
• CI job timing: steady ~10–11 min per run across the last 5 successful runs — no slowdown trend.
• Flaky signals: none observed (no same-job fail→pass pattern in this window).

No action needed.

[claude]

✅ healthy — 2026-07-21 07:52 UTC

• 60/60 runs in the last 24h succeeded or skipped as expected. No failures across CI, Security, Code Quality, Claude Code Review (Mac), or PR checks (feat(eval): the prompt-size instrument — count the real prompt with the real tokenizer #65, M1K3 hardening: conversation JSON import, speak() serialisation, voice effect-chain tuning #66, QA fixes & polish: sidebar nav + artifact fail-closed + visionOS icon #68).
• Nightly DMG: built clean both nights — Build · sign · notarize · publish completed with no skipped steps (secrets present), ~20–21 min runtime both runs (07-20 05:13→05:34, 07-21 05:04→05:24). No slowdown trend.
• Claude Code Assistant (issue_comment trigger) shows several skipped conclusions — expected behavior when a comment doesn't match the trigger condition, not a failure.
• No flaky signals observed (no job failed-then-passed on re-run).
• Nothing to fix.

— 2026-07-21

[claude]

⚠️ attention — 2 flaky-looking failures on an in-flight PR, master is clean

Window: 2026-07-21 07:51 → 2026-07-22 07:51 UTC (~104 runs)

• Master/push/scheduled workflows: all green. CI, Security, Code Quality, Scheduled, and PR checks on qa/kev-fixes/QA fixes & polish: sidebar nav + artifact fail-closed + visionOS icon #68 all succeeded.
• Nightly DMG: ✅ built successfully (29892714424, 2026-07-22 05:03 UTC).
• 2 CI failures, both on PR feat(security): pin and verify downloaded model weights #69 (feat/weight-integrity-pinning), both self-healed on the next push to the same PR — no fix commit visible between runs, and each failure is a different test, so this reads as flakiness rather than a real regression:
  • 20:44:03Z (29867025466, sha d16cda9): IntelligenceMCPToolsTests — "a fast brain returns its answer inline within the grace window" got the async-fallback "still working" message instead of the grounded answer. Timing/race against the grace-window deadline under CI load.
  • 21:40:27Z (29870890758, sha 2874e9e): ModelCacheIntegrityTests:152 — "a second tear replaces the quarantine instead of accumulating copies" — NSCocoaErrorDomain Code=4 moving a temp file into .m1k3-quarantine ("doesn't exist"). Not present in the runs immediately before (21:00) or after (22:05+).
  • Both suites passed clean in every surrounding run on the same PR. Worth a look if either recurs — the quarantine-folder one especially, since a real "folder doesn't exist" race could matter for the shipped feature (weight-integrity pinning) this PR is adding.
• No slow-job trend or other flakiness observed elsewhere.

2026-07-22 UTC