Skip to content

[BUG] codecov/patch reports far below its 80% target on a diff whose changed files carry dedicated lane tests #1647

Description

@DaubnerF

Problem

Extension tests run in five coverage lanes whose reports all reach Codecov in one CI step under a single flag. The default codecov/patch status is computed from that upload. The check can fail and misrepresent patch coverage to reviewers on a PR whose changed lines live in a single non-surviving coverage lane; PR #1505 is the only confirmed false red. The defect is in the repository's CI upload configuration on main, not in the PR where it surfaced, and the fix belongs on main.

Introduced on main at commit 99025b1fb (PR #1631, merged 2026-09-14T21:21:02Z, Zoo Code v3.82.1).

Evidence

  • The lanes are separate scripts: test:coverage:api through test:coverage:tree-sitter (src/package.json:450-454).
  • Every lane instruments all of src: the shared coverage config declares include: ["src/**/*.ts", "src/**/*.tsx", "eslint-rules/**/*.mjs"] (src/vitest.config.ts:22), while each lane selects only its own tests (src/vitest.core.config.ts:9). The same source file therefore appears in several uploaded reports.
  • Upload non-core coverage to Codecov went from uploading one merged src/coverage/unit/lcov.info to five overlapping extension lane lcovs (api, core, services, misc, tree-sitter) under flags: ${{ matrix.codecov-flag }} (.github/workflows/code-qa.yml:196-206). The matrix is per-OS, so the codecov-flag: ubuntu entry (:137) tags all five lanes.
  • The default patch status tolerates nothing below 80%: target: 80%, threshold: 0% (codecov.yml:17-18).
  • Observed on the checks of PR [Fix] Prevent unavailable tools from appearing in system prompts #1505: codecov/patch reported "15.45% of diff hit (target 80.00%)" for a diff whose changed files carry dedicated tests under src/core/prompts/tools/__tests__/. Its Codecov comment listed src/core/prompts/tools/effective-tool-policy.ts at 3.70% and src/core/prompts/tools/filter-tools-for-mode.ts at 0.00%.
  • Green 99.18% at head 5ae40b8fb (2026-09-14T20:07:56Z), culprit merged 2026-09-14T21:21:02Z, red 15.45% at a715cf07c (2026-09-15T10:06:48Z).
  • Analysis, not established: the reported figures are consistent with the sparsest lane's records for those files, pointing at how repeated per-file records merge inside one flag, not at a test gap. Per-lane lcov inspection at that head showed those lines covered by the lane that tests them. Codecov's server-side merge rule for repeated files under one flag is worth confirming before choosing an option.
  • Searching open and closed issues and discussions found no report; [ENHANCEMENT] Add per-area Turbo coverage tasks and a coverage merge task #117 anticipates a coverage-merge fix, open, unimplemented.

Desired behavior

The default patch status should count a changed line as covered when any lane executes it.

Proposed approach

Either give each lane its own flag (distinct flags plus matching flag_management entries), or merge the lane reports into one union tracefile before a single upload. The comment at .github/workflows/code-qa.yml:186-190 explains the layout avoids double-counting, so the webview flag must stay separate.

Acceptance criteria

  • A branch whose changed src lines are executed by one lane's tests passes codecov/patch with those lines counted as covered.
  • Genuinely uncovered new lines still fail the check.
  • The webview-ui, core-unit and core-integration flags keep working as configured in codecov.yml.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions