Honor SkillSpector analysis completeness

[Tomauskasz]

Summary

• Validate SkillSpector 2.10 execution_successful and analysis_completeness fields when present.
• Report incomplete analysis before checking the recommendation/severity invariant.
• Preserve compatibility with older SkillSpector reports that omit the new completeness fields.

Root cause

SkillSpector deliberately upgrades a LOW report from SAFE to CAUTION when analysis is incomplete. SkillEvaluator ignored the authoritative completeness fields and therefore rejected a valid incomplete LOW/CAUTION report as a false recommendation/severity contradiction.

The validator now fails closed on unsuccessful, malformed, contradictory, or incomplete markers. A complete LOW/CAUTION report remains invalid.

Verification

• Full suite: 5482 passed, 17 skipped, 4 deselected
• Focused current, legacy, and incomplete report-contract cases: 3 passed
• Current and legacy clean-report fixtures share one parameterized test path
• make lint: passed
• make build: passed
• git diff --check: passed

Compatibility

Older reports that omit both execution_successful and analysis_completeness retain the existing validation path.

AI assistance disclosure

Codex inspected the SkillEvaluator and SkillSpector report contracts, reproduced the failure, drafted the implementation and tests, and ran the verification commands. I reviewed the root cause, fail-closed behavior, compatibility boundary, diff, and final pull-request text.

[Tomauskasz]

End-to-end follow-up:

• Installed this fix with SkillEvaluator PR fix(tier3): decode native Codex exec calls #111, the pinned Harbor provenance backport, and SkillSpector PR #451.
• Direct SkillSpector scan: LOW/SAFE, 100% coverage, complete analysis, zero ledger exceptions.
• Installed Tier 1 security validation passed and exited 0.
• One four-case paired Codex/Terra full run completed with 8 expected and 8 scored attempts, no execution errors, and a complete report.
• All three tiers passed; with-skill score was 0.9471 versus 0.6613 baseline.

The full source suite for this PR also passed: 5160 passed, 17 skipped, 4 deselected. Lint and build passed.

[rng1995]

The new contract checks still accept contradictory incomplete evidence.

[rng1995]

These three flags do not cover the full completeness contract. A report with is_complete=true, status=complete, and both execution flags true is accepted as SAFE even when coverage_percent=0, entirely_uninspected_files=1, or ledger_exceptions contains a fatal entry. Validate those authoritative fields and reject contradictions so an incomplete SkillSpector report cannot fail open; add regressions for the contradictory cases.

[Tomauskasz]

Addressed in 6fd3a5d and updated onto current main in merge commit 4ffb55c. A claimed-complete report now requires present authoritative details with numeric 100% coverage, zero integer partial/uninspected counts, and empty ledger-exception/limitation lists. Missing or contradictory details fail closed; reports that omit analysis_completeness entirely retain the legacy path.

Verification after merging current main: 1,053 validator tests passed; the full suite passed with 5,482 passed, 17 skipped, and 4 deselected; Ruff passed; source and wheel builds passed; git diff --check passed. The merge also incorporates main's Gitleaks workflow repair.

Fresh CI, DCO, and Security runs were created, but GitHub marked them action_required before running. The contributor account cannot approve fork workflows, so they are waiting for maintainer approval.

[rng1995]

@Tomauskasz Gentle ping when you have a chance: the remaining completeness-contract review thread is still waiting for an update or response. Thank you for your patience throughout the review and for your contribution.

[wuzhenhua24]

Independent reproduction — this matches the root cause described here exactly, and I bisected the SkillSpector side.

Minimal reproducer. A skill with no findings at all is enough; the only trigger is a / in a heading, which the reference resolver records as an unresolved local reference:

---
name: repro
description: Minimal reproducer for the SkillSpector completeness contract mismatch. Use when reproducing the reported validation failure.
---

# Repro

## Overview

Prose only. No executable content.

### Input/Output Separation

Keep inputs and outputs distinct.

Behaviour across SkillSpector versions (same skill, same SkillEvaluator commit, --no-llm):

SkillSpector	score	severity	recommendation	is_complete	skillevaluator security-scan
2.9.6	0	LOW	SAFE	false	PASS
2.10.0	0	LOW	CAUTION	false (status: partial)	INCOMPLETE
2.11.0	0	LOW	CAUTION	false (status: partial)	INCOMPLETE

Failure message on 2.10.0+:

skillspector JSON field 'risk_assessment.recommendation' does not match the risk severity;
security scan did not complete

Three things worth noting:

1. The skill has zero findings and score: 0, yet the entire security scan result is discarded. The rejection is not scoped to one questionable claim in the report.
2. 2.9.6 already reports is_complete: false for this skill — it just doesn't escalate the recommendation. So the behavioural change in 2.10.0 is the fail-closed escalation, not completeness detection. That supports reading analysis_completeness before the recommendation/severity invariant, as this PR does.
3. The trigger surface is wide. ledger_exceptions is a single reference_unresolved from the reference_resolution phase, raised by a slash inside a Markdown heading. Prose-heavy skills hit this readily — the first real skill where I saw it had two such exceptions, one from a heading and one from a table cell.

Environment: SkillEvaluator 3bfba44 (main, reports 0.2.1), SkillSpector installed via uv tool install, macOS / Python 3.13.

Until this lands, we are pinning SkillSpector to v2.9.6 downstream, with a regression test that fails on 2.10.0+ so we can tell when the pin is safe to drop.

[Tomauskasz]

Follow-up from the complete three-lane review is in bdd69c1.

The ten contradictory/missing completeness-detail cases now share one parameter matrix and one scanner/assertion body. Runtime behavior and case coverage are unchanged; the test removes 23 net lines of duplicated mechanics.

Verification:

• focused completeness and legacy contract set: 12 passed;
• full serial suite: 5,482 passed, 17 skipped, 4 deselected;
• repository Ruff check passed;
• source and wheel builds passed;
• git diff --check passed.

[Tomauskasz]

Final test-only polish is in 1f25501.

The partial-report regression now overrides only fields that differ from the shared fixture. Current and legacy clean-report coverage now shares one parameterized test path. Production behavior and contract coverage are unchanged; the test file is 20 lines smaller.

Final verification: 5,482 passed, 17 skipped, 4 deselected; Ruff passed; source and wheel builds passed; git diff --check passed. The independent follow-up review found no issue.

Fresh CI, DCO, and Security runs for this head are action_required and are waiting for maintainer approval of the fork workflows.