Add rule codes + configurable severity policy to skill-validator check

[aaronpowell]

Summary

skill-validator check currently emits mostly plain-text validation messages from multiple profilers. This makes it hard to reliably suppress specific checks, downgrade/upgrade severity, or enforce policy in CI by stable rule identity.

This issue proposes adding stable rule codes and an rc file for severity overrides, scoped to check for v1.

Problem

Today most findings are emitted as string messages (errors/warnings) without a stable identifier. That creates friction for:

• ignoring selected rules temporarily,
• treating some findings as warning vs error by project policy,
• producing deterministic automation/reporting in CI,
• evolving wording without breaking message-based filters.

Goals

• Introduce a stable, machine-usable rule identity for all check findings.
• Keep existing human-readable output quality.
• Allow per-repo policy overrides via config file.
• Preserve current behavior when no config is present.

Scope (v1)

• In scope: skill-validator check only.
• Out of scope (for now): evaluate findings and verdict policy wiring.

Proposal

1. Structured finding model

Introduce a shared finding shape (conceptual):

• Code (stable rule id)
• DefaultSeverity (error|warning|info)
• EffectiveSeverity (post-policy)
• Message
• Scope (skill|agent|plugin|global)
• Target (name/path)
• optional Location (file, line, column)

2. Rule code taxonomy

Use stable namespaces (examples):

• SVSK### skill rules
• SVAG### agent rules
• SVPL### plugin rules
• SVRF### reference scanner rules
• SVEX### external dependency advisories
• SVCM### command/config/runtime rules

Existing reference scanner codes (HTTP-NOT-HTTPS, etc.) should map cleanly (alias/transition strategy) to avoid abrupt breakage.

3. RC file for policy overrides

Add repository config file, e.g. .skill-validatorrc.json.

Example:

{
  "rules": {
    "SVSK100": "off",
    "SVRF002": "warning",
    "SVPL060": "error"
  },
  "overrides": [
    {
      "files": ["plugins/experimental/**"],
      "rules": {
        "SVRF001": "warning",
        "SVEX001": "off"
      }
    }
  ],
  "treatWarningsAsErrors": false,
  "unknownRuleBehavior": "warn"
}

Suggested severities:

• off, info, warning, error

4. Behavior

• Exit non-zero iff any effective error findings remain.
• off findings are suppressed from normal output.
• Unknown configured rule codes: behavior controlled by unknownRuleBehavior (warn|error|ignore).
• No config => behavior equivalent to today.

5. CLI additions (proposed)

• --rules-config <path> (explicit config file)
• --no-rules-config (disable discovery)

Implementation plan

1. Add finding model + default rule registry.
2. Convert all check emitters to coded findings:
   • SkillProfiler, AgentProfiler, PluginProfiler
   • duplicate skill name check
   • aggregate description cap check
   • reference scanner
   • external dependency checker
   • command-level validation findings
3. Add policy engine (resolve effective severity via defaults + rc).
4. Add config loader/discovery + schema validation.
5. Update output rendering + exit-code logic.
6. Add tests and docs.

Testing expectations

• Rule code stability tests.
• Severity override tests (error->warning, warning->error, off).
• Path override precedence tests.
• Unknown rule handling tests.
• Backward-compatibility tests for default behavior with no rc file.

Docs updates

• eng/skill-validator/src/README.md: explain rule codes, config file, and examples.
• Add migration guidance from message-string filtering to code-based policy.

Open questions for team discussion

1. Should we preserve existing scanner code strings as permanent aliases or only for a transition period?
2. What precedence model should overrides use (recommendation: later override wins)?
3. Should we print suppressed findings in --verbose mode for debugging policy files?
4. Should treatWarningsAsErrors be global-only in v1, or also allow per-path overrides?
5. Do we want a --list-rules command to dump available rules/default severities?

Acceptance criteria

• Every check finding includes a stable rule code.
• .skill-validatorrc.json can disable/remap rule severities.
• Exit code reflects effective severities.
• Existing behavior is preserved when no config is present.
• Tests/docs updated accordingly.

[github-actions]

🎯 Issue Triage

Summary: This issue proposes adding stable rule codes and a configurable severity policy (.skill-validatorrc.json) to skill-validator check. It falls under engineering infrastructure (eng/skill-validator/), targeting the check command's profilers to support per-repo CI policy overrides and machine-readable rule identities.

Type: Enhancement (new feature for skill-validator tooling)

Assigned owners: @ViktorHofer and @JanKrivanek — matching the /eng/ CODEOWNERS path.

Confidence: High — the issue is clearly scoped to eng/skill-validator/, the /eng/ CODEOWNERS path unambiguously maps to these two owners, and there is a directly related prior issue (#600) covering JSON output for the same tool.

Similar issues: #600 ("Support JSON output for skills validator") — related but not a duplicate; both propose structured output improvements to skill-validator, but this issue focuses on rule codes and policy overrides rather than JSON serialization.

Triage details

• Area mapping: No area-skill-validator label exists in this repo. The label infrastructure was used for the closely related issue Support JSON output for skills validator #600 ("Support JSON output for skills validator") which also touched eng/skill-validator/. Applied the same pattern here.
• CODEOWNERS match: /eng/ → @ViktorHofer @JanKrivanek``
• Confidence rationale: Single clear area (engineering tooling), well-specified proposal, direct CODEOWNERS match, no ambiguity about scope.
• Suggested next steps: Team should discuss the open questions listed in the issue (alias strategy for existing scanner codes, override precedence model, --list-rules command) before implementation starts.

Generated by Issue Triage for issue #747 · sonnet46 1.7M · ◷