Skip to content

[BUG]: Malformed policy YAML bypasses the CLI error wrapper #128

Description

@rioyu123

What happened?

load_profile() and load_policy_file() document ValueError for malformed YAML, and the CLI converts FileNotFoundError/ValueError into a clean click.ClickException. However, both loaders call yaml.safe_load() without catching yaml.YAMLError.

As a result, malformed custom policy YAML raises a raw PyYAML parser/scanner exception and bypasses the CLI's intended error wrapper.

Reproduction steps

At main commit 3bfba44:

from pathlib import Path
from tempfile import TemporaryDirectory

from skillevaluator.validators.policy import load_policy_file

with TemporaryDirectory() as td:
    path = Path(td) / "broken-policy.yaml"
    path.write_text("severity_overrides: [", encoding="utf-8")
    load_policy_file(path)

Actual result:

yaml.parser.ParserError

Expected result:

  • the loader raises the documented ValueError;
  • CLI use of --policy reports a concise error containing the policy path;
  • validation does not continue with a fallback policy;
  • no raw PyYAML traceback is exposed.

A focused fix can wrap yaml.YAMLError consistently in the bundled-profile and custom-policy loaders.

SkillEvaluator version or commit

main@3bfba44e754be87073b2344233f9569b06509ce1

Environment

  • Windows 11
  • Python 3.13
  • source checkout with uv
  • no provider credentials or external services involved

Before submitting

  • I agree to follow this project's Code of Conduct.
  • I searched open issues and pull requests and found no duplicate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions