What happened?
load_profile() and load_policy_file() document ValueError for malformed YAML, and the CLI converts FileNotFoundError/ValueError into a clean click.ClickException. However, both loaders call yaml.safe_load() without catching yaml.YAMLError.
As a result, malformed custom policy YAML raises a raw PyYAML parser/scanner exception and bypasses the CLI's intended error wrapper.
Reproduction steps
At main commit 3bfba44:
from pathlib import Path
from tempfile import TemporaryDirectory
from skillevaluator.validators.policy import load_policy_file
with TemporaryDirectory() as td:
path = Path(td) / "broken-policy.yaml"
path.write_text("severity_overrides: [", encoding="utf-8")
load_policy_file(path)
Actual result:
Expected result:
- the loader raises the documented ValueError;
- CLI use of --policy reports a concise error containing the policy path;
- validation does not continue with a fallback policy;
- no raw PyYAML traceback is exposed.
A focused fix can wrap yaml.YAMLError consistently in the bundled-profile and custom-policy loaders.
SkillEvaluator version or commit
main@3bfba44e754be87073b2344233f9569b06509ce1
Environment
- Windows 11
- Python 3.13
- source checkout with uv
- no provider credentials or external services involved
Before submitting
What happened?
load_profile() and load_policy_file() document ValueError for malformed YAML, and the CLI converts FileNotFoundError/ValueError into a clean click.ClickException. However, both loaders call yaml.safe_load() without catching yaml.YAMLError.
As a result, malformed custom policy YAML raises a raw PyYAML parser/scanner exception and bypasses the CLI's intended error wrapper.
Reproduction steps
At main commit 3bfba44:
Actual result:
Expected result:
A focused fix can wrap yaml.YAMLError consistently in the bundled-profile and custom-policy loaders.
SkillEvaluator version or commit
main@3bfba44e754be87073b2344233f9569b06509ce1
Environment
Before submitting