Repository navigation
tools: stricter catalog validation, run_evals --skill, machine-readable Harbor job check - #75
Merged
Merged
Conversation
vchamarthi
requested review from
benjaminum,
napetrov and
xaleryb
as code owners
October 23, 2026 03:07
vchamarthi
force-pushed
the
feat/tooling-fixes
branch
from
October 23, 2026 03:10
77214fc to
bfdfc1a
Compare
vchamarthi
changed the title
The file used bare pytest-style functions, so the unittest-based Tool tests step collected none of them and a broken --skill filter stayed green. The unknown-skill test now checks the tool's own FAIL line, since argparse rejecting the flag on main satisfied the old assertion. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…rbor_skill check_harbor_job.load() now raises LoadError instead of exiting, and summarize() did not catch it, so a malformed trial result.json ended --report-only and the post-run summary in a traceback. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
reward_of() returns None for NaN, infinity and overflowing rewards, so the text report told the author to write a reward file that was already written. A present but non-finite or overflowing reward now gets its own message; rc and the --json rows are unchanged. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
The validate.yml row names each blocking step, but left out the two blocking steps this branch adds: Tool tests and the validate_skills.py self-test. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Contributor
|
Ran on the head: the
PR body: text mode now fails NaN/Infinity rewards that main passed, which "text output unchanged" does not cover. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Five fixes to the repo's own tooling. No skill text changed.
validate_skills.py: an unknownskills.yamlkey (e.g.intel-hw-clas) and anintel-hw-classoutsidecpu | gpu | npu | acceleratorare errors. A non-string$schema/schemainskills.sh.jsonis an error, with a--self-testthat CI runs.run_evals.py --skill NAMElimits--validateand scoring to one skill (0/3, not0/30).check_harbor_job.py --jsonprints a machine-readable verdict (ok,problems, per-trial results). Missing, malformed, non-object, deeply nested and NaN/Infinity/overflowing results become structured failures; text output and--allow-errorsare unchanged.compare_harbor_skill.py --report-onlyno longer raisesKeyError: 'candidate'when only the baseline arm exists; the delta cell prints—.Checklist
descriptioncheck: N/A, tools only.python3 tools/validate_skills.pypasses locally.Evidence
5fdf63epython3 -m unittest discover -s tools/tests -t tools/tests: 33 tests, OK on Python 3.11, 3.14, 3.15. The deep-nesting test accepts either failure message, since 3.14+ parses that depth instead of raisingRecursionError.python3 tools/validate_skills.py --self-test: PASS.run_evals.py --validate --skill dpnp-interop:OK 1 eval file(s), 3 case(s); plain--validateunchanged at 10 / 30.intel-hw-class: GPUgive no error;--skilland--jsonare unrecognized arguments;--report-onlycrashes).Blast radius
tools/validate_skills.py,run_evals.py,check_harbor_job.py,compare_harbor_skill.py, newtools/tests/, and twovalidate.ymlsteps (self-test, unittest discovery with a collected-count guard).