Numeric Sanity Checker skill - #213
Open
Tim (Timziito) wants to merge 4 commits into
Open
Conversation
Verifies the arithmetic in a report before it ships: do subtotals sum to the stated total, do percentages sum to 100, does a percent-change figure actually match the two numbers it's derived from.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new “Numeric Sanity Checker” submission to the skill gallery, intended to help agents verify common arithmetic relationships (sum-to-total, percentages-to-100, and percent change) before presenting numeric claims.
Changes:
- Introduces a new agent skill (
SKILL.md) describing when/how to sanity-check arithmetic in generated reports. - Adds a bundled Python checker script to validate sums, percentage totals, and percent-change calculations from a JSON input.
- Provides submission metadata and a human-facing README describing usage and scope.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| submissions/numeric-sanity-checker/SKILL.md | Defines the agent-facing procedure and guardrails for numeric sanity checks, including how to run the bundled script. |
| submissions/numeric-sanity-checker/scripts/check_numbers.py | Implements the JSON-driven checker for sum/percentages/percent-change validations. |
| submissions/numeric-sanity-checker/README.md | Human-facing overview and usage notes for the skill and its checker script. |
| submissions/numeric-sanity-checker/metadata.json | Registers the submission (name/description/platforms/tags/author/version/dates). |
Matches the invocation shown in SKILL.md and README.md, per Copilot review feedback on the PR.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
submissions/numeric-sanity-checker/scripts/check_numbers.py:114
main()passes the parsed JSON directly torun()without validating its shape, so inputs like a single JSON object (not a list) or invalid JSON will crash with a traceback. The script should emit a clear error and exit with a non-zero code, and it should close the file handle when reading from disk.
text = sys.stdin.read() if args.path == "-" else open(args.path, encoding="utf-8").read()
checks = json.loads(text)
results = run(checks)
Fixes the issues flagged in the automated review: see PR discussion for details.
| } | ||
|
|
||
|
|
||
| def check_percent_change(old: float, new: float, claimed_pct: float, tolerance: float = DEFAULT_PERCENT_CHANGE_TOLERANCE) -> dict: |
Collaborator
There was a problem hiding this comment.
During testing this only computed relative percent change, so a correctly-worded "margin improved 5 points (18%→23%)" is scored as a FAIL against ~28%. Percentage-point changes are common in real reports, but the check flags them as errors. Consider a points mode (or requiring the caller to state which is meant).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verifies the arithmetic in a report before it ships: do subtotals sum to the stated total, do percentages sum to 100, does a percent-change figure actually match the two numbers it's derived from.