Skip to content

Guard only json.loads with the bare ValueError catch in the policy and business-context loaders - #467

Merged
ARSH871-bot merged 3 commits into
mainfrom
fix/narrow-valueerror-catch
Nov 16, 2026
Merged

ARSH871-bot merged 3 commits into
mainfrom
fix/narrow-valueerror-catch

Conversation

@ARSH871-bot

Copy link
Copy Markdown
Owner

Follow-up to #446, from @shubhamkataria2005's note on it: the broad except ValueError in the two JSON loaders wrapped the validator as well as json.loads, so a plain ValueError from a validation bug reached the user as "holds a value that cannot be read". That blames the user's file for our mistake, and it hides the bug.

What changed

analysis/policy.py and analysis/business_context.py, the same shape in each:

  • json.loads keeps its own try: JSONDecodeError, the duplicate-key PolicyError / BusinessContextError passed through, RecursionError for deep nesting, and the bare ValueError (the real case: an integer past 4,300 digits).
  • load_policy() / load_business_context() moves out of it. The validators raise their own error type for every problem with a file, so anything else they raise now shows up as the bug it is.
  • load_policy() keeps a RecursionError guard of its own, because a deeply nested rule list does hit it (the existing deep-validation test). load_business_context() gets none: it does not recurse into nested values (checked: a 2,000-deep description is refused as "must be text").

No behaviour change for any input a user can send. I ran 16 malformed inputs through both validators on main (bad types, a 5,000-digit string, negative and float numbers, junk severity and kind, a wrong top-level shape, a bad subnet, a non-string device, None). None raised a bare ValueError: each was accepted or came back as the loader's own error. So nothing a user can send changes, and a validator bug now fails loudly instead of being blamed on them.

Before and after, shown failing first

The tests are in their own first commit (f4fe6cd) so the history shows them failing before the fix. Against main:

FAILED tests/test_policy_loader.py::test_a_bug_in_the_validator_is_not_relabelled_as_a_bad_file
FAILED tests/test_business_context.py::test_a_bug_in_the_validator_is_not_relabelled_as_a_bad_file
2 failed
  a BusinessContextError "... holds a value that cannot be read: a bug in the validator, not in the file"

After the fix commit they pass, with the 190 tests that already exercise these loaders (including the 5,000-digit, deep-JSON, deep-validation, duplicate-key and the /api/policy upload tests).

Mutation testing found a gap in my own change

Nine mutants, each run against the four test files that exercise the loaders, with Batfish and Ollama not needed. First run: 7 caught, 2 survived. Dropping except PolicyError: raise from the json.loads block changed nothing. Without it, the duplicate-key refusal is re-wrapped as "holds a value that cannot be read: ... appears twice", and the existing test only checks for the substring "appears twice". That guard's one remaining job had no test, so the third commit adds one per loader that also asserts "cannot be read" is not in the message.

Mutant Result
Validator back inside the json.loads try (the original bug), policy and context caught, by the new test
Drop except PolicyError: raise / except BusinessContextError: raise caught, by the new duplicate-key test (survived before commit 3)
Drop the ValueError guard on json.loads, policy and context caught, 5000-digit-int
Drop the RecursionError guard on json.loads, policy and context caught, deep-json and the /api/policy 400 test
Drop the RecursionError guard on load_policy() caught, deep-validation

Verified

CI-style (Batfish and Ollama off):  2272 passed, 54 skipped   (main 2268, so +4: the four new tests)
Live (Batfish and Ollama running):  2313 passed, 13 skipped, 0 failed   (+4, the same four)
ruff check --select E4,E7,E9,F analysis/ ai/ web/ tools/ tests/  ->  clean

What this does not do

  • It does not touch the empty-queries loader refusal from Shubham's other note on Close five paths to a false all-clear #442. He offered to take that one, and it is a separate change.
  • A validator bug is now an HTTP 500 on /api/policy rather than a 400 that names the file. That is deliberate: it is a bug, not a bad file, and the upload is still refused either way.
  • Two inputs are accepted by load_policy() today and I left them alone: a queries value that is a string, and a malformed address inside a query. I haven't checked what policy_compliance then does with them, and they are not part of this change.

load_policy_file() and load_business_context_file() wrap the validator in
the same try as json.loads, so a plain ValueError raised by a bug in the
validator is turned into "holds a value that cannot be read", blaming the
user's file for our mistake (Shubham's note on #446).

These two tests monkeypatch the validator to raise a plain ValueError and
assert it is not turned into a PolicyError / BusinessContextError. Both fail
on main for the right reason; the next commit makes them pass.
The broad `except ValueError` in load_policy_file() and
load_business_context_file() wrapped the validator as well as json.loads.
A plain ValueError from a validation bug therefore reached the user as
"holds a value that cannot be read", blaming their file for our mistake
(Shubham's note on #446).

The bare ValueError that really is about the file is json.loads's (an
integer past 4,300 digits), so only json.loads is guarded by it. The
validators raise PolicyError / BusinessContextError for every problem with a
file; load_policy() keeps a RecursionError guard of its own, because a
deeply nested rule list does hit it, while load_business_context() does not
recurse and gets none. Anything else a validator raises now surfaces as the
bug it is.
Found by mutating my own change: with 'except PolicyError: raise' removed
from the json.loads block, the duplicate-key refusal was re-wrapped as 'holds
a value that cannot be read: ... appears twice', and every test still passed,
because the parametrized test matches on a substring. That guard's one job is
now this, so it needs a test that can tell.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 5087466. That's exactly the narrowing I meant, and thank you for picking it up so quickly.

Checked:

  • Merged onto current main (2833a1e): 2313 passed, 13 skipped, ruff clean on CI's selection.
  • The one consequence you call deliberate, end to end through /api/policy. I made load_policy() raise a plain ValueError (a stand-in validator bug) and uploaded a file:
    validator bug   -> 500  "Internal Server Error"      (none of the bug's text reaches the user)
    duplicate key   -> 400  "the key 'policy_compliance' appears twice in one object. ..."
    
    So the bug is now visible as a bug, nothing internal leaks out, and a real file problem still gets a 400 that names it.
  • Your mutation table is the right evidence. The except PolicyError: raise mutant surviving until commit 3 is a good catch on your own change.

On "What this does not do": yes, the empty-queries refusal is mine, and I'm doing it now. Your two other cases (a queries value that's a string, and a malformed address inside a query) belong with it, since both are "the loader accepts a rule the check can't use". I'll check what policy_compliance does with each and include them.

@SamikaPerera SamikaPerera left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the change against main (ee769b4) and ran real bad files through the real loaders and upload endpoints.

What it changes. In analysis/policy.py (load_policy_file) and analysis/business_context.py (load_business_context_file), the validator call moves out of the try that guards json.loads. The except ValueError now covers only json.loads. load_policy() keeps its own RecursionError guard; load_business_context() has none.

Claims checked:

  • The two "validator bug" tests fail on main and pass here. Copied onto main's code, test_a_bug_in_the_validator_is_not_relabelled_as_a_bad_file fails in both tests/test_policy_loader.py and tests/test_business_context.py. The two duplicate-key tests pass on main too, which fits: they pin behaviour main already has against this PR's own restructure.
  • The mutation table holds. I mutated five guards, one at a time, and ran the four loader test files. All five were killed:
    • validator back inside the try (policy) and (context): killed by the validator-bug test
    • drop except PolicyError: raise, and drop except BusinessContextError: raise: killed by the duplicate-key test
    • drop the RecursionError guard around load_policy(): killed by deep-validation
  • "load_business_context() does not recurse into nested values": true. A 2,000-deep description is refused as 'description' must be text, got list.
  • "No behaviour change for any input a user can send": true for what I sent. Through /api/policy and /api/business-context, these return the same 400 and the same message here as on main: a duplicate key, invalid JSON, a 5,000-digit integer, and nesting 100,000 deep.
  • My #470 is unaffected. #467 and #470 (e7c3a69) merge with no conflict. On the merge, tests/test_upload_body_limit.py and the four loader files pass (231). An oversized upload to /api/upload, /api/policy and /api/business-context still returns each endpoint's own "larger than the 2 MB limit" message.

Run: this branch merged with current main (ee769b4), Batfish running: 2328 passed, 13 skipped, 0 failed.

One thing I found that this PR neither causes nor fixes, for a separate issue. The refusals that begin with {file_path} show the server's temporary file path to the user. On both main and this branch, uploading invalid JSON to /api/policy returns, for example:

C:\Users\<user>\AppData\Local\Temp\tmp<random>.json is not valid JSON: line 1, column 25: Expecting value

The same happens for all four kinds of unreadable file (invalid JSON, a 5,000-digit integer, nesting 100,000 deep, and a file that is not UTF-8), for policies and business contexts. That shows the Windows user name and the temp folder. It never shows the session cookie. The path comes from the web layer validating a temp copy of the upload and passing the loader's message through as the 400 detail. That is web code, so it is mine. I filed it as #474.

Not checked: the 16 malformed inputs in the description (I re-ran a different set), and the "190 tests" figure. My four-file set gives 192 on this branch, but I don't know which files the 190 counts.

@ARSH871-bot
ARSH871-bot merged commit b6c104d into main Nov 16, 2026
2 checks passed
@ARSH871-bot
ARSH871-bot deleted the fix/narrow-valueerror-catch branch November 16, 2026 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants