Repository navigation
Guard only json.loads with the bare ValueError catch in the policy and business-context loaders - #467
Conversation
load_policy_file() and load_business_context_file() wrap the validator in the same try as json.loads, so a plain ValueError raised by a bug in the validator is turned into "holds a value that cannot be read", blaming the user's file for our mistake (Shubham's note on #446). These two tests monkeypatch the validator to raise a plain ValueError and assert it is not turned into a PolicyError / BusinessContextError. Both fail on main for the right reason; the next commit makes them pass.
The broad `except ValueError` in load_policy_file() and load_business_context_file() wrapped the validator as well as json.loads. A plain ValueError from a validation bug therefore reached the user as "holds a value that cannot be read", blaming their file for our mistake (Shubham's note on #446). The bare ValueError that really is about the file is json.loads's (an integer past 4,300 digits), so only json.loads is guarded by it. The validators raise PolicyError / BusinessContextError for every problem with a file; load_policy() keeps a RecursionError guard of its own, because a deeply nested rule list does hit it, while load_business_context() does not recurse and gets none. Anything else a validator raises now surfaces as the bug it is.
Found by mutating my own change: with 'except PolicyError: raise' removed from the json.loads block, the duplicate-key refusal was re-wrapped as 'holds a value that cannot be read: ... appears twice', and every test still passed, because the parametrized test matches on a substring. That guard's one job is now this, so it needs a test that can tell.
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Approving 5087466. That's exactly the narrowing I meant, and thank you for picking it up so quickly.
Checked:
- Merged onto current
main(2833a1e): 2313 passed, 13 skipped, ruff clean on CI's selection. - The one consequence you call deliberate, end to end through
/api/policy. I madeload_policy()raise a plainValueError(a stand-in validator bug) and uploaded a file:So the bug is now visible as a bug, nothing internal leaks out, and a real file problem still gets a 400 that names it.validator bug -> 500 "Internal Server Error" (none of the bug's text reaches the user) duplicate key -> 400 "the key 'policy_compliance' appears twice in one object. ..." - Your mutation table is the right evidence. The
except PolicyError: raisemutant surviving until commit 3 is a good catch on your own change.
On "What this does not do": yes, the empty-queries refusal is mine, and I'm doing it now. Your two other cases (a queries value that's a string, and a malformed address inside a query) belong with it, since both are "the loader accepts a rule the check can't use". I'll check what policy_compliance does with each and include them.
SamikaPerera
left a comment
There was a problem hiding this comment.
I checked the change against main (ee769b4) and ran real bad files through the real loaders and upload endpoints.
What it changes. In analysis/policy.py (load_policy_file) and analysis/business_context.py (load_business_context_file), the validator call moves out of the try that guards json.loads. The except ValueError now covers only json.loads. load_policy() keeps its own RecursionError guard; load_business_context() has none.
Claims checked:
- The two "validator bug" tests fail on main and pass here. Copied onto main's code,
test_a_bug_in_the_validator_is_not_relabelled_as_a_bad_filefails in bothtests/test_policy_loader.pyandtests/test_business_context.py. The two duplicate-key tests pass on main too, which fits: they pin behaviour main already has against this PR's own restructure. - The mutation table holds. I mutated five guards, one at a time, and ran the four loader test files. All five were killed:
- validator back inside the
try(policy) and (context): killed by the validator-bug test - drop
except PolicyError: raise, and dropexcept BusinessContextError: raise: killed by the duplicate-key test - drop the
RecursionErrorguard aroundload_policy(): killed bydeep-validation
- validator back inside the
- "
load_business_context()does not recurse into nested values": true. A 2,000-deepdescriptionis refused as'description' must be text, got list. - "No behaviour change for any input a user can send": true for what I sent. Through
/api/policyand/api/business-context, these return the same 400 and the same message here as on main: a duplicate key, invalid JSON, a 5,000-digit integer, and nesting 100,000 deep. - My #470 is unaffected. #467 and #470 (
e7c3a69) merge with no conflict. On the merge,tests/test_upload_body_limit.pyand the four loader files pass (231). An oversized upload to/api/upload,/api/policyand/api/business-contextstill returns each endpoint's own "larger than the 2 MB limit" message.
Run: this branch merged with current main (ee769b4), Batfish running: 2328 passed, 13 skipped, 0 failed.
One thing I found that this PR neither causes nor fixes, for a separate issue. The refusals that begin with {file_path} show the server's temporary file path to the user. On both main and this branch, uploading invalid JSON to /api/policy returns, for example:
C:\Users\<user>\AppData\Local\Temp\tmp<random>.json is not valid JSON: line 1, column 25: Expecting value
The same happens for all four kinds of unreadable file (invalid JSON, a 5,000-digit integer, nesting 100,000 deep, and a file that is not UTF-8), for policies and business contexts. That shows the Windows user name and the temp folder. It never shows the session cookie. The path comes from the web layer validating a temp copy of the upload and passing the loader's message through as the 400 detail. That is web code, so it is mine. I filed it as #474.
Not checked: the 16 malformed inputs in the description (I re-ran a different set), and the "190 tests" figure. My four-file set gives 192 on this branch, but I don't know which files the 190 counts.
Follow-up to #446, from @shubhamkataria2005's note on it: the broad
except ValueErrorin the two JSON loaders wrapped the validator as well asjson.loads, so a plainValueErrorfrom a validation bug reached the user as "holds a value that cannot be read". That blames the user's file for our mistake, and it hides the bug.What changed
analysis/policy.pyandanalysis/business_context.py, the same shape in each:json.loadskeeps its owntry:JSONDecodeError, the duplicate-keyPolicyError/BusinessContextErrorpassed through,RecursionErrorfor deep nesting, and the bareValueError(the real case: an integer past 4,300 digits).load_policy()/load_business_context()moves out of it. The validators raise their own error type for every problem with a file, so anything else they raise now shows up as the bug it is.load_policy()keeps aRecursionErrorguard of its own, because a deeply nested rule list does hit it (the existingdeep-validationtest).load_business_context()gets none: it does not recurse into nested values (checked: a 2,000-deepdescriptionis refused as "must be text").No behaviour change for any input a user can send. I ran 16 malformed inputs through both validators on
main(bad types, a 5,000-digit string, negative and float numbers, junk severity and kind, a wrong top-level shape, a bad subnet, a non-string device,None). None raised a bareValueError: each was accepted or came back as the loader's own error. So nothing a user can send changes, and a validator bug now fails loudly instead of being blamed on them.Before and after, shown failing first
The tests are in their own first commit (
f4fe6cd) so the history shows them failing before the fix. Againstmain:After the fix commit they pass, with the 190 tests that already exercise these loaders (including the 5,000-digit, deep-JSON, deep-validation, duplicate-key and the
/api/policyupload tests).Mutation testing found a gap in my own change
Nine mutants, each run against the four test files that exercise the loaders, with Batfish and Ollama not needed. First run: 7 caught, 2 survived. Dropping
except PolicyError: raisefrom thejson.loadsblock changed nothing. Without it, the duplicate-key refusal is re-wrapped as "holds a value that cannot be read: ... appears twice", and the existing test only checks for the substring "appears twice". That guard's one remaining job had no test, so the third commit adds one per loader that also asserts "cannot be read" is not in the message.json.loadstry (the original bug), policy and contextexcept PolicyError: raise/except BusinessContextError: raiseValueErrorguard onjson.loads, policy and context5000-digit-intRecursionErrorguard onjson.loads, policy and contextdeep-jsonand the/api/policy400 testRecursionErrorguard onload_policy()deep-validationVerified
What this does not do
queriesloader refusal from Shubham's other note on Close five paths to a false all-clear #442. He offered to take that one, and it is a separate change./api/policyrather than a 400 that names the file. That is deliberate: it is a bug, not a bad file, and the upload is still refused either way.load_policy()today and I left them alone: aqueriesvalue that is a string, and a malformed address inside a query. I haven't checked whatpolicy_compliancethen does with them, and they are not part of this change.