Skip to content

Pin each spelling of the temp path that refusals hide, one at a time - #478

Open
SamikaPerera wants to merge 1 commit into
mainfrom
test/temp-path-spelling-cases
Open

SamikaPerera wants to merge 1 commit into
mainfrom
test/temp-path-spelling-cases

Conversation

@SamikaPerera

Copy link
Copy Markdown
Collaborator

Follows up on #475. Tests only; no product code changes.

What Shubham found. In his approving review of #475 (Mon 16 Nov 13:53), he dropped the doubled, repr() and as_posix() spellings from _refusal_without_temp_path() in web/main.py one at a time. Each mutant "survives all 14 tests". He suggested putting a backslash in a temp folder name the test controls.

Why the folder backslash was not enough. canary_tempdir already makes a folder named netwise-temp-canary\x, and the mutants still survived. The reason is in what the loaders write. I measured the real analysis/policy.py and analysis/business_context.py refusals for five bad files each:

  • invalid JSON
  • nesting 100,000 deep
  • bytes that are not UTF-8
  • an integer of 5,000 digits
  • a file that does not exist

Under Windows and under Linux (WSL Ubuntu, Python 3.12.3), all 10 refusals contain only the plain spelling. So none of the three mutants changes any refusal the endpoints can produce today, on either OS.

A second reason: repr()'s inner text is the doubled spelling. Each one replaced the other's text when one was removed, and the one test that sends a quoted path could not tell which did the work.

What the new tests do.

  • They call _refusal_without_temp_path() directly with a PureWindowsPath and a PurePosixPath whose folder holds a backslash. Pure paths behave the same on any OS, so the Windows case runs on CI's Linux too.
  • Each message holds exactly one spelling, and the result is compared exactly: cannot read 'my-policy.json': Permission denied.
  • Removing repr() would leave ''my-policy.json'' (quotes doubled). That is not a leak, but the exact comparison catches it.
Mutant main's 14 tests with this PR (21 tests) failing tests
none (control) 14 passed 21 passed —
drop doubled 14 passed 2 failed each_spelling_alone…[doubled-unquoted-windows], [doubled-unquoted-linux-backslash]
drop repr() 14 passed 2 failed [repr-windows], [repr-linux-backslash]
drop as_posix() 14 passed 1 failed test_a_forward_slash_spelling_of_a_windows_path_is_replaced

Measured with pytest on Windows 11 (Python 3.12.4). I also ran a standard-library harness under WSL Ubuntu, with the real loaders and the real function: the new cases change 2, 2 and 1 of 7 outputs, and no real loader output changes.

Counts.

  • This file: 21 passed.
  • Full suite, with Batfish not running: 2460 passed, 63 skipped.
  • ruff check is clean on the file, both with CI's selection and with the defaults.

Not covered.

  • The endpoints are still only tested with the plain spelling, because that is the only one the loaders write. If a loader starts writing another spelling, these tests pin the function, not the new message.
  • The function's own documented gaps (a resolved or 8.3 short path, or the folder quoted without the file name) are unchanged.
  • Not run: pytest under real Linux. WSL here has no pip, so only the standard-library harness ran there. CI will run the file on Linux.

This is a follow-up to #475 with no rush and is not needed for the week 7 deadline.

_refusal_without_temp_path() replaces four spellings of the temp file's
path: repr(), backslashes doubled, plain, and forward slashes. The
endpoint tests pass with any one of repr, doubled or as_posix removed,
because the real loaders only ever write the plain spelling, and repr's
inner text is the doubled one, so each covered for the other.

These tests call the function directly with pure Windows and Linux
paths, so a Windows path is a Windows path on CI's Linux too, and each
message holds exactly one spelling. Tests only.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed e45478a, merged onto main cceffec. Approving.

The point of the PR checks out. I dropped each spelling from _refusal_without_temp_path() and ran your file, then main's copy of it:

mutant          this PR (21 tests)   main's copy (14 tests)
drop repr()         2 failed             all pass
drop doubled        2 failed             all pass
drop as_posix()     1 failed             all pass
drop plain         12 failed            11 failed

So three of the four spellings had no test that could tell they were gone, as you said, and each now has one. Calling the function directly with PureWindowsPath / PurePosixPath is the right way to get the Windows case onto CI's Linux.

CI-style suite on the merge: 2460 passed, 63 skipped (7 more than main). Ruff clean.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at e45478a.

These tests call the replacement directly, so they work on Linux CI. #475 could only reach the backslash spellings on Windows. To check that each one is really pinned, I removed one spelling at a time from the replacement and ran this file:

spelling removed       result
doubled backslashes    2 failed, 19 passed
repr() form            2 failed, 19 passed
as_posix() form        1 failed, 20 passed

Every removal made the tests fail, so none of the three is covered only by another one. The canary tests are unchanged and still pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants