Skip to content

Fix three false SUPPORTED results in claim verification (v0.5.1) - #27

Merged
Shakargy merged 1 commit into
mainfrom
v0.5.1-false-support-hotfix
Sep 5, 2026
Merged

Shakargy merged 1 commit into
mainfrom
v0.5.1-false-support-hotfix

Conversation

@Shakargy

@Shakargy Shakargy commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Why

An external review of v0.5.0 reproduced three cases where dtc verify reported SUPPORTED with no justifying evidence. I re-ran all of them against released code before changing anything - all three reproduced:

admin-authorization  -> SUPPORTED  "1 of 1 administrative route(s) show an
                                    authorization check"   (no guard exists anywhere)
route-test-coverage  -> SUPPORTED  "1 of 1 routes have a referencing test"
                                    (the test never imports or calls the route)
billing-webhook-sig  -> SUPPORTED  (the constructEvent helper is never called)

The admin case is the worst: src/admin/permissions.ts satisfied the evaluator's "permission" token through its filename, because the matcher included the file path. A file's name became evidence about its behavior - the exact failure DevTime exists to catch, shipped inside DevTime.

Fixes

Authorization must be at the route's own call site. The extractor now captures each route's own arguments. Not evidence anymore: guards imported but never applied, named in a comment, named in a string, or applied to a different route in the same file. Authentication is distinguished from authorization (requireAuth establishes identity, not permission). Router-level and server-wide guards are reported as unresolved, never as protected. Missing evidence stays WEAK, never CONTRADICTED.

Test association requires an import, matched on exact module stems (superusers no longer counts as users). Route identity keeps the HTTP method. Name similarity becomes an explicit unverified suggestion that can never raise status. Presented as Route Test Association; the statement no longer says "exercised by tests" because static association is not execution coverage.

Webhook verification is connected per handler (same file). Helpers elsewhere, and calls inside test files, no longer protect production handlers. Mixed repos report "1 of 2 handlers verify a provider signature".

Routes in test/example/fixture directories are not application surface. Express's inventory went from 142 routes - nearly all from its own test/ and examples/ - to 9.

Honest trade-off, stated in the release notes

route-test-association now abstains far more often. On three real repos it reports zero associated routes, because their tests exercise routes through a running server (supertest, TestClient) rather than by importing modules. That's the correct trade - "I cannot establish this" beats a false "your routes are tested" - but it means v0.5.0's numbers were mostly name collisions. Request-based association is the next step, not a claim made today.

Verification

  • 157 tests (16 new). Every reproduction is now a permanent regression test, plus the adversarial cases: commented guards, unused imports, guards in strings, authn-only guards, two routes with one guarded, import stem collisions, signature calls in test files, and routes defined in fixtures.
  • Legitimate patterns still resolve: guard at call site -> SUPPORTED; test importing its route -> SUPPORTED; handler verifying signatures in its own file -> SUPPORTED.
  • README/VERIFICATION/LIMITATIONS updated, including a README line that quoted demo output which this fix made false.
  • build + twine clean.

The demo fixture was deliberately not edited to make its numbers look better - it honestly reports 0 of 3 because its tests don't import the route implementations.

Version 0.5.1. No tag, no publish - awaiting approval.

An external review of v0.5.0 reproduced three cases where dtc verify reported
SUPPORTED with no justifying evidence. All three reproduced against the
released code.

- admin-authorization matched a combined text that included the FILE PATH, so
  src/admin/permissions.ts satisfied its "permission" token and reported
  "1 of 1 administrative route(s) show an authorization check" on a repository
  with no guard anywhere. Security-adjacent, and the reason for this release.
- route-test-coverage accepted name similarity as evidence: an unrelated test
  named "formats users display names" supported GET /users. Imports matched by
  substring (superusers counted as users) and HTTP methods were merged.
- billing-webhook-signature accepted a verification helper anywhere in the
  repository, including one nothing called, as protection for every handler.

Fixes:

- The route extractor captures each route's own call arguments, so a guard is
  attributed to the route it actually wraps. Comments are stripped; imported
  but unapplied guards, guards in strings, and guards on a sibling route are no
  longer evidence. Authentication (requireAuth) is distinguished from
  authorization; router-level and server-wide guards are reported as
  unresolved, never as protected.
- Test association requires an import matched on exact module stems; route
  identity keeps the HTTP method; name similarity is reported as an unverified
  suggestion that cannot raise status. The claim is presented as Route Test
  Association and no longer says "exercised by tests": static association is
  not execution coverage.
- Webhook signature verification is connected per handler by file. Calls in
  unrelated modules or in test files no longer protect production handlers.
  Mixed repositories report per-handler counts.
- Routes defined in test, example, and fixture directories are excluded from
  the application route inventory (Express reported 142 such routes; 9 are
  application surface).

Honest trade-off: route-test-association now abstains on real repositories
whose tests exercise routes through a running server rather than by import.
Its v0.5.0 numbers were mostly name collisions. Request-based association is
the next step.

157 tests (16 new); every reproduction and adversarial case is now a permanent
regression test. Claim ids and JSON schema_version are unchanged. Version 0.5.1.
@Shakargy
Shakargy merged commit d83e51f into main Sep 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant