Repository navigation
feat: v0.4.0 — where your site contradicts itself - #27
Merged
Merged
Conversation
A flat 100-point budget divided among a variable number of gates meant seven maxed gates zeroed a layer whether it had seven gates or thirty. seo is about to grow from 16 gates to 24, which would have cut the fraction of the layer needed to floor it from 44% to 29% -- the v0.2.1 saturation bug returning in slower form. The same constant also put a floor of 70 under the 2-gate security layer. One maxed gate now costs exactly 1/N of its layer and a layer floors only when every one of its gates is maxed. GATE_CAP is unchanged and still does its job.
Additive on both axes: an older report contains neither the security layer nor the new reasons and still validates. tests/test_registry.py's LAYERS filter is widened here, before any security.* id exists, so the id scan sees Group A's literals the moment they land instead of reporting them as unemitted.
HSTS presence and max-age, X-Content-Type-Options, Content-Security-Policy absence, Referrer-Policy. All info severity, so all cost zero points and none can fail a build: the gap analysis puts header grading out of scope by identity, and the honest position is to report them as inventory facts rather than grade them. CSP absence is reported; CSP contents deliberately are not. HSTS_MIN_MAX_AGE is named in the finding text because it is OmniRank's own floor, not a standard, and no claim is made about any browser preload programme. Also updates the two registry counter tests (48->53 total, infrastructure 6->11) incrementally rather than leaving the suite red until Task 12, per the brief's own fallback for one-commit-at-a-time execution. Reworded the short-max-age fix text to drop the literal substring "preload list": the brief's own test asserted that phrase must be absent while its own verbatim implementation text contained it (as part of an honest disclaimer), so the two were mutually unsatisfiable as written. The disclaimer's substance is preserved without the flagged phrase.
The two security gates that carry error severity, because both break something observable: browsers block mixed active content outright, and an http:// origin serving 200 gives every page a live duplicate that splits its canonical signals. Mixed content is suppressed by upgrade-insecure-requests in the CSP -- the one purpose this tool parses CSP for -- and skips <link> rels that are not subresources, so an http canonical is not misreported as a blocked asset. The http probe returns notEvaluated when the origin refuses the connection, because an unreachable origin has not passed this gate. Two brief/test fixes found during TDD: the mixed-content finding's example list was capped at 3 items, so the fixture exercising all four recognised subresource types (script/link/img/iframe) always dropped the last one behind the ellipsis -- raised the cap to 5. Also updated the two registry counter tests again (53->55, templated 15->16, infrastructure 11->12) for this task's two new ids, per the same one-commit-at-a-time fallback used in Task 3.
urllib.robotparser only became RFC 9309 compliant in Python 3.14. On 3.11-3.13 -- the whole CI matrix -- it ignores path wildcards and matches rules in file order, so 'Disallow: /docs/' followed by 'Allow: /docs/public/' reports /docs/public/x as disallowed. That is a fabricated finding, not merely a missed one, and it would have passed locally on 3.14 exactly like the v0.3.0 read_text(newline=) bug. The stdlib stays the matcher. What is new is a runtime behaviour probe plus a structural scan of the published rules: evaluate when the two are compatible, refuse and let the caller report notEvaluated when they are not.
The release's headline check and the same class of self-contradiction as seo.noindex.in-sitemap, rated the same: the site submits a URL for crawling in one file and forbids the crawl in another, so one declaration is wrong and no external data is needed to know it. Reports notEvaluated -- never a pass -- when robots.txt is unreachable, when there is no sitemap to contradict, or when the published rules need a matcher capability this interpreter lacks. Also updates test_registry.py's two static counters (55->56 total, advisory 11->12) so the suite stays green at this commit rather than deferring both to Task 12, matching the fallback the Task 3/4 brief offered and Tasks 1-4 already used.
…irect or 404 Targets already in the crawled set cost no extra request. Targets outside it are probed once each, deduplicated, capped at MAX_CANONICAL_PROBES, and anything past the cap is reported as budget-exceeded rather than skipped. A 5xx or a transport failure is page-unreachable, never .not-found -- a transient origin error is not a missing page and OmniRank does not guess which one it saw. site.py grows three public aliases so this module reuses the exact noindex, URL-key and canonical predicates the site pass already uses, instead of growing a second opinion about what they mean. Keeps the page-unreachable dedup at its brief-documented default (one NotEvaluated entry per pointing page, not per target) since only the single-page case is asserted. Also updates test_registry.py's two static counters (56->59 total, templated 16->17, advisory 12->14) to keep the suite green at this commit.
…lternate Attached to the DECLARING page: engines discard an hreflang annotation whose target they cannot resolve, so it is the declaring page that loses its international targeting. Crawled-set only, x-default exempt, no network I/O. Also updates test_registry.py's two static counters (59->60 total, advisory 14->15) to keep the suite green at this commit.
Eight types, expressed as alternatives groups so 'Product needs name AND (offers OR aggregateRating OR review)' is one table entry. One finding id for the whole class: the fix is always the same action, so the tier is identical, and a page missing six properties is one bad template rather than six defects -- which is also what keeps GATE_CAP charging it once. warning, not error: SS6 of the gap analysis warns that a stale required-property table fabricates errors, and this table has no freshness test yet. RICH_RESULT_RULES_AS_OF is named in every finding so its age is inspectable, and the fix text states plainly that the markup is valid schema.org and that what is unmet is Google's rich-result requirement -- two different claims. Registry counters (test_registry.py) updated incrementally rather than deferred to Task 12, to keep the suite green at every commit: 60 -> 61 total ids, drafted 12 -> 13.
alt="" is never flagged -- it is the spec's own decorative marker, and a check that fires on it tells users to make their markup worse. role=presentation, role=none and aria-hidden are honoured for the same reason. Only the first heading skip is reported, and only an INCREASE of more than one level counts. Links inside <nav> are exempt and generic anchor text is info, because the word list is English-only by construction. seo.lang.missing is an error because a missing lang makes another gate lie: page.lang drives bands.resolve_band(), so without it a page in a script with no word separators is measured against the Latin word band and aeo.answer-block.length reports a problem that does not exist. Brief defect fixed: the brief's verbatim onpage.py used zip(levels, levels[1:]) and audit.py's gates import as a single-line parenthesized list, both of which ruff flags (RUF007 prefer itertools.pairwise, I001 unsorted import block). Switched to itertools.pairwise() and ruff's own multi-line import layout so `ruff check scripts/py` stays clean, per the global constraint. Registry counters (test_registry.py) updated incrementally to keep the suite green at every commit: 61 -> 66 total ids (the full v0.4.0 registry size), tier distribution now mechanical 4 / templated 18 / drafted 16 / advisory 16 / infrastructure 12 -- matching Task 12's brief exactly, since this is the last task that adds new registry entries.
v0.4.0 IS this release and it deliberately ships no writing, so '--write arrives in v0.4.0' would become a false statement printed to every user the moment the tag lands -- in a tool whose whole positioning is that it does not assert what it has not verified. The replacement names no version at all. Naming v0.5.0 would repeat the mistake one release later; the honest form states the condition writing ships under: the locator proven against real repositories, and the automation-architecture SS2.5 guarantees implemented and tested. Brief defects fixed: - cli.WRITE_UNAVAILABLE's brief text said "it modifies nothing", but the brief's own Step-1 test asserts the literal substring "writes nothing" is in the message -- the verbatim implementation would never satisfy its own test. Changed to "it writes nothing" so the string actually makes the claim the test checks for. - Grepped beyond the brief's three-test file list (as instructed) and found two more tests pinning "v0.4.0": test_fixes_schema.py's test_no_context_declines_on_a_tsx_location and test_no_context_declines_on_a_markdown_location_with_markdown_specific_wording exercise the same TSX_DEFERRED/MARKDOWN_DEFERRED strings via schema.no_context() and would have gone stale unnoticed. Corrected both to assert "v0." not in made.reason instead. - docs/faq.md's "Will OmniRank edit my files?" answer carried the identical "arrives in v0.4.0" promise and is user-facing prose read by exactly the audience this task protects; corrected it to name the same condition. - docs/fix-preview.md's fixTier count table (4/15/12/11/6) was already stale after Tasks 9-10 landed 18 new registry entries; updated to the current registry counts (4/18/16/16/12) while already in the file for the version fix. docs/wiki/*.md and README.md's "## What ships in v0.3.0" section were left untouched: the wiki is synced from a separate workflow outside these four briefs' file lists, and README.md's version section is explicitly Task 12's scope (it also renames the v0.3.0 heading, so half-editing it here would leave it internally inconsistent until Task 12 lands).
66 finding ids (48 + 18), 30 scoring gates across five layers, and the score model normalised by gate count so the seo layer stops saturating as it grows. Scores are not comparable with 0.3.0: a layer's budget is now divided by its own gate count, so one broken gate costs 1/N of the layer rather than a flat 15. Registry tests retitled from "forty_eight" to "sixty_six" (the counters themselves were already correct, updated incrementally at Tasks 9 and 10 to keep the suite green at every commit -- this task only renames/reshapes the assertions per the brief and adds the seo=24/security=2 scoring-gate-count guard that pins the v0.4.0 arithmetic). Also updated beyond the brief's literal file list, for the same honesty reason Task 11 existed: - skills/audit/SKILL.md said "there is no --write flag in 0.3.0" -- the exact stale version promise Task 11 targeted, just in a file that brief's grep didn't cover (a skill description, not a test-pinned string). Corrected to drop the version number. - skills/audit/references/gates.md gained full sections for the four v0.4.0 gate groups (security, indexability contradictions, schema-required, on-page) -- the brief said "shipping 14 undocumented gates in it would be its own small fabrication," and the previous commits had left exactly that. - docs/fix-preview.md's fixTier count table, left stale by Tasks 9-10, is now correct (was already touched in Task 11's commit).
…t's robots.txt (B1) RobotFileParser.can_fetch() discards the host and matches on path alone, so every sitemap URL was judged against the audited host's robots.txt even when a sitemap index legitimately listed blog./shop./cdn. URLs on other hosts -- fabricating error findings on the gate this release is named after. check_sitemap_vs_robots now splits sitemap_urls by robots.txt's own netloc: only same-host URLs are evaluated, and off-host URLs get a not-applicable notEvaluated entry instead of a guess. Reproduced and verified fixed against two real local HTTP servers on different hosts.
… the surface (B2) Report.score() admitted every (layer, gate) pair into seen_gates, including gates whose findings all cost 0 -- re-counting exactly the info-only gates registry.py's SCORING_GATES_BY_LAYER deliberately excludes from the scoring surface. That widened the denominator for free, so adding a harmless info finding on a not-yet-seen gate could RAISE a layer's score: verified before this fix, 4 security errors scored 67, and the SAME findings plus 4 missing- header info findings scored 78 -- worse headers, better score. Guard the seen_gates.add() on the gate's raw cost being nonzero. Also fixes test_one_maxed_gate_costs_exactly_one_gates_worth_of_its_layer, which pinned the bug: its `beta` gate is info-only and must NOT enter the surface, so the layer's whole budget belongs to the one real gate and the score is 0, not the 50 the test previously asserted (chosen over "make beta a warning" because that would give beta its own nonzero cost, contradicting the test's own "beta contributes nothing" comment and changing what's being demonstrated). Adds a 40,000-trial randomised property test, drawing findings from the real registry, asserting adding any finding never raises any layer's score or overall -- the guarantee score()'s docstring already claimed. It fails against the pre-fix code within the first 400 trials.
Commit 32209f7 fixed docs/ and skills/ but missed docs/wiki/, which syncs to the live GitHub wiki -- so on tagging, "This release writes nothing. --write arrives in v0.4.0" and four siblings (FAQ, Home, Roadmap, Troubleshooting) would go live as statements v0.4.0 itself falsifies. Fix-Preview.md's copy was worse: presented as a verbatim CLI transcript, it no longer matched the shipped NOTHING_WRITTEN/WRITE_UNAVAILABLE strings in cli.py at all. Grepped docs/wiki/ for v0.4.0, --write and 0.3.0 rather than trusting the five-file list -- also found and fixed Troubleshooting.md's unrelated "deferred to v0.4.0" (a Next.js metadataBase fix, same false-promise pattern, now "deferred until the locator is proven" in the code) and Claude-Code- Setup.md / Quick-Start.md's stale "0.3.0" claims about the installed version and the exit-code table. Every replacement names the CONDITION writing ships under -- the locator proven against real repositories, the write guarantees implemented and tested -- never a version number, exactly as 32209f7 did for docs/. Adds a parametrized regression test over every docs/wiki/*.md page that fails on "arrives in/targeted at/planned for/deferred to/wait(ing) for/until vX.Y" so a future version-numbered promise cannot recur unnoticed. Verified it fails against the pre-fix wiki on exactly the five affected pages.
…ty scoring 100 on zero pages (S4, S5)
S4: security.mixed-content.subresource claimed at error severity that flagged
subresources "are not loading for your visitors at all", but that is only
true for BLOCKABLE ("active") content -- script, iframe, and
link[rel=stylesheet|preload|modulepreload]. Browsers auto-upgrade
OPTIONALLY-BLOCKABLE ("passive") content -- img and icon-family links -- to
https instead of blocking it, so the same claim was false for those. Split
into two ids: subresource (error, active only) keeps the original claim,
which is now actually true; a new passive-subresource (warning) reports what
OmniRank actually knows -- the request asked for http://, not that it failed.
S5: audit_site() admitted `security` to layers_run on a successful
https-redirect probe ALONE (`pages or not https_not_evaluated`), so
mixed-content -- one of only two scoring gates -- was silently counted clean
when zero pages were ever parsed. Reproduced the real report this caused:
rust-lang.org showed `security: 100` with `urlsChecked: 1` and 0 pages
parsed, while aeo/perf were correctly absent. `security` now joins
layers_run the same way aeo/perf do -- only after at least one page was
fetched -- removing the site-level-probe shortcut entirely. A REAL
https-redirect finding still reaches the score map on its own merits
regardless (Report.score() admits any layer with an actual finding), so this
only removes the fabricated clean-100 path, not genuine signal.
Both verified against the pre-fix code: the zero-page audit test fails with
`security: 100` before the audit.py fix and passes after.
…stale gate counts (S1, S2, S3) S1: docs/audit-guide.md -- the doc README names as authoritative -- had ZERO v0.4.0 content. It still said "the four layers" (quoting a Layer literal without security), documented the flat pre-v0.4.0 scoring budget Task 1 replaced, and listed 0 of the 14 gates v0.4.0 added. Rewrote the layers table, added Security/Indexability-contradictions/On-page gate reference sections (mirroring skills/audit/references/gates.md, also brought current for the S4 mixed-content split and S5 layersRun fix), and rewrote Scoring and Worked example against the real per-layer-surface-normalised formula -- every number in both sections is real output, captured against a live local fixture and https://example.com/https://example.com/nope-xyz on this machine. test_docs.py's NEW_GATE_IDS was a hardcoded v0.3.0-era list that let test_audit_guide_documents_the_new_gates pass despite the gap; replaced with one representative id per v0.4.0 gate. S2: README's "why does my site score 0" FAQ said seven broken gates zeroes a layer -- the OPPOSITE of v0.4.0's headline change. Measured directly: 7 maxed seo gates now score 71, 12 score 50, 16 score 33, and only all 24 reach 0. S3: gate counts were stale in three primary docs, not just the wiki -- 28/15/13 (a v0.2.0/v0.2.1 figure) even after v0.4.0 added 14 gates. Recounted from the registry and schema directly: 42 gates in the failOn enum, 21 error-capable, 67 finding ids (66 before S4's mixed-content split). Fixed every site named in the review (README.md, docs/ci-integration.md, docs/troubleshooting.md, docs/wiki/Configuration-Reference.md, docs/wiki/CI-Recipes.md, docs/wiki/Audit-Skill.md -- the last of these also had zero v0.4.0 content, same as audit-guide.md, so it got the same gate/ scoring rewrite) and added every missing v0.4.0 gate table to README's own gate listing, which was missing Security, Indexability-contradictions and On-page entirely. Added a parametrized test computing the real counts from the registry and schema and asserting each doc still states them, so this cannot silently drift again.
…S7); fix unused import (S9) S7: .github/workflows/ci.yml's Python matrix stopped at 3.11-3.13, and the robots-sitemap contradiction gate's capability probe (omnirank/robots.py) degrades on all three -- `RobotsVerdict.evaluated` is always False there, so every `evaluated is True` assertion in the suite was skipped on every CI leg, never actually running the gate's real firing path. Added 3.14. A new workflow test pins the matrix contains it. S9: tests/test_config_schema.py imported pytest but never used it (ruff F401) -- invisible to CI's `ruff check scripts/py`, which does not lint tests/, but a real lint violation in a new file regardless. Removed.
The capability-probe cross-check asserted matcher_supports_wildcards() equals sys.version_info >= (3, 14), on the reasoning that both capabilities arrived in CPython's 3.14 RFC 9309 rewrite. CI disagreed: the change was backported, so 3.13.7 reports no wildcard support while the runner's 3.13.14 reports it. A version inference calls one of those wrong, which is exactly the failure the probe exists to prevent -- the test had hard-coded that bug into the regression test guarding against it. It now compares each probe against the matcher's own observed behaviour, computed independently, so a probe stuck on a constant still fails while whatever CPython backports next cannot break it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Broadens audit coverage rather than adding file-writing. Writing stays unshipped — a richer audit is what earns the right to edit files later.
The theme
"OmniRank finds where your site contradicts itself." Defects provable from the site's own declarations, needing no external truth and no guessing: a sitemap URL the site's own robots.txt forbids, a canonical pointing at a noindex page, a canonical pointing at a 404, a noindexed page declared as an hreflang alternate. The competitive analysis identified this as the project's most defensible category, and this release makes it the headline.
It found a real one on the first site tried:
danluu.com/simple-architectures/canonicalises to/simple-architecture— missing thes— and that URL 404s.What's new
securitylayer, previously 0 gates: HSTS,nosniff, CSP presence,Referrer-Policy, mixed content, andhttp→httpsredirect@typeexist" check, with the finding text naming whose requirement is unmet, since a page can be valid schema.org and still ineligiblealt, heading-order skips, generic/empty link text, missing<html lang>67 finding ids, 42 gates.
What review caught
Scoring was non-monotone — adding findings raised the score. Zero-cost
infogates widened the denominator, so a site missing all four security headers scored 72 while the same site with them scored 17. Worse headers, better score, on the layer this release adds.danluu.comreportedsecurity 87where the truth was67. A test was pinning the bug in place. There is now a property test over the real registry that fails at trial 386 against the old code.The flagship gate fabricated findings across hosts.
can_fetch()discards the host and matches on path alone, so a sitemap index legitimately listingblog./cdn.URLs had those paths judged by the audited host's robots.txt — a host whoserobots.txtwas never fetched. Off-host URLs are nownotEvaluated.The published wiki still promised
--writein v0.4.0 in five places, one a fabricated CLI transcript that didn't match the shipped string.Plus:
docs/audit-guide.mdstill documented the scoring model this release deleted, the README FAQ stated the opposite of the headline change, and gate counts were stale in three primary docs.A trap worth recording
urllib.robotparserbehaves differently across the CI matrix. CPython rewrote it for RFC 9309 in 3.14, and on 3.11–3.13 it ignores anAllow:override:A robots gate built on the 3.13 behaviour would report a perfectly crawlable URL as blocked — a fabricated finding on three of four CI legs, passing green on a 3.14 dev machine. The matcher is now behind a runtime capability probe that degrades to
notEvaluatedrather than guessing, verified under both interpreters. CI gains a 3.14 leg so the probe's firing branch is actually exercised.Verification
Allow-override,x-default-only hreflang, targets outside the crawled set, non-matchingUser-agentblocksalt=""is never flagged; it is a legitimate decorative markeraudit+fixrun — byte-identical