Skip to content

fix(qa): preserve report output and surface URL collection failures - #618

Closed
ANO3457 wants to merge 14 commits into
pyrite-wiki:devfrom
ANO3457:fix/qa-json-and-url-errors
Closed

ANO3457 wants to merge 14 commits into
pyrite-wiki:devfrom
ANO3457:fix/qa-json-and-url-errors

Conversation

@ANO3457

@ANO3457 ANO3457 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #611.

  • Write stale/compact formatted reports with plain stdout, preserving long titles and literal bracket markup in JSON, YAML, CSV and Markdown.
  • Surface URL collection failures through the existing structured CLI error contract, including SQLite and SQLAlchemy failures, instead of claiming no URLs exist.
  • Discover stored entry types so document and plugin entry sources are included; keep explicit type filters and bare string sources working.

Plan / claim

Claimed #611 after #604/#574 landed. This covers the groomed output, failure and source-discovery theme in one PR.

Testing

  • 48 passed: pytest tests/test_json_stdout.py tests/test_qa_commands_run.py tests/test_url_checker.py -q. Tests use the root CLI and real SQLite-backed QA reports, forced color, a 40-column terminal, long bracket titles, and mocked HTTP.
  • Local verify-red working-tree run: 25 red, 0 import-only, 0 unexpected pass, 0 n/a. CI should verify the pushed head.
  • Ruff check and format check passed for the three changed Python files; git diff --check passed.
  • The broader scripts/test-affected --base upstream/dev --run -n 4 run on Windows did NOT pass: 4049 passed, 152 failed, 107 skipped, 897 errors. Visible failures include unavailable POSIX fcntl, missing extension installs, and SQLite file-lock teardown errors (WinError 32). I am not claiming a clean broad local suite; Linux CI is still needed.

Notes for the reviewer

The existing 10,000-entry limit remains per entry type, including newly discovered types. A type with more than 10,000 entries can still be truncated; this PR does not add pagination. A mixed-type query-limit regression protects other types from being hidden by 10,000 newer documents.

Cold review caught a draft implementation that accidentally made this cap global; it was corrected before submission. The final four-file remote diff matches the tested local content. A changelog fragment is included.

@ANO3457

ANO3457 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Fixed the CI architecture-ratchet failure (test_no_surface_reaches_past_the_services): database exception translation now lives in URLChecker.collect_urls, while the CLI handles only the typed PyriteError contract.

The failing CI run had 8,775 passing tests and this one failure. After reproducing it locally, the JSON-output, QA, URL-checker tests plus the architecture assertion pass: 49 passed. Ruff check and format check pass. The uploaded tree at 4cc91b08b7de7386efb3785242f4c5dd65a522a2 matches the locally tested tree. Keeping this draft until CI and review are complete.

@ANO3457
ANO3457 marked this pull request as ready for review October 2, 2026 18:49
@markramm

markramm commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator

Meets #611's acceptance points. Small things:

  • One place for error translation: url_checker.py adds its own sqlite/SQLAlchemy → StorageError conversion; search_service.storage_error_from already does this and marks a database lock retryable (StorageBusyError). This PR always says not retryable, and its test pins that. Property: URL-check storage errors go through the existing helper.
  • Scope: URL checks now cover every entry type, not only the five that were hard-coded. That fits the theme, and the maintainer can accept it; please say so in the changelog.
  • The 10,000-per-type cap still truncates silently; pre-existing, fine to leave for a follow-up.
  • It will be squash-merged (history includes a merge from dev).

Automated review by an agent for the maintainer; comment only.

* test: cover retryable URL collection database locks

* test: cover retryable URL collection database locks

* fix: share URL collection storage error translation

* docs: clarify URL type coverage and retryable locks
@ANO3457 ANO3457 closed this Oct 3, 2026
@ANO3457
ANO3457 deleted the fix/qa-json-and-url-errors branch October 3, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

qa stale and qa compact print JSON through Rich; check-urls hides a database error as 'no URLs'

2 participants