Repository navigation
Check every fetcher's description and instructions, except a frozen pending list - #353
Conversation
…ing list TEXT_FORMAT_PENDING (one entry per line, capped by a frozen baseline file) replaces the opt-in list. A pending fetcher that already passes must leave it. Rewrite the descriptions of two Entra fetchers and the AI services fetcher's instructions so they are checked. State the limits in the guide and skills. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ions Hash-pin the frozen baseline in the subset test, drop API versions and vague back-references from the AI services instructions, state null when a list is unread on every affected field, add the percentage formula to the app registrations fetcher, name audit logs in the sign-in logs description, and remove the unused prefix matcher. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…-list rule and the default-policy note The rewrite dropped two sentences. A per-account list that could not be read is null on that account and sets metadata.partial_failure, and the summary counts built from such lists then cover only the accounts whose list was read (with two accounts and one deployment list failing, total_deployments is 1). Microsoft's default policy blocks jailbreaks but does not enable indirect-attack detection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bruceweygandt
left a comment
There was a problem hiding this comment.
Approved. Every fetcher's description and instructions are now checked unless it is on the pending list, and the list holds exactly the frozen baseline. A pending fetcher whose text passes is told to leave the list. On main merged in, the full suite passes, and none of the fetchers rewritten since this PR opened is still pending. The authoring guide and both skills match the test's limits.
I merged main and restored two sentences the azure_ai_services_configuration rewrite dropped (11fd306): a per-account list that could not be read is null on that account and sets metadata.partial_failure, and the summary counts then cover only the accounts read (two accounts, one deployment list failing: total_deployments is 1); and Microsoft's default policy blocks jailbreaks but does not enable indirect-attack detection. Full suite and CI pass.
Optional follow-ups:
- tests/test_contracts.py hashes the raw bytes of text_format_baseline.txt, and .gitattributes has no rule for it, so a Windows checkout with CRLF endings would fail. Normalize line endings before hashing, or add -text for the file.
- The comment calls the pending list shrink-only, but the test only checks it stays a subset of the baseline, so a fetcher could go back on after leaving. A comment saying not to add one back may be the better trade, since a count line would bring back the conflicts this PR removes.
Fixes FDE-1198
What
The description and instructions format check now covers every fetcher, new ones included, except a shrink-only pending list.
tests/test_contracts.py:TEXT_FORMAT_FETCHERS(opt-in) becomesTEXT_FORMAT_PENDING(opt-out), one fetcher per line grouped by platform, so a rewrite batch only deletes its own lines and no PRs share a count line.tests/text_format_baseline.txt: frozen list the pending list must stay within; never edited; its sha256 is pinned in the test, so any change fails it.azure/ai_services_configuration(one-sentence description; instructions in the Script / Access / Commands Executed / Output / Summary fields layout, with a test callingassert_summary_documented),azure/entra_app_registrationsandazure/entra_sign_in_logs(description).docs/authoring_fetcher_guide.md,create-fetcherandonboard-platformskills: state the hard limits (30-word one-sentence description, no markdown, 800-character field lines, 20,000-character instructions) and that the contract test checks every new fetcher.Pending
213 fetchers on other platforms still fail the check and are listed as pending (AWS 83, GCP 19, GitHub 16, Wiz 14, Datadog 13, and others). They are not changed here.
Checks
ruffon the changed test files,tests/test_contracts.pyand the three fetchers' tests (1202 passed). No full suite.🤖 Generated with Claude Code