Skip to content

Check every fetcher's description and instructions, except a frozen pending list - #353

Merged
bruceweygandt merged 6 commits into
mainfrom
fde-1198
Nov 17, 2026
Merged

bruceweygandt merged 6 commits into
mainfrom
fde-1198

Conversation

@Controgon

Copy link
Copy Markdown
Contributor

Fixes FDE-1198

What

The description and instructions format check now covers every fetcher, new ones included, except a shrink-only pending list.

  • tests/test_contracts.py: TEXT_FORMAT_FETCHERS (opt-in) becomes TEXT_FORMAT_PENDING (opt-out), one fetcher per line grouped by platform, so a rewrite batch only deletes its own lines and no PRs share a count line.
  • tests/text_format_baseline.txt: frozen list the pending list must stay within; never edited; its sha256 is pinned in the test, so any change fails it.
  • New tests: pending is a subset of the baseline and names real fetchers; a pending fetcher whose text already passes must leave the list; every other fetcher follows the format.
  • Three Azure fetchers rewritten so they are checked, text only, no version bumps: azure/ai_services_configuration (one-sentence description; instructions in the Script / Access / Commands Executed / Output / Summary fields layout, with a test calling assert_summary_documented), azure/entra_app_registrations and azure/entra_sign_in_logs (description).
  • docs/authoring_fetcher_guide.md, create-fetcher and onboard-platform skills: state the hard limits (30-word one-sentence description, no markdown, 800-character field lines, 20,000-character instructions) and that the contract test checks every new fetcher.

Pending

213 fetchers on other platforms still fail the check and are listed as pending (AWS 83, GCP 19, GitHub 16, Wiz 14, Datadog 13, and others). They are not changed here.

Checks

ruff on the changed test files, tests/test_contracts.py and the three fetchers' tests (1202 passed). No full suite.

🤖 Generated with Claude Code

Controgon and others added 4 commits November 13, 2026 11:15
…ing list

TEXT_FORMAT_PENDING (one entry per line, capped by a frozen baseline file)
replaces the opt-in list. A pending fetcher that already passes must leave it.
Rewrite the descriptions of two Entra fetchers and the AI services fetcher's
instructions so they are checked. State the limits in the guide and skills.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ions

Hash-pin the frozen baseline in the subset test, drop API versions and
vague back-references from the AI services instructions, state null when a
list is unread on every affected field, add the percentage formula to the
app registrations fetcher, name audit logs in the sign-in logs description,
and remove the unused prefix matcher.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
bruceweygandt and others added 2 commits November 17, 2026 13:27
…-list rule and the default-policy note

The rewrite dropped two sentences. A per-account list that could not be read
is null on that account and sets metadata.partial_failure, and the summary
counts built from such lists then cover only the accounts whose list was read
(with two accounts and one deployment list failing, total_deployments is 1).
Microsoft's default policy blocks jailbreaks but does not enable
indirect-attack detection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@bruceweygandt bruceweygandt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Every fetcher's description and instructions are now checked unless it is on the pending list, and the list holds exactly the frozen baseline. A pending fetcher whose text passes is told to leave the list. On main merged in, the full suite passes, and none of the fetchers rewritten since this PR opened is still pending. The authoring guide and both skills match the test's limits.

I merged main and restored two sentences the azure_ai_services_configuration rewrite dropped (11fd306): a per-account list that could not be read is null on that account and sets metadata.partial_failure, and the summary counts then cover only the accounts read (two accounts, one deployment list failing: total_deployments is 1); and Microsoft's default policy blocks jailbreaks but does not enable indirect-attack detection. Full suite and CI pass.

Optional follow-ups:

  1. tests/test_contracts.py hashes the raw bytes of text_format_baseline.txt, and .gitattributes has no rule for it, so a Windows checkout with CRLF endings would fail. Normalize line endings before hashing, or add -text for the file.
  2. The comment calls the pending list shrink-only, but the test only checks it stays a subset of the baseline, so a fetcher could go back on after leaving. A comment saying not to add one back may be the better trade, since a count line would bring back the conflicts this PR removes.

@bruceweygandt
bruceweygandt merged commit 582653f into main Nov 17, 2026
16 checks passed
@bruceweygandt
bruceweygandt deleted the fde-1198 branch November 17, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants