Repository navigation
sources: add federal-IT trade press, so the beat can see events - #16
Merged
Merged
Conversation
The us-public-sector-compliance beat resolved nine sources, all .gov primary-document publishers. It could see published rules and nothing else, which is why its first live brief missed the reinstatement of the FedRAMP director: across 50,983 stored articles there were three incidental FedRAMP mentions and none within two weeks of the event. config/feeds/domains/federal_it.json adds five trade feeds tagged into the beat's existing selectors: FedScoop, DefenseScoop, CyberScoop, Nextgov/FCW - Cybersecurity, Washington Technology. All five resolve and all five returned a non-zero live count (10/10/10/5/11 over a four-day lookback). FedScoop's feed carried the missed event itself, stored with 320 words of body text. tests/config/test_beats.py asserts the beat resolves at least one non-.gov source and names all five, so a feed-config edit that silently drops them fails CI. Verified by removing the config file and watching the test fail. The feed ceiling rises from 12 to 20; the AP/Reuters exclusion is re-asserted in the new test, unchanged. SOURCES.md records a basis for each, a new rule 5, a new "Publisher RSS - personal use only" class, and the three candidates rejected on their terms rather than on a technical failure: - Federal News Network: robots.txt disallows ClaudeBot, Claude-Web, anthropic-ai and Claude by name. Not a barrier to this client, but a plain statement about this use. - MeriTalk: terms forbid automated monitoring or indexing of the site with no RSS carve-out. - Government Executive: same terms as Nextgov/FCW and shippable on the same basis, left out because its feed is workforce news, not federal IT. Two things a reviewer must decide rather than accept. Nextgov/FCW and Washington Technology are configured under a GEMG feed licence limited to personal, non-commercial use; that covers today's terminal brief and not publication. And those two feeds' content:encoded arrives with escaped CDATA, so feedparser sanitizes the body away: 0 of 16 stored rows have non-empty content, and only the publisher's one-sentence dek survives. Recorded, not worked around; fixing it means changing src/rss/fetcher.py for every configured feed.
…tatus SOURCES.md quoted licence terms verbatim -- the GEMG feed grant, the "except as expressly permitted by GEMG" clause, MeriTalk's automated-access prohibition -- with no URL attached to any of them. In the file that decides whether a source may be used, in a public repo, that is the licensing equivalent of a synthesis claim with no citation, which src/prompts/ANALYSIS_RULES.md already forbids for the tool's own output. Adds a provenance table to the federal-IT trade press section: eleven claims, each with the exact URL, the read date, and whether it was confirmed by retrieving that URL. All eleven were re-fetched in this session, every one returned HTTP 200, and every quoted phrase was confirmed present by exact substring match against the retrieved page text. Nothing in the section is unverified. Two things the re-fetch turned up that re-reading the prose would not have: - Washington Technology's robots.txt does not say "Crawl-delay: 1" as this file claimed. It is 30 bytes, blocks GPTBot, and has no wildcard group at all; the crawl-delay was carried over in error from the govexec.com and nextgov.com files. Corrected, and the correction is recorded rather than quietly applied. - The GEMG terms live at govexec.com/about/terms-and-conditions/. The obvious paths -- /terms/, /terms-service/, /about/terms/, and the nextgov and washingtontechnology equivalents -- all 404, which is why the page is hard to find. The working URL is now recorded so the next reader does not repeat the search. The Scoop News Group basis is relabelled from "no terms page published" to "no terms page found", with the five URLs tried and the 404s recorded. That is a negative finding, not a positive grant, and the file now says so and notes it makes those three the weaker-based of the five. Also fixes a dangling reference: ANALYSIS_RULES.md is at src/prompts/ANALYSIS_RULES.md, not the repo root. No change to which sources ship, the feed config, or the tests.
…tput The GEMG feed grant is conditional on attribution including a link back to the original URL. The renderers do not emit article URLs -- a brief names its sources but does not link them. Defensible for a private terminal brief with no audience; not defensible for anything published, and it is a second condition on top of the personal-use limit. Verified against https://www.govexec.com/about/terms-and-conditions/ today. Both conditions must be met before GEMG-sourced content is published.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
backlog/009-federal-it-trade-press.md.The failure this closes
The compliance beat's first live brief missed the reinstatement of the FedRAMP director — the largest event in its declared domain that week. Across 50,983 articles there were 3 incidental FedRAMP mentions and zero in two weeks, because across 207 feeds there was not one federal-IT trade outlet. Every source matching "Fed" was a primary-document publisher: Federal Register, Federal Reserve, Federal Trade Commission.
The beat could see published rules. It could not see news.
The evidence that matters — not a count
Task 005 passed while being blind by reporting document volume. So the proof here is the specific event, verified independently against the corpus:
Five outlets added: FedScoop, DefenseScoop, CyberScoop, Nextgov/FCW–Cybersecurity, Washington Technology.
tests/config/test_beats.pynow asserts the beat resolves at least one non-.govtrade source, so a future config edit that silently drops them fails CI — the gate bites, verified by removing the file:Licensing — every claim now carries its source
The first pass quoted terms verbatim with no URL behind them. The reviewer independently flagged this, marking its confidence medium because its sandbox had no network and it could only confirm the quotes were internally consistent. I could not find the GEMG terms page at any of five plausible URLs.
All eleven licence and
robots.txtclaims were re-fetched, each returning HTTP 200, each quoted phrase confirmed by exact substring match, each recorded inSOURCES.mdwith its URL and read-date. I then verified the highest-stakes one myself, at the URL that was missing:https://www.govexec.com/about/terms-and-conditions/Re-fetching caught a factual error that re-reading would not have. The table claimed
Crawl-delay: 1for Washington Technology'srobots.txt. It says no such thing — 30 bytes,GPTBotonly, no wildcard group. The value had been carried over from a sibling file. Corrected, with the correction recorded rather than quietly applied.A negative claim was also softened: "publishes no terms page" → "no terms page found", with the five URLs tried and their 404s listed, plus a note that this makes the three Scoop News Group sources the weaker-based of the five.
Two limits on publication, recorded not waived
Nextgov/FCW and Washington Technology are personal-use-only. The grant covers today's private terminal brief. It does not reach a published product — that needs written permission via
feedback@govexec.com, which the terms provide as an explicit pathway.The attribution condition is not currently met. The grant asks for attribution "including at a minimum a linked logo, a link that returns to the original URL." The renderers do not emit article URLs — a brief names its sources but does not link them. Fine for a private brief with no audience; not fine for anything published. Recorded in
SOURCES.md.Rejected on terms
robots.txtblocklistsClaudeBot,Claude-Web,anthropic-aiby name. InsightWeaver sends none of those agents and the wildcard group permits the feed, so it is a stated wish rather than a barrier — but a wish about precisely this use. The file also notes the blocklist is a third-party template still containing its own authoring instructions, which cuts against the rejection and is recorded for that reason.Verified
make checkexit 0, 853 tests. Beat resolves 14 feeds. AP/Reuters remain excluded — the existing test asserting that still passes.Two of five outlets ship summary-only
content(dek, not full text); disclosed per outlet inSOURCES.md.