Skip to content

sources: add federal-IT trade press, so the beat can see events - #16

Merged
sam-aydlette merged 3 commits into
mainfrom
task/009-trade-press
Aug 28, 2026
Merged

sam-aydlette merged 3 commits into
mainfrom
task/009-trade-press

Conversation

@sam-aydlette

@sam-aydlette sam-aydlette commented Aug 27, 2026 •

Copy link
Copy Markdown
Owner

Implements backlog/009-federal-it-trade-press.md.

The failure this closes

The compliance beat's first live brief missed the reinstatement of the FedRAMP director — the largest event in its declared domain that week. Across 50,983 articles there were 3 incidental FedRAMP mentions and zero in two weeks, because across 207 feeds there was not one federal-IT trade outlet. Every source matching "Fed" was a primary-document publisher: Federal Register, Federal Reserve, Federal Trade Commission.

The beat could see published rules. It could not see news.

The evidence that matters — not a count

Task 005 passed while being blind by reporting document volume. So the proof here is the specific event, verified independently against the corpus:

[FedScoop] 2026-08-26  320w
FedRAMP Director reinstated at GSA after weeks of leave  [name redacted]

Five outlets added: FedScoop, DefenseScoop, CyberScoop, Nextgov/FCW–Cybersecurity, Washington Technology. tests/config/test_beats.py now asserts the beat resolves at least one non-.gov trade source, so a future config edit that silently drops them fails CI — the gate bites, verified by removing the file:

AssertionError: federal-IT trade outlets missing from the beat:
{'Nextgov/FCW - Cybersecurity', 'CyberScoop', 'FedScoop', 'DefenseScoop', 'Washington Technology'}

Licensing — every claim now carries its source

The first pass quoted terms verbatim with no URL behind them. The reviewer independently flagged this, marking its confidence medium because its sandbox had no network and it could only confirm the quotes were internally consistent. I could not find the GEMG terms page at any of five plausible URLs.

All eleven licence and robots.txt claims were re-fetched, each returning HTTP 200, each quoted phrase confirmed by exact substring match, each recorded in SOURCES.md with its URL and read-date. I then verified the highest-stakes one myself, at the URL that was missing:

https://www.govexec.com/about/terms-and-conditions/

"Individuals may use the Feeds free of charge — unless otherwise specified — for their personal, non-commercial use only."
"Obtain, copy, monitor, index or data mine through the use of a robot, spider, any automated device, or any manual process, the Sites or the contents (except as expressly permitted by GEMG)."

Re-fetching caught a factual error that re-reading would not have. The table claimed Crawl-delay: 1 for Washington Technology's robots.txt. It says no such thing — 30 bytes, GPTBot only, no wildcard group. The value had been carried over from a sibling file. Corrected, with the correction recorded rather than quietly applied.

A negative claim was also softened: "publishes no terms page" → "no terms page found", with the five URLs tried and their 404s listed, plus a note that this makes the three Scoop News Group sources the weaker-based of the five.

Two limits on publication, recorded not waived

Nextgov/FCW and Washington Technology are personal-use-only. The grant covers today's private terminal brief. It does not reach a published product — that needs written permission via feedback@govexec.com, which the terms provide as an explicit pathway.

The attribution condition is not currently met. The grant asks for attribution "including at a minimum a linked logo, a link that returns to the original URL." The renderers do not emit article URLs — a brief names its sources but does not link them. Fine for a private brief with no audience; not fine for anything published. Recorded in SOURCES.md.

Rejected on terms

  • MeriTalk — forbids obtaining, monitoring, indexing or data mining by automated means, with no RSS carve-out (verified; the string "RSS" appears nowhere on the page). A real loss: full-text and directly on-topic.
  • Federal News Network — robots.txt blocklists ClaudeBot, Claude-Web, anthropic-ai by name. InsightWeaver sends none of those agents and the wildcard group permits the feed, so it is a stated wish rather than a barrier — but a wish about precisely this use. The file also notes the blocklist is a third-party template still containing its own authoring instructions, which cuts against the rejection and is recorded for that reason.

Verified

make check exit 0, 853 tests. Beat resolves 14 feeds. AP/Reuters remain excluded — the existing test asserting that still passes.

Two of five outlets ship summary-only content (dek, not full text); disclosed per outlet in SOURCES.md.

The us-public-sector-compliance beat resolved nine sources, all .gov
primary-document publishers. It could see published rules and nothing
else, which is why its first live brief missed the reinstatement of the
FedRAMP director: across 50,983 stored articles there were three
incidental FedRAMP mentions and none within two weeks of the event.

config/feeds/domains/federal_it.json adds five trade feeds tagged into
the beat's existing selectors: FedScoop, DefenseScoop, CyberScoop,
Nextgov/FCW - Cybersecurity, Washington Technology. All five resolve and
all five returned a non-zero live count (10/10/10/5/11 over a four-day
lookback). FedScoop's feed carried the missed event itself, stored with
320 words of body text.

tests/config/test_beats.py asserts the beat resolves at least one
non-.gov source and names all five, so a feed-config edit that silently
drops them fails CI. Verified by removing the config file and watching
the test fail. The feed ceiling rises from 12 to 20; the AP/Reuters
exclusion is re-asserted in the new test, unchanged.

SOURCES.md records a basis for each, a new rule 5, a new
"Publisher RSS - personal use only" class, and the three candidates
rejected on their terms rather than on a technical failure:

- Federal News Network: robots.txt disallows ClaudeBot, Claude-Web,
  anthropic-ai and Claude by name. Not a barrier to this client, but a
  plain statement about this use.
- MeriTalk: terms forbid automated monitoring or indexing of the site
  with no RSS carve-out.
- Government Executive: same terms as Nextgov/FCW and shippable on the
  same basis, left out because its feed is workforce news, not federal
  IT.

Two things a reviewer must decide rather than accept. Nextgov/FCW and
Washington Technology are configured under a GEMG feed licence limited
to personal, non-commercial use; that covers today's terminal brief and
not publication. And those two feeds' content:encoded arrives with
escaped CDATA, so feedparser sanitizes the body away: 0 of 16 stored
rows have non-empty content, and only the publisher's one-sentence dek
survives. Recorded, not worked around; fixing it means changing
src/rss/fetcher.py for every configured feed.
…tatus

SOURCES.md quoted licence terms verbatim -- the GEMG feed grant, the
"except as expressly permitted by GEMG" clause, MeriTalk's
automated-access prohibition -- with no URL attached to any of them. In
the file that decides whether a source may be used, in a public repo,
that is the licensing equivalent of a synthesis claim with no citation,
which src/prompts/ANALYSIS_RULES.md already forbids for the tool's own
output.

Adds a provenance table to the federal-IT trade press section: eleven
claims, each with the exact URL, the read date, and whether it was
confirmed by retrieving that URL. All eleven were re-fetched in this
session, every one returned HTTP 200, and every quoted phrase was
confirmed present by exact substring match against the retrieved page
text. Nothing in the section is unverified.

Two things the re-fetch turned up that re-reading the prose would not
have:

- Washington Technology's robots.txt does not say "Crawl-delay: 1" as
  this file claimed. It is 30 bytes, blocks GPTBot, and has no wildcard
  group at all; the crawl-delay was carried over in error from the
  govexec.com and nextgov.com files. Corrected, and the correction is
  recorded rather than quietly applied.
- The GEMG terms live at govexec.com/about/terms-and-conditions/. The
  obvious paths -- /terms/, /terms-service/, /about/terms/, and the
  nextgov and washingtontechnology equivalents -- all 404, which is why
  the page is hard to find. The working URL is now recorded so the next
  reader does not repeat the search.

The Scoop News Group basis is relabelled from "no terms page published"
to "no terms page found", with the five URLs tried and the 404s
recorded. That is a negative finding, not a positive grant, and the file
now says so and notes it makes those three the weaker-based of the five.

Also fixes a dangling reference: ANALYSIS_RULES.md is at
src/prompts/ANALYSIS_RULES.md, not the repo root.

No change to which sources ship, the feed config, or the tests.
…tput

The GEMG feed grant is conditional on attribution including a link back to
the original URL. The renderers do not emit article URLs -- a brief names its
sources but does not link them. Defensible for a private terminal brief with
no audience; not defensible for anything published, and it is a second
condition on top of the personal-use limit.

Verified against https://www.govexec.com/about/terms-and-conditions/ today.
Both conditions must be met before GEMG-sourced content is published.
@sam-aydlette
sam-aydlette merged commit afaeb48 into main Aug 28, 2026
5 checks passed
@sam-aydlette
sam-aydlette deleted the task/009-trade-press branch August 28, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant