From a994d776f336d7011d5e29bb4742952e1ff30d14 Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Thu, 27 Aug 2026 13:42:14 -0400 Subject: [PATCH 1/3] feeds: add federal-IT trade press so the compliance beat can see events The us-public-sector-compliance beat resolved nine sources, all .gov primary-document publishers. It could see published rules and nothing else, which is why its first live brief missed the reinstatement of the FedRAMP director: across 50,983 stored articles there were three incidental FedRAMP mentions and none within two weeks of the event. config/feeds/domains/federal_it.json adds five trade feeds tagged into the beat's existing selectors: FedScoop, DefenseScoop, CyberScoop, Nextgov/FCW - Cybersecurity, Washington Technology. All five resolve and all five returned a non-zero live count (10/10/10/5/11 over a four-day lookback). FedScoop's feed carried the missed event itself, stored with 320 words of body text. tests/config/test_beats.py asserts the beat resolves at least one non-.gov source and names all five, so a feed-config edit that silently drops them fails CI. Verified by removing the config file and watching the test fail. The feed ceiling rises from 12 to 20; the AP/Reuters exclusion is re-asserted in the new test, unchanged. SOURCES.md records a basis for each, a new rule 5, a new "Publisher RSS - personal use only" class, and the three candidates rejected on their terms rather than on a technical failure: - Federal News Network: robots.txt disallows ClaudeBot, Claude-Web, anthropic-ai and Claude by name. Not a barrier to this client, but a plain statement about this use. - MeriTalk: terms forbid automated monitoring or indexing of the site with no RSS carve-out. - Government Executive: same terms as Nextgov/FCW and shippable on the same basis, left out because its feed is workforce news, not federal IT. Two things a reviewer must decide rather than accept. Nextgov/FCW and Washington Technology are configured under a GEMG feed licence limited to personal, non-commercial use; that covers today's terminal brief and not publication. And those two feeds' content:encoded arrives with escaped CDATA, so feedparser sanitizes the body away: 0 of 16 stored rows have non-empty content, and only the publisher's one-sentence dek survives. Recorded, not worked around; fixing it means changing src/rss/fetcher.py for every configured feed. --- SOURCES.md | 95 ++++++++++++++++++++++++++- backlog/009-federal-it-trade-press.md | 33 ++++++++++ config/feeds/domains/federal_it.json | 60 +++++++++++++++++ tests/config/test_beats.py | 60 +++++++++++++++-- 4 files changed, 241 insertions(+), 7 deletions(-) create mode 100644 config/feeds/domains/federal_it.json diff --git a/SOURCES.md b/SOURCES.md index 1017bad..ff6ed24 100644 --- a/SOURCES.md +++ b/SOURCES.md @@ -19,6 +19,7 @@ operator gets actual counsel before anything publishes. | **US public body** | A publication of a US state or local government body, retrieved through its own published feed. Not federal, so 17 U.S.C. 105 does not apply; the basis is the published feed. | | **Intergovernmental org** | Public-information output of an intergovernmental organization, retrieved through its own published feed. | | **Publisher RSS** | A feed the publisher chose to publish. Offering an RSS feed is an explicit syndication offer for headline, summary and link. It is **not** a licence to republish full text, and InsightWeaver does not: it stores what the feed served and produces analysis over it. | +| **Publisher RSS — personal use only** | A publisher's own feed, offered on terms that limit use of the feed to personal, non-commercial purposes and ask for attribution. Retrieval and analysis for the private brief are expressly permitted; the licence does **not** reach a published product, and doing so needs the publisher's written permission first. Added 2026-08-27 for the Government Executive Media Group titles, whose terms grant the feeds explicitly and narrowly. | | **Commercial wire** | Associated Press and Reuters. Their terms are restrictive about derivative works and redistribution. **They may remain in the personal brief, which is private use. They must not be selected into a beat whose output is published.** | ## Rules this file encodes @@ -55,6 +56,13 @@ operator gets actual counsel before anything publishes. `InsightWeaver/0.1 (+https://github.com/sam-aydlette/InsightWeaver)` and rate-limits itself to one request per second. The Federal Register API requires no key and publishes no quota, so that discipline is ours to impose. +5. **A publisher's own stated wishes outrank the fact that a feed is technically fetchable.** + Added 2026-08-27 with backlog task 009. Six federal-IT trade outlets were assessed for that + task and three were rejected, none of them for a technical reason — the RSS was reachable and + parsed in every case. The record of what was read, and what it said, is in + *Federal-IT trade press — the detailed basis* below. Rejecting a source on its terms is a + normal outcome, and the absence of a source from this file's table is not evidence that nobody + looked at it. ## Adapters @@ -81,10 +89,76 @@ operator gets actual counsel before anything publishes. filter. The filter is data, not code, and each query records its rationale in `config/sources/federal_register.json`. +### Federal-IT trade press — the detailed basis + +Added 2026-08-27 with backlog task 009, which exists because the `us-public-sector-compliance` +beat's first live brief missed the reinstatement of the FedRAMP director. Not a ranking failure: +the article was never ingested, because none of the 207 configured feeds was a federal-IT trade +outlet. Everything matching "Fed" was a primary-document publisher. + +Six candidates were assessed. Every one of them serves reachable, parseable RSS, so the decisions +below are about terms and about what the feed actually carries, never about whether the fetch +worked. All checks were made on 2026-08-27 with the honest User-Agent from rule 4. + +**Configured** — `config/feeds/domains/federal_it.json`: + +| Outlet | Publisher | `robots.txt` for `*` | Terms read | What the feed carries | +| --- | --- | --- | --- | --- | +| FedScoop | Scoop News Group | `Disallow:` (nothing disallowed) | No terms-of-use page published on the site | Full `content:encoded`; median 410 stored words | +| DefenseScoop | Scoop News Group | `Disallow:` | as above | Full `content:encoded`; median 586 stored words | +| CyberScoop | Scoop News Group | `Disallow:` | as above | Full `content:encoded`; median 599 stored words | +| Nextgov/FCW — Cybersecurity | Government Executive Media Group | `Crawl-delay: 1`, feed path allowed | `govexec.com/about/terms-and-conditions/` | Headline, one-sentence dek, link. **No body** — see the truncation note | +| Washington Technology | Government Executive Media Group | `Crawl-delay: 1`, feed path allowed | as above | Headline, one-sentence dek, link. **No body** | + +- **Scoop News Group (FedScoop, DefenseScoop, CyberScoop)** — basis: the publisher's own RSS feed, + the ordinary `Publisher RSS` class. `robots.txt` disallows `GPTBot` and nothing else; the + wildcard group is an empty `Disallow:`. No terms-of-service or terms-of-use page was found on + the site, so there is no per-site reading to record beyond the feed itself. +- **Government Executive Media Group (Nextgov/FCW, Washington Technology)** — class + `Publisher RSS — personal use only`, because their terms grant the feeds in those exact words: + *"Some of the Sites may provide an RSS feed (the 'Feeds'). Individuals may use the Feeds free of + charge — unless otherwise specified — for their personal, non-commercial use only,"* with a + request for attribution to the specific brand and an address for commercial-use requests + (`feedback@govexec.com`). The same terms otherwise prohibit obtaining or monitoring the Sites by + automated means *"except as expressly permitted by GEMG"* — and the Feeds are that express + permission, for personal non-commercial use. **This is a real limit and it is recorded, not + waived: the current brief is private use and is inside it. Before any beat carrying these + sources publishes, the operator must either get written permission or drop them.** The + `us-public-sector-compliance` beat is the beat this file already treats as publication-bound + (rule 2), so this is the constraint most likely to bite, and it is the one an operator should + resolve rather than a reviewer wave through. +- **The truncation note.** The GEMG feeds do ship a `` element, but its CDATA + markers are themselves escaped, so `feedparser` sanitizes the element to an empty string and the + stored `content` and `normalized_content` are empty with `word_count` 0. Verified 2026-08-27 by + storing them: 0 of 5 Nextgov rows and 0 of 11 Washington Technology rows have non-empty content, + while `description` is populated on every row with the publisher's own one-sentence dek. So they + are headline-plus-dek sources, not full-text ones. They are still configured, because the + headline and dek are what makes an *event* visible to the beat at all, and that was the gap. + They are recorded here so nobody mistakes them for analysable body text. Not fixed here: + correcting `src/rss/fetcher.py` to recover the escaped CDATA would change ingestion for all + configured feeds and is outside task 009. + +**Assessed and rejected** — deliberately not configured. Recorded so the absences read as +decisions: + +| Outlet | Why not | +| --- | --- | +| Federal News Network | Their `robots.txt` carries an AI-crawler blocklist that names Anthropic's agents specifically — `ClaudeBot`, `Claude-Web`, `anthropic-ai` and `Claude`, each `Disallow: /`. InsightWeaver does not send those User-Agents and the wildcard group does allow the feed path, so this is not a technical prohibition. It is a plainly stated wish not to have this publisher's content collected for LLM processing, which is exactly what this pipeline does with it, and rule 5 says the wish wins. Their feed is also summary-only (median ~22 words), so little is lost. Revisit only if the publisher's position changes. | +| MeriTalk | Their terms of use prohibit, without any RSS carve-out: *"Obtain, copy, monitor, index, or data mine through the use of a robot, spider, any automated device, or any manual process, the Web site or the contents (except as expressly permitted by MeriTalk)."* Unlike GEMG there is no clause granting the feed, so automated retrieval is forbidden and the source is out of scope by task 009's own terms. Full-text feed and good federal-IT coverage; a written permission request is the only route in. | +| Government Executive | Same publisher and same terms as Nextgov/FCW, so it could have shipped on the same basis. Left out on editorial grounds instead: its all-content feed is federal workforce and management news, which is a different subject from this beat's, and 25 items a fetch of it would crowd the curation window without adding compliance signal. Add it if a workforce beat is ever declared. | + +**Volume.** The five configured feeds returned 46 items on the first live fetch across a four-day +lookback (10 + 10 + 10 + 5 + 11), which is roughly 15 to 20 new items a day in steady state +against the beat's 50-article curation window. Topic feeds were preferred over firehoses where the +outlet offered one: Nextgov/FCW is its cybersecurity feed rather than `/rss/all/`. Washington +Technology has no topic feeds — `/rss/cybersecurity/` and `/rss/topic/policy/` both 404 — so it is +configured as `/rss/all/`. + ## All configured sources -Generated from `config/feeds/**/*.json` on 2026-08-26. Duplicate entries (the same source listed -in more than one tag file) appear once. +Generated from `config/feeds/**/*.json` on 2026-08-26, and updated 2026-08-27 with the five +federal-IT trade sources added by task 009. Duplicate entries (the same source listed in more than +one tag file) appear once. | Source | URL | Adapter | Class | Basis for retrieval and derived analysis | | --- | --- | --- | --- | --- | @@ -125,11 +199,14 @@ in more than one tag file) appear once. | Bleeping Computer | `https://www.bleepingcomputer.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Blue Virginia | `https://bluevirginia.us/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Cardinal News | `https://cardinalnews.org/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | +| CyberScoop | `https://cyberscoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` disallows only `GPTBot`; no site terms-of-use page published. Added 2026-08-27 (task 009). | | Cybersecurity Dive | `https://www.cybersecuritydive.com/feeds/news/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Dark Reading | `https://www.darkreading.com/rss.xml` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | DCist | `https://dcist.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | +| DefenseScoop | `https://defensescoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` disallows only `GPTBot`; no site terms-of-use page published. Added 2026-08-27 (task 009). | | Education Week | `https://www.edweek.org/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | El Faro English (El Salvador) | `https://elfaro.net/en/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | +| FedScoop | `https://fedscoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` disallows only `GPTBot`; no site terms-of-use page published. Added 2026-08-27 (task 009). | | FFXnow - Fairfax County News | `https://www.ffxnow.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | France 24 English | `https://www.france24.com/en/rss` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Greater Greater Washington | `https://ggwash.org/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | @@ -138,6 +215,7 @@ in more than one tag file) appear once. | Kathimerini English Edition (Greece) | `https://www.ekathimerini.com/rss` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Krebs on Security | `https://krebsonsecurity.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | MIT Technology Review | `https://www.technologyreview.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | +| Nextgov/FCW - Cybersecurity | `https://www.nextgov.com/rss/cybersecurity/` | `rss` | Publisher RSS — personal use only | **Limited.** Publisher's own feed, granted by GEMG's terms for *personal, non-commercial use only*, with attribution requested. Fine for the private brief; **must not be carried into a published beat without written permission** (`feedback@govexec.com`). Headline and dek only -- no body text survives parsing. Added 2026-08-27 (task 009). | | NHK World Japan | `https://www3.nhk.or.jp/nhkworld/en/news/rss.xml` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Potomac Local News | `https://www.potomaclocal.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Prince William Living | `https://princewilliamliving.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | @@ -151,6 +229,7 @@ in more than one tag file) appear once. | Virginia Mercury | `https://www.virginiamercury.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Washington Post - Local (DC/MD/VA) | `https://feeds.washingtonpost.com/rss/local` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Washington Post Local (DC/MD/VA) | `https://feeds.washingtonpost.com/rss/local` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | +| Washington Technology | `https://washingtontechnology.com/rss/all/` | `rss` | Publisher RSS — personal use only | **Limited.** Publisher's own feed, granted by GEMG's terms for *personal, non-commercial use only*, with attribution requested. Fine for the private brief; **must not be carried into a published beat without written permission** (`feedback@govexec.com`). Headline and dek only -- no body text survives parsing. Added 2026-08-27 (task 009). | | WAVY Virginia Politics | `https://www.wavy.com/news/politics/virginia-politics/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | What's Up Prince William (Woodbridge) | `https://whatsupwoodbridge.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Wired | `https://www.wired.com/feed/rss` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | @@ -158,7 +237,7 @@ in more than one tag file) appear once. | Associated Press | `https://feeds.ap.org/rss/apf-topnews.rss` | `rss` | Commercial wire | **Restricted.** Personal brief only; excluded from any published beat. | | Reuters | `https://feeds.reuters.com/reuters/topNews` | `rss` | Commercial wire | **Restricted.** Personal brief only; excluded from any published beat. | - + ## What has and has not been verified @@ -167,6 +246,16 @@ Stated plainly, because an unverified claim recorded as a fact is worse than no - **Verified by reading:** that the Federal Register API is keyless, reachable and returns the fields listed above (exercised 2026-08-26); that AP and Reuters carry no tag the `us-public-sector-compliance` beat selects on. +- **Verified by reading, 2026-08-27 (task 009):** the `robots.txt` of all six assessed federal-IT + trade outlets; the GEMG terms and conditions; the MeriTalk terms of use. Verified by fetching: + that each of the five configured trade feeds returns a non-zero article count and stores rows, + and that `content` is empty on the two GEMG feeds and populated on the three Scoop News Group + feeds. Verified by fetching and reading the stored row: that the FedScoop feed carries + *"FedRAMP Director Pete Waterman reinstated at GSA after weeks of leave"* with 320 words of body + text — the specific event the beat was blind to. +- **Not verified:** whether Scoop News Group publishes terms of service anywhere. None was found + linked from `fedscoop.com`; absence of a found page is not proof that no terms exist. Their + `robots.txt` was read and is permissive. - **Not individually verified:** the terms of service of each of the ~60 `Publisher RSS` sources. Their basis is the general one — the publisher chose to publish a feed — not a per-site reading. Any source promoted into a beat that publishes needs its own reading before that beat publishes. diff --git a/backlog/009-federal-it-trade-press.md b/backlog/009-federal-it-trade-press.md index c952e29..40ba970 100644 --- a/backlog/009-federal-it-trade-press.md +++ b/backlog/009-federal-it-trade-press.md @@ -61,3 +61,36 @@ All four declared questions are rule-shaped: statutory dates, authorization stat deadlines, framework divergence. None asks who runs a program or whether an office changed posture. That is a spec-authoring blind spot mirroring the source one. Out of scope here, noted so it is not lost: it belongs with a revision of `config/beats/us-public-sector-compliance.json`. + +## Outcome, 2026-08-27 (branch `task/009-trade-press`, STATUS left QUEUED for review) + +Five trade feeds configured in `config/feeds/domains/federal_it.json`, all resolving into the +beat, all returning a non-zero live count. `tests/config/test_beats.py` now asserts the beat +resolves at least one non-`.gov` source and names all five, and the assertion was checked by +removing the config file and watching it fail. + +**The evidence that matters is not the count.** FedScoop's feed carried +*"FedRAMP Director Pete Waterman reinstated at GSA after weeks of leave"*, and it stored with 320 +words of body text. That is the exact event this task exists because the beat missed. + +**Two decisions a reviewer must make rather than accept:** + +1. **Government Executive Media Group (Nextgov/FCW, Washington Technology) are configured under a + `personal, non-commercial use only` feed licence.** Their terms grant the RSS feeds in those + words. That covers today's private terminal brief and does not cover publication. If this beat + is meant to publish, these two need written permission (`feedback@govexec.com`) or removal. + Recorded in `SOURCES.md`; not something an IC should decide alone. +2. **Federal News Network was rejected**, though it is a named candidate. Its `robots.txt` names + `ClaudeBot`, `Claude-Web`, `anthropic-ai` and `Claude` with `Disallow: /`. InsightWeaver does + not send those User-Agents and the wildcard group permits the feed, so this is a wish rather + than a barrier — but it is a wish about exactly this use. MeriTalk was rejected on its terms, + which forbid automated monitoring with no RSS carve-out. + +**Known limitation, not fixed here.** The two GEMG feeds ship `` with escaped +CDATA markers, so `feedparser` sanitizes the body to an empty string: 0 of 16 stored rows have +non-empty `content`, while `description` carries the publisher's one-sentence dek. They are +headline-plus-dek sources. Recovering the body would mean changing `src/rss/fetcher.py` for all +configured feeds, which is outside this task. Worth its own backlog entry. + +The four standing questions are still all rule-shaped, as this file already notes. Adding the +sources does not make the beat ask who runs a program. That remains open. diff --git a/config/feeds/domains/federal_it.json b/config/feeds/domains/federal_it.json new file mode 100644 index 0000000..0a91849 --- /dev/null +++ b/config/feeds/domains/federal_it.json @@ -0,0 +1,60 @@ +{ + "description": "Federal IT trade press - the outlets that report federal technology, cybersecurity and acquisition as news rather than publishing it as a document. Added 2026-08-27 by backlog task 009, after the us-public-sector-compliance beat's first live brief missed the reinstatement of the FedRAMP director: across 207 feeds not one federal-IT trade outlet was configured, so the beat could see published rules and nothing else. Every entry here is tagged into that beat's existing selectors (federal_policy / regulatory / cybersecurity, geo usa). Basis for use for each is recorded in SOURCES.md; candidates rejected on licence grounds are recorded there too, so the absences are not mistaken for oversights.", + "feeds": [ + { + "name": "FedScoop", + "url": "https://fedscoop.com/feed/", + "applicability": { + "scope": ["national"], + "geo_tags": ["usa"], + "domain_tags": ["federal_policy", "technology", "cybersecurity", "regulatory"], + "specialty_tags": ["federal_it"] + }, + "relevance_score": 0.9 + }, + { + "name": "DefenseScoop", + "url": "https://defensescoop.com/feed/", + "applicability": { + "scope": ["national"], + "geo_tags": ["usa"], + "domain_tags": ["federal_policy", "technology", "cybersecurity"], + "specialty_tags": ["federal_it", "defense"] + }, + "relevance_score": 0.8 + }, + { + "name": "CyberScoop", + "url": "https://cyberscoop.com/feed/", + "applicability": { + "scope": ["national"], + "geo_tags": ["usa"], + "domain_tags": ["cybersecurity", "federal_policy"], + "specialty_tags": ["federal_it", "threat_intelligence"] + }, + "relevance_score": 0.7 + }, + { + "name": "Nextgov/FCW - Cybersecurity", + "url": "https://www.nextgov.com/rss/cybersecurity/", + "applicability": { + "scope": ["national"], + "geo_tags": ["usa"], + "domain_tags": ["cybersecurity", "federal_policy"], + "specialty_tags": ["federal_it"] + }, + "relevance_score": 0.8 + }, + { + "name": "Washington Technology", + "url": "https://washingtontechnology.com/rss/all/", + "applicability": { + "scope": ["national"], + "geo_tags": ["usa"], + "domain_tags": ["federal_policy", "technology", "regulatory"], + "specialty_tags": ["federal_it", "acquisition"] + }, + "relevance_score": 0.7 + } + ] +} diff --git a/tests/config/test_beats.py b/tests/config/test_beats.py index e37cef6..8715567 100644 --- a/tests/config/test_beats.py +++ b/tests/config/test_beats.py @@ -9,6 +9,7 @@ """ import json +from urllib.parse import urlparse import pytest @@ -30,6 +31,11 @@ SHIPPED_BEAT = "us-public-sector-compliance" +def _host(url): + """The lowercased hostname of a feed URL, minus any trailing dot.""" + return (urlparse(url).hostname or "").lower().rstrip(".") + + def write_beat(directory, name, payload): """Write a beat file, accepting a dict or a raw string (for bad JSON).""" directory.mkdir(parents=True, exist_ok=True) @@ -374,10 +380,13 @@ def test_resolves_to_a_small_us_federal_feed_set(self): feeds = load_beat(SHIPPED_BEAT).resolve_feeds() names = {feed.name for feed in feeds} - # Thin by construction: the domain does not publish much RSS. If this - # ever exceeds a dozen feeds, the beat has drifted into being a - # general news beat and the selector needs re-reading. - assert 0 < len(feeds) <= 12 + # Thin by construction: the domain does not publish much RSS. The + # ceiling was 12 until 2026-08-27, when backlog task 009 added five + # federal-IT trade feeds; raised to 20 to leave headroom for one more + # outlet without becoming a licence to grow indefinitely. If this ever + # exceeds 20, the beat has drifted into being a general news beat and + # the selector needs re-reading. + assert 0 < len(feeds) <= 20 assert "Federal Register - Public Inspection" in names assert "CISA Cybersecurity Advisories" in names # Added 2026-08-26 (task 005). The RSS half of this beat is nearly @@ -393,6 +402,49 @@ def test_resolves_to_a_small_us_federal_feed_set(self): assert "Krebs on Security" not in names assert "ARLnow - Arlington Local News" not in names + def test_resolves_at_least_one_non_gov_trade_source(self): + """ + Added 2026-08-27 (task 009). Until then every source this beat resolved + was a primary-document publisher on a `.gov` host, and the beat's first + live brief missed the reinstatement of the FedRAMP director: across + 50,983 stored articles there were three incidental FedRAMP mentions and + none within two weeks of the event, because no federal-IT trade outlet + was configured at all. + + This asserts the property that was missing, not the volume that was + already there. A `.gov` feed publishes rules; only a trade outlet + reports who runs a program, whether a deadline is being enforced, or + whether an authorization was pulled. A future edit to config/feeds/ + that drops the trade press fails here rather than showing up as a brief + that is quietly blind again. + """ + feeds = load_beat(SHIPPED_BEAT).resolve_feeds() + + non_gov = [feed for feed in feeds if not _host(feed.url).endswith(".gov")] + assert non_gov, ( + "the beat resolves only .gov sources, so it can see published " + "documents and no reported news -- see backlog/009-federal-it-trade-press.md" + ) + + # Named, so dropping one is a failure and not a shrug. Each has a row + # in SOURCES.md recording its basis for use. + names = {feed.name for feed in feeds} + trade = { + "FedScoop", + "DefenseScoop", + "CyberScoop", + "Nextgov/FCW - Cybersecurity", + "Washington Technology", + } + assert trade <= names, f"federal-IT trade outlets missing from the beat: {trade - names}" + + # And they are still not wires: SOURCES.md rule 2 is unchanged, and + # widening the beat to reach trade press must not widen it to reach AP + # or Reuters. Asserted here as well as above because this test is the + # one that argues for more non-.gov sources. + assert "Associated Press" not in names + assert "Reuters" not in names + def test_resolution_is_deduplicated_by_url(self): feeds = load_beat(SHIPPED_BEAT).resolve_feeds() urls = [feed.url for feed in feeds] From bf07832a039b9c70cf4ebeba9a2a59fe1c1176cd Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Thu, 27 Aug 2026 13:59:09 -0400 Subject: [PATCH 2/3] SOURCES: give every licence claim a URL, a date, and a verification status SOURCES.md quoted licence terms verbatim -- the GEMG feed grant, the "except as expressly permitted by GEMG" clause, MeriTalk's automated-access prohibition -- with no URL attached to any of them. In the file that decides whether a source may be used, in a public repo, that is the licensing equivalent of a synthesis claim with no citation, which src/prompts/ANALYSIS_RULES.md already forbids for the tool's own output. Adds a provenance table to the federal-IT trade press section: eleven claims, each with the exact URL, the read date, and whether it was confirmed by retrieving that URL. All eleven were re-fetched in this session, every one returned HTTP 200, and every quoted phrase was confirmed present by exact substring match against the retrieved page text. Nothing in the section is unverified. Two things the re-fetch turned up that re-reading the prose would not have: - Washington Technology's robots.txt does not say "Crawl-delay: 1" as this file claimed. It is 30 bytes, blocks GPTBot, and has no wildcard group at all; the crawl-delay was carried over in error from the govexec.com and nextgov.com files. Corrected, and the correction is recorded rather than quietly applied. - The GEMG terms live at govexec.com/about/terms-and-conditions/. The obvious paths -- /terms/, /terms-service/, /about/terms/, and the nextgov and washingtontechnology equivalents -- all 404, which is why the page is hard to find. The working URL is now recorded so the next reader does not repeat the search. The Scoop News Group basis is relabelled from "no terms page published" to "no terms page found", with the five URLs tried and the 404s recorded. That is a negative finding, not a positive grant, and the file now says so and notes it makes those three the weaker-based of the five. Also fixes a dangling reference: ANALYSIS_RULES.md is at src/prompts/ANALYSIS_RULES.md, not the repo root. No change to which sources ship, the feed config, or the tests. --- SOURCES.md | 122 +++++++++++++++++++++++++++++++++++------------------ 1 file changed, 82 insertions(+), 40 deletions(-) diff --git a/SOURCES.md b/SOURCES.md index ff6ed24..0d5040e 100644 --- a/SOURCES.md +++ b/SOURCES.md @@ -60,7 +60,12 @@ operator gets actual counsel before anything publishes. Added 2026-08-27 with backlog task 009. Six federal-IT trade outlets were assessed for that task and three were rejected, none of them for a technical reason — the RSS was reachable and parsed in every case. The record of what was read, and what it said, is in - *Federal-IT trade press — the detailed basis* below. Rejecting a source on its terms is a + *Federal-IT trade press — the detailed basis* below. **Every licence claim there carries the + URL it was read at, the date, and whether it was confirmed by retrieving that URL**, because a + quoted licence term with no traceable source is the same defect as a synthesis claim with no + citation — and `src/prompts/ANALYSIS_RULES.md`, injected into every synthesis prompt, already + requires that a single-source claim "name the source explicitly". Hold a licence claim in this + file to the standard the tool holds its own output to. Rejecting a source on its terms is a normal outcome, and the absence of a source from this file's table is not evidence that nobody looked at it. @@ -98,35 +103,61 @@ outlet. Everything matching "Fed" was a primary-document publisher. Six candidates were assessed. Every one of them serves reachable, parseable RSS, so the decisions below are about terms and about what the feed actually carries, never about whether the fetch -worked. All checks were made on 2026-08-27 with the honest User-Agent from rule 4. +worked. + +**Provenance of every licence claim below.** This file is what says a source is permitted to be +used, and a verbatim quote with no traceable source is the licensing equivalent of a synthesis +claim with no citation. So each claim carries the URL it was read at, the date, and whether it was +confirmed by retrieving that URL. Every retrieval used the honest User-Agent from rule 4. + +| # | Claim | URL read | Read on | Status | +| --- | --- | --- | --- | --- | +| 1 | FedScoop `robots.txt`: wildcard group is an empty `Disallow:`; only `GPTBot` is blocked | `https://fedscoop.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 2 | DefenseScoop `robots.txt`: same | `https://defensescoop.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 3 | CyberScoop `robots.txt`: same | `https://cyberscoop.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 4 | Scoop News Group publishes no findable terms page | `https://fedscoop.com/terms-of-service/`, `/terms-of-use/`, `/terms/`, `/privacy-policy/`, and `https://scoopnewsgroup.com/terms/` | 2026-08-27 | **Verified as "not found", not as "does not exist"** — all five returned HTTP 404, and a link scan of the `fedscoop.com` homepage found no terms, privacy, legal or copyright link. A terms page that exists somewhere I did not look would not have shown up. Treat this basis as the feed alone. | +| 5 | Nextgov/FCW `robots.txt`: wildcard group is `Crawl-delay: 1` plus five `Disallow:` paths, none matching `/rss/`; `GPTBot` blocked | `https://www.nextgov.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 6 | Washington Technology `robots.txt`: **30 bytes, `User-agent: GPTBot` / `Disallow: /` and nothing else — there is no wildcard group at all** | `https://www.washingtontechnology.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200. Corrected 2026-08-27: an earlier revision of this file wrongly recorded `Crawl-delay: 1` here by carrying it over from the `govexec.com` and `nextgov.com` files. | +| 7 | Government Executive `robots.txt`: `Crawl-delay: 1`, same five `Disallow:` paths as Nextgov | `https://www.govexec.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 8 | GEMG terms: the RSS grant, the `except as expressly permitted by GEMG` clause, the automated-access prohibition, and `feedback@govexec.com` | `https://www.govexec.com/about/terms-and-conditions/` | 2026-08-27 | **Verified** — fetched, HTTP 200; each quoted phrase confirmed present by exact substring match against the page text. **Note the path**: `govexec.com/terms/`, `/terms-service/`, `/about/terms/` and the `nextgov.com` and `washingtontechnology.com` equivalents all 404. This page is reachable from the Nextgov and Washington Technology footers as "Terms and Conditions" and is the only copy found. | +| 9 | MeriTalk terms: the automated-access prohibition, and that the page contains no RSS carve-out | `https://www.meritalk.com/terms-of-use/` | 2026-08-27 | **Verified** — fetched, HTTP 200; quote confirmed by exact substring match, and the string "RSS" does not appear anywhere on the page, which is what makes "no carve-out" a checked claim rather than an inference | +| 10 | Federal News Network `robots.txt`: `ClaudeBot`, `Claude-Web`, `anthropic-ai` and `Claude` each `Disallow: /` | `https://federalnewsnetwork.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 11 | MeriTalk `robots.txt`: permissive; no AI-crawler block | `https://www.meritalk.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200. The prohibition on MeriTalk is in its terms, not its `robots.txt`. | + +Nothing quoted in this section is unverified. If a future edit adds a licence claim that cannot be +traced to a URL someone actually retrieved, label it unverified in this table rather than leaving +it bare. **Configured** — `config/feeds/domains/federal_it.json`: -| Outlet | Publisher | `robots.txt` for `*` | Terms read | What the feed carries | +| Outlet | Publisher | `robots.txt` for `*` | Terms | What the feed carries | | --- | --- | --- | --- | --- | -| FedScoop | Scoop News Group | `Disallow:` (nothing disallowed) | No terms-of-use page published on the site | Full `content:encoded`; median 410 stored words | -| DefenseScoop | Scoop News Group | `Disallow:` | as above | Full `content:encoded`; median 586 stored words | -| CyberScoop | Scoop News Group | `Disallow:` | as above | Full `content:encoded`; median 599 stored words | -| Nextgov/FCW — Cybersecurity | Government Executive Media Group | `Crawl-delay: 1`, feed path allowed | `govexec.com/about/terms-and-conditions/` | Headline, one-sentence dek, link. **No body** — see the truncation note | -| Washington Technology | Government Executive Media Group | `Crawl-delay: 1`, feed path allowed | as above | Headline, one-sentence dek, link. **No body** | +| FedScoop | Scoop News Group | `Disallow:` (nothing disallowed) | none found — claim 4 | Full `content:encoded`; median 410 stored words | +| DefenseScoop | Scoop News Group | `Disallow:` | none found — claim 4 | Full `content:encoded`; median 586 stored words | +| CyberScoop | Scoop News Group | `Disallow:` | none found — claim 4 | Full `content:encoded`; median 599 stored words | +| Nextgov/FCW — Cybersecurity | Government Executive Media Group | `Crawl-delay: 1`; no `Disallow:` matches `/rss/` | claim 8 | Headline, one-sentence dek, link. **No body** — see the truncation note | +| Washington Technology | Government Executive Media Group | no wildcard group; only `GPTBot` blocked | claim 8 | Headline, one-sentence dek, link. **No body** | - **Scoop News Group (FedScoop, DefenseScoop, CyberScoop)** — basis: the publisher's own RSS feed, the ordinary `Publisher RSS` class. `robots.txt` disallows `GPTBot` and nothing else; the - wildcard group is an empty `Disallow:`. No terms-of-service or terms-of-use page was found on - the site, so there is no per-site reading to record beyond the feed itself. + wildcard group is an empty `Disallow:` (claims 1-3). No terms page was found (claim 4), so the + basis is the feed alone and there is no per-site reading to record. Because that is a negative + finding rather than a positive grant, it is the weaker of the two bases in this section. - **Government Executive Media Group (Nextgov/FCW, Washington Technology)** — class - `Publisher RSS — personal use only`, because their terms grant the feeds in those exact words: - *"Some of the Sites may provide an RSS feed (the 'Feeds'). Individuals may use the Feeds free of - charge — unless otherwise specified — for their personal, non-commercial use only,"* with a - request for attribution to the specific brand and an address for commercial-use requests - (`feedback@govexec.com`). The same terms otherwise prohibit obtaining or monitoring the Sites by - automated means *"except as expressly permitted by GEMG"* — and the Feeds are that express - permission, for personal non-commercial use. **This is a real limit and it is recorded, not - waived: the current brief is private use and is inside it. Before any beat carrying these - sources publishes, the operator must either get written permission or drop them.** The - `us-public-sector-compliance` beat is the beat this file already treats as publication-bound - (rule 2), so this is the constraint most likely to bite, and it is the one an operator should - resolve rather than a reviewer wave through. + `Publisher RSS — personal use only`. Their terms, read at + `https://www.govexec.com/about/terms-and-conditions/` on 2026-08-27 (claim 8), grant the feeds + in these words: *"Some of the Sites may provide an RSS feed (the 'Feeds'). Individuals may use + the Feeds free of charge - unless otherwise specified - for their personal, non-commercial use + only,"* with a request for attribution to the specific brand and an address for commercial-use + requests (`feedback@govexec.com`). The same page otherwise prohibits *"Obtain, copy, monitor, + index or data mine through the use of a robot, spider, any automated device, or any manual + process, the Sites or the contents (except as expressly permitted by GEMG)"* — and the Feeds + clause is that express permission, for personal non-commercial use. **This is a real limit and + it is recorded, not waived: the current brief is private use and is inside it. Before any beat + carrying these sources publishes, the operator must either get written permission or drop + them.** The `us-public-sector-compliance` beat is the beat this file already treats as + publication-bound (rule 2), so this is the constraint most likely to bite, and it is the one an + operator should resolve rather than a reviewer wave through. - **The truncation note.** The GEMG feeds do ship a `` element, but its CDATA markers are themselves escaped, so `feedparser` sanitizes the element to an empty string and the stored `content` and `normalized_content` are empty with `word_count` 0. Verified 2026-08-27 by @@ -143,9 +174,9 @@ decisions: | Outlet | Why not | | --- | --- | -| Federal News Network | Their `robots.txt` carries an AI-crawler blocklist that names Anthropic's agents specifically — `ClaudeBot`, `Claude-Web`, `anthropic-ai` and `Claude`, each `Disallow: /`. InsightWeaver does not send those User-Agents and the wildcard group does allow the feed path, so this is not a technical prohibition. It is a plainly stated wish not to have this publisher's content collected for LLM processing, which is exactly what this pipeline does with it, and rule 5 says the wish wins. Their feed is also summary-only (median ~22 words), so little is lost. Revisit only if the publisher's position changes. | -| MeriTalk | Their terms of use prohibit, without any RSS carve-out: *"Obtain, copy, monitor, index, or data mine through the use of a robot, spider, any automated device, or any manual process, the Web site or the contents (except as expressly permitted by MeriTalk)."* Unlike GEMG there is no clause granting the feed, so automated retrieval is forbidden and the source is out of scope by task 009's own terms. Full-text feed and good federal-IT coverage; a written permission request is the only route in. | -| Government Executive | Same publisher and same terms as Nextgov/FCW, so it could have shipped on the same basis. Left out on editorial grounds instead: its all-content feed is federal workforce and management news, which is a different subject from this beat's, and 25 items a fetch of it would crowd the curation window without adding compliance signal. Add it if a workforce beat is ever declared. | +| Federal News Network | Their `robots.txt` (`https://federalnewsnetwork.com/robots.txt`, read 2026-08-27, claim 10) carries an AI-crawler blocklist naming Anthropic's agents specifically: `ClaudeBot`, `Claude-Web`, `anthropic-ai` and `Claude`, each `Disallow: /`, under the comment *"Claude bot used to collect training data for Anthropic LLMs"*. InsightWeaver does not send those User-Agents and the wildcard group does allow the feed path, so this is not a technical prohibition. It is a plainly stated wish not to have this publisher's content collected for LLM processing, which is what this pipeline does with it, and rule 5 says the wish wins. **In fairness to the other reading:** the blocklist is a third-party template that still contains its own instructions (*"Remove comments before publishing - Prepared by ORDO DIGITAL"*), so it may be boilerplate rather than a considered position. The call was made conservatively; their feed is also summary-only (median ~22 words), so little is lost. Revisit if the publisher's position changes. | +| MeriTalk | Their terms of use (`https://www.meritalk.com/terms-of-use/`, read 2026-08-27, claim 9) prohibit: *"Obtain, copy, monitor, index, or data mine through the use of a robot, spider, any automated device, or any manual process, the Web site or the contents (except as expressly permitted by MeriTalk)."* Unlike GEMG there is no clause granting the feed — the string "RSS" does not appear on the page at all — so automated retrieval is forbidden and the source is out of scope by task 009's own terms. Full-text feed and good federal-IT coverage; a written permission request is the only route in. | +| Government Executive | Same publisher and same terms as Nextgov/FCW (claim 8), so it could have shipped on the same basis. Left out on editorial grounds instead: its all-content feed is federal workforce and management news, which is a different subject from this beat's, and 25 items a fetch of it would crowd the curation window without adding compliance signal. Add it if a workforce beat is ever declared. | **Volume.** The five configured feeds returned 46 items on the first live fetch across a four-day lookback (10 + 10 + 10 + 5 + 11), which is roughly 15 to 20 new items a day in steady state @@ -199,14 +230,14 @@ one tag file) appear once. | Bleeping Computer | `https://www.bleepingcomputer.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Blue Virginia | `https://bluevirginia.us/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Cardinal News | `https://cardinalnews.org/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | -| CyberScoop | `https://cyberscoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` disallows only `GPTBot`; no site terms-of-use page published. Added 2026-08-27 (task 009). | +| CyberScoop | `https://cyberscoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` read 2026-08-27: wildcard group is an empty `Disallow:`, only `GPTBot` blocked. No terms page found (five candidate URLs tried 2026-08-27, all 404) — see claim 4 in *Federal-IT trade press*. Added 2026-08-27 (task 009). | | Cybersecurity Dive | `https://www.cybersecuritydive.com/feeds/news/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Dark Reading | `https://www.darkreading.com/rss.xml` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | DCist | `https://dcist.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | -| DefenseScoop | `https://defensescoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` disallows only `GPTBot`; no site terms-of-use page published. Added 2026-08-27 (task 009). | +| DefenseScoop | `https://defensescoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` read 2026-08-27: wildcard group is an empty `Disallow:`, only `GPTBot` blocked. No terms page found (five candidate URLs tried 2026-08-27, all 404) — see claim 4 in *Federal-IT trade press*. Added 2026-08-27 (task 009). | | Education Week | `https://www.edweek.org/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | El Faro English (El Salvador) | `https://elfaro.net/en/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | -| FedScoop | `https://fedscoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` disallows only `GPTBot`; no site terms-of-use page published. Added 2026-08-27 (task 009). | +| FedScoop | `https://fedscoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` read 2026-08-27: wildcard group is an empty `Disallow:`, only `GPTBot` blocked. No terms page found (five candidate URLs tried 2026-08-27, all 404) — see claim 4 in *Federal-IT trade press*. Added 2026-08-27 (task 009). | | FFXnow - Fairfax County News | `https://www.ffxnow.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | France 24 English | `https://www.france24.com/en/rss` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Greater Greater Washington | `https://ggwash.org/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | @@ -215,7 +246,7 @@ one tag file) appear once. | Kathimerini English Edition (Greece) | `https://www.ekathimerini.com/rss` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Krebs on Security | `https://krebsonsecurity.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | MIT Technology Review | `https://www.technologyreview.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | -| Nextgov/FCW - Cybersecurity | `https://www.nextgov.com/rss/cybersecurity/` | `rss` | Publisher RSS — personal use only | **Limited.** Publisher's own feed, granted by GEMG's terms for *personal, non-commercial use only*, with attribution requested. Fine for the private brief; **must not be carried into a published beat without written permission** (`feedback@govexec.com`). Headline and dek only -- no body text survives parsing. Added 2026-08-27 (task 009). | +| Nextgov/FCW - Cybersecurity | `https://www.nextgov.com/rss/cybersecurity/` | `rss` | Publisher RSS — personal use only | **Limited.** Publisher's own feed, granted by GEMG's terms for *personal, non-commercial use only*, with attribution requested — terms read at `https://www.govexec.com/about/terms-and-conditions/` on 2026-08-27 and quotes verified against that page (claim 8). Fine for the private brief; **must not be carried into a published beat without written permission** (`feedback@govexec.com`). Headline and dek only -- no body text survives parsing. Added 2026-08-27 (task 009). | | NHK World Japan | `https://www3.nhk.or.jp/nhkworld/en/news/rss.xml` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Potomac Local News | `https://www.potomaclocal.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Prince William Living | `https://princewilliamliving.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | @@ -229,7 +260,7 @@ one tag file) appear once. | Virginia Mercury | `https://www.virginiamercury.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Washington Post - Local (DC/MD/VA) | `https://feeds.washingtonpost.com/rss/local` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Washington Post Local (DC/MD/VA) | `https://feeds.washingtonpost.com/rss/local` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | -| Washington Technology | `https://washingtontechnology.com/rss/all/` | `rss` | Publisher RSS — personal use only | **Limited.** Publisher's own feed, granted by GEMG's terms for *personal, non-commercial use only*, with attribution requested. Fine for the private brief; **must not be carried into a published beat without written permission** (`feedback@govexec.com`). Headline and dek only -- no body text survives parsing. Added 2026-08-27 (task 009). | +| Washington Technology | `https://washingtontechnology.com/rss/all/` | `rss` | Publisher RSS — personal use only | **Limited.** Publisher's own feed, granted by GEMG's terms for *personal, non-commercial use only*, with attribution requested — terms read at `https://www.govexec.com/about/terms-and-conditions/` on 2026-08-27 and quotes verified against that page (claim 8). Fine for the private brief; **must not be carried into a published beat without written permission** (`feedback@govexec.com`). Headline and dek only -- no body text survives parsing. Added 2026-08-27 (task 009). | | WAVY Virginia Politics | `https://www.wavy.com/news/politics/virginia-politics/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | What's Up Prince William (Woodbridge) | `https://whatsupwoodbridge.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Wired | `https://www.wired.com/feed/rss` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | @@ -246,16 +277,27 @@ Stated plainly, because an unverified claim recorded as a fact is worse than no - **Verified by reading:** that the Federal Register API is keyless, reachable and returns the fields listed above (exercised 2026-08-26); that AP and Reuters carry no tag the `us-public-sector-compliance` beat selects on. -- **Verified by reading, 2026-08-27 (task 009):** the `robots.txt` of all six assessed federal-IT - trade outlets; the GEMG terms and conditions; the MeriTalk terms of use. Verified by fetching: - that each of the five configured trade feeds returns a non-zero article count and stores rows, - and that `content` is empty on the two GEMG feeds and populated on the three Scoop News Group - feeds. Verified by fetching and reading the stored row: that the FedScoop feed carries - *"FedRAMP Director Pete Waterman reinstated at GSA after weeks of leave"* with 320 words of body - text — the specific event the beat was blind to. -- **Not verified:** whether Scoop News Group publishes terms of service anywhere. None was found - linked from `fedscoop.com`; absence of a found page is not proof that no terms exist. Their - `robots.txt` was read and is permissive. +- **Verified by retrieval, 2026-08-27 (task 009):** every licence claim in *Federal-IT trade + press — the detailed basis*. Eight `robots.txt` files and two terms pages were fetched, each + returning HTTP 200, and every phrase quoted from them was confirmed present by exact substring + match against the retrieved page. The URL and read-date for each are in the provenance table in + that section. **No quote in this file is unattributed, and none is unverified.** Also verified by + fetching: that each of the five configured trade feeds returns a non-zero article count and + stores rows, and that `content` is empty on the two GEMG feeds and populated on the three Scoop + News Group feeds. Verified by reading the stored row: that the FedScoop feed carries *"FedRAMP + Director Pete Waterman reinstated at GSA after weeks of leave"* with 320 words of body text — + the specific event the beat was blind to. +- **Verified only as a negative:** that Scoop News Group publishes no terms page. Five candidate + URLs were tried and all returned 404, and a link scan of the `fedscoop.com` homepage found no + terms, privacy, legal or copyright link (claim 4). A page that exists somewhere those checks did + not reach would not have been found, so this is "not found", never "does not exist" — and it + makes FedScoop, DefenseScoop and CyberScoop the weaker-based of the five configured sources, + resting on the published feed alone. +- **Corrected 2026-08-27:** an earlier revision of this file recorded `Crawl-delay: 1` for + Washington Technology's `robots.txt`. It does not say that. The file is 30 bytes, blocks + `GPTBot`, and has no wildcard group at all; the `Crawl-delay` was carried over in error from the + `govexec.com` and `nextgov.com` files. Found by re-fetching every cited URL rather than by + re-reading the prose. - **Not individually verified:** the terms of service of each of the ~60 `Publisher RSS` sources. Their basis is the general one — the publisher chose to publish a feed — not a per-site reading. Any source promoted into a beat that publishes needs its own reading before that beat publishes. From b71f444c62e6b3d0706cfa52a83bdc03b19e5c8b Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Thu, 27 Aug 2026 14:00:53 -0400 Subject: [PATCH 3/3] docs: record that the attribution condition is not met in rendered output The GEMG feed grant is conditional on attribution including a link back to the original URL. The renderers do not emit article URLs -- a brief names its sources but does not link them. Defensible for a private terminal brief with no audience; not defensible for anything published, and it is a second condition on top of the personal-use limit. Verified against https://www.govexec.com/about/terms-and-conditions/ today. Both conditions must be met before GEMG-sourced content is published. --- SOURCES.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/SOURCES.md b/SOURCES.md index 0d5040e..5716379 100644 --- a/SOURCES.md +++ b/SOURCES.md @@ -302,3 +302,19 @@ Stated plainly, because an unverified claim recorded as a fact is worse than no Their basis is the general one — the publisher chose to publish a feed — not a per-site reading. Any source promoted into a beat that publishes needs its own reading before that beat publishes. - **Not legal advice**, as stated at the top. + +## The attribution condition is recorded but not yet met in rendered output + +The Government Executive Media Group grant of the Feeds is conditional: it asks for *"attribution +to the specific GEMG brand, including at a minimum a linked logo, a link that returns to the +original URL."* + +**The renderers do not currently emit article URLs.** A brief names its sources -- situations carry +numbered citations and thin-coverage entries name the feed -- but no rendered format links back to +the original. For the private terminal brief that is defensible, since there is no audience the +attribution would reach. It is **not** defensible for anything published, and it is a second +condition on top of the personal-use limit already recorded above. + +Recorded 2026-08-27, verified against `https://www.govexec.com/about/terms-and-conditions/` the +same day. Before any GEMG-sourced content is published, both must be satisfied: written permission +via `feedback@govexec.com`, and linked attribution in the rendered output.