diff --git a/SOURCES.md b/SOURCES.md index 1017bad..5716379 100644 --- a/SOURCES.md +++ b/SOURCES.md @@ -19,6 +19,7 @@ operator gets actual counsel before anything publishes. | **US public body** | A publication of a US state or local government body, retrieved through its own published feed. Not federal, so 17 U.S.C. 105 does not apply; the basis is the published feed. | | **Intergovernmental org** | Public-information output of an intergovernmental organization, retrieved through its own published feed. | | **Publisher RSS** | A feed the publisher chose to publish. Offering an RSS feed is an explicit syndication offer for headline, summary and link. It is **not** a licence to republish full text, and InsightWeaver does not: it stores what the feed served and produces analysis over it. | +| **Publisher RSS — personal use only** | A publisher's own feed, offered on terms that limit use of the feed to personal, non-commercial purposes and ask for attribution. Retrieval and analysis for the private brief are expressly permitted; the licence does **not** reach a published product, and doing so needs the publisher's written permission first. Added 2026-08-27 for the Government Executive Media Group titles, whose terms grant the feeds explicitly and narrowly. | | **Commercial wire** | Associated Press and Reuters. Their terms are restrictive about derivative works and redistribution. **They may remain in the personal brief, which is private use. They must not be selected into a beat whose output is published.** | ## Rules this file encodes @@ -55,6 +56,18 @@ operator gets actual counsel before anything publishes. `InsightWeaver/0.1 (+https://github.com/sam-aydlette/InsightWeaver)` and rate-limits itself to one request per second. The Federal Register API requires no key and publishes no quota, so that discipline is ours to impose. +5. **A publisher's own stated wishes outrank the fact that a feed is technically fetchable.** + Added 2026-08-27 with backlog task 009. Six federal-IT trade outlets were assessed for that + task and three were rejected, none of them for a technical reason — the RSS was reachable and + parsed in every case. The record of what was read, and what it said, is in + *Federal-IT trade press — the detailed basis* below. **Every licence claim there carries the + URL it was read at, the date, and whether it was confirmed by retrieving that URL**, because a + quoted licence term with no traceable source is the same defect as a synthesis claim with no + citation — and `src/prompts/ANALYSIS_RULES.md`, injected into every synthesis prompt, already + requires that a single-source claim "name the source explicitly". Hold a licence claim in this + file to the standard the tool holds its own output to. Rejecting a source on its terms is a + normal outcome, and the absence of a source from this file's table is not evidence that nobody + looked at it. ## Adapters @@ -81,10 +94,102 @@ operator gets actual counsel before anything publishes. filter. The filter is data, not code, and each query records its rationale in `config/sources/federal_register.json`. +### Federal-IT trade press — the detailed basis + +Added 2026-08-27 with backlog task 009, which exists because the `us-public-sector-compliance` +beat's first live brief missed the reinstatement of the FedRAMP director. Not a ranking failure: +the article was never ingested, because none of the 207 configured feeds was a federal-IT trade +outlet. Everything matching "Fed" was a primary-document publisher. + +Six candidates were assessed. Every one of them serves reachable, parseable RSS, so the decisions +below are about terms and about what the feed actually carries, never about whether the fetch +worked. + +**Provenance of every licence claim below.** This file is what says a source is permitted to be +used, and a verbatim quote with no traceable source is the licensing equivalent of a synthesis +claim with no citation. So each claim carries the URL it was read at, the date, and whether it was +confirmed by retrieving that URL. Every retrieval used the honest User-Agent from rule 4. + +| # | Claim | URL read | Read on | Status | +| --- | --- | --- | --- | --- | +| 1 | FedScoop `robots.txt`: wildcard group is an empty `Disallow:`; only `GPTBot` is blocked | `https://fedscoop.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 2 | DefenseScoop `robots.txt`: same | `https://defensescoop.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 3 | CyberScoop `robots.txt`: same | `https://cyberscoop.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 4 | Scoop News Group publishes no findable terms page | `https://fedscoop.com/terms-of-service/`, `/terms-of-use/`, `/terms/`, `/privacy-policy/`, and `https://scoopnewsgroup.com/terms/` | 2026-08-27 | **Verified as "not found", not as "does not exist"** — all five returned HTTP 404, and a link scan of the `fedscoop.com` homepage found no terms, privacy, legal or copyright link. A terms page that exists somewhere I did not look would not have shown up. Treat this basis as the feed alone. | +| 5 | Nextgov/FCW `robots.txt`: wildcard group is `Crawl-delay: 1` plus five `Disallow:` paths, none matching `/rss/`; `GPTBot` blocked | `https://www.nextgov.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 6 | Washington Technology `robots.txt`: **30 bytes, `User-agent: GPTBot` / `Disallow: /` and nothing else — there is no wildcard group at all** | `https://www.washingtontechnology.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200. Corrected 2026-08-27: an earlier revision of this file wrongly recorded `Crawl-delay: 1` here by carrying it over from the `govexec.com` and `nextgov.com` files. | +| 7 | Government Executive `robots.txt`: `Crawl-delay: 1`, same five `Disallow:` paths as Nextgov | `https://www.govexec.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 8 | GEMG terms: the RSS grant, the `except as expressly permitted by GEMG` clause, the automated-access prohibition, and `feedback@govexec.com` | `https://www.govexec.com/about/terms-and-conditions/` | 2026-08-27 | **Verified** — fetched, HTTP 200; each quoted phrase confirmed present by exact substring match against the page text. **Note the path**: `govexec.com/terms/`, `/terms-service/`, `/about/terms/` and the `nextgov.com` and `washingtontechnology.com` equivalents all 404. This page is reachable from the Nextgov and Washington Technology footers as "Terms and Conditions" and is the only copy found. | +| 9 | MeriTalk terms: the automated-access prohibition, and that the page contains no RSS carve-out | `https://www.meritalk.com/terms-of-use/` | 2026-08-27 | **Verified** — fetched, HTTP 200; quote confirmed by exact substring match, and the string "RSS" does not appear anywhere on the page, which is what makes "no carve-out" a checked claim rather than an inference | +| 10 | Federal News Network `robots.txt`: `ClaudeBot`, `Claude-Web`, `anthropic-ai` and `Claude` each `Disallow: /` | `https://federalnewsnetwork.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200 | +| 11 | MeriTalk `robots.txt`: permissive; no AI-crawler block | `https://www.meritalk.com/robots.txt` | 2026-08-27 | **Verified** — fetched, HTTP 200. The prohibition on MeriTalk is in its terms, not its `robots.txt`. | + +Nothing quoted in this section is unverified. If a future edit adds a licence claim that cannot be +traced to a URL someone actually retrieved, label it unverified in this table rather than leaving +it bare. + +**Configured** — `config/feeds/domains/federal_it.json`: + +| Outlet | Publisher | `robots.txt` for `*` | Terms | What the feed carries | +| --- | --- | --- | --- | --- | +| FedScoop | Scoop News Group | `Disallow:` (nothing disallowed) | none found — claim 4 | Full `content:encoded`; median 410 stored words | +| DefenseScoop | Scoop News Group | `Disallow:` | none found — claim 4 | Full `content:encoded`; median 586 stored words | +| CyberScoop | Scoop News Group | `Disallow:` | none found — claim 4 | Full `content:encoded`; median 599 stored words | +| Nextgov/FCW — Cybersecurity | Government Executive Media Group | `Crawl-delay: 1`; no `Disallow:` matches `/rss/` | claim 8 | Headline, one-sentence dek, link. **No body** — see the truncation note | +| Washington Technology | Government Executive Media Group | no wildcard group; only `GPTBot` blocked | claim 8 | Headline, one-sentence dek, link. **No body** | + +- **Scoop News Group (FedScoop, DefenseScoop, CyberScoop)** — basis: the publisher's own RSS feed, + the ordinary `Publisher RSS` class. `robots.txt` disallows `GPTBot` and nothing else; the + wildcard group is an empty `Disallow:` (claims 1-3). No terms page was found (claim 4), so the + basis is the feed alone and there is no per-site reading to record. Because that is a negative + finding rather than a positive grant, it is the weaker of the two bases in this section. +- **Government Executive Media Group (Nextgov/FCW, Washington Technology)** — class + `Publisher RSS — personal use only`. Their terms, read at + `https://www.govexec.com/about/terms-and-conditions/` on 2026-08-27 (claim 8), grant the feeds + in these words: *"Some of the Sites may provide an RSS feed (the 'Feeds'). Individuals may use + the Feeds free of charge - unless otherwise specified - for their personal, non-commercial use + only,"* with a request for attribution to the specific brand and an address for commercial-use + requests (`feedback@govexec.com`). The same page otherwise prohibits *"Obtain, copy, monitor, + index or data mine through the use of a robot, spider, any automated device, or any manual + process, the Sites or the contents (except as expressly permitted by GEMG)"* — and the Feeds + clause is that express permission, for personal non-commercial use. **This is a real limit and + it is recorded, not waived: the current brief is private use and is inside it. Before any beat + carrying these sources publishes, the operator must either get written permission or drop + them.** The `us-public-sector-compliance` beat is the beat this file already treats as + publication-bound (rule 2), so this is the constraint most likely to bite, and it is the one an + operator should resolve rather than a reviewer wave through. +- **The truncation note.** The GEMG feeds do ship a `` element, but its CDATA + markers are themselves escaped, so `feedparser` sanitizes the element to an empty string and the + stored `content` and `normalized_content` are empty with `word_count` 0. Verified 2026-08-27 by + storing them: 0 of 5 Nextgov rows and 0 of 11 Washington Technology rows have non-empty content, + while `description` is populated on every row with the publisher's own one-sentence dek. So they + are headline-plus-dek sources, not full-text ones. They are still configured, because the + headline and dek are what makes an *event* visible to the beat at all, and that was the gap. + They are recorded here so nobody mistakes them for analysable body text. Not fixed here: + correcting `src/rss/fetcher.py` to recover the escaped CDATA would change ingestion for all + configured feeds and is outside task 009. + +**Assessed and rejected** — deliberately not configured. Recorded so the absences read as +decisions: + +| Outlet | Why not | +| --- | --- | +| Federal News Network | Their `robots.txt` (`https://federalnewsnetwork.com/robots.txt`, read 2026-08-27, claim 10) carries an AI-crawler blocklist naming Anthropic's agents specifically: `ClaudeBot`, `Claude-Web`, `anthropic-ai` and `Claude`, each `Disallow: /`, under the comment *"Claude bot used to collect training data for Anthropic LLMs"*. InsightWeaver does not send those User-Agents and the wildcard group does allow the feed path, so this is not a technical prohibition. It is a plainly stated wish not to have this publisher's content collected for LLM processing, which is what this pipeline does with it, and rule 5 says the wish wins. **In fairness to the other reading:** the blocklist is a third-party template that still contains its own instructions (*"Remove comments before publishing - Prepared by ORDO DIGITAL"*), so it may be boilerplate rather than a considered position. The call was made conservatively; their feed is also summary-only (median ~22 words), so little is lost. Revisit if the publisher's position changes. | +| MeriTalk | Their terms of use (`https://www.meritalk.com/terms-of-use/`, read 2026-08-27, claim 9) prohibit: *"Obtain, copy, monitor, index, or data mine through the use of a robot, spider, any automated device, or any manual process, the Web site or the contents (except as expressly permitted by MeriTalk)."* Unlike GEMG there is no clause granting the feed — the string "RSS" does not appear on the page at all — so automated retrieval is forbidden and the source is out of scope by task 009's own terms. Full-text feed and good federal-IT coverage; a written permission request is the only route in. | +| Government Executive | Same publisher and same terms as Nextgov/FCW (claim 8), so it could have shipped on the same basis. Left out on editorial grounds instead: its all-content feed is federal workforce and management news, which is a different subject from this beat's, and 25 items a fetch of it would crowd the curation window without adding compliance signal. Add it if a workforce beat is ever declared. | + +**Volume.** The five configured feeds returned 46 items on the first live fetch across a four-day +lookback (10 + 10 + 10 + 5 + 11), which is roughly 15 to 20 new items a day in steady state +against the beat's 50-article curation window. Topic feeds were preferred over firehoses where the +outlet offered one: Nextgov/FCW is its cybersecurity feed rather than `/rss/all/`. Washington +Technology has no topic feeds — `/rss/cybersecurity/` and `/rss/topic/policy/` both 404 — so it is +configured as `/rss/all/`. + ## All configured sources -Generated from `config/feeds/**/*.json` on 2026-08-26. Duplicate entries (the same source listed -in more than one tag file) appear once. +Generated from `config/feeds/**/*.json` on 2026-08-26, and updated 2026-08-27 with the five +federal-IT trade sources added by task 009. Duplicate entries (the same source listed in more than +one tag file) appear once. | Source | URL | Adapter | Class | Basis for retrieval and derived analysis | | --- | --- | --- | --- | --- | @@ -125,11 +230,14 @@ in more than one tag file) appear once. | Bleeping Computer | `https://www.bleepingcomputer.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Blue Virginia | `https://bluevirginia.us/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Cardinal News | `https://cardinalnews.org/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | +| CyberScoop | `https://cyberscoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` read 2026-08-27: wildcard group is an empty `Disallow:`, only `GPTBot` blocked. No terms page found (five candidate URLs tried 2026-08-27, all 404) — see claim 4 in *Federal-IT trade press*. Added 2026-08-27 (task 009). | | Cybersecurity Dive | `https://www.cybersecuritydive.com/feeds/news/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Dark Reading | `https://www.darkreading.com/rss.xml` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | DCist | `https://dcist.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | +| DefenseScoop | `https://defensescoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` read 2026-08-27: wildcard group is an empty `Disallow:`, only `GPTBot` blocked. No terms page found (five candidate URLs tried 2026-08-27, all 404) — see claim 4 in *Federal-IT trade press*. Added 2026-08-27 (task 009). | | Education Week | `https://www.edweek.org/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | El Faro English (El Salvador) | `https://elfaro.net/en/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | +| FedScoop | `https://fedscoop.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). `robots.txt` read 2026-08-27: wildcard group is an empty `Disallow:`, only `GPTBot` blocked. No terms page found (five candidate URLs tried 2026-08-27, all 404) — see claim 4 in *Federal-IT trade press*. Added 2026-08-27 (task 009). | | FFXnow - Fairfax County News | `https://www.ffxnow.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | France 24 English | `https://www.france24.com/en/rss` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Greater Greater Washington | `https://ggwash.org/feed` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | @@ -138,6 +246,7 @@ in more than one tag file) appear once. | Kathimerini English Edition (Greece) | `https://www.ekathimerini.com/rss` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Krebs on Security | `https://krebsonsecurity.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | MIT Technology Review | `https://www.technologyreview.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | +| Nextgov/FCW - Cybersecurity | `https://www.nextgov.com/rss/cybersecurity/` | `rss` | Publisher RSS — personal use only | **Limited.** Publisher's own feed, granted by GEMG's terms for *personal, non-commercial use only*, with attribution requested — terms read at `https://www.govexec.com/about/terms-and-conditions/` on 2026-08-27 and quotes verified against that page (claim 8). Fine for the private brief; **must not be carried into a published beat without written permission** (`feedback@govexec.com`). Headline and dek only -- no body text survives parsing. Added 2026-08-27 (task 009). | | NHK World Japan | `https://www3.nhk.or.jp/nhkworld/en/news/rss.xml` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Potomac Local News | `https://www.potomaclocal.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Prince William Living | `https://princewilliamliving.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | @@ -151,6 +260,7 @@ in more than one tag file) appear once. | Virginia Mercury | `https://www.virginiamercury.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Washington Post - Local (DC/MD/VA) | `https://feeds.washingtonpost.com/rss/local` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Washington Post Local (DC/MD/VA) | `https://feeds.washingtonpost.com/rss/local` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | +| Washington Technology | `https://washingtontechnology.com/rss/all/` | `rss` | Publisher RSS — personal use only | **Limited.** Publisher's own feed, granted by GEMG's terms for *personal, non-commercial use only*, with attribution requested — terms read at `https://www.govexec.com/about/terms-and-conditions/` on 2026-08-27 and quotes verified against that page (claim 8). Fine for the private brief; **must not be carried into a published beat without written permission** (`feedback@govexec.com`). Headline and dek only -- no body text survives parsing. Added 2026-08-27 (task 009). | | WAVY Virginia Politics | `https://www.wavy.com/news/politics/virginia-politics/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | What's Up Prince William (Woodbridge) | `https://whatsupwoodbridge.com/feed/` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | | Wired | `https://www.wired.com/feed/rss` | `rss` | Publisher RSS | Publisher's own published RSS feed (an explicit syndication offer). | @@ -158,7 +268,7 @@ in more than one tag file) appear once. | Associated Press | `https://feeds.ap.org/rss/apf-topnews.rss` | `rss` | Commercial wire | **Restricted.** Personal brief only; excluded from any published beat. | | Reuters | `https://feeds.reuters.com/reuters/topNews` | `rss` | Commercial wire | **Restricted.** Personal brief only; excluded from any published beat. | - + ## What has and has not been verified @@ -167,7 +277,44 @@ Stated plainly, because an unverified claim recorded as a fact is worse than no - **Verified by reading:** that the Federal Register API is keyless, reachable and returns the fields listed above (exercised 2026-08-26); that AP and Reuters carry no tag the `us-public-sector-compliance` beat selects on. +- **Verified by retrieval, 2026-08-27 (task 009):** every licence claim in *Federal-IT trade + press — the detailed basis*. Eight `robots.txt` files and two terms pages were fetched, each + returning HTTP 200, and every phrase quoted from them was confirmed present by exact substring + match against the retrieved page. The URL and read-date for each are in the provenance table in + that section. **No quote in this file is unattributed, and none is unverified.** Also verified by + fetching: that each of the five configured trade feeds returns a non-zero article count and + stores rows, and that `content` is empty on the two GEMG feeds and populated on the three Scoop + News Group feeds. Verified by reading the stored row: that the FedScoop feed carries *"FedRAMP + Director Pete Waterman reinstated at GSA after weeks of leave"* with 320 words of body text — + the specific event the beat was blind to. +- **Verified only as a negative:** that Scoop News Group publishes no terms page. Five candidate + URLs were tried and all returned 404, and a link scan of the `fedscoop.com` homepage found no + terms, privacy, legal or copyright link (claim 4). A page that exists somewhere those checks did + not reach would not have been found, so this is "not found", never "does not exist" — and it + makes FedScoop, DefenseScoop and CyberScoop the weaker-based of the five configured sources, + resting on the published feed alone. +- **Corrected 2026-08-27:** an earlier revision of this file recorded `Crawl-delay: 1` for + Washington Technology's `robots.txt`. It does not say that. The file is 30 bytes, blocks + `GPTBot`, and has no wildcard group at all; the `Crawl-delay` was carried over in error from the + `govexec.com` and `nextgov.com` files. Found by re-fetching every cited URL rather than by + re-reading the prose. - **Not individually verified:** the terms of service of each of the ~60 `Publisher RSS` sources. Their basis is the general one — the publisher chose to publish a feed — not a per-site reading. Any source promoted into a beat that publishes needs its own reading before that beat publishes. - **Not legal advice**, as stated at the top. + +## The attribution condition is recorded but not yet met in rendered output + +The Government Executive Media Group grant of the Feeds is conditional: it asks for *"attribution +to the specific GEMG brand, including at a minimum a linked logo, a link that returns to the +original URL."* + +**The renderers do not currently emit article URLs.** A brief names its sources -- situations carry +numbered citations and thin-coverage entries name the feed -- but no rendered format links back to +the original. For the private terminal brief that is defensible, since there is no audience the +attribution would reach. It is **not** defensible for anything published, and it is a second +condition on top of the personal-use limit already recorded above. + +Recorded 2026-08-27, verified against `https://www.govexec.com/about/terms-and-conditions/` the +same day. Before any GEMG-sourced content is published, both must be satisfied: written permission +via `feedback@govexec.com`, and linked attribution in the rendered output. diff --git a/backlog/009-federal-it-trade-press.md b/backlog/009-federal-it-trade-press.md index c952e29..40ba970 100644 --- a/backlog/009-federal-it-trade-press.md +++ b/backlog/009-federal-it-trade-press.md @@ -61,3 +61,36 @@ All four declared questions are rule-shaped: statutory dates, authorization stat deadlines, framework divergence. None asks who runs a program or whether an office changed posture. That is a spec-authoring blind spot mirroring the source one. Out of scope here, noted so it is not lost: it belongs with a revision of `config/beats/us-public-sector-compliance.json`. + +## Outcome, 2026-08-27 (branch `task/009-trade-press`, STATUS left QUEUED for review) + +Five trade feeds configured in `config/feeds/domains/federal_it.json`, all resolving into the +beat, all returning a non-zero live count. `tests/config/test_beats.py` now asserts the beat +resolves at least one non-`.gov` source and names all five, and the assertion was checked by +removing the config file and watching it fail. + +**The evidence that matters is not the count.** FedScoop's feed carried +*"FedRAMP Director Pete Waterman reinstated at GSA after weeks of leave"*, and it stored with 320 +words of body text. That is the exact event this task exists because the beat missed. + +**Two decisions a reviewer must make rather than accept:** + +1. **Government Executive Media Group (Nextgov/FCW, Washington Technology) are configured under a + `personal, non-commercial use only` feed licence.** Their terms grant the RSS feeds in those + words. That covers today's private terminal brief and does not cover publication. If this beat + is meant to publish, these two need written permission (`feedback@govexec.com`) or removal. + Recorded in `SOURCES.md`; not something an IC should decide alone. +2. **Federal News Network was rejected**, though it is a named candidate. Its `robots.txt` names + `ClaudeBot`, `Claude-Web`, `anthropic-ai` and `Claude` with `Disallow: /`. InsightWeaver does + not send those User-Agents and the wildcard group permits the feed, so this is a wish rather + than a barrier — but it is a wish about exactly this use. MeriTalk was rejected on its terms, + which forbid automated monitoring with no RSS carve-out. + +**Known limitation, not fixed here.** The two GEMG feeds ship `` with escaped +CDATA markers, so `feedparser` sanitizes the body to an empty string: 0 of 16 stored rows have +non-empty `content`, while `description` carries the publisher's one-sentence dek. They are +headline-plus-dek sources. Recovering the body would mean changing `src/rss/fetcher.py` for all +configured feeds, which is outside this task. Worth its own backlog entry. + +The four standing questions are still all rule-shaped, as this file already notes. Adding the +sources does not make the beat ask who runs a program. That remains open. diff --git a/config/feeds/domains/federal_it.json b/config/feeds/domains/federal_it.json new file mode 100644 index 0000000..0a91849 --- /dev/null +++ b/config/feeds/domains/federal_it.json @@ -0,0 +1,60 @@ +{ + "description": "Federal IT trade press - the outlets that report federal technology, cybersecurity and acquisition as news rather than publishing it as a document. Added 2026-08-27 by backlog task 009, after the us-public-sector-compliance beat's first live brief missed the reinstatement of the FedRAMP director: across 207 feeds not one federal-IT trade outlet was configured, so the beat could see published rules and nothing else. Every entry here is tagged into that beat's existing selectors (federal_policy / regulatory / cybersecurity, geo usa). Basis for use for each is recorded in SOURCES.md; candidates rejected on licence grounds are recorded there too, so the absences are not mistaken for oversights.", + "feeds": [ + { + "name": "FedScoop", + "url": "https://fedscoop.com/feed/", + "applicability": { + "scope": ["national"], + "geo_tags": ["usa"], + "domain_tags": ["federal_policy", "technology", "cybersecurity", "regulatory"], + "specialty_tags": ["federal_it"] + }, + "relevance_score": 0.9 + }, + { + "name": "DefenseScoop", + "url": "https://defensescoop.com/feed/", + "applicability": { + "scope": ["national"], + "geo_tags": ["usa"], + "domain_tags": ["federal_policy", "technology", "cybersecurity"], + "specialty_tags": ["federal_it", "defense"] + }, + "relevance_score": 0.8 + }, + { + "name": "CyberScoop", + "url": "https://cyberscoop.com/feed/", + "applicability": { + "scope": ["national"], + "geo_tags": ["usa"], + "domain_tags": ["cybersecurity", "federal_policy"], + "specialty_tags": ["federal_it", "threat_intelligence"] + }, + "relevance_score": 0.7 + }, + { + "name": "Nextgov/FCW - Cybersecurity", + "url": "https://www.nextgov.com/rss/cybersecurity/", + "applicability": { + "scope": ["national"], + "geo_tags": ["usa"], + "domain_tags": ["cybersecurity", "federal_policy"], + "specialty_tags": ["federal_it"] + }, + "relevance_score": 0.8 + }, + { + "name": "Washington Technology", + "url": "https://washingtontechnology.com/rss/all/", + "applicability": { + "scope": ["national"], + "geo_tags": ["usa"], + "domain_tags": ["federal_policy", "technology", "regulatory"], + "specialty_tags": ["federal_it", "acquisition"] + }, + "relevance_score": 0.7 + } + ] +} diff --git a/tests/config/test_beats.py b/tests/config/test_beats.py index e37cef6..8715567 100644 --- a/tests/config/test_beats.py +++ b/tests/config/test_beats.py @@ -9,6 +9,7 @@ """ import json +from urllib.parse import urlparse import pytest @@ -30,6 +31,11 @@ SHIPPED_BEAT = "us-public-sector-compliance" +def _host(url): + """The lowercased hostname of a feed URL, minus any trailing dot.""" + return (urlparse(url).hostname or "").lower().rstrip(".") + + def write_beat(directory, name, payload): """Write a beat file, accepting a dict or a raw string (for bad JSON).""" directory.mkdir(parents=True, exist_ok=True) @@ -374,10 +380,13 @@ def test_resolves_to_a_small_us_federal_feed_set(self): feeds = load_beat(SHIPPED_BEAT).resolve_feeds() names = {feed.name for feed in feeds} - # Thin by construction: the domain does not publish much RSS. If this - # ever exceeds a dozen feeds, the beat has drifted into being a - # general news beat and the selector needs re-reading. - assert 0 < len(feeds) <= 12 + # Thin by construction: the domain does not publish much RSS. The + # ceiling was 12 until 2026-08-27, when backlog task 009 added five + # federal-IT trade feeds; raised to 20 to leave headroom for one more + # outlet without becoming a licence to grow indefinitely. If this ever + # exceeds 20, the beat has drifted into being a general news beat and + # the selector needs re-reading. + assert 0 < len(feeds) <= 20 assert "Federal Register - Public Inspection" in names assert "CISA Cybersecurity Advisories" in names # Added 2026-08-26 (task 005). The RSS half of this beat is nearly @@ -393,6 +402,49 @@ def test_resolves_to_a_small_us_federal_feed_set(self): assert "Krebs on Security" not in names assert "ARLnow - Arlington Local News" not in names + def test_resolves_at_least_one_non_gov_trade_source(self): + """ + Added 2026-08-27 (task 009). Until then every source this beat resolved + was a primary-document publisher on a `.gov` host, and the beat's first + live brief missed the reinstatement of the FedRAMP director: across + 50,983 stored articles there were three incidental FedRAMP mentions and + none within two weeks of the event, because no federal-IT trade outlet + was configured at all. + + This asserts the property that was missing, not the volume that was + already there. A `.gov` feed publishes rules; only a trade outlet + reports who runs a program, whether a deadline is being enforced, or + whether an authorization was pulled. A future edit to config/feeds/ + that drops the trade press fails here rather than showing up as a brief + that is quietly blind again. + """ + feeds = load_beat(SHIPPED_BEAT).resolve_feeds() + + non_gov = [feed for feed in feeds if not _host(feed.url).endswith(".gov")] + assert non_gov, ( + "the beat resolves only .gov sources, so it can see published " + "documents and no reported news -- see backlog/009-federal-it-trade-press.md" + ) + + # Named, so dropping one is a failure and not a shrug. Each has a row + # in SOURCES.md recording its basis for use. + names = {feed.name for feed in feeds} + trade = { + "FedScoop", + "DefenseScoop", + "CyberScoop", + "Nextgov/FCW - Cybersecurity", + "Washington Technology", + } + assert trade <= names, f"federal-IT trade outlets missing from the beat: {trade - names}" + + # And they are still not wires: SOURCES.md rule 2 is unchanged, and + # widening the beat to reach trade press must not widen it to reach AP + # or Reuters. Asserted here as well as above because this test is the + # one that argues for more non-.gov sources. + assert "Associated Press" not in names + assert "Reuters" not in names + def test_resolution_is_deduplicated_by_url(self): feeds = load_beat(SHIPPED_BEAT).resolve_feeds() urls = [feed.url for feed in feeds]