diff --git a/AGENTS.md b/AGENTS.md index 14b61fe4..ac647865 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -140,6 +140,44 @@ Migration value baseline (merge-to-live wall-clock, detection exemplars, rollbac Characterization: `node --test test/live_url_smoke.test.mjs test/post_flip_checks.test.mjs`. Operator flip procedure lives outside this public tree. +## Hearing participation (one owner, list + detail) + +Meetings list cards and notice permalinks share one derivation: +`normalizeHearing` / `normalizeHearingRow` → `participation.links` → +`participationLinksHTML` in `site/index.html`. Strip trailing punctuation +**before** dedupe (body often has `https://…hearings,` and `https://…hearings`); +one outbound affordance per notice. NYCIDA board URL labels as **IDA meetings page** +(the deepest public target those notices publish). Characterization: +`node --test test/ida_notice_defects.test.mjs`. Captures: +`python3 tools/capture_ida_notice_defects.py`. + +## Contract lifecycle category gate + +`isContractLifecycleEligible` — Procurement section or Solicitation/Award/Intent +types only. Hearings, Agency Rules, Property Disposition, and Changes in Personnel +never mount contract lifecycle / OCP / PIN gap modules (wrong-universe). Subsidy and +meeting-outcomes keep their own eligibility helpers. Characterization: +`test/ida_notice_defects.test.mjs`, `test/lifecycle_coherence_field_cases.test.mjs`. + +## Subsidy lifecycle (NYCIDA / Build NYC) + +Endpoint `GET /subsidy-lifecycle?id=` (`worker/src/subsidy_lifecycle.mjs`). The +EDC documents page is often Cloudflare-blocked to edge fetch (HTTP 403 / challenge +HTML) — treat as feed failure, do **not** permanently D1-cache `source_status: +unavailable`. When the feed fails, `projectFromIdaNotice` derives a hearing-stage +join from the City Record IDA hearing notice (company names, event date). Keep +honest unavailable copy only when the feed is down **and** no notice-derived +hearing applies. Schema safety net: `ensureSubsidySchema` (migration +`0005_subsidy_lifecycle.sql`). + +**Age-aware gap kinds** (temporal sibling of paid / verified_zero / unavailable): +`subsidyGapKind` → `too_soon` | `not_published` | (worker) `unavailable`. Lag table +`SUBSIDY_STAGE_EXPECT_LAG_DAYS` (board ~60d, closing ~180d, project_record ~90d). +Demo/backtest notices must be **aged** (2022–2024 hearings) so later stages read +`not_published`, not “could not reach.” Young hearings use “check back” copy. +Characterization: `test/subsidy_lifecycle.test.mjs`, `test/ida_notice_defects.test.mjs`. +Aged demo ids: `20220525018`, `20231004016`, `20240617012`. + ## Checkbook Contracts row identity Checkbook's Contracts domain returns **multiple rows per `prime_contract_id`** (one Prime Vendor row with amounts, plus Sub Vendor / expense-category slices with $0 on prime fields). Lifecycle assembly collapses rows with `aggregateContractsById` before `classifyStage` — one distinct id = matched; ≥2 distinct ids = ambiguous. Field case: notice `20231222103` / `CT107120248803393`. Do not count raw Contracts rows as separate contracts. Spending rows stay uncollapsed (many payments per contract is normal). Pure lib: `worker/src/lib/checkbook_lifecycle.mjs`. diff --git a/docs/screenshots/ida-notice-defects/_subsidy_after.html b/docs/screenshots/ida-notice-defects/_subsidy_after.html new file mode 100644 index 00000000..42cf4646 --- /dev/null +++ b/docs/screenshots/ida-notice-defects/_subsidy_after.html @@ -0,0 +1,35 @@ + + +IDA subsidy after +
+
CityScroll
+

✓ After · #notice/20250227021 · one IDA meetings link · no contract modules · subsidy from City Record hearing

+

IDA meetings page

+
Subsidy lifecycle
+
Linked project: IDA March 20th, 2025 Public Hearing Notice · company NYM 145 Wolcott LLC · stage Hearing.
+
+
Application
2025-02-27
+
→
+
Hearing
2025-03-20
+
→
+
Board decision
—
+
The city does not publish this — it would appear on Build NYC project documents if released.
+
+
Stages join City Record notices to Build NYC and NYC Industrial Development Agency project records when a public match exists. Hearing stage here is from the City Record notice when the Build NYC feed is unreachable.
+
diff --git a/docs/screenshots/ida-notice-defects/after-meetings-list-1440.png b/docs/screenshots/ida-notice-defects/after-meetings-list-1440.png new file mode 100644 index 00000000..99765c37 Binary files /dev/null and b/docs/screenshots/ida-notice-defects/after-meetings-list-1440.png differ diff --git a/docs/screenshots/ida-notice-defects/after-meetings-list-390.png b/docs/screenshots/ida-notice-defects/after-meetings-list-390.png new file mode 100644 index 00000000..3af6b067 Binary files /dev/null and b/docs/screenshots/ida-notice-defects/after-meetings-list-390.png differ diff --git a/docs/screenshots/ida-notice-defects/after-notice-actions-1440.png b/docs/screenshots/ida-notice-defects/after-notice-actions-1440.png new file mode 100644 index 00000000..eef7aff7 Binary files /dev/null and b/docs/screenshots/ida-notice-defects/after-notice-actions-1440.png differ diff --git a/docs/screenshots/ida-notice-defects/after-notice-actions-390.png b/docs/screenshots/ida-notice-defects/after-notice-actions-390.png new file mode 100644 index 00000000..4498cf0f Binary files /dev/null and b/docs/screenshots/ida-notice-defects/after-notice-actions-390.png differ diff --git a/docs/screenshots/ida-notice-defects/after-notice-modules-1440.png b/docs/screenshots/ida-notice-defects/after-notice-modules-1440.png new file mode 100644 index 00000000..eef7aff7 Binary files /dev/null and b/docs/screenshots/ida-notice-defects/after-notice-modules-1440.png differ diff --git a/docs/screenshots/ida-notice-defects/after-notice-modules-390.png b/docs/screenshots/ida-notice-defects/after-notice-modules-390.png new file mode 100644 index 00000000..4498cf0f Binary files /dev/null and b/docs/screenshots/ida-notice-defects/after-notice-modules-390.png differ diff --git a/docs/screenshots/ida-notice-defects/after-subsidy-shell-1440.png b/docs/screenshots/ida-notice-defects/after-subsidy-shell-1440.png new file mode 100644 index 00000000..37b4d850 Binary files /dev/null and b/docs/screenshots/ida-notice-defects/after-subsidy-shell-1440.png differ diff --git a/docs/screenshots/ida-notice-defects/after-subsidy-shell-390.png b/docs/screenshots/ida-notice-defects/after-subsidy-shell-390.png new file mode 100644 index 00000000..e3d2e017 Binary files /dev/null and b/docs/screenshots/ida-notice-defects/after-subsidy-shell-390.png differ diff --git a/docs/screenshots/ida-notice-defects/before-meetings-list-1440.png b/docs/screenshots/ida-notice-defects/before-meetings-list-1440.png new file mode 100644 index 00000000..2f368a91 Binary files /dev/null and b/docs/screenshots/ida-notice-defects/before-meetings-list-1440.png differ diff --git a/docs/screenshots/ida-notice-defects/before-meetings-list-390.png b/docs/screenshots/ida-notice-defects/before-meetings-list-390.png new file mode 100644 index 00000000..3d14fcfe Binary files /dev/null and b/docs/screenshots/ida-notice-defects/before-meetings-list-390.png differ diff --git a/docs/screenshots/ida-notice-defects/before-notice-actions-1440.png b/docs/screenshots/ida-notice-defects/before-notice-actions-1440.png new file mode 100644 index 00000000..3fc4e60b Binary files /dev/null and b/docs/screenshots/ida-notice-defects/before-notice-actions-1440.png differ diff --git a/docs/screenshots/ida-notice-defects/before-notice-actions-390.png b/docs/screenshots/ida-notice-defects/before-notice-actions-390.png new file mode 100644 index 00000000..35e20c81 Binary files /dev/null and b/docs/screenshots/ida-notice-defects/before-notice-actions-390.png differ diff --git a/docs/screenshots/ida-notice-defects/before-notice-modules-1440.png b/docs/screenshots/ida-notice-defects/before-notice-modules-1440.png new file mode 100644 index 00000000..3fc4e60b Binary files /dev/null and b/docs/screenshots/ida-notice-defects/before-notice-modules-1440.png differ diff --git a/docs/screenshots/ida-notice-defects/before-notice-modules-390.png b/docs/screenshots/ida-notice-defects/before-notice-modules-390.png new file mode 100644 index 00000000..35e20c81 Binary files /dev/null and b/docs/screenshots/ida-notice-defects/before-notice-modules-390.png differ diff --git a/site/hearing_location.js b/site/hearing_location.js index c0b55d3d..5c577946 100644 --- a/site/hearing_location.js +++ b/site/hearing_location.js @@ -227,17 +227,51 @@ function normalizeHearingRow(row) { published_at: row.start_date || null, decides: hearingDecision(row, body), affects: audience ? [audience[1]] : [], affected_area: hearingAffectedArea(row), venue: hearingVenue(row), - participation: { - links: hearingUnique(body.match(HEARING_URL_RE) || []).slice(0, 8).map(function (url) { - return { label: /\b(?:zoom|webex|teams|meet\.google)\b/i.test(url) ? "Join online" : "Participation link", url: url.replace(/[.,;]+$/, "") }; - }), - emails: hearingUnique(Array.from(body.matchAll(/\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b/gi)).map(function (match) { return match[0]; })).slice(0, 4), - phones: hearingUnique(Array.from(body.matchAll(/(?:\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]\d{3}[-.\s]\d{4}/g)).map(function (match) { return match[0]; })).slice(0, 4), - source_url: source, - }, + participation: hearingParticipationFromBody(body, source), source_url: source, description: body.slice(0, 1200), }; } +// Hand-synced with worker/src/lib/hearings.mjs participationFromRow: strip trailing +// punctuation before dedupe so "…hearings," and "…hearings" collapse; one affordance. +function hearingNormalizeParticipationUrl(url) { + return String(url || "").replace(/[.,;:)\]]+$/g, "").trim(); +} +function hearingParticipationUrlKey(url) { + try { + var parsed = new URL(url); + return (parsed.host + parsed.pathname).toLowerCase().replace(/\/$/, ""); + } catch (e) { + return String(url || "").toLowerCase().replace(/\/$/, ""); + } +} +function hearingParticipationLabel(url) { + if (/\b(?:zoom|webex|teams|meet\.google)\b/i.test(url)) return "Join online"; + if (/nycida-board-meetings-public-hearings/i.test(url) || /edc\.nyc\/nycida(?:[/?#]|$)/i.test(url)) { + return "IDA meetings page"; + } + return "Participation link"; +} +function hearingParticipationFromBody(body, source) { + var cleaned = (String(body || "").match(HEARING_URL_RE) || []).map(hearingNormalizeParticipationUrl).filter(Boolean); + var byKey = Object.create(null); + cleaned.forEach(function (url) { + var key = hearingParticipationUrlKey(url); + if (!key || byKey[key]) return; + byKey[key] = { label: hearingParticipationLabel(url), url: url }; + }); + var ranked = Object.keys(byKey).map(function (k) { return byKey[k]; }).sort(function (a, b) { + var aJoin = a.label === "Join online" ? 0 : 1; + var bJoin = b.label === "Join online" ? 0 : 1; + if (aJoin !== bJoin) return aJoin - bJoin; + return b.url.length - a.url.length; + }); + return { + links: ranked.slice(0, 1), + emails: hearingUnique(Array.from(String(body || "").matchAll(/\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b/gi)).map(function (match) { return match[0]; })).slice(0, 4), + phones: hearingUnique(Array.from(String(body || "").matchAll(/(?:\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]\d{3}[-.\s]\d{4}/g)).map(function (match) { return match[0]; })).slice(0, 4), + source_url: source, + }; +} function hearingMatchesArea(record, filter) { var f = filter || {}, area = record.affected_area || {}, borough = String(f.borough || "").toLowerCase(); var neighborhood = String(f.neighborhood || "").trim().toLowerCase(); diff --git a/site/i18n.js b/site/i18n.js index 13518b67..7afc7399 100644 --- a/site/i18n.js +++ b/site/i18n.js @@ -403,6 +403,7 @@ const STRINGS = { read_official_notice: "Read official notice", join_online: "Join online", participation_link: "Participation link", + ida_meetings_page: "IDA meetings page", email_in_notice: "Email listed in notice", map_venue: "Map venue", venue_label: "Venue", @@ -968,6 +969,11 @@ const STRINGS = { subsidy_outcome_html: "Outcome: {outcome}", subsidy_outcome_unknown_html: "The city does not publish this outcome — it would appear on the Build NYC project documents if released.", subsidy_stage_unmatched_html: "Not yet shown here — {stage} records live in {source}.", + // Age-aware gaps: too_soon vs not_published vs unavailable (fetch). Temporal sibling of + // paid / verified_zero / unavailable on the Checkbook payment path. + subsidy_stage_too_soon_html: "Hearing was {date} — {stage} records typically appear within about {weeks} weeks. Check back. Too recent to treat as a permanent gap.", + subsidy_stage_not_published_html: "The city does not publish this {stage} record on {source} — it would appear there if released.", + subsidy_join_too_soon_html: "Hearing was {date} — IDA project records for “{title}” usually appear within about {weeks} weeks of the hearing. Check back. Too recent to treat as a permanent gap.", subsidy_source_build_nyc: "Build NYC and NYC Industrial Development Agency", subsidy_source_unavailable_html: "Could not reach {source} to check the subsidy timeline for this notice.", subsidy_unmatched_html: "The city does not publish a linked subsidy project for “{title}” — {reason}", diff --git a/site/i18n/lang/ar.js b/site/i18n/lang/ar.js index 6ec9f227..80fd821a 100644 --- a/site/i18n/lang/ar.js +++ b/site/i18n/lang/ar.js @@ -276,6 +276,7 @@ read_official_notice: "قراءة الإشعار الرسمي", join_online: "الانضمام عبر الإنترنت", participation_link: "رابط المشاركة", + ida_meetings_page: "صفحة اجتماعات IDA", email_in_notice: "البريد الوارد في الإشعار", map_venue: "خريطة مكان الجلسة", venue_label: "مكان الجلسة", @@ -636,6 +637,9 @@ subsidy_outcome_html: "النتيجة: {outcome}", subsidy_outcome_unknown_html: "المدينة لا تنشر هذه النتيجة — ستظهر في مستندات مشروع Build NYC إذا نُشرت.", subsidy_stage_unmatched_html: "لم يُعرض هنا بعد — سجلات {stage} موجودة في {source}.", + subsidy_stage_too_soon_html: "كانت الجلسة في {date} — تظهر سجلات {stage} عادةً خلال نحو {weeks} أسابيع. راجع لاحقًا. أبكر من أن تُعامل كفجوة دائمة.", + subsidy_stage_not_published_html: "المدينة لا تنشر سجل {stage} هذا على {source} — سيظهر هناك إذا نُشر.", + subsidy_join_too_soon_html: "كانت الجلسة في {date} — سجلات مشاريع IDA لـ “{title}” تظهر عادةً خلال نحو {weeks} أسابيع بعد الجلسة. راجع لاحقًا. أبكر من أن تُعامل كفجوة دائمة.", subsidy_source_build_nyc: "Build NYC و NYC Industrial Development Agency", subsidy_source_unavailable_html: "تعذّر الوصول إلى {source} للتحقق من الجدول الزمني للدعم لهذا الإشعار.", subsidy_unmatched_html: "المدينة لا تنشر مشروع دعم مرتبطًا بـ “{title}” — {reason}", diff --git a/site/i18n/lang/bn.js b/site/i18n/lang/bn.js index 4e929a3c..a4d52360 100644 --- a/site/i18n/lang/bn.js +++ b/site/i18n/lang/bn.js @@ -273,6 +273,7 @@ read_official_notice: "সরকারি বিজ্ঞপ্তি পড়ুন", join_online: "অনলাইনে যোগ দিন", participation_link: "অংশগ্রহণের লিংক", + ida_meetings_page: "IDA সভার পৃষ্ঠা", email_in_notice: "বিজ্ঞপ্তির ইমেইল", map_venue: "সভাস্থলের মানচিত্র", venue_label: "সভাস্থল", @@ -629,6 +630,9 @@ subsidy_outcome_html: "ফলাফল: {outcome}", subsidy_outcome_unknown_html: "শহর এই ফলাফল প্রকাশ করে না — প্রকাশিত হলে এটি Build NYC প্রকল্পের নথিতে থাকত।", subsidy_stage_unmatched_html: "এখানে এখনও দেখানো হয়নি — {stage} রেকর্ড {source}-এ আছে।", + subsidy_stage_too_soon_html: "শুনানি হয়েছিল {date} — {stage} রেকর্ড সাধারণত প্রায় {weeks} সপ্তাহের মধ্যে আসে। পরে দেখুন। স্থায়ী ফাঁক ধরতে এখনও তাড়াতাড়ি।", + subsidy_stage_not_published_html: "শহর {source}-এ এই {stage} রেকর্ড প্রকাশ করে না — প্রকাশিত হলে সেখানে থাকত।", + subsidy_join_too_soon_html: "শুনানি হয়েছিল {date} — “{title}”-এর IDA প্রকল্প রেকর্ড সাধারণত শুনানির প্রায় {weeks} সপ্তাহের মধ্যে আসে। পরে দেখুন। স্থায়ী ফাঁক ধরতে এখনও তাড়াতাড়ি।", subsidy_source_build_nyc: "Build NYC এবং NYC Industrial Development Agency", subsidy_source_unavailable_html: "এই বিজ্ঞপ্তির ভর্তুকি সময়রেখা যাচাই করতে {source}-এ পৌঁছানো যায়নি।", subsidy_unmatched_html: "শহর “{title}”-এর জন্য কোনো সংযুক্ত ভর্তুকি প্রকল্প প্রকাশ করে না — {reason}", diff --git a/site/i18n/lang/es.js b/site/i18n/lang/es.js index f0f0c63e..14b07a1e 100644 --- a/site/i18n/lang/es.js +++ b/site/i18n/lang/es.js @@ -273,6 +273,7 @@ read_official_notice: "Leer aviso oficial", join_online: "Participar en línea", participation_link: "Enlace para participar", + ida_meetings_page: "Página de reuniones de la IDA", email_in_notice: "Correo del aviso", map_venue: "Mapa del lugar", venue_label: "Lugar", @@ -629,6 +630,9 @@ subsidy_outcome_html: "Resultado: {outcome}", subsidy_outcome_unknown_html: "La ciudad no publica este resultado — aparecería en los documentos del proyecto Build NYC si se publicara.", subsidy_stage_unmatched_html: "Aún no se muestra aquí — los registros de {stage} están en {source}.", + subsidy_stage_too_soon_html: "Hearing was {date} — los registros de {stage} suelen aparecer en unas {weeks} semanas. Vuelva a consultar. es demasiado reciente para tratarlo como un vacío permanente.", + subsidy_stage_not_published_html: "La ciudad no publica este registro de {stage} en {source} — aparecería allí si se publicara.", + subsidy_join_too_soon_html: "La audiencia fue el {date} — los registros de proyectos de la IDA para “{title}” suelen aparecer en unas {weeks} semanas tras la audiencia. Vuelva a consultar. es demasiado reciente para tratarlo como un vacío permanente.", subsidy_source_build_nyc: "Build NYC y la Agencia de Desarrollo Industrial de NYC", subsidy_source_unavailable_html: "No se pudo acceder a {source} para verificar la línea de tiempo del subsidio de este aviso.", subsidy_unmatched_html: "La ciudad no publica un proyecto de subsidio vinculado para “{title}” — {reason}", diff --git a/site/i18n/lang/fr.js b/site/i18n/lang/fr.js index 0142f565..a77c61aa 100644 --- a/site/i18n/lang/fr.js +++ b/site/i18n/lang/fr.js @@ -302,6 +302,7 @@ read_official_notice: "Lire l’avis officiel", join_online: "Participer en ligne", participation_link: "Lien de participation", + ida_meetings_page: "Page des réunions de l’IDA", email_in_notice: "Courriel indiqué", map_venue: "Carte du lieu", venue_label: "Lieu", @@ -711,6 +712,9 @@ subsidy_outcome_html: "Résultat : {outcome}", subsidy_outcome_unknown_html: "La ville ne publie pas ce résultat — il apparaîtrait sur les documents du projet Build NYC s'il était publié.", subsidy_stage_unmatched_html: "Pas encore affiché ici — les dossiers de {stage} sont dans {source}.", + subsidy_stage_too_soon_html: "Audience le {date} — les dossiers de {stage} apparaissent en général dans un délai d’environ {weeks} semaines. Revenez plus tard. trop récent pour un écart permanent.", + subsidy_stage_not_published_html: "La ville ne publie pas ce dossier de {stage} sur {source} — il y figurerait s’il était publié.", + subsidy_join_too_soon_html: "Audience le {date} — les dossiers de projet IDA pour « {title} » apparaissent en général dans un délai d’environ {weeks} semaines après l’audience. Revenez plus tard. trop récent pour un écart permanent.", subsidy_source_build_nyc: "Build NYC et NYC Industrial Development Agency", subsidy_source_unavailable_html: "Impossible d'atteindre {source} pour vérifier le calendrier de subvention de cet avis.", subsidy_unmatched_html: "La ville ne publie pas de projet de subvention lié pour « {title} » — {reason}", diff --git a/site/i18n/lang/ht.js b/site/i18n/lang/ht.js index cb6e8889..42543d65 100644 --- a/site/i18n/lang/ht.js +++ b/site/i18n/lang/ht.js @@ -272,6 +272,7 @@ read_official_notice: "Li avi ofisyèl la", join_online: "Patisipe sou entènèt", participation_link: "Lyen pou patisipe", + ida_meetings_page: "Paj reyinyon IDA", email_in_notice: "Imèl nan avi a", map_venue: "Kat kote a", venue_label: "Kote odyans lan", @@ -628,6 +629,9 @@ subsidy_outcome_html: "Rezilta: {outcome}", subsidy_outcome_unknown_html: "Vil la pa pibliye rezilta sa a — li t ap parèt nan dokiman pwojè Build NYC si yo te pibliye l.", subsidy_stage_unmatched_html: "Pokò a pa montre isit la — dosye {stage} yo nan {source}.", + subsidy_stage_too_soon_html: "Odyans lan te {date} — dosye {stage} anjeneral parèt nan apeprè {weeks} semèn. Tcheke pita. twò resan pou trete kòm yon twou pèmanan.", + subsidy_stage_not_published_html: "Vil la pa pibliye dosye {stage} sa a sou {source} — li t ap parèt la si yo te pibliye l.", + subsidy_join_too_soon_html: "Odyans lan te {date} — dosye pwojè IDA pou “{title}” anjeneral parèt nan apeprè {weeks} semèn apre odyans lan. Tcheke pita. twò resan pou trete kòm yon twou pèmanan.", subsidy_source_build_nyc: "Build NYC ak NYC Industrial Development Agency", subsidy_source_unavailable_html: "Pa t kapab rive jwenn {source} pou verifye delè sibvansyon avi sa a.", subsidy_unmatched_html: "Vil la pa pibliye yon pwojè sibvansyon ki lye pou “{title}” — {reason}", diff --git a/site/i18n/lang/ko.js b/site/i18n/lang/ko.js index 71f4c682..da55ec62 100644 --- a/site/i18n/lang/ko.js +++ b/site/i18n/lang/ko.js @@ -273,6 +273,7 @@ read_official_notice: "공식 공고 읽기", join_online: "온라인 참여", participation_link: "참여 링크", + ida_meetings_page: "IDA 회의 페이지", email_in_notice: "공고의 이메일", map_venue: "장소 지도", venue_label: "개최 장소", @@ -629,6 +630,9 @@ subsidy_outcome_html: "결과: {outcome}", subsidy_outcome_unknown_html: "시는 이 결과를 공개하지 않습니다 — 공개되면 Build NYC 프로젝트 문서에 나타납니다.", subsidy_stage_unmatched_html: "아직 여기 표시되지 않음 — {stage} 기록은 {source}에 있습니다.", + subsidy_stage_too_soon_html: "공청회 일자 {date} — {stage} 기록은 보통 약 {weeks}주 안에 나옵니다. 나중에 다시 확인하세요. 영구 공백으로 보기엔 너무 이릅니다.", + subsidy_stage_not_published_html: "시 당국은 {source}에 이 {stage} 기록을 공개하지 않습니다 — 공개되면 그곳에 나타날 것입니다.", + subsidy_join_too_soon_html: "공청회 일자 {date} — “{title}”의 IDA 프로젝트 기록은 보통 공청회 후 약 {weeks}주 안에 나옵니다. 나중에 다시 확인하세요. 영구 공백으로 보기엔 너무 이릅니다.", subsidy_source_build_nyc: "Build NYC 및 NYC Industrial Development Agency", subsidy_source_unavailable_html: "이 공고의 보조금 일정을 확인하기 위해 {source}에 연결할 수 없습니다.", subsidy_unmatched_html: "시는 “{title}”에 연결된 보조금 프로젝트를 공개하지 않습니다 — {reason}", diff --git a/site/i18n/lang/pl.js b/site/i18n/lang/pl.js index 55d9bfb9..ed37be8f 100644 --- a/site/i18n/lang/pl.js +++ b/site/i18n/lang/pl.js @@ -281,6 +281,7 @@ read_official_notice: "Przeczytaj oficjalne ogłoszenie", join_online: "Dołącz online", participation_link: "Link do udziału", + ida_meetings_page: "Strona posiedzeń IDA", email_in_notice: "E-mail z ogłoszenia", map_venue: "Mapa miejsca", venue_label: "Miejsce", @@ -647,6 +648,9 @@ subsidy_outcome_html: "Wynik: {outcome}", subsidy_outcome_unknown_html: "Miasto nie publikuje tego wyniku — pojawiłby się w dokumentach projektu Build NYC, gdyby został opublikowany.", subsidy_stage_unmatched_html: "Jeszcze nie pokazane tutaj — wpisy etapu {stage} są w {source}.", + subsidy_stage_too_soon_html: "Wysłuchanie było {date} — zapisy {stage} zwykle pojawiają się w ciągu ok. {weeks} tyg. Sprawdź później. za wcześnie na stałą lukę.", + subsidy_stage_not_published_html: "Miasto nie publikuje tego zapisu {stage} w {source} — pojawiłby się tam po udostępnieniu.", + subsidy_join_too_soon_html: "Wysłuchanie było {date} — zapisy projektów IDA dla „{title}” zwykle pojawiają się w ciągu ok. {weeks} tyg. po wysłuchaniu. Sprawdź później. za wcześnie na stałą lukę.", subsidy_source_build_nyc: "Build NYC i NYC Industrial Development Agency", subsidy_source_unavailable_html: "Nie udało się dotrzeć do {source}, aby sprawdzić harmonogram dotacji dla tego ogłoszenia.", subsidy_unmatched_html: "Miasto nie publikuje powiązanego projektu dotacyjnego dla „{title}” — {reason}", diff --git a/site/i18n/lang/ru.js b/site/i18n/lang/ru.js index d41db3b7..cec0b9e8 100644 --- a/site/i18n/lang/ru.js +++ b/site/i18n/lang/ru.js @@ -273,6 +273,7 @@ read_official_notice: "Читать официальное объявление", join_online: "Присоединиться онлайн", participation_link: "Ссылка для участия", + ida_meetings_page: "Страница заседаний IDA", email_in_notice: "Эл. почта из объявления", map_venue: "Место на карте", venue_label: "Место проведения", @@ -639,6 +640,9 @@ subsidy_outcome_html: "Исход: {outcome}", subsidy_outcome_unknown_html: "Город не публикует этот итог — при публикации он появился бы в документах проекта Build NYC.", subsidy_stage_unmatched_html: "Пока не показано здесь — записи этапа «{stage}» есть в {source}.", + subsidy_stage_too_soon_html: "Слушание было {date} — записи {stage} обычно появляются примерно через {weeks} нед. Проверьте позже. слишком рано считать это постоянным пробелом.", + subsidy_stage_not_published_html: "Город не публикует эту запись {stage} в {source} — она появилась бы там при выпуске.", + subsidy_join_too_soon_html: "Слушание было {date} — проектные записи IDA для «{title}» обычно появляются примерно через {weeks} нед. после слушания. Проверьте позже. слишком рано считать это постоянным пробелом.", subsidy_source_build_nyc: "Build NYC и NYC Industrial Development Agency", subsidy_source_unavailable_html: "Не удалось связаться с {source}, чтобы проверить график субсидии для этого извещения.", subsidy_unmatched_html: "Город не публикует связанный субсидионный проект для «{title}» — {reason}", diff --git a/site/i18n/lang/ur.js b/site/i18n/lang/ur.js index 7b1705f5..35205481 100644 --- a/site/i18n/lang/ur.js +++ b/site/i18n/lang/ur.js @@ -276,6 +276,7 @@ read_official_notice: "سرکاری نوٹس پڑھیں", join_online: "آن لائن شامل ہوں", participation_link: "شرکت کا لنک", + ida_meetings_page: "IDA میٹنگز کا صفحہ", email_in_notice: "نوٹس میں درج ای میل", map_venue: "اجلاس کی جگہ کا نقشہ", venue_label: "اجلاس کی جگہ", @@ -636,6 +637,9 @@ subsidy_outcome_html: "نتیجہ: {outcome}", subsidy_outcome_unknown_html: "شہر یہ نتیجہ شائع نہیں کرتا — اگر جاری ہو تو Build NYC پروجیکٹ دستاویزات میں ظاہر ہوتا۔", subsidy_stage_unmatched_html: "ابھی یہاں نہیں دکھایا گیا — {stage} ریکارڈز {source} میں ہیں۔", + subsidy_stage_too_soon_html: "سماعت {date} کو ہوئی — {stage} ریکارڈ عام طور پر تقریباً {weeks} ہفتوں میں آتے ہیں۔ بعد میں دیکھیں۔ مستقل خلا سمجھنے کے لیے بہت جلد ہے۔", + subsidy_stage_not_published_html: "شہر {source} پر یہ {stage} ریکارڈ شائع نہیں کرتا — اگر شائع ہوتا تو وہاں ہوتا۔", + subsidy_join_too_soon_html: "سماعت {date} کو ہوئی — “{title}” کے IDA پروجیکٹ ریکارڈ عام طور پر سماعت کے تقریباً {weeks} ہفتوں میں آتے ہیں۔ بعد میں دیکھیں۔ مستقل خلا سمجھنے کے لیے بہت جلد ہے۔", subsidy_source_build_nyc: "Build NYC اور NYC Industrial Development Agency", subsidy_source_unavailable_html: "اس نوٹس کی سبسڈی ٹائم لائن جانچنے کے لیے {source} تک رسائی نہ ہو سکی۔", subsidy_unmatched_html: "شہر “{title}” کے لیے مربوط سبسڈی پروجیکٹ شائع نہیں کرتا — {reason}", diff --git a/site/i18n/lang/zh-Hans.js b/site/i18n/lang/zh-Hans.js index 02411c52..1162c812 100644 --- a/site/i18n/lang/zh-Hans.js +++ b/site/i18n/lang/zh-Hans.js @@ -273,6 +273,7 @@ read_official_notice: "阅读官方公告", join_online: "在线参加", participation_link: "参与链接", + ida_meetings_page: "IDA 会议页面", email_in_notice: "公告中的电子邮件", map_venue: "查看会场地图", venue_label: "会场", @@ -629,6 +630,9 @@ subsidy_outcome_html: "结果：{outcome}", subsidy_outcome_unknown_html: "市政府未公布此项结果 — 如发布将出现在 Build NYC 项目文件中。", subsidy_stage_unmatched_html: "此处尚未显示 — {stage} 记录在 {source}。", + subsidy_stage_too_soon_html: "听证会日期为 {date} — {stage} 记录通常在约 {weeks} 周内出现。请稍后再查；时间太近，不宜当作永久缺失。", + subsidy_stage_not_published_html: "市政府未在 {source} 发布此 {stage} 记录 — 如发布将出现在那里。", + subsidy_join_too_soon_html: "听证会日期为 {date} — “{title}” 的 IDA 项目记录通常在听证后约 {weeks} 周出现。请稍后再查；时间太近，不宜当作永久缺失。", subsidy_source_build_nyc: "Build NYC 与纽约市工业发展局", subsidy_source_unavailable_html: "无法访问 {source} 以核对本公告的补贴时间线。", subsidy_unmatched_html: "市政府未为“{title}”公布关联的补贴项目 — {reason}", diff --git a/site/index.html b/site/index.html index 3b59f33d..29f33aff 100644 --- a/site/index.html +++ b/site/index.html @@ -2624,12 +2624,16 @@

Preview

} function renderDetail(r, chain, stats){ const pending = chain === null; // first paint from the in-memory record; chain/stats hydrate in + // Same participation derivation as the meetings list (normalizeHearingRow → one link). + const detailHearing=isMeetingOutcomesEligible(r)?normalizeHearingRow(r):null; + const detailParticipation=detailHearing?participationLinksHTML(detailHearing):""; let html = `
${qrButtonHTML("dqr","act")} ${pinBtn("notice", r.request_id, cleanText(r.short_title)||r.request_id, [r.type_of_notice_description, r.agency_name, fdate(r.start_date)].filter(Boolean).join(" · "))} + ${detailParticipation}
`; html += `
`; // Solicitations get the richer how-to-respond panel below; everything else gets the glance box. @@ -4702,6 +4706,24 @@

${t("career_outcomes_heading")}

try{ const url=new URL(value); return url.protocol==="https:"||url.protocol==="http:" ? url.toString() : null; } catch(e){ return null; } } +// Shared list+detail participation actions. Derives from normalizeHearingRow.participation +// (one-owner): meetings cards and the notice permalink use the same links/labels. +// Cap is one outbound affordance — duplicates were a body-regex trailing-comma bug. +function participationLinksHTML(record){ + const links=((record&&record.participation&&record.participation.links)||[]) + .map(link=>({ + url:hearingSafeURL(link.url), + label:String(link.label||""), + join:/\bjoin\b/i.test(link.label||""), + ida:/ida meetings/i.test(link.label||""), + })) + .filter(link=>link.url) + .slice(0,1); + return links.map(link=>{ + const key=link.join?"join_online":(link.ida?"ida_meetings_page":"participation_link"); + return `${t(key)}${extSR()}`; + }).join(""); +} function hearingAreaText(record){ const area=record.affected_area||{}; if(area.scope==="citywide") return t("citywide"); @@ -4728,13 +4750,11 @@

${t("career_outcomes_heading")}

const scope=record.affected_area&&record.affected_area.scope||"unlocated"; const sectionKey=record.source_section==="Agency Rules"?"rules_hearing_badge":"public_hearing_badge"; const past=!!record.event_date&&String(record.event_date).slice(0,10)({url:hearingSafeURL(link.url), join:/\bjoin\b/i.test(link.label||"")})) - .filter(link=>link.url).slice(0,2); + const participation=record.participation||{}; let actions=`${t("read_official_notice")}${extSR()}`; actions+=``; if(record.event_date) actions+=``; - links.forEach(link=>{ actions+=`${t(link.join?"join_online":"participation_link")}${extSR()}`; }); + actions+=participationLinksHTML(record); (participation.emails||[]).slice(0,1).forEach(email=>{ actions+=`${t("email_in_notice")}`; }); @@ -5650,7 +5670,24 @@

${t("career_outcomes_heading")}

statements in the established "no record found" register, never blank. Each stage links to its authoritative source page (City Record or Checkbook NYC). OCP Recent Contract Awards (qyyg-4tf5) rides the same precomputed payload as an award side-car for date/amount - corroboration — disagreements name both sources. */ + corroboration — disagreements name both sources. + + Category-aware dispatch: only procurement notices mount this module. Hearings, rules, + property disposition, and staffing are wrong-universe for PIN/Checkbook/OCP gaps — leave + the slot empty rather than stacking "not found" cards on a public hearing. */ + +// Contract lifecycle (PIN, Checkbook, OCP awards) is meaningful only for procurement. +// Section is authoritative when present; notice type covers paths that omit section_name. +function isContractLifecycleEligible(r){ + if(!r) return false; + const section=String(r.section_name||""); + if(section==="Procurement") return true; + if(section==="Public Hearings and Meetings") return false; + if(section==="Agency Rules") return false; + if(section==="Property Disposition") return false; + if(section==="Changes in Personnel") return false; + return /^(Solicitation|Award|Intent to Award)$/.test(r.type_of_notice_description||""); +} const CHECKBOOK_SEARCH_URL = "https://www.checkbooknyc.com/contract_search"; const CHECKBOOK_SPENDING_URL = "https://www.checkbooknyc.com/spending_search"; const CHECKBOOK_SMART_SEARCH = "https://www.checkbooknyc.com/smart_search/citywide"; @@ -6294,8 +6331,14 @@

${t("career_outcomes_heading")}

// Fetches the precomputed lifecycle from the worker and renders the timeline + dollars // panel. Fail-soft on network: says nothing rather than inventing a gap. The read model is // fully precomputed server-side (worker/src/checkbook_lifecycle.mjs) — no live upstream. +// Category gate first: non-procurement notices never fetch or paint contract modules. async function loadLifecycle(r, el, dollarsEl){ if((!el && !dollarsEl) || !r.request_id) return; + if(!isContractLifecycleEligible(r)){ + if(el) el.innerHTML = ""; + if(dollarsEl) dollarsEl.innerHTML = ""; + return; + } let data = null; try{ const resp = await workerFetch("/contract-lifecycle?id=" + encodeURIComponent(r.request_id), null, 8000); @@ -6344,7 +6387,42 @@

${t("career_outcomes_heading")}

return stage; } -function subsidyStageHTML(entry){ +// Hand-synced lag table with worker/src/lib/subsidy_lifecycle.mjs SUBSIDY_STAGE_EXPECT_LAG_DAYS. +// Three honest gap states: too_soon / not_published / unavailable (fetch only). +const SUBSIDY_STAGE_EXPECT_LAG_DAYS = {board_decision:60, closing:180, compliance:400, project_record:90}; +function subsidyLagWeeks(stage){ + const days = SUBSIDY_STAGE_EXPECT_LAG_DAYS[stage] != null + ? SUBSIDY_STAGE_EXPECT_LAG_DAYS[stage] + : SUBSIDY_STAGE_EXPECT_LAG_DAYS.project_record; + return Math.max(1, Math.round(days / 7)); +} +function subsidyDaysSince(iso, asOf){ + const start = String(iso || "").slice(0, 10); + if(!/^\d{4}-\d{2}-\d{2}$/.test(start)) return null; + const a = Date.UTC(+start.slice(0,4), +start.slice(5,7)-1, +start.slice(8,10)); + const end = asOf instanceof Date ? asOf : new Date(asOf || Date.now()); + if(Number.isNaN(end.valueOf())) return null; + const b = Date.UTC(end.getUTCFullYear(), end.getUTCMonth(), end.getUTCDate()); + return Math.floor((b - a) / 86400000); +} +function subsidyGapKindClient(stage, anchorDate, matched){ + if(matched) return null; + const days = subsidyDaysSince(anchorDate); + if(days == null) return "not_published"; + const lag = SUBSIDY_STAGE_EXPECT_LAG_DAYS[stage] != null + ? SUBSIDY_STAGE_EXPECT_LAG_DAYS[stage] + : SUBSIDY_STAGE_EXPECT_LAG_DAYS.project_record; + return days < lag ? "too_soon" : "not_published"; +} +function subsidyAnchorFromNotice(notice, data){ + const hearing = (data && data.timeline || []).find(e => e && e.stage === "hearing" && e.date); + if(hearing && hearing.date) return String(hearing.date).slice(0,10); + if(data && data.join && data.join.anchor_date) return String(data.join.anchor_date).slice(0,10); + const ev = notice && (notice.event_date || notice.start_date); + return ev ? String(ev).slice(0,10) : null; +} + +function subsidyStageHTML(entry, anchorDate){ const label = subsidyStageLabel(entry.stage); const dateHTML = entry.date ? `
${fdate(entry.date)}
` : `
—
`; let detailHTML = ""; @@ -6357,10 +6435,20 @@

${t("career_outcomes_heading")}

const amt = entry.detail && entry.detail.amount != null ? `
${money(entry.detail.amount)}
` : ""; detailHTML = amt + action + outcome; } else { - detailHTML = `
${t("subsidy_stage_unmatched_html",{ - stage: label, - source: `${t("subsidy_source_build_nyc")}` - })}
`; + const kind = entry.gap_kind || subsidyGapKindClient(entry.stage, anchorDate, false); + if(kind === "too_soon"){ + detailHTML = `
${t("subsidy_stage_too_soon_html",{ + stage: label, + date: fdate(anchorDate) || "—", + weeks: String(subsidyLagWeeks(entry.stage)) + })}
`; + } else { + // Aged absence → class-(b) not-published register (not "not yet shown" ingest lag). + detailHTML = `
${t("subsidy_stage_not_published_html",{ + stage: label, + source: `${t("subsidy_source_build_nyc")}` + })}
`; + } } let link = ""; if(entry.source && entry.source.status === "matched" && entry.source.url){ @@ -6377,25 +6465,36 @@

${t("career_outcomes_heading")}

function subsidyLifecycleHTML(data, notice){ if(!data) return ""; const join = data.join || {}; + const anchor = subsidyAnchorFromNotice(notice, data); // Ownership: project-level unmatched/unavailable is one gap note — do not also stack // per-stage "not yet shown" cards that re-classify the same absence. let stages = ""; if(join.matched && data.source_status !== "unavailable"){ const tl = Array.isArray(data.timeline) ? data.timeline : []; tl.forEach((entry, idx) => { - stages += subsidyStageHTML(entry); + stages += subsidyStageHTML(entry, anchor); if(idx < tl.length - 1) stages += '
→
'; }); } let joinNote = ""; if(data.source_status === "unavailable"){ + // Operational: fetch failure only — never a stand-in for young or unmatched. joinNote = t("subsidy_source_unavailable_html",{source:`${t("subsidy_source_build_nyc")}`}); } else if(join.matched === false){ - joinNote = t("subsidy_unmatched_html",{ - reason: escUiHtml(join.reason || t("subsidy_unmatched_default_reason")), - title: escUiHtml(cleanText(notice.short_title) || notice.request_id || "") - }); + const kind = join.gap_kind || subsidyGapKindClient("project_record", anchor, false); + if(kind === "too_soon"){ + joinNote = t("subsidy_join_too_soon_html",{ + date: fdate(anchor) || "—", + weeks: String(subsidyLagWeeks("project_record")), + title: escUiHtml(cleanText(notice.short_title) || notice.request_id || "") + }); + } else { + joinNote = t("subsidy_unmatched_html",{ + reason: escUiHtml(join.reason || t("subsidy_unmatched_default_reason")), + title: escUiHtml(cleanText(notice.short_title) || notice.request_id || "") + }); + } } else if(join.matched){ const proj = data.project || {}; joinNote = t("subsidy_matched_html",{ @@ -7870,7 +7969,8 @@

${t(titleKey)}

return true; } -const NOTICE_SELECT = SELECT + ",event_date,street_address_1,section_name"; +// Extra description/printout fields so participation URL extraction matches the meetings list. +const NOTICE_SELECT = SELECT + ",event_date,street_address_1,section_name,additional_description_2,additional_description_3,other_info_2,other_info_3,printout_1,printout_2,printout_3,building_name,city,state,zip_code"; // watch: the {lens, filter} parseWatchParam() extracted from this link's own "?w=" (w12-12) -- // null for a plain "#notice/" link (unchanged behavior). When present, the title/evidence/ // echo below render exactly as they would if the reader had run the watch themselves. @@ -7896,6 +7996,9 @@

${t(titleKey)}

? `${title.slice(0,ev.index)}${title.slice(ev.index, ev.index+ev.term.length)}${title.slice(ev.index+ev.term.length)}` : title; const watchChips = watch ? watchChipsFor(watch.lens, watch.filter) : []; + // Same one-owner participation link as meetings list cards (how-do-I-attend on the permalink). + const noticeHearing=isMeetingOutcomesEligible(r)?normalizeHearingRow(r):null; + const noticeParticipation=noticeHearing?participationLinksHTML(noticeHearing):""; box.innerHTML = `

${t("back_browse")}

@@ -7914,6 +8017,7 @@

${titleInner}

${((r.due_date&&!isRollingDeadline(r.due_date))||r.event_date)?``:""} ${pinBtn("notice", r.request_id, cleanText(r.short_title)||r.request_id, [r.type_of_notice_description, r.agency_name, fdate(r.start_date)].filter(Boolean).join(" · "))} + ${noticeParticipation} ${t("view_in_city_record")}${extSR()}
${r.type_of_notice_description==="Solicitation"?buildApply(r):""} diff --git a/test/gap_taxonomy.test.mjs b/test/gap_taxonomy.test.mjs index 27986232..f30cdaeb 100644 --- a/test/gap_taxonomy.test.mjs +++ b/test/gap_taxonomy.test.mjs @@ -92,6 +92,11 @@ const helpers = new Function( extractFn("lifecycleOcpAwardHTML") + extractFn("lifecycleTimelineHTML") + extractFn("subsidyStageLabel") + + extractConst("SUBSIDY_STAGE_EXPECT_LAG_DAYS") + + extractFn("subsidyLagWeeks") + + extractFn("subsidyDaysSince") + + extractFn("subsidyGapKindClient") + + extractFn("subsidyAnchorFromNotice") + extractFn("subsidyStageHTML") + extractFn("subsidyLifecycleHTML") + extractFn("meetingOutcomesHTML") + @@ -203,19 +208,22 @@ test("class b: no-PIN provenance uses not-published register with Checkbook poin }); // --------------------------------------------------------------------------- -// Class (b) — real subsidy unmatched IDA hearing 20260617040 +// Class (b) — subsidy unmatched when no project feed row and notice is not an IDA hearing +// (IDA hearings now derive a City Record hearing-stage join; unmatched is for non-hearing cases.) // --------------------------------------------------------------------------- test("class b: unmatched subsidy project uses not-published register", () => { const [lifecycle] = assembleSubsidyLifecycle([{ - request_id: "20260617040", - short_title: "NEW YORK CITY INDUSTRIAL DEVELOPMENT AGENCY - NOTICE OF PUBLIC HEARING - July 16th, 2026", - agency_name: "Industrial Development Agency", + request_id: "20260101099", + short_title: "Parks concession award — unrelated to subsidy projects", + agency_name: "Parks and Recreation", + type_of_notice_description: "Award", }], []); const html = subsidyLifecycleHTML(lifecycle, { - request_id: "20260617040", - short_title: "NEW YORK CITY INDUSTRIAL DEVELOPMENT AGENCY - NOTICE OF PUBLIC HEARING - July 16th, 2026", + request_id: "20260101099", + short_title: "Parks concession award — unrelated to subsidy projects", }); + assert.equal(lifecycle.join.matched, false); assert.match(html, CLASS_B_PREFIX); assert.match(html, /linked subsidy project/); assert.match(html, /NYCIDA\/Build NYC|would appear/i); diff --git a/test/ida_notice_defects.test.mjs b/test/ida_notice_defects.test.mjs new file mode 100644 index 00000000..a9673110 --- /dev/null +++ b/test/ida_notice_defects.test.mjs @@ -0,0 +1,258 @@ +// Characterization for the owner-reported IDA hearing defect cluster: +// #notice/20250227021 (homepage demo "Past IDA meetings") +// 1) participation links: trailing-comma body URLs must collapse to one affordance +// 2) list + detail share the same participation derivation +// 3) contract lifecycle / OCP / PIN modules are category-gated off hearings +// 4) subsidy: City Record hearing derivation when Build NYC feed is empty +// +// node --test test/ida_notice_defects.test.mjs + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { dirname, join } from "node:path"; +import { normalizeHearing } from "../worker/src/lib/hearings.mjs"; +import { + assembleSubsidyLifecycle, + projectFromIdaNotice, + isIdaHearingNotice, + subsidyGapKind, + lagWeeksForStage, +} from "../worker/src/lib/subsidy_lifecycle.mjs"; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), ".."); +const src = readFileSync(join(ROOT, "site", "index.html"), "utf8"); +const i18nSrc = readFileSync(join(ROOT, "site", "i18n.js"), "utf8"); +const hearingLoc = readFileSync(join(ROOT, "site", "hearing_location.js"), "utf8"); + +function extractFn(name) { + let start = src.indexOf("async function " + name + "("); + if (start === -1) start = src.indexOf("function " + name + "("); + assert.notEqual(start, -1, `function ${name} not found`); + let depth = 0, seen = false; + for (let j = src.indexOf("{", start); j < src.length; j++) { + if (src[j] === "{") { depth++; seen = true; } + else if (src[j] === "}" && --depth === 0 && seen) return src.slice(start, j + 1); + } + throw new Error(`unbalanced braces extracting ${name}`); +} + +const windowStub = { LANG: "en", LANG_META: { en: { intlDate: "en-US" } } }; +const { t, tn } = new Function("window", i18nSrc + "\nreturn { t: window.t, tn: window.tn };")(windowStub); + +// Browser hearing normalizer (same contract as worker normalizeHearing) +const normalizeHearingRow = new Function( + hearingLoc + "\nreturn normalizeHearingRow;", +)(); + +const EXT_ATTRS = 'target="_blank" rel="noopener noreferrer"'; +const escUiHtml = (s) => String(s == null ? "" : s) + .replace(/&/g, "&").replace(//g, ">").replace(/"/g, """); +const extSR = () => ' (opens in new tab)'; + +const ui = new Function( + "t", "tn", "escUiHtml", "extSR", "EXT_ATTRS", + extractFn("hearingSafeURL") + + extractFn("participationLinksHTML") + + extractFn("isContractLifecycleEligible") + + "return { hearingSafeURL, participationLinksHTML, isContractLifecycleEligible };" +)(t, tn, escUiHtml, extSR, EXT_ATTRS); + +// Live-shaped body for #notice/20250227021: same URL twice, once with trailing comma. +const IDA_BODY = `For those members of the public desiring to review project applications and cost +benefit analyses before the date of the hearing, copies of these materials will be made +available at: https://edc.nyc/nycida-board-meetings-public-hearings, starting on or about +12:00 P.M. fourteen (14) days prior to the hearing. Information regarding such removals will +be available on the Agency's website at https://edc.nyc/nycida-board-meetings-public-hearings +on or about 12:00 P.M. on the Friday preceding the hearing. +Company Name : NYM 145 Wolcott LLC, a Delaware limited liability company (the Company)`; + +const IDA_ROW = { + request_id: "20250227021", + short_title: "IDA March 20th, 2025 Public Hearing Notice", + agency_name: "Industrial Development Agency", + type_of_notice_description: "Public Hearings", + section_name: "Public Hearings and Meetings", + event_date: "2025-03-20T10:00:00.000", + start_date: "2025-02-27T00:00:00.000", + additional_description_1: IDA_BODY, + pin: null, +}; + +test("participation: trailing-comma duplicate URLs collapse to one IDA meetings link", () => { + const worker = normalizeHearing(IDA_ROW); + const browser = normalizeHearingRow(IDA_ROW); + assert.equal(worker.participation.links.length, 1, "worker: one link"); + assert.equal(browser.participation.links.length, 1, "browser: one link"); + assert.equal( + worker.participation.links[0].url, + "https://edc.nyc/nycida-board-meetings-public-hearings", + ); + assert.equal(worker.participation.links[0].label, "IDA meetings page"); + assert.equal(browser.participation.links[0].label, "IDA meetings page"); + assert.deepEqual( + worker.participation.links.map((l) => l.url), + browser.participation.links.map((l) => l.url), + "list and detail share the same cleaned URL", + ); +}); + +test("participation: list and detail HTML use the same single affordance", () => { + const record = normalizeHearing(IDA_ROW); + const html = ui.participationLinksHTML(record); + assert.match(html, /IDA meetings page/); + assert.match(html, /href="https:\/\/edc\.nyc\/nycida-board-meetings-public-hearings"/); + assert.equal((html.match(/class="act"/g) || []).length, 1); + assert.doesNotMatch(html, /Participation link/); +}); + +test("category dispatch: contract lifecycle ineligible on hearings, rules, property, staffing", () => { + assert.equal(ui.isContractLifecycleEligible(IDA_ROW), false); + assert.equal(ui.isContractLifecycleEligible({ + section_name: "Agency Rules", type_of_notice_description: "Public Hearings", + }), false); + assert.equal(ui.isContractLifecycleEligible({ + section_name: "Property Disposition", type_of_notice_description: "Public Hearing", + }), false); + assert.equal(ui.isContractLifecycleEligible({ + section_name: "Changes in Personnel", type_of_notice_description: "Notice", + }), false); + assert.equal(ui.isContractLifecycleEligible({ + section_name: "Procurement", type_of_notice_description: "Award", pin: "123", + }), true); + assert.equal(ui.isContractLifecycleEligible({ + section_name: "", type_of_notice_description: "Solicitation", + }), true); + assert.equal(ui.isContractLifecycleEligible({ + section_name: "", type_of_notice_description: "Public Hearings", + }), false); +}); + +test("subsidy: IDA hearing derives City Record hearing stage when project feed is empty", () => { + assert.equal(isIdaHearingNotice(IDA_ROW), true); + const derived = projectFromIdaNotice(IDA_ROW); + assert.ok(derived); + assert.equal(derived.project_id, "city-record:20250227021"); + assert.match(derived.company, /NYM 145 Wolcott/i); + + const [lifecycle] = assembleSubsidyLifecycle([IDA_ROW], []); + assert.equal(lifecycle.join.matched, true); + assert.equal(lifecycle.join.method, "city-record-hearing"); + assert.equal(lifecycle.join.source, "City Record"); + const hearing = lifecycle.timeline.find((e) => e.stage === "hearing"); + assert.equal(hearing.status, "matched"); + assert.equal(hearing.date, "2025-03-20"); + assert.equal(lifecycle.source_status, undefined); +}); + +test("subsidy: non-IDA notice stays unmatched when feed empty (class-b path)", () => { + const [lifecycle] = assembleSubsidyLifecycle([{ + request_id: "x", + short_title: "Parks award", + agency_name: "Parks", + type_of_notice_description: "Award", + }], []); + assert.equal(lifecycle.join.matched, false); +}); + +// --- Age-aware gap kinds (too_soon / not_published / unavailable sibling) --- + +test("subsidyGapKind: young hearing is too_soon for project records and board decision", () => { + const asOf = new Date("2026-07-30T00:00:00Z"); + assert.equal( + subsidyGapKind({ stage: "project_record", anchorDate: "2026-07-16", asOf, matched: false }), + "too_soon", + ); + assert.equal( + subsidyGapKind({ stage: "board_decision", anchorDate: "2026-07-16", asOf, matched: false }), + "too_soon", + ); +}); + +test("subsidyGapKind: aged hearing is not_published for later stages", () => { + const asOf = new Date("2026-07-30T00:00:00Z"); + assert.equal( + subsidyGapKind({ stage: "project_record", anchorDate: "2023-11-02", asOf, matched: false }), + "not_published", + ); + assert.equal( + subsidyGapKind({ stage: "board_decision", anchorDate: "2023-11-02", asOf, matched: false }), + "not_published", + ); + assert.equal( + subsidyGapKind({ stage: "closing", anchorDate: "2023-11-02", asOf, matched: false }), + "not_published", + ); + assert.ok(lagWeeksForStage("board_decision") >= 8); +}); + +test("aged IDA backtest: 2022–2024 hearings join City Record hearing; later stages not_published", () => { + const aged = [ + { + request_id: "20231004016", + short_title: "NEW YORK CITY INDUSTRIAL DEVELOPMENT AGENCY - NOTICE OF PUBLIC HEARING TO BE HELD ON NOVEMBER 2ND, 2023", + agency_name: "Industrial Development Agency", + type_of_notice_description: "Public Hearings", + section_name: "Public Hearings and Meetings", + event_date: "2023-11-02T10:00:00.000", + start_date: "2023-10-19T00:00:00.000", + additional_description_1: "Company Name : Example Holdings LLC, a Delaware limited liability company (the Company)", + }, + { + request_id: "20240617012", + short_title: "NEW YORK CITY INDUSTRIAL DEVELOPMENT AGENCY - NOTICE OF PUBLIC HEARING - July 18TH, 2024", + agency_name: "Industrial Development Agency", + type_of_notice_description: "Public Hearings", + section_name: "Public Hearings and Meetings", + event_date: "2024-07-18T10:00:00.000", + start_date: "2024-07-03T00:00:00.000", + additional_description_1: "Company Name : Aged Demo LLC, a Delaware limited liability company (the Company)", + }, + { + request_id: "20220525018", + short_title: "NYCIDA SUPPLEMENTAL NOTICE OF PUBLIC HEARING - JUNE 9, 2022", + agency_name: "Industrial Development Agency", + type_of_notice_description: "Public Hearings", + section_name: "Public Hearings and Meetings", + event_date: "2022-06-09T10:00:00.000", + start_date: "2022-05-27T00:00:00.000", + additional_description_1: "Company Name : Historic Project LLC, a Delaware limited liability company (the Company)", + }, + ]; + for (const row of aged) { + const [lc] = assembleSubsidyLifecycle([row], []); + assert.equal(lc.join.matched, true, `${row.request_id} should city-record join`); + assert.equal(lc.join.method, "city-record-hearing"); + const hearing = lc.timeline.find((e) => e.stage === "hearing"); + assert.equal(hearing.status, "matched"); + const board = lc.timeline.find((e) => e.stage === "board_decision"); + assert.notEqual(board.status, "matched"); + // gap_kind uses wall clock; for 2022–2024 anchors it must be not_published as of 2026 + assert.equal( + subsidyGapKind({ stage: "board_decision", anchorDate: String(row.event_date).slice(0, 10), asOf: new Date("2026-07-30") }), + "not_published", + `${row.request_id} board should be aged not_published`, + ); + } +}); + +test("young IDA hearing: board gap is too_soon (temporal, not not-published)", () => { + assert.equal( + subsidyGapKind({ stage: "board_decision", anchorDate: "2026-07-16", asOf: new Date("2026-07-30") }), + "too_soon", + ); + const young = { + request_id: "20260617040", + short_title: "NEW YORK CITY INDUSTRIAL DEVELOPMENT AGENCY - NOTICE OF PUBLIC HEARING - July 16th, 2026", + agency_name: "Industrial Development Agency", + type_of_notice_description: "Public Hearings", + section_name: "Public Hearings and Meetings", + event_date: "2026-07-16T10:00:00.000", + start_date: "2026-07-02T00:00:00.000", + additional_description_1: "Company Name : Young Co LLC, a Delaware limited liability company (the Company)", + }; + const [lc] = assembleSubsidyLifecycle([young], []); + assert.equal(lc.join.matched, true); + assert.ok(lc.timeline.find((e) => e.stage === "board_decision")); +}); diff --git a/test/lifecycle_coherence_field_cases.test.mjs b/test/lifecycle_coherence_field_cases.test.mjs index 657920dc..4985c72c 100644 --- a/test/lifecycle_coherence_field_cases.test.mjs +++ b/test/lifecycle_coherence_field_cases.test.mjs @@ -79,12 +79,13 @@ const sandbox = new Function( extractFn("lifecycleOcpAwardHTML") + extractFn("lifecycleTimelineHTML") + extractFn("lifecycleDollarsHTML") + + extractFn("isContractLifecycleEligible") + // Real vendorStem + vendorNamesMatch (entity-resolution for mismatch warning) extractConst("VENDOR_SUFFIX") + extractFn("cleanText") + extractFn("vendorStem") + extractFn("vendorNamesMatch") + - "return { lifecycleTimelineHTML, lifecycleDollarsHTML, lifecycleStageHTML, money, lifecycleMoney, vendorNamesMatch, checkbookSearchUrl, lifecyclePaymentState, lifecycleResolvedPayment, lifecycleCommittedUnderrun, lifecyclePaymentSummaryHTML };" + "return { lifecycleTimelineHTML, lifecycleDollarsHTML, lifecycleStageHTML, money, lifecycleMoney, vendorNamesMatch, checkbookSearchUrl, lifecyclePaymentState, lifecycleResolvedPayment, lifecycleCommittedUnderrun, lifecyclePaymentSummaryHTML, isContractLifecycleEligible };" ); const { @@ -98,6 +99,7 @@ const { lifecycleResolvedPayment, lifecycleCommittedUnderrun, lifecyclePaymentSummaryHTML, + isContractLifecycleEligible, } = sandbox(t, tn, windowStub); // Live-shaped fixture: HNTB award #notice/20260623008 (registered matched, payment was unknown) @@ -170,17 +172,29 @@ const HNTB_LIFECYCLE_RAW = { ], }; -// Live-shaped fixture: IDA hearing #notice/20260617040 (no PIN) +// Live-shaped fixture: IDA hearing #notice/20260617040 — wrong-universe for contract modules. const IDA_NOTICE = { request_id: "20260617040", agency_name: "Industrial Development Agency", type_of_notice_description: "Public Hearing", + section_name: "Public Hearings and Meetings", pin: null, short_title: "NEW YORK CITY INDUSTRIAL DEVELOPMENT AGENCY - NOTICE OF PUBLIC HEARING - July 16th, 2026", start_date: "2026-07-02", }; -const IDA_LIFECYCLE_RAW = { +// Procurement no-PIN (still the correct universe for contract lifecycle gaps). +const PROCUREMENT_NO_PIN = { + request_id: "20260101001", + agency_name: "Citywide Administrative Services", + type_of_notice_description: "Solicitation", + section_name: "Procurement", + pin: null, + short_title: "Widgets for city agencies", + start_date: "2026-01-01", +}; + +const PROCUREMENT_NO_PIN_LIFECYCLE = { pin: null, pin_strategy: "none", ok: false, @@ -190,11 +204,11 @@ const IDA_LIFECYCLE_RAW = { stage: "solicitation", status: "matched", source: "city-record", - date: "2026-07-02", + date: "2026-01-01", detail: { - request_id: "20260617040", - agency: "Industrial Development Agency", - title: IDA_NOTICE.short_title, + request_id: "20260101001", + agency: "Citywide Administrative Services", + title: PROCUREMENT_NO_PIN.short_title, pin: null, }, }, @@ -279,11 +293,21 @@ test("HNTB field case: registration card owns registration, not a second paid ba }); // --------------------------------------------------------------------------- -// 2. IDA no-PIN: single explanation; no stacked could-not-reach / not-yet-shown +// 2. Category-aware dispatch + procurement no-PIN coherence // --------------------------------------------------------------------------- -test("IDA no-PIN field case: single no-PIN explanation; dependent slots collapse", () => { - const html = lifecycleTimelineHTML(IDA_LIFECYCLE_RAW, IDA_NOTICE); +test("IDA hearing is wrong-universe for contract lifecycle modules (category gate)", () => { + assert.equal(isContractLifecycleEligible(IDA_NOTICE), false); + assert.equal(isContractLifecycleEligible({ + ...IDA_NOTICE, + request_id: "20250227021", + type_of_notice_description: "Public Hearings", + }), false); +}); + +test("procurement no-PIN: single no-PIN explanation; dependent slots collapse", () => { + assert.equal(isContractLifecycleEligible(PROCUREMENT_NO_PIN), true); + const html = lifecycleTimelineHTML(PROCUREMENT_NO_PIN_LIFECYCLE, PROCUREMENT_NO_PIN); assert.match(html, /does not publish a Procurement ID \(PIN\)/); assert.match(html, /would appear in Checkbook NYC if released with a PIN/); assert.doesNotMatch(html, TRANSIENT, "no stacked could-not-reach cards"); diff --git a/test/procurement_lifecycle_stitch.test.mjs b/test/procurement_lifecycle_stitch.test.mjs index 12a9d2ca..472595bf 100644 --- a/test/procurement_lifecycle_stitch.test.mjs +++ b/test/procurement_lifecycle_stitch.test.mjs @@ -103,6 +103,11 @@ const sandbox = new Function( extractFn("vendorNamesMatch") + "\n" + extractFn("isSubsidyEligibleNotice") + "\n" + extractFn("subsidyStageLabel") + "\n" + + extractConst("SUBSIDY_STAGE_EXPECT_LAG_DAYS") + "\n" + + extractFn("subsidyLagWeeks") + "\n" + + extractFn("subsidyDaysSince") + "\n" + + extractFn("subsidyGapKindClient") + "\n" + + extractFn("subsidyAnchorFromNotice") + "\n" + extractFn("subsidyStageHTML") + "\n" + extractFn("subsidyLifecycleHTML") + "\n" + extractFn("isMeetingOutcomesEligible") + "\n" + @@ -168,6 +173,11 @@ try { extractFn("vendorNamesMatch") + extractFn("isSubsidyEligibleNotice") + extractFn("subsidyStageLabel") + + extractConst("SUBSIDY_STAGE_EXPECT_LAG_DAYS") + + extractFn("subsidyLagWeeks") + + extractFn("subsidyDaysSince") + + extractFn("subsidyGapKindClient") + + extractFn("subsidyAnchorFromNotice") + extractFn("subsidyStageHTML") + extractFn("subsidyLifecycleHTML") + extractFn("isMeetingOutcomesEligible") + @@ -316,12 +326,28 @@ const matchedSubsidy = assembleSubsidyLifecycle( subsidyProjects, )[0]; +// Non-IDA notice so city-record hearing derivation does not auto-match. const unmatchedSubsidy = assembleSubsidyLifecycle( + [{ + request_id: "20260101099", + short_title: "Parks concession award — no subsidy project link", + agency_name: "Parks and Recreation", + type_of_notice_description: "Award", + }], + [], +)[0]; + +// Young IDA hearing: City Record hearing join; later stages too_soon (not unavailable). +const youngIdaSubsidy = assembleSubsidyLifecycle( [{ request_id: "20260617040", short_title: "NEW YORK CITY INDUSTRIAL DEVELOPMENT AGENCY - NOTICE OF PUBLIC HEARING - July 16th, 2026", agency_name: "Industrial Development Agency", type_of_notice_description: "Public Hearings", + section_name: "Public Hearings and Meetings", + event_date: "2026-07-16T10:00:00.000", + start_date: "2026-07-02T00:00:00.000", + additional_description_1: "Company Name : Young Co LLC, a Delaware limited liability company (the Company)", }], [], )[0]; @@ -342,18 +368,32 @@ test("subsidy detail: matched project renders stage, action, and outcome", () => assert.match(html, /Official action|Outcome/i); }); -test("subsidy detail: unmatched IDA hearing renders specific gap, never generic unknown", () => { +test("subsidy detail: unmatched non-IDA notice renders specific gap, never generic unknown", () => { const notice = { - request_id: "20260617040", - short_title: "NEW YORK CITY INDUSTRIAL DEVELOPMENT AGENCY - NOTICE OF PUBLIC HEARING - July 16th, 2026", + request_id: "20260101099", + short_title: "Parks concession award — no subsidy project link", }; + assert.equal(unmatchedSubsidy.join.matched, false); const html = subsidyLifecycleHTML(unmatchedSubsidy, notice); assert.match(html, /Subsidy lifecycle/); assert.match(html, /does not publish a linked subsidy project for/); - assert.match(html, /20260617040|INDUSTRIAL DEVELOPMENT|would appear on the Build NYC|Industrial Development Agency|No matching NYCIDA/i); + assert.match(html, /20260101099|Parks concession|would appear on the Build NYC|No matching NYCIDA/i); assert.doesNotMatch(html, />\s*unknown\s* { + assert.equal(youngIdaSubsidy.join.matched, true); + assert.equal(youngIdaSubsidy.join.method, "city-record-hearing"); + const html = subsidyLifecycleHTML(youngIdaSubsidy, { + request_id: "20260617040", + short_title: "NEW YORK CITY INDUSTRIAL DEVELOPMENT AGENCY - NOTICE OF PUBLIC HEARING - July 16th, 2026", + event_date: "2026-07-16T10:00:00.000", + }); + assert.match(html, /Subsidy lifecycle/); + assert.match(html, /Hearing|Linked project/i); + assert.doesNotMatch(html, /Could not reach/i); +}); + // --------------------------------------------------------------------------- // Council meeting outcomes: matched + unmatched // --------------------------------------------------------------------------- diff --git a/test/standards/stray_english_allowlist.txt b/test/standards/stray_english_allowlist.txt index 16853b91..da0c925c 100644 --- a/test/standards/stray_english_allowlist.txt +++ b/test/standards/stray_english_allowlist.txt @@ -45,7 +45,7 @@ Goods and Services Services (other than human services) # ── SODA query fragments / column expressions -,event_date,street_address_1,section_name +,event_date,street_address_1,section_name,additional_description_2,additional_description_3,other_info_2,other_info_3,printout_1,printout_2,printout_3,building_name,city,state,zip_code agency_name,sum(contract_amount) AS total type_of_notice_description AS t,count(request_id) AS n vendor_name,sum(contract_amount) AS total diff --git a/tools/capture_ida_notice_defects.py b/tools/capture_ida_notice_defects.py new file mode 100644 index 00000000..39d596ce --- /dev/null +++ b/tools/capture_ida_notice_defects.py @@ -0,0 +1,168 @@ +#!/usr/bin/env python3 +"""Before/after evidence for the IDA public-hearing defect cluster. + +Field case: #notice/20250227021 (homepage demo "Past IDA meetings"). + +Captures production (before) and the local site tree (after) for: + - meetings list card participation affordance + - notice detail modules (participation + absence of contract lifecycle) + +Also writes an offline subsidy after-shell (City Record hearing derivation) +because that fix ships with the Worker, not the static site alone. + + python3 tools/capture_ida_notice_defects.py +""" + +from __future__ import annotations + +import functools +import time +from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +import threading + +from playwright.sync_api import sync_playwright + +ROOT = Path(__file__).resolve().parents[1] +OUT = ROOT / "docs" / "screenshots" / "ida-notice-defects" +VIEWPORTS = ((390, 844), (1440, 900)) +NOTICE_ID = "20250227021" +PROD = "https://cityscroll.org/" +MEETINGS_HASH = "#meetings?when=past&q=IDA" +NOTICE_HASH = f"#notice/{NOTICE_ID}" + + +class QuietHandler(SimpleHTTPRequestHandler): + def log_message(self, _format: str, *_args: object) -> None: + pass + + +class StaticServer: + def __init__(self) -> None: + handler = functools.partial(QuietHandler, directory=str(ROOT / "site")) + self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler) + self.thread = threading.Thread(target=self.server.serve_forever, daemon=True) + + def __enter__(self) -> str: + self.thread.start() + return f"http://127.0.0.1:{self.server.server_port}/" + + def __exit__(self, *_exc: object) -> None: + self.server.shutdown() + self.thread.join(timeout=5) + self.server.server_close() + + +def wait_list(page) -> None: + page.wait_for_selector(".hcard, .fcard, #meetingsfeed .fcard", timeout=45000) + # Prefer an IDA-related card when present + page.wait_for_timeout(800) + + +def wait_notice(page) -> None: + page.wait_for_selector(".rolename, #noticeview .panel", timeout=45000) + # Lifecycle / subsidy slots paint async + page.wait_for_timeout(2500) + + +def shot(page, path: Path) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + page.screenshot(path=str(path), full_page=True) + + +def capture_live(browser, base: str, phase: str) -> None: + for width, height in VIEWPORTS: + context = browser.new_context(viewport={"width": width, "height": height}) + page = context.new_page() + page.goto(base + "index.html" + MEETINGS_HASH, wait_until="domcontentloaded", timeout=60000) + wait_list(page) + # Scroll first card into view + page.evaluate("() => { const c=document.querySelector('.hcard,.fcard'); if(c) c.scrollIntoView({block:'center'}); }") + shot(page, OUT / f"{phase}-meetings-list-{width}.png") + + page.goto(base + "index.html" + NOTICE_HASH, wait_until="domcontentloaded", timeout=60000) + wait_notice(page) + page.evaluate("() => { const a=document.querySelector('.actions'); if(a) a.scrollIntoView({block:'center'}); }") + shot(page, OUT / f"{phase}-notice-actions-{width}.png") + page.evaluate( + """() => { + const el = document.querySelector('#nlifecycle,#dlifecycle,#nsubsidy,#dsubsidy'); + if (el) el.scrollIntoView({block:'start'}); + else window.scrollTo(0, 400); + }""" + ) + shot(page, OUT / f"{phase}-notice-modules-{width}.png") + context.close() + + +SUBSIDY_AFTER_SHELL = """ + +IDA subsidy after +
+
CityScroll
+

✓ After · #notice/20250227021 · one IDA meetings link · no contract modules · subsidy from City Record hearing

+

IDA meetings page

+
Subsidy lifecycle
+
Linked project: IDA March 20th, 2025 Public Hearing Notice · company NYM 145 Wolcott LLC · stage Hearing.
+
+
Application
2025-02-27
+
→
+
Hearing
2025-03-20
+
→
+
Board decision
—
+
The city does not publish this — it would appear on Build NYC project documents if released.
+
+
Stages join City Record notices to Build NYC and NYC Industrial Development Agency project records when a public match exists. Hearing stage here is from the City Record notice when the Build NYC feed is unreachable.
+
+""" + + +def capture_subsidy_after(browser) -> None: + shell = OUT / "_subsidy_after.html" + shell.parent.mkdir(parents=True, exist_ok=True) + shell.write_text(SUBSIDY_AFTER_SHELL, encoding="utf-8") + for width, height in VIEWPORTS: + context = browser.new_context(viewport={"width": width, "height": height}) + page = context.new_page() + page.goto(shell.as_uri(), wait_until="load") + shot(page, OUT / f"after-subsidy-shell-{width}.png") + context.close() + + +def main() -> None: + OUT.mkdir(parents=True, exist_ok=True) + with sync_playwright() as p: + browser = p.chromium.launch() + print("Capturing production (before)…") + capture_live(browser, PROD, "before") + print("Capturing local site (after)…") + with StaticServer() as base: + # Give the server a beat + time.sleep(0.2) + capture_live(browser, base, "after") + print("Capturing subsidy after shell…") + capture_subsidy_after(browser) + browser.close() + print(f"Wrote screenshots under {OUT.relative_to(ROOT)}") + for path in sorted(OUT.glob("*.png")): + print(f" {path.name} {path.stat().st_size} bytes") + + +if __name__ == "__main__": + main() diff --git a/worker/src/lib/hearings.mjs b/worker/src/lib/hearings.mjs index ff36a876..1bd1f92c 100644 --- a/worker/src/lib/hearings.mjs +++ b/worker/src/lib/hearings.mjs @@ -168,11 +168,50 @@ function decisionSummary(row, body) { return title || "The notice does not give a short plain-language summary."; } +// Strip trailing punctuation that the body regex often captures (e.g. URL + ","). +// Dedupe runs on the cleaned form so "…hearings," and "…hearings" collapse to one link. +function normalizeParticipationUrl(url) { + return String(url || "").replace(/[.,;:)\]]+$/g, "").trim(); +} + +function participationUrlKey(url) { + try { + const parsed = new URL(url); + return `${parsed.host}${parsed.pathname}`.toLowerCase().replace(/\/$/, ""); + } catch { + return String(url || "").toLowerCase().replace(/\/$/, ""); + } +} + +// Honest labels: meeting-join platforms vs generic board calendars vs other. +// The common NYCIDA notice only publishes the board-meetings landing page — that is +// the deepest public target for those hearings, so label it as such rather than a +// vague "Participation link". +function participationLabel(url) { + if (/\b(?:zoom|webex|teams|meet\.google)\b/i.test(url)) return "Join online"; + if (/nycida-board-meetings-public-hearings/i.test(url) || /edc\.nyc\/nycida(?:[/?#]|$)/i.test(url)) { + return "IDA meetings page"; + } + return "Participation link"; +} + +// One outbound participation affordance per notice: prefer a live join URL, else the +// most specific cleaned URL the body published (longest path wins among equals). function participationFromRow(row, body, sourceUrl) { - const links = unique(body.match(URL_RE) || []).slice(0, 8).map((url) => ({ - label: /\b(?:zoom|webex|teams|meet\.google)\b/i.test(url) ? "Join online" : "Participation link", - url: url.replace(/[.,;]+$/, ""), - })); + const cleaned = (body.match(URL_RE) || []).map(normalizeParticipationUrl).filter(Boolean); + const byKey = new Map(); + for (const url of cleaned) { + const key = participationUrlKey(url); + if (!key || byKey.has(key)) continue; + byKey.set(key, { label: participationLabel(url), url }); + } + const ranked = [...byKey.values()].sort((a, b) => { + const aJoin = a.label === "Join online" ? 0 : 1; + const bJoin = b.label === "Join online" ? 0 : 1; + if (aJoin !== bJoin) return aJoin - bJoin; + return b.url.length - a.url.length; + }); + const links = ranked.slice(0, 1); const emails = unique([...body.matchAll(/\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b/gi)].map((match) => match[0])).slice(0, 4); const phones = unique([...body.matchAll(/(?:\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]\d{3}[-.\s]\d{4}/g)].map((match) => match[0])).slice(0, 4); return { diff --git a/worker/src/lib/subsidy_lifecycle.mjs b/worker/src/lib/subsidy_lifecycle.mjs index 0c930445..8a7a3e02 100644 --- a/worker/src/lib/subsidy_lifecycle.mjs +++ b/worker/src/lib/subsidy_lifecycle.mjs @@ -287,6 +287,8 @@ export function matchProjectToNotice(notice, projects = []) { let bestScore = 0; for (const project of projects) { + // Synthetic City Record-derived rows are a fallback, never preferred over a feed match. + if (String(project.project_id || "").startsWith("city-record:")) continue; const titleScore = overlapScore(noticeTitle, project.project_name); const companyScore = noticeCompany && project.company ? overlapScore(noticeCompany, project.company) * 0.8 @@ -300,16 +302,172 @@ export function matchProjectToNotice(notice, projects = []) { return bestScore >= 0.35 ? best : null; } +// When the Build NYC document feed is unreachable (or has no row for this hearing), derive a +// hearing-stage project from the City Record notice itself. IDA public-hearing notices are the +// public hearing stage — they list companies and applications. This is not a substitute for +// board/closing/compliance documents; those stay explicit unknowns until the feed recovers. +// Typical lag after a hearing before later Build NYC / IDA stages are public enough +// that absence is "not published" rather than "too soon". Conservative generic weeks — +// not measured publication stats. Temporal sibling of paid / verified_zero / unavailable. +export const SUBSIDY_STAGE_EXPECT_LAG_DAYS = { + board_decision: 60, // ~one board cycle after the hearing + closing: 180, // closing packages often land months later + compliance: 400, // first annual compliance window + project_record: 90, // whole-project join after hearing +}; + +export function daysSinceIso(iso, asOf = new Date()) { + const start = String(iso || "").slice(0, 10); + if (!/^\d{4}-\d{2}-\d{2}$/.test(start)) return null; + const a = Date.UTC(+start.slice(0, 4), +start.slice(5, 7) - 1, +start.slice(8, 10)); + const end = asOf instanceof Date ? asOf : new Date(asOf); + if (Number.isNaN(end.valueOf())) return null; + const b = Date.UTC(end.getUTCFullYear(), end.getUTCMonth(), end.getUTCDate()); + return Math.floor((b - a) / 86400000); +} + +export function lagWeeksForStage(stage) { + const days = SUBSIDY_STAGE_EXPECT_LAG_DAYS[stage] ?? SUBSIDY_STAGE_EXPECT_LAG_DAYS.project_record; + return Math.max(1, Math.round(days / 7)); +} + +// Three honest gap states for subsidy slots (when not matched): +// too_soon — typical publication lag has not elapsed +// not_published — lag elapsed; treat as class-(b) absence +// unavailable — feed/fetch failure (set by the worker path, not here) +export function subsidyGapKind({ stage = "project_record", anchorDate = null, asOf = new Date(), matched = false } = {}) { + if (matched) return null; + const days = daysSinceIso(anchorDate, asOf); + if (days == null) return "not_published"; + const lag = SUBSIDY_STAGE_EXPECT_LAG_DAYS[stage] ?? SUBSIDY_STAGE_EXPECT_LAG_DAYS.project_record; + if (days < lag) return "too_soon"; + return "not_published"; +} + +export function subsidyAnchorDate(notice = {}, lifecycle = null) { + const fromTimeline = (lifecycle?.timeline || []).find( + (e) => e && e.stage === STAGE_HEARING && e.date, + ); + return toDate( + fromTimeline?.date + || notice.event_date + || notice.start_date + || null, + ); +} + +export function isIdaHearingNotice(notice = {}) { + const agency = String(notice.agency_name || notice.agency || ""); + const title = String(notice.short_title || notice.title || ""); + const type = String(notice.type_of_notice_description || notice.notice_type || ""); + const section = String(notice.section_name || notice.source_section || ""); + const body = plainText([ + notice.additional_description_1, + notice.additional_description_2, + notice.additional_description_3, + notice.other_info_1, + ].filter(Boolean).join(" ")); + const ida = /industrial development|nycida|build nyc|economic development corporation/i.test( + `${agency} ${title} ${body}`, + ) || /\bIDA\b/.test(title); + if (!ida) return false; + const hearingType = /hearing|public hearing|meeting/i.test(`${type} ${title}`); + const hearingSection = section === "Public Hearings and Meetings" + || (section === "Agency Rules" && /public hearing/i.test(type)); + return hearingType || hearingSection || !!notice.event_date; +} + +export function projectFromIdaNotice(notice = {}) { + if (!isIdaHearingNotice(notice)) return null; + const requestId = String(notice.request_id || "").trim(); + if (!requestId) return null; + const body = plainText([ + notice.additional_description_1, + notice.additional_description_2, + notice.additional_description_3, + notice.other_info_1, + notice.other_info_2, + notice.other_info_3, + ].filter(Boolean).join(" ")); + const companies = unique( + [...body.matchAll(/Company Name\s*:\s*([^,(\n]{3,100})/gi)] + .map((match) => plainText(match[1])) + .filter(Boolean), + ); + const company = companies[0] || ""; + const cityRecordUrl = `https://a856-cityrecord.nyc.gov/RequestDetail/${encodeURIComponent(requestId)}`; + const hearingDate = toDate(notice.event_date || notice.start_date); + return { + request_id: requestId, + project_id: `city-record:${requestId}`, + project_name: plainText(notice.short_title || notice.title || `IDA hearing ${requestId}`), + company, + project_address: "", + location_text: body.slice(0, 400), + bbl: null, + bbls: [], + requested_benefit: moneyState(null), + estimated_cost: moneyState(null), + application: { + date: toDate(notice.start_date), + status: companies.length ? "filed" : "", + url: cityRecordUrl, + }, + hearing: { + date: hearingDate, + status: hearingDate ? "held" : "scheduled", + venue: "", + url: cityRecordUrl, + }, + board_decision: { date: null, outcome: "", body: "", url: "" }, + closing: { date: null, status: "", amount: null, url: "" }, + compliance: { year: "", date: null, status: "", url: "" }, + _derived_from: "city-record-hearing", + }; +} + +function withGapKinds(timeline, anchorDate) { + return (timeline || []).map((entry) => { + if (!entry || entry.status === "matched") return entry; + return { + ...entry, + gap_kind: subsidyGapKind({ + stage: entry.stage, + anchorDate, + matched: false, + }), + }; + }); +} + export function assembleSubsidyLifecycle(notices = [], projects = []) { return (notices || []).map((notice = {}) => { - const matched = matchProjectToNotice(notice, projects); + let matched = matchProjectToNotice(notice, projects); if (!matched) { + // Prefer an explicit synthetic row from the caller; otherwise derive from the notice. + matched = (projects || []).find( + (p) => p && p.request_id === String(notice.request_id || "").trim() + && String(p.project_id || "").startsWith("city-record:"), + ) || projectFromIdaNotice(notice); + } + const anchor = subsidyAnchorDate(notice, null); + if (!matched) { + const bareTimeline = STAGES.map((stage) => ({ + stage, + status: UNKNOWN, + date: null, + official_action: stage === STAGE_APPLICATION ? "application_review" : stage === STAGE_HEARING ? "public_hearing" : stage === STAGE_BOARD_DECISION ? "board_decision" : stage === STAGE_CLOSING ? "closing" : "annual_compliance", + outcome: UNKNOWN, + source: docState(), + detail: null, + })); return { request_id: String(notice.request_id || "").trim() || null, project: null, stage: STAGE_APPLICATION, join: { matched: false, + gap_kind: subsidyGapKind({ stage: "project_record", anchorDate: anchor, matched: false }), reason: "No matching NYCIDA/Build NYC project record was linked to this notice from public sources.", }, company: { @@ -336,21 +494,20 @@ export function assembleSubsidyLifecycle(notices = [], projects = []) { closing: docState(), compliance: docState(), }, - timeline: STAGES.map((stage) => ({ - stage, - status: UNKNOWN, - date: null, - official_action: stage === STAGE_APPLICATION ? "application_review" : stage === STAGE_HEARING ? "public_hearing" : stage === STAGE_BOARD_DECISION ? "board_decision" : stage === STAGE_CLOSING ? "closing" : "annual_compliance", - outcome: UNKNOWN, - source: docState(), - detail: null, - })), + timeline: withGapKinds(bareTimeline, anchor), }; } const place = buildPlace(notice, matched); - const timeline = buildTimeline(matched); + const hearingAnchor = matched.hearing?.date || anchor || subsidyAnchorDate(notice, null); + const timeline = withGapKinds(buildTimeline(matched), hearingAnchor); const current = resolveStage(matched); + const fromCityRecord = matched._derived_from === "city-record-hearing" + || String(matched.project_id || "").startsWith("city-record:"); + const joinSource = fromCityRecord ? "City Record" : "Build NYC"; + const joinMethod = fromCityRecord + ? "city-record-hearing" + : (matched.request_id ? "request_id" : "title_or_company_overlap"); return { request_id: String(notice.request_id || "").trim() || null, project: { @@ -361,14 +518,15 @@ export function assembleSubsidyLifecycle(notices = [], projects = []) { stage: current, join: { matched: true, - method: matched.request_id ? "request_id" : "title_or_company_overlap", - source: "Build NYC", + method: joinMethod, + source: joinSource, project_reference: matched.project_id || matched.request_id || null, + anchor_date: hearingAnchor || null, }, company: { status: matched.company ? "matched" : UNKNOWN, value: matched.company || null, - source: matched.company ? "Build NYC" : null, + source: matched.company ? joinSource : null, }, place: { status: place.status, diff --git a/worker/src/subsidy_lifecycle.mjs b/worker/src/subsidy_lifecycle.mjs index 8d632803..5bf267f9 100644 --- a/worker/src/subsidy_lifecycle.mjs +++ b/worker/src/subsidy_lifecycle.mjs @@ -48,22 +48,61 @@ function parseEmbeddedProjectJson(text) { return []; } +// Cloudflare challenge HTML is not a project feed — treat as source failure so callers can +// fall back to City Record notice derivation instead of caching an empty "ok" feed forever. +function looksLikeBotChallenge(text) { + return /just a moment|cf-browser-verification|challenge-platform|_cf_chl|cdn-cgi\/challenge/i + .test(String(text || "")); +} + async function fetchSubsidyProjects() { let text; try { - const r = await fetch(SUBSIDY_SOURCE, { headers: { Accept: "application/json, text/html" } }); + const r = await fetch(SUBSIDY_SOURCE, { + headers: { + Accept: "application/json, text/html", + "User-Agent": "CityScrollBot/1.0 (+https://cityscroll.org; subsidy-lifecycle)", + }, + }); if (!r.ok) return { projects: [], ok: false, sourceError: `Feed returned ${r.status}` }; text = await r.text(); } catch (error) { return { projects: [], ok: false, sourceError: String(error?.message || error) }; } + if (looksLikeBotChallenge(text)) { + return { projects: [], ok: false, sourceError: "Feed blocked by bot challenge" }; + } + const jsonRows = parseJsonishText(text); const scriptRows = jsonRows.length ? jsonRows : parseEmbeddedProjectJson(text); const projects = parseNYCIDAProjects(Array.isArray(scriptRows) ? scriptRows : []); + // Empty body with no project JSON is a soft failure when the page is the docs landing + // shell rather than a machine-readable feed (common when the host changes markup). + if (!projects.length && !jsonRows.length && !scriptRows.length) { + return { projects: [], ok: false, sourceError: "Feed returned no project records" }; + } return { projects, ok: true }; } +// Runtime safety net if migration 0005 was not applied (same posture as PASSPort D1). +export async function ensureSubsidySchema(env) { + if (!env?.DB) return; + try { + await env.DB.prepare(` + CREATE TABLE IF NOT EXISTS subsidy_lifecycle ( + request_id TEXT PRIMARY KEY, + agency TEXT, + lifecycle TEXT, + computed_at TEXT + ) + `).run(); + await env.DB.prepare( + "CREATE INDEX IF NOT EXISTS idx_subsidy_lifecycle_agency ON subsidy_lifecycle(agency)", + ).run(); + } catch { /* non-fatal */ } +} + export async function fetchNoticeRow(env, requestId) { if (env.DB) { try { @@ -87,11 +126,15 @@ export async function fetchNoticeRow(env, requestId) { async function cacheGet(env, requestId) { if (!env.DB) return null; try { + await ensureSubsidySchema(env); const row = await env.DB.prepare( "SELECT lifecycle FROM subsidy_lifecycle WHERE request_id = ?", ).bind(requestId).first(); if (row && row.lifecycle) { const lifecycle = JSON.parse(row.lifecycle); + // Never serve a permanently cached feed-unavailable row — recompute so City Record + // derivation or a recovered feed can replace the operational error. + if (lifecycle && lifecycle.source_status === "unavailable") return null; if (lifecycle && Array.isArray(lifecycle.timeline)) return lifecycle; } } catch { /* miss */ } @@ -99,8 +142,11 @@ async function cacheGet(env, requestId) { } async function cachePut(env, requestId, agency, lifecycle) { - if (!env.DB) return; + if (!env.DB || !lifecycle) return; + // Do not materialize unavailable — genuine transient failures must not stick forever. + if (lifecycle.source_status === "unavailable") return; try { + await ensureSubsidySchema(env); await env.DB.prepare( `INSERT OR REPLACE INTO subsidy_lifecycle (request_id, agency, lifecycle, computed_at) VALUES (?, ?, ?, ?)`, @@ -112,12 +158,24 @@ export async function computeLifecycle(env, requestId, noticeRow) { const notice = noticeRow === undefined ? await fetchNoticeRow(env, requestId) : noticeRow; if (!notice) return { lifecycle: null, ok: false, sourceUnavailable: false }; const { projects, ok } = await fetchSubsidyProjects(); - if (!ok) { - const fallback = assembleSubsidyLifecycle([notice], [])[0] ?? null; - if (fallback) fallback.source_status = "unavailable"; - return { lifecycle: fallback, ok: false, sourceUnavailable: true }; + // assembleSubsidyLifecycle derives a City Record hearing project for IDA notices when + // the Build NYC feed is empty or blocked — that is a real public join, not "unavailable". + const [lifecycle] = assembleSubsidyLifecycle([notice], ok ? projects : []); + if (!lifecycle) return { lifecycle: null, ok: false, sourceUnavailable: !ok }; + if (!ok && !lifecycle.join?.matched) { + // Feed down and no notice-derived hearing row → honest operational unavailable. + lifecycle.source_status = "unavailable"; + return { lifecycle, ok: false, sourceUnavailable: true }; + } + // Feed down but City Record hearing matched: partial source, not a blank failure. + if (!ok && lifecycle.join?.matched) { + lifecycle.source_status = "ok"; + lifecycle.join = { + ...lifecycle.join, + feed_status: "unavailable", + feed_note: "Build NYC document feed unreachable; hearing stage from City Record notice.", + }; } - const [lifecycle] = assembleSubsidyLifecycle([notice], projects); return { lifecycle, ok: true, sourceUnavailable: false }; } @@ -126,8 +184,7 @@ export async function getOrCompute(env, requestId) { if (cached) return { lifecycle: cached, ok: true, sourceUnavailable: cached.source_status === "unavailable" }; const { lifecycle, ok, sourceUnavailable } = await computeLifecycle(env, requestId); - // Cache matched and explicit-gap rows so the notice detail always gets a structured - // lifecycle (join.matched false, source_status unavailable, etc.) — never a blank. + // Cache matched and explicit taxonomy-gap rows; never unavailable (see cachePut). if (lifecycle) { const row = await fetchNoticeRow(env, requestId).catch(() => null); await cachePut(env, requestId, row && row.agency_name, lifecycle);