Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,44 @@ Migration value baseline (merge-to-live wall-clock, detection exemplars, rollbac
Characterization: `node --test test/live_url_smoke.test.mjs test/post_flip_checks.test.mjs`. Operator flip procedure lives outside this public tree.


## Hearing participation (one owner, list + detail)

Meetings list cards and notice permalinks share one derivation:
`normalizeHearing` / `normalizeHearingRow` → `participation.links` →
`participationLinksHTML` in `site/index.html`. Strip trailing punctuation
**before** dedupe (body often has `https://…hearings,` and `https://…hearings`);
one outbound affordance per notice. NYCIDA board URL labels as **IDA meetings page**
(the deepest public target those notices publish). Characterization:
`node --test test/ida_notice_defects.test.mjs`. Captures:
`python3 tools/capture_ida_notice_defects.py`.

## Contract lifecycle category gate

`isContractLifecycleEligible` — Procurement section or Solicitation/Award/Intent
types only. Hearings, Agency Rules, Property Disposition, and Changes in Personnel
never mount contract lifecycle / OCP / PIN gap modules (wrong-universe). Subsidy and
meeting-outcomes keep their own eligibility helpers. Characterization:
`test/ida_notice_defects.test.mjs`, `test/lifecycle_coherence_field_cases.test.mjs`.

## Subsidy lifecycle (NYCIDA / Build NYC)

Endpoint `GET /subsidy-lifecycle?id=` (`worker/src/subsidy_lifecycle.mjs`). The
EDC documents page is often Cloudflare-blocked to edge fetch (HTTP 403 / challenge
HTML) — treat as feed failure, do **not** permanently D1-cache `source_status:
unavailable`. When the feed fails, `projectFromIdaNotice` derives a hearing-stage
join from the City Record IDA hearing notice (company names, event date). Keep
honest unavailable copy only when the feed is down **and** no notice-derived
hearing applies. Schema safety net: `ensureSubsidySchema` (migration
`0005_subsidy_lifecycle.sql`).

**Age-aware gap kinds** (temporal sibling of paid / verified_zero / unavailable):
`subsidyGapKind` → `too_soon` | `not_published` | (worker) `unavailable`. Lag table
`SUBSIDY_STAGE_EXPECT_LAG_DAYS` (board ~60d, closing ~180d, project_record ~90d).
Demo/backtest notices must be **aged** (2022–2024 hearings) so later stages read
`not_published`, not “could not reach.” Young hearings use “check back” copy.
Characterization: `test/subsidy_lifecycle.test.mjs`, `test/ida_notice_defects.test.mjs`.
Aged demo ids: `20220525018`, `20231004016`, `20240617012`.

## Checkbook Contracts row identity

Checkbook's Contracts domain returns **multiple rows per `prime_contract_id`** (one Prime Vendor row with amounts, plus Sub Vendor / expense-category slices with $0 on prime fields). Lifecycle assembly collapses rows with `aggregateContractsById` before `classifyStage` — one distinct id = matched; ≥2 distinct ids = ambiguous. Field case: notice `20231222103` / `CT107120248803393`. Do not count raw Contracts rows as separate contracts. Spending rows stay uncollapsed (many payments per contract is normal). Pure lib: `worker/src/lib/checkbook_lifecycle.mjs`.
Expand Down
35 changes: 35 additions & 0 deletions docs/screenshots/ida-notice-defects/_subsidy_after.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
<!doctype html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>IDA subsidy after</title>
<style>
:root{--paper:#f4efe5;--ink:#161512;--muted:#69635a;--rule:#c8bfb0;--green:#285d49;--ox:#7b1f2b}
body{margin:0;background:var(--paper);color:var(--ink);font:16px/1.45 ui-sans-serif,system-ui,sans-serif}
.page{max-width:880px;margin:0 auto;padding:20px}
.brand{font:900 24px/1 Georgia,serif}.brand span{color:var(--ox)}
.chain-h{font:700 12px/1 ui-sans-serif,system-ui,sans-serif;letter-spacing:.12em;text-transform:uppercase;margin:18px 0 10px}
.note{font:13px/1.45 ui-sans-serif,system-ui,sans-serif;color:var(--muted);padding:10px 12px;background:#faf6ee;border-inline-start:3px solid var(--rule);border-radius:0 7px 7px 0;margin:8px 0}
.chain{display:flex;flex-wrap:wrap;gap:0;align-items:stretch}
.stage{flex:1 1 120px;min-width:110px}
.box{border:1px solid var(--rule);border-top:3px solid var(--green);border-radius:8px;background:#fff;padding:11px 12px;min-height:100px}
.stage-name{font:700 11px/1 ui-sans-serif,system-ui,sans-serif;letter-spacing:.1em;text-transform:uppercase;color:var(--green)}
.when{font-size:13px;margin-top:4px}
.lc-norecord{font:12px/1.4 ui-sans-serif,system-ui,sans-serif;color:var(--muted);margin-top:6px}
.connector{display:flex;align-items:center;padding:0 4px;color:var(--muted)}
.fix{border:2px solid var(--green);background:#f0faf5;color:var(--green);font:700 11px/1.3 ui-sans-serif,system-ui,sans-serif;padding:6px 8px;border-radius:6px;margin:8px 0}
.act{display:inline-block;margin:4px 6px 4px 0;padding:8px 12px;border:1px solid var(--rule);border-radius:6px;background:#fff;text-decoration:none;color:var(--ink);font:600 13px/1 ui-sans-serif,system-ui,sans-serif}
</style></head><body><div class="page">
<div class="brand">City<span>Scroll</span></div>
<p class="fix">✓ After · #notice/20250227021 · one IDA meetings link · no contract modules · subsidy from City Record hearing</p>
<p><a class="act" href="https://edc.nyc/nycida-board-meetings-public-hearings">IDA meetings page</a></p>
<div class="chain-h">Subsidy lifecycle</div>
<div class="note">Linked project: <b>IDA March 20th, 2025 Public Hearing Notice</b> · company <b>NYM 145 Wolcott LLC</b> · stage Hearing.</div>
<div class="chain">
<div class="stage"><div class="box"><div class="stage-name">Application</div><div class="when">2025-02-27</div></div></div>
<div class="connector">→</div>
<div class="stage"><div class="box"><div class="stage-name">Hearing</div><div class="when">2025-03-20</div></div></div>
<div class="connector">→</div>
<div class="stage"><div class="box" style="border-top-color:var(--rule)"><div class="stage-name" style="color:var(--muted)">Board decision</div><div class="when">—</div>
<div class="lc-norecord">The city does not publish this — it would appear on Build NYC project documents if released.</div></div></div>
</div>
<div class="note">Stages join City Record notices to Build NYC and NYC Industrial Development Agency project records when a public match exists. Hearing stage here is from the City Record notice when the Build NYC feed is unreachable.</div>
</div></body></html>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
50 changes: 42 additions & 8 deletions site/hearing_location.js
Original file line number Diff line number Diff line change
Expand Up @@ -227,17 +227,51 @@ function normalizeHearingRow(row) {
published_at: row.start_date || null, decides: hearingDecision(row, body),
affects: audience ? [audience[1]] : [], affected_area: hearingAffectedArea(row),
venue: hearingVenue(row),
participation: {
links: hearingUnique(body.match(HEARING_URL_RE) || []).slice(0, 8).map(function (url) {
return { label: /\b(?:zoom|webex|teams|meet\.google)\b/i.test(url) ? "Join online" : "Participation link", url: url.replace(/[.,;]+$/, "") };
}),
emails: hearingUnique(Array.from(body.matchAll(/\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b/gi)).map(function (match) { return match[0]; })).slice(0, 4),
phones: hearingUnique(Array.from(body.matchAll(/(?:\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]\d{3}[-.\s]\d{4}/g)).map(function (match) { return match[0]; })).slice(0, 4),
source_url: source,
},
participation: hearingParticipationFromBody(body, source),
source_url: source, description: body.slice(0, 1200),
};
}
// Hand-synced with worker/src/lib/hearings.mjs participationFromRow: strip trailing
// punctuation before dedupe so "…hearings," and "…hearings" collapse; one affordance.
function hearingNormalizeParticipationUrl(url) {
return String(url || "").replace(/[.,;:)\]]+$/g, "").trim();
}
function hearingParticipationUrlKey(url) {
try {
var parsed = new URL(url);
return (parsed.host + parsed.pathname).toLowerCase().replace(/\/$/, "");
} catch (e) {
return String(url || "").toLowerCase().replace(/\/$/, "");
}
}
function hearingParticipationLabel(url) {
if (/\b(?:zoom|webex|teams|meet\.google)\b/i.test(url)) return "Join online";
if (/nycida-board-meetings-public-hearings/i.test(url) || /edc\.nyc\/nycida(?:[/?#]|$)/i.test(url)) {
return "IDA meetings page";
}
return "Participation link";
}
function hearingParticipationFromBody(body, source) {
var cleaned = (String(body || "").match(HEARING_URL_RE) || []).map(hearingNormalizeParticipationUrl).filter(Boolean);
var byKey = Object.create(null);
cleaned.forEach(function (url) {
var key = hearingParticipationUrlKey(url);
if (!key || byKey[key]) return;
byKey[key] = { label: hearingParticipationLabel(url), url: url };
});
var ranked = Object.keys(byKey).map(function (k) { return byKey[k]; }).sort(function (a, b) {
var aJoin = a.label === "Join online" ? 0 : 1;
var bJoin = b.label === "Join online" ? 0 : 1;
if (aJoin !== bJoin) return aJoin - bJoin;
return b.url.length - a.url.length;
});
return {
links: ranked.slice(0, 1),
emails: hearingUnique(Array.from(String(body || "").matchAll(/\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b/gi)).map(function (match) { return match[0]; })).slice(0, 4),
phones: hearingUnique(Array.from(String(body || "").matchAll(/(?:\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]\d{3}[-.\s]\d{4}/g)).map(function (match) { return match[0]; })).slice(0, 4),
source_url: source,
};
}
function hearingMatchesArea(record, filter) {
var f = filter || {}, area = record.affected_area || {}, borough = String(f.borough || "").toLowerCase();
var neighborhood = String(f.neighborhood || "").trim().toLowerCase();
Expand Down
6 changes: 6 additions & 0 deletions site/i18n.js
Original file line number Diff line number Diff line change
Expand Up @@ -403,6 +403,7 @@ const STRINGS = {
read_official_notice: "Read official notice",
join_online: "Join online",
participation_link: "Participation link",
ida_meetings_page: "IDA meetings page",
email_in_notice: "Email listed in notice",
map_venue: "Map venue",
venue_label: "Venue",
Expand Down Expand Up @@ -968,6 +969,11 @@ const STRINGS = {
subsidy_outcome_html: "Outcome: {outcome}",
subsidy_outcome_unknown_html: "The city does not publish this outcome — it would appear on the Build NYC project documents if released.",
subsidy_stage_unmatched_html: "Not yet shown here — {stage} records live in {source}.",
// Age-aware gaps: too_soon vs not_published vs unavailable (fetch). Temporal sibling of
// paid / verified_zero / unavailable on the Checkbook payment path.
subsidy_stage_too_soon_html: "Hearing was {date} — {stage} records typically appear within about {weeks} weeks. Check back. Too recent to treat as a permanent gap.",
subsidy_stage_not_published_html: "The city does not publish this {stage} record on {source} — it would appear there if released.",
subsidy_join_too_soon_html: "Hearing was {date} — IDA project records for “{title}” usually appear within about {weeks} weeks of the hearing. Check back. Too recent to treat as a permanent gap.",
subsidy_source_build_nyc: "Build NYC and NYC Industrial Development Agency",
subsidy_source_unavailable_html: "Could not reach {source} to check the subsidy timeline for this notice.",
subsidy_unmatched_html: "The city does not publish a linked subsidy project for “{title}” — {reason}",
Expand Down
4 changes: 4 additions & 0 deletions site/i18n/lang/ar.js
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,7 @@
read_official_notice: "قراءة الإشعار الرسمي",
join_online: "الانضمام عبر الإنترنت",
participation_link: "رابط المشاركة",
ida_meetings_page: "صفحة اجتماعات IDA",
email_in_notice: "البريد الوارد في الإشعار",
map_venue: "خريطة مكان الجلسة",
venue_label: "مكان الجلسة",
Expand Down Expand Up @@ -636,6 +637,9 @@
subsidy_outcome_html: "النتيجة: {outcome}",
subsidy_outcome_unknown_html: "المدينة لا تنشر هذه النتيجة — ستظهر في مستندات مشروع Build NYC إذا نُشرت.",
subsidy_stage_unmatched_html: "لم يُعرض هنا بعد — سجلات {stage} موجودة في {source}.",
subsidy_stage_too_soon_html: "كانت الجلسة في {date} — تظهر سجلات {stage} عادةً خلال نحو {weeks} أسابيع. راجع لاحقًا. أبكر من أن تُعامل كفجوة دائمة.",
subsidy_stage_not_published_html: "المدينة لا تنشر سجل {stage} هذا على {source} — سيظهر هناك إذا نُشر.",
subsidy_join_too_soon_html: "كانت الجلسة في {date} — سجلات مشاريع IDA لـ “{title}” تظهر عادةً خلال نحو {weeks} أسابيع بعد الجلسة. راجع لاحقًا. أبكر من أن تُعامل كفجوة دائمة.",
subsidy_source_build_nyc: "Build NYC و NYC Industrial Development Agency",
subsidy_source_unavailable_html: "تعذّر الوصول إلى {source} للتحقق من الجدول الزمني للدعم لهذا الإشعار.",
subsidy_unmatched_html: "المدينة لا تنشر مشروع دعم مرتبطًا بـ “{title}” — {reason}",
Expand Down
4 changes: 4 additions & 0 deletions site/i18n/lang/bn.js
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,7 @@
read_official_notice: "সরকারি বিজ্ঞপ্তি পড়ুন",
join_online: "অনলাইনে যোগ দিন",
participation_link: "অংশগ্রহণের লিংক",
ida_meetings_page: "IDA সভার পৃষ্ঠা",
email_in_notice: "বিজ্ঞপ্তির ইমেইল",
map_venue: "সভাস্থলের মানচিত্র",
venue_label: "সভাস্থল",
Expand Down Expand Up @@ -629,6 +630,9 @@
subsidy_outcome_html: "ফলাফল: {outcome}",
subsidy_outcome_unknown_html: "শহর এই ফলাফল প্রকাশ করে না — প্রকাশিত হলে এটি Build NYC প্রকল্পের নথিতে থাকত।",
subsidy_stage_unmatched_html: "এখানে এখনও দেখানো হয়নি — {stage} রেকর্ড {source}-এ আছে।",
subsidy_stage_too_soon_html: "শুনানি হয়েছিল {date} — {stage} রেকর্ড সাধারণত প্রায় {weeks} সপ্তাহের মধ্যে আসে। পরে দেখুন। স্থায়ী ফাঁক ধরতে এখনও তাড়াতাড়ি।",
subsidy_stage_not_published_html: "শহর {source}-এ এই {stage} রেকর্ড প্রকাশ করে না — প্রকাশিত হলে সেখানে থাকত।",
subsidy_join_too_soon_html: "শুনানি হয়েছিল {date} — “{title}”-এর IDA প্রকল্প রেকর্ড সাধারণত শুনানির প্রায় {weeks} সপ্তাহের মধ্যে আসে। পরে দেখুন। স্থায়ী ফাঁক ধরতে এখনও তাড়াতাড়ি।",
subsidy_source_build_nyc: "Build NYC এবং NYC Industrial Development Agency",
subsidy_source_unavailable_html: "এই বিজ্ঞপ্তির ভর্তুকি সময়রেখা যাচাই করতে {source}-এ পৌঁছানো যায়নি।",
subsidy_unmatched_html: "শহর “{title}”-এর জন্য কোনো সংযুক্ত ভর্তুকি প্রকল্প প্রকাশ করে না — {reason}",
Expand Down
4 changes: 4 additions & 0 deletions site/i18n/lang/es.js
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,7 @@
read_official_notice: "Leer aviso oficial",
join_online: "Participar en línea",
participation_link: "Enlace para participar",
ida_meetings_page: "Página de reuniones de la IDA",
email_in_notice: "Correo del aviso",
map_venue: "Mapa del lugar",
venue_label: "Lugar",
Expand Down Expand Up @@ -629,6 +630,9 @@
subsidy_outcome_html: "Resultado: {outcome}",
subsidy_outcome_unknown_html: "La ciudad no publica este resultado — aparecería en los documentos del proyecto Build NYC si se publicara.",
subsidy_stage_unmatched_html: "Aún no se muestra aquí — los registros de {stage} están en {source}.",
subsidy_stage_too_soon_html: "Hearing was {date} — los registros de {stage} suelen aparecer en unas {weeks} semanas. Vuelva a consultar. es demasiado reciente para tratarlo como un vacío permanente.",
subsidy_stage_not_published_html: "La ciudad no publica este registro de {stage} en {source} — aparecería allí si se publicara.",
subsidy_join_too_soon_html: "La audiencia fue el {date} — los registros de proyectos de la IDA para “{title}” suelen aparecer en unas {weeks} semanas tras la audiencia. Vuelva a consultar. es demasiado reciente para tratarlo como un vacío permanente.",
subsidy_source_build_nyc: "Build NYC y la Agencia de Desarrollo Industrial de NYC",
subsidy_source_unavailable_html: "No se pudo acceder a {source} para verificar la línea de tiempo del subsidio de este aviso.",
subsidy_unmatched_html: "La ciudad no publica un proyecto de subsidio vinculado para “{title}” — {reason}",
Expand Down
4 changes: 4 additions & 0 deletions site/i18n/lang/fr.js
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,7 @@
read_official_notice: "Lire l’avis officiel",
join_online: "Participer en ligne",
participation_link: "Lien de participation",
ida_meetings_page: "Page des réunions de l’IDA",
email_in_notice: "Courriel indiqué",
map_venue: "Carte du lieu",
venue_label: "Lieu",
Expand Down Expand Up @@ -711,6 +712,9 @@
subsidy_outcome_html: "Résultat : {outcome}",
subsidy_outcome_unknown_html: "La ville ne publie pas ce résultat — il apparaîtrait sur les documents du projet Build NYC s'il était publié.",
subsidy_stage_unmatched_html: "Pas encore affiché ici — les dossiers de {stage} sont dans {source}.",
subsidy_stage_too_soon_html: "Audience le {date} — les dossiers de {stage} apparaissent en général dans un délai d’environ {weeks} semaines. Revenez plus tard. trop récent pour un écart permanent.",
subsidy_stage_not_published_html: "La ville ne publie pas ce dossier de {stage} sur {source} — il y figurerait s’il était publié.",
subsidy_join_too_soon_html: "Audience le {date} — les dossiers de projet IDA pour « {title} » apparaissent en général dans un délai d’environ {weeks} semaines après l’audience. Revenez plus tard. trop récent pour un écart permanent.",
subsidy_source_build_nyc: "Build NYC et NYC Industrial Development Agency",
subsidy_source_unavailable_html: "Impossible d'atteindre {source} pour vérifier le calendrier de subvention de cet avis.",
subsidy_unmatched_html: "La ville ne publie pas de projet de subvention lié pour « {title} » — {reason}",
Expand Down
4 changes: 4 additions & 0 deletions site/i18n/lang/ht.js
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,7 @@
read_official_notice: "Li avi ofisyèl la",
join_online: "Patisipe sou entènèt",
participation_link: "Lyen pou patisipe",
ida_meetings_page: "Paj reyinyon IDA",
email_in_notice: "Imèl nan avi a",
map_venue: "Kat kote a",
venue_label: "Kote odyans lan",
Expand Down Expand Up @@ -628,6 +629,9 @@
subsidy_outcome_html: "Rezilta: {outcome}",
subsidy_outcome_unknown_html: "Vil la pa pibliye rezilta sa a — li t ap parèt nan dokiman pwojè Build NYC si yo te pibliye l.",
subsidy_stage_unmatched_html: "Pokò a pa montre isit la — dosye {stage} yo nan {source}.",
subsidy_stage_too_soon_html: "Odyans lan te {date} — dosye {stage} anjeneral parèt nan apeprè {weeks} semèn. Tcheke pita. twò resan pou trete kòm yon twou pèmanan.",
subsidy_stage_not_published_html: "Vil la pa pibliye dosye {stage} sa a sou {source} — li t ap parèt la si yo te pibliye l.",
subsidy_join_too_soon_html: "Odyans lan te {date} — dosye pwojè IDA pou “{title}” anjeneral parèt nan apeprè {weeks} semèn apre odyans lan. Tcheke pita. twò resan pou trete kòm yon twou pèmanan.",
subsidy_source_build_nyc: "Build NYC ak NYC Industrial Development Agency",
subsidy_source_unavailable_html: "Pa t kapab rive jwenn {source} pou verifye delè sibvansyon avi sa a.",
subsidy_unmatched_html: "Vil la pa pibliye yon pwojè sibvansyon ki lye pou “{title}” — {reason}",
Expand Down
Loading
Loading