Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,10 @@ prefer otherwise.
token; the inbound-email parser has a daily surface ceiling, a per-sender limit, size clamps,
and auto-reply/loop guards; email sending has per-run and per-day caps. An unconfigured or
capped route degrades, it doesn't fall open.
- **Abuse of write routes** (`/subscribe`, `/feedback`, `/inv`, `/batch`): Turnstile where a
human is asserted, per-IP and per-address daily rate limits, strict validation, size clamps,
TTLs on everything stored.
- **Abuse of write routes** (`/subscribe`, `/feedback`, `/inv`, `/batch`): per-IP and
per-address daily rate limits, strict validation, size clamps, TTLs on everything stored;
`/feedback` still uses Turnstile; `/subscribe` relies on rate limits + double opt-in (CAPTCHA
can return if abuse shows in the sends dashboard).
- **Email**: double opt-in (nothing stored until the signed confirm link is clicked), signed
one-click unsubscribe (RFC 8058), From is always the app's own identity.
- **No open redirect**: `/r` accepts a validated slug + record id and constructs the target
Expand Down
8 changes: 4 additions & 4 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ Browser (cityscroll.org — canonical Worker mirror of static GitHub Pages)
├── /mcp MCP for AI assistants: search/get/preview_watch/create_watch (metered)
├── /checkbook Checkbook NYC proxy + expiration pipeline (fc:* cache)
├── /forecast Checkbook contract-expiration estimate timeline
├── /subscribe /confirm /unsubscribe double-opt-in email (Turnstile-gated)
├── /subscribe /confirm /unsubscribe double-opt-in email (rate-limited)
├── /feedback operator feedback form (Turnstile-gated, fails closed)
├── /feed.xml /feed.json /feed.ics standing feeds from any saved search
├── /batch watchlist cross-reference
Expand Down Expand Up @@ -188,7 +188,7 @@ Bottom-up, the way it's built: public Socrata feeds and Checkbook are the ground
- `cityscroll.org` / `www.cityscroll.org` are the canonical site hosts (custom-domain routes in `worker/wrangler.toml`). The Worker normally reverse-proxies the GitHub Pages origin at `crol-list.org` byte-for-byte (`worker/src/mirror.mjs`). Origin redirects are manual; a redirect back to CityScroll trips a circuit breaker and retries through GitHub's public repository source seam.
- Direct visitors to `crol-list.org` / `www.crol-list.org` receive a 301 to the matching CityScroll path and query. The mirror's independent redirect-loop failover keeps the canonical site available if an origin fetch is redirected back at the Worker. Fragments remain client-side and are retained by conforming browsers.
- New feed, confirmation, redirect, and API URLs mint on CityScroll. Existing calendar UIDs retain `@crol-list` and Atom entries retain `tag:crol-list.org,2026:` so calendar and feed clients do not create duplicates. Outbound alerts are sent from `alerts@cityscroll.org` with Reply-To `alerts@crol-list.org` (still-routable); inbound operational routing remains on `@crol-list.org` (`subscribe@`, `feedback@`) unless separately redirected by provider policy.
- Secrets are stored outside the repository (Wrangler secret bindings). Bindings referenced by code include `ANTHROPIC_API_KEY`, `RESEND_API_KEY`, `TURNSTILE_SECRET`, `TOKEN_SECRET`, `USAGE_KEY`, `ANALYTICS_READ_TOKEN`, and `ANALYTICS_DEV_KEY`. The production analytics write gate `ANALYTICS_ENVIRONMENT=production` is a non-secret var in `wrangler.toml` (beta overrides it to `preview`); a missing or non-production value drops Analytics Engine writes. The developer key authenticates short-lived HMAC exclusions. Spend guards are vars in `wrangler.toml`: `MAX_PER_RUN=25`, `MAX_SENDS_PER_DAY=50` (under Resend's free 100/day); `/subscribe` and `/feedback` fail closed (503) if their secrets are absent.
- Secrets are stored outside the repository (Wrangler secret bindings). Bindings referenced by code include `ANTHROPIC_API_KEY`, `RESEND_API_KEY`, `TURNSTILE_SECRET`, `TOKEN_SECRET`, `USAGE_KEY`, `ANALYTICS_READ_TOKEN`, and `ANALYTICS_DEV_KEY`. The production analytics write gate `ANALYTICS_ENVIRONMENT=production` is a non-secret var in `wrangler.toml` (beta overrides it to `preview`); a missing or non-production value drops Analytics Engine writes. The developer key authenticates short-lived HMAC exclusions. Spend guards are vars in `wrangler.toml`: `MAX_PER_RUN=25`, `MAX_SENDS_PER_DAY=50` (under Resend's free 100/day); `/subscribe` fails closed (503) without `TOKEN_SECRET` + `RESEND_API_KEY` + `SUBS` (no Turnstile on this path); `/feedback` still fails closed without Turnstile + Resend + `FEEDBACK`.
- GitHub Actions is path-filtered in `CI` using `dorny/paths-filter`; worker/docs/frontend jobs run only when their lanes changed.
PR and merge tests include source-contract verification against committed fixtures:
`tools/verify_source_contracts.mjs`, plus a blocking 20-sample p95 browser-performance
Expand Down Expand Up @@ -231,7 +231,7 @@ Bottom-up, the way it's built: public Socrata feeds and Checkbook are the ground

1. A visitor loads `site/index.html` (inline CSS + vanilla JS) at canonical `cityscroll.org`, mirrored from the static GitHub Pages origin — no application backend required.
2. Picking a lens fires queries direct from the browser to CORS-open public APIs: Socrata SODA for City Record notices and ABO awards, plus GeoSearch/MapPLUTO for BBL and rezoning geometry. Checkbook queries use the schema-agnostic worker proxy.
3. Server-only features route to `api.cityscroll.org`: `/nl` (plain English → filters via Claude Haiku, metered by `NL_METER`), `/subscribe`→`/confirm`→`/unsubscribe` (double-opt-in, Turnstile-gated, fails closed), feeds, `/batch`, `/agencies`, `/inv`, `/stats`, `/feedback`, keyed `/admin/*` and `/usage`.
3. Server-only features route to `api.cityscroll.org`: `/nl` (plain English → filters via Claude Haiku, metered by `NL_METER`), `/subscribe`→`/confirm`→`/unsubscribe` (double-opt-in, rate-limited, fails closed without token/send secrets), feeds, `/batch`, `/agencies`, `/inv`, `/stats`, `/feedback` (still Turnstile-gated), keyed `/admin/*` and `/usage`.
4. The forecasting layer (`/checkbook` + `/forecast`) parses historical Checkbook NYC contract terms into estimated expirations (`fc:<stem>` in `ALERT_STATE`) and renders them in the profile timeline. Official procurement-plan rows are disabled; the cleanup job removes stale `plan:` keys.
5. Subscriptions land in KV `SUBS`; legacy aggregate integers accrue in stats counters, while bounded page and interaction events accrue in Analytics Engine without visitor identifiers. The only personal data is the double-opted-in subscription email.
6. The daily cron (13:00 UTC) first refreshes the D1 notices mirror from Socrata (cursored, fail-soft — a failed ingest never blocks alerts), pre-warms prior-cycle match sets for freshly-ingested Award notices, rebuilds the hearings, Property, and versioned whole-profile vendor projections in KV, then replays active subscriptions and forecast milestones, sending digests and early-warning emails via Resend — hard-capped at 25/run, 50/day. Each cache job is fail-soft; Money digests exclude data-entry-error amounts (≥ $10B) and label rolling year-2090 deadlines honestly.
Expand All @@ -246,4 +246,4 @@ Bottom-up, the way it's built: public Socrata feeds and Checkbook are the ground
**A:** The core search, CORS-open Socrata data (including ABO authority awards), maps, and local workspace still work. Worker-backed extras go dark — email alerts, feeds, `/nl` search, forecasting, stats, Checkbook payment lookups, and NYCHA contract matches.

**Q:** What stops a hostile script from running up the bill on the paid routes?
**A:** Layered ceilings that fail closed: `/nl` is metered per-day in KV `NL_METER`; email sends are hard-capped by `MAX_PER_RUN=25` / `MAX_SENDS_PER_DAY=50` (under Resend's free tier); `/subscribe` and `/feedback` are Turnstile-gated with per-IP/per-address rate-limit counters and return 503 if their secrets are missing.
**A:** Layered ceilings that fail closed: `/nl` is metered per-day in KV `NL_METER`; email sends are hard-capped by `MAX_PER_RUN=25` / `MAX_SENDS_PER_DAY=50` (under Resend's free tier); `/subscribe` uses per-IP/per-address rate limits and double opt-in (no CAPTCHA; can return behind a config flag if abuse appears); `/feedback` stays Turnstile-gated; both return 503 if their required secrets are missing.
Binary file modified docs/readme/data-page.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/readme/homepage.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/readme/money-search.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/readme/procurement-lifecycle.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/readme/vendor-profile.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion site/clarity.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"use strict";

// Operator: paste the Clarity project id between the quotes to enable. Leave empty for dormant.
const CONFIGURED_PROJECT_ID = "";
const CONFIGURED_PROJECT_ID = "xusuca7gsv";

const TAG_ORIGIN = "https://www.clarity.ms/tag/";
const META_NAME = "crol-clarity-project-id";
Expand Down
11 changes: 6 additions & 5 deletions site/i18n.js
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@ const STRINGS = {
freq_weekly: "Weekly",
preview_digest_btn: "Preview today's digest",
subscribe_btn: "Subscribe →",
subscribe_confirm_note: "We email a confirmation link — alerts begin only after you click it, so no one can sign you up but you.",
subscribe_confirm_note: "We'll email a link to confirm.",
// Homepage primary conversion (under masthead tagline) — short words for the reading-level ratchet
home_cta_prompt: "Want email updates?",
home_cta_submit: "Sign up",
Expand All @@ -323,17 +323,18 @@ const STRINGS = {
loading_data: "Loading…",
retry_open_data: "Could not reach NYC Open Data. Retry in a moment.",
nothing_found: "Nothing found. Try a broader keyword or \"All RFPs\".",
check_inbox: "Check your inbox.",
sent_confirm_to: "We sent a confirmation link to {email} — your alert starts once you click it.",
check_inbox: "Check your inbox to confirm.",
sent_confirm_to: "Sent to {email}.",
turnstile_fail: "The human check didn't pass — try it again.",
rate_limited: "Too many attempts — give it a minute.",
bad_email: "That email address looks off.",
channel_unsupported: "Text alerts aren't available yet — choose Email.",
not_configured: "Subscriptions aren't switched on yet.",
send_failed: "Couldn't send the email just now — try again.",
generic_error: "Something went wrong — please try again.",
complete_human_check: "Complete the “I’m human” check above first.",
sending_confirm_link: "Sending your confirmation link…",
// Turnstile on signup is interaction-only (invisible for most visitors); this is the race/load case.
complete_human_check: "Still verifying — try again in a moment.",
sending_confirm_link: "Sending…",
cant_reach_server: "Couldn't reach the server — try again.",

// Deadline chips (N-01: numbers under ten spelled out; {n} receives already-spelled value)
Expand Down
10 changes: 5 additions & 5 deletions site/i18n/lang/ar.js
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@
freq_weekly: "أسبوعيًا",
preview_digest_btn: "معاينة ملخص اليوم",
subscribe_btn: "اشترك ←",
subscribe_confirm_note: "نرسل رابط تأكيد عبر البريد الإلكتروني — لا تبدأ التنبيهات إلا بعد أن تنقر عليه، فلا يمكن لأحد سواك تسجيل اشتراكك.",
subscribe_confirm_note: "سنرسل رابطًا للتأكيد.",
home_cta_prompt: "هل تريد تحديثات بالبريد؟",
home_cta_submit: "اشترك",
home_cta_topics: "أو اختر مواضيع",
Expand All @@ -218,17 +218,17 @@
loading_data: "جارٍ التحميل…",
retry_open_data: "تعذّر الاتصال بـ NYC Open Data. أعد المحاولة بعد لحظة.",
nothing_found: "لم يُعثر على شيء. جرّب كلمة مفتاحية أوسع أو \"كل طلبات تقديم العروض\".",
check_inbox: "تحقق من بريدك الوارد.",
sent_confirm_to: "أرسلنا رابط تأكيد إلى {email} — يبدأ تنبيهك بمجرد أن تنقر عليه.",
check_inbox: "تحقق من بريدك الوارد للتأكيد.",
sent_confirm_to: "أُرسل إلى {email}.",
turnstile_fail: "لم يجتز التحقق من أنك إنسان — حاول مرة أخرى.",
rate_limited: "محاولات كثيرة جدًا — انتظر قليلًا.",
bad_email: "عنوان البريد الإلكتروني هذا يبدو غير صحيح.",
channel_unsupported: "تنبيهات الرسائل النصية غير متاحة بعد — اختر البريد الإلكتروني.",
not_configured: "الاشتراكات غير مفعّلة بعد.",
send_failed: "تعذّر إرسال البريد الإلكتروني الآن — حاول مرة أخرى.",
generic_error: "حدث خطأ ما — يُرجى المحاولة مرة أخرى.",
complete_human_check: "أكمل التحقق من \"أنا إنسان\" أعلاه أولًا.",
sending_confirm_link: "جارٍ إرسال رابط التأكيد…",
complete_human_check: "ما زلنا نتحقق — حاول مرة أخرى بعد لحظة.",
sending_confirm_link: "جارٍ الإرسال…",
cant_reach_server: "تعذّر الوصول إلى الخادم — حاول مرة أخرى.",
closes_today: "يُغلق اليوم",
closes_in_1_day: "يُغلق خلال يوم واحد",
Expand Down
10 changes: 5 additions & 5 deletions site/i18n/lang/bn.js
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@
freq_weekly: "সাপ্তাহিক",
preview_digest_btn: "আজকের সারসংক্ষেপ দেখুন",
subscribe_btn: "সাবস্ক্রাইব করুন →",
subscribe_confirm_note: "আমরা একটি নিশ্চিতকরণ লিংক ইমেইল করি — আপনি সেটিতে ক্লিক করার পরই সতর্কতা শুরু হয়, তাই আপনি ছাড়া আর কেউ আপনাকে সাইন আপ করাতে পারবে না।",
subscribe_confirm_note: "নিশ্চিত করতে আমরা একটি লিংক ইমেইল করব।",
home_cta_prompt: "ইমেইলে আপডেট চান?",
home_cta_submit: "নিবন্ধন",
home_cta_topics: "অথবা বিষয় বেছে নিন",
Expand All @@ -216,17 +216,17 @@
loading_data: "লোড হচ্ছে…",
retry_open_data: "NYC Open Data-তে পৌঁছানো যায়নি। একটু পরে আবার চেষ্টা করুন।",
nothing_found: "কিছু পাওয়া যায়নি। একটি বিস্তৃত মূল শব্দ বা \"সব RFP\" চেষ্টা করুন।",
check_inbox: "আপনার ইনবক্স দেখুন।",
sent_confirm_to: "আমরা {email}-এ একটি নিশ্চিতকরণ লিংক পাঠিয়েছি — আপনি সেটিতে ক্লিক করলেই আপনার সতর্কতা শুরু হবে।",
check_inbox: "নিশ্চিত করতে আপনার ইনবক্স দেখুন।",
sent_confirm_to: "{email}-এ পাঠানো হয়েছে।",
turnstile_fail: "মানব যাচাই সফল হয়নি — আবার চেষ্টা করুন।",
rate_limited: "অনেকবার চেষ্টা করা হয়েছে — একটু অপেক্ষা করুন।",
bad_email: "সেই ইমেইল ঠিকানাটি ঠিক মনে হচ্ছে না।",
channel_unsupported: "SMS সতর্কতা এখনও উপলব্ধ নয় — ইমেইল বেছে নিন।",
not_configured: "সাবস্ক্রিপশন এখনও চালু করা হয়নি।",
send_failed: "এই মুহূর্তে ইমেইল পাঠানো যায়নি — আবার চেষ্টা করুন।",
generic_error: "কিছু একটা ভুল হয়েছে — অনুগ্রহ করে আবার চেষ্টা করুন।",
complete_human_check: "প্রথমে উপরের \"আমি মানুষ\" যাচাই সম্পূর্ণ করুন।",
sending_confirm_link: "আপনার নিশ্চিতকরণ লিংক পাঠানো হচ্ছে…",
complete_human_check: "এখনও যাচাই হচ্ছে — একটু পরে আবার চেষ্টা করুন।",
sending_confirm_link: "পাঠানো হচ্ছে…",
cant_reach_server: "সার্ভারে পৌঁছানো যায়নি — আবার চেষ্টা করুন।",
closes_today: "আজ বন্ধ হচ্ছে",
closes_in_1_day: "এক দিনে বন্ধ হচ্ছে",
Expand Down
10 changes: 5 additions & 5 deletions site/i18n/lang/es.js
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@
freq_weekly: "Semanal",
preview_digest_btn: "Ver el resumen de hoy",
subscribe_btn: "Suscribirse →",
subscribe_confirm_note: "Le enviamos un enlace de confirmación — las alertas comienzan solo después de que lo haga clic, así que nadie puede suscribirle excepto usted.",
subscribe_confirm_note: "Le enviaremos un enlace para confirmar.",
home_cta_prompt: "¿Quiere alertas por correo?",
home_cta_submit: "Registrarse",
home_cta_topics: "o elija temas",
Expand All @@ -216,17 +216,17 @@
loading_data: "Cargando…",
retry_open_data: "No se pudo conectar a NYC Open Data. Intente de nuevo en un momento.",
nothing_found: "No se encontró nada. Pruebe con una palabra clave más amplia o \"Todas las RFP\".",
check_inbox: "Revise su bandeja de entrada.",
sent_confirm_to: "Le enviamos un enlace de confirmación a {email} — su alerta comienza cuando lo haga clic.",
check_inbox: "Revise su bandeja de entrada para confirmar.",
sent_confirm_to: "Enviado a {email}.",
turnstile_fail: "La verificación de humano no pasó — inténtelo de nuevo.",
rate_limited: "Demasiados intentos — espere un momento.",
bad_email: "Esa dirección de correo no parece correcta.",
channel_unsupported: "Las alertas por SMS aún no están disponibles — elija Correo.",
not_configured: "Las suscripciones aún no están activadas.",
send_failed: "No se pudo enviar el correo ahora — inténtelo de nuevo.",
generic_error: "Algo salió mal — inténtelo de nuevo.",
complete_human_check: "Complete la verificación “Soy humano” de arriba primero.",
sending_confirm_link: "Enviando su enlace de confirmación…",
complete_human_check: "Aún verificando — intente de nuevo en un momento.",
sending_confirm_link: "Enviando…",
cant_reach_server: "No se pudo conectar al servidor — inténtelo de nuevo.",
closes_today: "cierra hoy",
closes_in_1_day: "cierra en un día",
Expand Down
Loading
Loading