From 3ad4b004a80fdd795d1c5b8552f845ec4546f458 Mon Sep 17 00:00:00 2001 From: Luis Tanafranca <80248146+ltanafranca1004@users.noreply.github.com> Date: Sun, 9 Aug 2026 17:23:06 -0700 Subject: [PATCH 1/2] fix(events-crawler): stricter relevance for NVCL and Capilano MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Savar's 2026-08-09 review flagged that NVCL's and Capilano's generic tech-help content (Drop-in technology help, Windows/Mac laptop help, MS Office: Intro to Word/Excel) is too broad — it teaches skills the audience already has, unlike immigration/settlement content. Confirmed with him directly, including dropping the MS Office / spreadsheet suite. Rather than revert last week's loosening globally (the technology-help term is genuine, wanted content at Surrey, which no one flagged), add a per-source STRICT filter: relevance.ts splits into CORE_TERMS and DIGITAL_LITERACY_TERMS, and isSettlementRelevant(title, { strict }) uses CORE only. Sources opt in via Source.strictRelevance, set on nvcl + capilano. The default union is byte-equivalent to before, so the other sources are unaffected. Verified with dryrun --no-filter --titles over every filtered source's whole window (cap raised locally): NVCL drops 6 distinct digital titles (13 rows) and keeps its 5 genuine ones (English Corner x3, newcomer circletime, Open door community hub); Capilano unchanged; the other five filtered sources have zero flips. Also folds in the latent left-word-boundary fixes from BACKLOG (a global regex hardening, opposite failure direction): a leading \b on lease, rental, tenant, resume — plus renting (swallowed by "parenting"), which the original sweep missed. Zero live incidence, so zero flips on any source; locked in relevance_test.ts as DROP fixtures paired with real-word keeps. Re-verified the alternative set: only these 5 \b changes moved, nothing dropped in the split. Co-Authored-By: Claude Opus 4.8 --- BACKLOG.md | 29 +++- .../events-crawler/adapters/bibliocommons.ts | 2 +- .../events-crawler/adapters/capilano.ts | 2 +- .../events-crawler/adapters/communico.ts | 2 +- .../events-crawler/adapters/livewhale.ts | 2 +- .../functions/events-crawler/adapters/nvcl.ts | 2 +- .../events-crawler/adapters/surrey.ts | 2 +- .../events-crawler/adapters/tribe.ts | 2 +- supabase/functions/events-crawler/dryrun.ts | 15 +- .../functions/events-crawler/lib/relevance.ts | 149 +++++++++++------- .../events-crawler/lib/relevance_test.ts | 61 ++++++- .../functions/events-crawler/lib/sources.ts | 8 + .../functions/events-crawler/lib/types.ts | 8 + 13 files changed, 212 insertions(+), 72 deletions(-) diff --git a/BACKLOG.md b/BACKLOG.md index d233016..1dab9a2 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -526,6 +526,19 @@ and the filter gained qualified office-suite terms, `tech(nology)? (cafe|help|su titles of the seven filtered sources' 4-month windows, 9 distinct titles (56 rows) flipped DROP → KEEP and nothing else moved. `lib/relevance_test.ts` now locks in the trap set. +**Follow-up 2026-08-09 — per-source strict filter for NVCL + Capilano.** Savar reviewed the widened +output and flagged that generic tech-help ("Drop-in technology help", "Windows/Mac laptop help", +"MS Office: Intro to Word/Excel") is too broad for NVCL and Capilano specifically — it teaches skills +the audience already has, unlike the immigration/settlement content. **Confirmed with him directly, +including dropping the MS Office / spreadsheet suite — not an assumption.** Rather than revert the +loosening globally (the `technology help` term is genuine, wanted content at Surrey, which no one +flagged), `relevance.ts` now has a STRICT variant — CORE_TERMS only, no digital-literacy group — gated +per-source by `Source.strictRelevance`, set on `nvcl` + `capilano`. Re-captured all seven filtered +sources with `--no-filter --titles` (cap raised locally): NVCL drops 6 distinct digital titles (13 +rows) and keeps its 5 genuine ones (English Corner ×3, newcomer circletime, Open door community hub); +Capilano unchanged (its one on-mission item, the International Student Orientation, matches a core +term); the other five sources have **zero flips**. + **Correction to the NVCL measurement this entry used to carry.** It claimed NVCL's whole 4-month window was 27 distinct titles matching **none** of the filter, for 0 rows. Re-measured 2026-08-08 with the walk untruncated: the window is **61 distinct titles**, and the filter @@ -555,7 +568,17 @@ live feeds contain "Excel in Your Studies", "Crossword Club", "Wordplay for Todd "PowerPoint Karaoke Night". Add terms narrowly, add each one's target **and** its near-misses to `lib/relevance_test.ts`, and re-capture every source with `--no-filter --titles` to see the flips. -**events-crawler — `relevance.ts` terms without a left word boundary (latent false positives)** +**events-crawler — `relevance.ts` terms without a left word boundary (latent false positives) — RESOLVED 2026-08-09** +Fixed in the NVCL/Capilano strict-relevance PR (folded in per the recommendation below, on Savar's +sign-off): `lease`→`\blease\b`, `rental`→`\brental`, `tenant`→`\btenant`, `resume`→`\bresume`, **plus a +fifth the original sweep missed — `renting`→`\brenting`** (swallowed by "pa**renting**", e.g. +"Parenting Support Circle"). Each swallow example is now a DROP fixture in `lib/relevance_test.ts`, +each paired with a real-word KEEP (`Rental Housing`, `Tenant Rights`, `Renting 101`, `Lease +Agreement`, `Résumé Clinic`) so the narrowing is proven not to cost a genuine hit. Re-captured all +seven filtered sources with `--no-filter --titles` (cap raised locally): **zero flips from the `\b` +edits on any source** — the false-keeps still have zero live incidence, so nothing currently ingested +moved. The original finding, kept for the record: + Found 2026-08-09 during a manual review of PRs #96/#97, which merged under an explicit no-CodeRabbit-review exception (the free-tier review quota was exhausted; CodeRabbit will not review a merged PR — `@coderabbitai review` answers "Already reviewed" on #96 and "Pull request is @@ -571,6 +594,7 @@ English word ends in "newcomer" — but four are substrings of common words: |---|---|---| | `lease\b` | P·**lease** | `Please Note: Library Closed Monday` | | `rental` | Pa·**rental** | `Parental Controls: Keeping Kids Safe Online` | +| `renting` | Pa·**renting** | `Parenting Support Circle` (missed by the original sweep) | | `tenant` | Lieu·**tenant** | `Lieutenant Governor Reading Award Ceremony` | | `resume` | P·**resume**·d | `Presumed Innocent: Film Screening` | @@ -584,7 +608,8 @@ Fix is mechanical (`\blease\b`, `\brental`, `\btenant`, `\bresume`) but **adding regex**, so the monotonicity argument that covers term *additions* does not apply — it can drop an existing keep. Re-run the corpus comparison the way the `\btech` fix was verified, and add each swallow example to `lib/relevance_test.ts` as a DROP fixture. Best folded into whichever PR next -touches `relevance.ts` (e.g. the near-misses above) rather than done as its own change. +touches `relevance.ts` (e.g. the near-misses above) rather than done as its own change. — Done exactly +that; see the RESOLVED note at the top of this entry. **Client-only tab state elsewhere — same back-navigation bug Community just fixed** Found while fixing Community (this PR): tab state held in `useState` never reaches the URL, diff --git a/supabase/functions/events-crawler/adapters/bibliocommons.ts b/supabase/functions/events-crawler/adapters/bibliocommons.ts index 43418d9..0beaeab 100644 --- a/supabase/functions/events-crawler/adapters/bibliocommons.ts +++ b/supabase/functions/events-crawler/adapters/bibliocommons.ts @@ -135,7 +135,7 @@ function candidatesFromPage(page: BiblioPage, source: Source, ctx: AdapterContex // past MAX_TITLE_CHARS would otherwise be invisible to the filter. const fullTitle = clean(def.title); if (!fullTitle) continue; - if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) continue; + if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) continue; const title = fullTitle.slice(0, MAX_TITLE_CHARS); const startIso = offsetIsoToUtc(ev.indexStart); diff --git a/supabase/functions/events-crawler/adapters/capilano.ts b/supabase/functions/events-crawler/adapters/capilano.ts index 9acaa7d..73cb776 100644 --- a/supabase/functions/events-crawler/adapters/capilano.ts +++ b/supabase/functions/events-crawler/adapters/capilano.ts @@ -207,7 +207,7 @@ function parseListing(html: string, source: Source, ctx: AdapterContext): PagePa } // Window before relevance, so both filters see every parsed row. - if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) continue; + if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) continue; if (startMs < ctx.nowMs || startMs > ctx.windowEndMs) continue; // End date and end time are independent: a multi-day item has date-stamp2 but often no diff --git a/supabase/functions/events-crawler/adapters/communico.ts b/supabase/functions/events-crawler/adapters/communico.ts index cea889a..dccf1e3 100644 --- a/supabase/functions/events-crawler/adapters/communico.ts +++ b/supabase/functions/events-crawler/adapters/communico.ts @@ -140,7 +140,7 @@ export async function fetchEvents(source: Source, ctx: AdapterContext): Promise< const title = fullTitle.slice(0, MAX_TITLE_CHARS); const link = clean(firstGroup(block, LINK_RE)); if (!title || !link) continue; - if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) continue; + if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) continue; // Two entity layers: the XML escaping around , then the HTML inside it. const descriptionHtml = decodeEntities(firstGroup(block, DESCRIPTION_RE)); diff --git a/supabase/functions/events-crawler/adapters/livewhale.ts b/supabase/functions/events-crawler/adapters/livewhale.ts index a31b7c4..aec4348 100644 --- a/supabase/functions/events-crawler/adapters/livewhale.ts +++ b/supabase/functions/events-crawler/adapters/livewhale.ts @@ -103,7 +103,7 @@ function toCandidate( const startIso = toIsoUtc(ev.date_utc); if (!title || !url || !startIso) return null; // NOT NULL columns - if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) return null; + if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) return null; const startMs = Date.parse(startIso); // The feed only ever looks forward, but bound both edges anyway — an all-day event diff --git a/supabase/functions/events-crawler/adapters/nvcl.ts b/supabase/functions/events-crawler/adapters/nvcl.ts index d848025..11e1e52 100644 --- a/supabase/functions/events-crawler/adapters/nvcl.ts +++ b/supabase/functions/events-crawler/adapters/nvcl.ts @@ -241,7 +241,7 @@ function parsePage(html: string, source: Source, ctx: AdapterContext): PageParse // Track the window before relevance, so the walk can stop on dates even when a page // happens to contain nothing relevant. - if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) continue; + if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) continue; if (startMs < ctx.nowMs || startMs > ctx.windowEndMs) continue; candidates.push({ diff --git a/supabase/functions/events-crawler/adapters/surrey.ts b/supabase/functions/events-crawler/adapters/surrey.ts index 9d4a14a..c9a35e2 100644 --- a/supabase/functions/events-crawler/adapters/surrey.ts +++ b/supabase/functions/events-crawler/adapters/surrey.ts @@ -155,7 +155,7 @@ function parsePage(html: string, source: Source, ctx: AdapterContext): PageParse // Track the window before relevance, so the walk can stop on dates even when a page // happens to contain nothing relevant. - if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) continue; + if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) continue; if (startMs < ctx.nowMs || startMs > ctx.windowEndMs) continue; let endIso = offsetIsoToUtc(times[1]); diff --git a/supabase/functions/events-crawler/adapters/tribe.ts b/supabase/functions/events-crawler/adapters/tribe.ts index 67ed539..9fe835b 100644 --- a/supabase/functions/events-crawler/adapters/tribe.ts +++ b/supabase/functions/events-crawler/adapters/tribe.ts @@ -78,7 +78,7 @@ async function tribeEventToRow( // Cheap rejections first, so a filtered or far-future event costs no HEAD probe or // Pexels lookup. Backstop for the server-side ?end_date — see fetchEvents. - if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) return null; + if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) return null; if (Date.parse(eventDatetime) > ctx.windowEndMs) return null; // location + event_type from the venue NAME first, then venue presence, then the diff --git a/supabase/functions/events-crawler/dryrun.ts b/supabase/functions/events-crawler/dryrun.ts index 99a0603..7b98b16 100644 --- a/supabase/functions/events-crawler/dryrun.ts +++ b/supabase/functions/events-crawler/dryrun.ts @@ -64,17 +64,22 @@ function printRow(row: EventRow, i: number): void { * * `filtered` is the source's REGISTRY setting, not the setting this run used. A settlement * agency is never relevance-filtered, so printing KEEP/DROP beside its events would invite - * exactly the wrong conclusion — those rows are ingested either way. Hence `n/a`. + * exactly the wrong conclusion — those rows are ingested either way. Hence `n/a`. `strict` + * is likewise the registry setting, so the verdict matches what production would give a + * strict source (NVCL/Capilano) even under --no-filter. */ -function printTitle(row: EventRow, filtered: boolean): void { - const verdict = !filtered ? 'n/a ' : isSettlementRelevant(row.title) ? 'KEEP' : 'DROP'; +function printTitle(row: EventRow, filtered: boolean, strict: boolean): void { + const verdict = !filtered ? 'n/a ' : isSettlementRelevant(row.title, { strict }) ? 'KEEP' : 'DROP'; console.log(`${verdict} ${row.event_datetime.slice(0, 10)} ${row.title}`); } if (Deno.args.includes('--list')) { console.log('Sources:'); for (const s of SOURCES) { - const flags = [s.enabled ? 'enabled' : 'DISABLED', s.relevanceFilter ? 'relevance-filtered' : ''] + const flags = [ + s.enabled ? 'enabled' : 'DISABLED', + s.relevanceFilter ? (s.strictRelevance ? 'relevance-filtered (strict)' : 'relevance-filtered') : '', + ] .filter(Boolean) .join(', '); console.log(` ${s.slug.padEnd(18)} ${s.kind.padEnd(15)} ${flags}`); @@ -111,7 +116,7 @@ const rows = await ADAPTERS[source.kind](source, ctx); if (Deno.args.includes('--json')) { console.log(JSON.stringify(rows, null, 2)); } else if (Deno.args.includes('--titles')) { - rows.forEach((row) => printTitle(row, Boolean(registered.relevanceFilter))); + rows.forEach((row) => printTitle(row, Boolean(registered.relevanceFilter), Boolean(registered.strictRelevance))); } else { rows.forEach(printRow); } diff --git a/supabase/functions/events-crawler/lib/relevance.ts b/supabase/functions/events-crawler/lib/relevance.ts index 54fa947..a505055 100644 --- a/supabase/functions/events-crawler/lib/relevance.ts +++ b/supabase/functions/events-crawler/lib/relevance.ts @@ -16,6 +16,18 @@ // irrelevant ones is actively harmful. Revisit by adding terms, not by widening to // descriptions. // +// TWO STRENGTHS. The default filter (CORE_TERMS + DIGITAL_LITERACY_TERMS) treats generic +// digital-literacy / tech-help content as on-mission. A STRICT filter (CORE_TERMS only) +// drops that whole group. It exists because most tech-help programming — "Drop-in +// technology help", "Windows laptop help", "MS Office: Intro to Word" — teaches skills the +// app's users already have; it is not settlement-specific the way immigration/legal/ +// language content is. Sources opt into strict via `Source.strictRelevance` (lib/types.ts). +// Set on NVCL and Capilano after Savar's 2026-08-09 review of the live output flagged their +// tech-help sessions as too broad — CONFIRMED with him directly, including dropping the +// MS Office / spreadsheet suite, not an assumption. Kept per-source deliberately: the same +// `technology help` term is genuine, wanted content at Surrey, so the fix could not be a +// global revert without regressing a source no one flagged. +// // Measured keep-rates on the live feeds when this landed (2026-07-31): // West Van 8/50 · VPL 6/100 · SFU 5/200 · NVDPL 16/100 · Surrey 2/10 (page 1) // @@ -24,8 +36,9 @@ // --titles`, with MAX_PER_ORG raised locally so the walk isn't truncated): // West Van 6/50 → 6 · VPL 22/200 → 22 · SFU 3/111 → 3 · NVDPL 11/82 → 20 // Surrey 31/192 → 46 · NVCL 10/200 → 42 · Capilano 1/10 → 1 -// Across all 395 distinct titles, 9 flipped DROP → KEEP and nothing else moved — a term -// added here is a union alternative, so it can only add keeps, never drop an existing one. +// Adding a default term is a union alternative, so it can only add keeps, never drop an +// existing one. Adding a `\b` (below) is the opposite — it NARROWS — so those edits are +// re-verified against the captured corpus, not assumed safe. /** * Accents are stripped before matching (NFD, then drop combining marks), so a single @@ -40,64 +53,86 @@ function foldAccents(value: string): string { } /** - * Grouped for reviewability — the groups are purely documentation, the union is what - * runs. Terms are matched case-insensitively against the accent-folded title. + * The settlement-specific terms, always in force. Grouped for reviewability — the groups + * are purely documentation, the union is what runs. Terms are matched case-insensitively + * against the accent-folded title. + * + * A leading `\b` on `tenant`, `rental`, `renting`, `lease` and `resume`: each is a + * substring of a common unrelated word — Lieu·tenant, Pa·rental, Pa·renting, P·lease, + * P·resume·d — so without it the filter wrongly KEEPS "Lieutenant Governor Reading Award", + * "Parental Controls", "Parenting Support Circle", "Please Note: Library Closed" and + * "Presumed Innocent: Film Screening". Zero live incidence in the captured corpus today, + * but latent; see lib/relevance_test.ts for the swallow examples and the real-word keeps + * that prove the boundary doesn't cost a genuine hit. + */ +const CORE_TERMS = [ + // Status & settlement + 'newcomer|immigrant|immigration|refugee|asylum|settlement|migrant|international\\s+student', + // Legal & documents + 'citizenship|permanent\\s+resident|pr\\s+card|work\\s+permit|study\\s+permit|visa\\b', + 'social\\s+insurance|sin\\s+clinic|legal\\s+clinic|notary', + // Language + '\\besl\\b|\\blinc\\b|english\\s+(conversation|class|corner|practice|language)', + 'conversation\\s+(circle|club|cafe)|language\\s+(exchange|cafe)|speaking\\s+english|literacy', + // Employment + 'job\\s+(search|fair|club|help)|\\bresume|cover\\s+letter|interview\\s+skills|career', + 'employment|workbc|hiring|credential|foreign[-\\s]trained|workplace', + // Finance + 'tax\\s+clinic|income\\s+tax|banking|bank\\s+account|budgeting|financial\\s+literacy', + 'credit\\s+score', + // Housing + 'housing|\\btenant|tenancy|\\brental|\\brenting|landlord|\\blease\\b', + // Named settlement programs. Kept to programs run under the same name by more than + // one source, so this doesn't become a list of one-off event names: the Open Door + // community hub is NVCL's and NVDPL's shared newcomer drop-in, 29 occurrences across + // the two 4-month windows, and was the single largest miss in the corpus. Qualified + // so it cannot catch "Open Door Poetry Reading" or "Open Doors Heritage Tour". + 'open\\s+door\\s+(community|drop)', + // Health-system navigation + 'health\\s+card|family\\s+doctor|\\bmsp\\b|medical\\s+insurance', +]; + +/** + * Digital-literacy / tech-help terms. In force under the default filter, dropped under the + * strict filter (see the header). Split out as its own group precisely so a source can opt + * out of it without losing the settlement terms above. */ -const RELEVANCE_RE = new RegExp( - [ - // Status & settlement - 'newcomer|immigrant|immigration|refugee|asylum|settlement|migrant|international\\s+student', - // Legal & documents - 'citizenship|permanent\\s+resident|pr\\s+card|work\\s+permit|study\\s+permit|visa\\b', - 'social\\s+insurance|sin\\s+clinic|legal\\s+clinic|notary', - // Language - '\\besl\\b|\\blinc\\b|english\\s+(conversation|class|corner|practice|language)', - 'conversation\\s+(circle|club|cafe)|language\\s+(exchange|cafe)|speaking\\s+english|literacy', - // Employment - 'job\\s+(search|fair|club|help)|resume|cover\\s+letter|interview\\s+skills|career', - 'employment|workbc|hiring|credential|foreign[-\\s]trained|workplace', - // Finance - 'tax\\s+clinic|income\\s+tax|banking|bank\\s+account|budgeting|financial\\s+literacy', - 'credit\\s+score', - // Housing - 'housing|tenant|tenancy|rental|renting|landlord|lease\\b', - // Digital literacy - // `\b` before tech: without it the alternative matches the tail of "Biotech Help" or - // "Fintech Support", both plausible at SFU. It was already missing on the narrower - // `tech` this widened. - 'digital\\s+literacy|computer\\s+(basics|skills|help)|\\btech(nology)?\\s+(cafe|help|support)', - 'device\\s+clinic|laptop\\s+help|online\\s+safety|internet\\s+basics', - // Digital literacy — office suite. Libraries title these by product, not by the - // category words above, so the group missed them entirely: NVCL's "MS Office learn - // and practice: Intro to Word" and Surrey's office sessions are settlement-adjacent - // employment content by any reading. Every office word is QUALIFIED — behind `ms`, - // `microsoft`, or `intro to` — because the bare forms are traps: `\bexcel\b` catches - // "Excel in Your Studies", `\bword\b` catches "Crossword Club" and "Wordplay for - // Toddlers", and `\bpowerpoint\b` catches "PowerPoint Karaoke Night". All four are - // real titles from the live feeds. See lib/relevance_test.ts. - '\\bms\\s+(office|word|excel|powerpoint|outlook)\\b', - 'microsoft\\s+(office|word|excel|powerpoint|outlook|365)\\b', - 'intro(duction)?\\s+to\\s+(excel|word|powerpoint|outlook)\\b|\\bspreadsheet', - // Named settlement programs. Kept to programs run under the same name by more than - // one source, so this doesn't become a list of one-off event names: the Open Door - // community hub is NVCL's and NVDPL's shared newcomer drop-in, 29 occurrences across - // the two 4-month windows, and was the single largest miss in the corpus. Qualified - // so it cannot catch "Open Door Poetry Reading" or "Open Doors Heritage Tour". - 'open\\s+door\\s+(community|drop)', - // Health-system navigation - 'health\\s+card|family\\s+doctor|\\bmsp\\b|medical\\s+insurance', - ].join('|'), - 'i', -); +const DIGITAL_LITERACY_TERMS = [ + // Digital literacy + // `\b` before tech: without it the alternative matches the tail of "Biotech Help" or + // "Fintech Support", both plausible at SFU. It was already missing on the narrower + // `tech` this widened. + 'digital\\s+literacy|computer\\s+(basics|skills|help)|\\btech(nology)?\\s+(cafe|help|support)', + 'device\\s+clinic|laptop\\s+help|online\\s+safety|internet\\s+basics', + // Office suite. Libraries title these by product, not by the category words above, so the + // group missed them entirely: NVCL's "MS Office learn and practice: Intro to Word" and + // Surrey's office sessions. Every office word is QUALIFIED — behind `ms`, `microsoft`, or + // `intro to` — because the bare forms are traps: `\bexcel\b` catches "Excel in Your + // Studies", `\bword\b` catches "Crossword Club" and "Wordplay for Toddlers", and + // `\bpowerpoint\b` catches "PowerPoint Karaoke Night". All four are real titles from the + // live feeds. See lib/relevance_test.ts. + '\\bms\\s+(office|word|excel|powerpoint|outlook)\\b', + 'microsoft\\s+(office|word|excel|powerpoint|outlook|365)\\b', + 'intro(duction)?\\s+to\\s+(excel|word|powerpoint|outlook)\\b|\\bspreadsheet', +]; + +// Default: settlement terms + digital literacy. Strict: settlement terms only. Alternation +// order does not affect a .test(), so the default union is byte-equivalent to the single +// array this replaced, apart from the intended `\b` additions above. +const RELEVANCE_RE = new RegExp([...CORE_TERMS, ...DIGITAL_LITERACY_TERMS].join('|'), 'i'); +const STRICT_RELEVANCE_RE = new RegExp(CORE_TERMS.join('|'), 'i'); /** - * Whether a library/university event's title indicates settlement-relevant content. + * Whether a library/university event's title indicates settlement-relevant content. Pass + * `{ strict: true }` (wired from `Source.strictRelevance`) to also drop generic + * digital-literacy / tech-help content — see the header for why that is per-source. * - * Note what is deliberately absent: a bare `orientation` would catch West Van's - * "Recording Studio Orientation", so SFU's flagship "International Student Orientation" - * is caught by `international student` instead. Likewise bare `health`, `family` and - * `community` are excluded as far too broad for a library calendar. + * Note what is deliberately absent from CORE_TERMS: a bare `orientation` would catch West + * Van's "Recording Studio Orientation", so SFU's flagship "International Student + * Orientation" is caught by `international student` instead. Likewise bare `health`, + * `family` and `community` are excluded as far too broad for a library calendar. */ -export function isSettlementRelevant(title: string): boolean { - return RELEVANCE_RE.test(foldAccents(title)); +export function isSettlementRelevant(title: string, opts?: { strict?: boolean }): boolean { + const re = opts?.strict ? STRICT_RELEVANCE_RE : RELEVANCE_RE; + return re.test(foldAccents(title)); } diff --git a/supabase/functions/events-crawler/lib/relevance_test.ts b/supabase/functions/events-crawler/lib/relevance_test.ts index e72aa09..0f1dfbf 100644 --- a/supabase/functions/events-crawler/lib/relevance_test.ts +++ b/supabase/functions/events-crawler/lib/relevance_test.ts @@ -37,8 +37,20 @@ const KEEP: Record = { employment: [ 'Canadian Job Search Workshop', 'Ask a Career Counsellor: Resume Workshop', - // Accent folding is load-bearing here — a plain `resume` term misses "Résumé". + // Accent folding is load-bearing here — a plain `resume` term misses "Résumé". The + // leading `\b` on `resume` must not cost these: "Resume"/"Résumé" start a word. 'Résumé Clinic with S.U.C.C.E.S.S.', + 'Resume Writing Workshop', + ], + // Housing terms carry a leading `\b` (tenant, rental, renting, lease) so they don't fire + // on Lieu·tenant / Pa·rental / Pa·renting / P·lease. These prove the boundary still keeps + // the genuine article, where the word starts the match — the DROP block below proves it + // rejects the swallowing words. + housing: [ + 'Rental Housing Info Session', + 'Tenant Rights Workshop', + 'Renting 101: Know Your Rights', + 'Lease Agreement Basics for Newcomers', ], 'digital literacy': [ 'Tech Help', @@ -106,6 +118,17 @@ const DROP: Record = { // Why `orientation` is deliberately absent — see isSettlementRelevant's doc comment. 'Recording Studio Orientation', ], + // Left-word-boundary swallows: each housing/employment term is a substring of a common + // unrelated word, so before the leading `\b` these titles were wrongly KEPT. Constructed, + // not captured (zero live incidence in the corpus today) — but latent, and the `\b` fix + // that added this group NARROWS the regex, so these lock it in. See BACKLOG.md. + 'left-word-boundary swallows (lease/rental/renting/tenant/resume)': [ + 'Please Note: Library Closed Monday', // P·lease → lease + 'Parental Controls: Keeping Kids Safe Online', // Pa·rental → rental + 'Parenting Support Circle', // Pa·renting → renting + 'Lieutenant Governor Reading Award Ceremony', // Lieu·tenant → tenant + 'Presumed Innocent: Film Screening', // P·resume·d → resume + ], 'ordinary library programming': [ 'Babytime', 'Family Storytime', @@ -136,3 +159,39 @@ Deno.test('isSettlementRelevant rejects near-misses and general programming', () } } }); + +// STRICT MODE (Source.strictRelevance, set on NVCL + Capilano — see relevance.ts). Strict +// drops the whole digital-literacy / tech-help group while keeping every settlement term. +// All STRICT_FLIP titles are REAL, captured from NVCL's live window on 2026-08-09; they are +// exactly the rows Savar's review flagged as too broad. + +/** Kept by the DEFAULT filter, dropped by STRICT — the digital-literacy group. */ +const STRICT_FLIP: string[] = [ + 'Drop-in technology help', + 'Drop-in technology help in Farsi کمک به رفع اشکال تکنولوژی به زبان فارسی', + 'Mac laptop help', + 'Windows laptop help', + 'MS Office learn and practice: Intro to Word', +]; + +/** Kept by BOTH filters — settlement content strict must never drop. */ +const STRICT_KEEP: string[] = [ + 'Open door community hub', // named program + 'Fall 2026 New International Student Orientation', // international student + 'English Corner (virtual) — September 2026', // language + 'Circletime success: Welcoming newcomer children at circletime', // newcomer + 'Résumé Clinic with S.U.C.C.E.S.S.', // employment +]; + +Deno.test('strict mode drops the digital-literacy group the default filter keeps', () => { + for (const title of STRICT_FLIP) { + assert(isSettlementRelevant(title), `default filter should KEEP: ${title}`); + assert(!isSettlementRelevant(title, { strict: true }), `strict filter should DROP: ${title}`); + } +}); + +Deno.test('strict mode keeps genuine settlement content', () => { + for (const title of STRICT_KEEP) { + assert(isSettlementRelevant(title, { strict: true }), `strict filter should KEEP: ${title}`); + } +}); diff --git a/supabase/functions/events-crawler/lib/sources.ts b/supabase/functions/events-crawler/lib/sources.ts index b0348ed..ce8ba99 100644 --- a/supabase/functions/events-crawler/lib/sources.ts +++ b/supabase/functions/events-crawler/lib/sources.ts @@ -129,6 +129,10 @@ export const SOURCES: Source[] = [ host: 'www.nvcl.ca', enabled: true, relevanceFilter: true, + // Strict: its digital programming is generic device help ("Drop-in technology help", + // "Windows/Mac laptop help", "MS Office: Intro to Word"), which Savar's 2026-08-09 review + // flagged as too broad for the audience. See lib/relevance.ts. + strictRelevance: true, // The listing's venue slot is present but empty on every row — see adapters/nvcl.ts. defaultLocation: 'North Vancouver City Library', }, @@ -144,6 +148,10 @@ export const SOURCES: Source[] = [ host: 'www.capilanou.ca', enabled: true, relevanceFilter: true, + // Strict, same as NVCL (Savar's 2026-08-09 review). No effect on the current 10-item + // window — the one on-mission item, the International Student Orientation, matches a + // core settlement term — but guards against future generic tech-help workshops. + strictRelevance: true, }, ]; diff --git a/supabase/functions/events-crawler/lib/types.ts b/supabase/functions/events-crawler/lib/types.ts index 5bed1b1..ba496c8 100644 --- a/supabase/functions/events-crawler/lib/types.ts +++ b/supabase/functions/events-crawler/lib/types.ts @@ -43,6 +43,14 @@ export interface Source { * published is already on-mission and filtering would only lose events. */ relevanceFilter?: boolean; + /** + * Use the STRICT relevance filter (settlement terms only, no generic digital-literacy / + * tech-help) instead of the default. Only meaningful alongside `relevanceFilter`. Set on + * sources whose tech programming is generic device help rather than newcomer-specific — + * NVCL and Capilano, per Savar's 2026-08-09 review. See lib/relevance.ts for the rationale + * and why this is scoped per-source rather than a global change. + */ + strictRelevance?: boolean; /** * Location to use when the source's feed carries no location data at all. Only for * feeds where the field genuinely does not exist — never to paper over a parse miss, From 9c5697d9e7e24cbc4b31c66a45a890e27f605e8d Mon Sep 17 00:00:00 2001 From: Luis Tanafranca <80248146+ltanafranca1004@users.noreply.github.com> Date: Sun, 9 Aug 2026 17:38:22 -0700 Subject: [PATCH 2/2] fix(events-crawler): guard bare `literacy` so strict drops "Digital Literacy" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit caught a completeness gap in strict mode: bare `literacy` in CORE_TERMS (kept for Family/Adult reading literacy) also matches "Digital Literacy", so STRICT_RELEVANCE_RE would re-admit exactly the digital-literacy content strict is meant to drop. Latent — no such title in the current NVCL/Capilano windows, so the corpus diff didn't surface it — but real. Add a `(? --- supabase/functions/events-crawler/lib/relevance.ts | 13 ++++++++++--- .../functions/events-crawler/lib/relevance_test.ts | 7 +++++++ 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/supabase/functions/events-crawler/lib/relevance.ts b/supabase/functions/events-crawler/lib/relevance.ts index a505055..a13ccda 100644 --- a/supabase/functions/events-crawler/lib/relevance.ts +++ b/supabase/functions/events-crawler/lib/relevance.ts @@ -73,7 +73,12 @@ const CORE_TERMS = [ 'social\\s+insurance|sin\\s+clinic|legal\\s+clinic|notary', // Language '\\besl\\b|\\blinc\\b|english\\s+(conversation|class|corner|practice|language)', - 'conversation\\s+(circle|club|cafe)|language\\s+(exchange|cafe)|speaking\\s+english|literacy', + // `literacy` here means reading/language literacy (Family/Adult Literacy). The + // `(? {