Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -526,6 +526,19 @@ and the filter gained qualified office-suite terms, `tech(nology)? (cafe|help|su
titles of the seven filtered sources' 4-month windows, 9 distinct titles (56 rows) flipped
DROP → KEEP and nothing else moved. `lib/relevance_test.ts` now locks in the trap set.

**Follow-up 2026-08-09 — per-source strict filter for NVCL + Capilano.** Savar reviewed the widened
output and flagged that generic tech-help ("Drop-in technology help", "Windows/Mac laptop help",
"MS Office: Intro to Word/Excel") is too broad for NVCL and Capilano specifically — it teaches skills
the audience already has, unlike the immigration/settlement content. **Confirmed with him directly,
including dropping the MS Office / spreadsheet suite — not an assumption.** Rather than revert the
loosening globally (the `technology help` term is genuine, wanted content at Surrey, which no one
flagged), `relevance.ts` now has a STRICT variant — CORE_TERMS only, no digital-literacy group — gated
per-source by `Source.strictRelevance`, set on `nvcl` + `capilano`. Re-captured all seven filtered
sources with `--no-filter --titles` (cap raised locally): NVCL drops 6 distinct digital titles (13
rows) and keeps its 5 genuine ones (English Corner ×3, newcomer circletime, Open door community hub);
Capilano unchanged (its one on-mission item, the International Student Orientation, matches a core
term); the other five sources have **zero flips**.

**Correction to the NVCL measurement this entry used to carry.** It claimed NVCL's whole
4-month window was 27 distinct titles matching **none** of the filter, for 0 rows. Re-measured
2026-08-08 with the walk untruncated: the window is **61 distinct titles**, and the filter
Expand Down Expand Up @@ -555,7 +568,17 @@ live feeds contain "Excel in Your Studies", "Crossword Club", "Wordplay for Todd
"PowerPoint Karaoke Night". Add terms narrowly, add each one's target **and** its near-misses to
`lib/relevance_test.ts`, and re-capture every source with `--no-filter --titles` to see the flips.

**events-crawler — `relevance.ts` terms without a left word boundary (latent false positives)**
**events-crawler — `relevance.ts` terms without a left word boundary (latent false positives) — RESOLVED 2026-08-09**
Fixed in the NVCL/Capilano strict-relevance PR (folded in per the recommendation below, on Savar's
sign-off): `lease`→`\blease\b`, `rental`→`\brental`, `tenant`→`\btenant`, `resume`→`\bresume`, **plus a
fifth the original sweep missed — `renting`→`\brenting`** (swallowed by "pa**renting**", e.g.
"Parenting Support Circle"). Each swallow example is now a DROP fixture in `lib/relevance_test.ts`,
each paired with a real-word KEEP (`Rental Housing`, `Tenant Rights`, `Renting 101`, `Lease
Agreement`, `Résumé Clinic`) so the narrowing is proven not to cost a genuine hit. Re-captured all
seven filtered sources with `--no-filter --titles` (cap raised locally): **zero flips from the `\b`
edits on any source** — the false-keeps still have zero live incidence, so nothing currently ingested
moved. The original finding, kept for the record:

Found 2026-08-09 during a manual review of PRs #96/#97, which merged under an explicit
no-CodeRabbit-review exception (the free-tier review quota was exhausted; CodeRabbit will not
review a merged PR — `@coderabbitai review` answers "Already reviewed" on #96 and "Pull request is
Expand All @@ -571,6 +594,7 @@ English word ends in "newcomer" — but four are substrings of common words:
|---|---|---|
| `lease\b` | P·**lease** | `Please Note: Library Closed Monday` |
| `rental` | Pa·**rental** | `Parental Controls: Keeping Kids Safe Online` |
| `renting` | Pa·**renting** | `Parenting Support Circle` (missed by the original sweep) |
| `tenant` | Lieu·**tenant** | `Lieutenant Governor Reading Award Ceremony` |
| `resume` | P·**resume**·d | `Presumed Innocent: Film Screening` |

Expand All @@ -584,7 +608,8 @@ Fix is mechanical (`\blease\b`, `\brental`, `\btenant`, `\bresume`) but **adding
regex**, so the monotonicity argument that covers term *additions* does not apply — it can drop an
existing keep. Re-run the corpus comparison the way the `\btech` fix was verified, and add each
swallow example to `lib/relevance_test.ts` as a DROP fixture. Best folded into whichever PR next
touches `relevance.ts` (e.g. the near-misses above) rather than done as its own change.
touches `relevance.ts` (e.g. the near-misses above) rather than done as its own change. — Done exactly
that; see the RESOLVED note at the top of this entry.

**Client-only tab state elsewhere — same back-navigation bug Community just fixed**
Found while fixing Community (this PR): tab state held in `useState` never reaches the URL,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ function candidatesFromPage(page: BiblioPage, source: Source, ctx: AdapterContex
// past MAX_TITLE_CHARS would otherwise be invisible to the filter.
const fullTitle = clean(def.title);
if (!fullTitle) continue;
if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) continue;
if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) continue;
const title = fullTitle.slice(0, MAX_TITLE_CHARS);

const startIso = offsetIsoToUtc(ev.indexStart);
Expand Down
2 changes: 1 addition & 1 deletion supabase/functions/events-crawler/adapters/capilano.ts
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ function parseListing(html: string, source: Source, ctx: AdapterContext): PagePa
}

// Window before relevance, so both filters see every parsed row.
if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) continue;
if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) continue;
if (startMs < ctx.nowMs || startMs > ctx.windowEndMs) continue;

// End date and end time are independent: a multi-day item has date-stamp2 but often no
Expand Down
2 changes: 1 addition & 1 deletion supabase/functions/events-crawler/adapters/communico.ts
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ export async function fetchEvents(source: Source, ctx: AdapterContext): Promise<
const title = fullTitle.slice(0, MAX_TITLE_CHARS);
const link = clean(firstGroup(block, LINK_RE));
if (!title || !link) continue;
if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) continue;
if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) continue;

// Two entity layers: the XML escaping around <description>, then the HTML inside it.
const descriptionHtml = decodeEntities(firstGroup(block, DESCRIPTION_RE));
Expand Down
2 changes: 1 addition & 1 deletion supabase/functions/events-crawler/adapters/livewhale.ts
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ function toCandidate(
const startIso = toIsoUtc(ev.date_utc);
if (!title || !url || !startIso) return null; // NOT NULL columns

if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) return null;
if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) return null;

const startMs = Date.parse(startIso);
// The feed only ever looks forward, but bound both edges anyway — an all-day event
Expand Down
2 changes: 1 addition & 1 deletion supabase/functions/events-crawler/adapters/nvcl.ts
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ function parsePage(html: string, source: Source, ctx: AdapterContext): PageParse

// Track the window before relevance, so the walk can stop on dates even when a page
// happens to contain nothing relevant.
if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) continue;
if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) continue;
if (startMs < ctx.nowMs || startMs > ctx.windowEndMs) continue;

candidates.push({
Expand Down
2 changes: 1 addition & 1 deletion supabase/functions/events-crawler/adapters/surrey.ts
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ function parsePage(html: string, source: Source, ctx: AdapterContext): PageParse

// Track the window before relevance, so the walk can stop on dates even when a page
// happens to contain nothing relevant.
if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) continue;
if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) continue;
if (startMs < ctx.nowMs || startMs > ctx.windowEndMs) continue;

let endIso = offsetIsoToUtc(times[1]);
Expand Down
2 changes: 1 addition & 1 deletion supabase/functions/events-crawler/adapters/tribe.ts
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ async function tribeEventToRow(

// Cheap rejections first, so a filtered or far-future event costs no HEAD probe or
// Pexels lookup. Backstop for the server-side ?end_date — see fetchEvents.
if (source.relevanceFilter && !isSettlementRelevant(fullTitle)) return null;
if (source.relevanceFilter && !isSettlementRelevant(fullTitle, { strict: source.strictRelevance })) return null;
if (Date.parse(eventDatetime) > ctx.windowEndMs) return null;

// location + event_type from the venue NAME first, then venue presence, then the
Expand Down
15 changes: 10 additions & 5 deletions supabase/functions/events-crawler/dryrun.ts
Original file line number Diff line number Diff line change
Expand Up @@ -64,17 +64,22 @@ function printRow(row: EventRow, i: number): void {
*
* `filtered` is the source's REGISTRY setting, not the setting this run used. A settlement
* agency is never relevance-filtered, so printing KEEP/DROP beside its events would invite
* exactly the wrong conclusion — those rows are ingested either way. Hence `n/a`.
* exactly the wrong conclusion — those rows are ingested either way. Hence `n/a`. `strict`
* is likewise the registry setting, so the verdict matches what production would give a
* strict source (NVCL/Capilano) even under --no-filter.
*/
function printTitle(row: EventRow, filtered: boolean): void {
const verdict = !filtered ? 'n/a ' : isSettlementRelevant(row.title) ? 'KEEP' : 'DROP';
function printTitle(row: EventRow, filtered: boolean, strict: boolean): void {
const verdict = !filtered ? 'n/a ' : isSettlementRelevant(row.title, { strict }) ? 'KEEP' : 'DROP';
console.log(`${verdict} ${row.event_datetime.slice(0, 10)} ${row.title}`);
}

if (Deno.args.includes('--list')) {
console.log('Sources:');
for (const s of SOURCES) {
const flags = [s.enabled ? 'enabled' : 'DISABLED', s.relevanceFilter ? 'relevance-filtered' : '']
const flags = [
s.enabled ? 'enabled' : 'DISABLED',
s.relevanceFilter ? (s.strictRelevance ? 'relevance-filtered (strict)' : 'relevance-filtered') : '',
]
.filter(Boolean)
.join(', ');
console.log(` ${s.slug.padEnd(18)} ${s.kind.padEnd(15)} ${flags}`);
Expand Down Expand Up @@ -111,7 +116,7 @@ const rows = await ADAPTERS[source.kind](source, ctx);
if (Deno.args.includes('--json')) {
console.log(JSON.stringify(rows, null, 2));
} else if (Deno.args.includes('--titles')) {
rows.forEach((row) => printTitle(row, Boolean(registered.relevanceFilter)));
rows.forEach((row) => printTitle(row, Boolean(registered.relevanceFilter), Boolean(registered.strictRelevance)));
} else {
rows.forEach(printRow);
}
Expand Down
Loading