Skip to content

Add per-destination event filtering - #56

Merged
ctuncay merged 2 commits into
mainfrom
feat/destination-event-filtering
Jul 16, 2026
Merged

Add per-destination event filtering#56
ctuncay merged 2 commits into
mainfrom
feat/destination-event-filtering

Conversation

@ctuncay

@ctuncay ctuncay commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an optional per-destination filter block to the emulator config:

destinations:
  - id: my-projection
    description: ...
    sources: [my-source]
    endpoint: http://backend:8080/api/v1/context/projection/my-projection
    filter:
      column: event_type
      values: ["OrderCreated", "OrderUpdated"]

Records whose column value is not in values are skipped and committed instead of delivered. Destinations without a filter are untouched.

Motivation: consuming apps register many HTTP destinations over the same topic, and every destination receives every record just to discard the irrelevant ones. On Sumori's local stack (34 destinations), ~91% of order-event deliveries are receive-and-discard waste. Filtering at the emulator removes that fan-out; the consumer's own accept() decoding remains the correctness gate — the filter is purely an optimization.

Fail-open semantics: records missing the filter column, or whose value is not a string, are delivered. A config mistake degrades to the pre-filter behaviour, never to silent data loss. Empty values lists are rejected at config parse time.

Compatibility — safe to release and roll out

  • Backward compatible (new emulator, old configs): filter is optional. Any existing config runs unchanged with identical delivery behaviour — the filtering code path is only reached when a destination declares a filter. Projects that don't use filtering are unaffected by upgrading the image.
  • Forward compatible (old emulator, new configs): current released versions parse destination objects leniently and ignore unknown keys, so a config that already contains filter blocks still loads on an old image — it just delivers everything, which is the pre-filter behaviour. Config rollout and image rollout can therefore happen in either order.
  • Consumer-group cursor/state format is untouched; a filtered record advances the cursor exactly like a delivered one, so upgrading or rolling back the image mid-stream is safe.

Testing

  • Unit: 6 projector specs (match / skip / fail-open on missing column and non-string values), 3 config parse specs (incl. empty-values rejection).
  • E2E: file-source test proving a filtered record is committed and does not block its partition.
  • Full suite locally on macOS: 286 examples, 0 failures, including the PostgreSQL, MySQL and SQL Server connector suites (recipe documented in the new CLAUDE.md).

🤖 Generated with Claude Code

Adds an optional filter: {column, values} block to destination configs. Records whose column value is not in values are skipped and committed instead of delivered, cutting receive-and-discard fan-out when many destinations share one topic.

Fail-open semantics: records missing the filter column, or whose value is not a string, are delivered — a config mistake degrades to the pre-filter behaviour, never to silent data loss. Destinations without a filter are untouched.

Includes projector/config/e2e test coverage, README config docs, and a CLAUDE.md with the dev-env setup and the verified macOS recipe for running the full DB-backed test suite locally.
@ctuncay ctuncay self-assigned this Jul 16, 2026
Comment thread src/Ambar/Emulator/Projector.hs Outdated
Review feedback on PR #56: a fail-open (missing column, null or non-string value, non-object record) looked exactly like a normal match, so a typo'd filter column silently restored full delivery and the no-op filter could go unnoticed indefinitely.

matchesFilter is now filterVerdict returning Matched / Skipped / FailedOpen with the reason. On the first FailedOpen a destination logs a warning naming the column and cause; subsequent fail-opens on that destination stay quiet so a permanently mismatched column cannot flood the log at full event rate (one warning per destination per emulator run). Delivery behaviour is unchanged — fail-open still delivers.
@ctuncay
ctuncay merged commit b1c1742 into main Jul 16, 2026
1 check passed
@ctuncay
ctuncay deleted the feat/destination-event-filtering branch July 16, 2026 17:35
ctuncay added a commit that referenced this pull request Jul 16, 2026
Reverts #56 due to push back from Luis, will revisit the code change once Marcelo is back.
@ctuncay
ctuncay restored the feat/destination-event-filtering branch July 16, 2026 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants