Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,15 @@ import { createEmailScanner } from "@textfilters/email";
const scanner = createEmailScanner();
const codePoints = Array.from("contact user@example.com");
const result = scanner.scan({ text: "contact user@example.com", codePoints });
const hasEmail = scanner.check({
text: "contact user@example.com",
codePoints,
});

scanner.scan({ text: "contact user@example.com", codePoints }, (match) => {
console.log(match.range);
return false;
});
```

The default shared instance is exported as `filter`. It has stable `name: "email"`.
Expand All @@ -98,8 +107,9 @@ False-positive guards avoid package scopes, social handles, prose-only `at` and
See [docs/architecture.md](docs/architecture.md) for the scanner flow, module map,
and change guide.

Run `npm run benchmark:email` from this package to compare clean, direct,
obfuscated, and late-match email censoring cases on the same machine.
Run `npm run benchmark:email` from this package to compare scanner setup,
`check()`, clean, no-dot `@` text, direct, obfuscated, and late-match email
censoring cases on the same machine.

## Related Textfilters Packages

Expand Down
47 changes: 30 additions & 17 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,11 @@ The package provides composable email censoring for direct addresses and common

The default `filter` export is a shared instance with default domain rules. `emailFilter(options?)` is an alias for `createEmailFilter(options?)`.

`createEmailScanner(options?)` exposes the same matching behavior as a range scanner. `scanEmailRanges(text, options?)` returns code point ranges directly for callers that want to compose masking through `@textfilters/core`.
`createEmailScanner(options?)` exposes the same matching behavior as a range
scanner. It supports `check(input)` for boolean checks and `scan(input, sink)`
for allocation-aware range streaming with early stop support. `scanEmailRanges`
returns code point ranges directly for callers that want to compose masking
through `@textfilters/core`.

`EmailFilterOptions` supports:

Expand Down Expand Up @@ -65,27 +69,36 @@ graph TD

## File Responsibilities

| File or directory | Responsibility | Out of scope |
| --------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------- |
| `src/index.ts` | Public entrypoint exports. | Scanner details. |
| `src/types.ts` | Public constants, options, filter types, and scanner contract types. | Internal token and metadata shapes. |
| `src/normalization.ts` | Code point metadata, NFKC lowercase folding, and character predicates. | Email matching policy. |
| `src/scanner.ts` | Scanner factory, cheap prefilter, option normalization, and final range merge. | Tokenization and rule internals. |
| `src/scanner/matching/direct.ts` | Direct literal candidate discovery around `@`. | Exclusions and boundary policy. |
| `src/scanner/matching/obfuscated.ts` | Optional obfuscated candidate discovery through the token stream. | Public API construction or masking. |
| `src/scanner/matching/candidate.ts` | Shared candidate validation, exclusions, and surrounding-boundary checks. | Candidate discovery. |
| `src/scanner/context/bare-at-policy.ts` | Bare `at` phrase policy for prose, labels, and accepted address lists. | Direct `user@example.com` matching. |
| `src/scanner/context` | Introducer, label, phrase, and address-list helpers for bare `at` policy. | Direct `user@example.com` matching. |
| `src/scanner/rules` | Local/domain validation, scanner lexicon, and exclusions. | Metadata construction. |
| `src/scanner/core/types.ts` | Shared internal token, punctuation, candidate, and scanner option types. | Public types. |
| `src/filter.ts` | Factory, shared instance, alias, and code point masking orchestration. | Tokenization and domain validation. |
| `tests/*.spec.ts` | Public behavior grouped by matching, contexts, prose guards, and integration. | Exhaustive RFC email validation. |
| File or directory | Responsibility | Out of scope |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| `src/index.ts` | Public entrypoint exports. | Scanner details. |
| `src/types.ts` | Public constants, options, filter types, and scanner contract types. | Internal token and metadata shapes. |
| `src/normalization.ts` | Code point metadata, NFKC lowercase folding, and character predicates. | Email matching policy. |
| `src/scanner.ts` | Scanner factory, cheap prefilter, boolean checks, sink streaming, option normalization, and final range merge. | Tokenization and rule internals. |
| `src/scanner/matching/direct.ts` | Direct literal candidate discovery around `@`. | Exclusions and boundary policy. |
| `src/scanner/matching/obfuscated.ts` | Optional obfuscated candidate discovery through the token stream. | Public API construction or masking. |
| `src/scanner/matching/candidate.ts` | Shared candidate validation, exclusions, and surrounding-boundary checks. | Candidate discovery. |
| `src/scanner/context/bare-at-policy.ts` | Bare `at` phrase policy for prose, labels, and accepted address lists. | Direct `user@example.com` matching. |
| `src/scanner/context` | Introducer, label, phrase, and address-list helpers for bare `at` policy. | Direct `user@example.com` matching. |
| `src/scanner/rules` | Local/domain validation, scanner lexicon, and exclusions. | Metadata construction. |
| `src/scanner/core/types.ts` | Shared internal token, punctuation, candidate, and scanner option types. | Public types. |
| `src/filter.ts` | Factory, shared instance, alias, and code point masking orchestration. | Tokenization and domain validation. |
| `tests/*.spec.ts` | Public behavior grouped by matching, contexts, prose guards, and integration. | Exhaustive RFC email validation. |

## Scanner Flow

Direct email scanning searches for normalized `@` characters, expands a local part to the left, expands a domain to the right, and emits an internal direct candidate. This path does not tokenize text and remains the small path for literal `user@example.com` matching.

Before metadata creation, the scanner checks for cheap direct or obfuscated email candidate signals. Clearly clean text returns no ranges without tokenization; candidate text still runs through the same direct, obfuscated, exclusion, and false-positive guards as before.
Before metadata creation, the scanner checks for cheap direct or obfuscated
email candidate signals. Shared-style hints such as `hasAtSign`, `hasDot`,
`hasNonAscii`, and text length can reject clearly clean input before metadata
allocation. Candidate text still runs through the same direct, obfuscated,
exclusion, and false-positive guards as before.

Boolean `check()` scans direct `@` windows first and stops after the first valid
candidate. Sink-based `scan(input, sink)` emits ranges as they are found and
stops when the sink returns `false`; the legacy `scan(input)` result shape is
preserved.

Obfuscated scanning is isolated under `src/scanner/matching/obfuscated.ts` and only runs when `matchObfuscated` is not `false`. It tokenizes words and separator tokens while ignoring whitespace around bracketed separators. It accepts `at`, `dot`, `@`, and `.` separator forms when they appear between a plausible local part and a valid dotted domain, then emits an internal obfuscated candidate.

Expand Down
25 changes: 24 additions & 1 deletion scripts/benchmark-email.mjs
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
import { performance } from "node:perf_hooks";
import { createEmailFilter } from "../dist/index.js";
import { createEmailFilter, createEmailScanner } from "../dist/index.js";

const ITERATIONS = 1_000;
const SETUP_ITERATIONS = 100;

const SHORT_CLEAN = "Hello world";
const LONG_CLEAN = "The quick brown fox jumps over the lazy dog. ".repeat(50);
const NO_DOT_AT = "Contact user@example for details";
const DIRECT_EMAIL = "Contact user@example.com for details";
const OBFUSCATED_EMAIL = "Contact user [at] example [dot] com for details";
const LATE_MATCH =
Expand Down Expand Up @@ -38,11 +39,33 @@ function printResults(results) {
}

const filter = createEmailFilter();
const scanner = createEmailScanner();
const input = (text) => ({ text, codePoints: Array.from(text) });
const hintedInput = (text) => ({
text,
codePoints: Array.from(text),
hints: {
textLength: text.length,
hasNonAscii: /[^\x00-\x7f]/u.test(text),
hasAtSign: text.includes("@"),
hasDot: text.includes("."),
},
});

printResults([
bench("createEmailFilter()", () => createEmailFilter(), SETUP_ITERATIONS),
bench("createEmailScanner()", () => createEmailScanner(), SETUP_ITERATIONS),
bench("check short clean", () => scanner.check(input(SHORT_CLEAN))),
bench("check hinted short clean", () =>
scanner.check(hintedInput(SHORT_CLEAN)),
),
bench("check long clean", () => scanner.check(input(LONG_CLEAN))),
bench("check no-dot at text", () => scanner.check(input(NO_DOT_AT))),
bench("check direct email", () => scanner.check(input(DIRECT_EMAIL))),
bench("check late-match email", () => scanner.check(input(LATE_MATCH))),
bench("censor short clean", () => filter.censor(SHORT_CLEAN)),
bench("censor long clean", () => filter.censor(LONG_CLEAN)),
bench("censor no-dot at text", () => filter.censor(NO_DOT_AT)),
bench("censor direct email", () => filter.censor(DIRECT_EMAIL)),
bench("censor obfuscated email", () => filter.censor(OBFUSCATED_EMAIL)),
bench("censor late-match email", () => filter.censor(LATE_MATCH)),
Expand Down
4 changes: 4 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
@@ -1,15 +1,19 @@
export { createEmailFilter, emailFilter, filter } from "./filter.js";
export {
EMAIL_FILTER_NAME,
type EmailRangeMatch,
type EmailRangeMatchSink,
type EmailRangeScanner,
type EmailRangeScanResult,
type EmailFilter,
type EmailFilterOptions,
type EmailScanInput,
} from "./types.js";
export {
checkEmailRanges,
collectEmailRanges,
createEmailScanner,
scanEmailRangeMatches,
scanEmailRanges,
type EmailScannerConfig,
} from "./scanner.js";
Loading