content'))).toBe(-1);
+ });
+
+ test('returns -1 across many marker-free chunks', () => {
+ const scan = createHeadEndScanner();
+ for (let i = 0; i < 20; i++) {
+ expect(scan(bytes('plain markup with heads and bodies spelled out
'))).toBe(-1);
+ }
+ });
+});
+
describe('guardedFetch against a loopback rig (real socket)', () => {
let server: Server;
let port: number;
@@ -111,6 +184,16 @@ describe('guardedFetch against a loopback rig (real socket)', () => {
referer?: string;
} | null = null;
+ const HUGE_PAGE_HEAD =
+ 'Huge Page';
+ let hugeStream = { closedEarly: false, finishedAll: false };
+ // Resolves when the huge-streaming socket closes, so the test waits on that
+ // event instead of polling a flag. Reassigned per request, with the resolver
+ // captured in that request's own close handler, so a late close from an
+ // earlier run resolves its own promise rather than the current run's (keeps
+ // the rig correct if this suite ever gains test.retry or test.concurrent).
+ let hugeClosed: Promise = Promise.resolve();
+
// The rig binds loopback; a public hostname resolves to it via the injected
// resolver, and the guard is told to treat that one loopback address as
// public so post-admission behavior can be exercised over a real socket.
@@ -193,6 +276,96 @@ describe('guardedFetch against a loopback rig (real socket)', () => {
res.end('{"not":"html"}');
return;
}
+ if (path === '/huge-streaming') {
+ // GitHub-shaped page: the head arrives in the first write, then a body
+ // far past the byte cap keeps streaming until the client hangs up.
+ res.writeHead(200, { 'Content-Type': 'text/html' });
+ res.write(HUGE_PAGE_HEAD);
+ const observed = { closedEarly: false, finishedAll: false };
+ hugeStream = observed;
+ const closed = Promise.withResolvers();
+ hugeClosed = closed.promise;
+ const filler = Buffer.alloc(64 * 1024, 0x78);
+ let written = 0;
+ const timer = setInterval(() => {
+ if (res.destroyed || res.writableEnded) return;
+ if (written >= 8 * 1024 * 1024) {
+ observed.finishedAll = true;
+ clearInterval(timer);
+ res.end();
+ return;
+ }
+ written += filler.byteLength;
+ res.write(filler);
+ }, 1);
+ res.on('close', () => {
+ clearInterval(timer);
+ if (!res.writableFinished) observed.closedEarly = true;
+ closed.resolve();
+ });
+ return;
+ }
+ if (path === '/marker-in-head-script') {
+ // A spelled inside a head-level ` +
+ `RealTitle${'z'.repeat(1024)}`,
+ );
+ return;
+ }
+ if (path === '/head-end-past-cap-multichunk') {
+ // A non-marker preamble in one write, then the head-end in a later
+ // write. Only counting the preamble (`received`) pushes the total past
+ // the cap, so the guard must add it to the in-chunk offset.
+ res.writeHead(200, { 'Content-Type': 'text/html' });
+ res.write('a'.repeat(80));
+ setTimeout(() => res.end(`${'b'.repeat(37)}tail`), 20);
+ return;
+ }
+ if (path === '/head-split-across-writes') {
+ res.writeHead(200, { 'Content-Type': 'text/html' });
+ res.write('Split res.end('ad>after the boundary'), 20);
+ return;
+ }
+ if (path === '/body-open-no-head-close') {
+ res.writeHead(200, { 'Content-Type': 'text/html' });
+ res.end(`NoClose${'y'.repeat(64 * 1024)}`);
+ return;
+ }
+ if (path === '/upper-head') {
+ res.writeHead(200, { 'Content-Type': 'text/html' });
+ res.end(`Caps${'z'.repeat(64 * 1024)}`);
+ return;
+ }
+ if (path === '/marker-after-cap') {
+ res.writeHead(200, { 'Content-Type': 'text/html' });
+ res.end(`${'x'.repeat(2048)}`);
+ return;
+ }
+ if (path === '/small-no-markers') {
+ res.writeHead(200, { 'Content-Type': 'text/html' });
+ res.end('Wholeevery byte returned
');
+ return;
+ }
+ if (path === '/json-with-marker-bytes') {
+ res.writeHead(200, { 'Content-Type': 'application/json' });
+ res.end('{"snippet":"","trailer":"kept"}');
+ return;
+ }
+ if (path === '/gzip-huge-head-first') {
+ // Tiny on the wire; decompressed far past the cap used by the test —
+ // but the head ends early, so the scan (which runs on DECOMPRESSED
+ // bytes) must admit it where the old whole-body read rejected it.
+ res.writeHead(200, { 'Content-Type': 'text/html', 'Content-Encoding': 'gzip' });
+ res.end(
+ gzipSync(`Zipped${'w'.repeat(200 * 1024)}`),
+ );
+ return;
+ }
if (path === '/slow') {
const timer = setTimeout(() => {
if (!res.writableEnded) {
@@ -335,4 +508,124 @@ describe('guardedFetch against a loopback rig (real socket)', () => {
});
expect(result).toEqual({ ok: false, reason: 'timeout' });
});
+
+ test('succeeds on a huge streaming page whose head arrives first, cancelling the rest', async () => {
+ const result = await guardedFetch(`http://rig.example:${port}/huge-streaming`, withRig);
+ expect(result.ok).toBe(true);
+ if (!result.ok) return;
+ // The body is the exact head prefix — nothing past head-end is buffered,
+ // regardless of how the runtime coalesced the filler into chunks.
+ expect(new TextDecoder().decode(result.body)).toBe(HUGE_PAGE_HEAD);
+ expect(result.body.byteLength).toBeLessThanOrEqual(DEFAULT_MAX_BYTES);
+ // The reader hung up on the still-streaming remainder rather than consuming
+ // the multi-MB body: wait for the socket-close event, failing loudly if it
+ // never arrives instead of polling a flag on a fixed interval.
+ let timer: ReturnType | undefined;
+ const timeout = new Promise<'timeout'>((resolve) => {
+ timer = setTimeout(() => resolve('timeout'), 2000);
+ });
+ const outcome = await Promise.race([hugeClosed.then(() => 'closed' as const), timeout]);
+ clearTimeout(timer);
+ expect(outcome).toBe('closed');
+ expect(hugeStream.closedEarly).toBe(true);
+ expect(hugeStream.finishedAll).toBe(false);
+ });
+
+ test('keeps the real head when a is spelled inside a head-level RealTitle",
+ );
+ });
+
+ test('detects a split across two response writes', async () => {
+ const result = await guardedFetch(`http://rig.example:${port}/head-split-across-writes`, {
+ ...withRig,
+ maxBytes: 4 * 1024,
+ });
+ expect(result.ok).toBe(true);
+ if (!result.ok) return;
+ expect(new TextDecoder().decode(result.body)).toBe('Split');
+ });
+
+ test('terminates at an opening {
+ const result = await guardedFetch(`http://rig.example:${port}/body-open-no-head-close`, {
+ ...withRig,
+ maxBytes: 4 * 1024,
+ });
+ expect(result.ok).toBe(true);
+ if (!result.ok) return;
+ // 64KB of body filler against a 4KB cap: only head-end termination admits it.
+ expect(new TextDecoder().decode(result.body)).toBe('NoClose)', async () => {
+ const result = await guardedFetch(`http://rig.example:${port}/upper-head`, {
+ ...withRig,
+ maxBytes: 4 * 1024,
+ });
+ expect(result.ok).toBe(true);
+ if (!result.ok) return;
+ expect(new TextDecoder().decode(result.body)).toBe('Caps');
+ });
+
+ test('still rejects as oversized when the cap is reached before head-end', async () => {
+ const result = await guardedFetch(`http://rig.example:${port}/marker-after-cap`, {
+ ...withRig,
+ maxBytes: 1024,
+ });
+ expect(result).toEqual({ ok: false, reason: 'oversized' });
+ });
+
+ test('rejects as oversized when head-end clears the cap only after earlier chunks count', async () => {
+ // 80-byte preamble in chunk one, then at offset 44 of chunk two:
+ // oversized only if the guard adds the 80 already received to the in-chunk
+ // offset (80 + 44 = 124 > 100). A check that dropped `received` would admit
+ // an over-cap body across writes.
+ const result = await guardedFetch(`http://rig.example:${port}/head-end-past-cap-multichunk`, {
+ ...withRig,
+ maxBytes: 100,
+ });
+ expect(result).toEqual({ ok: false, reason: 'oversized' });
+ });
+
+ test('returns an under-cap page with no head markers whole, as before', async () => {
+ const result = await guardedFetch(`http://rig.example:${port}/small-no-markers`, withRig);
+ expect(result.ok).toBe(true);
+ if (!result.ok) return;
+ expect(new TextDecoder().decode(result.body)).toBe(
+ 'Wholeevery byte returned
',
+ );
+ });
+
+ test('reads non-HTML content in full even when it contains marker-shaped bytes', async () => {
+ const result = await guardedFetch(`http://rig.example:${port}/json-with-marker-bytes`, {
+ ...withRig,
+ allowContentType: (mimeType) => mimeType === 'application/json',
+ });
+ expect(result.ok).toBe(true);
+ if (!result.ok) return;
+ expect(new TextDecoder().decode(result.body)).toBe(
+ '{"snippet":"","trailer":"kept"}',
+ );
+ });
+
+ test('applies the head-end scan to DECOMPRESSED bytes (gzip page far past the cap)', async () => {
+ const result = await guardedFetch(`http://rig.example:${port}/gzip-huge-head-first`, {
+ ...withRig,
+ maxBytes: 8 * 1024,
+ });
+ expect(result.ok).toBe(true);
+ if (!result.ok) return;
+ expect(new TextDecoder().decode(result.body)).toBe('Zipped');
+ });
});
diff --git a/packages/server/src/link-preview/guarded-fetch.ts b/packages/server/src/link-preview/guarded-fetch.ts
index 313064e9d..4fc569284 100644
--- a/packages/server/src/link-preview/guarded-fetch.ts
+++ b/packages/server/src/link-preview/guarded-fetch.ts
@@ -30,7 +30,14 @@
* re-run the full scheme+resolve+validate admission on each hop's target,
* bounded to a small hop count. Response reads are bounded on both time
* (AbortSignal) and DECOMPRESSED size, and admit only allowlisted content
- * types. Decompression contract: the request advertises
+ * types. HTML reads additionally stop at head-end (`` close or ``'s closing `>`, or the terminator that
+ * ends a `` reports
+ * the offset just past that terminator, mid-tag), or -1 while the head has not
+ * ended. Chunks are accumulated and re-scanned by `findHeadEndOffset` (the same
+ * context-aware walk the metadata extractor uses), so markup-shaped bytes inside
+ * a head-level script, comment, quoted attribute value, or `` are skipped
+ * rather than mistaken for the end, and a boundary split across chunks is
+ * detected once its bytes arrive. The accumulated view is latin1 (1:1 with
+ * bytes; the markup tokens are ASCII), so the offset it yields is a true byte
+ * offset.
+ *
+ * Re-scanning the whole accumulation per chunk is O(n^2) in the accumulated
+ * size; the fetch layer's byte cap (`DEFAULT_MAX_BYTES`) bounds that
+ * accumulation, so raising the cap raises this scan's worst-case cost
+ * quadratically.
+ */
+export function createHeadEndScanner(): (chunk: Uint8Array) => number {
+ let html = '';
+ return (chunk) => {
+ const consumed = html.length;
+ html += Buffer.from(chunk).toString('latin1');
+ const end = findHeadEndOffset(html);
+ // The previous accumulation held no boundary, so a newly found one always
+ // lands within this chunk: its offset is in (consumed, html.length].
+ return end === -1 ? -1 : end - consumed;
+ };
+}
+
function readCappedBody(
message: IncomingMessage,
maxBytes: number,
signal: AbortSignal,
+ scanHeadEnd: boolean,
): Promise<{ ok: true; body: Uint8Array } | { ok: false; reason: GuardRejectReason }> {
const encoding = (message.headers['content-encoding'] ?? '').trim().toLowerCase();
const decompressor = makeDecompressor(encoding);
@@ -358,6 +406,7 @@ function readCappedBody(
return new Promise((settle) => {
const chunks: Buffer[] = [];
+ const findHeadEnd = scanHeadEnd ? createHeadEndScanner() : null;
let received = 0;
let settled = false;
@@ -384,6 +433,24 @@ function readCappedBody(
);
}
source.on('data', (chunk: Buffer) => {
+ if (findHeadEnd) {
+ const endInChunk = findHeadEnd(chunk);
+ if (endInChunk !== -1) {
+ // Head-end seen: the head prefix is a complete successful body.
+ // Realize the head-first contract (only a cap reached BEFORE head-end
+ // is oversized) by cap-checking just the bytes up to the marker
+ // (`received + endInChunk`), then truncating there and stopping all
+ // further reads via teardown. The whole-chunk cap check below is
+ // bypassed, so bytes past head-end never count toward the cap.
+ if (received + endInChunk > maxBytes) {
+ resolveOnce({ ok: false, reason: 'oversized' });
+ return;
+ }
+ chunks.push(chunk.subarray(0, endInChunk));
+ resolveOnce({ ok: true, body: new Uint8Array(Buffer.concat(chunks)) });
+ return;
+ }
+ }
received += chunk.byteLength;
if (received > maxBytes) {
resolveOnce({ ok: false, reason: 'oversized' });
@@ -404,9 +471,12 @@ function rejectWith(reason: GuardRejectReason): GuardedFetchFailure {
}
/**
- * Fetch a URL under the SSRF guard. Returns the bounded response body plus the
- * derived content-type and final URL on success, or a bounded reason code on
- * any violation — never a partial body.
+ * Fetch a URL under the SSRF guard. On success returns the bounded response
+ * body plus the derived content-type and final URL; on any violation, a bounded
+ * reason code. For an HTML response the body is the head prefix (everything up
+ * to `</head>`/`<body>`) — all the metadata extractor reads — while any other
+ * admitted content (the favicon path's images) is returned whole. A failure
+ * never yields a partial body.
*/
export async function guardedFetch(
rawUrl: string,
@@ -453,7 +523,15 @@ export async function guardedFetch(
return rejectWith('non-html');
}
- const bodyResult = await readCappedBody(response, maxBytes, signal);
+ // Head-first streaming applies only to HTML pages; any other admitted
+ // content (the favicon path's images) may legitimately contain
+ // marker-shaped bytes and must be read in full.
+ const bodyResult = await readCappedBody(
+ response,
+ maxBytes,
+ signal,
+ contentType === 'text/html',
+ );
if (!bodyResult.ok) return rejectWith(bodyResult.reason);
return { ok: true, body: bodyResult.body, contentType, finalUrl: target.logicalUrl };
}
diff --git a/packages/server/src/link-preview/html-metadata.test.ts b/packages/server/src/link-preview/html-metadata.test.ts
index 6a9b135d78517d777910365797236dff66796472..b025f8bd51de70babce374978be8eb9d07647186 100644
GIT binary patch
delta 3094
zcmdT`-EJF26jpLU(kLm3lA0eudYXi;-S{Vw;HGg_N~1_Y0!4xdBqSupyJx*a*R$K*
zS=$jpmUx1qx#ogLK)eWWjf8jr1ij&!Gc&t(9k<j{uNpb_{>+>?=R4o|&iVJ^JMp->
z*yrEZ4>#)&aN(aL^Us8T5d;~^pT7KMeFPvelT`<YGG(630Q;C;(PE5z2-3I*3_5ub
zU<xwLg~ufNz$CO=A=z@Fs(%^*PcsRbOp*1t=t6+f3u{gtsx>%v;hPnpzg7z_LxgV5
zQlE^NH6z*-CmGO2UvDJK#vls64p|H`<QdQhhvZyzMfX_R;v&iE%#bChAJY;OL7b8?
z4|yLm2<V1^7Ym6(Hpp=Q3KG`E`(}eaXjEEDQVjSI23&?9V(7&F6)@rJYcGm3?Hh$G
zdy(5TTgWIyJb~ocJOyytW>H%kciWQ72;IvPxOU4vaA3b!zTl{Xg`Ia#DUzo`Z|=C0
zC64-a%h@>bdN}>`ueCQ{4_Lp+y?Dp9=}?bXc0V2aRI4>*eD)}3QPn1Za;^Qj56;*l
zr(QVXW9Fqikvtak{+d7PheaT!&}np-I47uRO{+M(F!62eh)n5k`nkrb=ke(4oG?rn
zeK7%Xg2KRTu$QbmO&|65^3n(6*x>*0exQ>_Arx?{lhxf7pD;3p9ksFK+!z~DPu|*8
z7fWl)<tDWNTdKCbx#&AT?``~68Le1oI#_jPN(ickTtnGl1C}DuCKo<SuZX)L-TRhL
zQ|z3s3JX+<Nt{yc>4mYWO&CM7P7E%KO*0YIM&;g;<&jjkZ`}HP=xF4D3LrsGR%e{7
zYDO9|JV0~ZBqhM%MNoAL!%hwMiMh6<oo0lhD?`QBw$7gK*G$S>7=$Qv!IbV&O?gP@
zshgF?sPMQf0h0{(Wn9<UbPLbzb>a9d5wnMUL?SZ*QifD*87IiO$heR6%E(Sc>K8%V
zmV&DVjTkx2!&1g3azob1xZP8V4Pg;Sn6U_tOjbLe70iI1=DdvKI%sh+i6ddWCY@tp
z$)JOzVpicuBqQS|^iVRNN!C=!Yq#zhsVRK!vlO!9GM5%*=cwJXua#T5<9uc5Vil~Q
z+wp9BP#fEuBl}tlNy(jv%Jsu}_+!#AW~Js}GmE%~)dLFMklb+cl+>AVL0q&9`GkTj
zN#eZ!%v?^TvP__W^UIGvBK&*m$=yx$&+_^k2A4aHMoCDE2B>yr_f-PnWPT9fQ6Ad{
zP0a1&5~0tc9Mzx8x0SnkcXd($ot;;W3kP$<P%ugTzPz{nR2v>3PyLb$wiRmq{GzCb
z`Q>x&5>UzZ+b32Hf#gxo7K4zJ<|oXfW-+O3nxiIqA~B^Qib*}m<Pp}i;LpgX(Q2?g
z&3)Mh0Hux=R?8MJL0mmtTX|3BsmP#lsDp^CBx%)zkYyzM5@={cU;VgO8O`>fXU#=r
zgQn99g=rj4jx48$h{iUW<!B7m9hPQ19u?D=KAVOHGdR+$$e^3X1O4tDi#et)cuZ*5
z0v2W1=rI~g=W@^haow%4n7t@GvztyYh&UVPV&hQ#adT%MJn9BCV;f~MgRu)3VUNaD
cAW-!@P>vMZ?b8<>toxufGraD+p3L<Be>x`b8vp<R
delta 30
mcmZq5`{qAEgC(gn&uXHU;l{iR0+ZEs_$N*m-rOR(TnqrbCJU7S
diff --git a/packages/server/src/link-preview/html-metadata.ts b/packages/server/src/link-preview/html-metadata.ts
index b0778e353..94782ab20 100644
--- a/packages/server/src/link-preview/html-metadata.ts
+++ b/packages/server/src/link-preview/html-metadata.ts
@@ -284,19 +284,29 @@ function findCloseTag(
return null;
}
+interface HeadScan {
+ rawTitle: string;
+ metaContent: Map<string, string>;
+ faviconHref: string | undefined;
+ /**
+ * Index just past the head-end boundary (`</head>`'s closing `>`, or the
+ * terminator that ends a `<body` open tag's name, mid-tag for an attributed
+ * `<body class="a">`), or -1 when the head does not end within `html`.
+ */
+ headEndOffset: number;
+}
+
/**
- * Extract the preview fields from a page's head. Title prefers `og:title` over
- * `<title>`; description prefers `og:description` over the `description` meta;
- * site name is `og:site_name`. Only content inside the head (before `</head>`
- * or `<body>`) counts, so a stray `<title>`/`<meta>` in the body cannot spoof
- * the card; a document with no explicit `<head>` (title before `<body>`) still
- * yields its head fields. Script/style content and HTML comments are skipped
- * wholesale, so markup-shaped strings inside them cannot contribute fields.
+ * One context-aware pass over a document head that both collects the metadata
+ * fields AND reports where the head ends. `extractHtmlMetadata` (fields) and
+ * `findHeadEndOffset` (boundary) are thin views over this single walk, so the
+ * streaming fetch and the parser can never disagree on where the head stops.
*/
-export function extractHtmlMetadata(html: string): RawHtmlMetadata {
+function scanHead(html: string): HeadScan {
let rawTitle = '';
const metaContent = new Map<string, string>();
let faviconHref: string | undefined;
+ let headEndOffset = -1;
try {
const lower = asciiLowerCase(html);
@@ -334,7 +344,16 @@ export function extractHtmlMetadata(html: string): RawHtmlMetadata {
if (next === 0x2f /* / */) {
const { name, end } = readTagName(lower, lt + 2);
- if (name === 'head') break;
+ if (name === 'head') {
+ // Record the boundary only once a `>` closes the tag in-buffer. A
+ // bare trailing `</head` (end-of-input) is left pending so a chunk
+ // boundary cannot mistake a still-growing `</header>` for head-end.
+ if (end < len) {
+ const gt = lower.indexOf('>', end);
+ if (gt !== -1) headEndOffset = gt + 1;
+ }
+ break;
+ }
const gt = lower.indexOf('>', end);
if (gt === -1) break;
i = gt + 1;
@@ -349,7 +368,18 @@ export function extractHtmlMetadata(html: string): RawHtmlMetadata {
const { name, end: nameEnd } = readTagName(lower, lt + 1);
- if (name === 'body') break;
+ if (name === 'body') {
+ // Head-end at the opening body tag, but only once a real terminator
+ // (`>`, `/`, or whitespace) settles the name in-buffer; a bare trailing
+ // `<body` waits for the next bytes rather than matching at a boundary.
+ if (nameEnd < len) {
+ const term = lower.charCodeAt(nameEnd);
+ if (term === 0x3e /* > */ || term === 0x2f /* / */ || isWhitespaceCode(term)) {
+ headEndOffset = nameEnd + 1;
+ }
+ }
+ break;
+ }
if (name === 'title') {
const contentStart = scanAttributes(html, lower, nameEnd, null);
@@ -408,6 +438,21 @@ export function extractHtmlMetadata(html: string): RawHtmlMetadata {
// request handler.
}
+ return { rawTitle, metaContent, faviconHref, headEndOffset };
+}
+
+/**
+ * Extract the preview fields from a page's head. Title prefers `og:title` over
+ * `<title>`; description prefers `og:description` over the `description` meta;
+ * site name is `og:site_name`. Only content inside the head (before `</head>`
+ * or `<body>`) counts, so a stray `<title>`/`<meta>` in the body cannot spoof
+ * the card; a document with no explicit `<head>` (title before `<body>`) still
+ * yields its head fields. Script/style content and HTML comments are skipped
+ * wholesale, so markup-shaped strings inside them cannot contribute fields.
+ */
+export function extractHtmlMetadata(html: string): RawHtmlMetadata {
+ const { rawTitle, metaContent, faviconHref } = scanHead(html);
+
const title = sanitizeText(metaContent.get('og:title') ?? rawTitle, MAX_TITLE);
const description = sanitizeText(
metaContent.get('og:description') ?? metaContent.get('description') ?? '',
@@ -423,6 +468,20 @@ export function extractHtmlMetadata(html: string): RawHtmlMetadata {
return result;
}
+/**
+ * Byte offset just past the end of a document head (`</head>`'s closing `>`, or
+ * the terminator that ends a `<body` open tag's name, mid-tag for an attributed
+ * `<body class="a">`), found by the SAME context-aware walk the metadata
+ * extractor uses, so markup-shaped bytes inside a head-level script, comment,
+ * quoted attribute value, or `<title>` never count as the boundary. Returns -1
+ * when the head has not ended within `html`, which lets a streaming caller keep
+ * reading; a bare trailing `</head`/`<body` (no terminator yet) stays -1 until
+ * the next bytes settle it, so a chunk boundary can never split a false match.
+ */
+export function findHeadEndOffset(html: string): number {
+ return scanHead(html).headEndOffset;
+}
+
/**
* The domain shown on the card, derived from the request URL only. `URL`
* normalizes IDN hosts to punycode, so a look-alike Unicode host renders as its