diff --git a/.gitignore b/.gitignore index c7e0409..0f808e5 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,4 @@ tests/fixtures/private-raw/* tests/e2e/.extension-out/ tests/e2e/certs/ !tests/e2e/test-extension.pem +native-host/*.generated.json diff --git a/AGENTS.md b/AGENTS.md index 0b103a3..b99f77b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -58,11 +58,14 @@ media-manifest.jsonl (when image download enabled — append-only download l - **Environment detection:** `isDevMode = !chrome.runtime.getManifest().update_url` — packed CWS extensions have `update_url`, unpacked don't. Used to switch seenIds storage between session (dev) and local (production). - **Volatile dev cache:** In dev mode (unpacked), `seenIds` is stored in `chrome.storage.session`, which clears on extension reload. This eliminates the need to manually clear storage during development. Production behavior is unchanged (persisted to `chrome.storage.local`). - **Dedup in service worker:** Multiple tabs feed the same service worker. `seenIds` Set (max 50,000, FIFO eviction) prevents duplicates. In production, persisted to `chrome.storage.local` across sessions; in dev mode, uses volatile `chrome.storage.session`. Both host and daemon also load seen IDs from existing JSONL files on startup. +- **Image backfill bypass:** When image download is enabled, duplicate tweets with photo URLs may be forwarded to the daemon once per service-worker session so skipped images can be downloaded. These forwards emit `IMAGE_BACKFILL` trace events and do not increment `sessionCount` or `allTimeCount`; toggling image download from off to on clears the session-only backfill cache so tweets can be rechecked. - **Jittered flush:** Batch flush uses `setTimeout` with randomized interval (30s base + up to 50% jitter = 30–45s), re-randomized each cycle. Avoids clockwork-regular patterns. - **Path validation:** When the user sets a custom output directory, the service worker sends a `TEST_PATH` request to the HTTP daemon, which attempts `makedirs` + write/delete of a temp file before accepting the path. - **Error resilience:** The native host logs crashes to `~/.xtap/host-error.log` with Python version and traceback. The HTTP daemon returns error status codes and logs startup diagnostics (Python version, output dir, token status). When the daemon is unreachable, the extension shows a red "!" badge and buffers tweets until the next successful reprobe (30s cooldown). The popup auto-refreshes every 2 seconds to reflect transport state changes. - **Daemon debug logging:** Set `XTAP_LOG_LEVEL=debug` to get per-request logging (method, path, duration, tweet counts, tracebacks). Configured via environment variable in the service template (launchd/systemd). Re-run `install.sh` after changing. - **Image download tuning (env vars, all optional):** `XTAP_IMAGE_DELAY_MS` (default 100) sets the inter-request delay for the background image worker. `XTAP_MAX_FILE_MB` (default 50) caps the size of a single image; the worker aborts mid-stream and deletes the `.part` file when exceeded. `XTAP_MAX_MEDIA_MB` (default unset = unlimited) caps the cumulative bytes downloaded per process; further jobs log `skipped:quota`. Bad values fall back to defaults with a stderr warning instead of crashing the daemon. +- **Video download tuning (env vars, optional):** `XTAP_MAX_VIDEO_MB` (default 500) caps the direct MP4 fallback download size and deletes the `.part` file if exceeded; set `0` or less to disable the cap. Bad values fall back to the default with a stderr warning. +- **Daemon connection lifecycle (env vars, optional):** `XTAP_CONN_TIMEOUT_S` (default 15) is the per-connection socket timeout on `DaemonHandler` — required because handler threads are non-daemon (`daemon_threads = False`) and `server_close()` joins them at shutdown; without it one idle connection blocks shutdown until SIGKILL. The timeout is per blocking read, not a total request deadline, so `XTAP_SHUTDOWN_GRACE_S` (default 10) additionally bounds the shutdown join and force-exits if a trickle peer keeps a handler alive. Both always resolve to a finite positive value — bad or `<=0` input falls back to the default. ## Stealth Constraints @@ -73,14 +76,14 @@ media-manifest.jsonl (when image download enabled — append-only download l 3. **No expando properties** — XHR URL tracking uses a `WeakMap`, never attaches properties to instances. 4. **No DOM footprint** — no injected elements, no visible page modifications. The only transient artifact is the `` tag, removed within milliseconds by the bridge script. 5. **No console output in page context** — all logging happens in the service worker, which runs outside the page's JavaScript environment. -6. **Minimal permissions** — only `storage` and `nativeMessaging`. Host permissions scoped to `x.com`, `twitter.com`, and `127.0.0.1` (local daemon only). No `webRequest`, no `tabs`, no `scripting`, no web-accessible resources. The debug dashboard is an internal extension page (`chrome-extension://` origin), not a web-accessible resource. +6. **Minimal permissions** — only `storage`, `nativeMessaging`, and `alarms`. Host permissions scoped to `x.com`, `twitter.com`, and `127.0.0.1` (local daemon only). No `webRequest`, no `tabs`, no `scripting`, no web-accessible resources. The debug dashboard is an internal extension page (`chrome-extension://` origin), not a web-accessible resource. 7. **Random event channel** — per-page-load name, meta tag removed immediately after reading. 8. **Only `open()` patched on XHR** — `send()` is not patched, so non-GraphQL XHR calls have clean stack traces. **Any change that adds network requests to X/Twitter domains must be rejected.** **Carve-outs (opt-in only, daemon-side, off by default):** -- **Video download** (`/download-video`) — user-initiated via popup button; calls `pbs.twimg.com` / `video.twimg.com` and runs yt-dlp. +- **Video download** (`/download-video`) — user-initiated via popup button; requires an `https://x.com/.../status/` or `https://twitter.com/.../status/` URL; calls `pbs.twimg.com` / `video.twimg.com` for direct MP4 fallback and runs yt-dlp when available. - **Image download** (`image_download:true` on `/tweets`) — opt-in toggle in popup. The daemon fetches `pbs.twimg.com` photos in a single background worker. Hostname allowlist enforced; redirects blocked; per-file size cap; never enabled by default. The browser-side capture path stays passive. These carve-outs run on the daemon, not the extension, so the page itself never originates the request. diff --git a/README.md b/README.md index 21a65db..9ca5e4a 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ xTap is a browser extension (Chrome + Firefox) that silently intercepts the Grap - **Structured output** — each tweet saved as a clean JSON object with author, metrics, media, and more - **Article support** — long-form X articles are captured when X returns the full body, with inline image references and Draft.js block structure - **Video download** — download videos from tweets using yt-dlp (or direct MP4 fallback) via the extension popup. Requires the HTTP daemon. **Note:** unlike passive capture, video downloads make additional network requests to X and are not stealth. -- **Image download** — opt-in toggle in the popup ("Download images automatically") fetches photos from `pbs.twimg.com` to `/media//` as you browse. Daemon-side; rate-limited; logs to `media-manifest.jsonl`. **Note:** also not stealth — adds requests to the Twitter CDN. +- **Image download** — opt-in toggle in the popup ("Download images automatically") fetches photos from `pbs.twimg.com` to `/media//` as you browse. If enabled after browsing, already-seen photo tweets are rechecked for media download without bumping the captured-tweet counter. Daemon-side; rate-limited; logs to `media-manifest.jsonl`. **Note:** also not stealth — adds requests to the Twitter CDN. - **Pause / resume** — click the extension icon to toggle capture on the fly - **Live counter** — badge on the extension icon shows tweets captured this session - **Multi-tab aware** — multiple X tabs feed into the same service worker with shared deduplication @@ -102,7 +102,7 @@ Even though passive interception is inherently low-risk, xTap avoids leaving unn - **Random event channel** — the MAIN↔ISOLATED world bridge uses a `CustomEvent` with a per-page-load random name; the `` beacon that communicates the name is removed immediately after the bridge reads it - **Zero DOM footprint** — no injected UI, no page modifications; everything lives in the popup and service worker - **Zero console output in page context** — all logging happens in the service worker and parser, which run outside the page's JavaScript environment -- **Minimal permissions** — only `storage` and `nativeMessaging`; no `webRequest`, no host permissions beyond `x.com` / `twitter.com` / `127.0.0.1` +- **Minimal permissions** — only `storage`, `nativeMessaging`, and `alarms` (the alarm wakes the service worker to deliver buffered tweets; it is not observable by web pages); no `webRequest`, no host permissions beyond `x.com` / `twitter.com` / `127.0.0.1` - **Jittered flush timing** — batches are flushed on a randomized interval to avoid a clockwork-regular pattern These measures don't make detection impossible — a determined page script could still compare prototype references or probe for patched behavior — but they avoid the low-hanging signals that fingerprinting scripts typically check. More importantly, there's nothing to detect server-side because xTap generates zero network activity of its own. @@ -261,20 +261,36 @@ If the extension shows "Not connected" or a red "!" badge: The easiest way to change where tweets are saved is through the extension popup — click the xTap icon and enter your preferred path in the **Output directory** field. -Alternatively, set the `XTAP_OUTPUT_DIR` environment variable before launching your browser: +Alternatively, set the `XTAP_OUTPUT_DIR` environment variable and re-run the installer — the daemon runs as a system service (launchd/systemd/Scheduled Task), so the variable must be baked into the service definition; exporting it in a shell or before launching the browser has no effect: ```bash export XTAP_OUTPUT_DIR="$HOME/Documents/xtap-data" +./native-host/install.sh [chrome|firefox] ``` +On Windows, set it as a *user* environment variable (`[Environment]::SetEnvironmentVariable('XTAP_OUTPUT_DIR', 'D:\\path', 'User')`) and re-run `install.ps1`. + | Setting | Default | Description | |---|---|---| | Popup "Output directory" | *(empty — uses default)* | Overrides the output path per-session | -| `XTAP_OUTPUT_DIR` env var | `~/Downloads/xtap` | Fallback when no popup setting is configured | +| `XTAP_OUTPUT_DIR` env var | `~/Downloads/xtap` | Fallback when no popup setting is configured (set at install time) | | Debug Dashboard | — | Accessible via popup link; shows live capture events, transport health, debug logging and discovery mode toggles, and parser sandbox | > **macOS note:** On macOS, the HTTP daemon (installed via `install.sh`) runs outside browser TCC sandboxes and can write to protected paths like `~/Documents` and iCloud Drive after a one-time macOS permission prompt. +### Download tuning + +These daemon environment variables are optional. Re-run the installer after changing them so the service definition picks them up. + +| Setting | Default | Description | +|---|---:|---| +| `XTAP_IMAGE_DELAY_MS` | `100` | Delay between background image requests | +| `XTAP_MAX_FILE_MB` | `50` | Max size for one downloaded image | +| `XTAP_MAX_MEDIA_MB` | *(unlimited)* | Max cumulative image bytes per daemon process | +| `XTAP_MAX_VIDEO_MB` | `500` | Max size for one direct MP4 fallback video download; set `0` or less to disable the cap | +| `XTAP_CONN_TIMEOUT_S` | `15` | Daemon per-connection socket timeout — drops idle/stalled connections; bad or `<=0` values fall back to the default (the timeout is never disabled) | +| `XTAP_SHUTDOWN_GRACE_S` | `10` | Max seconds the daemon waits for in-flight requests at shutdown before force-exiting; bad or `<=0` values fall back to the default | + ## Output Format Output is written to daily files (`tweets-YYYY-MM-DD.jsonl`). Each line is a self-contained JSON object: diff --git a/background.js b/background.js index b65833b..ebf0147 100644 --- a/background.js +++ b/background.js @@ -8,6 +8,7 @@ const FLUSH_INTERVAL_MS = 30_000; const MAX_SEEN_IDS = 50_000; const HTTP_TIMEOUT_MS = 10_000; const MAX_BUFFER_SIZE = 2000; +const IMAGE_BACKFILL_FLAG = '__xtap_image_backfill'; let captureEnabled = true; let buffer = []; @@ -184,11 +185,20 @@ const _origLog = console.log; const _origWarn = console.warn; const _origError = console.error; +const MAX_LOG_BUFFER = 5000; + function debugLog(level, args) { if (!debugLogging) return; const ts = new Date().toISOString(); - const text = args.map(a => typeof a === 'string' ? a : JSON.stringify(a)).join(' '); + const text = args.map(a => { + if (typeof a === 'string') return a; + try { return JSON.stringify(a); } catch { return String(a); } + }).join(' '); logBuffer.push(`${ts} [${level}] ${text}`); + // Cap so an unreachable daemon can't grow the buffer unboundedly. + if (logBuffer.length > MAX_LOG_BUFFER) { + logBuffer.splice(0, logBuffer.length - MAX_LOG_BUFFER); + } } console.log = (...args) => { _origLog(...args); debugLog('LOG', args); }; @@ -212,7 +222,7 @@ async function httpFetch(method, path, body) { opts.signal = controller.signal; try { const resp = await fetch(url, opts); - return await resp.json(); + return { status: resp.status, data: await resp.json() }; } finally { clearTimeout(timeout); } @@ -290,6 +300,7 @@ async function initTransport() { httpToken = cached.httpToken; httpPort = cached.httpPort; transport = 'http'; + updateBadge(); // clear a stale '!' from a previous daemon-down session console.log('[xTap] Using HTTP transport (cached token)'); return; } @@ -304,6 +315,7 @@ async function initTransport() { httpPort = result.port; transport = 'http'; await chrome.storage.local.set({ httpToken, httpPort }); + updateBadge(); // clear a stale '!' from a previous daemon-down session console.log('[xTap] Using HTTP transport (token from native host)'); return; } @@ -353,7 +365,19 @@ async function sendToHost(msg) { } try { - return await httpFetch('POST', path, body); + const { status, data } = await httpFetch('POST', path, body); + if (status === 401 || status === 403) { + // The daemon is alive but our token is stale (e.g. rotated by a + // reinstall). Reset the transport so the next flush re-probes and + // fetches a fresh token via the native host — otherwise we'd retry + // the same dead credentials forever. + console.error(`[xTap] Daemon rejected credentials (HTTP ${status}), resetting transport`); + transport = 'none'; + updateTransportBadge(); + return null; + } + if (data && typeof data === 'object') data.httpStatus = status; + return data; } catch (e) { console.error('[xTap] HTTP send failed:', e.message); transport = 'none'; @@ -368,6 +392,31 @@ function scheduledFlush() { if (buffer.length > 0 || logBuffer.length > 0) flush(); } +// The setTimeout flush timer dies with the service worker (MV3 kills it ~30s +// after the last event), stranding the final partial batch until a later +// session. A chrome.alarms backstop survives SW termination and wakes the SW +// to deliver whatever is still buffered, then clears itself. +const FLUSH_ALARM = 'xtap-flush'; +let flushAlarmSet = false; + +function ensureFlushAlarm() { + if (flushAlarmSet || !chrome.alarms) return; + flushAlarmSet = true; + chrome.alarms.create(FLUSH_ALARM, { periodInMinutes: 1 }); +} + +chrome.alarms?.onAlarm.addListener((alarm) => { + if (alarm.name !== FLUSH_ALARM) return; + (async () => { + await ready; + if (buffer.length > 0 || logBuffer.length > 0) await flush(); + if (buffer.length === 0 && logBuffer.length === 0) { + chrome.alarms.clear(FLUSH_ALARM); + flushAlarmSet = false; + } + })(); +}); + async function flushLogs() { if (logBuffer.length === 0) return; if (transport === 'none') return; @@ -426,6 +475,32 @@ async function reprobeTransport() { return false; } +// Cap per POST so a backlog (e.g. after a daemon outage) can't exceed the +// daemon's 10 MB body limit and wedge in a 413-retry loop. +const MAX_TWEETS_PER_POST = 200; +let flushInFlight = false; + +function tweetForHost(tweet) { + if (!tweet || !tweet[IMAGE_BACKFILL_FLAG]) return tweet; + const { [IMAGE_BACKFILL_FLAG]: _ignored, ...clean } = tweet; + return clean; +} + +function isBufferedImageBackfill(tweet) { + return !!(tweet && tweet[IMAGE_BACKFILL_FLAG]); +} + +// Remove the delivered/dropped tweets by object identity, never by position. +// An overflow eviction in enqueueTweets can splice the buffer front while +// flush() awaits the POST, so buffer[0..batch.length) may no longer be the +// tweets we sent — a positional splice would then discard never-sent tweets +// (their ids are already in seenIds, so the loss is permanent). Reassigning +// `buffer` is safe: no other code runs between the await and here. +function removeDelivered(batch) { + const delivered = new Set(batch); + buffer = buffer.filter(t => !delivered.has(t)); +} + async function flush() { if (buffer.length === 0 && logBuffer.length === 0) return; @@ -433,25 +508,59 @@ async function flush() { if (!(await reprobeTransport())) return; } - if (buffer.length > 0) { - const batch = buffer.splice(0); - const message = { tweets: batch }; - if (outputDir) message.outputDir = outputDir; - if (imageDownload) message.imageDownload = true; - + if (buffer.length > 0 && !flushInFlight) { + flushInFlight = true; try { - const resp = await sendToHost(message); - if (!resp || !resp.ok) { - console.error('[xTap] Host rejected tweets:', resp?.error || 'no response'); - buffer.unshift(...batch); - await saveState(); - } else { + // Send from the front without removing — the batch stays in `buffer` + // (and thus in persisted state) until the daemon acks, so SW death + // mid-POST can't lose it. The daemon dedups by ID, so a death after + // the POST but before the ack only costs a duplicate send, not data. + let postCap = MAX_TWEETS_PER_POST; + let splitRemaining = 0; + while (buffer.length > 0) { + const batch = buffer.slice(0, postCap); + const message = { tweets: batch.map(tweetForHost) }; + if (outputDir) message.outputDir = outputDir; + if (imageDownload) message.imageDownload = true; + + const resp = await sendToHost(message); + if (!resp || !resp.ok) { + // 413: the batch is too large in bytes (count cap doesn't bound + // huge article tweets). Retrying it unchanged would wedge the + // queue forever — halve and retry; a single tweet that alone + // exceeds the daemon's body limit can never be delivered, so + // drop it with a trace event. + if (resp?.httpStatus === 413) { + if (batch.length > 1) { + postCap = Math.max(1, Math.floor(batch.length / 2)); + splitRemaining = batch.length; + continue; + } + const oversized = batch[0]; + removeDelivered([oversized]); + console.error(`[xTap] Dropping tweet ${oversized.id}: exceeds daemon body limit`); + emitTraceEvent({ timestamp: Date.now(), endpoint: 'flush', tweetId: oversized.id, status: 'DROPPED_OVERSIZED', reason: 'single tweet exceeds daemon body limit' }); + postCap = MAX_TWEETS_PER_POST; // only the fat batch pays the split cost + splitRemaining = 0; + await saveState(); + continue; + } + // Anything else: batch stays buffered for the next flush. + console.error('[xTap] Host rejected tweets:', resp?.error || 'no response'); + break; + } + removeDelivered(batch); + if (splitRemaining > 0) { + splitRemaining -= batch.length; + if (splitRemaining <= 0) { + postCap = MAX_TWEETS_PER_POST; // restore after a fat batch forced a split + splitRemaining = 0; + } + } await saveState(); } - } catch (e) { - console.error('[xTap] Send failed, buffering tweets back:', e); - buffer.unshift(...batch); - await saveState(); + } finally { + flushInFlight = false; } } @@ -479,6 +588,10 @@ function emitTraceEvent(event) { function enqueueTweets(tweets, endpoint = 'unknown') { let newCount = 0; + let queuedCount = 0; + let backfillCount = 0; + let skippedCount = 0; + let droppedBackfillCount = 0; for (const tweet of tweets) { // Always cache for video lookup (even dupes — updates with latest data) if (tweet.id) { @@ -490,13 +603,29 @@ function enqueueTweets(tweets, endpoint = 'unknown') { } } + const wasSeen = !!(tweet.id && seenIds.has(tweet.id)); if (!dedupTweet(tweet, seenIds, { imageBackfill: imageDownload, imageCheckedIds })) { + skippedCount++; emitTraceEvent({ timestamp: Date.now(), endpoint, tweetId: tweet.id, status: 'DEDUPLICATED', reason: 'seenIds' }); continue; } + + const isImageBackfill = wasSeen && !tweet.is_article; + if (isImageBackfill) { + backfillCount++; + tweet[IMAGE_BACKFILL_FLAG] = true; + } else { + newCount++; + } buffer.push(tweet); - newCount++; - emitTraceEvent({ timestamp: Date.now(), endpoint, tweetId: tweet.id, status: 'ACCEPTED', reason: null }); + queuedCount++; + emitTraceEvent({ + timestamp: Date.now(), + endpoint, + tweetId: tweet.id, + status: isImageBackfill ? 'IMAGE_BACKFILL' : 'ACCEPTED', + reason: null, + }); } // FIFO eviction if seenIds grows too large @@ -509,20 +638,31 @@ function enqueueTweets(tweets, endpoint = 'unknown') { imageCheckedIds = new Set(arr.slice(arr.length - MAX_SEEN_IDS)); } - const dupeCount = tweets.length - newCount; - if (dupeCount > 0) { - console.log(`[xTap] Dedup: ${newCount} new, ${dupeCount} duplicates skipped (seenIds: ${seenIds.size})`); - } - sessionCount += newCount; allTimeCount += newCount; updateBadge(); + if (queuedCount > 0) ensureFlushAlarm(); if (buffer.length > MAX_BUFFER_SIZE) { - const overflow = buffer.length - MAX_BUFFER_SIZE; - buffer.splice(0, overflow); - console.warn(`[xTap] Buffer overflow: dropped ${overflow} oldest tweets (cap: ${MAX_BUFFER_SIZE})`); - emitTraceEvent({ timestamp: Date.now(), endpoint, tweetId: null, status: 'BUFFER_OVERFLOW', reason: `dropped ${overflow}` }); + let droppedOldestCount = 0; + while (buffer.length > MAX_BUFFER_SIZE) { + const backfillIndex = buffer.findIndex(isBufferedImageBackfill); + if (backfillIndex !== -1) { + const [dropped] = buffer.splice(backfillIndex, 1); + if (dropped?.id) imageCheckedIds.delete(dropped.id); + droppedBackfillCount++; + } else { + buffer.shift(); + droppedOldestCount++; + } + } + const droppedTotal = droppedOldestCount + droppedBackfillCount; + console.warn(`[xTap] Buffer overflow: dropped ${droppedTotal} tweets (${droppedBackfillCount} image backfill, ${droppedOldestCount} oldest; cap: ${MAX_BUFFER_SIZE})`); + emitTraceEvent({ timestamp: Date.now(), endpoint, tweetId: null, status: 'BUFFER_OVERFLOW', reason: `dropped ${droppedTotal}` }); + } + + if (skippedCount > 0 || backfillCount > 0 || droppedBackfillCount > 0) { + console.log(`[xTap] Dedup: ${newCount} new, ${backfillCount} image backfill, ${skippedCount} duplicates skipped, ${droppedBackfillCount} backfill dropped (seenIds: ${seenIds.size})`); } } @@ -658,7 +798,7 @@ chrome.runtime.onMessage.addListener((msg, _sender, sendResponse) => { } else { await clearStagedPayload(stageKey); } - // Flush after WAL commit so buffer.splice in flush() can't race with + // Flush after WAL commit so acknowledged batch removal can't race with // the persist-then-clear sequence above. if (buffer.length >= BATCH_SIZE) flush(); } catch (e) { @@ -738,7 +878,9 @@ chrome.runtime.onMessage.addListener((msg, _sender, sendResponse) => { } if (msg.type === 'SET_IMAGE_DOWNLOAD') { + const wasOff = !imageDownload; imageDownload = !!msg.imageDownload; + if (wasOff && imageDownload) imageCheckedIds.clear(); chrome.storage.local.set({ imageDownload }); sendResponse({ imageDownload }); return true; @@ -815,8 +957,9 @@ chrome.runtime.onMessage.addListener((msg, _sender, sendResponse) => { type: 'DOWNLOAD_STATUS', downloadId: msg.downloadId, }); - // Clean up finished downloads from active map - if (resp?.status === 'done' || resp?.status === 'error') { + // Clean up finished downloads from active map. 'unknown' means the + // daemon restarted and lost the download — stop resuming it. + if (resp?.status === 'done' || resp?.status === 'error' || resp?.status === 'unknown') { for (const [tid, did] of activeDownloads) { if (did === msg.downloadId) { activeDownloads.delete(tid); break; } } @@ -845,6 +988,12 @@ restoreState().catch((e) => { readyResolve(); updateBadge(); await initTransport(); + // Deliver anything restored/recovered right away — the SW may idle out + // before the first timer tick, and nothing else flushes at startup. + if (buffer.length > 0 || logBuffer.length > 0) { + if (buffer.length > 0) ensureFlushAlarm(); + flush(); + } function scheduleNextFlush() { const jitter = Math.random() * FLUSH_INTERVAL_MS * 0.5; flushTimer = setTimeout(() => { scheduledFlush(); scheduleNextFlush(); }, FLUSH_INTERVAL_MS + jitter); diff --git a/content-bridge.js b/content-bridge.js index 82cefbe..a78277f 100644 --- a/content-bridge.js +++ b/content-bridge.js @@ -20,15 +20,24 @@ }); } + // The MAIN world script normally runs within milliseconds; if the beacon + // hasn't appeared after the full window, it never will (injection blocked) + // — stop polling instead of spinning for the lifetime of the tab. + const BEACON_RETRY_MS = 50; + const BEACON_MAX_ATTEMPTS = 200; // ~10s + let beaconAttempts = 0; + function findBeacon() { const meta = document.querySelector('meta[name="__cfg"]'); if (meta) { const eventName = meta.content; meta.remove(); // Clean up — no trace left in DOM start(eventName); - } else { - // MAIN world script hasn't run yet, retry - requestAnimationFrame(findBeacon); + } else if (++beaconAttempts < BEACON_MAX_ATTEMPTS) { + // MAIN world script hasn't run yet, retry. setTimeout, not rAF — + // rAF callbacks never fire in hidden/background tabs, which would + // drop every event until the tab becomes visible. + setTimeout(findBeacon, BEACON_RETRY_MS); } } diff --git a/content-main.js b/content-main.js index 083e1aa..dcd874b 100644 --- a/content-main.js +++ b/content-main.js @@ -63,11 +63,17 @@ xhrPatched.add(this); this.addEventListener('load', function () { try { + const mappedUrl = xhrUrls.get(this); + if (!mappedUrl) return; const data = JSON.parse(this.responseText); - dispatchData(xhrUrls.get(this), data); + dispatchData(mappedUrl, data); } catch (_) {} }); } + } else { + // XHR object reused for a non-GraphQL request — drop the stale mapping + // so the load listener above ignores this response. + xhrUrls.delete(this); } return nativeOpen.call(this, method, url, ...rest); }; diff --git a/debug.css b/debug.css index edef61d..7552a46 100644 --- a/debug.css +++ b/debug.css @@ -157,8 +157,10 @@ h2 { } .status-ACCEPTED { color: #4ade80; } +.status-IMAGE_BACKFILL { color: #60a5fa; } .status-DEDUPLICATED { color: #8b98a5; } .status-PARSER_ERROR { color: #f87171; } +.status-DROPPED_OVERSIZED { color: #f87171; } /* Sandbox */ .sandbox-row { diff --git a/lib/dedup.js b/lib/dedup.js index 63b3152..adaddb4 100644 --- a/lib/dedup.js +++ b/lib/dedup.js @@ -11,7 +11,9 @@ * Returns false → tweet is a duplicate, should be skipped. * * - Tweets without an id are always new (and not tracked). - * - Article tweets bypass dedup — they may enrich a previously captured row. + * - Article tweets may bypass tweet-ID dedup once — they may enrich a + * previously captured non-article row, but repeated full article payloads + * should not be appended forever. * - When `options.imageBackfill` is set and the tweet has photo media, * a duplicate is also let through once per session (tracked in * `options.imageCheckedIds`) so the daemon can download images that @@ -23,6 +25,8 @@ export function dedupTweet(tweet, seenIds, options = {}) { const isDup = !!(tweet.id && seenIds.has(tweet.id)); const isArticle = !!tweet.is_article; + const articleKey = isArticle && tweet.id ? articleSeenKey(tweet.id) : null; + const isArticleDup = !!(articleKey && seenIds.has(articleKey)); const wantsImageBackfill = ( imageBackfill && !!tweet.id && @@ -31,21 +35,26 @@ export function dedupTweet(tweet, seenIds, options = {}) { hasPhotoMedia(tweet) ); - if (isDup && !isArticle && !wantsImageBackfill) { + if (isDup && (!isArticle || isArticleDup) && !wantsImageBackfill) { return false; } if (tweet.id) seenIds.add(tweet.id); + if (articleKey) seenIds.add(articleKey); if (wantsImageBackfill) imageCheckedIds.add(tweet.id); return true; } +function articleSeenKey(id) { + return `article:${id}`; +} + function hasPhotoMedia(tweet) { const media = tweet.media; if (!Array.isArray(media)) return false; for (const m of media) { - if (m && m.type === 'photo') return true; + if (m && m.type === 'photo' && m.url) return true; } return false; } diff --git a/lib/tweet-parser.js b/lib/tweet-parser.js index 44d6e08..8870dfd 100644 --- a/lib/tweet-parser.js +++ b/lib/tweet-parser.js @@ -17,8 +17,7 @@ export function extractTweets(endpoint, data) { if (!result) return []; const raw = unwrapTweetResult(result); if (!raw) return []; - const tweet = normalizeTweet(raw); - return tweet ? [tweet] : []; + return normalizeWithQuoted(raw); } const instructions = findInstructions(endpoint, data); @@ -151,8 +150,7 @@ function extractTweetsFromEntry(entry, skippedTypes) { // Single tweet item if (content.entryType === 'TimelineTimelineItem' || content.__typename === 'TimelineTimelineItem') { - const tweet = extractFromItemContent(content.itemContent); - if (tweet) tweets.push(tweet); + tweets.push(...extractFromItemContent(content.itemContent)); return tweets; } @@ -161,8 +159,7 @@ function extractTweetsFromEntry(entry, skippedTypes) { const items = content.items || []; for (const item of items) { const itemContent = item.item?.itemContent || item.itemContent; - const tweet = extractFromItemContent(itemContent); - if (tweet) tweets.push(tweet); + tweets.push(...extractFromItemContent(itemContent)); } return tweets; } @@ -170,8 +167,7 @@ function extractTweetsFromEntry(entry, skippedTypes) { // Fallback: try itemContent directly (may be at content.itemContent or content.item.itemContent) const fallbackItemContent = content.itemContent || content.item?.itemContent; if (fallbackItemContent) { - const tweet = extractFromItemContent(fallbackItemContent); - if (tweet) tweets.push(tweet); + tweets.push(...extractFromItemContent(fallbackItemContent)); return tweets; } @@ -187,7 +183,7 @@ function extractTweetsFromEntry(entry, skippedTypes) { } function extractFromItemContent(itemContent) { - if (!itemContent) return null; + if (!itemContent) return []; // Track non-tweet item types we encounter if (itemContent.itemType !== 'TimelineTweet' && itemContent.__typename !== 'TimelineTweet') { @@ -197,19 +193,51 @@ function extractFromItemContent(itemContent) { extractFromItemContent._seenTypes.add(itemType); console.log(`[xTap:parser] Skipping non-tweet itemType: "${itemType}"`); } - return null; + return []; } const tweetResults = itemContent.tweet_results; if (!tweetResults?.result) { console.warn('[xTap:parser] TimelineTweet has no tweet_results.result | keys:', Object.keys(itemContent).join(', ')); - return null; + return []; } const raw = unwrapTweetResult(tweetResults.result); - if (!raw) return null; + if (!raw) return []; + + return normalizeWithQuoted(raw); +} - return normalizeTweet(raw); +/** + * Normalize a tweet plus any inline quoted tweet it carries. X delivers the + * full quoted payload at quoted_status_result.result — capturing only + * legacy.quoted_status_id_str would discard the quoted content entirely. + */ +function normalizeWithQuoted(raw) { + const out = []; + const tweet = normalizeTweet(raw); + if (!tweet) return out; + out.push(tweet); + + // For retweets the quoted payload sits on the retweeted original. + const sources = [raw]; + const rtResult = raw.legacy?.retweeted_status_result?.result; + if (rtResult) { + const rtRaw = unwrapTweetResult(rtResult); + if (rtRaw) sources.push(rtRaw); + } + for (const source of sources) { + const quotedResult = source.quoted_status_result?.result; + if (!quotedResult) continue; + const quotedRaw = unwrapTweetResult(quotedResult); + if (!quotedRaw) continue; + const quoted = normalizeTweet(quotedRaw); + // Dedupe by id — a retweet-of-a-quote can carry the same quoted payload + // on both the wrapper and the original, and article tweets bypass all + // downstream dedup layers. + if (quoted && !out.some(t => t.id === quoted.id)) out.push(quoted); + } + return out; } /** diff --git a/manifest.firefox.json b/manifest.firefox.json index 20572d0..c36c1ac 100644 --- a/manifest.firefox.json +++ b/manifest.firefox.json @@ -1,11 +1,12 @@ { "manifest_version": 3, "name": "xTap", - "version": "0.23.2", + "version": "0.24.0", "description": "Passively captures tweets from X/Twitter as you browse", "permissions": [ "storage", - "nativeMessaging" + "nativeMessaging", + "alarms" ], "host_permissions": [ "*://*.x.com/*", diff --git a/manifest.json b/manifest.json index c793aa1..0774826 100644 --- a/manifest.json +++ b/manifest.json @@ -1,9 +1,9 @@ { "manifest_version": 3, "name": "xTap", - "version": "0.23.2", + "version": "0.24.0", "description": "Passively captures tweets from X/Twitter as you browse", - "permissions": ["storage", "nativeMessaging"], + "permissions": ["storage", "nativeMessaging", "alarms"], "host_permissions": ["*://*.x.com/*", "*://*.twitter.com/*", "http://127.0.0.1/*"], "background": { "service_worker": "background.js", diff --git a/native-host/com.xtap.daemon.plist b/native-host/com.xtap.daemon.plist index d3af63d..9328e9c 100644 --- a/native-host/com.xtap.daemon.plist +++ b/native-host/com.xtap.daemon.plist @@ -17,6 +17,8 @@ __PATH__ XTAP_LOG_LEVEL __LOG_LEVEL__ + XTAP_OUTPUT_DIR + __OUTPUT_DIR__ RunAtLoad diff --git a/native-host/com.xtap.daemon.service b/native-host/com.xtap.daemon.service index 3c12caf..1d3b15b 100644 --- a/native-host/com.xtap.daemon.service +++ b/native-host/com.xtap.daemon.service @@ -7,6 +7,7 @@ ExecStart=__PYTHON_PATH__ __DAEMON_PATH__ Environment=HOME=__HOME_DIR__ Environment=PATH=__PATH__ Environment=XTAP_LOG_LEVEL=__LOG_LEVEL__ +Environment="XTAP_OUTPUT_DIR=__OUTPUT_DIR__" Restart=always RestartSec=5 diff --git a/native-host/com.xtap.host.firefox.json b/native-host/com.xtap.host.firefox.json index eb1b92e..3a4667d 100644 --- a/native-host/com.xtap.host.firefox.json +++ b/native-host/com.xtap.host.firefox.json @@ -1,6 +1,6 @@ { "name": "com.xtap.host", - "description": "xTap native messaging host — writes captured tweets to JSONL", + "description": "xTap native messaging host — passes the HTTP daemon auth token to the extension", "path": "/path/to/xtap_host.py", "type": "stdio", "allowed_extensions": ["xtap@mkubicek.dev"] diff --git a/native-host/com.xtap.host.json b/native-host/com.xtap.host.json index f8cf71a..d1bbbb7 100644 --- a/native-host/com.xtap.host.json +++ b/native-host/com.xtap.host.json @@ -1,6 +1,6 @@ { "name": "com.xtap.host", - "description": "xTap native messaging host — writes captured tweets to JSONL", + "description": "xTap native messaging host — passes the HTTP daemon auth token to the extension", "path": "/path/to/xtap_host.py", "type": "stdio", "allowed_origins": ["chrome-extension://EXTENSION_ID/"] diff --git a/native-host/install.ps1 b/native-host/install.ps1 index 970e7a5..3d102a0 100644 --- a/native-host/install.ps1 +++ b/native-host/install.ps1 @@ -18,7 +18,11 @@ $HostName = "com.xtap.host" $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Definition $HostPy = Join-Path $ScriptDir "xtap_host.py" $BatPath = Join-Path $ScriptDir "xtap_host.bat" -$ManifestPath = Join-Path $ScriptDir "$HostName.json" +# Per-browser file: Chrome needs allowed_origins, Firefox allowed_extensions. +# A shared file would let the second install break the first browser (both +# registry keys would point at a manifest missing their key). The generated +# name also avoids clobbering the repo's template JSON files. +$ManifestPath = Join-Path $ScriptDir "$HostName.$Browser.generated.json" if (-not $ExtensionId -and $Browser -eq "firefox") { $ExtensionId = "xtap@mkubicek.dev" @@ -43,7 +47,7 @@ if (-not (Get-Command python -ErrorAction SilentlyContinue)) { # Write manifest (path must point to the .bat wrapper) $manifestData = @{ name = $HostName - description = "xTap native messaging host -- writes captured tweets to JSONL" + description = "xTap native messaging host -- passes the HTTP daemon auth token to the extension" path = $BatPath type = "stdio" } @@ -128,5 +132,8 @@ Write-Host " Stop-ScheduledTask -TaskName $TaskName" Write-Host "" $outputDir = if ($env:XTAP_OUTPUT_DIR) { $env:XTAP_OUTPUT_DIR } else { Join-Path $HOME "Downloads\xtap" } -Write-Host "Output directory (set XTAP_OUTPUT_DIR to change):" +Write-Host "Output directory:" Write-Host " $outputDir" +Write-Host "To change it, set XTAP_OUTPUT_DIR as a *user* environment variable" +Write-Host "(scheduled tasks do not see shell-session variables), then re-run:" +Write-Host " [Environment]::SetEnvironmentVariable('XTAP_OUTPUT_DIR', 'D:\path', 'User')" diff --git a/native-host/install.sh b/native-host/install.sh index fc28b4c..7fe6fe7 100755 --- a/native-host/install.sh +++ b/native-host/install.sh @@ -114,7 +114,7 @@ if [ "$BROWSER" = "firefox" ]; then cat > "$MANIFEST_PATH" < "$MANIFEST_PATH" </\>/g') + _OUT_SED=$(printf '%s' "$_OUT_XML" | sed -e 's/[\\&|]/\\&/g') # Substitute plist template mkdir -p "$HOME/Library/LaunchAgents" @@ -185,6 +193,7 @@ if [ "$OS" = "Darwin" ]; then -e "s|__HOME_DIR__|${HOME}|g" \ -e "s|__PATH__|${USER_PATH}|g" \ -e "s|__LOG_LEVEL__|${XTAP_LOG_LEVEL}|g" \ + -e "s|__OUTPUT_DIR__|${_OUT_SED}|g" \ "$PLIST_TEMPLATE" > "$PLIST_DEST" # Load daemon @@ -226,6 +235,13 @@ if [ "$OS" = "Linux" ]; then # Capture user's PATH so the daemon can find yt-dlp and other tools USER_PATH="$PATH" XTAP_LOG_LEVEL="${XTAP_LOG_LEVEL:-info}" + # Bake the output dir into the service environment — systemd does not + # inherit shell exports, so this is the only way XTAP_OUTPUT_DIR reaches + # the daemon (re-run install.sh after changing it). The template quotes the + # Environment= value (paths with spaces); escape sed metachars and any + # embedded double quotes so the substitution can't corrupt the unit file. + XTAP_OUTPUT_DIR="${XTAP_OUTPUT_DIR:-$HOME/Downloads/xtap}" + _OUT_SED=$(printf '%s' "$XTAP_OUTPUT_DIR" | sed -e 's/[\\&|]/\\&/g' -e 's/"/\\\\"/g') # Substitute service template mkdir -p "$SERVICE_DIR" @@ -235,11 +251,15 @@ if [ "$OS" = "Linux" ]; then -e "s|__HOME_DIR__|${HOME}|g" \ -e "s|__PATH__|${USER_PATH}|g" \ -e "s|__LOG_LEVEL__|${XTAP_LOG_LEVEL}|g" \ + -e "s|__OUTPUT_DIR__|${_OUT_SED}|g" \ "$SERVICE_TEMPLATE" > "$SERVICE_DEST" # Reload and enable systemctl --user daemon-reload systemctl --user enable --now "$SERVICE_NAME" + # enable --now does not restart an already-running unit — restart so a + # re-install actually applies the freshly baked environment. + systemctl --user try-restart "$SERVICE_NAME" 2>/dev/null || true echo "" echo "HTTP daemon installed:" @@ -253,7 +273,7 @@ if [ "$OS" = "Linux" ]; then fi echo "" -echo "Output directory (set XTAP_OUTPUT_DIR to change):" +echo "Output directory (set XTAP_OUTPUT_DIR and re-run install to change):" echo " ${XTAP_OUTPUT_DIR:-$HOME/Downloads/xtap}" echo "" echo "Debug logging (re-run install after setting):" diff --git a/native-host/xtap_core.py b/native-host/xtap_core.py index 43b785d..7013031 100644 --- a/native-host/xtap_core.py +++ b/native-host/xtap_core.py @@ -16,16 +16,24 @@ from urllib.parse import urlparse -DEFAULT_OUTPUT_DIR = os.environ.get('XTAP_OUTPUT_DIR', os.path.expanduser('~/Downloads/xtap')) +# realpath so the default dir and an explicit outputDir naming the same +# physical directory (e.g. via a symlinked home) share one seen-ID set — +# resolve_output_dir keys per-directory state by this string. +DEFAULT_OUTPUT_DIR = os.path.realpath( + os.environ.get('XTAP_OUTPUT_DIR', os.path.expanduser('~/Downloads/xtap'))) # Allowed roots for outputDir validation: user's home + DEFAULT_OUTPUT_DIR # (the latter covers XTAP_OUTPUT_DIR pointing outside home, e.g. /data/xtap) _ALLOWED_ROOTS = tuple(dict.fromkeys([ os.path.realpath(os.path.expanduser('~')), - os.path.realpath(DEFAULT_OUTPUT_DIR), + DEFAULT_OUTPUT_DIR, # already realpath-resolved at definition ])) +def _article_seen_key(tweet_id): + return f'article:{tweet_id}' + + def validate_output_dir(path): """Validate that a resolved path is under an allowed root directory. @@ -51,59 +59,94 @@ def validate_output_dir(path): def load_seen_ids(out_dir): """Build a set of tweet IDs from all existing JSONL files in the output directory.""" seen = set() - for path in glob.glob(os.path.join(out_dir, 'tweets-*.jsonl')): - with open(path, 'r') as f: - for line in f: - line = line.strip() - if not line: - continue - try: - tweet_id = json.loads(line).get('id') - if tweet_id: - seen.add(tweet_id) - except (json.JSONDecodeError, KeyError): - continue + escaped_dir = glob.escape(out_dir) + # Files evicted by iCloud ("Optimize Mac Storage") become hidden + # .name.icloud placeholders the glob below can't read — dedup silently + # loses their IDs, so at least make the degradation visible. + placeholders = glob.glob(os.path.join(escaped_dir, '.tweets-*.jsonl.icloud')) + if placeholders: + names = ', '.join(os.path.basename(p) for p in placeholders[:5]) + print(f'[xtap] WARNING: {len(placeholders)} tweets file(s) evicted by ' + f'iCloud — their IDs cannot be deduplicated against: {names}', + file=sys.stderr) + for path in glob.glob(os.path.join(escaped_dir, 'tweets-*.jsonl')): + try: + # errors='replace': a corrupt file must not crash the daemon at + # startup — unreadable lines just fail JSON parsing and are skipped. + with open(path, 'r', encoding='utf-8', errors='replace') as f: + for line in f: + line = line.strip() + if not line: + continue + try: + tweet = json.loads(line) + tweet_id = tweet.get('id') + if tweet_id: + seen.add(tweet_id) + if tweet.get('is_article'): + seen.add(_article_seen_key(tweet_id)) + except (json.JSONDecodeError, KeyError): + continue + except OSError as e: + print(f'[xtap] WARNING: cannot read {path}: {e}', file=sys.stderr) return seen -def resolve_output_dir(msg_dir, default_dir, seen_ids, custom_dirs): - """Resolve output directory from message, loading seen IDs for new custom dirs. +def resolve_output_dir(msg_dir, default_dir, seen_ids_by_dir): + """Resolve output directory from message, scoping seen IDs per directory. - Returns the resolved output directory path. + seen_ids_by_dir maps out_dir -> set of tweet IDs already written there. + Per-directory scoping matters: after switching to a new output directory, + tweets already captured into the old one must still be written to the new + archive (a shared set would silently suppress them). + + Returns (out_dir, seen_ids) for the resolved directory. """ if msg_dir: out_dir = validate_output_dir(os.path.expanduser(msg_dir)) os.makedirs(out_dir, exist_ok=True) - if out_dir != default_dir and out_dir not in custom_dirs: - seen_ids.update(load_seen_ids(out_dir)) - custom_dirs.add(out_dir) else: out_dir = default_dir - return out_dir + if out_dir not in seen_ids_by_dir: + seen_ids_by_dir[out_dir] = load_seen_ids(out_dir) + return out_dir, seen_ids_by_dir[out_dir] def write_tweets(tweets, out_dir, seen_ids): - """Write tweets to JSONL, deduplicating against seen_ids. Returns (count, dupes).""" + """Write tweets to JSONL, deduplicating against seen_ids. Returns (count, dupes). + + seen_ids is committed only after the file has been flushed and closed — + if the write fails, the extension retries the batch, and pre-committed IDs + would make the retry look like all-duplicates (silent data loss). A partial + flush before a failure can yield duplicate lines on retry; duplicates are + recoverable, dropped tweets are not. + """ out_file = os.path.join(out_dir, f'tweets-{date.today().isoformat()}.jsonl') count = 0 dupes = 0 - with open(out_file, 'a') as f: + written_ids = set() + with open(out_file, 'a', encoding='utf-8') as f: for tweet in tweets: tid = tweet.get('id') - if tid and tid in seen_ids and not tweet.get('is_article'): - dupes += 1 - continue - if tid: - seen_ids.add(tid) + article_key = _article_seen_key(tid) if tid and tweet.get('is_article') else None + if tid and (tid in seen_ids or tid in written_ids): + if not article_key or article_key in seen_ids or article_key in written_ids: + dupes += 1 + continue f.write(json.dumps(tweet, ensure_ascii=False) + '\n') + if tid: + written_ids.add(tid) + if article_key: + written_ids.add(article_key) count += 1 + seen_ids |= written_ids return count, dupes def write_log(lines, out_dir): """Append debug log lines to daily log file. Returns logged count.""" log_file = os.path.join(out_dir, f'debug-{date.today().isoformat()}.log') - with open(log_file, 'a') as f: + with open(log_file, 'a', encoding='utf-8') as f: for line in lines: f.write(line + '\n') return len(lines) @@ -116,7 +159,7 @@ def write_dump(filename, content, out_dir): if not safe_name or safe_name in ('.', '..'): raise ValueError(f'Invalid dump filename: {filename!r}') dump_file = os.path.join(out_dir, safe_name) - with open(dump_file, 'w') as f: + with open(dump_file, 'w', encoding='utf-8') as f: f.write(content) return dump_file @@ -126,7 +169,7 @@ def test_path(out_dir): os.makedirs(out_dir, exist_ok=True) test_file = os.path.join(out_dir, f'.xtap-write-test-{threading.get_ident()}') try: - with open(test_file, 'w') as f: + with open(test_file, 'w', encoding='utf-8') as f: f.write('ok') finally: try: @@ -137,6 +180,11 @@ def test_path(out_dir): # --- Video download --- +VIDEO_REQUEST_TIMEOUT_S = 60 +VIDEO_CHUNK_SIZE = 64 * 1024 +DEFAULT_MAX_VIDEO_MB = 500 +TWEET_STATUS_HOSTS = frozenset({'x.com', 'twitter.com'}) + _ytdlp_path = None _ytdlp_checked = False _downloads = {} @@ -168,30 +216,106 @@ def get_download_status(download_id): def _date_prefix(post_date): """Convert ISO date string to yyyy.mm.dd prefix, or empty string on failure.""" - if not post_date: + if not isinstance(post_date, str): return '' + match = re.match(r'^(\d{4})-(\d{2})-(\d{2})(?:T|$)', post_date) + if not match: + return '' + year, month, day = match.groups() try: - # Handle both "2024-01-15T12:34:56.000Z" and "2024-01-15" - dt = post_date[:10].replace('-', '.') - return dt + '_' - except Exception: + datetime.strptime(f'{year}-{month}-{day}', '%Y-%m-%d') + except ValueError: return '' + return f'{year}.{month}.{day}_' + + +def _tweet_status_id(tweet_url): + """Validate an X/Twitter status URL and return its numeric tweet ID.""" + if not isinstance(tweet_url, str): + raise ValueError('tweetUrl must be an X/Twitter status URL') + parsed = urlparse(tweet_url) + host = (parsed.hostname or '').lower() + if parsed.scheme != 'https' or host not in TWEET_STATUS_HOSTS: + raise ValueError(f'tweetUrl not allowed: {tweet_url!r}') + match = re.search(r'/(?:[^/?#]+/)*status/(\d+)(?:/|$)', parsed.path) + if not match: + raise ValueError(f'tweetUrl must be a status URL: {tweet_url!r}') + return match.group(1) + + +def validate_tweet_url(tweet_url): + """Public validation hook for daemon request handling.""" + _tweet_status_id(tweet_url) + return tweet_url + + +def _safe_video_path(video_dir, filename): + if (not filename or os.path.basename(filename) != filename or + '/' in filename or '\\' in filename): + raise ValueError(f'unsafe video filename: {filename!r}') + base = os.path.realpath(video_dir) + candidate = os.path.realpath(os.path.join(video_dir, filename)) + try: + if os.path.commonpath([base, candidate]) != base: + raise ValueError(f'video path escapes output directory: {filename!r}') + except ValueError: + raise ValueError(f'video path escapes output directory: {filename!r}') + return candidate + + +def _video_max_bytes(): + max_video_mb = _env_float('XTAP_MAX_VIDEO_MB', float(DEFAULT_MAX_VIDEO_MB), label='video') + if max_video_mb <= 0: + return None + return int(max_video_mb * 1024 * 1024) -def download_direct(direct_url, tweet_id, video_dir, post_date=''): # pragma: no cover +def download_direct(direct_url, tweet_id, video_dir, post_date=''): """Download video via direct CDN URL. Returns the file path.""" + # Same trust boundary as the image downloader: only fetch from Twitter's + # CDN over https — the URL crosses the daemon's HTTP API and must not be + # able to point the daemon at file:// or internal hosts. + parsed = urlparse(direct_url) + host = (parsed.hostname or '').lower() + if parsed.scheme != 'https' or not (host == 'twimg.com' or host.endswith('.twimg.com')): + raise ValueError(f'direct URL not allowed: {direct_url!r}') os.makedirs(video_dir, exist_ok=True) prefix = _date_prefix(post_date) filename = f'{prefix}{tweet_id}.mp4' - filepath = os.path.join(video_dir, filename) + filepath = _safe_video_path(video_dir, filename) tmp_path = filepath + '.part' - urllib.request.urlretrieve(direct_url, tmp_path) - os.replace(tmp_path, filepath) + max_bytes = _video_max_bytes() + # _NO_REDIRECT_OPENER (not bare urlopen) so a redirect can't bounce the + # daemon off the allowlisted host — the host check only validates the + # initial URL. Socket-level timeout so a stalled CDN can't hang the + # download thread (and the popup's polling) forever. + try: + with _NO_REDIRECT_OPENER.open(direct_url, timeout=VIDEO_REQUEST_TIMEOUT_S) as resp, \ + open(tmp_path, 'wb') as f: + headers = getattr(resp, 'headers', {}) or {} + content_length = headers.get('Content-Length') + if max_bytes is not None and content_length and content_length.isdigit(): + if int(content_length) > max_bytes: + raise ValueError('video exceeds XTAP_MAX_VIDEO_MB') + written = 0 + while True: + chunk = resp.read(VIDEO_CHUNK_SIZE) + if not chunk: + break + written += len(chunk) + if max_bytes is not None and written > max_bytes: + raise ValueError('video exceeds XTAP_MAX_VIDEO_MB') + f.write(chunk) + os.replace(tmp_path, filepath) + except Exception: + _safe_unlink(tmp_path) + raise return filepath def start_download(download_id, tweet_url, direct_url, out_dir, post_date=''): # pragma: no cover """Start a background download. Returns immediately; poll get_download_status().""" + tweet_id = _tweet_status_id(tweet_url) video_dir = os.path.join(out_dir, 'videos') os.makedirs(video_dir, exist_ok=True) @@ -210,9 +334,6 @@ def run(): elif direct_url: with _downloads_lock: _downloads[download_id]['progress'] = 0 - # Extract tweet ID from URL - m = re.search(r'/status/(\d+)', tweet_url) - tweet_id = m.group(1) if m else download_id path = download_direct(direct_url, tweet_id, video_dir, post_date) with _downloads_lock: _downloads[download_id].update( @@ -304,15 +425,16 @@ def _download_with_ytdlp(download_id, tweet_url, video_dir, post_date=''): # pr Downloads into a .downloading/ staging subdirectory so partial files are not visible in video_dir until the download is fully complete. """ + tweet_id = _tweet_status_id(tweet_url) staging_dir = os.path.join(video_dir, '.downloading') os.makedirs(staging_dir, exist_ok=True) prefix = _date_prefix(post_date) # Pin the tweet status ID in the filename rather than relying on %(id)s, # which some yt-dlp Twitter sub-extractors (amplify/broadcast/card) fill # with a media or broadcast ID instead of the tweet ID. - m = re.search(r'/status/(\d+)', tweet_url) - id_part = m.group(1) if m else '%(id)s' - output_template = os.path.join(staging_dir, prefix + '%(title)s [' + id_part + '].%(ext)s') + output_template = _safe_video_path( + staging_dir, + prefix + '%(title)s [' + tweet_id + '].%(ext)s') cmd = [ _ytdlp_path, '--newline', '--progress', @@ -347,8 +469,15 @@ def _download_with_ytdlp(download_id, tweet_url, video_dir, post_date=''): # pr # Move completed file from staging dir to final video_dir final_path = tracker.final_path if final_path and os.path.isfile(final_path): - dest_path = os.path.join(video_dir, os.path.basename(final_path)) - shutil.move(final_path, dest_path) + staging_real = os.path.realpath(staging_dir) + final_real = os.path.realpath(final_path) + try: + if os.path.commonpath([staging_real, final_real]) != staging_real: + raise RuntimeError('yt-dlp output escaped staging directory') + except ValueError: + raise RuntimeError('yt-dlp output escaped staging directory') + dest_path = _safe_video_path(video_dir, os.path.basename(final_path)) + shutil.move(final_real, dest_path) final_path = dest_path with _downloads_lock: _downloads[download_id].update( @@ -379,14 +508,14 @@ def _env_int(name, default): return default -def _env_float(name, default): +def _env_float(name, default, label='image'): raw = (os.environ.get(name) or '').strip() if not raw: return default try: return float(raw) except ValueError: - print(f'[xtap:image] invalid {name}={raw!r}, using {default}', file=sys.stderr) + print(f'[xtap:{label}] invalid {name}={raw!r}, using {default}', file=sys.stderr) return default @@ -426,6 +555,14 @@ def _is_safe_rel_path(out_dir, rel_path): return False +def _strip_article_text_local_path(article, rel_path): + text = article.get('text') + if not isinstance(text, str): + return + pattern = re.compile(r'!\[[^\]\n]*\]\(' + re.escape(rel_path) + r'\)') + article['text'] = pattern.sub('', text) + + def collect_image_jobs(tweets, out_dir): """Compute the list of image download jobs for a batch of tweets. @@ -437,9 +574,9 @@ def collect_image_jobs(tweets, out_dir): Article media items (under `tweet.article.media[]`) already have `local_path` set by the JS parser because that path is also embedded in the article's rendered Markdown text (`![](media//file.png)`). - Those entries are validated here but not mutated; if the supplied - `local_path` would escape `out_dir`, the field is stripped so the - unsafe path never lands in the JSONL. + Those entries are validated here; if the supplied `local_path` would + escape `out_dir`, the field and its rendered Markdown image reference + are stripped so the unsafe path never lands in the JSONL. Path components are validated against `out_dir`: tweet IDs must match [0-9]+, filenames must be plain basenames, and the final resolved @@ -480,6 +617,7 @@ def collect_image_jobs(tweets, out_dir): if not _is_safe_rel_path(out_dir, rel_path): # Strip the unsafe path so it doesn't end up in the JSONL. item.pop('local_path', None) + _strip_article_text_local_path(article, rel_path) continue pending.append({'tweet_id': tweet_id, 'url': url, 'rel_path': rel_path}) @@ -671,7 +809,7 @@ def _log(self, out_dir, tweet_id, url, dest_path, status, size): manifest_path = os.path.join(out_dir, 'media-manifest.jsonl') try: os.makedirs(out_dir, exist_ok=True) - with open(manifest_path, 'a') as f: + with open(manifest_path, 'a', encoding='utf-8') as f: f.write(json.dumps(entry, ensure_ascii=False) + '\n') except OSError as e: print(f'[xtap:image] manifest write failed: {e}', file=sys.stderr) diff --git a/native-host/xtap_daemon.py b/native-host/xtap_daemon.py index 181c9ff..b3c3621 100755 --- a/native-host/xtap_daemon.py +++ b/native-host/xtap_daemon.py @@ -6,6 +6,7 @@ import os import platform import signal +import socket import sys import time import uuid @@ -16,12 +17,44 @@ validate_output_dir, write_tweets, write_log, write_dump, test_path, check_ytdlp, start_download, get_download_status, - collect_image_jobs, get_image_downloader) + collect_image_jobs, get_image_downloader, + validate_tweet_url) -VERSION = '0.23.2' +VERSION = '0.24.0' BIND_HOST = '127.0.0.1' BIND_PORT = int(os.environ.get('XTAP_DAEMON_PORT', 17381)) -MAX_BODY_SIZE = 10 * 1024 * 1024 # 10 MB +MAX_BODY_SIZE = 10 * 1024 * 1024 # 10 MB (extension caps POSTs via MAX_TWEETS_PER_POST in background.js) +MAX_DRAIN_SIZE = 64 * 1024 * 1024 # cap bytes drained from an oversized request before replying 413 +MAX_DRAIN_IDLE_TIMEOUT_S = 0.25 # don't hang on clients that declare a body and send none + + +def _env_timeout_s(name, default): + # Always finite — bad/<=0 values fall back to the default rather than + # disabling the timeout (disabling would reintroduce the shutdown-hang + # these guards exist to prevent). + raw = (os.environ.get(name) or '').strip() + if not raw: + return default + try: + value = float(raw) + except ValueError: + return default + return value if value > 0 else default + + +# Per-connection socket timeout (see DaemonHandler.timeout). Default 15s: +# above the extension's 10s client timeout (HTTP_TIMEOUT_MS) so a legit +# localhost request is never cut off, and below launchd's ~20s SIGTERM-> +# SIGKILL window so an idle connection can't push shutdown past it. +CONN_TIMEOUT_S = _env_timeout_s('XTAP_CONN_TIMEOUT_S', 15.0) + +# Bounded shutdown join. The socket timeout above is per blocking read, not a +# total request deadline — a trickle peer sending one byte per read window +# keeps its handler thread alive forever, and server_close() would join it +# indefinitely. After this grace period the daemon force-exits: legitimate +# JSONL writes finish in well under a second, so only stuck peers are cut off, +# and 10s stays below launchd's ~20s SIGTERM->SIGKILL escalation. +SHUTDOWN_GRACE_S = _env_timeout_s('XTAP_SHUTDOWN_GRACE_S', 10.0) XTAP_DIR = os.path.expanduser('~/.xtap') XTAP_SECRET = os.path.join(XTAP_DIR, 'secret') @@ -40,7 +73,7 @@ def log_debug(msg): def load_token(): try: - with open(XTAP_SECRET, 'r') as f: + with open(XTAP_SECRET, 'r', encoding='utf-8') as f: return f.read().strip() except FileNotFoundError: log_info(f'FATAL: {XTAP_SECRET} not found. Run install.sh first.') @@ -49,12 +82,33 @@ def load_token(): # Module-level state shared across requests _token = None -_seen_ids = set() -_custom_dirs = set() +_seen_ids_by_dir = {} _state_lock = threading.Lock() +class XtapHTTPServer(ThreadingHTTPServer): + # daemon_threads defaults to True on ThreadingHTTPServer, which makes + # server_close()'s thread-join a no-op (daemon threads aren't tracked) — + # an in-flight handler is then killed mid-JSONL-write at interpreter exit. + # False + block_on_close (default True) makes server_close() actually wait + # for in-flight writes to finish before the process exits. + daemon_threads = False + + class DaemonHandler(BaseHTTPRequestHandler): + # Socket timeout for stalled/idle connections. Required by the + # daemon_threads = False change above: without it, a peer that opens a + # connection to 127.0.0.1 and never finishes sending its request line/body + # (a port scanner, a stalled fetch — this is pre-auth, so the token can't + # gate it) parks a non-daemon handler thread in a blocking read forever. + # server_close() then blocks joining that thread on shutdown until + # launchd/systemd escalates to SIGKILL, killing in-flight JSONL writes — + # the exact failure daemon_threads = False exists to prevent. The timeout + # makes each blocking read raise so idle threads exit. It is per-read, not + # a total deadline — a trickle sender can still hold a thread, which is + # why shutdown is additionally bounded by SHUTDOWN_GRACE_S in main(). + timeout = CONN_TIMEOUT_S + def log_message(self, format, *args): # Log to stderr (captured by launchd/systemd) log_info(f'{self.client_address[0]} - {format % args}') @@ -108,18 +162,41 @@ def _validate_content_length(self): self._send_json({'ok': False, 'error': 'Content-Length must not be negative'}, 400) return -1 if length > MAX_BODY_SIZE: + # Drain the (bounded) body before replying. An unread request body + # on a closed HTTP/1.0 socket triggers a TCP RST, which surfaces to + # the extension's fetch() as a network error rather than a 413 — so + # the client's batch-splitting recovery never sees the 413 and the + # oversized batch wedges the queue. Draining lets the 413 arrive + # cleanly. Capped so a bogus huge Content-Length can't tie us up. + self._drain_body(min(length, MAX_DRAIN_SIZE)) self._send_json({'ok': False, 'error': 'Payload too large'}, 413) return -1 return length + def _drain_body(self, n): + old_timeout = self.connection.gettimeout() + self.connection.settimeout(MAX_DRAIN_IDLE_TIMEOUT_S) + remaining = n + try: + while remaining > 0: + try: + chunk = self.rfile.read(min(65536, remaining)) + except socket.timeout: + break + if not chunk: + break + remaining -= len(chunk) + finally: + self.connection.settimeout(old_timeout) + def do_POST(self): log_debug(f'POST {self.path} (Content-Length: {self.headers.get("Content-Length", "?")})') - length = self._validate_content_length() - if length < 0: + if not self._check_auth(): return - if not self._check_auth(): + length = self._validate_content_length() + if length < 0: return try: @@ -155,15 +232,16 @@ def _handle_tweets(self, body): # silently turning the feature on. image_download = body.get('image_download') is True with _state_lock: - out_dir = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids, _custom_dirs) + out_dir, seen_ids = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids_by_dir) tweets = body.get('tweets', []) - # Compute jobs only when we'll actually fetch — collect_image_jobs - # may strip unsafe article local_paths from the tweets, so we - # must call it before write_tweets to keep the JSONL clean. - pending_images = collect_image_jobs(tweets, out_dir) if image_download else [] - count, dupes = write_tweets(tweets, out_dir, _seen_ids) + # Always collect before write_tweets: collect_image_jobs strips + # unsafe article local_paths from the tweets so traversal paths + # never land in the JSONL — sanitization must not depend on the + # image_download toggle. + pending_images = collect_image_jobs(tweets, out_dir) + count, dupes = write_tweets(tweets, out_dir, seen_ids) queued = 0 - if pending_images: + if image_download and pending_images: get_image_downloader().enqueue(pending_images, out_dir) queued = len(pending_images) log_debug(f' Tweets: {count} written, {dupes} dupes, {queued} images queued -> {out_dir}') @@ -178,10 +256,12 @@ def _handle_tweets(self, body): def _handle_log(self, body): try: msg_dir = body.get('outputDir', '').strip() - with _state_lock: - out_dir = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids, _custom_dirs) lines = body.get('lines', []) - logged = write_log(lines, out_dir) + with _state_lock: + out_dir, _ = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids_by_dir) + # Write under the lock — concurrent appends through separate + # file objects can interleave mid-line at buffer boundaries. + logged = write_log(lines, out_dir) self._send_json({'ok': True, 'logged': logged}) except ValueError as e: self._send_json({'ok': False, 'error': str(e)}, 400) @@ -194,7 +274,7 @@ def _handle_dump(self, body): try: msg_dir = body.get('outputDir', '').strip() with _state_lock: - out_dir = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids, _custom_dirs) + out_dir, _ = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids_by_dir) filename = body.get('filename', 'dump.json') content = body.get('content', '') path = write_dump(filename, content, out_dir) @@ -232,8 +312,9 @@ def _handle_download_video(self, body): direct_url = body.get('directUrl', '') post_date = body.get('postDate', '') msg_dir = body.get('outputDir', '').strip() + validate_tweet_url(tweet_url) with _state_lock: - out_dir = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids, _custom_dirs) + out_dir, _ = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids_by_dir) download_id = str(uuid.uuid4()) start_download(download_id, tweet_url, direct_url, out_dir, post_date) log_debug(f' Download started: {download_id} -> {tweet_url}') @@ -261,9 +342,9 @@ def _setup_stdio(): return os.makedirs(XTAP_DIR, exist_ok=True) if sys.stdout is None: - sys.stdout = open(os.path.join(XTAP_DIR, 'daemon-stdout.log'), 'a') + sys.stdout = open(os.path.join(XTAP_DIR, 'daemon-stdout.log'), 'a', encoding='utf-8') if sys.stderr is None: - sys.stderr = open(os.path.join(XTAP_DIR, 'daemon-stderr.log'), 'a') + sys.stderr = open(os.path.join(XTAP_DIR, 'daemon-stderr.log'), 'a', encoding='utf-8') def _log_startup_diagnostics(): @@ -290,7 +371,7 @@ def _log_startup_diagnostics(): def main(): - global _token, _seen_ids + global _token _setup_stdio() @@ -298,13 +379,13 @@ def main(): # Initialize output directory and seen IDs os.makedirs(DEFAULT_OUTPUT_DIR, exist_ok=True) - _seen_ids = load_seen_ids(DEFAULT_OUTPUT_DIR) + _seen_ids_by_dir[DEFAULT_OUTPUT_DIR] = load_seen_ids(DEFAULT_OUTPUT_DIR) _log_startup_diagnostics() - log_info(f' Seen IDs: {len(_seen_ids)} loaded') + log_info(f' Seen IDs: {len(_seen_ids_by_dir[DEFAULT_OUTPUT_DIR])} loaded') try: - server = ThreadingHTTPServer((BIND_HOST, BIND_PORT), DaemonHandler) + server = XtapHTTPServer((BIND_HOST, BIND_PORT), DaemonHandler) except OSError as e: log_info(f'FATAL: Cannot bind to {BIND_HOST}:{BIND_PORT} — {e}') log_info(f' Is another instance already running?') @@ -312,7 +393,11 @@ def main(): def shutdown(signum, frame): log_info(f'Received signal {signum}, shutting down...') - server.shutdown() + # shutdown() must run on a different thread than serve_forever(): the + # signal handler interrupts the serve_forever thread, and shutdown() + # blocks until serve_forever exits — calling it inline deadlocks until + # launchd/systemd escalates to SIGKILL (killing in-flight writes). + threading.Thread(target=server.shutdown, daemon=True).start() signal.signal(signal.SIGINT, shutdown) if platform.system() == 'Windows': @@ -322,6 +407,25 @@ def shutdown(signum, frame): log_info(f'Listening on {BIND_HOST}:{BIND_PORT}') server.serve_forever() + # Join in-flight non-daemon handler threads so any active JSONL write can + # finish cleanly — but bound the join: a peer that trickles bytes defeats + # the per-read socket timeout and would pin server_close() forever, + # holding the SIGTERM window open until launchd/systemd SIGKILLs us + # mid-write anyway. Force-exit after the grace period instead: real + # writes have long finished; only stuck peers are cut off. + closer = threading.Thread(target=server.server_close, daemon=True) + closer.start() + closer.join(SHUTDOWN_GRACE_S) + if closer.is_alive(): + log_info(f'Shutdown grace ({SHUTDOWN_GRACE_S}s) expired with handler ' + f'threads still running — forcing exit') + sys.stderr.flush() + try: + sys.stdout.flush() + except (OSError, ValueError): + pass + os._exit(0) + log_info('Shutdown complete') if __name__ == '__main__': diff --git a/native-host/xtap_host.py b/native-host/xtap_host.py index 19ba26c..7b62184 100755 --- a/native-host/xtap_host.py +++ b/native-host/xtap_host.py @@ -32,6 +32,16 @@ def read_exact(stream, size): return buf +def drain_exact(stream, size): + """Discard exactly *size* bytes so the next frame starts at a boundary.""" + remaining = size + while remaining > 0: + chunk = stream.read(min(65536, remaining)) + if not chunk: + raise EOFError('unexpected end of stream') + remaining -= len(chunk) + + def read_message(): try: header = read_exact(sys.stdin.buffer, 4) @@ -39,6 +49,10 @@ def read_message(): raise EOFError('stdin closed') length = struct.unpack(' MAX_MESSAGE_BYTES: + # Drain the payload before raising — leaving it in the pipe would make + # the next read interpret payload bytes as a length header, and the + # stream would never realign. + drain_exact(sys.stdin.buffer, length) raise ValueError(f'message too large: {length} bytes') data = read_exact(sys.stdin.buffer, length) return json.loads(data) @@ -63,7 +77,7 @@ def _main(): if msg.get('type') == 'GET_TOKEN': try: - with open(XTAP_SECRET, 'r') as f: + with open(XTAP_SECRET, 'r', encoding='utf-8') as f: token = f.read().strip() send_message({'ok': True, 'token': token, 'port': XTAP_PORT}) except FileNotFoundError: @@ -83,7 +97,7 @@ def main(): _main() except Exception: os.makedirs(XTAP_DIR, exist_ok=True) - with open(XTAP_ERROR_LOG, 'a') as f: + with open(XTAP_ERROR_LOG, 'a', encoding='utf-8') as f: f.write(f'\n--- {datetime.now().isoformat()} ---\n') f.write(f'Python: {sys.version}\n') f.write(f'Script: {os.path.abspath(__file__)}\n') diff --git a/popup.js b/popup.js index 227a947..db38f28 100644 --- a/popup.js +++ b/popup.js @@ -186,6 +186,10 @@ function pollDownload(downloadId) { } else if (resp.status === 'error') { clearInterval(pollTimer); showDownloadResult('error', resp.error || 'Download failed'); + } else if (resp.status === 'unknown') { + // Daemon restarted and lost the download state — stop polling. + clearInterval(pollTimer); + showDownloadResult('error', 'Download state lost — try again'); } }); }, 500); diff --git a/scripts/build-firefox-manifest.js b/scripts/build-firefox-manifest.js index aa68eb5..c3d4ed3 100755 --- a/scripts/build-firefox-manifest.js +++ b/scripts/build-firefox-manifest.js @@ -23,9 +23,8 @@ manifest.browser_specific_settings = { } }; -fs.writeFileSync( - path.join(root, 'manifest.firefox.json'), - JSON.stringify(manifest, null, 2) + '\n' -); +// Optional output path override (used by tests to diff without overwriting) +const outPath = process.argv[2] || path.join(root, 'manifest.firefox.json'); +fs.writeFileSync(outPath, JSON.stringify(manifest, null, 2) + '\n'); -console.log('Generated manifest.firefox.json'); +console.log(`Generated ${outPath}`); diff --git a/tests/content-bridge.test.mjs b/tests/content-bridge.test.mjs new file mode 100644 index 0000000..92f9d47 --- /dev/null +++ b/tests/content-bridge.test.mjs @@ -0,0 +1,129 @@ +/** + * Tests for content-bridge.js — beacon polling, give-up bound, and event relay. + * Run with: node --test tests/content-bridge.test.mjs + * + * Evaluates the bridge in a vm context with a stubbed document, fake timers, + * and a recorded chrome.runtime.sendMessage. + */ + +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import vm from 'node:vm'; + +const bridgeSource = readFileSync(new URL('../content-bridge.js', import.meta.url), 'utf8'); + +const BEACON_MAX_ATTEMPTS = 200; // mirrors content-bridge.js + +function setup({ meta = null, documentElement = {} } = {}) { + const state = { + meta, // set to an object to make the beacon findable + metaRemoved: false, + listeners: {}, // eventName -> [handlers] + scheduled: [], // pending fake-timer callbacks: {fn, ms} + sent: [], // chrome.runtime.sendMessage payloads + rafCalls: 0, + }; + + const document = { + documentElement, + querySelector(sel) { + assert.equal(sel, 'meta[name="__cfg"]'); + if (!state.meta) return null; + return { + content: state.meta.content, + remove() { state.metaRemoved = true; }, + }; + }, + addEventListener(name, handler) { + (state.listeners[name] ||= []).push(handler); + }, + }; + + const sandbox = { + document, + chrome: { runtime: { sendMessage: (msg) => state.sent.push(msg) } }, + // Fake timers: collect callbacks; tests drain them explicitly. + setTimeout(fn, ms) { state.scheduled.push({ fn, ms }); return state.scheduled.length; }, + // rAF must NOT be used — it never fires in hidden/background tabs. + requestAnimationFrame() { state.rafCalls++; }, + JSON, + }; + + vm.runInNewContext(bridgeSource, sandbox); + return state; +} + +// Run every currently-scheduled timer once (new timers queue for the next round). +function drainOnce(state) { + const batch = state.scheduled.splice(0); + for (const { fn } of batch) fn(); + return batch.length; +} + +describe('beacon discovery', () => { + it('finds the beacon immediately, removes the meta tag, and relays events', () => { + const state = setup({ meta: { content: 'evt-abc' } }); + + assert.equal(state.metaRemoved, true, 'the meta tag must leave no DOM trace'); + assert.equal(state.scheduled.length, 0, 'no retry needed when the beacon is present'); + assert.ok(state.listeners['evt-abc'], 'bridge must listen on the beacon event name'); + + state.listeners['evt-abc'][0]({ + detail: JSON.stringify({ url: 'https://x.com/i/api/graphql/X/HomeTimeline', endpoint: 'HomeTimeline', data: { a: 1 } }), + }); + // JSON round-trip: the message object is created inside the vm realm, so + // a direct deepEqual would fail on cross-realm prototypes. + assert.deepEqual(JSON.parse(JSON.stringify(state.sent)), [{ + type: 'GRAPHQL_RESPONSE', + url: 'https://x.com/i/api/graphql/X/HomeTimeline', + endpoint: 'HomeTimeline', + data: { a: 1 }, + }]); + }); + + it('retries via setTimeout (not rAF) and finds the beacon on a later attempt', () => { + const state = setup(); + assert.equal(state.scheduled.length, 1, 'a miss must schedule exactly one retry'); + assert.equal(state.scheduled[0].ms, 50, 'retry interval is BEACON_RETRY_MS'); + assert.equal(state.rafCalls, 0, + 'rAF callbacks never fire in hidden tabs — polling must use setTimeout'); + + drainOnce(state); // miss #2 + state.meta = { content: 'evt-late' }; + drainOnce(state); // finds it + assert.ok(state.listeners['evt-late'], 'beacon found on retry must start the bridge'); + assert.equal(state.scheduled.length, 0, 'no further polling after the beacon is found'); + }); + + it('gives up after BEACON_MAX_ATTEMPTS instead of polling forever', () => { + const state = setup(); + let totalRetries = 1; // the initial miss already scheduled one + for (let round = 0; round < BEACON_MAX_ATTEMPTS + 10 && state.scheduled.length > 0; round++) { + totalRetries += drainOnce(state); + } + // Initial call misses and schedules retry 1; retries 1..199 miss and + // schedule the next; retry 199 pushes beaconAttempts to 200, which stops. + assert.equal(totalRetries - 1, BEACON_MAX_ATTEMPTS - 1, + 'polling must stop at the attempt bound'); + assert.equal(state.scheduled.length, 0, 'no timer may remain after give-up'); + assert.deepEqual(Object.keys(state.listeners), [], + 'no event listener without a beacon'); + }); + + it('ignores malformed event payloads without throwing', () => { + const state = setup({ meta: { content: 'evt-x' } }); + state.listeners['evt-x'][0]({ detail: 'not json {' }); + assert.deepEqual(state.sent, [], 'malformed payloads are dropped silently'); + }); + + it('waits for DOMContentLoaded when documentElement is not ready', () => { + const state = setup({ documentElement: null }); + assert.equal(state.scheduled.length, 0, 'no polling before the document exists'); + assert.ok(state.listeners['DOMContentLoaded'], 'must defer to DOMContentLoaded'); + + state.meta = { content: 'evt-dcl' }; + state.listeners['DOMContentLoaded'][0](); + assert.ok(state.listeners['evt-dcl'], 'polling starts once the document is ready'); + }); +}); diff --git a/tests/dedup-xhr.test.mjs b/tests/dedup-xhr.test.mjs index cbe2fa9..d671127 100644 --- a/tests/dedup-xhr.test.mjs +++ b/tests/dedup-xhr.test.mjs @@ -39,10 +39,11 @@ describe('dedupTweet (Bug 1: undefined poisoning)', () => { assert.equal(dedupTweet({ id: '2', text: 'second' }, seenIds), true); }); - it('does not deduplicate article tweets even with same id', () => { + it('lets one full article enrich a previously seen non-article row', () => { const seenIds = new Set(); assert.equal(dedupTweet({ id: '1', text: 'stub' }, seenIds), true); assert.equal(dedupTweet({ id: '1', text: 'full article', is_article: true }, seenIds), true); + assert.equal(dedupTweet({ id: '1', text: 'same full article', is_article: true }, seenIds), false); }); it('respects pre-existing seenIds', () => { @@ -109,6 +110,26 @@ describe('dedupTweet (image-backfill bypass)', () => { assert.ok(!imageCheckedIds.has('1')); }); + it('does not bypass dedup for URL-less photo media', () => { + const seenIds = new Set(['1']); + const imageCheckedIds = new Set(); + const opts = { imageBackfill: true, imageCheckedIds }; + assert.equal( + dedupTweet({ id: '1', text: 'photo without URL', media: [{ type: 'photo' }] }, seenIds, opts), + false + ); + assert.ok(!imageCheckedIds.has('1')); + }); + + it('handles null, empty, and mixed media defensively', () => { + const seenIds = new Set(['1']); + const opts = () => ({ imageBackfill: true, imageCheckedIds: new Set() }); + assert.equal(dedupTweet({ id: '1', media: null }, seenIds, opts()), false); + assert.equal(dedupTweet({ id: '1', media: [] }, seenIds, opts()), false); + assert.equal(dedupTweet({ id: '1', media: [null, { type: 'photo', url: 'x' }] }, seenIds, opts()), true); + assert.equal(dedupTweet({ id: '1', media: [{ type: 'video' }, { type: 'photo', url: 'x' }] }, seenIds, opts()), true); + }); + it('does not bypass when imageBackfill is off (default)', () => { const seenIds = new Set(['1']); assert.equal(dedupTweet(photoTweet('1'), seenIds), false); @@ -133,6 +154,18 @@ describe('dedupTweet (image-backfill bypass)', () => { assert.equal(dedupTweet({ id: '1', text: 'first' }, seenIds), true); assert.equal(dedupTweet({ id: '1', text: 'dupe' }, seenIds), false); }); + + it('documents article tweets with photo media and imageBackfill', () => { + const seenIds = new Set(['1']); + const imageCheckedIds = new Set(); + const tweet = { + id: '1', + is_article: true, + media: [{ type: 'photo', url: 'https://pbs.twimg.com/media/x.jpg:orig' }], + }; + assert.equal(dedupTweet(tweet, seenIds, { imageBackfill: true, imageCheckedIds }), true); + assert.ok(imageCheckedIds.has('1')); + }); }); // --------------------------------------------------------------------------- @@ -266,3 +299,21 @@ describe('XHR listener stacking — production content-main.js (Bug 2)', () => { assert.equal(env.dispatched.length, 0); }); }); + +describe('XHR stale mapping — GraphQL open followed by non-GraphQL re-open', () => { + it('drops the stale mapping so the non-GraphQL response is not dispatched', () => { + const env = createBrowserEnv(); + runInNewContext(contentMainCode, { ...env, console }); + + const xhr = new env.XMLHttpRequest(); + env.XMLHttpRequest.prototype.open.call( + xhr, 'GET', 'https://x.com/i/api/graphql/abc/TweetDetail' + ); + env.XMLHttpRequest.prototype.open.call( + xhr, 'GET', 'https://x.com/i/api/1.1/jot/client_event.json' + ); + fireLoad(xhr, JSON.stringify({ not: 'graphql' })); + assert.equal(env.dispatched.length, 0, + 'response of a non-GraphQL request must not be dispatched under the stale GraphQL URL'); + }); +}); diff --git a/tests/fixtures/sanitized/timeline-basic/expected.jsonl b/tests/fixtures/sanitized/timeline-basic/expected.jsonl index 191e6e7..7aa06dc 100644 --- a/tests/fixtures/sanitized/timeline-basic/expected.jsonl +++ b/tests/fixtures/sanitized/timeline-basic/expected.jsonl @@ -1,38 +1,46 @@ -{"id":"2768333734645649238","url":"https://x.com/user_4631fa03/status/2768333734645649238","created_at":"2026-04-18T07:08:24.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Joule haze haze core unity nexus haze kraft. glyph valve node vortex wave bloom delta wave. solar grid kraft index query ratio kite jade. zonal trace kraft micro prism flux jade nexus.","lang":"en","metrics":{"likes":154,"retweets":5,"replies":25,"views":11206,"bookmarks":74,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/6f2ae143d008.jpg:orig","alt_text":null}],"urls":[{"display":"example.com/san_efd1caf28d","expanded":"https://example.com/sanitized/2c91f97644","shortened":"https://t.co/SAN_721a4a73"}],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"2768333734645649238","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1084900394861970120","url":"https://x.com/user_4631fa03/status/1084900394861970120","created_at":"2026-04-18T07:30:36.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Echo logic kraft prism apex ultra kraft jade. nexus vortex bloom zonal drift bloom node foxtrot. node mesa gamma haze orbit logic apex valve. relay gamma jade helix terra vortex query ultra.","lang":"en","metrics":{"likes":10,"retweets":0,"replies":5,"views":3442,"bookmarks":5,"quotes":0},"media":[],"urls":[{"display":"example.com/san_f39d162a50","expanded":"https://example.com/sanitized/981eee8a2a","shortened":"https://t.co/SAN_086d32a6"}],"hashtags":[],"mentions":[],"in_reply_to":"2768333734645649238","quoted_tweet_id":"2469075055515812845","conversation_id":"2768333734645649238","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"8745401311155908942","url":"https://x.com/user_4631fa03/status/8745401311155908942","created_at":"2026-04-18T10:20:40.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Prism echo echo kraft wave apex kite zonal. solar sigma beta delta pulse foxtrot quartz vortex. grid core ionic gamma unity solar zonal quartz. prism kraft yield flux query ratio sigma prism.","lang":"en","metrics":{"likes":17,"retweets":1,"replies":10,"views":1336,"bookmarks":13,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/1781a12effe8.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"8745401311155908942","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1375722608427664103","url":"https://x.com/user_4631fa03/status/1375722608427664103","created_at":"2026-04-18T10:20:41.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Orbit flux node index orbit micro unity nexus. trace orbit kraft zonal valve joule unity zonal. ratio kite joule trace logic lunar ultra logic. xenon trace foxtrot valve nexus grid bloom glyph.","lang":"en","metrics":{"likes":4,"retweets":1,"replies":2,"views":1251,"bookmarks":3,"quotes":0},"media":[],"urls":[{"display":"example.com/san_f39d162a50","expanded":"https://example.com/sanitized/a84c690255","shortened":"https://t.co/SAN_924572c7"}],"hashtags":[],"mentions":[],"in_reply_to":"8745401311155908942","quoted_tweet_id":"7829999841354085292","conversation_id":"8745401311155908942","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"8835360242311239849","url":"https://x.com/user_4631fa03/status/8835360242311239849","created_at":"2026-04-18T10:20:41.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Solar delta flux query widen widen logic node. ultra prism apex glyph joule solar node yield. ionic ratio index quartz relay sigma optic orbit. jade pulse gamma pulse ratio trace yield node.","lang":"en","metrics":{"likes":5,"retweets":0,"replies":3,"views":705,"bookmarks":4,"quotes":0},"media":[],"urls":[{"display":"example.com/san_da2570e1c3","expanded":"https://example.com/sanitized/86473004e4","shortened":"https://t.co/SAN_07e57cb8"}],"hashtags":[],"mentions":[],"in_reply_to":"1375722608427664103","quoted_tweet_id":null,"conversation_id":"8745401311155908942","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"4082130165462299434","url":"https://x.com/user_b924e844/status/4082130165462299434","created_at":"2026-04-18T00:10:58.000Z","author":{"id":"317476795064","username":"user_b924e844","display_name":"User bd0aa5","verified":false,"is_blue_verified":true,"follower_count":238295998},"text":"Kite core pulse gamma sigma nexus.","lang":"en","metrics":{"likes":30360,"retweets":1970,"replies":2508,"views":11414572,"bookmarks":2327,"quotes":144},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/a6a0a5bbd04e.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"4082130165462299434","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"7843560721270987274","url":"https://x.com/user_177791b9/status/7843560721270987274","created_at":"2026-04-18T03:18:09.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":57993},"text":"Jade jade echo gamma apex ratio apex terra. kite orbit zonal logic bloom pulse jade drift. bloom sigma apex pulse ionic haze mesa lunar. sigma glyph kite trace flux ratio grid haze.","lang":"en","metrics":{"likes":42,"retweets":0,"replies":9,"views":4132,"bookmarks":7,"quotes":0},"media":[],"urls":[],"hashtags":[],"mentions":[{"id":"317476795064","username":"user_b924e844"}],"in_reply_to":"4082130165462299434","quoted_tweet_id":null,"conversation_id":"4082130165462299434","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"7883792353538994169","url":"https://x.com/user_dcc6212f/status/7883792353538994169","created_at":"2026-04-17T17:54:16.000Z","author":{"id":"189535729767","username":"user_dcc6212f","display_name":"User 8f90c2","verified":false,"is_blue_verified":true,"follower_count":47071},"text":"Echo kraft glyph helix yield echo mesa kraft. pulse micro micro delta unity micro core flux. orbit delta beta ionic glyph bloom ultra bloom. node quartz logic optic prism valve foxtrot optic.","lang":"en","metrics":{"likes":4197,"retweets":178,"replies":89,"views":953458,"bookmarks":1196,"quotes":13},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/e27f62c92aa1.mp4","alt_text":null,"duration_ms":153878}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7883792353538994169","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"4525395474206381929","url":"https://x.com/user_4631fa03/status/4525395474206381929","created_at":"2026-04-18T05:20:56.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Gamma lunar valve kite trace vortex echo quartz. glyph nexus sigma ultra node ionic mesa micro.","lang":"en","metrics":{"likes":154,"retweets":1,"replies":36,"views":20266,"bookmarks":20,"quotes":2},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"2365697391495111868","conversation_id":"4525395474206381929","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1048209185645476420","url":"https://x.com/user_87e0a70d/status/1048209185645476420","created_at":"2026-04-18T03:52:15.000Z","author":{"id":"103858680551","username":"user_87e0a70d","display_name":"User e83e9a","verified":false,"is_blue_verified":true,"follower_count":22173},"text":"Vortex optic quartz terra glyph sigma mesa joule. kraft lunar gamma ultra mesa unity widen wave. beta relay echo grid kraft haze echo ultra. sigma vortex foxtrot yield micro delta quartz delta.","lang":"en","metrics":{"likes":264,"retweets":34,"replies":43,"views":14976,"bookmarks":107,"quotes":13},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/e3c68b576ec5.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1048209185645476420","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1767626127106045159","url":"https://x.com/user_e57091aa/status/1767626127106045159","created_at":"2026-04-16T21:18:05.000Z","author":{"id":"100508716875","username":"user_e57091aa","display_name":"User c7b576","verified":false,"is_blue_verified":true,"follower_count":485174},"text":"Kite pulse sigma xenon terra ionic delta unity. valve quartz vortex trace kraft helix joule beta. query zonal.","lang":"en","metrics":{"likes":373,"retweets":19,"replies":23,"views":111146,"bookmarks":59,"quotes":7},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"2236680614779141805","conversation_id":"1767626127106045159","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"9651080193327180310","url":"https://x.com/user_85536508/status/9651080193327180310","created_at":"2026-04-16T19:57:55.000Z","author":{"id":"209260777194","username":"user_85536508","display_name":"User c624c9","verified":false,"is_blue_verified":true,"follower_count":594545},"text":"Ionic flux jade haze vortex lunar delta index. yield alpha valve core index core echo trace. zonal core bloom zonal yield wave index kite. xenon relay logic echo ratio logic kraft xenon.","lang":"en","metrics":{"likes":0,"retweets":671,"replies":0,"views":10,"bookmarks":0,"quotes":0},"media":[],"urls":[],"hashtags":[],"mentions":[{"id":"797042816569","username":"user_ebe970cb"}],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"9651080193327180310","is_retweet":true,"retweeted_tweet_id":"2409614990081141699","is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1146959566041199923","url":"https://x.com/user_bc30c2e1/status/1146959566041199923","created_at":"2026-04-18T04:11:35.000Z","author":{"id":"101740612170","username":"user_bc30c2e1","display_name":"User a20888","verified":false,"is_blue_verified":true,"follower_count":26987},"text":"Nexus terra prism beta foxtrot bloom sigma orbit. kraft foxtrot echo vortex orbit mesa prism xenon. beta echo alpha alpha joule yield kraft kraft. node grid gamma grid ratio unity zonal vortex.","lang":"en","metrics":{"likes":71,"retweets":11,"replies":5,"views":6733,"bookmarks":71,"quotes":3},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/3ba649a3340a.mp4","alt_text":null,"duration_ms":132550}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1146959566041199923","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"9852928959577338607","url":"https://x.com/user_ff0a2240/status/9852928959577338607","created_at":"2026-04-17T19:31:23.000Z","author":{"id":"192103818576","username":"user_ff0a2240","display_name":"User 1fe083","verified":false,"is_blue_verified":true,"follower_count":231796},"text":"Kraft grid index wave relay ratio unity helix. pulse beta core solar kite apex ultra node. wave joule.","lang":"en","metrics":{"likes":3134,"retweets":65,"replies":34,"views":1226019,"bookmarks":899,"quotes":2},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/57fefb90cbe8.mp4","alt_text":null,"duration_ms":20093}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"8296520841586648572","conversation_id":"9852928959577338607","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"3468601991063024541","url":"https://x.com/user_c3e2c4c6/status/3468601991063024541","created_at":"2026-04-17T08:02:50.000Z","author":{"id":"354032513890","username":"user_c3e2c4c6","display_name":"User 3e3e19","verified":false,"is_blue_verified":true,"follower_count":13096},"text":"Grid widen ionic ionic echo core foxtrot orbit. flux kite yield wave prism ionic micro index. core haze helix yield micro grid vortex query. core glyph quartz drift mesa foxtrot quartz node.","lang":"en","metrics":{"likes":459,"retweets":127,"replies":8,"views":22775,"bookmarks":413,"quotes":1},"media":[{"type":"animated_gif","url":"https://video.twimg.com/sanitized/967d2d6865e0.mp4","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"3468601991063024541","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"6198977326281265147","url":"https://x.com/user_9c7ca1b3/status/6198977326281265147","created_at":"2026-04-18T06:58:02.000Z","author":{"id":"523051486301","username":"user_9c7ca1b3","display_name":"User 74f1a5","verified":false,"is_blue_verified":true,"follower_count":2274},"text":"Ultra kite unity apex prism pulse sigma echo. delta widen ratio ionic index core beta lunar. vortex prism orbit widen node joule delta logic. gamma gamma kraft solar glyph beta xenon ionic.","lang":"en","metrics":{"likes":446,"retweets":54,"replies":14,"views":15839,"bookmarks":50,"quotes":14},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/7b81270ddd67.mp4","alt_text":null,"duration_ms":57662}],"urls":[],"hashtags":["FSDEurope"],"mentions":[{"id":"179057625331","username":"user_3748af22"},{"id":"965407876276","username":"user_0b64533b"}],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"6198977326281265147","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"8744248205180156214","url":"https://x.com/user_b63b44cb/status/8744248205180156214","created_at":"2026-04-17T22:06:58.000Z","author":{"id":"496882092350","username":"user_b63b44cb","display_name":"User 4f3bb4","verified":false,"is_blue_verified":true,"follower_count":16371},"text":"Joule flux drift logic gamma alpha orbit nexus. query foxtrot quartz xenon alpha haze valve logic. gamma echo haze beta zonal index unity flux. nexus ultra yield relay.","lang":"en","metrics":{"likes":619,"retweets":18,"replies":77,"views":76388,"bookmarks":28,"quotes":1},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/35242b0dcf82.mp4","alt_text":null,"duration_ms":27416}],"urls":[],"hashtags":[],"mentions":[{"id":"317476795064","username":"user_b924e844"}],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"8744248205180156214","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"3295315316533426843","url":"https://x.com/user_a3b63c0c/status/3295315316533426843","created_at":"2026-04-18T08:23:15.000Z","author":{"id":"700858783701","username":"user_a3b63c0c","display_name":"User 3d7f29","verified":false,"is_blue_verified":true,"follower_count":106534},"text":"Unity lunar xenon beta beta bloom optic micro. apex index jade zonal foxtrot flux glyph jade. node quartz valve beta logic orbit ionic terra. beta bloom yield valve ionic zonal sigma solar.","lang":"en","metrics":{"likes":89,"retweets":25,"replies":8,"views":5789,"bookmarks":21,"quotes":3},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/fa3b15f9df65.mp4","alt_text":null,"duration_ms":40783}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"3295315316533426843","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1053712734066915642","url":"https://x.com/user_4ee882b3/status/1053712734066915642","created_at":"2026-04-17T15:35:06.000Z","author":{"id":"548044251674","username":"user_4ee882b3","display_name":"User c2fabb","verified":false,"is_blue_verified":false,"follower_count":959587},"text":"Sigma kraft lunar pulse beta micro foxtrot prism. jade kite node xenon ultra vortex glyph delta. logic logic.","lang":"en","metrics":{"likes":187,"retweets":26,"replies":7,"views":22885,"bookmarks":38,"quotes":7},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/ccdaf4ea28e1.jpg:orig","alt_text":null}],"urls":[{"display":"example.com/san_c33c0e81f3","expanded":"https://example.com/sanitized/32cac8908c","shortened":"https://t.co/SAN_f8593acf"}],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1053712734066915642","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1092838877727611300","url":"https://x.com/user_76376d61/status/1092838877727611300","created_at":"2026-04-17T15:32:52.000Z","author":{"id":"880434958610","username":"user_76376d61","display_name":"User faaa33","verified":false,"is_blue_verified":true,"follower_count":2496327},"text":"Apex unity gamma helix ionic terra zonal optic. optic nexus sigma index micro alpha lunar alpha. solar kraft yield prism index delta orbit pulse. echo valve sigma ionic joule yield prism delta.","lang":"en","metrics":{"likes":670,"retweets":55,"replies":50,"views":96421,"bookmarks":236,"quotes":5},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"8445923126500780810","conversation_id":"1092838877727611300","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1087819610196554182","url":"https://x.com/user_780df61e/status/1087819610196554182","created_at":"2026-04-17T17:36:14.000Z","author":{"id":"241051283392","username":"user_780df61e","display_name":"User f91397","verified":false,"is_blue_verified":true,"follower_count":145172},"text":"Lunar foxtrot kraft apex relay gamma kite ultra. query bloom ionic apex kraft quartz ratio valve. haze zonal vortex beta pulse widen gamma helix. alpha haze ratio lunar relay pulse sigma terra.","lang":"en","metrics":{"likes":1011,"retweets":134,"replies":43,"views":155755,"bookmarks":395,"quotes":53},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/5b3254df60eb.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[{"id":"914249346434","username":"user_090c461a"}],"in_reply_to":null,"quoted_tweet_id":"3619715370568280138","conversation_id":"1087819610196554182","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"7763791146638902904","url":"https://x.com/user_9ab3c02f/status/7763791146638902904","created_at":"2026-04-17T13:19:17.000Z","author":{"id":"156794147791","username":"user_9ab3c02f","display_name":"User 745ae9","verified":false,"is_blue_verified":false,"follower_count":18453},"text":"Echo mesa pulse trace flux pulse vortex index. jade nexus vortex echo query solar flux terra. node mesa lunar bloom trace prism terra beta. prism ionic joule solar node xenon solar delta.","lang":"en","metrics":{"likes":163,"retweets":41,"replies":12,"views":14201,"bookmarks":117,"quotes":6},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/8316b1d4eedb.mp4","alt_text":null,"duration_ms":80079}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7763791146638902904","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1064790687975197401","url":"https://x.com/user_27908ca4/status/1064790687975197401","created_at":"2026-04-17T17:13:22.000Z","author":{"id":"600055494068","username":"user_27908ca4","display_name":"User 00314a","verified":false,"is_blue_verified":true,"follower_count":16702},"text":"Valve jade trace echo relay unity mesa kraft. quartz haze vortex echo drift valve gamma trace. bloom core grid optic orbit lunar yield ratio. haze quartz ratio pulse delta vortex flux jade.","lang":"en","metrics":{"likes":16369,"retweets":1272,"replies":235,"views":4452872,"bookmarks":1728,"quotes":46},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/6bc1142090c8.mp4","alt_text":null,"duration_ms":25400}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1064790687975197401","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"7847249670910741773","url":"https://x.com/user_fca1c00e/status/7847249670910741773","created_at":"2026-04-16T19:51:49.000Z","author":{"id":"597528245006","username":"user_fca1c00e","display_name":"User 2b08fa","verified":false,"is_blue_verified":true,"follower_count":73248},"text":"Orbit helix ionic widen bloom alpha node flux. alpha kite xenon wave logic micro.","lang":"en","metrics":{"likes":4069,"retweets":425,"replies":108,"views":261887,"bookmarks":652,"quotes":62},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/1b535f79d26e.mp4","alt_text":null,"duration_ms":405845}],"urls":[],"hashtags":[],"mentions":[{"id":"846116008513","username":"user_8706cf85"}],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7847249670910741773","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1139811845225147422","url":"https://x.com/user_9defae5d/status/1139811845225147422","created_at":"2026-04-17T21:00:28.000Z","author":{"id":"752846022597","username":"user_9defae5d","display_name":"User 49a70d","verified":false,"is_blue_verified":true,"follower_count":4026082},"text":"Xenon grid lunar terra micro delta ultra query. query ultra.","lang":"en","metrics":{"likes":11998,"retweets":348,"replies":294,"views":2830541,"bookmarks":1885,"quotes":24},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/32a0858c7ba1.mp4","alt_text":null,"duration_ms":48200}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1139811845225147422","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"8838033385258358422","url":"https://x.com/user_143e26b0/status/8838033385258358422","created_at":"2026-04-17T06:45:00.000Z","author":{"id":"104209884508","username":"user_143e26b0","display_name":"User a3ae97","verified":false,"is_blue_verified":true,"follower_count":231412},"text":"Echo joule gamma delta mesa alpha trace trace. mesa sigma core ionic vortex quartz solar drift. kite echo index foxtrot node bloom prism.","lang":"en","metrics":{"likes":7418,"retweets":632,"replies":80,"views":563486,"bookmarks":13813,"quotes":14},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/e058e69774a2.mp4","alt_text":null,"duration_ms":989900}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"8838033385258358422","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"2324730829770412414","url":"https://x.com/user_b924e844/status/2324730829770412414","created_at":"2026-04-17T22:42:45.000Z","author":{"id":"317476795064","username":"user_b924e844","display_name":"User bd0aa5","verified":false,"is_blue_verified":true,"follower_count":238295998},"text":"Unity haze nexus grid bloom xenon drift nexus.","lang":"en","metrics":{"likes":77166,"retweets":6121,"replies":5487,"views":23451769,"bookmarks":1384,"quotes":435},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"3456067991854997259","conversation_id":"2324730829770412414","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1522641576348854435","url":"https://x.com/user_d687596a/status/1522641576348854435","created_at":"2026-04-17T07:31:01.000Z","author":{"id":"428921884383","username":"user_d687596a","display_name":"User d02c1d","verified":false,"is_blue_verified":true,"follower_count":5371},"text":"Foxtrot nexus mesa ratio pulse valve zonal ionic. optic widen lunar beta widen terra logic quartz. node drift gamma bloom trace alpha zonal pulse. haze prism delta kraft wave helix flux kite.","lang":"en","metrics":{"likes":6615,"retweets":216,"replies":277,"views":568881,"bookmarks":573,"quotes":27},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/59f952440273.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1522641576348854435","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"6263674550993747526","url":"https://x.com/user_31de149a/status/6263674550993747526","created_at":"2026-04-17T23:00:01.000Z","author":{"id":"109193805778","username":"user_31de149a","display_name":"User ff846b","verified":false,"is_blue_verified":true,"follower_count":2252},"text":"Zonal valve orbit logic joule prism solar joule. kite logic.","lang":"en","metrics":{"likes":729,"retweets":273,"replies":329,"views":130052,"bookmarks":104,"quotes":1},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/a4e453c51741.mp4","alt_text":null,"duration_ms":25147}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"6263674550993747526","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1125293230092406943","url":"https://x.com/user_46a851d0/status/1125293230092406943","created_at":"2026-04-17T05:12:42.000Z","author":{"id":"114807699959","username":"user_46a851d0","display_name":"User 17e593","verified":false,"is_blue_verified":true,"follower_count":32983},"text":"Widen sigma haze delta beta orbit haze kraft. delta lunar orbit joule trace vortex alpha optic. flux kite.","lang":"en","metrics":{"likes":2744,"retweets":321,"replies":188,"views":369291,"bookmarks":157,"quotes":92},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/d1b19d45bad3.mp4","alt_text":null,"duration_ms":45289}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1125293230092406943","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"3869031856844220752","url":"https://x.com/user_ae6fa065/status/3869031856844220752","created_at":"2026-04-16T20:17:55.000Z","author":{"id":"370687453523","username":"user_ae6fa065","display_name":"User 1bc521","verified":false,"is_blue_verified":true,"follower_count":34763},"text":"Wave grid alpha beta trace vortex trace micro. mesa drift gamma helix micro valve unity solar. flux echo optic haze relay apex prism index. solar yield core ionic prism terra vortex wave.","lang":"en","metrics":{"likes":4252,"retweets":839,"replies":365,"views":1561389,"bookmarks":510,"quotes":375},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/c4da05b6d821.mp4","alt_text":null,"duration_ms":9280}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"3869031856844220752","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"7057157674853009481","url":"https://x.com/user_19d30911/status/7057157674853009481","created_at":"2026-04-17T15:14:48.000Z","author":{"id":"248952525840","username":"user_19d30911","display_name":"User 0e779b","verified":false,"is_blue_verified":true,"follower_count":25143},"text":"Joule joule node index gamma pulse trace kite. terra nexus drift mesa.","lang":"en","metrics":{"likes":127,"retweets":12,"replies":10,"views":4565,"bookmarks":124,"quotes":2},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/7350be6d01f4.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7057157674853009481","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1090250687255528987","url":"https://x.com/user_427de521/status/1090250687255528987","created_at":"2026-04-17T14:07:06.000Z","author":{"id":"810929584413","username":"user_427de521","display_name":"User ec9b5e","verified":false,"is_blue_verified":true,"follower_count":29105},"text":"Unity widen prism relay apex widen delta logic. trace kraft kite orbit foxtrot ionic valve orbit. grid zonal relay kite drift jade haze drift. ultra sigma flux kite jade core micro terra.","lang":"en","metrics":{"likes":220,"retweets":11,"replies":10,"views":68536,"bookmarks":156,"quotes":5},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/8a5a8b58b5f0.mp4","alt_text":null,"duration_ms":55660}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"1041554574205013017","conversation_id":"1090250687255528987","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"5810820280941882540","url":"https://x.com/user_41c91310/status/5810820280941882540","created_at":"2026-04-17T21:45:31.000Z","author":{"id":"457108228717","username":"user_41c91310","display_name":"User 65905d","verified":false,"is_blue_verified":true,"follower_count":87660},"text":"Gamma sigma ratio sigma delta trace grid index. optic logic foxtrot kite echo kite ratio prism. zonal logic index kite.","lang":"en","metrics":{"likes":2034,"retweets":117,"replies":29,"views":109238,"bookmarks":92,"quotes":5},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/641459a6d307.mp4","alt_text":null,"duration_ms":11742}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"5810820280941882540","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"7172332670700521790","url":"https://x.com/user_c3eaee62/status/7172332670700521790","created_at":"2026-04-17T17:45:00.000Z","author":{"id":"962206826969","username":"user_c3eaee62","display_name":"User 4f40ff","verified":false,"is_blue_verified":true,"follower_count":21972},"text":"Apex wave unity micro valve core lunar glyph. joule solar jade xenon terra optic alpha index. vortex echo delta logic quartz ultra helix ultra. ionic ratio wave core bloom valve node echo.","lang":"en","metrics":{"likes":454,"retweets":24,"replies":6,"views":33541,"bookmarks":58,"quotes":3},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/0ef85c2b699a.mp4","alt_text":null,"duration_ms":24666}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7172332670700521790","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"1003861653946784235","url":"https://x.com/user_eca18d38/status/1003861653946784235","created_at":"2026-04-17T14:48:25.000Z","author":{"id":"985789943932","username":"user_eca18d38","display_name":"User 14253b","verified":false,"is_blue_verified":true,"follower_count":1251},"text":"Echo trace delta kite ultra wave flux grid. apex apex haze.","lang":"en","metrics":{"likes":1064,"retweets":25,"replies":39,"views":99781,"bookmarks":915,"quotes":2},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/7f5787dd213e.mp4","alt_text":null,"duration_ms":17917}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1003861653946784235","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"5222422539262490066","url":"https://x.com/user_df56e8e9/status/5222422539262490066","created_at":"2026-04-17T23:02:06.000Z","author":{"id":"759883384437","username":"user_df56e8e9","display_name":"User 130332","verified":false,"is_blue_verified":true,"follower_count":1977},"text":"Echo micro quartz logic trace ratio gamma query. widen ionic relay drift gamma beta haze logic. core ultra core ultra apex gamma lunar lunar. prism quartz lunar node delta echo yield ultra.","lang":"es","metrics":{"likes":2133,"retweets":90,"replies":63,"views":627247,"bookmarks":385,"quotes":4},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/1012044c2e02.mp4","alt_text":null,"duration_ms":44746}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"5222422539262490066","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"9727384604865208385","url":"https://x.com/user_4a9f5ff6/status/9727384604865208385","created_at":"2026-04-17T15:13:27.000Z","author":{"id":"793385184366","username":"user_4a9f5ff6","display_name":"User fd5225","verified":false,"is_blue_verified":true,"follower_count":10291},"text":"Delta core foxtrot echo zonal query bloom jade. index ratio apex logic glyph beta sigma sigma. optic helix query valve sigma vortex node solar.","lang":"es","metrics":{"likes":39868,"retweets":409,"replies":9379,"views":28413190,"bookmarks":5936,"quotes":757},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/371f237ed3be.jpg:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/b82dafe77b46.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"9727384604865208385","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline","captured_at":"2025-01-01T00:00:00.000Z"} +{"id":"2768333734645649238","url":"https://x.com/user_4631fa03/status/2768333734645649238","created_at":"2026-04-18T07:08:24.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Joule haze haze core unity nexus haze kraft. glyph valve node vortex wave bloom delta wave. solar grid kraft index query ratio kite jade. zonal trace kraft micro prism flux jade nexus.","lang":"en","metrics":{"likes":154,"retweets":5,"replies":25,"views":11206,"bookmarks":74,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/6f2ae143d008.jpg:orig","alt_text":null}],"urls":[{"display":"example.com/san_efd1caf28d","expanded":"https://example.com/sanitized/2c91f97644","shortened":"https://t.co/SAN_721a4a73"}],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"2768333734645649238","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1084900394861970120","url":"https://x.com/user_4631fa03/status/1084900394861970120","created_at":"2026-04-18T07:30:36.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Echo logic kraft prism apex ultra kraft jade. nexus vortex bloom zonal drift bloom node foxtrot. node mesa gamma haze orbit logic apex valve. relay gamma jade helix terra vortex query ultra.","lang":"en","metrics":{"likes":10,"retweets":0,"replies":5,"views":3442,"bookmarks":5,"quotes":0},"media":[],"urls":[{"display":"example.com/san_f39d162a50","expanded":"https://example.com/sanitized/981eee8a2a","shortened":"https://t.co/SAN_086d32a6"}],"hashtags":[],"mentions":[],"in_reply_to":"2768333734645649238","quoted_tweet_id":"2469075055515812845","conversation_id":"2768333734645649238","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"8745401311155908942","url":"https://x.com/user_4631fa03/status/8745401311155908942","created_at":"2026-04-18T10:20:40.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Prism echo echo kraft wave apex kite zonal. solar sigma beta delta pulse foxtrot quartz vortex. grid core ionic gamma unity solar zonal quartz. prism kraft yield flux query ratio sigma prism.","lang":"en","metrics":{"likes":17,"retweets":1,"replies":10,"views":1336,"bookmarks":13,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/1781a12effe8.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"8745401311155908942","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1375722608427664103","url":"https://x.com/user_4631fa03/status/1375722608427664103","created_at":"2026-04-18T10:20:41.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Orbit flux node index orbit micro unity nexus. trace orbit kraft zonal valve joule unity zonal. ratio kite joule trace logic lunar ultra logic. xenon trace foxtrot valve nexus grid bloom glyph.","lang":"en","metrics":{"likes":4,"retweets":1,"replies":2,"views":1251,"bookmarks":3,"quotes":0},"media":[],"urls":[{"display":"example.com/san_f39d162a50","expanded":"https://example.com/sanitized/a84c690255","shortened":"https://t.co/SAN_924572c7"}],"hashtags":[],"mentions":[],"in_reply_to":"8745401311155908942","quoted_tweet_id":"7829999841354085292","conversation_id":"8745401311155908942","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"7829999841354085292","url":"https://x.com/user_4631fa03/status/7829999841354085292","created_at":"2026-04-18T08:59:47.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Quartz gamma gamma quartz delta grid delta kite. helix nexus terra drift gamma alpha mesa lunar. query alpha widen relay pulse micro quartz solar. foxtrot wave relay kite mesa grid logic alpha.","lang":"en","metrics":{"likes":96,"retweets":14,"replies":23,"views":11362,"bookmarks":71,"quotes":2},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/fb76a74cd106.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7829999841354085292","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"8835360242311239849","url":"https://x.com/user_4631fa03/status/8835360242311239849","created_at":"2026-04-18T10:20:41.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Solar delta flux query widen widen logic node. ultra prism apex glyph joule solar node yield. ionic ratio index quartz relay sigma optic orbit. jade pulse gamma pulse ratio trace yield node.","lang":"en","metrics":{"likes":5,"retweets":0,"replies":3,"views":705,"bookmarks":4,"quotes":0},"media":[],"urls":[{"display":"example.com/san_da2570e1c3","expanded":"https://example.com/sanitized/86473004e4","shortened":"https://t.co/SAN_07e57cb8"}],"hashtags":[],"mentions":[],"in_reply_to":"1375722608427664103","quoted_tweet_id":null,"conversation_id":"8745401311155908942","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"4082130165462299434","url":"https://x.com/user_b924e844/status/4082130165462299434","created_at":"2026-04-18T00:10:58.000Z","author":{"id":"317476795064","username":"user_b924e844","display_name":"User bd0aa5","verified":false,"is_blue_verified":true,"follower_count":238295998},"text":"Kite core pulse gamma sigma nexus.","lang":"en","metrics":{"likes":30360,"retweets":1970,"replies":2508,"views":11414572,"bookmarks":2327,"quotes":144},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/a6a0a5bbd04e.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"4082130165462299434","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"7843560721270987274","url":"https://x.com/user_177791b9/status/7843560721270987274","created_at":"2026-04-18T03:18:09.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":57993},"text":"Jade jade echo gamma apex ratio apex terra. kite orbit zonal logic bloom pulse jade drift. bloom sigma apex pulse ionic haze mesa lunar. sigma glyph kite trace flux ratio grid haze.","lang":"en","metrics":{"likes":42,"retweets":0,"replies":9,"views":4132,"bookmarks":7,"quotes":0},"media":[],"urls":[],"hashtags":[],"mentions":[{"id":"317476795064","username":"user_b924e844"}],"in_reply_to":"4082130165462299434","quoted_tweet_id":null,"conversation_id":"4082130165462299434","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"7883792353538994169","url":"https://x.com/user_dcc6212f/status/7883792353538994169","created_at":"2026-04-17T17:54:16.000Z","author":{"id":"189535729767","username":"user_dcc6212f","display_name":"User 8f90c2","verified":false,"is_blue_verified":true,"follower_count":47071},"text":"Echo kraft glyph helix yield echo mesa kraft. pulse micro micro delta unity micro core flux. orbit delta beta ionic glyph bloom ultra bloom. node quartz logic optic prism valve foxtrot optic.","lang":"en","metrics":{"likes":4197,"retweets":178,"replies":89,"views":953458,"bookmarks":1196,"quotes":13},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/e27f62c92aa1.mp4","alt_text":null,"duration_ms":153878}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7883792353538994169","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"4525395474206381929","url":"https://x.com/user_4631fa03/status/4525395474206381929","created_at":"2026-04-18T05:20:56.000Z","author":{"id":"370991728694","username":"user_4631fa03","display_name":"User fe8a05","verified":false,"is_blue_verified":true,"follower_count":754198},"text":"Gamma lunar valve kite trace vortex echo quartz. glyph nexus sigma ultra node ionic mesa micro.","lang":"en","metrics":{"likes":154,"retweets":1,"replies":36,"views":20266,"bookmarks":20,"quotes":2},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"2365697391495111868","conversation_id":"4525395474206381929","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"2365697391495111868","url":"https://x.com/user_54b76fcc/status/2365697391495111868","created_at":"2026-04-18T05:11:19.000Z","author":{"id":"797741283176","username":"user_54b76fcc","display_name":"User 43b424","verified":false,"is_blue_verified":false,"follower_count":171},"text":"Orbit unity ionic gamma valve kite orbit prism. vortex optic glyph haze xenon delta bloom drift. ultra delta micro.","lang":"en","metrics":{"likes":0,"retweets":0,"replies":2,"views":21209,"bookmarks":0,"quotes":1},"media":[],"urls":[],"hashtags":[],"mentions":[{"id":"370991728694","username":"user_4631fa03"},{"id":"518206027438","username":"user_b8d59361"}],"in_reply_to":"4203858264205422490","quoted_tweet_id":null,"conversation_id":"5210406254797030936","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1048209185645476420","url":"https://x.com/user_87e0a70d/status/1048209185645476420","created_at":"2026-04-18T03:52:15.000Z","author":{"id":"103858680551","username":"user_87e0a70d","display_name":"User e83e9a","verified":false,"is_blue_verified":true,"follower_count":22173},"text":"Vortex optic quartz terra glyph sigma mesa joule. kraft lunar gamma ultra mesa unity widen wave. beta relay echo grid kraft haze echo ultra. sigma vortex foxtrot yield micro delta quartz delta.","lang":"en","metrics":{"likes":264,"retweets":34,"replies":43,"views":14976,"bookmarks":107,"quotes":13},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/e3c68b576ec5.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1048209185645476420","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1767626127106045159","url":"https://x.com/user_e57091aa/status/1767626127106045159","created_at":"2026-04-16T21:18:05.000Z","author":{"id":"100508716875","username":"user_e57091aa","display_name":"User c7b576","verified":false,"is_blue_verified":true,"follower_count":485174},"text":"Kite pulse sigma xenon terra ionic delta unity. valve quartz vortex trace kraft helix joule beta. query zonal.","lang":"en","metrics":{"likes":373,"retweets":19,"replies":23,"views":111146,"bookmarks":59,"quotes":7},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"2236680614779141805","conversation_id":"1767626127106045159","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"2236680614779141805","url":"https://x.com/user_d70966be/status/2236680614779141805","created_at":"2026-04-15T21:06:25.000Z","author":{"id":"138736846558","username":"user_d70966be","display_name":"User 949b23","verified":false,"is_blue_verified":false,"follower_count":8576},"text":"Ionic prism apex optic solar drift zonal ionic. vortex orbit orbit glyph prism index optic mesa. vortex quartz zonal echo echo helix zonal index. jade pulse jade jade query unity ionic ratio.","lang":"en","metrics":{"likes":289,"retweets":36,"replies":16,"views":89210,"bookmarks":58,"quotes":5},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/739c8985241e.jpg:orig","alt_text":null}],"urls":[{"display":"example.com/san_500b623496","expanded":"https://example.com/sanitized/a5a247c344","shortened":"https://t.co/SAN_3dc5f6d2"}],"hashtags":[],"mentions":[{"id":"410279411712","username":"user_264367d4"}],"in_reply_to":null,"quoted_tweet_id":"2949823258772645662","conversation_id":"2236680614779141805","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"9651080193327180310","url":"https://x.com/user_85536508/status/9651080193327180310","created_at":"2026-04-16T19:57:55.000Z","author":{"id":"209260777194","username":"user_85536508","display_name":"User c624c9","verified":false,"is_blue_verified":true,"follower_count":594545},"text":"Ionic flux jade haze vortex lunar delta index. yield alpha valve core index core echo trace. zonal core bloom zonal yield wave index kite. xenon relay logic echo ratio logic kraft xenon.","lang":"en","metrics":{"likes":0,"retweets":671,"replies":0,"views":10,"bookmarks":0,"quotes":0},"media":[],"urls":[],"hashtags":[],"mentions":[{"id":"797042816569","username":"user_ebe970cb"}],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"9651080193327180310","is_retweet":true,"retweeted_tweet_id":"2409614990081141699","is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1146959566041199923","url":"https://x.com/user_bc30c2e1/status/1146959566041199923","created_at":"2026-04-18T04:11:35.000Z","author":{"id":"101740612170","username":"user_bc30c2e1","display_name":"User a20888","verified":false,"is_blue_verified":true,"follower_count":26987},"text":"Nexus terra prism beta foxtrot bloom sigma orbit. kraft foxtrot echo vortex orbit mesa prism xenon. beta echo alpha alpha joule yield kraft kraft. node grid gamma grid ratio unity zonal vortex.","lang":"en","metrics":{"likes":71,"retweets":11,"replies":5,"views":6733,"bookmarks":71,"quotes":3},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/3ba649a3340a.mp4","alt_text":null,"duration_ms":132550}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1146959566041199923","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"9852928959577338607","url":"https://x.com/user_ff0a2240/status/9852928959577338607","created_at":"2026-04-17T19:31:23.000Z","author":{"id":"192103818576","username":"user_ff0a2240","display_name":"User 1fe083","verified":false,"is_blue_verified":true,"follower_count":231796},"text":"Kraft grid index wave relay ratio unity helix. pulse beta core solar kite apex ultra node. wave joule.","lang":"en","metrics":{"likes":3134,"retweets":65,"replies":34,"views":1226019,"bookmarks":899,"quotes":2},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/57fefb90cbe8.mp4","alt_text":null,"duration_ms":20093}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"8296520841586648572","conversation_id":"9852928959577338607","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"8296520841586648572","url":"https://x.com/user_f75a7f55/status/8296520841586648572","created_at":"2026-04-16T14:08:07.000Z","author":{"id":"566986560365","username":"user_f75a7f55","display_name":"User 719fb2","verified":false,"is_blue_verified":true,"follower_count":299514},"text":"Widen orbit micro valve.","lang":"en","metrics":{"likes":5997,"retweets":226,"replies":219,"views":1808075,"bookmarks":281,"quotes":40},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"9892606599816188485","conversation_id":"8296520841586648572","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"3468601991063024541","url":"https://x.com/user_c3e2c4c6/status/3468601991063024541","created_at":"2026-04-17T08:02:50.000Z","author":{"id":"354032513890","username":"user_c3e2c4c6","display_name":"User 3e3e19","verified":false,"is_blue_verified":true,"follower_count":13096},"text":"Grid widen ionic ionic echo core foxtrot orbit. flux kite yield wave prism ionic micro index. core haze helix yield micro grid vortex query. core glyph quartz drift mesa foxtrot quartz node.","lang":"en","metrics":{"likes":459,"retweets":127,"replies":8,"views":22775,"bookmarks":413,"quotes":1},"media":[{"type":"animated_gif","url":"https://video.twimg.com/sanitized/967d2d6865e0.mp4","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"3468601991063024541","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"6198977326281265147","url":"https://x.com/user_9c7ca1b3/status/6198977326281265147","created_at":"2026-04-18T06:58:02.000Z","author":{"id":"523051486301","username":"user_9c7ca1b3","display_name":"User 74f1a5","verified":false,"is_blue_verified":true,"follower_count":2274},"text":"Ultra kite unity apex prism pulse sigma echo. delta widen ratio ionic index core beta lunar. vortex prism orbit widen node joule delta logic. gamma gamma kraft solar glyph beta xenon ionic.","lang":"en","metrics":{"likes":446,"retweets":54,"replies":14,"views":15839,"bookmarks":50,"quotes":14},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/7b81270ddd67.mp4","alt_text":null,"duration_ms":57662}],"urls":[],"hashtags":["FSDEurope"],"mentions":[{"id":"179057625331","username":"user_3748af22"},{"id":"965407876276","username":"user_0b64533b"}],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"6198977326281265147","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"8744248205180156214","url":"https://x.com/user_b63b44cb/status/8744248205180156214","created_at":"2026-04-17T22:06:58.000Z","author":{"id":"496882092350","username":"user_b63b44cb","display_name":"User 4f3bb4","verified":false,"is_blue_verified":true,"follower_count":16371},"text":"Joule flux drift logic gamma alpha orbit nexus. query foxtrot quartz xenon alpha haze valve logic. gamma echo haze beta zonal index unity flux. nexus ultra yield relay.","lang":"en","metrics":{"likes":619,"retweets":18,"replies":77,"views":76388,"bookmarks":28,"quotes":1},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/35242b0dcf82.mp4","alt_text":null,"duration_ms":27416}],"urls":[],"hashtags":[],"mentions":[{"id":"317476795064","username":"user_b924e844"}],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"8744248205180156214","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"3295315316533426843","url":"https://x.com/user_a3b63c0c/status/3295315316533426843","created_at":"2026-04-18T08:23:15.000Z","author":{"id":"700858783701","username":"user_a3b63c0c","display_name":"User 3d7f29","verified":false,"is_blue_verified":true,"follower_count":106534},"text":"Unity lunar xenon beta beta bloom optic micro. apex index jade zonal foxtrot flux glyph jade. node quartz valve beta logic orbit ionic terra. beta bloom yield valve ionic zonal sigma solar.","lang":"en","metrics":{"likes":89,"retweets":25,"replies":8,"views":5789,"bookmarks":21,"quotes":3},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/fa3b15f9df65.mp4","alt_text":null,"duration_ms":40783}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"3295315316533426843","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1053712734066915642","url":"https://x.com/user_4ee882b3/status/1053712734066915642","created_at":"2026-04-17T15:35:06.000Z","author":{"id":"548044251674","username":"user_4ee882b3","display_name":"User c2fabb","verified":false,"is_blue_verified":false,"follower_count":959587},"text":"Sigma kraft lunar pulse beta micro foxtrot prism. jade kite node xenon ultra vortex glyph delta. logic logic.","lang":"en","metrics":{"likes":187,"retweets":26,"replies":7,"views":22885,"bookmarks":38,"quotes":7},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/ccdaf4ea28e1.jpg:orig","alt_text":null}],"urls":[{"display":"example.com/san_c33c0e81f3","expanded":"https://example.com/sanitized/32cac8908c","shortened":"https://t.co/SAN_f8593acf"}],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1053712734066915642","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1092838877727611300","url":"https://x.com/user_76376d61/status/1092838877727611300","created_at":"2026-04-17T15:32:52.000Z","author":{"id":"880434958610","username":"user_76376d61","display_name":"User faaa33","verified":false,"is_blue_verified":true,"follower_count":2496327},"text":"Apex unity gamma helix ionic terra zonal optic. optic nexus sigma index micro alpha lunar alpha. solar kraft yield prism index delta orbit pulse. echo valve sigma ionic joule yield prism delta.","lang":"en","metrics":{"likes":670,"retweets":55,"replies":50,"views":96421,"bookmarks":236,"quotes":5},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"8445923126500780810","conversation_id":"1092838877727611300","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"8445923126500780810","url":"https://x.com/user_32fd6cc2/status/8445923126500780810","created_at":"2026-04-16T22:17:43.000Z","author":{"id":"723990033263","username":"user_32fd6cc2","display_name":"User 6bf654","verified":false,"is_blue_verified":true,"follower_count":300316},"text":"Haze prism jade core flux trace orbit wave. node micro zonal logic bloom trace grid solar. relay alpha optic haze mesa kite trace valve. kraft beta lunar orbit flux kite mesa delta.","lang":"en","metrics":{"likes":325,"retweets":38,"replies":18,"views":173800,"bookmarks":141,"quotes":22},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/88b1ce9cf703.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[{"id":"620580012773","username":"user_b88a2264"},{"id":"496710990389","username":"user_2f339796"}],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"8445923126500780810","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1087819610196554182","url":"https://x.com/user_780df61e/status/1087819610196554182","created_at":"2026-04-17T17:36:14.000Z","author":{"id":"241051283392","username":"user_780df61e","display_name":"User f91397","verified":false,"is_blue_verified":true,"follower_count":145172},"text":"Lunar foxtrot kraft apex relay gamma kite ultra. query bloom ionic apex kraft quartz ratio valve. haze zonal vortex beta pulse widen gamma helix. alpha haze ratio lunar relay pulse sigma terra.","lang":"en","metrics":{"likes":1011,"retweets":134,"replies":43,"views":155755,"bookmarks":395,"quotes":53},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/5b3254df60eb.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[{"id":"914249346434","username":"user_090c461a"}],"in_reply_to":null,"quoted_tweet_id":"3619715370568280138","conversation_id":"1087819610196554182","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"3619715370568280138","url":"https://x.com/user_780df61e/status/3619715370568280138","created_at":"2026-04-17T16:28:06.000Z","author":{"id":"241051283392","username":"user_780df61e","display_name":"User f91397","verified":false,"is_blue_verified":true,"follower_count":145172},"text":"Kite helix widen relay vortex ionic bloom xenon. ratio zonal optic micro flux ultra mesa pulse. apex xenon widen wave orbit logic jade valve. orbit orbit valve wave drift gamma echo gamma.","lang":"en","metrics":{"likes":611,"retweets":46,"replies":44,"views":181722,"bookmarks":62,"quotes":21},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/81fec0f45538.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[{"id":"914249346434","username":"user_090c461a"}],"in_reply_to":null,"quoted_tweet_id":"1392040302113823531","conversation_id":"3619715370568280138","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"7763791146638902904","url":"https://x.com/user_9ab3c02f/status/7763791146638902904","created_at":"2026-04-17T13:19:17.000Z","author":{"id":"156794147791","username":"user_9ab3c02f","display_name":"User 745ae9","verified":false,"is_blue_verified":false,"follower_count":18453},"text":"Echo mesa pulse trace flux pulse vortex index. jade nexus vortex echo query solar flux terra. node mesa lunar bloom trace prism terra beta. prism ionic joule solar node xenon solar delta.","lang":"en","metrics":{"likes":163,"retweets":41,"replies":12,"views":14201,"bookmarks":117,"quotes":6},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/8316b1d4eedb.mp4","alt_text":null,"duration_ms":80079}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7763791146638902904","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1064790687975197401","url":"https://x.com/user_27908ca4/status/1064790687975197401","created_at":"2026-04-17T17:13:22.000Z","author":{"id":"600055494068","username":"user_27908ca4","display_name":"User 00314a","verified":false,"is_blue_verified":true,"follower_count":16702},"text":"Valve jade trace echo relay unity mesa kraft. quartz haze vortex echo drift valve gamma trace. bloom core grid optic orbit lunar yield ratio. haze quartz ratio pulse delta vortex flux jade.","lang":"en","metrics":{"likes":16369,"retweets":1272,"replies":235,"views":4452872,"bookmarks":1728,"quotes":46},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/6bc1142090c8.mp4","alt_text":null,"duration_ms":25400}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1064790687975197401","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"7847249670910741773","url":"https://x.com/user_fca1c00e/status/7847249670910741773","created_at":"2026-04-16T19:51:49.000Z","author":{"id":"597528245006","username":"user_fca1c00e","display_name":"User 2b08fa","verified":false,"is_blue_verified":true,"follower_count":73248},"text":"Orbit helix ionic widen bloom alpha node flux. alpha kite xenon wave logic micro.","lang":"en","metrics":{"likes":4069,"retweets":425,"replies":108,"views":261887,"bookmarks":652,"quotes":62},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/1b535f79d26e.mp4","alt_text":null,"duration_ms":405845}],"urls":[],"hashtags":[],"mentions":[{"id":"846116008513","username":"user_8706cf85"}],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7847249670910741773","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1139811845225147422","url":"https://x.com/user_9defae5d/status/1139811845225147422","created_at":"2026-04-17T21:00:28.000Z","author":{"id":"752846022597","username":"user_9defae5d","display_name":"User 49a70d","verified":false,"is_blue_verified":true,"follower_count":4026082},"text":"Xenon grid lunar terra micro delta ultra query. query ultra.","lang":"en","metrics":{"likes":11998,"retweets":348,"replies":294,"views":2830541,"bookmarks":1885,"quotes":24},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/32a0858c7ba1.mp4","alt_text":null,"duration_ms":48200}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1139811845225147422","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"8838033385258358422","url":"https://x.com/user_143e26b0/status/8838033385258358422","created_at":"2026-04-17T06:45:00.000Z","author":{"id":"104209884508","username":"user_143e26b0","display_name":"User a3ae97","verified":false,"is_blue_verified":true,"follower_count":231412},"text":"Echo joule gamma delta mesa alpha trace trace. mesa sigma core ionic vortex quartz solar drift. kite echo index foxtrot node bloom prism.","lang":"en","metrics":{"likes":7418,"retweets":632,"replies":80,"views":563486,"bookmarks":13813,"quotes":14},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/e058e69774a2.mp4","alt_text":null,"duration_ms":989900}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"8838033385258358422","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"2324730829770412414","url":"https://x.com/user_b924e844/status/2324730829770412414","created_at":"2026-04-17T22:42:45.000Z","author":{"id":"317476795064","username":"user_b924e844","display_name":"User bd0aa5","verified":false,"is_blue_verified":true,"follower_count":238295998},"text":"Unity haze nexus grid bloom xenon drift nexus.","lang":"en","metrics":{"likes":77166,"retweets":6121,"replies":5487,"views":23451769,"bookmarks":1384,"quotes":435},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"3456067991854997259","conversation_id":"2324730829770412414","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"3456067991854997259","url":"https://x.com/user_12970566/status/3456067991854997259","created_at":"2026-04-17T22:24:07.000Z","author":{"id":"887732988441","username":"user_12970566","display_name":"User c44f93","verified":false,"is_blue_verified":true,"follower_count":1585601},"text":"Mesa glyph zonal mesa drift beta mesa sigma. drift wave bloom glyph haze kite wave gamma. apex zonal node lunar helix.","lang":"en","metrics":{"likes":2828,"retweets":350,"replies":365,"views":23643851,"bookmarks":152,"quotes":40},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"7561604691929603606","conversation_id":"3456067991854997259","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1522641576348854435","url":"https://x.com/user_d687596a/status/1522641576348854435","created_at":"2026-04-17T07:31:01.000Z","author":{"id":"428921884383","username":"user_d687596a","display_name":"User d02c1d","verified":false,"is_blue_verified":true,"follower_count":5371},"text":"Foxtrot nexus mesa ratio pulse valve zonal ionic. optic widen lunar beta widen terra logic quartz. node drift gamma bloom trace alpha zonal pulse. haze prism delta kraft wave helix flux kite.","lang":"en","metrics":{"likes":6615,"retweets":216,"replies":277,"views":568881,"bookmarks":573,"quotes":27},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/59f952440273.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1522641576348854435","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"6263674550993747526","url":"https://x.com/user_31de149a/status/6263674550993747526","created_at":"2026-04-17T23:00:01.000Z","author":{"id":"109193805778","username":"user_31de149a","display_name":"User ff846b","verified":false,"is_blue_verified":true,"follower_count":2252},"text":"Zonal valve orbit logic joule prism solar joule. kite logic.","lang":"en","metrics":{"likes":729,"retweets":273,"replies":329,"views":130052,"bookmarks":104,"quotes":1},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/a4e453c51741.mp4","alt_text":null,"duration_ms":25147}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"6263674550993747526","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1125293230092406943","url":"https://x.com/user_46a851d0/status/1125293230092406943","created_at":"2026-04-17T05:12:42.000Z","author":{"id":"114807699959","username":"user_46a851d0","display_name":"User 17e593","verified":false,"is_blue_verified":true,"follower_count":32983},"text":"Widen sigma haze delta beta orbit haze kraft. delta lunar orbit joule trace vortex alpha optic. flux kite.","lang":"en","metrics":{"likes":2744,"retweets":321,"replies":188,"views":369291,"bookmarks":157,"quotes":92},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/d1b19d45bad3.mp4","alt_text":null,"duration_ms":45289}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1125293230092406943","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"3869031856844220752","url":"https://x.com/user_ae6fa065/status/3869031856844220752","created_at":"2026-04-16T20:17:55.000Z","author":{"id":"370687453523","username":"user_ae6fa065","display_name":"User 1bc521","verified":false,"is_blue_verified":true,"follower_count":34763},"text":"Wave grid alpha beta trace vortex trace micro. mesa drift gamma helix micro valve unity solar. flux echo optic haze relay apex prism index. solar yield core ionic prism terra vortex wave.","lang":"en","metrics":{"likes":4252,"retweets":839,"replies":365,"views":1561389,"bookmarks":510,"quotes":375},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/c4da05b6d821.mp4","alt_text":null,"duration_ms":9280}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"3869031856844220752","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"7057157674853009481","url":"https://x.com/user_19d30911/status/7057157674853009481","created_at":"2026-04-17T15:14:48.000Z","author":{"id":"248952525840","username":"user_19d30911","display_name":"User 0e779b","verified":false,"is_blue_verified":true,"follower_count":25143},"text":"Joule joule node index gamma pulse trace kite. terra nexus drift mesa.","lang":"en","metrics":{"likes":127,"retweets":12,"replies":10,"views":4565,"bookmarks":124,"quotes":2},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/7350be6d01f4.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7057157674853009481","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1090250687255528987","url":"https://x.com/user_427de521/status/1090250687255528987","created_at":"2026-04-17T14:07:06.000Z","author":{"id":"810929584413","username":"user_427de521","display_name":"User ec9b5e","verified":false,"is_blue_verified":true,"follower_count":29105},"text":"Unity widen prism relay apex widen delta logic. trace kraft kite orbit foxtrot ionic valve orbit. grid zonal relay kite drift jade haze drift. ultra sigma flux kite jade core micro terra.","lang":"en","metrics":{"likes":220,"retweets":11,"replies":10,"views":68536,"bookmarks":156,"quotes":5},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/8a5a8b58b5f0.mp4","alt_text":null,"duration_ms":55660}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"1041554574205013017","conversation_id":"1090250687255528987","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1041554574205013017","url":"https://x.com/user_dc9c9f09/status/1041554574205013017","created_at":"2026-04-17T03:29:29.000Z","author":{"id":"738414954028","username":"user_dc9c9f09","display_name":"User 80aeed","verified":false,"is_blue_verified":true,"follower_count":1700910},"text":"Terra orbit core echo query jade kraft node. nexus ionic apex yield core ultra gamma alpha. prism ionic ionic micro core haze helix ionic. valve trace relay jade node sigma haze prism.","lang":"en","metrics":{"likes":5593,"retweets":611,"replies":321,"views":4200675,"bookmarks":3909,"quotes":596},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1041554574205013017","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"5810820280941882540","url":"https://x.com/user_41c91310/status/5810820280941882540","created_at":"2026-04-17T21:45:31.000Z","author":{"id":"457108228717","username":"user_41c91310","display_name":"User 65905d","verified":false,"is_blue_verified":true,"follower_count":87660},"text":"Gamma sigma ratio sigma delta trace grid index. optic logic foxtrot kite echo kite ratio prism. zonal logic index kite.","lang":"en","metrics":{"likes":2034,"retweets":117,"replies":29,"views":109238,"bookmarks":92,"quotes":5},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/641459a6d307.mp4","alt_text":null,"duration_ms":11742}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"5810820280941882540","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"7172332670700521790","url":"https://x.com/user_c3eaee62/status/7172332670700521790","created_at":"2026-04-17T17:45:00.000Z","author":{"id":"962206826969","username":"user_c3eaee62","display_name":"User 4f40ff","verified":false,"is_blue_verified":true,"follower_count":21972},"text":"Apex wave unity micro valve core lunar glyph. joule solar jade xenon terra optic alpha index. vortex echo delta logic quartz ultra helix ultra. ionic ratio wave core bloom valve node echo.","lang":"en","metrics":{"likes":454,"retweets":24,"replies":6,"views":33541,"bookmarks":58,"quotes":3},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/0ef85c2b699a.mp4","alt_text":null,"duration_ms":24666}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7172332670700521790","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"1003861653946784235","url":"https://x.com/user_eca18d38/status/1003861653946784235","created_at":"2026-04-17T14:48:25.000Z","author":{"id":"985789943932","username":"user_eca18d38","display_name":"User 14253b","verified":false,"is_blue_verified":true,"follower_count":1251},"text":"Echo trace delta kite ultra wave flux grid. apex apex haze.","lang":"en","metrics":{"likes":1064,"retweets":25,"replies":39,"views":99781,"bookmarks":915,"quotes":2},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/7f5787dd213e.mp4","alt_text":null,"duration_ms":17917}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1003861653946784235","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"5222422539262490066","url":"https://x.com/user_df56e8e9/status/5222422539262490066","created_at":"2026-04-17T23:02:06.000Z","author":{"id":"759883384437","username":"user_df56e8e9","display_name":"User 130332","verified":false,"is_blue_verified":true,"follower_count":1977},"text":"Echo micro quartz logic trace ratio gamma query. widen ionic relay drift gamma beta haze logic. core ultra core ultra apex gamma lunar lunar. prism quartz lunar node delta echo yield ultra.","lang":"es","metrics":{"likes":2133,"retweets":90,"replies":63,"views":627247,"bookmarks":385,"quotes":4},"media":[{"type":"video","url":"https://video.twimg.com/sanitized/1012044c2e02.mp4","alt_text":null,"duration_ms":44746}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"5222422539262490066","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} +{"id":"9727384604865208385","url":"https://x.com/user_4a9f5ff6/status/9727384604865208385","created_at":"2026-04-17T15:13:27.000Z","author":{"id":"793385184366","username":"user_4a9f5ff6","display_name":"User fd5225","verified":false,"is_blue_verified":true,"follower_count":10291},"text":"Delta core foxtrot echo zonal query bloom jade. index ratio apex logic glyph beta sigma sigma. optic helix query valve sigma vortex node solar.","lang":"es","metrics":{"likes":39868,"retweets":409,"replies":9379,"views":28413190,"bookmarks":5936,"quotes":757},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/371f237ed3be.jpg:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/b82dafe77b46.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"9727384604865208385","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"HomeTimeline"} diff --git a/tests/fixtures/sanitized/timeline-basic/manifest.json b/tests/fixtures/sanitized/timeline-basic/manifest.json index 1394c1b..928ead7 100644 --- a/tests/fixtures/sanitized/timeline-basic/manifest.json +++ b/tests/fixtures/sanitized/timeline-basic/manifest.json @@ -6,7 +6,7 @@ "fixture": "fixture.json", "expected": "expected.jsonl" }, - "tweet_count": 38, + "tweet_count": 46, "remap_stats": { "tweet_ids": 166, "user_ids": 77, diff --git a/tests/fixtures/sanitized/user-highlights/expected.jsonl b/tests/fixtures/sanitized/user-highlights/expected.jsonl index 854c518..a33de5b 100644 --- a/tests/fixtures/sanitized/user-highlights/expected.jsonl +++ b/tests/fixtures/sanitized/user-highlights/expected.jsonl @@ -1,20 +1,21 @@ -{"id":"112789460439","url":"https://x.com/user_177791b9/status/112789460439","created_at":"2026-04-18T08:04:28.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Gamma alpha helix zonal haze apex jade prism. pulse foxtrot kite foxtrot nexus apex nexus mesa. query valve yield widen jade beta grid apex. relay foxtrot jade jade apex unity mesa flux.","lang":"en","metrics":{"likes":130,"retweets":7,"replies":6,"views":21766,"bookmarks":15,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/830c7f4db13f.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"4151314283632272422","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"113423394470","url":"https://x.com/user_177791b9/status/113423394470","created_at":"2026-04-17T13:34:09.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Grid kite zonal xenon beta flux prism echo. pulse trace jade widen joule helix ratio node. vortex delta logic kraft core orbit relay mesa. terra joule jade grid index pulse ultra glyph.","lang":"en","metrics":{"likes":238,"retweets":4,"replies":13,"views":9603,"bookmarks":5,"quotes":1},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"9585919719397654236","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"522848797238","url":"https://x.com/user_177791b9/status/522848797238","created_at":"2026-04-16T06:37:18.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Gamma xenon joule grid joule wave echo gamma. trace kraft haze alpha echo grid ionic alpha. apex terra yield ratio joule glyph drift drift. nexus mesa ratio jade joule pulse joule flux.","lang":"en","metrics":{"likes":279,"retweets":8,"replies":11,"views":11145,"bookmarks":6,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/714afaca1c5b.png:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/ee36c649996e.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1088040851420267035","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"314010794471","url":"https://x.com/user_177791b9/status/314010794471","created_at":"2026-04-16T05:25:02.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Echo query grid quartz trace kraft apex trace. jade alpha nexus optic solar yield delta yield.","lang":"en","metrics":{"likes":89,"retweets":3,"replies":1,"views":6271,"bookmarks":2,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/c455cf40593c.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1136487012791746054","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"978379419722","url":"https://x.com/user_177791b9/status/978379419722","created_at":"2026-04-15T18:47:00.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Trace foxtrot trace kite logic trace mesa delta. orbit mesa kite wave trace mesa pulse query. xenon foxtrot drift ratio logic grid optic micro. ultra kite nexus ratio lunar drift sigma pulse.","lang":"en","metrics":{"likes":202,"retweets":4,"replies":10,"views":22504,"bookmarks":5,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/d9a6f7ede307.jpg:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/7da0453b2506.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"8702284508520760048","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"683918383060","url":"https://x.com/user_177791b9/status/683918383060","created_at":"2026-04-15T04:33:15.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Wave kite jade valve kite drift index bloom. sigma ionic node zonal lunar.","lang":"en","metrics":{"likes":114,"retweets":3,"replies":2,"views":17702,"bookmarks":3,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/ac878f047101.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"5902950981874640231","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"105909587479","url":"https://x.com/user_177791b9/status/105909587479","created_at":"2026-04-14T17:28:23.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Grid alpha core helix logic pulse kite kraft. haze ratio index ionic kraft foxtrot gamma ionic. grid query micro bloom terra zonal kraft kite. query zonal foxtrot beta.","lang":"en","metrics":{"likes":169,"retweets":4,"replies":7,"views":11748,"bookmarks":4,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/f52f8fdbf464.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1029168857203280126","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"983679547696","url":"https://x.com/user_177791b9/status/983679547696","created_at":"2026-04-14T09:45:25.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Flux foxtrot kite prism nexus valve nexus delta. foxtrot trace relay nexus logic unity relay query. jade terra nexus foxtrot drift foxtrot grid xenon. prism joule ultra.","lang":"en","metrics":{"likes":515,"retweets":9,"replies":56,"views":29705,"bookmarks":56,"quotes":4},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/7543a25cdf2d.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1382352788479161962","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"107017025453","url":"https://x.com/user_177791b9/status/107017025453","created_at":"2026-04-13T15:07:44.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Micro gamma alpha drift index query ionic sigma. haze ionic mesa trace node beta valve index. vortex grid kite foxtrot optic pulse glyph relay. micro logic nexus kraft glyph solar lunar zonal.","lang":"en","metrics":{"likes":106,"retweets":2,"replies":4,"views":14140,"bookmarks":3,"quotes":2},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/8612ee4fcc0b.png:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/9f6786200cc0.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"3289504580961117446","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"513112497893","url":"https://x.com/user_177791b9/status/513112497893","created_at":"2026-04-12T14:02:20.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Kite drift valve vortex ratio foxtrot pulse zonal. prism widen joule solar ultra sigma drift wave. logic solar glyph grid kite kraft quartz core. solar flux valve yield vortex delta optic relay.","lang":"en","metrics":{"likes":111,"retweets":5,"replies":1,"views":20385,"bookmarks":43,"quotes":0},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"5631690148853576547","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"190051451151","url":"https://x.com/user_177791b9/status/190051451151","created_at":"2026-04-12T09:40:56.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Wave bloom echo index prism widen jade micro. sigma kraft kraft prism sigma vortex apex relay. ratio trace ultra bloom drift nexus index haze. drift gamma drift.","lang":"en","metrics":{"likes":65,"retweets":2,"replies":11,"views":8774,"bookmarks":14,"quotes":0},"media":[],"urls":[{"display":"example.com/san_750213f67b","expanded":"https://example.com/sanitized/5832f748e0","shortened":"https://t.co/SAN_bd989186"}],"hashtags":[],"mentions":[{"id":"268851338333","username":"user_a08297da"}],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1022431468294884315","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"278753212238","url":"https://x.com/user_177791b9/status/278753212238","created_at":"2026-04-08T16:35:48.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Alpha nexus grid solar helix yield pulse foxtrot. optic bloom solar sigma micro vortex terra widen. joule zonal valve wave glyph delta kite ratio. prism optic kite yield beta unity drift unity.","lang":"en","metrics":{"likes":97,"retweets":3,"replies":5,"views":8687,"bookmarks":1,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/5576f3b9f8ad.png:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/4be2b44d4b06.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"9650784833371052014","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"748215133609","url":"https://x.com/user_177791b9/status/748215133609","created_at":"2026-04-08T05:27:18.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Grid foxtrot helix alpha quartz drift lunar bloom. alpha zonal delta logic node unity quartz bloom. flux nexus lunar drift bloom widen vortex zonal. nexus wave grid prism flux apex kite kite.","lang":"en","metrics":{"likes":150,"retweets":5,"replies":4,"views":8552,"bookmarks":3,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/59c0ee517cfc.png:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/93aabb9521a2.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"4289154262641847974","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"765682364341","url":"https://x.com/user_177791b9/status/765682364341","created_at":"2026-04-07T16:33:38.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Ratio quartz kraft mesa orbit unity quartz orbit. alpha kite beta pulse mesa quartz terra kite. lunar prism valve glyph delta jade.","lang":"en","metrics":{"likes":81,"retweets":5,"replies":0,"views":15295,"bookmarks":2,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/6a3938e59780.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7962489132433808741","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"481882888662","url":"https://x.com/user_177791b9/status/481882888662","created_at":"2026-04-07T16:07:26.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Optic sigma trace unity optic unity glyph sigma. echo sigma beta sigma beta query quartz apex. micro delta flux orbit foxtrot joule drift alpha. bloom nexus ratio grid grid.","lang":"en","metrics":{"likes":397,"retweets":9,"replies":19,"views":21127,"bookmarks":54,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/890bcccddac1.jpg:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/20ac97724514.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"2066555943268566099","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"830872526425","url":"https://x.com/user_177791b9/status/830872526425","created_at":"2026-04-05T06:52:40.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Kite relay terra micro pulse kraft core foxtrot. ultra flux glyph beta gamma joule sigma lunar. kraft wave glyph index node haze jade flux. wave delta pulse pulse zonal ultra kraft logic.","lang":"en","metrics":{"likes":103,"retweets":3,"replies":6,"views":31310,"bookmarks":23,"quotes":1},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1623606950670884975","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"642797153431","url":"https://x.com/user_177791b9/status/642797153431","created_at":"2026-04-04T16:23:45.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Echo xenon apex flux widen haze pulse ionic. haze echo widen node wave widen kraft valve. prism grid jade orbit helix trace core xenon. kite trace beta pulse unity mesa delta unity.","lang":"en","metrics":{"likes":92,"retweets":2,"replies":4,"views":8215,"bookmarks":57,"quotes":0},"media":[],"urls":[{"display":"example.com/san_9f1c8fe39b","expanded":"https://example.com/sanitized/e31f425564","shortened":"https://t.co/SAN_35f64d1e"}],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"7962979722680293702","conversation_id":"5146709938762188802","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"104231197370","url":"https://x.com/user_177791b9/status/104231197370","created_at":"2026-04-01T14:35:56.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Prism beta echo mesa nexus unity micro jade. helix joule kraft micro widen zonal grid optic. glyph orbit prism logic solar joule ionic logic. sigma beta vortex bloom lunar terra yield gamma.","lang":"en","metrics":{"likes":153,"retweets":3,"replies":10,"views":20619,"bookmarks":1,"quotes":2},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/56ca8d48f7f8.jpg:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/2e6c3717bba2.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"6417664262552259848","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"294604607458","url":"https://x.com/user_177791b9/status/294604607458","created_at":"2026-03-29T07:04:48.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Bloom wave nexus query ratio solar ionic yield. solar drift yield kite delta terra alpha quartz. sigma trace kraft core helix zonal foxtrot valve. kite yield query vortex nexus helix ionic orbit.","lang":"en","metrics":{"likes":113,"retweets":10,"replies":4,"views":22154,"bookmarks":23,"quotes":1},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7639736522467378281","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} -{"id":"501095708138","url":"https://x.com/user_177791b9/status/501095708138","created_at":"2026-03-27T17:44:30.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Gamma wave micro drift grid joule relay ratio. jade pulse prism query glyph joule kraft gamma. bloom mesa jade index widen lunar prism wave. widen core flux ratio logic query flux ultra.","lang":"en","metrics":{"likes":170,"retweets":11,"replies":20,"views":51118,"bookmarks":146,"quotes":8},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/f57021b08802.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1101363954456126703","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets","captured_at":"2025-01-01T00:00:00.000Z"} +{"id":"112789460439","url":"https://x.com/user_177791b9/status/112789460439","created_at":"2026-04-18T08:04:28.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Gamma alpha helix zonal haze apex jade prism. pulse foxtrot kite foxtrot nexus apex nexus mesa. query valve yield widen jade beta grid apex. relay foxtrot jade jade apex unity mesa flux.","lang":"en","metrics":{"likes":130,"retweets":7,"replies":6,"views":21766,"bookmarks":15,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/830c7f4db13f.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"4151314283632272422","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"113423394470","url":"https://x.com/user_177791b9/status/113423394470","created_at":"2026-04-17T13:34:09.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Grid kite zonal xenon beta flux prism echo. pulse trace jade widen joule helix ratio node. vortex delta logic kraft core orbit relay mesa. terra joule jade grid index pulse ultra glyph.","lang":"en","metrics":{"likes":238,"retweets":4,"replies":13,"views":9603,"bookmarks":5,"quotes":1},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"9585919719397654236","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"522848797238","url":"https://x.com/user_177791b9/status/522848797238","created_at":"2026-04-16T06:37:18.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Gamma xenon joule grid joule wave echo gamma. trace kraft haze alpha echo grid ionic alpha. apex terra yield ratio joule glyph drift drift. nexus mesa ratio jade joule pulse joule flux.","lang":"en","metrics":{"likes":279,"retweets":8,"replies":11,"views":11145,"bookmarks":6,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/714afaca1c5b.png:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/ee36c649996e.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1088040851420267035","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"314010794471","url":"https://x.com/user_177791b9/status/314010794471","created_at":"2026-04-16T05:25:02.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Echo query grid quartz trace kraft apex trace. jade alpha nexus optic solar yield delta yield.","lang":"en","metrics":{"likes":89,"retweets":3,"replies":1,"views":6271,"bookmarks":2,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/c455cf40593c.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1136487012791746054","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"978379419722","url":"https://x.com/user_177791b9/status/978379419722","created_at":"2026-04-15T18:47:00.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Trace foxtrot trace kite logic trace mesa delta. orbit mesa kite wave trace mesa pulse query. xenon foxtrot drift ratio logic grid optic micro. ultra kite nexus ratio lunar drift sigma pulse.","lang":"en","metrics":{"likes":202,"retweets":4,"replies":10,"views":22504,"bookmarks":5,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/d9a6f7ede307.jpg:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/7da0453b2506.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"8702284508520760048","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"683918383060","url":"https://x.com/user_177791b9/status/683918383060","created_at":"2026-04-15T04:33:15.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Wave kite jade valve kite drift index bloom. sigma ionic node zonal lunar.","lang":"en","metrics":{"likes":114,"retweets":3,"replies":2,"views":17702,"bookmarks":3,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/ac878f047101.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"5902950981874640231","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"105909587479","url":"https://x.com/user_177791b9/status/105909587479","created_at":"2026-04-14T17:28:23.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Grid alpha core helix logic pulse kite kraft. haze ratio index ionic kraft foxtrot gamma ionic. grid query micro bloom terra zonal kraft kite. query zonal foxtrot beta.","lang":"en","metrics":{"likes":169,"retweets":4,"replies":7,"views":11748,"bookmarks":4,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/f52f8fdbf464.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1029168857203280126","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"983679547696","url":"https://x.com/user_177791b9/status/983679547696","created_at":"2026-04-14T09:45:25.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Flux foxtrot kite prism nexus valve nexus delta. foxtrot trace relay nexus logic unity relay query. jade terra nexus foxtrot drift foxtrot grid xenon. prism joule ultra.","lang":"en","metrics":{"likes":515,"retweets":9,"replies":56,"views":29705,"bookmarks":56,"quotes":4},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/7543a25cdf2d.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1382352788479161962","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"107017025453","url":"https://x.com/user_177791b9/status/107017025453","created_at":"2026-04-13T15:07:44.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Micro gamma alpha drift index query ionic sigma. haze ionic mesa trace node beta valve index. vortex grid kite foxtrot optic pulse glyph relay. micro logic nexus kraft glyph solar lunar zonal.","lang":"en","metrics":{"likes":106,"retweets":2,"replies":4,"views":14140,"bookmarks":3,"quotes":2},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/8612ee4fcc0b.png:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/9f6786200cc0.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"3289504580961117446","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"513112497893","url":"https://x.com/user_177791b9/status/513112497893","created_at":"2026-04-12T14:02:20.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Kite drift valve vortex ratio foxtrot pulse zonal. prism widen joule solar ultra sigma drift wave. logic solar glyph grid kite kraft quartz core. solar flux valve yield vortex delta optic relay.","lang":"en","metrics":{"likes":111,"retweets":5,"replies":1,"views":20385,"bookmarks":43,"quotes":0},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"5631690148853576547","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"190051451151","url":"https://x.com/user_177791b9/status/190051451151","created_at":"2026-04-12T09:40:56.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Wave bloom echo index prism widen jade micro. sigma kraft kraft prism sigma vortex apex relay. ratio trace ultra bloom drift nexus index haze. drift gamma drift.","lang":"en","metrics":{"likes":65,"retweets":2,"replies":11,"views":8774,"bookmarks":14,"quotes":0},"media":[],"urls":[{"display":"example.com/san_750213f67b","expanded":"https://example.com/sanitized/5832f748e0","shortened":"https://t.co/SAN_bd989186"}],"hashtags":[],"mentions":[{"id":"268851338333","username":"user_a08297da"}],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1022431468294884315","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"278753212238","url":"https://x.com/user_177791b9/status/278753212238","created_at":"2026-04-08T16:35:48.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Alpha nexus grid solar helix yield pulse foxtrot. optic bloom solar sigma micro vortex terra widen. joule zonal valve wave glyph delta kite ratio. prism optic kite yield beta unity drift unity.","lang":"en","metrics":{"likes":97,"retweets":3,"replies":5,"views":8687,"bookmarks":1,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/5576f3b9f8ad.png:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/4be2b44d4b06.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"9650784833371052014","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"748215133609","url":"https://x.com/user_177791b9/status/748215133609","created_at":"2026-04-08T05:27:18.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Grid foxtrot helix alpha quartz drift lunar bloom. alpha zonal delta logic node unity quartz bloom. flux nexus lunar drift bloom widen vortex zonal. nexus wave grid prism flux apex kite kite.","lang":"en","metrics":{"likes":150,"retweets":5,"replies":4,"views":8552,"bookmarks":3,"quotes":0},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/59c0ee517cfc.png:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/93aabb9521a2.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"4289154262641847974","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"765682364341","url":"https://x.com/user_177791b9/status/765682364341","created_at":"2026-04-07T16:33:38.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Ratio quartz kraft mesa orbit unity quartz orbit. alpha kite beta pulse mesa quartz terra kite. lunar prism valve glyph delta jade.","lang":"en","metrics":{"likes":81,"retweets":5,"replies":0,"views":15295,"bookmarks":2,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/6a3938e59780.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7962489132433808741","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"481882888662","url":"https://x.com/user_177791b9/status/481882888662","created_at":"2026-04-07T16:07:26.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Optic sigma trace unity optic unity glyph sigma. echo sigma beta sigma beta query quartz apex. micro delta flux orbit foxtrot joule drift alpha. bloom nexus ratio grid grid.","lang":"en","metrics":{"likes":397,"retweets":9,"replies":19,"views":21127,"bookmarks":54,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/890bcccddac1.jpg:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/20ac97724514.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"2066555943268566099","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"830872526425","url":"https://x.com/user_177791b9/status/830872526425","created_at":"2026-04-05T06:52:40.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Kite relay terra micro pulse kraft core foxtrot. ultra flux glyph beta gamma joule sigma lunar. kraft wave glyph index node haze jade flux. wave delta pulse pulse zonal ultra kraft logic.","lang":"en","metrics":{"likes":103,"retweets":3,"replies":6,"views":31310,"bookmarks":23,"quotes":1},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1623606950670884975","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"642797153431","url":"https://x.com/user_177791b9/status/642797153431","created_at":"2026-04-04T16:23:45.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Echo xenon apex flux widen haze pulse ionic. haze echo widen node wave widen kraft valve. prism grid jade orbit helix trace core xenon. kite trace beta pulse unity mesa delta unity.","lang":"en","metrics":{"likes":92,"retweets":2,"replies":4,"views":8215,"bookmarks":57,"quotes":0},"media":[],"urls":[{"display":"example.com/san_9f1c8fe39b","expanded":"https://example.com/sanitized/e31f425564","shortened":"https://t.co/SAN_35f64d1e"}],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":"7962979722680293702","conversation_id":"5146709938762188802","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"633754965947","url":"https://x.com/user_177791b9/status/633754965947","created_at":"2026-04-04T16:22:13.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Echo optic kite solar haze lunar node foxtrot. micro grid jade grid optic pulse.","lang":"en","metrics":{"likes":162,"retweets":2,"replies":17,"views":16152,"bookmarks":135,"quotes":1},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/7f81644c00ba.jpg:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/73fdb3caca49.jpg:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/2ee4442f46e5.jpg:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/2f9a6859ad16.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7962979722680293702","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":true,"source_endpoint":"UserHighlightsTweets"} +{"id":"104231197370","url":"https://x.com/user_177791b9/status/104231197370","created_at":"2026-04-01T14:35:56.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Prism beta echo mesa nexus unity micro jade. helix joule kraft micro widen zonal grid optic. glyph orbit prism logic solar joule ionic logic. sigma beta vortex bloom lunar terra yield gamma.","lang":"en","metrics":{"likes":153,"retweets":3,"replies":10,"views":20619,"bookmarks":1,"quotes":2},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/56ca8d48f7f8.jpg:orig","alt_text":null},{"type":"photo","url":"https://pbs.twimg.com/sanitized/2e6c3717bba2.jpg:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"6417664262552259848","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"294604607458","url":"https://x.com/user_177791b9/status/294604607458","created_at":"2026-03-29T07:04:48.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Bloom wave nexus query ratio solar ionic yield. solar drift yield kite delta terra alpha quartz. sigma trace kraft core helix zonal foxtrot valve. kite yield query vortex nexus helix ionic orbit.","lang":"en","metrics":{"likes":113,"retweets":10,"replies":4,"views":22154,"bookmarks":23,"quotes":1},"media":[],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"7639736522467378281","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} +{"id":"501095708138","url":"https://x.com/user_177791b9/status/501095708138","created_at":"2026-03-27T17:44:30.000Z","author":{"id":"568793795593","username":"user_177791b9","display_name":"User 610e3e","verified":false,"is_blue_verified":true,"follower_count":58086},"text":"Gamma wave micro drift grid joule relay ratio. jade pulse prism query glyph joule kraft gamma. bloom mesa jade index widen lunar prism wave. widen core flux ratio logic query flux ultra.","lang":"en","metrics":{"likes":170,"retweets":11,"replies":20,"views":51118,"bookmarks":146,"quotes":8},"media":[{"type":"photo","url":"https://pbs.twimg.com/sanitized/f57021b08802.png:orig","alt_text":null}],"urls":[],"hashtags":[],"mentions":[],"in_reply_to":null,"quoted_tweet_id":null,"conversation_id":"1101363954456126703","is_retweet":false,"retweeted_tweet_id":null,"is_subscriber_only":false,"source_endpoint":"UserHighlightsTweets"} diff --git a/tests/fixtures/sanitized/user-highlights/manifest.json b/tests/fixtures/sanitized/user-highlights/manifest.json index 33ece90..5b4980c 100644 --- a/tests/fixtures/sanitized/user-highlights/manifest.json +++ b/tests/fixtures/sanitized/user-highlights/manifest.json @@ -6,7 +6,7 @@ "fixture": "fixture.json", "expected": "expected.jsonl" }, - "tweet_count": 20, + "tweet_count": 21, "remap_stats": { "tweet_ids": 43, "user_ids": 49, diff --git a/tests/manifest.test.mjs b/tests/manifest.test.mjs index 9df4355..b7dac88 100644 --- a/tests/manifest.test.mjs +++ b/tests/manifest.test.mjs @@ -2,6 +2,7 @@ import test from 'node:test'; import assert from 'node:assert/strict'; import { execFileSync } from 'node:child_process'; import fs from 'node:fs'; +import os from 'node:os'; import path from 'node:path'; import url from 'node:url'; @@ -10,11 +11,25 @@ const root = path.resolve(__dirname, '..'); const chromeManifestPath = path.join(root, 'manifest.json'); const firefoxManifestPath = path.join(root, 'manifest.firefox.json'); -// Regenerate to ensure it's up-to-date -execFileSync('node', [path.join(root, 'scripts', 'build-firefox-manifest.js')]); +// Regenerate into a temp file — never overwrite the checked-in manifest, +// otherwise a stale commit can't be detected (it would be silently fixed +// locally while the release zip ships the stale file). +const tmpOut = path.join( + fs.mkdtempSync(path.join(os.tmpdir(), 'xtap-manifest-')), + 'manifest.firefox.json' +); +execFileSync('node', [path.join(root, 'scripts', 'build-firefox-manifest.js'), tmpOut]); + +const generated = fs.readFileSync(tmpOut, 'utf8'); +const checkedIn = fs.readFileSync(firefoxManifestPath, 'utf8'); const chrome = JSON.parse(fs.readFileSync(chromeManifestPath, 'utf8')); -const firefox = JSON.parse(fs.readFileSync(firefoxManifestPath, 'utf8')); +const firefox = JSON.parse(checkedIn); + +test('checked-in manifest.firefox.json is up to date', () => { + assert.equal(checkedIn, generated, + 'manifest.firefox.json is stale — run: node scripts/build-firefox-manifest.js'); +}); test('Firefox manifest uses background scripts instead of service_worker', () => { assert.equal(firefox.manifest_version, 3); @@ -35,6 +50,18 @@ test('Firefox manifest preserves permissions from Chrome manifest', () => { assert.deepEqual(firefox.host_permissions, chrome.host_permissions); }); +test('manifest permissions stay within stealth allowlist', () => { + const allowedPermissions = ['alarms', 'nativeMessaging', 'storage']; + const allowedHosts = ['*://*.twitter.com/*', '*://*.x.com/*', 'http://127.0.0.1/*']; + for (const manifest of [chrome, firefox]) { + assert.deepEqual([...manifest.permissions].sort(), allowedPermissions); + assert.deepEqual([...manifest.host_permissions].sort(), allowedHosts); + assert.ok(!manifest.permissions.includes('webRequest')); + assert.ok(!manifest.permissions.includes('tabs')); + assert.ok(!manifest.permissions.includes('scripting')); + } +}); + test('Firefox manifest version matches Chrome manifest', () => { assert.equal(firefox.version, chrome.version); }); diff --git a/tests/startup-flush.test.mjs b/tests/startup-flush.test.mjs new file mode 100644 index 0000000..c029a38 --- /dev/null +++ b/tests/startup-flush.test.mjs @@ -0,0 +1,154 @@ +/** + * Tests for the service-worker startup path in background.js — the Init block + * that the transport-flush/staging-wal rigs strip out. + * Run with: node --test tests/startup-flush.test.mjs + * + * Runs the FULL source (imports stripped, Init kept) in a vm with pre-seeded + * storage, and asserts that a buffer restored from a previous session is + * delivered immediately at startup — without any flush timer or alarm firing. + * MV3 can kill the SW ~30s after wake, so waiting for the first timer tick + * risks stranding the restored batch for another session. + */ + +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import vm from 'node:vm'; +import { dedupTweet } from '../lib/dedup.js'; + +const bgSource = readFileSync(new URL('../background.js', import.meta.url), 'utf8'); +const testSource = bgSource.replace(/^import\b.*$/gm, ''); + +// The Init block must still be present — if the marker moves, this rig would +// silently stop testing the startup path. +assert.ok(bgSource.includes('// --- Init ---'), 'Init marker missing from background.js'); + +function createMockStorage(initial = {}) { + let data = { ...initial }; + return { + get(keys) { + if (keys === null) return Promise.resolve({ ...data }); + const result = {}; + const list = Array.isArray(keys) ? keys : [keys]; + for (const k of list) if (k in data) result[k] = data[k]; + return Promise.resolve(result); + }, + set(items) { Object.assign(data, items); return Promise.resolve(); }, + remove(keys) { + const list = Array.isArray(keys) ? keys : [keys]; + for (const k of list) delete data[k]; + return Promise.resolve(); + }, + setAccessLevel() { return Promise.resolve(); }, + }; +} + +function until(cond, ms = 2000) { + return new Promise((resolve, reject) => { + const started = Date.now(); + (function poll() { + if (cond()) return resolve(); + if (Date.now() - started > ms) return reject(new Error('condition not met in time')); + setTimeout(poll, 5); + })(); + }); +} + +function bootSW({ sessionSeed = {}, localSeed = {} } = {}) { + const sessionStore = createMockStorage(sessionSeed); + const localStore = createMockStorage(localSeed); + const fetches = []; // { url, method, body } + const alarms = { created: [], listener: null }; + + const sandbox = { + console: { log() {}, warn() {}, error() {} }, + // unref so the SW's recurring flush timer (30-45s, rescheduled forever by + // scheduleNextFlush in the Init block) can't keep the test process alive. + setTimeout: (fn, ms) => { + const t = globalThis.setTimeout(fn, ms); + if (typeof t?.unref === 'function') t.unref(); + return t; + }, + clearTimeout: globalThis.clearTimeout, + Math, JSON, Date, Promise, Set, Map, Array, Object, + AbortController: globalThis.AbortController, + AbortSignal: globalThis.AbortSignal, + fetch: async (url, opts = {}) => { + fetches.push({ url, method: opts.method || 'GET', body: opts.body }); + return { ok: true, status: 200, json: async () => ({ ok: true }) }; + }, + extractTweets: () => [], + dedupTweet, + chrome: { + runtime: { + getManifest: () => ({}), // dev mode -> seenIds/tweetBuffer in session storage + connectNative() { throw new Error('not available'); }, + onMessage: { addListener() {} }, + lastError: null, + }, + storage: { session: sessionStore, local: localStore }, + action: { setBadgeText() {}, setBadgeBackgroundColor() {} }, + alarms: { + create(name, info) { alarms.created.push({ name, info }); }, + clear() {}, + onAlarm: { addListener(fn) { alarms.listener = fn; } }, + }, + }, + }; + + vm.runInNewContext(testSource, sandbox); + return { fetches, alarms, sessionStore, localStore }; +} + +describe('service worker startup', () => { + it('flushes a restored buffer immediately, before any timer or alarm fires', async () => { + const sw = bootSW({ + sessionSeed: { tweetBuffer: [{ id: 'restored-1', text: 'from last session' }] }, + localSeed: { httpToken: 'cached-token', httpPort: 17381 }, + }); + + // Init chain: restoreState -> initTransport (probes /status with the + // cached token) -> startup flush POSTs the restored buffer. All of this + // must happen with zero timer ticks and zero alarm firings. + await until(() => sw.fetches.some(f => f.url.includes('/tweets'))); + + const post = sw.fetches.find(f => f.url.includes('/tweets')); + const tweets = JSON.parse(post.body).tweets; + assert.deepEqual(tweets.map(t => t.id), ['restored-1'], + 'the restored batch must be delivered by the startup flush itself'); + + assert.ok(sw.fetches.some(f => f.url.includes('/status')), + 'startup must validate the cached token against the daemon first'); + + // The alarm backstop must be armed for the restored buffer (in case the + // startup flush fails and the SW dies before the timer tick). + assert.ok(sw.alarms.created.some(a => a.name === 'xtap-flush'), + 'a restored buffer must arm the flush alarm backstop'); + + // The batch drains and is removed from persisted state after the ack. + const deadline = Date.now() + 2000; + let persisted; + do { + persisted = await sw.sessionStore.get(['tweetBuffer']); + if ((persisted.tweetBuffer || []).length === 0) break; + await new Promise(r => setTimeout(r, 5)); + } while (Date.now() < deadline); + assert.deepEqual(persisted.tweetBuffer, [], + 'the acked startup batch must leave persisted state'); + }); + + it('does not POST tweets at startup when nothing was restored', async () => { + const sw = bootSW({ + localSeed: { httpToken: 'cached-token', httpPort: 17381 }, + }); + + // Let the init chain settle (transport probe happens regardless). + await until(() => sw.fetches.some(f => f.url.includes('/status'))); + await new Promise(r => setTimeout(r, 50)); + + assert.equal(sw.fetches.filter(f => f.url.includes('/tweets')).length, 0, + 'an empty buffer must not trigger a startup POST'); + assert.equal(sw.alarms.created.length, 0, + 'no alarm needed when nothing is buffered'); + }); +}); diff --git a/tests/test_xtap_core.py b/tests/test_xtap_core.py index e736861..de233fe 100644 --- a/tests/test_xtap_core.py +++ b/tests/test_xtap_core.py @@ -1,9 +1,12 @@ """Tests for native-host/xtap_core.py""" +import builtins +import io import json import os import sys import threading +import urllib.error import pytest @@ -33,6 +36,31 @@ def test_none(self): def test_non_string_returns_empty(self): assert xtap_core._date_prefix(12345) == '' + def test_path_like_values_return_empty(self): + assert xtap_core._date_prefix('/tmp/pwn') == '' + assert xtap_core._date_prefix('../../../x') == '' + + def test_invalid_dates_return_empty(self): + assert xtap_core._date_prefix('2024-99-99') == '' + assert xtap_core._date_prefix('2024-01-15evil') == '' + + +class TestTweetUrlValidation: + def test_accepts_x_and_twitter_status_urls(self): + assert xtap_core._tweet_status_id('https://x.com/user/status/123') == '123' + assert xtap_core._tweet_status_id('https://twitter.com/i/web/status/456?lang=en') == '456' + + def test_rejects_non_status_or_offsite_urls(self): + for url in ( + 'http://x.com/user/status/123', + 'https://x.com.evil.example/user/status/123', + 'https://evil.example/status/123', + 'https://x.com/user', + 'https://x.com/user/status/notdigits', + None): + with pytest.raises(ValueError): + xtap_core.validate_tweet_url(url) + # --------------------------------------------------------------------------- # load_seen_ids @@ -51,6 +79,15 @@ def test_single_file(self, tmp_path): ) assert xtap_core.load_seen_ids(str(tmp_path)) == {'111', '222'} + def test_article_file_loads_article_seen_key(self, tmp_path): + f = tmp_path / 'tweets-2024-01-15.jsonl' + f.write_text( + json.dumps({'id': '111', 'text': 'article', 'is_article': True}) + '\n', + encoding='utf-8') + assert xtap_core.load_seen_ids(str(tmp_path)) == { + '111', xtap_core._article_seen_key('111') + } + def test_multiple_files(self, tmp_path): (tmp_path / 'tweets-2024-01-15.jsonl').write_text( json.dumps({'id': '111'}) + '\n' @@ -120,12 +157,30 @@ def test_consecutive_calls_dedup(self, tmp_path): assert count == 0 assert dupes == 1 - def test_article_bypasses_dedup(self, tmp_path): + def test_article_can_enrich_previously_seen_non_article(self, tmp_path): seen = {'1'} tweets = [{'id': '1', 'text': 'article', 'is_article': True}] count, dupes = xtap_core.write_tweets(tweets, str(tmp_path), seen) assert count == 1 assert dupes == 0 + assert xtap_core._article_seen_key('1') in seen + + def test_repeated_full_article_is_deduped(self, tmp_path): + seen = {'1', xtap_core._article_seen_key('1')} + tweets = [{'id': '1', 'text': 'same article', 'is_article': True}] + count, dupes = xtap_core.write_tweets(tweets, str(tmp_path), seen) + assert count == 0 + assert dupes == 1 + + def test_same_article_twice_in_one_batch_is_deduped(self, tmp_path): + seen = {'1'} + tweets = [ + {'id': '1', 'text': 'article', 'is_article': True}, + {'id': '1', 'text': 'same article', 'is_article': True}, + ] + count, dupes = xtap_core.write_tweets(tweets, str(tmp_path), seen) + assert count == 1 + assert dupes == 1 def test_tweet_without_id_written(self, tmp_path): seen = set() @@ -163,20 +218,18 @@ def test_unicode_preserved(self, tmp_path): class TestResolveOutputDir: def test_falsy_msg_dir(self, tmp_path): default = str(tmp_path / 'default') - result = xtap_core.resolve_output_dir('', default, set(), set()) + result, _ = xtap_core.resolve_output_dir('', default, {}) assert result == default def test_none_msg_dir(self, tmp_path): default = str(tmp_path / 'default') - result = xtap_core.resolve_output_dir(None, default, set(), set()) + result, _ = xtap_core.resolve_output_dir(None, default, {}) assert result == default def test_custom_dir_created(self, tmp_path, monkeypatch): monkeypatch.setattr(xtap_core, '_ALLOWED_ROOTS', (os.path.realpath(str(tmp_path)),)) custom = str(tmp_path / 'custom') - seen = set() - custom_dirs = set() - result = xtap_core.resolve_output_dir(custom, '/default', seen, custom_dirs) + result, _ = xtap_core.resolve_output_dir(custom, '/default', {}) assert os.path.realpath(result) == os.path.realpath(custom) assert os.path.isdir(result) @@ -184,20 +237,19 @@ def test_custom_dir_no_reload(self, tmp_path, monkeypatch): monkeypatch.setattr(xtap_core, '_ALLOWED_ROOTS', (os.path.realpath(str(tmp_path)),)) custom = str(tmp_path / 'custom') os.makedirs(custom) - seen = set() - custom_dirs = set() - # First call adds to custom_dirs - result = xtap_core.resolve_output_dir(custom, '/default', seen, custom_dirs) - assert result in custom_dirs - # Second call — custom_dirs already has it, load_seen_ids not called again - old_size = len(custom_dirs) - xtap_core.resolve_output_dir(custom, '/default', seen, custom_dirs) - assert len(custom_dirs) == old_size + calls = [] + orig_load = xtap_core.load_seen_ids + monkeypatch.setattr(xtap_core, 'load_seen_ids', + lambda d: (calls.append(d), orig_load(d))[1]) + seen_by_dir = {} + result, _ = xtap_core.resolve_output_dir(custom, '/default', seen_by_dir) + assert result in seen_by_dir + # Second call — already loaded, load_seen_ids not called again + xtap_core.resolve_output_dir(custom, '/default', seen_by_dir) + assert len(calls) == 1 def test_tilde_expansion(self, tmp_path): - seen = set() - custom_dirs = set() - result = xtap_core.resolve_output_dir('~/xtap-test-dir', '/default', seen, custom_dirs) + result, _ = xtap_core.resolve_output_dir('~/xtap-test-dir', '/default', {}) expected = os.path.expanduser('~/xtap-test-dir') assert result == expected # Clean up @@ -211,9 +263,7 @@ def test_custom_dir_loads_seen_ids(self, tmp_path, monkeypatch): (tmp_path / 'custom' / 'tweets-2024-01-15.jsonl').write_text( json.dumps({'id': '999'}) + '\n' ) - seen = set() - custom_dirs = set() - xtap_core.resolve_output_dir(custom, '/default', seen, custom_dirs) + _, seen = xtap_core.resolve_output_dir(custom, '/default', {}) assert '999' in seen @@ -250,7 +300,7 @@ def test_absolute_path_outside_home_rejected(self): def test_resolve_output_dir_rejects_traversal(self): with pytest.raises(ValueError, match='outside allowed directories'): - xtap_core.resolve_output_dir('/etc/evil', '/default', set(), set()) + xtap_core.resolve_output_dir('/etc/evil', '/default', {}) def test_default_output_dir_allowed(self): result = xtap_core.validate_output_dir(xtap_core.DEFAULT_OUTPUT_DIR) @@ -701,15 +751,21 @@ def test_rejects_non_numeric_tweet_id(self, tmp_path): def test_rejects_article_local_path_with_traversal(self, tmp_path): tweets = [{ 'id': '999', - 'article': {'media': [{ - 'url': 'https://pbs.twimg.com/media/x.jpg', - 'local_path': '../../../../etc/passwd', - }]}, + 'article': { + 'text': 'before\n![x](../../../../etc/passwd)\nafter', + 'media': [{ + 'url': 'https://pbs.twimg.com/media/x.jpg', + 'local_path': '../../../../etc/passwd', + }], + }, }] pending = xtap_core.collect_image_jobs(tweets, str(tmp_path)) assert pending == [] # Unsafe local_path is stripped so the JSONL doesn't carry it. assert 'local_path' not in tweets[0]['article']['media'][0] + assert '../../../../etc/passwd' not in tweets[0]['article']['text'] + assert 'before' in tweets[0]['article']['text'] + assert 'after' in tweets[0]['article']['text'] def test_rejects_article_local_path_absolute(self, tmp_path): tweets = [{ @@ -1139,3 +1195,452 @@ def test_get_returns_same_instance(self, monkeypatch): b = xtap_core.get_image_downloader() assert a is b xtap_core.reset_image_downloader() + + +# --------------------------------------------------------------------------- +# write_tweets — failure atomicity (seen_ids must not be poisoned) +# --------------------------------------------------------------------------- + + +class TestWriteTweetsFailureAtomicity: + def test_failed_batch_does_not_poison_seen_ids(self, tmp_path): + """If writing a batch raises, none of its IDs may be committed to + seen_ids — otherwise the extension's retry of the same batch is + counted as all-dupes and the tweets are silently lost.""" + seen = set() + bad_batch = [ + {'id': '1', 'text': 'fine'}, + {'id': '2', 'text': {'not', 'serializable'}}, # json.dumps raises + ] + with pytest.raises(TypeError): + xtap_core.write_tweets(bad_batch, str(tmp_path), seen) + + # The retry (same IDs, now serializable) must be written, not deduped. + count, dupes = xtap_core.write_tweets( + [{'id': '1', 'text': 'fine'}, {'id': '2', 'text': 'fixed'}], + str(tmp_path), seen) + assert count == 2 + assert dupes == 0 + files = list(tmp_path.glob('tweets-*.jsonl')) + ids = [json.loads(line)['id'] + for line in files[0].read_text(encoding='utf-8').strip().split('\n')] + assert '2' in ids + + def test_same_id_twice_in_one_batch_deduped(self, tmp_path): + seen = set() + count, dupes = xtap_core.write_tweets( + [{'id': '1', 'text': 'a'}, {'id': '1', 'text': 'b'}], + str(tmp_path), seen) + assert count == 1 + assert dupes == 1 + + def test_successful_batch_commits_seen_ids(self, tmp_path): + seen = set() + xtap_core.write_tweets([{'id': '1', 'text': 'a'}], str(tmp_path), seen) + assert seen == {'1'} + + +# --------------------------------------------------------------------------- +# Explicit UTF-8 encoding (Windows runs Python with a cp1252 locale default) +# --------------------------------------------------------------------------- + + +class TestExplicitUtf8Encoding: + """Simulates a non-UTF-8 locale by defaulting text-mode open() calls that + lack an explicit encoding= to cp1252 — the behavior of Python <= 3.15 on + Windows, where emoji in tweets raise UnicodeEncodeError unless every + open() passes encoding='utf-8'.""" + + @pytest.fixture(autouse=True) + def _cp1252_default_open(self, monkeypatch): + real_open = builtins.open + def cp1252_open(file, mode='r', *args, **kwargs): + if 'b' not in mode and kwargs.get('encoding') is None: + kwargs['encoding'] = 'cp1252' + return real_open(file, mode, *args, **kwargs) + monkeypatch.setattr(builtins, 'open', cp1252_open) + + def test_write_tweets_emoji(self, tmp_path): + seen = set() + count, _ = xtap_core.write_tweets( + [{'id': '1', 'text': 'Hello \U0001f30d'}], str(tmp_path), seen) + assert count == 1 + files = list(tmp_path.glob('tweets-*.jsonl')) + assert '\U0001f30d' in files[0].read_text(encoding='utf-8') + + def test_load_seen_ids_emoji(self, tmp_path): + (tmp_path / 'tweets-2024-01-15.jsonl').write_text( + json.dumps({'id': '1', 'text': '\U0001f30d'}, ensure_ascii=False) + '\n', + encoding='utf-8') + assert xtap_core.load_seen_ids(str(tmp_path)) == {'1'} + + def test_write_log_emoji(self, tmp_path): + assert xtap_core.write_log(['line \U0001f30d'], str(tmp_path)) == 1 + logs = list(tmp_path.glob('debug-*.log')) + assert '\U0001f30d' in logs[0].read_text(encoding='utf-8') + + def test_write_dump_emoji(self, tmp_path): + xtap_core.write_dump('d.json', '{"x": "\U0001f30d"}', str(tmp_path)) + assert '\U0001f30d' in (tmp_path / 'd.json').read_text(encoding='utf-8') + + +# --------------------------------------------------------------------------- +# load_seen_ids — robustness (daemon calls this at startup; a crash loops) +# --------------------------------------------------------------------------- + + +class TestLoadSeenIdsRobustness: + def test_undecodable_file_does_not_crash(self, tmp_path): + (tmp_path / 'tweets-2024-01-14.jsonl').write_bytes(b'\xff\xfe\x00garbage\n') + (tmp_path / 'tweets-2024-01-15.jsonl').write_text( + json.dumps({'id': '111'}) + '\n', encoding='utf-8') + assert xtap_core.load_seen_ids(str(tmp_path)) == {'111'} + + def test_glob_metacharacters_in_dir_name(self, tmp_path): + d = tmp_path / 'xtap [archive]' + d.mkdir() + (d / 'tweets-2024-01-15.jsonl').write_text( + json.dumps({'id': '111'}) + '\n', encoding='utf-8') + assert xtap_core.load_seen_ids(str(d)) == {'111'} + + def test_icloud_placeholder_warns(self, tmp_path, capsys): + """Evicted iCloud files become hidden .name.icloud placeholders that + the glob misses — dedup silently degrades. At minimum warn.""" + (tmp_path / '.tweets-2024-01-15.jsonl.icloud').write_bytes(b'') + xtap_core.load_seen_ids(str(tmp_path)) + assert 'icloud' in capsys.readouterr().err.lower() + + def test_unreadable_file_warns_and_continues(self, tmp_path, monkeypatch, capsys): + """A file whose open() raises OSError (permissions, iCloud eviction + race) must not crash daemon startup — warn and keep the IDs from the + readable files.""" + good = tmp_path / 'tweets-2024-01-15.jsonl' + good.write_text(json.dumps({'id': '111'}) + '\n', encoding='utf-8') + bad = tmp_path / 'tweets-2024-01-14.jsonl' + bad.write_text(json.dumps({'id': '222'}) + '\n', encoding='utf-8') + + real_open = builtins.open + + def deny_bad(file, *args, **kwargs): + if str(file) == str(bad): + raise PermissionError(13, 'Permission denied', str(file)) + return real_open(file, *args, **kwargs) + + monkeypatch.setattr(builtins, 'open', deny_bad) + assert xtap_core.load_seen_ids(str(tmp_path)) == {'111'} + err = capsys.readouterr().err + assert 'cannot read' in err + assert str(bad) in err + + +# --------------------------------------------------------------------------- +# Per-directory dedup (switching outputDir must not suppress tweets) +# --------------------------------------------------------------------------- + + +class TestPerDirectoryDedup: + """seen-ID sets are scoped per output directory: pointing the extension + at a new directory must re-write tweets there even if they were already + captured into the old directory.""" + + def test_new_dir_gets_tweets_already_in_default(self, tmp_path, monkeypatch): + monkeypatch.setattr(xtap_core, '_ALLOWED_ROOTS', (os.path.realpath(str(tmp_path)),)) + default = str(tmp_path / 'default') + os.makedirs(default) + seen_by_dir = {default: set()} + + out, seen = xtap_core.resolve_output_dir('', default, seen_by_dir) + assert out == default + xtap_core.write_tweets([{'id': '1', 'text': 'a'}], out, seen) + + custom = str(tmp_path / 'custom') + out2, seen2 = xtap_core.resolve_output_dir(custom, default, seen_by_dir) + count, dupes = xtap_core.write_tweets([{'id': '1', 'text': 'a'}], out2, seen2) + assert count == 1 + assert dupes == 0 + + def test_dedup_still_applies_within_each_dir(self, tmp_path, monkeypatch): + monkeypatch.setattr(xtap_core, '_ALLOWED_ROOTS', (os.path.realpath(str(tmp_path)),)) + default = str(tmp_path / 'default') + os.makedirs(default) + seen_by_dir = {default: set()} + custom = str(tmp_path / 'custom') + + for _ in range(2): + out, seen = xtap_core.resolve_output_dir(custom, default, seen_by_dir) + count, dupes = xtap_core.write_tweets([{'id': '1', 'text': 'a'}], out, seen) + assert count == 0 + assert dupes == 1 + + def test_existing_ids_loaded_per_dir(self, tmp_path, monkeypatch): + monkeypatch.setattr(xtap_core, '_ALLOWED_ROOTS', (os.path.realpath(str(tmp_path)),)) + default = str(tmp_path / 'default') + os.makedirs(default) + custom = tmp_path / 'custom' + custom.mkdir() + (custom / 'tweets-2024-01-15.jsonl').write_text( + json.dumps({'id': '999'}) + '\n', encoding='utf-8') + seen_by_dir = {default: set()} + + out, seen = xtap_core.resolve_output_dir(str(custom), default, seen_by_dir) + count, dupes = xtap_core.write_tweets([{'id': '999', 'text': 'a'}], out, seen) + assert count == 0 + assert dupes == 1 + + +# --------------------------------------------------------------------------- +# download_direct — URL trust boundary + stall timeout +# --------------------------------------------------------------------------- + + +class TestDownloadDirect: + def test_rejects_non_https_and_non_twimg_urls(self, tmp_path): + for url in ('file:///etc/passwd', + 'http://127.0.0.1:17381/secret', + 'https://evil.example.com/x.mp4', + 'https://twimg.com.evil.example/x.mp4'): + with pytest.raises(ValueError, match='not allowed'): + xtap_core.download_direct(url, '123', str(tmp_path)) + assert list(tmp_path.iterdir()) == [] # nothing fetched or written + + def test_passes_socket_timeout_and_renames_part(self, tmp_path, monkeypatch): + captured = {} + + class FakeResp(io.BytesIO): + headers = {} + + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + def fake_open(url, timeout=None): + captured['url'] = url + captured['timeout'] = timeout + return FakeResp(b'video-bytes') + + # Must go through the no-redirect opener, not bare urlopen. + monkeypatch.setattr(xtap_core._NO_REDIRECT_OPENER, 'open', fake_open) + path = xtap_core.download_direct( + 'https://video.twimg.com/ext_tw_video/1/pu/vid/x.mp4', '42', str(tmp_path)) + assert captured['timeout'] == xtap_core.VIDEO_REQUEST_TIMEOUT_S + assert os.path.basename(path) == '42.mp4' + with open(path, 'rb') as f: + assert f.read() == b'video-bytes' + assert not os.path.exists(path + '.part') + + def test_post_date_cannot_escape_video_dir(self, tmp_path, monkeypatch): + class FakeResp(io.BytesIO): + headers = {} + + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + monkeypatch.setattr( + xtap_core._NO_REDIRECT_OPENER, 'open', + lambda *a, **kw: FakeResp(b'video-bytes')) + path = xtap_core.download_direct( + 'https://video.twimg.com/ext_tw_video/1/pu/vid/x.mp4', + '42', + str(tmp_path), + post_date='/tmp/pwn') + + assert os.path.dirname(path) == os.path.realpath(str(tmp_path)) + assert os.path.basename(path) == '42.mp4' + + def test_rejects_oversize_content_length_and_removes_part(self, tmp_path, monkeypatch): + class FakeResp(io.BytesIO): + headers = {'Content-Length': '1000000'} + + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + monkeypatch.setenv('XTAP_MAX_VIDEO_MB', '0.0001') # ~104 bytes + monkeypatch.setattr( + xtap_core._NO_REDIRECT_OPENER, 'open', + lambda *a, **kw: FakeResp(b'')) + + with pytest.raises(ValueError, match='XTAP_MAX_VIDEO_MB'): + xtap_core.download_direct( + 'https://video.twimg.com/ext_tw_video/1/pu/vid/x.mp4', + '42', + str(tmp_path)) + + assert not (tmp_path / '42.mp4').exists() + assert not (tmp_path / '42.mp4.part').exists() + + def test_aborts_midstream_oversize_and_removes_part(self, tmp_path, monkeypatch): + class FakeResp(io.BytesIO): + headers = {} + + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + monkeypatch.setenv('XTAP_MAX_VIDEO_MB', '0.0001') # ~104 bytes + monkeypatch.setattr( + xtap_core._NO_REDIRECT_OPENER, 'open', + lambda *a, **kw: FakeResp(b'x' * 200)) + + with pytest.raises(ValueError, match='XTAP_MAX_VIDEO_MB'): + xtap_core.download_direct( + 'https://video.twimg.com/ext_tw_video/1/pu/vid/x.mp4', + '42', + str(tmp_path)) + + assert not (tmp_path / '42.mp4').exists() + assert not (tmp_path / '42.mp4.part').exists() + + def test_stream_failure_removes_part(self, tmp_path, monkeypatch): + class BrokenResp: + headers = {} + + def __init__(self): + self.calls = 0 + + def read(self, _n=-1): + self.calls += 1 + if self.calls == 1: + return b'partial' + raise OSError('stream broke') + + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + monkeypatch.setattr( + xtap_core._NO_REDIRECT_OPENER, 'open', + lambda *a, **kw: BrokenResp()) + + with pytest.raises(OSError, match='stream broke'): + xtap_core.download_direct( + 'https://video.twimg.com/ext_tw_video/1/pu/vid/x.mp4', + '42', + str(tmp_path)) + + assert not (tmp_path / '42.mp4').exists() + assert not (tmp_path / '42.mp4.part').exists() + + def test_redirect_off_allowlist_is_blocked(self, tmp_path): + # A twimg.com URL that 302-redirects off-host must not be followed — + # the host check only validates the initial URL, so the no-redirect + # opener is the real guard. _NoRedirectHandler raises on any 3xx. + handler = xtap_core._NoRedirectHandler() + req = urllib.request.Request('https://video.twimg.com/x.mp4') + with pytest.raises(urllib.error.HTTPError): + handler.redirect_request( + req, io.BytesIO(b''), 302, + 'Found', {}, 'http://169.254.169.254/latest/meta-data/') + + def test_download_direct_uses_no_redirect_opener(self): + # Guard against a regression to bare urllib.request.urlopen. + import inspect + src = inspect.getsource(xtap_core.download_direct) + assert '_NO_REDIRECT_OPENER.open' in src + assert 'urllib.request.urlopen' not in src + + +class TestVideoMaxBytes: + """XTAP_MAX_VIDEO_MB boundary values: 0/negative disable the cap, + non-numeric falls back to the 500 MB default. A regression here silently + re-enables the cap for users who set 0, or disables it for everyone.""" + + def test_unset_uses_default(self, monkeypatch): + monkeypatch.delenv('XTAP_MAX_VIDEO_MB', raising=False) + assert xtap_core._video_max_bytes() == \ + xtap_core.DEFAULT_MAX_VIDEO_MB * 1024 * 1024 + + def test_zero_disables_cap(self, monkeypatch): + monkeypatch.setenv('XTAP_MAX_VIDEO_MB', '0') + assert xtap_core._video_max_bytes() is None + + def test_negative_disables_cap(self, monkeypatch): + monkeypatch.setenv('XTAP_MAX_VIDEO_MB', '-1') + assert xtap_core._video_max_bytes() is None + + def test_non_numeric_falls_back_to_default(self, monkeypatch, capsys): + monkeypatch.setenv('XTAP_MAX_VIDEO_MB', 'lots') + assert xtap_core._video_max_bytes() == \ + xtap_core.DEFAULT_MAX_VIDEO_MB * 1024 * 1024 + assert 'XTAP_MAX_VIDEO_MB' in capsys.readouterr().err + + def test_disabled_cap_allows_body_over_default_chunks(self, tmp_path, monkeypatch): + """With the cap disabled, download_direct must stream a body larger + than any Content-Length check would allow, not raise mid-stream.""" + monkeypatch.setenv('XTAP_MAX_VIDEO_MB', '0') + body = b'v' * (2 * xtap_core.VIDEO_CHUNK_SIZE + 17) + monkeypatch.setattr( + xtap_core._NO_REDIRECT_OPENER, 'open', + lambda *a, **kw: _FakeResponse(body, content_length=len(body))) + path = xtap_core.download_direct( + 'https://video.twimg.com/ext_tw_video/1/pu/vid/x.mp4', + '42', str(tmp_path)) + assert os.path.getsize(path) == len(body) + assert not os.path.exists(path + '.part') + + +class TestVideoDownloadValidation: + def test_start_download_rejects_offsite_tweet_url_before_registering(self, tmp_path): + with xtap_core._downloads_lock: + xtap_core._downloads.pop('bad-url', None) + + with pytest.raises(ValueError, match='tweetUrl'): + xtap_core.start_download( + 'bad-url', + 'http://169.254.169.254/latest/meta-data/', + '', + str(tmp_path)) + + with xtap_core._downloads_lock: + assert 'bad-url' not in xtap_core._downloads + + def test_ytdlp_output_template_stays_under_staging_dir(self, tmp_path, monkeypatch): + captured = {} + + class FakeProc: + def __init__(self, cmd, **_kwargs): + captured['cmd'] = cmd + self.stdout = iter([]) + self.returncode = 0 + + def wait(self): + return 0 + + download_id = 'safe-template' + with xtap_core._downloads_lock: + xtap_core._downloads[download_id] = { + 'status': 'downloading', + 'progress': None, + 'path': None, + 'error': None, + } + monkeypatch.setattr(xtap_core, '_ytdlp_path', '/usr/bin/yt-dlp') + monkeypatch.setattr(xtap_core.subprocess, 'Popen', FakeProc) + + try: + xtap_core._download_with_ytdlp( + download_id, + 'https://x.com/user/status/42', + str(tmp_path), + post_date='/tmp/pwn') + + output_template = captured['cmd'][captured['cmd'].index('-o') + 1] + staging_dir = os.path.realpath(str(tmp_path / '.downloading')) + assert os.path.commonpath([staging_dir, os.path.realpath(output_template)]) == staging_dir + assert os.path.basename(output_template).startswith('%(title)s [42]') + finally: + # A failed assertion must not leak the entry into shared module + # state for subsequent tests. + with xtap_core._downloads_lock: + xtap_core._downloads.pop(download_id, None) diff --git a/tests/test_xtap_daemon.py b/tests/test_xtap_daemon.py index cd61e18..8dd1741 100644 --- a/tests/test_xtap_daemon.py +++ b/tests/test_xtap_daemon.py @@ -2,8 +2,13 @@ import json import os +import platform +import signal +import socket +import subprocess import sys import threading +import time import pytest import urllib.request @@ -30,6 +35,18 @@ def _set_module_token(): xtap_daemon._token = old +@pytest.fixture(autouse=True) +def _isolate_default_output_dir(tmp_path, monkeypatch): + """Point the daemon's default output dir at a tmp dir. resolve_output_dir + lazily loads seen IDs for it, and tests must never touch the user's real + ~/Downloads/xtap (slow with a large archive; blocked by macOS TCC in + sandboxed runs).""" + default = tmp_path / 'default-out' + default.mkdir() + monkeypatch.setattr(xtap_daemon, 'DEFAULT_OUTPUT_DIR', str(default)) + monkeypatch.setattr(xtap_daemon, '_seen_ids_by_dir', {}) + + @pytest.fixture() def daemon_url(): """Start DaemonHandler on an ephemeral port and return its base URL.""" @@ -146,6 +163,60 @@ def test_oversized_content_length(self, daemon_url): assert 'too large' in body['error'].lower() conn.close() + def test_oversized_body_drained_so_413_arrives_cleanly(self, daemon_url, monkeypatch): + """An actual over-limit body (not just the header) must still get a + parseable 413 — the daemon drains the body so the socket close doesn't + RST the upload mid-stream and surface as a network error instead.""" + import http.client + monkeypatch.setattr(xtap_daemon, 'MAX_BODY_SIZE', 1000) + payload = b'{"tweets":[' + b'0' * 4000 + b']}' # well over 1000 + port = int(daemon_url.rsplit(':', 1)[1]) + conn = http.client.HTTPConnection('127.0.0.1', port) + conn.request('POST', '/tweets', body=payload, + headers={'Content-Type': 'application/json', + 'Authorization': f'Bearer {TEST_TOKEN}'}) + resp = conn.getresponse() # must not raise (no RST) + body = json.loads(resp.read()) + assert resp.status == 413 + assert 'too large' in body['error'].lower() + conn.close() + + def test_413_arrives_when_client_stalls_mid_drain(self, daemon_url, monkeypatch): + """_drain_body's idle-timeout escape: a client that declares an + oversized body but stops sending must still get its 413 within the + drain idle window (MAX_DRAIN_IDLE_TIMEOUT_S) instead of pinning the + handler thread until the full body arrives.""" + monkeypatch.setattr(xtap_daemon, 'MAX_BODY_SIZE', 1000) + port = int(daemon_url.rsplit(':', 1)[1]) + s = socket.create_connection(('127.0.0.1', port), timeout=5) + try: + s.sendall(b'POST /tweets HTTP/1.1\r\n' + b'Host: 127.0.0.1\r\n' + b'Authorization: Bearer ' + TEST_TOKEN.encode() + b'\r\n' + b'Content-Type: application/json\r\n' + b'Content-Length: 500000\r\n' + b'\r\n' + b'partial-body-then-silence') + s.settimeout(5) + start = time.time() + data = s.recv(65536) + elapsed = time.time() - start + assert b'413' in data.split(b'\r\n', 1)[0], \ + f'expected a 413 status line, got: {data[:80]!r}' + # Idle timeout is 0.25s; anything near the 5s recv timeout means + # the drain blocked instead of bailing on the stalled client. + assert elapsed < 2, \ + f'413 took {elapsed:.1f}s — drain idle timeout did not fire' + finally: + s.close() + + +class TestServerShutdownConfig: + def test_handler_threads_not_daemonized(self): + """daemon_threads must be False so server_close() joins in-flight + handlers instead of the OS killing them mid-write at exit.""" + assert xtap_daemon.XtapHTTPServer.daemon_threads is False + # --------------------------------------------------------------------------- # Tests — Auth @@ -165,9 +236,14 @@ def test_wrong_token_rejected(self, daemon_url): assert status == 401 assert body['error'] == 'Unauthorized' - def test_content_length_checked_before_auth(self, daemon_url): - """Oversized request should be rejected with 413 even without auth.""" + def test_auth_checked_before_oversized_body_drain(self, daemon_url, monkeypatch): + """Unauthenticated oversized requests must not force body draining.""" import http.client + drained = [] + monkeypatch.setattr( + xtap_daemon.DaemonHandler, + '_drain_body', + lambda _self, n: drained.append(n)) huge_length = xtap_daemon.MAX_BODY_SIZE + 1 conn = http.client.HTTPConnection('127.0.0.1', int(daemon_url.rsplit(':', 1)[1])) conn.putrequest('POST', '/tweets') @@ -177,8 +253,9 @@ def test_content_length_checked_before_auth(self, daemon_url): conn.endheaders() resp = conn.getresponse() body = json.loads(resp.read()) - # Should get 413, not 401 - assert resp.status == 413 + assert resp.status == 401 + assert body['error'] == 'Unauthorized' + assert drained == [] conn.close() @@ -331,6 +408,29 @@ def test_dump_rejects_dotdot_filename(self, daemon_url): assert 'Invalid dump filename' in body['error'] +class TestDownloadVideo: + def test_rejects_offsite_tweet_url_before_starting_download(self, daemon_url, monkeypatch): + called = [] + monkeypatch.setattr( + xtap_daemon, + 'start_download', + lambda *args, **kwargs: called.append((args, kwargs))) + + status, body = _post( + daemon_url, '/download-video', + body={ + 'tweetUrl': 'http://169.254.169.254/latest/meta-data/', + 'directUrl': '', + }, + token=TEST_TOKEN, + ) + + assert status == 400 + assert body['ok'] is False + assert 'tweetUrl' in body['error'] + assert called == [] + + # --------------------------------------------------------------------------- # Tests — Concurrency (issue #8) # --------------------------------------------------------------------------- @@ -436,3 +536,187 @@ def writer(): del xtap_core._downloads[download_id] assert not errors, f'Found incoherent reads: {errors[:5]}' + + +# --------------------------------------------------------------------------- +# Tests — signal shutdown (the handler must not deadlock serve_forever) +# --------------------------------------------------------------------------- + +def _start_daemon(tmp_path, extra_env=None): + """Launch xtap_daemon.py as a subprocess and wait for it to listen. + + Returns (proc, port, stderr_lines). Caller must kill proc if still alive. + """ + port = _free_port() + xtap_dir = tmp_path / '.xtap' + if not xtap_dir.exists(): + xtap_dir.mkdir() + (xtap_dir / 'secret').write_text('test-token') + env = {**os.environ, + 'HOME': str(tmp_path), + 'XTAP_DAEMON_PORT': str(port), + 'XTAP_OUTPUT_DIR': str(tmp_path / 'out'), + **(extra_env or {})} + daemon_py = os.path.join( + os.path.dirname(os.path.abspath(xtap_daemon.__file__)), 'xtap_daemon.py') + proc = subprocess.Popen([sys.executable, daemon_py], env=env, + stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, + text=True) + stderr_lines = [] + threading.Thread( + target=lambda: stderr_lines.extend(proc.stderr), daemon=True).start() + deadline = time.time() + 10 + while not any('Listening' in line for line in stderr_lines): + if proc.poll() is not None: + pytest.fail(f'daemon exited early: {"".join(stderr_lines)}') + if time.time() > deadline: + proc.kill() + pytest.fail(f'daemon never started: {"".join(stderr_lines)}') + time.sleep(0.05) + return proc, port, stderr_lines + + +class TestSignalShutdown: + + @pytest.mark.skipif(platform.system() == 'Windows', reason='POSIX signals only') + def test_sigterm_exits_promptly(self, tmp_path): + """server.shutdown() invoked directly inside a signal handler runs on + the same thread as serve_forever() and deadlocks — launchd/systemd + then SIGKILLs mid-write. The daemon must exit cleanly within 5s.""" + proc, _port, _stderr = _start_daemon(tmp_path) + try: + proc.send_signal(signal.SIGTERM) + try: + proc.wait(timeout=5) + except subprocess.TimeoutExpired: + pytest.fail('daemon did not exit within 5s of SIGTERM (shutdown deadlock)') + finally: + if proc.poll() is None: + proc.kill() + proc.wait(timeout=5) + + @pytest.mark.skipif(platform.system() == 'Windows', reason='POSIX signals only') + def test_idle_connection_does_not_block_shutdown(self, tmp_path): + """A peer that opens a connection but never sends a request parks a + non-daemon handler thread; with daemon_threads=False, server_close() + joins it on shutdown. Without a socket timeout on DaemonHandler that + join blocks until launchd/systemd SIGKILLs mid-write. The handler + timeout must let the idle thread expire so shutdown still completes.""" + # Short conn timeout so the test is fast (real default 15s). Long + # shutdown grace so this test pins the SOCKET-timeout path, not the + # force-exit backstop. + proc, port, _stderr = _start_daemon( + tmp_path, {'XTAP_CONN_TIMEOUT_S': '2', 'XTAP_SHUTDOWN_GRACE_S': '30'}) + idle = None + try: + # Open a connection and send nothing — the handler thread blocks in + # readline() waiting for a request line that never comes. + idle = socket.create_connection(('127.0.0.1', port), timeout=5) + time.sleep(0.2) # let the daemon accept + spawn the handler thread + + proc.send_signal(signal.SIGTERM) + try: + # 2s conn timeout + generous margin for the shutdown join. + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + pytest.fail('idle connection blocked shutdown past the socket ' + 'timeout — DaemonHandler.timeout is not taking effect') + finally: + if idle is not None: + idle.close() + if proc.poll() is None: + proc.kill() + proc.wait(timeout=5) + + @pytest.mark.skipif(platform.system() == 'Windows', reason='POSIX signals only') + def test_trickle_connection_does_not_block_shutdown(self, tmp_path): + """The per-read socket timeout does NOT bound a peer that trickles one + byte per read window — that thread stays alive indefinitely. The + SHUTDOWN_GRACE_S backstop must force-exit the daemon anyway instead of + letting server_close() join the trickle thread until SIGKILL.""" + proc, port, stderr_lines = _start_daemon( + tmp_path, {'XTAP_CONN_TIMEOUT_S': '1', 'XTAP_SHUTDOWN_GRACE_S': '2'}) + stop_trickle = threading.Event() + trickle_sock = None + try: + trickle_sock = socket.create_connection(('127.0.0.1', port), timeout=5) + + def trickle(sock): + # One header byte every 0.3s — each byte resets the 1s + # per-read timeout, so the handler thread never times out. + sock.sendall(b'P') + while not stop_trickle.wait(0.3): + try: + sock.sendall(b'X') + except OSError: + return + + t = threading.Thread(target=trickle, args=(trickle_sock,), daemon=True) + t.start() + time.sleep(0.5) # ensure the handler thread is mid-read + + proc.send_signal(signal.SIGTERM) + try: + # 2s grace + margin. Without the backstop this hangs forever. + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + pytest.fail('trickle connection blocked shutdown — the ' + 'SHUTDOWN_GRACE_S force-exit backstop is not working') + assert any('grace' in line for line in stderr_lines), \ + 'force-exit path must log that the grace period expired' + finally: + stop_trickle.set() + if trickle_sock is not None: + trickle_sock.close() + if proc.poll() is None: + proc.kill() + proc.wait(timeout=5) + + +def _free_port(): + s = socket.socket() + s.bind(('127.0.0.1', 0)) + port = s.getsockname()[1] + s.close() + return port + + +# --------------------------------------------------------------------------- +# Tests — article local_path sanitization must not depend on image toggle +# --------------------------------------------------------------------------- + +class TestArticlePathSanitization: + + def test_unsafe_local_path_stripped_without_image_download( + self, daemon_url, tmp_path, monkeypatch): + """Traversal-laden article local_path values must be stripped from the + JSONL even when image_download is off — downstream consumers join + out_dir + local_path.""" + monkeypatch.setattr(xtap_core, '_ALLOWED_ROOTS', + (os.path.realpath(str(tmp_path)),)) + out_dir = str(tmp_path / 'out') + body = { + 'outputDir': out_dir, + 'tweets': [{ + 'id': '777000777', + 'text': 'article', + 'is_article': True, + 'article': { + 'text': 'before\n![evil](../../../../etc/evil.jpg)\nafter', + 'media': [{ + 'url': 'https://pbs.twimg.com/media/abc.jpg', + 'local_path': '../../../../etc/evil.jpg', + }], + }, + }], + } + status, resp = _post(daemon_url, '/tweets', body, token=TEST_TOKEN) + assert status == 200 + assert resp['ok'] is True + files = list((tmp_path / 'out').glob('tweets-*.jsonl')) + assert len(files) == 1 + line = json.loads(files[0].read_text(encoding='utf-8').strip()) + assert 'local_path' not in line['article']['media'][0] + assert '../../../../etc/evil.jpg' not in line['article']['text'] + assert 'before' in line['article']['text'] + assert 'after' in line['article']['text'] diff --git a/tests/test_xtap_host.py b/tests/test_xtap_host.py index ea1b06a..4ba8039 100644 --- a/tests/test_xtap_host.py +++ b/tests/test_xtap_host.py @@ -141,3 +141,45 @@ def test_missing_secret_returns_error(self, monkeypatch, tmp_path): resp = json.loads(out_buf.read(resp_len)) assert resp['ok'] is False assert 'not found' in resp['error'] + + +# --------------------------------------------------------------------------- +# Oversized message recovery +# --------------------------------------------------------------------------- + +class TestOversizedMessageRecovery: + + def test_stream_realigned_after_oversized_message(self, monkeypatch, tmp_path): + """An oversized frame's payload must be drained from the pipe before + continuing — otherwise the next read interprets payload bytes as a + length header and the stream never recovers.""" + xtap_dir = tmp_path / '.xtap' + xtap_dir.mkdir() + (xtap_dir / 'secret').write_text('tok') + monkeypatch.setattr(xtap_host, 'XTAP_SECRET', str(xtap_dir / 'secret')) + monkeypatch.setattr(xtap_host, 'XTAP_DIR', str(xtap_dir)) + + big = b'x' * (xtap_host.MAX_MESSAGE_BYTES + 1) + raw = (struct.pack(' body }; +} + +function tick() { + return new Promise(r => setTimeout(r, 0)); +} + +function setup() { + const sessionStore = createMockStorage(); + const localStore = createMockStorage(); + let messageListener = null; + const fetchHolder = { + impl: async () => okResponse(), + }; + const alarms = { + created: [], + cleared: [], + listener: null, + create(name, info) { this.created.push({ name, info }); }, + clear(name) { this.cleared.push(name); }, + onAlarm: null, // set below so addListener can reach `alarms` + }; + alarms.onAlarm = { addListener: (fn) => { alarms.listener = fn; } }; + + const sandbox = { + console: { log() {}, warn() {}, error() {} }, + setTimeout: globalThis.setTimeout, + clearTimeout: globalThis.clearTimeout, + AbortController: globalThis.AbortController, + AbortSignal: globalThis.AbortSignal, + fetch: (...args) => fetchHolder.impl(...args), + extractTweets: () => [], + dedupTweet, + chrome: { + runtime: { + getManifest: () => ({}), // no update_url → isDevMode = true + connectNative() { throw new Error('not available'); }, + onMessage: { addListener(fn) { messageListener = fn; } }, + lastError: null, + }, + storage: { + session: sessionStore, + local: localStore, + }, + action: { + setBadgeText() {}, + setBadgeBackgroundColor() {}, + }, + alarms, + }, + }; + + vm.runInNewContext(testSource, sandbox); + + const env = sandbox._internals; + env.sessionStore = sessionStore; + env.localStore = localStore; + env.fetchHolder = fetchHolder; + env.alarms = alarms; + env.sendMessage = (msg) => new Promise((resolve) => { + const returned = messageListener(msg, {}, resolve); + if (returned !== true) resolve(undefined); + }); + return env; +} + +const photoTweet = (id) => ({ + id, + text: 'pic', + media: [{ type: 'photo', url: 'https://pbs.twimg.com/media/x.jpg:orig' }], +}); + +// --------------------------------------------------------------------------- +// Daemon rejection handling +// --------------------------------------------------------------------------- + +describe('daemon rejection handling', () => { + it('a 401 response resets transport so credentials are re-probed', async () => { + const env = setup(); + env.transport = 'http'; + env.httpToken = 'stale-token'; + env.httpPort = 17381; + // Daemon answers with valid JSON but HTTP 401 (token rotated by reinstall) + env.fetchHolder.impl = async () => okResponse({ ok: false, error: 'Unauthorized' }, 401); + env.buffer = [{ id: '1', text: 'one' }]; + + await env.flush(); + + assert.equal(env.buffer.length, 1, 'batch must be preserved for retry'); + assert.equal(env.transport, 'none', + 'transport must reset on auth failure — otherwise flush retries the same ' + + 'stale token forever and never refreshes via the native host'); + }); + + it('a 500 response keeps the batch and keeps the transport', async () => { + const env = setup(); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + env.fetchHolder.impl = async () => okResponse({ ok: false, error: 'disk full' }, 500); + env.buffer = [{ id: '1', text: 'one' }]; + + await env.flush(); + + assert.equal(env.buffer.length, 1); + // Daemon is alive (it answered) — a transient server error is not a + // transport failure, so no costly reprobe cycle. + assert.equal(env.transport, 'http'); + }); +}); + +// --------------------------------------------------------------------------- +// Image backfill accounting +// --------------------------------------------------------------------------- + +describe('image backfill accounting', () => { + it('posts backfill duplicates without incrementing capture counters', () => { + const env = setup(); + env.imageDownload = true; + env.seenIds = new Set(['1']); + + env.enqueueTweets([photoTweet('1')], 'HomeTimeline'); + + assert.equal(env.buffer.length, 1, 'duplicate photo tweet must still reach the daemon'); + assert.equal(env.buffer[0].id, '1'); + assert.equal(env.sessionCount, 0); + assert.equal(env.allTimeCount, 0); + assert.ok(env.alarms.created.some(a => a.name === 'xtap-flush'), + 'backfill-only batches still need a flush alarm'); + + const event = env.traceEvents.find(e => e.tweetId === '1'); + assert.equal(event.status, 'IMAGE_BACKFILL'); + }); + + it('strips the internal backfill marker before POSTing to the daemon', async () => { + const env = setup(); + env.imageDownload = true; + env.seenIds = new Set(['1']); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + let posted = null; + env.fetchHolder.impl = async (_url, opts) => { + posted = JSON.parse(opts.body); + return okResponse({ ok: true }); + }; + + env.enqueueTweets([photoTweet('1')], 'HomeTimeline'); + await env.flush(); + + assert.equal(posted.tweets.length, 1); + assert.equal(posted.tweets[0].id, '1'); + assert.equal(posted.tweets[0].__xtap_image_backfill, undefined); + }); + + it('drops image backfill before real buffered tweets when the buffer is full', () => { + const env = setup(); + env.imageDownload = true; + env.seenIds = new Set(['1']); + env.buffer = Array.from({ length: 2000 }, (_, i) => ({ id: `real-${i}`, text: 'real' })); + + env.enqueueTweets([photoTweet('1')], 'HomeTimeline'); + + assert.equal(env.buffer.length, 2000); + assert.equal(env.buffer[0].id, 'real-0'); + assert.equal(env.buffer.some(t => t.id === '1'), false); + assert.equal(env.imageCheckedIds.has('1'), false, + 'dropped backfill should be retryable later in the same service-worker session'); + assert.ok(env.traceEvents.some(e => e.status === 'BUFFER_OVERFLOW')); + }); + + it('counts fresh captures while image download is enabled', () => { + const env = setup(); + env.imageDownload = true; + + env.enqueueTweets([photoTweet('1')], 'HomeTimeline'); + + assert.equal(env.buffer.length, 1); + assert.equal(env.sessionCount, 1); + assert.equal(env.allTimeCount, 1); + + const event = env.traceEvents.find(e => e.tweetId === '1'); + assert.equal(event.status, 'ACCEPTED'); + }); + + it('lets one article duplicate enrich a previous non-article capture', () => { + const env = setup(); + env.imageDownload = true; + env.seenIds = new Set(['1']); + + env.enqueueTweets([{ ...photoTweet('1'), is_article: true }], 'TweetResultByRestId'); + + assert.equal(env.buffer.length, 1); + assert.equal(env.sessionCount, 1); + assert.equal(env.allTimeCount, 1); + + const event = env.traceEvents.find(e => e.tweetId === '1'); + assert.equal(event.status, 'ACCEPTED'); + }); + + it('deduplicates repeated full article captures after the enrichment write', () => { + const env = setup(); + env.imageDownload = true; + env.seenIds = new Set(['1']); + + env.enqueueTweets([{ ...photoTweet('1'), is_article: true }], 'TweetResultByRestId'); + env.enqueueTweets([{ ...photoTweet('1'), is_article: true }], 'TweetResultByRestId'); + + assert.equal(env.buffer.length, 1); + assert.equal(env.sessionCount, 1); + assert.equal(env.allTimeCount, 1); + assert.ok(env.traceEvents.some(e => e.tweetId === '1' && e.status === 'DEDUPLICATED')); + }); + + it('clears imageCheckedIds when image download flips from off to on', async () => { + const env = setup(); + env.imageDownload = false; + env.imageCheckedIds = new Set(['1', '2']); + + const resp = await env.sendMessage({ type: 'SET_IMAGE_DOWNLOAD', imageDownload: true }); + + assert.equal(resp.imageDownload, true); + assert.equal(env.imageDownload, true); + assert.equal(env.imageCheckedIds.size, 0); + }); +}); + +// --------------------------------------------------------------------------- +// Flush in-flight durability +// --------------------------------------------------------------------------- + +describe('flush in-flight durability', () => { + it('keeps the in-flight batch in persisted state until the daemon acks', async () => { + const env = setup(); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + const posted = []; + let resolveFetch; + env.fetchHolder.impl = (url, opts) => { + posted.push(JSON.parse(opts.body).tweets.map(t => t.id)); + return new Promise(r => { resolveFetch = r; }); + }; + env.buffer = [{ id: '1', text: 'one' }]; + + const flushP = env.flush(); + await tick(); + + // A concurrent GRAPHQL_RESPONSE handler enqueues + persists while the + // POST is in flight (the common case while scrolling). + env.enqueueTweets([{ id: '2', text: 'two' }], 'test'); + await env.saveState(); + + // If the SW were killed right now, persisted state must still contain + // tweet 1 — its WAL entry was already cleared and seenIds blocks recapture. + const persisted = await env.sessionStore.get(null); + assert.ok(persisted.tweetBuffer.some(t => t.id === '1'), + 'in-flight batch missing from persisted buffer — SW death here loses it'); + + // Ack the first POST; subsequent POSTs (draining tweet 2) ack immediately. + const firstResolve = resolveFetch; + env.fetchHolder.impl = (url, opts) => { + posted.push(JSON.parse(opts.body).tweets.map(t => t.id)); + return Promise.resolve(okResponse()); + }; + firstResolve(okResponse()); + await flushP; + + // Drained exactly once each, nothing left behind. + assert.deepEqual(posted, [['1'], ['2']]); + assert.equal(env.buffer.length, 0); + const after = await env.sessionStore.get(null); + assert.deepEqual(after.tweetBuffer, []); + }); + + it('does not double-send when flush is invoked concurrently', async () => { + const env = setup(); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + let calls = 0; + const resolvers = []; + env.fetchHolder.impl = () => { calls++; return new Promise(r => resolvers.push(r)); }; + env.buffer = [{ id: '1', text: 'one' }]; + + const p1 = env.flush(); + await tick(); + const p2 = env.flush(); + await tick(); + + assert.equal(calls, 1, 'second flush must not re-send the in-flight batch'); + + resolvers.forEach(r => r(okResponse())); + await p1; + await p2; + assert.equal(env.buffer.length, 0); + }); + + it('splits a backlog larger than MAX_TWEETS_PER_POST into capped POSTs', async () => { + const env = setup(); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + const posted = []; + env.fetchHolder.impl = async (url, opts) => { + posted.push(JSON.parse(opts.body).tweets.length); + return okResponse(); + }; + env.buffer = Array.from({ length: 250 }, (_, i) => ({ id: String(i), text: 't' })); + + await env.flush(); + + assert.deepEqual(posted, [200, 50], 'backlog must drain in <=200-tweet POSTs'); + assert.equal(env.buffer.length, 0); + }); + + it('splits the batch on 413 instead of wedging the queue', async () => { + const env = setup(); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + const posted = []; + // Daemon accepts at most 2 tweets per POST (simulated byte limit). + env.fetchHolder.impl = async (url, opts) => { + const n = JSON.parse(opts.body).tweets.length; + posted.push(n); + if (n > 2) return okResponse({ ok: false, error: 'Payload too large' }, 413); + return okResponse(); + }; + env.buffer = Array.from({ length: 5 }, (_, i) => ({ id: String(i), text: 't' })); + + await env.flush(); + + assert.equal(env.buffer.length, 0, 'queue must drain despite 413s'); + assert.deepEqual(posted, [5, 2, 2, 1], 'batch must halve on 413, then drain'); + }); + + it('drops a single tweet that alone exceeds the body limit', async () => { + const env = setup(); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + env.fetchHolder.impl = async (url, opts) => { + const ids = JSON.parse(opts.body).tweets.map(t => t.id); + if (ids.includes('huge')) return okResponse({ ok: false, error: 'Payload too large' }, 413); + return okResponse(); + }; + env.buffer = [{ id: 'huge', text: 'giant article' }, { id: 'normal', text: 'ok' }]; + + await env.flush(); + + assert.equal(env.buffer.length, 0); + const dropped = env.traceEvents.filter(e => e.status === 'DROPPED_OVERSIZED'); + assert.equal(dropped.length, 1, 'oversized tweet must be dropped with a trace event'); + assert.equal(dropped[0].tweetId, 'huge'); + }); + + it('keeps buffer order when the send fails mid-scroll', async () => { + const env = setup(); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + let rejectFetch; + env.fetchHolder.impl = () => new Promise((_, rej) => { rejectFetch = rej; }); + env.buffer = [{ id: '1', text: 'one' }]; + + const flushP = env.flush(); + await tick(); + env.enqueueTweets([{ id: '2', text: 'two' }], 'test'); + + rejectFetch(new Error('connection refused')); + await flushP; + + assert.deepEqual(env.buffer.map(t => t.id), ['1', '2']); + }); + + it('overflow eviction during an in-flight POST must not discard unsent tweets', async () => { + const env = setup(); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + const posted = []; + let resolveFirst; + // Hold the first POST open so a buffer overflow can race it. + env.fetchHolder.impl = (url, opts) => { + posted.push(JSON.parse(opts.body).tweets.map(t => t.id)); + return new Promise(r => { resolveFirst = r; }); + }; + // Buffer sits at the overflow cap; the first 200-tweet POST covers t0..t199. + env.buffer = Array.from({ length: 2000 }, (_, i) => ({ id: `t${i}`, text: 'x' })); + + const flushP = env.flush(); + await tick(); + + // Remaining POSTs (draining the rest of the backlog) ack immediately. + env.fetchHolder.impl = (url, opts) => { + posted.push(JSON.parse(opts.body).tweets.map(t => t.id)); + return Promise.resolve(okResponse()); + }; + // 50 fresh tweets arrive mid-POST → overflow eviction shifts 50 (t0..t49) + // off the buffer front, sliding it out from under the in-flight batch. + env.enqueueTweets(Array.from({ length: 50 }, (_, i) => ({ id: `n${i}`, text: 'x' })), 'test'); + + resolveFirst(okResponse()); + await flushP; + + const sent = new Set(posted.flat()); + // t200..t249 were never in the acked batch and were never evicted. A + // positional splice(0, batch.length) after the ack would delete them (the + // buffer front shifted by 50), silently losing 50 never-sent tweets. + for (let i = 200; i < 250; i++) { + assert.ok(sent.has(`t${i}`), `t${i} was discarded unsent by the ack removal`); + } + assert.equal(env.buffer.length, 0, 'queue must fully drain'); + }); +}); + +// --------------------------------------------------------------------------- +// Download status cleanup (daemon restart loses download state -> 'unknown') +// --------------------------------------------------------------------------- + +describe('download status cleanup', () => { + function statusEnv(daemonStatus) { + const env = setup(); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + env.fetchHolder.impl = async () => okResponse({ ok: true, status: daemonStatus }); + env.activeDownloads.set('tweet-1', 'dl-1'); + return env; + } + + for (const finished of ['done', 'error', 'unknown']) { + it(`'${finished}' clears the activeDownloads entry`, async () => { + const env = statusEnv(finished); + const resp = await env.sendMessage({ type: 'DOWNLOAD_STATUS', downloadId: 'dl-1' }); + assert.equal(resp.status, finished); + assert.equal(env.activeDownloads.size, 0, + `'${finished}' must stop the popup from resuming a dead download`); + }); + } + + it("'downloading' keeps the activeDownloads entry", async () => { + const env = statusEnv('downloading'); + const resp = await env.sendMessage({ type: 'DOWNLOAD_STATUS', downloadId: 'dl-1' }); + assert.equal(resp.status, 'downloading'); + assert.equal(env.activeDownloads.get('tweet-1'), 'dl-1', + 'an in-progress download must stay resumable'); + }); + + it('does not clear entries for a different downloadId', async () => { + const env = statusEnv('done'); + env.activeDownloads.set('tweet-2', 'dl-2'); + await env.sendMessage({ type: 'DOWNLOAD_STATUS', downloadId: 'dl-1' }); + assert.equal(env.activeDownloads.has('tweet-1'), false); + assert.equal(env.activeDownloads.get('tweet-2'), 'dl-2'); + }); +}); + +// --------------------------------------------------------------------------- +// Debug log buffering (cap + stringify fallback) +// --------------------------------------------------------------------------- + +describe('debug log buffering', () => { + it('caps logBuffer at MAX_LOG_BUFFER, dropping oldest lines', () => { + const env = setup(); + env.debugLogging = true; + for (let i = 0; i < 5050; i++) env.debugLog('LOG', [`line-${i}`]); + assert.equal(env.logBuffer.length, 5000, + 'an unreachable daemon must not grow the log buffer unboundedly'); + assert.ok(env.logBuffer[0].includes('line-50'), 'oldest lines are dropped first'); + assert.ok(env.logBuffer[4999].includes('line-5049'), 'newest lines are kept'); + }); + + it('falls back to String() when an argument cannot be JSON.stringified', () => { + const env = setup(); + env.debugLogging = true; + const circular = {}; + circular.self = circular; + env.debugLog('LOG', [circular]); + assert.equal(env.logBuffer.length, 1, 'a circular argument must not throw'); + assert.ok(env.logBuffer[0].includes('[object Object]'), + 'unstringifiable args fall back to String(a)'); + }); + + it('buffers nothing while debug logging is off', () => { + const env = setup(); + env.debugLog('LOG', ['dropped']); + assert.equal(env.logBuffer.length, 0); + }); +}); + +// --------------------------------------------------------------------------- +// Flush alarm (MV3: setTimeout dies with the SW; chrome.alarms survives) +// --------------------------------------------------------------------------- + +describe('flush alarm', () => { + it('schedules an alarm when tweets are buffered', () => { + const env = setup(); + env.enqueueTweets([{ id: '1', text: 'one' }], 'test'); + assert.ok(env.alarms.created.some(a => a.name === 'xtap-flush'), + 'buffered tweets need a chrome.alarms backstop — the setTimeout flush ' + + 'timer dies with the service worker'); + }); + + it('the alarm listener flushes the restored buffer', async () => { + const env = setup(); + env.readyResolve(); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + let calls = 0; + env.fetchHolder.impl = async () => { calls++; return okResponse(); }; + env.buffer = [{ id: '1', text: 'one' }]; + + assert.ok(env.alarms.listener, 'no onAlarm listener registered at top level'); + env.alarms.listener({ name: 'xtap-flush' }); + await tick(); + await tick(); + + assert.equal(calls, 1, 'alarm did not trigger a flush'); + assert.equal(env.buffer.length, 0); + }); + + it('the alarm listener clears the alarm once the buffer is drained', async () => { + const env = setup(); + env.readyResolve(); + env.transport = 'http'; + env.httpToken = 't'; + env.httpPort = 17381; + env.buffer = []; + + assert.ok(env.alarms.listener, 'no onAlarm listener registered at top level'); + env.alarms.listener({ name: 'xtap-flush' }); + await tick(); + await tick(); + + assert.ok(env.alarms.cleared.includes('xtap-flush'), + 'a drained buffer must clear the alarm so the SW stops being woken'); + }); +}); diff --git a/tests/tweet-parser.test.mjs b/tests/tweet-parser.test.mjs index 199fae2..eb2e3f8 100644 --- a/tests/tweet-parser.test.mjs +++ b/tests/tweet-parser.test.mjs @@ -1009,3 +1009,110 @@ describe('normalizeTweet URLs', () => { assert.equal(t.urls[0].shortened, 'https://t.co/abc'); }); }); + +// --------------------------------------------------------------------------- +// Quoted tweets — the inline quoted payload must be captured, not just its ID +// --------------------------------------------------------------------------- + +function makeQuotedRawTweet() { + return makeRawTweet({ + rest_id: '200', + core: { + user_results: { + result: { + rest_id: 'u2', + core: { screen_name: 'quoteduser', name: 'Quoted User' }, + legacy: { followers_count: 7, verified: false }, + }, + }, + }, + legacy: { + id_str: '200', + user_id_str: 'u2', + full_text: 'the quoted text', + conversation_id_str: '200', + }, + }); +} + +describe('quoted tweet extraction', () => { + it('extracts the quoted tweet alongside the quoting tweet', () => { + const quoting = makeRawTweet({ + legacy: { quoted_status_id_str: '200' }, + quoted_status_result: { result: makeQuotedRawTweet() }, + }); + const tweets = extractTweets('HomeTimeline', makeTimelineResponse(quoting)); + + assert.equal(tweets.length, 2); + const main = tweets.find(t => t.id === '100'); + const quoted = tweets.find(t => t.id === '200'); + assert.ok(main); + assert.ok(quoted, 'inline quoted tweet payload was discarded'); + assert.equal(main.quoted_tweet_id, '200'); + assert.equal(quoted.text, 'the quoted text'); + assert.equal(quoted.author.username, 'quoteduser'); + }); + + it('unwraps TweetWithVisibilityResults-wrapped quoted tweets', () => { + const quoting = makeRawTweet({ + legacy: { quoted_status_id_str: '200' }, + quoted_status_result: { + result: { __typename: 'TweetWithVisibilityResults', tweet: makeQuotedRawTweet() }, + }, + }); + const tweets = extractTweets('HomeTimeline', makeTimelineResponse(quoting)); + assert.equal(tweets.length, 2); + assert.ok(tweets.some(t => t.id === '200')); + }); + + it('ignores tombstoned quoted tweets', () => { + const quoting = makeRawTweet({ + legacy: { quoted_status_id_str: '200' }, + quoted_status_result: { result: { __typename: 'TweetTombstone' } }, + }); + const tweets = extractTweets('HomeTimeline', makeTimelineResponse(quoting)); + assert.equal(tweets.length, 1); + assert.equal(tweets[0].id, '100'); + }); + + it('extracts the quoted tweet for TweetResultByRestId responses', () => { + const quoting = makeRawTweet({ + legacy: { quoted_status_id_str: '200' }, + quoted_status_result: { result: makeQuotedRawTweet() }, + }); + const data = { data: { tweetResult: { result: quoting } } }; + const tweets = extractTweets('TweetResultByRestId', data); + assert.equal(tweets.length, 2); + assert.ok(tweets.some(t => t.id === '200')); + }); + + it('extracts the quoted tweet carried by a retweeted original', () => { + const original = makeRawTweet({ + rest_id: '300', + legacy: { id_str: '300', quoted_status_id_str: '200' }, + quoted_status_result: { result: makeQuotedRawTweet() }, + }); + const rt = makeRawTweet({ + legacy: { retweeted_status_result: { result: original } }, + }); + const tweets = extractTweets('HomeTimeline', makeTimelineResponse(rt)); + assert.ok(tweets.some(t => t.id === '200'), + 'quoted payload on the retweeted original was discarded'); + }); + + it('does not emit the quoted tweet twice when wrapper and original both carry it', () => { + const original = makeRawTweet({ + rest_id: '300', + legacy: { id_str: '300' }, + quoted_status_result: { result: makeQuotedRawTweet() }, + }); + const rt = makeRawTweet({ + quoted_status_result: { result: makeQuotedRawTweet() }, + legacy: { retweeted_status_result: { result: original } }, + }); + const tweets = extractTweets('HomeTimeline', makeTimelineResponse(rt)); + assert.equal(tweets.filter(t => t.id === '200').length, 1, + 'same quoted tweet must be emitted once per entry'); + }); +}); +