Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,4 @@ test-results/

# Docs
docs/
.worktrees/
49 changes: 48 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ Extract videos from X (formerly Twitter) tweets.
- ✅ Extract videos from public X/Twitter tweets
- ✅ Supports multiple formats (mp4, webm, gif, etc.)
- ✅ Automatic format selection (highest quality)
- ✅ Download videos directly or just get of URL
- ✅ Download videos directly or just get the URL
- ✅ Clip videos to a specific time range (`--from` and `--to`)
- ⚠️ Downloading videos from private tweets (experimental alpha features)
- ❌ Windows support

Expand Down Expand Up @@ -59,6 +60,11 @@ x-dl https://x.com/user/status/123456
- mp4/webm/gif files: direct download
- HLS (m3u8) playlists: downloads via ffmpeg to produce mp4
- If direct download fails with 401/403 auth errors and `--profile` is used, automatically retries using authenticated Playwright requests
- **Clipping:**
- `--from` and `--to` (MM:SS format) trim videos to a specific time range
- HLS streams are clipped during download with ffmpeg re-encoding
- MP4 streams download full video, then clip locally
- Clipped files get a `_clip` suffix in the filename
- **Auth:** with `--profile`, Playwright reuses cookies/session from a persistent profile directory
- **ffmpeg:** checked at runtime and auto-installed when possible

Expand Down Expand Up @@ -158,6 +164,8 @@ x-dl install --with-deps
| `--headed` | Show browser window for debugging |
| `--profile [dir]` | Use a persistent browser profile for authenticated extraction (default: `~/.x-dl-profile`) |
| `--login` | Open X in a persistent profile and wait for you to log in |
| `--from <MM:SS>` | Clip start time in minutes and seconds (e.g. `00:30`) |
| `--to <MM:SS>` | Clip end time in minutes and seconds (e.g. `01:30`) |
| `--help, -h` | Show help message |

**Note:** The `-o` option accepts any file extension. If you specify a path with an extension (e.g., `video.mp4`, `video.webm`), that format will be used. Otherwise, the format is auto-detected from the extracted video.
Expand Down Expand Up @@ -198,6 +206,17 @@ x-dl --profile ~/.x-dl-profile https://x.com/user/status/123456
x-dl --timeout 60 https://x.com/user/status/123456
```

**Clip a video to a specific time range:**
```bash
# Download only the 30s–90s portion of a video
x-dl --from 00:30 --to 01:30 https://x.com/user/status/123456

# Download from 1 minute to the end
x-dl --from 01:00 https://x.com/user/status/123456
```

Clipped files are saved with a `_clip` suffix, e.g. `username_123456_clip.mp4`. Both `--from` and `--to` are optional — omitting `--from` starts from the beginning, omitting `--to` runs to the end.

## Output

When extracting a video, the tool will:
Expand Down Expand Up @@ -239,8 +258,36 @@ When extracting a video, the tool will:
✅ Video saved to: ~/Downloads/Remotion_2013626968386765291.mp4
```

### Example Output with Clipping

```
🎬 x-dl - X/Twitter Video Extractor

🔍 Checking for Playwright (Chromium)...
✅ Playwright Chromium is ready
🔍 Checking for ffmpeg...
✅ ffmpeg is ready

🎬 Extracting video from: https://x.com/Remotion/status/2013626968386765291
📝 Tweet: @Remotion (ID: 2013626968386765291)
🌐 Opening tweet in browser...
✅ Page loaded
🔍 Looking for video...
✅ Video extracted: https://video.twimg.com/ext_tw_video/...
📋 Suggested filename: Remotion_2013626968386765291_clip.mp4
📥 Downloading HLS video via ffmpeg...
⠋ Downloading HLS...
✅ HLS download completed

✅ Video saved to: ~/Downloads/Remotion_2013626968386765291_clip.mp4
```

## Limitations

- **Public tweets only**: Private or protected tweets cannot be extracted
- **Clipping requires ffmpeg**: `--from` and `--to` require ffmpeg for processing
- **Clipping time format**: Times must be in MM:SS format (e.g., `00:30`, not `0:30` or `30`)

- **Public tweets only**: Private or protected tweets cannot be extracted
- **Time-limited URLs**: Video URLs may expire after some time
- **Rate limiting**: X may rate-limit excessive requests
Expand Down
9 changes: 7 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "x-dl",
"version": "0.4.3",
"version": "0.4.4",
"description": "Extract videos from X/Twitter tweets",
"main": "./src/index.ts",
"type": "module",
Expand All @@ -24,7 +24,12 @@
"release:major": "bun version major && git push && git push --tags",
"version:tags": "git tag -l --sort=-version:refname"
},
"keywords": ["twitter", "x", "video", "extractor"],
"keywords": [
"twitter",
"x",
"video",
"extractor"
],
"author": "",
"license": "MIT",
"dependencies": {
Expand Down
204 changes: 200 additions & 4 deletions src/ffmpeg.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ export interface DownloadHlsOptions {
playlistUrl: string;
outputPath: string;
timeout?: number;
clipFromSecs?: number;
clipDurationSecs?: number;
}

export interface FfmpegCapabilities {
Expand All @@ -19,7 +21,7 @@ export interface FfmpegCapabilityCheckResult extends FfmpegCapabilities {
}

export async function downloadHlsWithFfmpeg(options: DownloadHlsOptions): Promise<string> {
const { playlistUrl, outputPath, timeout } = options;
const { playlistUrl, outputPath, timeout, clipFromSecs, clipDurationSecs } = options;

console.log('📥 Downloading HLS video via ffmpeg...');

Expand All @@ -31,19 +33,24 @@ export async function downloadHlsWithFfmpeg(options: DownloadHlsOptions): Promis
}

const timeoutMs = timeout ?? 120000;
const noProgressTimeoutMs = 30000;
const noProgressTimeoutMs = 60000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 No-progress timeout kills ffmpeg during slow HLS seek before encoding can start

When clipping an HLS stream with --from, the code uses slow/accurate seek (-ss after -i), which requires ffmpeg to download and decode the entire HLS stream up to the seek point before writing any output. During this seek phase, the output file either doesn't exist or contains only a tiny header and does not grow. The 60-second noProgressTimeoutMs poll at src/ffmpeg.ts:75-107 monitors output file size growth and will kill ffmpeg after 60 seconds of no growth, even though ffmpeg is actively downloading and decoding HLS segments.

Root Cause and Impact

The timeline for --from 05:00 on an HLS stream is:

  1. lastProgressTime initialized to Date.now() at line 38
  2. ffmpeg spawned — begins downloading HLS segments and decoding
  3. Output file either doesn't exist yet or has a tiny ftyp header (~32 bytes)
  4. 60 seconds later, poll at line 86/96 fires: now - lastProgressTime > noProgressTimeoutMs → ffmpeg killed with error "FFMPEG stuck: no progress for 60 seconds"

The previous 30-second timeout was fine because non-clipping HLS downloads use -c copy (stream copy), which writes output immediately. The PR increased it to 60s, but slow seek on HLS requires downloading the full input up to the seek point, which can take minutes for non-trivial --from values.

Impact: The HLS clipping feature is broken for any --from value where the HLS content before the seek point takes more than ~60 seconds to download and decode. For example, --from 05:00 would require downloading ~5 minutes of HLS segments before producing output, but ffmpeg is killed after 60 seconds.

Prompt for agents
In src/ffmpeg.ts, the noProgressTimeoutMs at line 36 needs to account for HLS slow seek. When clipping is active (clipFromSecs is defined), ffmpeg uses -ss after -i which means it must download and decode the entire stream up to the seek point before any output is written. The 60-second no-progress timeout kills ffmpeg before it can start encoding.

Possible fixes:
1. When isClipping is true (line 42), either disable the no-progress polling entirely, or scale the noProgressTimeoutMs based on clipFromSecs (e.g. clipFromSecs * 2 * 1000 + 60000) to give ffmpeg enough time to seek through the input.
2. Alternatively, monitor ffmpeg's stderr for progress information instead of polling output file size, since ffmpeg can report processing progress even during the seek phase.
3. As a simpler fix, set noProgressTimeoutMs to a much larger value (e.g. the full timeoutMs) when clipping is active, and rely solely on the overall timeout.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

let lastFileSize = 0;
let lastProgressTime = Date.now();
let rejected = false;

return new Promise((resolve, reject) => {
const isClipping = clipFromSecs !== undefined || clipDurationSecs !== undefined;
const args = [
'-y',
'-hide_banner',
'-loglevel', 'error',
'-i', playlistUrl,
'-c', 'copy',
'-bsf:a', 'aac_adtstoasc',
// Use -ss after -i (slow/accurate seek) — fast seek on HLS produces empty output
...(clipFromSecs !== undefined ? ['-ss', String(clipFromSecs)] : []),
...(clipDurationSecs !== undefined ? ['-t', String(clipDurationSecs)] : []),
...(isClipping
? ['-c:v', 'libx264', '-c:a', 'aac', '-movflags', '+faststart']
: ['-c', 'copy', '-bsf:a', 'aac_adtstoasc']),
outputPath,
];

Expand Down Expand Up @@ -136,6 +143,69 @@ export async function downloadHlsWithFfmpeg(options: DownloadHlsOptions): Promis
});
}

export interface ClipLocalFileOptions {
inputPath: string;
outputPath: string;
clipFrom?: string;
clipTo?: string;
}

export function mmssToSeconds(mmss: string): number {
const [mm, ss] = mmss.split(':').map(Number);
return mm * 60 + ss;
}

export async function clipLocalFile(options: ClipLocalFileOptions): Promise<void> {
const { inputPath, outputPath, clipFrom, clipTo } = options;

const startSecs = clipFrom ? mmssToSeconds(clipFrom) : 0;
const duration = clipTo ? mmssToSeconds(clipTo) - startSecs : undefined;

return new Promise((resolve, reject) => {
const args = [
'-y',
'-hide_banner',
'-loglevel', 'error',
...(clipFrom ? ['-ss', String(startSecs)] : []),
'-i', inputPath,
...(duration !== undefined ? ['-t', String(duration)] : []),
'-c:v', 'libx264', '-c:a', 'aac', '-movflags', '+faststart',
outputPath,
];

const ffmpeg = spawn('ffmpeg', args);
let stderr = '';

if (ffmpeg.stderr) {
ffmpeg.stderr.on('data', (data: Buffer) => { stderr += data.toString(); });
}

const spinnerChars = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
let spinnerIndex = 0;
const spinnerInterval = setInterval(() => {
process.stdout.write(`\r${spinnerChars[spinnerIndex]} Clipping...`);
spinnerIndex = (spinnerIndex + 1) % spinnerChars.length;
}, 80);

ffmpeg.on('close', (code: number | null) => {
clearInterval(spinnerInterval);
if (code === 0) {
process.stdout.write('\r✅ Clip complete\n');
resolve();
} else {
process.stdout.write('\r\x1b[K');
reject(new Error(stderr.trim() || `ffmpeg exited with code ${code ?? 'null (signal)'}`));
}
});

ffmpeg.on('error', (err: Error) => {
clearInterval(spinnerInterval);
process.stdout.write('\r\x1b[K');
reject(new Error(`Failed to start ffmpeg: ${err.message}`));
});
});
}

export async function checkFfmpegCapabilities(): Promise<FfmpegCapabilityCheckResult> {
const results: FfmpegCapabilityCheckResult = {
available: false,
Expand Down Expand Up @@ -199,6 +269,132 @@ export async function checkFfmpegCapabilities(): Promise<FfmpegCapabilityCheckRe
return results;
}

export interface DownloadMp4Options {
videoUrl: string;
outputPath: string;
clipFrom?: string;
clipTo?: string;
timeout?: number;
}

export async function downloadMp4WithFfmpeg(options: DownloadMp4Options): Promise<string> {
const { videoUrl, outputPath, clipFrom, clipTo, timeout } = options;

console.log('📥 Downloading video via ffmpeg...');

const fs = await import('node:fs');

if (fs.existsSync(outputPath)) {
console.log(`⚠️ File already exists, removing: ${outputPath}`);
fs.unlinkSync(outputPath);
}

const timeoutMs = timeout ?? 120000;
const noProgressTimeoutMs = 60000;
let lastFileSize = 0;
let lastProgressTime = Date.now();
let rejected = false;

const fromSecs = clipFrom ? mmssToSeconds(clipFrom) : 0;
const toSecs = clipTo ? mmssToSeconds(clipTo) : undefined;
const duration = toSecs !== undefined ? toSecs - fromSecs : undefined;

return new Promise((resolve, reject) => {
const args = [
'-y',
'-hide_banner',
'-loglevel', 'error',
...(clipFrom ? ['-ss', String(fromSecs)] : []),
'-i', videoUrl,
...(duration !== undefined ? ['-t', String(duration)] : []),
'-c:v', 'libx264', '-c:a', 'aac', '-movflags', '+faststart',
outputPath,
];

const ffmpeg = spawn('ffmpeg', args);

let stderr = '';

if (ffmpeg.stderr) {
ffmpeg.stderr.on('data', (data: Buffer) => {
stderr += data.toString();
});
}

const spinnerChars = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
let spinnerIndex = 0;

const spinnerInterval = setInterval(() => {
process.stdout.write(`\r${spinnerChars[spinnerIndex]} Downloading...`);
spinnerIndex = (spinnerIndex + 1) % spinnerChars.length;
}, 80);

const pollInterval = setInterval(() => {
try {
const now = Date.now();
if (fs.existsSync(outputPath)) {
const currentFileSize = fs.statSync(outputPath).size;
if (currentFileSize > lastFileSize) {
lastFileSize = currentFileSize;
lastProgressTime = now;
} else if (now - lastProgressTime > noProgressTimeoutMs) {
clearInterval(spinnerInterval);
clearInterval(pollInterval);
clearTimeout(timeoutHandle);
rejected = true;
ffmpeg.kill();
process.stdout.write('\r\x1b[K');
reject(new Error(`FFMPEG stuck: no progress for ${noProgressTimeoutMs / 1000} seconds`));
}
} else if (now - lastProgressTime > noProgressTimeoutMs) {
clearInterval(spinnerInterval);
clearInterval(pollInterval);
clearTimeout(timeoutHandle);
rejected = true;
ffmpeg.kill();
process.stdout.write('\r\x1b[K');
reject(new Error(`FFMPEG stuck: no progress for ${noProgressTimeoutMs / 1000} seconds`));
}
} catch (_err) {}
}, 2000);

const timeoutHandle = setTimeout(() => {
clearInterval(spinnerInterval);
clearInterval(pollInterval);
rejected = true;
ffmpeg.kill();
process.stdout.write('\r\x1b[K');
reject(new Error(`FFMPEG download timed out after ${timeoutMs / 1000} seconds`));
}, timeoutMs);

ffmpeg.on('close', (code: number | null) => {
clearInterval(spinnerInterval);
clearInterval(pollInterval);
clearTimeout(timeoutHandle);

if (code === 0 && !rejected) {
process.stdout.write('\r✅ Download completed\n');
resolve(outputPath);
} else if (!rejected) {
process.stdout.write('\r\x1b[K');
const error = stderr.trim() || `ffmpeg exited with code ${code ?? 'null (signal)'}`;
reject(new Error(`Failed to download: ${error}`));
}
});

ffmpeg.on('error', (err: Error) => {
clearInterval(spinnerInterval);
clearInterval(pollInterval);
clearTimeout(timeoutHandle);
if (!rejected) {
rejected = true;
process.stdout.write('\r\x1b[K');
reject(new Error(`Failed to start ffmpeg: ${err.message}`));
}
});
});
}

export function hasRequiredFfmpegCapabilities(capabilities: FfmpegCapabilities): { has: boolean; missing: string[] } {
const required = {
protocol: 'https',
Expand Down
Loading