feat(scripts): discord-dump.py — channel/thread→jsonl exporter - #226
Closed
alex-nsheaps[bot] wants to merge 5 commits into
Closed
alex-nsheaps[bot] wants to merge 5 commits into
alex-nsheaps[bot] wants to merge 5 commits into
Conversation
…orter Stdlib-only Python script that dumps Discord messages to JSON Lines. CLI / env-var inputs: - --guild/-g (or DISCORD_GUILD_ID) — required - --channel/-c (or DISCORD_CHANNEL_ID) — required unless --all - --thread/-t (or DISCORD_THREAD_ID) — required when --channel is forum and --forum / --with-threads / --thread is not specified - --all — dump every textual channel in the guild - --with-threads / --forum — also dump every active + archived-public thread (one jsonl per thread) - --start / --end — UTC date or ISO8601 filter - --user / --exclude-user — author-id allow/deny (repeatable) - --token-env (default DISCORD_BOT_TOKEN) Resilience: - Full pagination via the \`before=\` cursor (limit 100) - Honors 429 with retry_after from body or Retry-After header - Proactive sleep when X-RateLimit-Remaining: 0 - Exponential backoff + jitter for 5xx / network / timeout (cap 60s, --max-retries 8) Output is one raw Message JSON per line — embeds, attachments, reactions, and components preserved verbatim for downstream tooling. Tested by dumping #behavior channel: 5748 messages in one run, 0 lost messages despite dozens of 429 hits.
alex-nsheaps
Bot
added
the
request-review
Discord's GET /channels/{id}/messages does NOT return a thread's starter
message (the post body with msg-id == thread-id). When the target channel
is a thread (type 10/11/12), fetch it explicitly via
GET /channels/{id}/messages/{id} and append after the pagination walk.
Verified against thread 1497431286661517353 (forum thread, type 11):
5752 messages now include the starter (was 5751 from /messages walk only).
After each dump, query GET /channels/{id}/messages?before=<min_id>&limit=100.
If the API returns 0 messages older than the dump's min id, the dump is
provably complete from the start. For threads, also verify the starter id
(== thread id) is present. Exits non-zero on FAIL.
Tested against #behavior (152 msgs): PASS — oldest msg
1490890554513555559 @ 2026-04-07T01:46:47Z, no older messages, starter
present.
Discord's GET /channels/{id}/messages returns newest-first 100 at a time;
the previous implementation wrote per-page chunks reversed to oldest→newest
WITHIN each page, so the file's first line was a recent message and the
true oldest message was buried mid-file. That's not display order.
Accumulate all fetched messages (+ optional thread starter) in memory and
sort by snowflake id before writing. Output is now oldest-at-top end-to-end,
matching how Discord displays the channel.
For typical channels this is fine — Discord caps channels at single-digit
millions of messages and this script is for ad-hoc dumps; streaming-sort
isn't needed yet.
3 tasks
Contributor
Author
|
Closing — moved into ai-mktpl/plugins/discord per Nate Discord 17:45Z. New PR: nsheaps/ai-mktpl#588. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New
scripts/discord-dump.py: stdlib-only Python script for dumping a Discord channel/thread (or whole guild) to JSON Lines with full pagination, rate-limit handling, retry on transient failures, and (for threads) explicit fetch of the starter message that Discord doesn't return fromGET /channels/{id}/messages.Per Nate's Discord ask.
What it does
--guild/--channel/--thread(also env:DISCORD_GUILD_ID/DISCORD_CHANNEL_ID/DISCORD_THREAD_ID)--allfor every textual channel in the guild--with-threads/--forumto also dump active + archived-public threads--start/--endUTC date filter (YYYY-MM-DDor ISO8601)--user/--exclude-userauthor-id allow/deny (repeatable)$DISCORD_BOT_TOKEN(override with--token-env)Resilience
before=cursor (limit 100)429from both bodyretry_afterandRetry-AfterheaderX-RateLimit-Remaining: 0(usesX-RateLimit-Reset-After)--max-retries 8)GET /channels/{id}/messagesdoes NOT include the post-body message whose id equals the thread id. When target is a thread (type 10/11/12), script fetches/channels/{id}/messages/{id}explicitly and appends.Output
One raw Message JSON per line — embeds/attachments/reactions/components preserved verbatim.
Test
Dumped two channels end-to-end (see alex PR #34):
1490890535878131792, type 0): 152 messages.1497431286661517353, type 11): 5752 messages, including the starter.Commits
8f9ce7finitial script + README050e255fix: fetch thread-starter message explicitlyTest plan
--helprenders--with-threadsdiscovers active + archived threads--forumerrors helpfully when used on non-forum channel--start/--endfiltering correctness$DISCORD_BOT_TOKENand from custom--token-env