面向 Discourse 论坛的数据挖掘工具包：抓取 → 清洗 → 分类 → 评分 → 分析，全流程参数化可复用。
实战出身：2026 年 7 月 TRAE AI 创造力大赛论坛，抓取并分析了 14,139 个参赛项目。
合规：仅使用 Discourse 官方公开 API，默认限速 1.0s（对齐论坛 60 次/分钟配额），只处理公开内容。
I needed to analyze 14,000+ projects submitted to a Discourse-based forum. Existing tools either:
- Only call APIs (discourse_api) but don't clean or analyze data
- Are generic web scrapers (Firecrawl) that don't understand Discourse's HTML structure
- Require manual work for every new forum
So I built this. It encodes 6 iterations of classification refinement and 3 iterations of scoring formula redesign into reusable, parameterized code.
Forum URL → Crawl → Clean HTML → Classify → Score → Analyze → Report
| Module | What it handles | Key lessons encoded |
|---|---|---|
| Crawler | Discourse API (list + detail), 503 retry, incremental crawl | List API truncates fields; must fetch detail per topic |
| HTML Cleaner | Strip tags, extract code blocks, images, links | Discourse HTML has quotes, oneboxes, mentions that need special handling |
| Link Classifier | Classify URLs as demo/source/doc/social/invalid | localhost links, Punycode domains, Vercel needs VPN in China |
| Keyword Classifier | 18 categories, 89 subcategories, 3-layer mechanism | Generic keywords like "interaction" caused 39.7% misclassification |
| Scoring | 3-layer filtering: signal → value → recommendation | Mechanical formulas (views×0.01 + likes×2) can't replace human judgment |
| Stats | Category distribution, engagement, demo availability, creator stats | 82% of creators submitted only 1 project |
# Install
pip install requests
# Crawl a Discourse forum
python -m src.main crawl --url https://forum.example.com --category 40 --output data/raw.json
# Clean and structure
python -m src.main clean --input data/raw.json --output data/cleaned.json
# Classify by category
python -m src.main classify --input data/cleaned.json --output data/classified.json
# Score and filter
python -m src.main score --input data/classified.json --output data/scored.json --top 20
# Generate statistics report
python -m src.main analyze --input data/scored.json --output report.md --format markdown
# Or run the full pipeline at once
python -m src.main pipeline --url https://forum.example.com --category 40 --output-dir output/This is the core innovation. Generic keyword matching fails badly on forum data. Here's why and how we fixed it:
Layer 1 - Title-first matching: If a high-specificity keyword appears in the title, classify immediately. Titles have the strongest signal.
Layer 2 - Precision keyword matching: Scan title + body using ONLY high-specificity compound terms. For Gaming, use "Unity3D", "game engine", "RPG", "level design" — never just "game" or "interactive".
Layer 3 - Dual negative filtering: If a negative keyword appears in BOTH title and body, reduce confidence by 70%. This catches edge cases where a project mentions a category but doesn't belong to it.
The lesson: Keyword specificity > keyword coverage. It's better to leave 10% unclassified than to misclassify 40%.
Layer 1 - Signal scoring (0-10): Content quality (40%) + engagement (30%) + demo availability (20%) + creator signals (10%). Content quality is assessed by checking description clarity, feature specificity, technical depth, and use case clarity.
Layer 2 - Value classification: A (product worth learning), B (demand worth entering), C (creative worth reusing).
Layer 3 - Human review: The tool narrows 14,000 items to a few hundred. A human reviews the top candidates.
The lesson: Automated scoring can narrow scope but cannot replace human judgment. The goal is to reduce manual work, not eliminate it.
discourse-miner/
├── src/
│ ├── crawler.py # Discourse API crawler
│ ├── html_cleaner.py # HTML parsing and cleaning
│ ├── link_classifier.py # URL classification
│ ├── main.py # CLI entry point
│ ├── classifiers/
│ │ └── keyword_classifier.py # 3-layer classification
│ └── analyzers/
│ ├── scoring.py # 3-layer scoring
│ └── stats.py # Statistics generation
├── examples/
│ └── full_pipeline.py # Complete usage example
├── config/
│ └── categories.json # Custom category definitions
├── docs/
│ └── lessons_learned.md # Detailed pitfall documentation
├── requirements.txt
├── LICENSE
└── README.md
- Community managers who want to understand what their forum is talking about
- Competition organizers who need to review hundreds of submissions
- Researchers studying online community dynamics
- Content creators looking for data-driven story angles
我需要分析一个基于 Discourse 的论坛上 14,000+ 个提交项目。现有的工具要么只调 API 不管清洗分析，要么是通用爬虫不懂 Discourse 的数据结构。
这个工具编码了6轮分类迭代和3轮评分公式重设计的经验，把踩过的坑变成可复用的代码。
| 模块 | 解决什么问题 | 踩过的坑 |
|---|---|---|
| 爬虫 | Discourse API 两层抓取（列表+详情），503重试，增量爬取 | 列表API字段截断，必须逐条请求详情 |
| HTML清洗 | 去标签、提取代码块、图片、链接 | Discourse的引用、onebox、@提及需要特殊处理 |
| 链接分类 | 把URL分为Demo/源码/文档/社交/无效 | localhost链接、Punycode域名、Vercel国内需VPN |
| 关键词分类 | 18个一级分类、89个二级分类、三重分类机制 | 泛化词"互动""交互"导致39.7%误判 |
| 评分筛选 | 三层筛选：信号分→价值分类→人工实测 | 机械公式（浏览量×0.01+点赞×2）无法替代人工判断 |
| 统计分析 | 分类分布、互动数据、Demo可用率、创作者画像 | 82%的创作者只提交了1个项目 |
# 安装依赖
pip install requests
# 爬取论坛数据
python -m src.main crawl --url https://forum.example.com --category 40 --output data/raw.json
# 清洗结构化
python -m src.main clean --input data/raw.json --output data/cleaned.json
# 分类
python -m src.main classify --input data/cleaned.json --output data/classified.json
# 评分筛选
python -m src.main score --input data/classified.json --output data/scored.json --top 20
# 生成统计报告
python -m src.main analyze --input data/scored.json --output report.md --format markdown
# 一键全流程
python -m src.main pipeline --url https://forum.example.com --category 40 --output-dir output/第一层 - 标题优先匹配：高特异性关键词出现在标题中，直接分类。标题信号最强。
第二层 - 精准关键词匹配：只用高特异性复合词。游戏赛道用"Unity3D""游戏引擎""RPG""关卡设计"，不用"游戏""互动"。
第三层 - 双重负向过滤：负向关键词同时出现在标题和正文中，置信度降70%。
核心教训：关键词特异性比覆盖面更重要。宁可漏掉10%，也不要误判40%。
第一层 - 信号分（0-10）：内容质量40% + 互动数据30% + Demo可用性20% + 创作者信号10%
第二层 - 价值分类：A类（产品好可学习）、B类（需求对可入场）、C类（创意新可复用）
第三层 - 人工实测：工具把14,000个项目缩小到几百个，人工体验这几个
核心教训：自动化评分能缩小范围，但不能替代人工判断。
用这个工具分析了 TRAE AI创造力大赛的 14,139 个项目：
- 爬取耗时约60分钟（1.8条/秒）
- 分类准确率从 v1 的约60%提升到 v6 的95%+
- 从14,139个项目中筛出2000个高信号项目
- 最终人工实测约100个Demo
- 产出8份分析文档，发现公考AI是最大蓝海赛道
- 社区管理员：了解论坛用户在讨论什么
- 竞赛组织者：批量审核数百个提交项目
- 研究人员：研究在线社区动态
- 内容创作者：用数据驱动找选题角度
- 仅抓取公开可访问的数据（
/latest.json、/c.json、/t/<id>.json等官方公开端点），不模拟登录、不绕过验证码或频率限制。 - 使用前请阅读并遵守目标论坛的服务条款与
robots.txt。 - 默认请求间隔 1.0 秒（对齐 Discourse 默认每分钟 60 次限制），429/503 时指数退避并遵循
Retry-After。 --api-key仅用于已获论坛授权的私有论坛场景，请勿使用他人凭据。- 抓取结果中的用户名等公开元数据仅用于统计与分析，不要用于骚扰、人肉或任何违背目标平台规则的行为。
- 本工具只处理"公开可见"的内容；是否合规取决于使用场景，请自行确认。
MIT License - see LICENSE
Issues and PRs welcome. If you've crawled a Discourse forum and hit a problem not covered here, your experience will help improve the tool.