Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
<img width="400" alt="logo" src="https://github.com/user-attachments/assets/ea105e9e-00d0-4d48-a2a4-006cc4e89848" />

[![CLI](https://img.shields.io/github/v/release/superdoc-dev/docx-corpus?filter=cli-v*&label=cli)](https://github.com/superdoc-dev/docx-corpus/releases)
[![CDX Filter](https://img.shields.io/github/v/release/superdoc-dev/docx-corpus?filter=cdx-filter-v*&label=cdx-filter)](https://github.com/superdoc-dev/docx-corpus/releases)
[![codecov](https://codecov.io/gh/superdoc-dev/docx-corpus/graph/badge.svg)](https://codecov.io/gh/superdoc-dev/docx-corpus)
[![CLI](https://img.shields.io/github/v/release/superdoc/docx-corpus?filter=cli-v*&label=cli)](https://github.com/superdoc/docx-corpus/releases)
[![CDX Filter](https://img.shields.io/github/v/release/superdoc/docx-corpus?filter=cdx-filter-v*&label=cdx-filter)](https://github.com/superdoc/docx-corpus/releases)
[![codecov](https://codecov.io/gh/superdoc/docx-corpus/graph/badge.svg)](https://codecov.io/gh/superdoc/docx-corpus)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

The largest open corpus of classified Word documents. 736K+ real `.docx` files from the public web, classified into 10 document types and 9 topics across 46+ languages.
Expand All @@ -12,7 +12,7 @@ The entire document AI research ecosystem runs on scanned images and PDFs. The f
**[docxcorp.us](https://docxcorp.us)** · **[HuggingFace](https://huggingface.co/datasets/superdoc-dev/docx-corpus)** · **[API](https://api.docxcorp.us/stats)**

> [!IMPORTANT]
> **WARC trailing bytes fixed for new scrapes** — Documents scraped before [`477d1b9`](https://github.com/superdoc-dev/docx-corpus/commit/477d1b9) may contain a 4-byte WARC record terminator (`\r\n\r\n`) after ZIP EOCD. Future scrapes are fixed; use `python3 scripts/fix_trailing_warc_bytes.py` to inspect or repair older local copies. This was confirmed to trigger Word for Mac's unreadable-content warning for the sample in [#8](https://github.com/superdoc-dev/docx-corpus/issues/8).
> **WARC trailing bytes fixed for new scrapes** — Documents scraped before [`477d1b9`](https://github.com/superdoc/docx-corpus/commit/477d1b9) may contain a 4-byte WARC record terminator (`\r\n\r\n`) after ZIP EOCD. Future scrapes are fixed; use `python3 scripts/fix_trailing_warc_bytes.py` to inspect or repair older local copies. This was confirmed to trigger Word for Mac's unreadable-content warning for the sample in [#8](https://github.com/superdoc/docx-corpus/issues/8).

## How It Works

Expand All @@ -33,7 +33,7 @@ Common Crawl (3B+ URLs/month)
## Quick Start

```bash
git clone https://github.com/superdoc-dev/docx-corpus.git
git clone https://github.com/superdoc/docx-corpus.git
cd docx-corpus
bun install
```
Expand Down
4 changes: 2 additions & 2 deletions apps/site/content/classification.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ secondaryQueries:

docx-corpus uses a two-dimensional taxonomy. Every classified document has one document_type label (its form/structure) and one document_topic label (its subject domain). The labels are independent; a `legal` document can have any topic, and a `government` topic can have any document_type.

Taxonomy version: `docx-corpus-v2` (2.0.0). Source: [scripts/classification/taxonomy.json](https://github.com/superdoc-dev/docx-corpus/blob/main/scripts/classification/taxonomy.json).
Taxonomy version: `docx-corpus-v2` (2.0.0). Source: [scripts/classification/taxonomy.json](https://github.com/superdoc/docx-corpus/blob/main/scripts/classification/taxonomy.json).

## Document types (10)

Expand Down Expand Up @@ -114,7 +114,7 @@ See [/quality](/quality) for confidence distributions and the rest of the datase

## Reproducing this

Code in [scripts/classification/](https://github.com/superdoc-dev/docx-corpus/tree/main/scripts/classification). Run order: `sample.py` -> `label.py` -> `train.py` -> `classify.py`. Pipeline notes in [CLAUDE.md](https://github.com/superdoc-dev/docx-corpus/blob/main/scripts/classification/CLAUDE.md).
Code in [scripts/classification/](https://github.com/superdoc/docx-corpus/tree/main/scripts/classification). Run order: `sample.py` -> `label.py` -> `train.py` -> `classify.py`. Pipeline notes in [CLAUDE.md](https://github.com/superdoc/docx-corpus/blob/main/scripts/classification/CLAUDE.md).

---

Expand Down
2 changes: 1 addition & 1 deletion apps/site/content/quality.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ This page is for researchers deciding whether docx-corpus is fit for their task.

## Validation

Every file that enters the corpus passes these checks (in [`packages/scraper/validation.ts`](https://github.com/superdoc-dev/docx-corpus/blob/main/packages/scraper/validation.ts)):
Every file that enters the corpus passes these checks (in [`packages/scraper/validation.ts`](https://github.com/superdoc/docx-corpus/blob/main/packages/scraper/validation.ts)):

1. Minimum size: at least 100 bytes.
2. ZIP magic bytes: starts with `PK\x03\x04`.
Expand Down
2 changes: 1 addition & 1 deletion apps/site/src/pages/classification.astro
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ const description = 'Two-dimensional taxonomy of 10 document types and 9 topics.
<section class="prose">
<div class="section-label">How labels are produced</div>
<p>We follow the <a href="https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1">FineWeb-Edu</a> pattern: sample a small, stratified set of documents across the five most-common languages, label each with an LLM, then train a lightweight multilingual text classifier on those labels and apply it at scale.</p>
<p>The classifier is a fine-tuned XLM-RoBERTa, trained on roughly 3,500 LLM-labeled examples. Two independent classifiers are trained, one per dimension. Code is open: see the <a href="https://github.com/superdoc-dev/docx-corpus/tree/main/scripts/classification">classification scripts</a> on GitHub.</p>
<p>The classifier is a fine-tuned XLM-RoBERTa, trained on roughly 3,500 LLM-labeled examples. Two independent classifiers are trained, one per dimension. Code is open: see the <a href="https://github.com/superdoc/docx-corpus/tree/main/scripts/classification">classification scripts</a> on GitHub.</p>
</section>

<section class="prose">
Expand Down
2 changes: 1 addition & 1 deletion apps/site/src/pages/index.astro
Original file line number Diff line number Diff line change
Expand Up @@ -863,7 +863,7 @@ const structuredData = {
<div class="section-label">About the dataset</div>
<div class="about-prose">
<p>docx-corpus is the largest open corpus of classified Word documents on the public web. It contains 736,000+ real .docx files collected from Common Crawl, validated, deduplicated, and labeled by document type and topic. It is the first large-scale research dataset for DOCX, the world's most-used document creation format, which has historically been absent from document AI work built on scanned images and PDFs.</p>
<p>Each document has a content hash, file size, detected language, and a type and topic label from a fine-tuned XLM-RoBERTa text classifier. The corpus spans 46+ languages and uses a 10-type, 9-topic taxonomy. Available as a <a href="https://huggingface.co/datasets/superdoc-dev/docx-corpus">HuggingFace dataset</a>, a REST API at <a href="https://api.docxcorp.us/stats">api.docxcorp.us</a>, and downloadable manifest files. Built by <a href="https://superdoc.dev/?utm_source=docxcorp.us&amp;utm_medium=referral&amp;utm_campaign=about">SuperDoc</a>. Source on <a href="https://github.com/superdoc-dev/docx-corpus">GitHub</a>. Dataset is <a href="https://opendatacommons.org/licenses/by/1-0/">ODC-BY</a>; source code is MIT.</p>
<p>Each document has a content hash, file size, detected language, and a type and topic label from a fine-tuned XLM-RoBERTa text classifier. The corpus spans 46+ languages and uses a 10-type, 9-topic taxonomy. Available as a <a href="https://huggingface.co/datasets/superdoc-dev/docx-corpus">HuggingFace dataset</a>, a REST API at <a href="https://api.docxcorp.us/stats">api.docxcorp.us</a>, and downloadable manifest files. Built by <a href="https://superdoc.dev/?utm_source=docxcorp.us&amp;utm_medium=referral&amp;utm_campaign=about">SuperDoc</a>. Source on <a href="https://github.com/superdoc/docx-corpus">GitHub</a>. Dataset is <a href="https://opendatacommons.org/licenses/by/1-0/">ODC-BY</a>; source code is MIT.</p>
</div>
<div class="about-browse">
<div>
Expand Down
2 changes: 1 addition & 1 deletion apps/site/src/pages/llms.txt.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ ${topics.map(t => `- [${t.label}](${absoluteUrl(routes.topic(t.id))}): ${t.count
## Links

- Homepage: ${absoluteUrl(routes.home())}
- GitHub: https://github.com/superdoc-dev/docx-corpus
- GitHub: https://github.com/superdoc/docx-corpus
- HuggingFace: https://huggingface.co/datasets/superdoc-dev/docx-corpus
- API: https://api.docxcorp.us
- Takedown requests: help@docxcorp.us
Expand Down
2 changes: 1 addition & 1 deletion packages/scraper/commoncrawl/warc.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import type { RateLimiter } from "../rate-limiter";
import type { CdxRecord } from "./cdx-r2";

const WARC_BASE_URL = "https://data.commoncrawl.org";
const USER_AGENT = "docx-corpus (https://github.com/superdoc-dev/docx-corpus)";
const USER_AGENT = "docx-corpus (https://github.com/superdoc/docx-corpus)";

export interface WarcResult {
content: Uint8Array;
Expand Down
2 changes: 1 addition & 1 deletion scripts/export-hf.py
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@
## Links

- **Website**: [docxcorp.us](https://docxcorp.us)
- **GitHub**: [superdoc-dev/docx-corpus](https://github.com/superdoc-dev/docx-corpus)
- **GitHub**: [superdoc/docx-corpus](https://github.com/superdoc/docx-corpus)
- **Built by**: 🦋 [SuperDoc](https://superdoc.dev)
"""

Expand Down
Loading