Convert Word (.docx) documents into JSONL suitable for chat‑style fine‑tuning (e.g., OpenAI format), with a Windows‑friendly GUI (Tkinter) and a CLI.
Pages are processed one‑by‑one through GPT to clean OCR noise, split/condense into Q&A pairs, and emitted as streaming JSONL lines.
A topic tag is injected as a system message (not echoed), which helps separate domains during training and reduces cross‑topic interference.
- DOCX → JSONL using
python-docx(optimized for PDF→Word conversions). - Prompt‑driven cleanup & structuring via GPT (editable JSON template).
- System tag injection:
{"role":"system","content":"This conversation is about [TAG]."}
➜ Tag isn’t echoed in answers, but conditions the conversation. - Queue + rate control: set Requests‑Per‑Minute (RPM) and Max inflight concurrency.
- File‑cache of raw (untagged) model outputs to avoid reprocessing the same pages.
- Streaming writer: safely appends one JSON object per line to
.jsonl. - GUI (Tkinter) & CLI (Windows‑first, same convention as your other repos).
- Logging to
./logs/…with rotating files.
- Python 3.10+
- Git (optional, for versioning)
- An OpenAI API key
cd \path\to\word-to-lora
-
Create & activate venv (same as your other projects) py -m venv .venv ..venv\Scripts\activate
-
Install dependencies py -m pip install --upgrade pip pip install -r requirements.txt
-
Create .env (repo root) Only the API key lives here. OPENAI_API_KEY=sk-your-real-key-here .env is in .gitignore so secrets aren’t committed.
Run It Option A — CLI (fastest for a quick test)
py -m src.cli --in tests\data\sample.docx --out example_output\out.jsonl --tag [Pro Tools] Common flags: --in (required): path to .docx --out (required): path to output .jsonl --prompt (default prompts\page_prompt.json) --rpm requests per minute throttle (default 60) --max-inflight concurrent requests (default 5) --tag optional topic tag (free‑form; normalized to [PROTOOLS]‑style and injected as system message)
Option B — GUI (Tkinter)
py -m src.app In the window:
Input .docx → pick your file Output .jsonl → choose a path Prompt JSON → leave default unless you customized Tag (optional) → e.g., Pro Tools or [Pro Tools] (auto‑normalized) RPM / Max inflight → tune speed/rate limits
Start
Output Format Each line in the JSONL is a single training example with a messages array. The tag is injected as the first system message:
{ "messages": [ {"role": "system", "content": "This conversation is about [PROTOOLS]."}, {"role": "user", "content": "What is the best online source for information about Avid systems?"}, {"role": "assistant", "content": "The Avid website (www.avid.com) is your best online source for information to help you get the most out of your Avid system."} ] } The app does not prefix the tag on user or assistant messages—only the system message. If your old cache contains tagged content, the pipeline strips the leading normalized tag to keep outputs clean.
RPM & Concurrency (What they mean) RPM (Requests Per Minute): global throttle. With RPM = 60, the app sends ≤ 60 requests/min (~1/s).
Max inflight: how many requests run in parallel. With 5 inflight, up to 5 pages are processed at once.
They work together. Effective throughput is roughly: min( RPM , (60 / avg_request_time_seconds) * MaxInflight ) Start with RPM 60, Inflight 5–8. Reduce if you hit rate limits; raise cautiously if stable.
Configuration API key: from .env → OPENAI_API_KEY
(Optional) custom endpoint: set OPENAI_BASE_URL as a standard env var (not required in .env) Model default in src/config.py: gpt-4o-mini (change if you like)
Prompt Template Edit prompts/page_prompt.json. It contains:
system: role instructions (dataset formatter) user_instructions: bullet‑pointed rules (clean OCR, produce messages, return JSON only) optional few‑shot examples The app sends the page text plus your instructions and requests JSON‑only output.
Caching & Re‑runs Cache path: .\cache\
Keyed by document hash + page index + page content hash. Stores raw, untagged model output (so you can re‑emit with different tags without paying tokens again). Safe to delete for a clean rebuild.
Logs Written to .\logs\run_YYYYMMDD-HHMMSS.log Includes page start/finish, retries, and errors.
Training Tips (multi‑topic, avoid forgetting) Mix topics in your training set; don’t over‑represent a single domain. Use smaller learning rates / fewer epochs for incremental updates.
Keep the system tag consistent across examples of a topic, and include the same system message at inference when you want that behavior.
Troubleshooting Auth error / missing key Ensure .env exists with a valid OPENAI_API_KEY, and your venv is active.
Rate limit errors Lower --rpm and/or --max-inflight.
Schema check failed Your prompt likely produced non‑JSON or a different structure. Revisit prompts\page_prompt.json. The app expects exactly:
{"messages":[{"role":"user","content":"..."},{"role":"assistant","content":"..."} ...]} Weird “pages” DOCX doesn’t natively store page layout; this app detects explicit page breaks from PDF→Word conversions. If a file lacks breaks, the parser falls back to chunking by size.
Development Run lint/tests as desired (placeholders exist under tests/).
Packaging is minimal; you run modules directly:
py -m src.cli ... py -m src.app If you later want a pip‑installable package, you can switch to a src/word_to_lora/ package layout and update setup.py accordingly.
License MIT © 2025 Tiān Jié Héng Feel free to fork, and fuck off! :)