Repository navigation
Korean text rendering: plan, eval, synth, AI Hub parser - #2
Open
its-janghoon wants to merge 83 commits into
Open
its-janghoon wants to merge 83 commits into
its-janghoon wants to merge 83 commits into
Conversation
Folders for configs, eval, infra, synth and paper data; the plan; a check that keeps AI Hub data and account details out of this public repository; and a script that creates and verifies the private bucket.
unseen_words.txt is frozen and filters every training source. unseen_syllables.txt holds 120 syllables outside KS X 1001, sampled with a fixed seed.
… coverage manifest
… backgrounds (507)
…ison, baseline eval script
Pairs images with labels of the same split and set (base vs 230216_add stems collide with different content). Drops unseen-word/syllable images, md5/dHash duplicates and unplaceable boxes. Crop windows are searched over size, trainer aspect buckets and position so text stays legible at 512 and no legible unlabeled text is shown. Report writes numbers only.
…rbatim The VLM sees only region positions, never the label text, and returns scene and per-region style. Captions (ko_long, ko_short, en) are composed in code from the AI Hub label so a VLM misread cannot reach them; scene strings that echo a label or quote fall back to a category phrase. Resumable; preview HTML stays on the box.
AI Hub ignores Range requests, so the fetcher restarts a file instead of resuming. The probe lists a dataset and checks approval on its smallest file. hf_relay.sh downloads Hugging Face files on a host with direct internet and copies them to the box over the tailnet while the box routes through the Korean exit node.
…k, cost guard precompute.py caches VAE latents and Qwen3-4B caption embeddings per aspect bucket. train_lora.py trains a LoRA on Z-Image Base from that cache with 30-min checkpoints, SIGTERM checkpointing and resume; eval_baseline.py gains --lora so a Base LoRA can be scored on Base or Turbo. train_full.py is the same loop with FSDP2 for p5 (not launched). Configs for g6, g6e, p5 and a smoke run; docs/path-b-runbook.md with the cost table.
…1725); source_image.ocr is fully labeled
… 0.339->0.295, unseen 0.344->0.302; pilot gate met
…d eval scripts/glyph.py renders the target text (Noto Sans KR Bold) into its box and feeds it to Z-Image omni mode as a clean condition image; no new parameters. precompute --glyph caches glyph latents and omni caption segments; train_lora glyph.enabled with per-batch dropout; eval_baseline --glyph uses a default scene layout. run_glyph_box.sh chains it after r64 v2.
paddleocr lives in a separate venv on the box, so the reaction check crashed after generating its images. The chain runs one GPU job at a time, so the main-venv qwen3vl engine fits.
qwen3vl guessed plausible words from the scene: it read a stroke-less glyph-reaction signboard as "약국". A text now counts as right only when both engines read it right, and the CER is the worse of the two. - eval/metrics.py: combine() - eval_baseline.py: --ocr dual (new default), per-engine lines/scores kept in scores.jsonl; paddle venv found by default on the gpu-box - rescore_dual.py: re-score finished runs, running only the missing engine - chain scripts use --ocr dual - glyph reaction check images (model output) and dual scores
Omni-mode glyph conditioning did not learn to copy glyph strokes (swap test at step 3141 and 5000: 0/10 exact under dual OCR, share following the glyph went down). Switch to adding the glyph latent to the same-position tokens through a zero-init input projection. - glyph.py / train_lora.py / eval_baseline.py: channel-concat path - configs/g6e-glyph-ch1.yaml, scripts/run_glyph_ch_box.sh - scripts/glyph_swap_box.sh: prompt-A / glyph-B swap test - paper/runs: swap results (3141, 5000), side-by-side vs r64v2
ch1 swap test (step 5245): 10/10 images drew the prompt text, 0/10 the glyph. Prompt and glyph always agreed in training, so the glyph was never needed. - scripts/precompute_mtxt.py: sidecar <shard>.mtxt.safetensors with quoted text -> "*" - train_lora.py: glyph.mask_caption share of glyph samples use the masked caption; glyph.init_proj carries the ch1 projection over - configs/g6e-glyph-ch2.yaml, scripts/run_glyph_ch2_box.sh
- ch2 swap at step 2650 (caption text kept): 10/10 drew prompt text A
- ch2 swap at step 3001 with masked caption ("*"): 0/10 drew glyph B,
gibberish instead (dual CER 0.98)
- scripts/glyph_swap_masked_box.sh: masked-caption swap test
- docs/render-harmonize-plan.md: font-render text, harmonize with
Z-Image inpaint, gate on dual OCR + human review
- ch2 stopped at step 3285
- scripts/render_harmonize.py: blank-surface scene from Base, Noto Sans KR text in the scene's default box (colour and weight from the prompt), inpaint over a band around the box with strokes masked out and pasted back pixel-exact; one eval folder per config (plain, b<pad>-s<strength>) - scripts/run_render_harmonize_box.sh: 20-prompt trial + dual-OCR scoring
- render_harmonize.py: VAE tiling + empty_cache (1024px inpaint decode ran out of GPU memory) - paper/runs/render-harmonize-trial: reports, scores, settings, 8-prompt sheet (images stay on the box) - best dual EM 0.40 (b16-s0.3, b48-s0.3); misses come from scene text the blank prompt did not stop and from OCR misreading clean font text
The first harmonize trial kept the font strokes as pasted pixels, so every
config looked pasted. This trial re-draws the strokes from the font composite:
- zr24-s<s>: Z-Image Base + r64 v2 LoRA inpaints the text box (+24 px),
strokes included, strength 0.35 / 0.5 / 0.65
- qwen: Qwen-Image-Edit-2511 (Apache-2.0) edits the whole composite,
transformer weights stored in fp8 to fit the 46 GB card
score_crops.py scores the text box only with dual OCR, so text the scene
model drew by itself no longer counts against the pasted word.
Box-only dual EM (20 prompts): plain 0.30, b16-s0.3 0.35, zr24-s0.35 0.40,
zr24-s0.5 0.25, zr24-s0.65 0.10, qwen 0.20. Qwen changes letters
(커피 -> 머머, 주차장 -> 수라장). Sheets for the human naturalness check:
paper/runs/rerender-trial/sheet-{full,zoom}.png.
docs/reward-training-plan.md: plan C, region-masked DPO then GRPO with a
three-level reward, gated on a jamo-swap reward check.
make_jamo_pairs.py draws the target and one-jamo corruptions (vowel, tense, initial, final) into the same blank scene with the font and harmonizes them the plan-B way, so right and wrong images differ only in that stroke. reward.py scores the text box: per-syllable Qwen3-VL yes/no against a font reference (P(yes) from logits), dual OCR word match, and extra-text penalty; --check ranks the pairs and passes at >= 95%.
The first inpaint decode ran out of the 44 GB card right after a scene was generated; empty the cache there and use expandable segments in the chain.
498 jamo-swap pairs on font-composited scenes. Reward picks the correct image in 78.5% overall; vowel swaps 66.5%, unseen-syllable words 58.8%. Adds check.json, a sheet of 4 failing pairs, and the script that builds the sheet.
Adds two layers read by Qwen3-VL with teacher forcing: log P(target) for the whole text crop, and log P(syllable) for each syllable cell alone, which removes the word prior that hid wrong vowels. Total = syl_mean + 2 * lik. Rescored the existing 498 pairs only. Still below the 0.95 gate; weights were tuned on the same pairs. Results in paper/runs/reward-check/README.md.
ResNet-18 over all 11,172 syllables, trained on 34 OFL fonts with photo backgrounds and edge clutter (scripts/syllable_cls.py, fetch_ko_fonts.sh, run_syl_cls_box.sh). Score = mean log P(target syllable) per cell, declared before the run; Noto Sans KR (the pairs' font) held out. v6 (clean backgrounds) 0.954, reads 39% of right images exactly. v7 (+ photo patches) 0.998, reads 89%; on the 60 prompts not used to pick the v7 changes: 1.000 and 83%.
eval/make_train_prompts.py builds 2,000 training prompts (four length buckets, 500 each) disjoint from the eval set; train_prompts_c1.jsonl is its output. run_c1_rollout_box.sh draws 4 seeds per prompt with the r64 v2 LoRA on Base and scores them with reward_cls.py (syllable classifier per PaddleOCR cell, minus extra cells). Smoke N=200 (800 images): dual EM 0.31; reward spread > 0.1 inside 194 of 200 groups, so best/worst pairs exist for DPO. Human check sheet for r64 v2 images is in paper/runs/human-check-r64v2 (labels pending).
One page per experiment with the dual-OCR numbers from paper/runs, the negative results (glyph conditioning, render-and-harmonize), the reward check from v1 to the syllable classifier, and cautions for the paper (40-prompt noise, short-word scoring bias, PaddleOCR-only runs not rescored, no ablation for r64 v2). AI usage log updated.
The paper notes live in the open-papers repo (015-korean-text-rendering), which copies this repo's paper/ results at d402765. This reverts the summary and its AI-usage row here, and masks the GPU box host name that was still in four glyph swap reports.
dpo_pairs.py: best vs worst classifier reward per rollout prompt (min gap 2.0), VAE latents of both images, one latent mask over every line box either image has, the prompt's caption embedding; 1 prompt in 20 held out. On the 200-prompt smoke rollout: 176 pairs, median gap 7.7. train_dpo.py: Diffusion-DPO in the flow-matching form of train_lora.py. Winner and loser share noise and sigma; errors are masked means with text cells at 5x; the reference is a frozen second adapter holding the same r64 v2 weights the trained one starts from. Held-out pref_acc / mean d every 50 steps. configs/g6e-dpo-c1.yaml: beta 2000, lr 1e-5, 8 pairs per step, 2 epochs. run_dpo_box.sh: waits for the 2,000-prompt rollout, builds pairs, runs a 2-step smoke, then waits for ~/runs/dpo-go (human check of the reward on r64 v2 images) before training and the dual-OCR + classifier final eval. Checked on CPU only (GPU busy with the rollout): both files compile, the pair picker on the smoke rewards, and two-adapter switching in peft.
Janghoon Lee marked 7 of the 20 sheet images right (1, 2, 4, 5, 9, 11, 18). The classifier verdict (every target syllable aligned to a cell read as that syllable) agrees on 18 of 20 and never calls a human-wrong image right; it misses 빵집 (alignment lost among 15 extra cells) and 신선한 (read 신션한). The reward alone separates the groups with AUC 0.714, because it also subtracts extra cells and two human-right images carry 14-15 of them. Dual OCR exact match agrees with the human on 19 of 20. scripts/human_check_compare.py makes the comparison.
Held-out pref_acc 0.72 -> 0.86, but the 220-prompt dual-OCR eval is within noise of r64 v2 (CER 0.284 vs 0.281, EM 0.295 vs 0.305) and the classifier reward on fresh samples is slightly lower. Images not committed.
Same 1,746 pairs as c1. run_dpo_box.sh takes CFG/RUN from env and draws the same 8 peek prompts after the final eval.
Stronger DPO (beta 500, lr 5e-5, 4 epochs) raised the classifier reward (-7.25 -> -5.25) mostly by cutting extra text cells (5.53 -> 2.41), but dual-OCR target accuracy got worse vs r64 v2 (CER 0.281 -> 0.314, EM 0.305 -> 0.259). Images not committed; host masked.
Plan C (reward DPO) dropped after c2. Score other open base models on the same 220-prompt eval with dual OCR and the syllable-classifier reward, no training. Qwen-Image-2512 first (Apache-2.0, ungated): fp8 weight storage on the GPU like rerender.py, true CFG 4, 50 steps, 8-prompt smoke first.
dual CER 0.459 (r64 v2 0.281), EM 12.7%, unseen EM 10%. Worse than Z-Image Base; host masked, images not committed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's done
What's not done
What to review
Intentions
Build the data and eval groundwork so the pilot LoRA can start as soon as AWS access lands, without any AI Hub data or private host detail entering git.